Linear discriminant analysis: Difference between revisions
No edit summary Tag: Reverted |
m →LDA for two classes: training as part of supervised learning |
||
(43 intermediate revisions by 28 users not shown) | |||
Line 1: | Line 1: | ||
{{ |
{{Short description|Method used in statistics, pattern recognition, and other fields}} |
||
{{Distinguish|latent Dirichlet allocation}} |
{{Distinguish|latent Dirichlet allocation}} |
||
[[File:Linear discriminant analysis plot.png|thumb|Linear discriminant analysis on a two dimensional space with two classes. The Bayes boundary is calculated based on the true data generation parameters, the estimated boundary on the realised data points.<ref>{{Cite web |last=Holtel |first=Frederik |date=2023-02-20 |title=Linear Discriminant Analysis (LDA) Can Be So Easy |url=https://towardsdatascience.com/linear-discriminant-analysis-lda-can-be-so-easy-b3f46e32f982 |access-date=2024-05-18 |website=Medium |language=en}}</ref>]] |
|||
combination may be used as a [[linear classifier]], or, more commonly, for [[dimensionality reduction]] before later [[statistical classification|classification]]. |
|||
'''Linear discriminant analysis''' ('''LDA'''), '''normal discriminant analysis''' ('''NDA'''), or '''discriminant function analysis''' is a generalization of '''Fisher's linear discriminant''', a method used in [[statistics]] and other fields, to find a [[linear combination]] of features that characterizes or separates two or more classes of objects or events. The resulting combination may be used as a [[linear classifier]], or, more commonly, for [[dimensionality reduction]] before later [[statistical classification|classification]]. |
|||
⚫ | LDA is closely related to [[analysis of variance]] (ANOVA) and [[regression analysis]], which also attempt to express one [[dependent variable]] as a linear combination of other features or measurements.<ref name="Fisher:1936">{{cite journal |last=Fisher |first=R. A. |author-link=Ronald Fisher |title=The Use of Multiple Measurements in Taxonomic Problems |journal=[[Annals of Eugenics]] |volume=7 |pages=179–188 |year=1936 |hdl=2440/15227|doi=10.1111/j.1469-1809.1936.tb02137.x |issue=2|url=https://digital.library.adelaide.edu.au/dspace/bitstream/2440/15227/1/138.pdf |hdl-access=free }} |
||
</ref><ref name="McLachlan:2004">{{cite book |title=Discriminant Analysis and Statistical Pattern Recognition |first1=G. J. |last1=McLachlan |publisher=Wiley Interscience |isbn=978-0-471-69115-0 |year=2004 |mr=1190469}}</ref> However, ANOVA uses [[categorical variable|categorical]] [[independent variables]] and a [[continuous variable|continuous]] [[dependent variable]], whereas discriminant analysis has continuous [[independent variables]] and a categorical dependent variable (''i.e.'' the class label).<ref>Analyzing Quantitative Data: An Introduction for Social Researchers, Debra Wetcher-Hendricks, p.288</ref> [[Logistic regression]] and [[probit regression]] are more similar to LDA than ANOVA is, as they also explain a categorical variable by the values of continuous independent variables. These other methods are preferable in applications where it is not reasonable to assume that the independent variables are normally distributed, which is a fundamental assumption of the LDA method. |
|||
LDA is also closely related to [[principal component analysis]] (PCA) and [[factor analysis]] in that they both look for linear combinations of variables which best explain the data.<ref name="Martinez:2001">{{cite journal |last1=Martinez |first1=A. M. |last2=Kak |first2=A. C. |title=PCA versus LDA |journal=[[IEEE Transactions on Pattern Analysis and Machine Intelligence]] |volume=23 |issue=2 |pages=228–233 |year=2001 |url=http://www.ece.osu.edu/~aleix/pami01.pdf |doi=10.1109/34.908974 |access-date=2010-06-30 |archive-date=2008-10-11 |archive-url=https://web.archive.org/web/20081011071459/http://www.ece.osu.edu/~aleix/pami01.pdf |url-status=dead }}</ref> LDA explicitly attempts to model the difference between the classes of data. PCA, in contrast, does not take into account any difference in class, and factor analysis builds the feature combinations based on differences rather than similarities. Discriminant analysis is also different from factor analysis in that it is not an interdependence technique: a distinction between independent variables and dependent variables (also called criterion variables) must be made. |
|||
LDA works when the measurements made on independent variables for each observation are continuous quantities. When dealing with categorical independent variables, the equivalent technique is discriminant correspondence analysis.<ref name="Abdi 2007">Abdi, H. (2007) [http://www.utdallas.edu/~herve/Abdi-DCA2007-pretty.pdf "Discriminant correspondence analysis."] In: N.J. Salkind (Ed.): ''Encyclopedia of Measurement and Statistic''. Thousand Oaks (CA): Sage. pp. 270–275.</ref><ref name="Perriere 2003">{{cite journal | last1 = Perriere | first1 = G. | last2 = Thioulouse | first2 = J. | year = 2003 | title = Use of Correspondence Discriminant Analysis to predict the subcellular location of bacterial proteins | journal = Computer Methods and Programs in Biomedicine | volume = 70 | issue = 2| pages = 99–105 | doi=10.1016/s0169-2607(02)00011-1| pmid = 12507786 }}</ref> |
|||
Discriminant analysis is used when groups are known a priori (unlike in [[cluster analysis]]). Each case must have a score on one or more quantitative predictor measures, and a score on a group measure.<ref name="buy">Büyüköztürk, Ş. & Çokluk-Bökeoğlu, Ö. (2008). [https://ejer.com.tr/wp-content/uploads/2021/01/ejer_2008_issue_33.pdf Discriminant function analysis: Concept and application]. Egitim Arastirmalari - Eurasian Journal of Educational Research, 33, 73-92. </ref> In simple terms, discriminant function analysis is classification - the act of distributing things into groups, classes or categories of the same type. |
|||
⚫ | LDA is closely related to [[analysis of variance]] (ANOVA) and [[regression analysis]], which also attempt to express one [[dependent variable]] as a linear combination of other features or measurements.<ref name="Fisher:1936">{{cite journal |last=Fisher |first=R. A. |author-link=Ronald Fisher |title= |
||
==History== |
==History== |
||
The original [[dichotomy|dichotomous]] discriminant analysis was developed by Sir [[Ronald Fisher]] in 1936.<ref name="cohen">Cohen et al. Applied Multiple Regression/Correlation Analysis for the Behavioural Sciences 3rd ed. (2003). Taylor & Francis Group.</ref> It is different from an [[ANOVA]] or [[MANOVA]], which is used to predict one (ANOVA) or multiple (MANOVA) continuous dependent variables by one or more independent categorical variables. Discriminant function analysis is useful in determining whether a set of variables is effective in predicting category membership.<ref name="green"> |
The original [[dichotomy|dichotomous]] discriminant analysis was developed by Sir [[Ronald Fisher]] in 1936.<ref name="cohen">Cohen et al. Applied Multiple Regression/Correlation Analysis for the Behavioural Sciences 3rd ed. (2003). Taylor & Francis Group.</ref> It is different from an [[ANOVA]] or [[MANOVA]], which is used to predict one (ANOVA) or multiple (MANOVA) continuous dependent variables by one or more independent categorical variables. Discriminant function analysis is useful in determining whether a set of variables is effective in predicting category membership.<ref name="green">{{cite journal | url=https://www.tandfonline.com/doi/pdf/10.1198/tas.2005.s139 | doi=10.1198/tas.2005.s139 | title=Using SPSS for Windows and Macintosh: Analyzing and Understanding Data | date=2005 | last1=Hansen | first1=John | journal=The American Statistician | volume=59 | page=113 }}</ref> |
||
==LDA for two classes== |
==LDA for two classes== |
||
Consider a set of observations <math> { \vec x } </math> (also called features, attributes, variables or measurements) for each sample of an object or event with known class <math>y</math>. This set of samples is called the [[training set]]. The classification problem is then to find a good predictor for the class <math>y</math> of any sample of the same distribution (not necessarily from the training set) given only an observation <math> \vec x </math>.<ref name="Venables:2002">{{cite book |title=Modern Applied Statistics with S |first1=W. N. |last1=Venables |first2=B. D. |last2=Ripley |author-link2=Brian Ripley |publisher=Springer Verlag |isbn=978-0-387-95457-8 |year=2002 |edition=4th}} |
Consider a set of observations <math> { \vec x } </math> (also called features, attributes, variables or measurements) for each sample of an object or event with known class <math>y</math>. This set of samples is called the [[training set]] in a [[supervised learning]] context. The classification problem is then to find a good predictor for the class <math>y</math> of any sample of the same distribution (not necessarily from the training set) given only an observation <math> \vec x </math>.<ref name="Venables:2002">{{cite book |title=Modern Applied Statistics with S |first1=W. N. |last1=Venables |first2=B. D. |last2=Ripley |author-link2=Brian Ripley |publisher=Springer Verlag |isbn=978-0-387-95457-8 |year=2002 |edition=4th}} |
||
</ref>{{rp|338}} |
</ref>{{rp|338}} |
||
LDA approaches the problem by assuming that the conditional [[probability density function]]s <math>p(\vec x|y=0)</math> and <math>p(\vec x|y=1)</math> are both [[Multivariate normal distribution|the normal distribution]] with mean and [[covariance]] parameters <math>\left(\vec \mu_0, \Sigma_0\right)</math> and <math>\left(\vec \mu_1, \Sigma_1\right)</math>, respectively. Under this assumption, the Bayes optimal solution is to predict points as being from the second class if the log of the likelihood ratios is bigger than some threshold T, so that: |
LDA approaches the problem by assuming that the conditional [[probability density function]]s <math>p(\vec x|y=0)</math> and <math>p(\vec x|y=1)</math> are both [[Multivariate normal distribution|the normal distribution]] with mean and [[covariance]] parameters <math>\left(\vec \mu_0, \Sigma_0\right)</math> and <math>\left(\vec \mu_1, \Sigma_1\right)</math>, respectively. Under this assumption, the [[Bayes classifier|Bayes-optimal solution]] is to predict points as being from the second class if the log of the likelihood ratios is bigger than some threshold T, so that: |
||
: <math> (\vec x- \vec \mu_0)^T \Sigma_0^{-1} ( \vec x- \vec \mu_0) + \ln|\Sigma_0| - (\vec x- \vec \mu_1)^T \Sigma_1^{-1} ( \vec x- \vec \mu_1) - \ln|\Sigma_1| \ > \ T </math> |
: <math> \frac{1}{2} (\vec x - \vec \mu_0)^\mathrm{T} \Sigma_0^{-1} ( \vec x - \vec \mu_0) + \frac{1}{2} \ln|\Sigma_0| - \frac{1}{2} (\vec x - \vec \mu_1)^\mathrm{T} \Sigma_1^{-1} ( \vec x - \vec \mu_1) - \frac{1}{2} \ln|\Sigma_1| \ > \ T </math> |
||
Without any further assumptions, the resulting classifier is referred to as |
Without any further assumptions, the resulting classifier is referred to as [[quadratic classifier|quadratic discriminant analysis]] (QDA). |
||
LDA instead makes the additional simplifying [[homoscedastic]]ity assumption (''i.e.'' that the class covariances are identical, so <math>\Sigma_0 = \Sigma_1 = \Sigma</math>) and that the covariances have full rank. |
LDA instead makes the additional simplifying [[homoscedastic]]ity assumption (''i.e.'' that the class covariances are identical, so <math>\Sigma_0 = \Sigma_1 = \Sigma</math>) and that the covariances have full rank. |
||
In this case, several terms cancel: |
In this case, several terms cancel: |
||
:<math> {\vec x}^T \Sigma_0^{-1} \vec x = {\vec x}^T \Sigma_1^{-1} \vec x</math> |
:<math> {\vec x}^\mathrm{T} \Sigma_0^{-1} \vec x = {\vec x}^\mathrm{T} \Sigma_1^{-1} \vec x</math> |
||
:<math>{\vec x}^T {\Sigma_i}^{-1} \vec{\mu}_i = {\vec{\mu}_i}^T{\Sigma_i}^{-1} \vec x</math> because <math>\Sigma_i</math> is [[Hermitian matrix|Hermitian]] |
:<math>{\vec x}^\mathrm{T} {\Sigma_i}^{-1} \vec{\mu}_i = {\vec{\mu}_i}^\mathrm{T}{\Sigma_i}^{-1} \vec x</math> because <math>\Sigma_i</math> is [[Hermitian matrix|Hermitian]] |
||
and the above decision criterion |
and the above decision criterion |
||
becomes a threshold on the [[dot product]] |
becomes a threshold on the [[dot product]] |
||
:<math> \vec w |
:<math> {\vec w}^\mathrm{T} \vec x > c </math> |
||
for some threshold constant ''c'', where |
for some threshold constant ''c'', where |
||
:<math>\vec w = \Sigma^{-1} (\vec \mu_1 - \vec \mu_0)</math> |
:<math>\vec w = \Sigma^{-1} (\vec \mu_1 - \vec \mu_0)</math> |
||
:<math> c = \ |
:<math> c = \frac12 \, {\vec w}^\mathrm{T} (\vec \mu_1 + \vec \mu_0)</math> |
||
This means that the criterion of an input <math> \vec{ x }</math> being in a class <math>y</math> is purely a function of this linear combination of the known observations. |
This means that the criterion of an input <math> \vec{ x }</math> being in a class <math>y</math> is purely a function of this linear combination of the known observations. |
||
It is often useful to see this conclusion in geometrical terms: the criterion of an input <math> \vec{ x }</math> being in a class <math>y</math> is purely a function of projection of multidimensional-space point <math> \vec{ x }</math> onto vector <math> \vec{ w }</math> (thus, we only consider its direction). In other words, the observation belongs to <math>y</math> if corresponding <math> \vec{ x }</math> is located on a certain side of a hyperplane perpendicular to <math> \vec{ w }</math>. The location of the plane is defined by the threshold c. |
It is often useful to see this conclusion in geometrical terms: the criterion of an input <math> \vec{ x }</math> being in a class <math>y</math> is purely a function of projection of multidimensional-space point <math> \vec{ x }</math> onto vector <math> \vec{ w }</math> (thus, we only consider its direction). In other words, the observation belongs to <math>y</math> if corresponding <math> \vec{ x }</math> is located on a certain side of a hyperplane perpendicular to <math> \vec{ w }</math>. The location of the plane is defined by the threshold <math>c</math>. |
||
==Assumptions== |
==Assumptions== |
||
Line 43: | Line 52: | ||
*[[Multivariate normal distribution|Multivariate normality]]: Independent variables are normal for each level of the grouping variable.<ref name="green"/><ref name="buy"/> |
*[[Multivariate normal distribution|Multivariate normality]]: Independent variables are normal for each level of the grouping variable.<ref name="green"/><ref name="buy"/> |
||
*Homogeneity of variance/covariance ([[homoscedasticity]]): Variances among group variables are the same across levels of predictors. Can be tested with [[Box's M test|Box's M]] statistic.<ref name="green"/> It has been suggested, however, that linear discriminant analysis be used when covariances are equal, and that [[quadratic classifier#Quadratic discriminant analysis|quadratic discriminant analysis]] may be used when covariances are not equal.<ref name="buy"/> |
*Homogeneity of variance/covariance ([[homoscedasticity]]): Variances among group variables are the same across levels of predictors. Can be tested with [[Box's M test|Box's M]] statistic.<ref name="green"/> It has been suggested, however, that linear discriminant analysis be used when covariances are equal, and that [[quadratic classifier#Quadratic discriminant analysis|quadratic discriminant analysis]] may be used when covariances are not equal.<ref name="buy"/> |
||
*[[Multicollinearity]]: Predictive power can decrease with an increased correlation between predictor variables.<ref name="buy"/> |
|||
*[[statistical independence|Independence]]: Participants are assumed to be randomly sampled, and a participant's score on one variable is assumed to be independent of scores on that variable for all other participants.<ref name="green"/><ref name="buy"/> |
*[[statistical independence|Independence]]: Participants are assumed to be randomly sampled, and a participant's score on one variable is assumed to be independent of scores on that variable for all other participants.<ref name="green"/><ref name="buy"/> |
||
Line 54: | Line 62: | ||
Each function is given a discriminant score{{clarify|date=April 2019}} to determine how well it predicts group placement. |
Each function is given a discriminant score{{clarify|date=April 2019}} to determine how well it predicts group placement. |
||
*Structure Correlation Coefficients: The correlation between each predictor and the discriminant score of each function. This is a zero-order correlation (i.e., not corrected for the other predictors). |
*Structure Correlation Coefficients: The correlation between each predictor and the discriminant score of each function. This is a zero-order correlation (i.e., not corrected for the other predictors).<ref>Garson, G. D. (2008). Discriminant function analysis. https://web.archive.org/web/20080312065328/http://www2.chass.ncsu.edu/garson/pA765/discrim.htm.</ref> |
||
*Standardized Coefficients: Each predictor's weight in the linear combination that is the discriminant function. Like in a regression equation, these coefficients are partial (i.e., corrected for the other predictors). Indicates the unique contribution of each predictor in predicting group assignment. |
*Standardized Coefficients: Each predictor's weight in the linear combination that is the discriminant function. Like in a regression equation, these coefficients are partial (i.e., corrected for the other predictors). Indicates the unique contribution of each predictor in predicting group assignment. |
||
*Functions at Group Centroids: Mean discriminant scores for each grouping variable are given for each function. The farther apart the means are, the less error there will be in classification. |
*Functions at Group Centroids: Mean discriminant scores for each grouping variable are given for each function. The farther apart the means are, the less error there will be in classification. |
||
Line 60: | Line 68: | ||
==Discrimination rules== |
==Discrimination rules== |
||
*[[Maximum likelihood]]: Assigns x to the group that maximizes population (group) density.<ref name="har">Hardle, W., Simar, L. (2007). ''[https://pdfs.semanticscholar.org/e49a/3ed1560a0979f5ca0146c1aca7ef98e64af7.pdf Applied Multivariate Statistical Analysis]''. Springer Berlin Heidelberg. pp. 289-303.</ref> |
*[[Maximum likelihood]]: Assigns <math>x</math> to the group that maximizes population (group) density.<ref name="har">Hardle, W., Simar, L. (2007). ''[https://pdfs.semanticscholar.org/e49a/3ed1560a0979f5ca0146c1aca7ef98e64af7.pdf Applied Multivariate Statistical Analysis]''. Springer Berlin Heidelberg. pp. 289-303.</ref> |
||
*Bayes Discriminant Rule: Assigns x to the group that maximizes <math>\pi_i f_i(x)</math>, where ''π<sub>i</sub>'' represents the [[prior probability]] of that classification, and <math>f_i(x)</math> represents the population density.<ref name="har"/> |
*Bayes Discriminant Rule: Assigns <math>x</math> to the group that maximizes <math>\pi_i f_i(x)</math>, where ''π<sub>i</sub>'' represents the [[prior probability]] of that classification, and <math>f_i(x)</math> represents the population density.<ref name="har"/> |
||
*[[ |
*[[#Fisher's linear discriminant|Fisher's linear discriminant rule]]: Maximizes the ratio between ''SS''<sub>between</sub> and ''SS''<sub>within</sub>, and finds a linear combination of the predictors to predict group.<ref name="har"/> |
||
==Eigenvalues== |
==Eigenvalues== |
||
Line 71: | Line 79: | ||
Some suggest the use of eigenvalues as [[effect size]] measures, however, this is generally not supported.<ref name="green"/> Instead, the [[canonical correlation]] is the preferred measure of effect size. It is similar to the eigenvalue, but is the square root of the ratio of ''SS''<sub>between</sub> and ''SS''<sub>total</sub>. It is the correlation between groups and the function.<ref name="green"/> |
Some suggest the use of eigenvalues as [[effect size]] measures, however, this is generally not supported.<ref name="green"/> Instead, the [[canonical correlation]] is the preferred measure of effect size. It is similar to the eigenvalue, but is the square root of the ratio of ''SS''<sub>between</sub> and ''SS''<sub>total</sub>. It is the correlation between groups and the function.<ref name="green"/> |
||
Another popular measure of effect size is the percent of variance{{clarify|date=April 2012}} for each function. This is calculated by: (''λ<sub>x</sub>/Σλ<sub>i</sub>'') X 100 where ''λ<sub>x</sub>'' is the eigenvalue for the function and Σ''λ<sub>i</sub>'' is the sum of all eigenvalues. This tells us how strong the prediction is for that particular function compared to the others.<ref name="green"/> |
Another popular measure of effect size is the percent of variance{{clarify|date=April 2012}} for each function. This is calculated by: (''λ<sub>x</sub>/Σλ<sub>i</sub>'') X 100 where ''λ<sub>x</sub>'' is the eigenvalue for the function and Σ''λ<sub>i</sub>'' is the sum of all eigenvalues. This tells us how strong the prediction is for that particular function compared to the others.<ref name="green"/> |
||
Percent correctly classified can also be analyzed as an effect size. The kappa value can describe this while correcting for chance agreement.<ref name="green"/>{{clarify|date=April 2012|reason=Kappa normalizes across all categorizes rather than biased by a significantly good or poorly performing classes|text=Kappa normalizes across all categorizes rather than biased by a significantly good or poorly performing classes.|pre-text=|post-text=}}<ref>{{Cite journal|last=Israel|first=Steven A.|date=June 2006|title=Performance Metrics: How and When|journal=Geocarto International|volume=21|issue=2|pages=23–32|doi=10.1080/10106040608542380|s2cid=122376081|issn=1010-6049}}</ref> |
Percent correctly classified can also be analyzed as an effect size. The kappa value can describe this while correcting for chance agreement.<ref name="green"/>{{clarify|date=April 2012|reason=Kappa normalizes across all categorizes rather than biased by a significantly good or poorly performing classes|text=Kappa normalizes across all categorizes rather than biased by a significantly good or poorly performing classes.|pre-text=|post-text=}}<ref>{{Cite journal|last=Israel|first=Steven A.|date=June 2006|title=Performance Metrics: How and When|journal=Geocarto International|volume=21|issue=2|pages=23–32|doi=10.1080/10106040608542380|bibcode=2006GeoIn..21...23I |s2cid=122376081|issn=1010-6049}}</ref> |
||
==Canonical discriminant analysis for ''k'' classes== |
==Canonical discriminant analysis for ''k'' classes== |
||
Line 79: | Line 87: | ||
The terms ''Fisher's linear discriminant'' and ''LDA'' are often used interchangeably, although [[Ronald A. Fisher|Fisher's]] original article<ref name="Fisher:1936" /> actually describes a slightly different discriminant, which does not make some of the assumptions of LDA such as [[normal distribution|normally distributed]] classes or equal class [[covariance]]s. |
The terms ''Fisher's linear discriminant'' and ''LDA'' are often used interchangeably, although [[Ronald A. Fisher|Fisher's]] original article<ref name="Fisher:1936" /> actually describes a slightly different discriminant, which does not make some of the assumptions of LDA such as [[normal distribution|normally distributed]] classes or equal class [[covariance]]s. |
||
Suppose two classes of observations have [[mean]]s <math> \vec \mu_0, \vec \mu_1 </math> and covariances <math>\Sigma_0,\Sigma_1 </math>. Then the linear combination of features <math> \vec w |
Suppose two classes of observations have [[mean]]s <math> \vec \mu_0, \vec \mu_1 </math> and covariances <math>\Sigma_0,\Sigma_1 </math>. Then the linear combination of features <math> {\vec w}^\mathrm{T} \vec x </math> will have [[mean]]s <math> {\vec w}^\mathrm{T} \vec \mu_i </math> and [[variance]]s <math> {\vec w}^\mathrm{T} \Sigma_i \vec w </math> for <math> i=0,1 </math>. Fisher defined the separation between these two [[probability distribution|distributions]] to be the ratio of the variance between the classes to the variance within the classes: |
||
:<math>S=\frac{\sigma_{\text{between}}^2}{\sigma_{\text{within}}^2}= \frac{(\vec w \cdot \vec \mu_1 - \vec w \cdot \vec \mu_0)^2}{\vec w^T \Sigma_1 \vec w + \vec w^T \Sigma_0 \vec w} = \frac{(\vec w \cdot (\vec \mu_1 - \vec \mu_0))^2}{\vec w^T (\Sigma_0+\Sigma_1) \vec w} </math> |
:<math>S=\frac{\sigma_{\text{between}}^2}{\sigma_{\text{within}}^2}= \frac{(\vec w \cdot \vec \mu_1 - \vec w \cdot \vec \mu_0)^2}{{\vec w}^\mathrm{T} \Sigma_1 \vec w + {\vec w}^\mathrm{T} \Sigma_0 \vec w} = \frac{(\vec w \cdot (\vec \mu_1 - \vec \mu_0))^2}{{\vec w}^\mathrm{T} (\Sigma_0+\Sigma_1) \vec w} </math> |
||
This measure is, in some sense, a measure of the [[signal-to-noise ratio]] for the class labelling. It can be shown that the maximum separation occurs when |
This measure is, in some sense, a measure of the [[signal-to-noise ratio]] for the class labelling. It can be shown that the maximum separation occurs when |
||
Line 88: | Line 96: | ||
When the assumptions of LDA are satisfied, the above equation is equivalent to LDA. |
When the assumptions of LDA are satisfied, the above equation is equivalent to LDA. |
||
[[File:Fisher2classes.png|thumb|Fisher's Linear Discriminant visualised as an axis]] |
|||
Be sure to note that the vector <math>\vec w</math> is the [[surface normal|normal]] to the discriminant [[hyperplane]]. As an example, in a two dimensional problem, the line that best divides the two groups is perpendicular to <math>\vec w</math>. |
Be sure to note that the vector <math>\vec w</math> is the [[surface normal|normal]] to the discriminant [[hyperplane]]. As an example, in a two dimensional problem, the line that best divides the two groups is perpendicular to <math>\vec w</math>. |
||
Generally, the data points to be discriminated are projected onto <math>\vec w</math>; then the threshold that best separates the data is chosen from analysis of the one-dimensional distribution. There is no general rule for the threshold. However, if projections of points from both classes exhibit approximately the same distributions, a good choice would be the hyperplane between projections of the two means, <math>\vec w \cdot \vec \mu_0 </math> and <math>\vec w \cdot \vec \mu_1 </math>. In this case the parameter c in threshold condition <math> \vec w \cdot \vec x > c </math> can be found explicitly: |
Generally, the data points to be discriminated are projected onto <math>\vec w</math>; then the threshold that best separates the data is chosen from analysis of the one-dimensional distribution. There is no general rule for the threshold. However, if projections of points from both classes exhibit approximately the same distributions, a good choice would be the hyperplane between projections of the two means, <math>\vec w \cdot \vec \mu_0 </math> and <math>\vec w \cdot \vec \mu_1 </math>. In this case the parameter c in threshold condition <math> \vec w \cdot \vec x > c </math> can be found explicitly: |
||
:<math> c = \vec w \cdot \frac12 (\vec \mu_0 + \vec \mu_1) = \frac{1}{2} \vec\mu_1^T \Sigma^{-1}_{1} \vec\mu_1 - \frac{1}{2} \vec\mu_0^T \Sigma^{-1}_{0} \vec\mu_0 </math>. |
:<math> c = \vec w \cdot \frac12 (\vec \mu_0 + \vec \mu_1) = \frac{1}{2} \vec\mu_1^\mathrm{T} \Sigma^{-1}_{1} \vec\mu_1 - \frac{1}{2} \vec\mu_0^\mathrm{T} \Sigma^{-1}_{0} \vec\mu_0 </math>. |
||
[[Otsu's method]] is related to Fisher's linear discriminant, and was created to binarize the histogram of pixels in a grayscale image by optimally picking the black/white threshold that minimizes intra-class variance and maximizes inter-class variance within/between grayscales assigned to black and white pixel classes. |
[[Otsu's method]] is related to Fisher's linear discriminant, and was created to binarize the histogram of pixels in a grayscale image by optimally picking the black/white threshold that minimizes intra-class variance and maximizes inter-class variance within/between grayscales assigned to black and white pixel classes. |
||
==Multiclass LDA== |
==Multiclass LDA== |
||
[[File:4class3ddiscriminant.png|thumb|Visualisation for one-versus-all LDA axes for 4 classes in 3d]] |
|||
⚫ | In the case where there are more than two classes, the analysis used in the derivation of the Fisher discriminant can be extended to find a [[Linear subspace|subspace]] which appears to contain all of the class variability.<ref name="garson">Garson, G. D. (2008). Discriminant function analysis. {{cite web |url=http://www2.chass.ncsu.edu/garson/pa765/discrim.htm |title= |
||
[[File:3dProjections.png|thumb|Projections along linear discriminant axes for 4 classes]] |
|||
⚫ | In the case where there are more than two classes, the analysis used in the derivation of the Fisher discriminant can be extended to find a [[Linear subspace|subspace]] which appears to contain all of the class variability.<ref name="garson">Garson, G. D. (2008). Discriminant function analysis. {{cite web |url=http://www2.chass.ncsu.edu/garson/pa765/discrim.htm |title=PA 765: Discriminant Function Analysis |access-date=2008-03-04 |url-status=dead |archive-url=https://web.archive.org/web/20080312065328/http://www2.chass.ncsu.edu/garson/pA765/discrim.htm |archive-date=2008-03-12 }} .</ref> This generalization is due to [[C. R. Rao]].<ref name="Rao:1948">{{cite journal |last=Rao |first=R. C. |author-link=Calyampudi Radhakrishna Rao |title=The utilization of multiple measurements in problems of biological classification |journal=Journal of the Royal Statistical Society, Series B |volume=10 |issue=2 |pages=159–203 |year=1948 |doi=10.1111/j.2517-6161.1948.tb00008.x |jstor=2983775}}</ref> Suppose that each of C classes has a mean <math> \mu_i </math> and the same covariance <math> \Sigma </math>. Then the scatter between class variability may be defined by the sample covariance of the class means |
||
:<math> \Sigma_b = \frac{1}{C} \sum_{i=1}^C (\mu_i-\mu) (\mu_i-\mu)^T </math> |
:<math> \Sigma_b = \frac{1}{C} \sum_{i=1}^C (\mu_i-\mu) (\mu_i-\mu)^\mathrm{T} </math> |
||
where <math> \mu </math> is the mean of the class means. The class separation in a direction <math> \vec w </math> in this case will be given by |
where <math> \mu </math> is the mean of the class means. The class separation in a direction <math> \vec w </math> in this case will be given by |
||
:<math> S = \frac{\vec w^T \Sigma_b \vec w}{\vec w^T \Sigma \vec w} </math> |
:<math> S = \frac{{\vec w}^\mathrm{T} \Sigma_b \vec w}{{\vec w}^\mathrm{T} \Sigma \vec w} </math> |
||
This means that when <math> \vec w </math> is an [[eigenvector]] of <math> \Sigma^{-1} \Sigma_b </math> the separation will be equal to the corresponding [[eigenvalue]]. |
This means that when <math> \vec w </math> is an [[eigenvector]] of <math> \Sigma^{-1} \Sigma_b </math> the separation will be equal to the corresponding [[eigenvalue]]. |
||
Line 114: | Line 124: | ||
==Incremental LDA== |
==Incremental LDA== |
||
The typical implementation of the LDA technique requires that all the samples are available in advance. However, there are situations where the entire data set is not available and the input data are observed as a stream. In this case, it is desirable for the LDA feature extraction to have the ability to update the computed LDA features by observing the new samples without running the algorithm on the whole data set. For example, in many real-time applications such as mobile robotics or on-line face recognition, it is important to update the extracted LDA features as soon as new observations are available. An LDA feature extraction technique that can update the LDA features by simply observing new samples is an ''incremental LDA algorithm'', and this idea has been extensively studied over the last two decades.<ref name=":0">{{Cite journal|title = Fast incremental LDA feature extraction|journal = Pattern Recognition|date = 2015-06-01|pages = 1999–2012|volume = 48|issue = 6|doi = 10.1016/j.patcog.2014.12.012|first1 = Youness|last1 = Aliyari Ghassabeh|first2 = Frank|last2 = Rudzicz|first3 = Hamid Abrishami|last3 = Moghaddam}}</ref> Chatterjee and Roychowdhury proposed an incremental self-organized LDA algorithm for updating the LDA features.<ref name=":1">{{Cite journal|title = On self-organizing algorithms and networks for class-separability features|journal = IEEE Transactions on Neural Networks|date = 1997-05-01|issn = 1045-9227|pages = 663–678|volume = 8|issue = 3|doi = 10.1109/72.572105|pmid = 18255669|first1 = C.|last1 = Chatterjee|first2 = V.P.|last2 = Roychowdhury}}</ref> In other work, Demir and Ozmehmet proposed online local learning algorithms for updating LDA features incrementally using error-correcting and the Hebbian learning rules.<ref>{{Cite journal|title = Online Local Learning Algorithms for Linear Discriminant Analysis|journal = Pattern |
The typical implementation of the LDA technique requires that all the samples are available in advance. However, there are situations where the entire data set is not available and the input data are observed as a stream. In this case, it is desirable for the LDA feature extraction to have the ability to update the computed LDA features by observing the new samples without running the algorithm on the whole data set. For example, in many real-time applications such as mobile robotics or on-line face recognition, it is important to update the extracted LDA features as soon as new observations are available. An LDA feature extraction technique that can update the LDA features by simply observing new samples is an ''incremental LDA algorithm'', and this idea has been extensively studied over the last two decades.<ref name=":0">{{Cite journal|title = Fast incremental LDA feature extraction|journal = Pattern Recognition|date = 2015-06-01|pages = 1999–2012|volume = 48|issue = 6|doi = 10.1016/j.patcog.2014.12.012|first1 = Youness|last1 = Aliyari Ghassabeh|first2 = Frank|last2 = Rudzicz|first3 = Hamid Abrishami|last3 = Moghaddam|bibcode = 2015PatRe..48.1999A}}</ref> Chatterjee and Roychowdhury proposed an incremental self-organized LDA algorithm for updating the LDA features.<ref name=":1">{{Cite journal|title = On self-organizing algorithms and networks for class-separability features|journal = IEEE Transactions on Neural Networks|date = 1997-05-01|issn = 1045-9227|pages = 663–678|volume = 8|issue = 3|doi = 10.1109/72.572105|pmid = 18255669|first1 = C.|last1 = Chatterjee|first2 = V.P.|last2 = Roychowdhury}}</ref> In other work, Demir and Ozmehmet proposed online local learning algorithms for updating LDA features incrementally using error-correcting and the Hebbian learning rules.<ref>{{Cite journal|title = Online Local Learning Algorithms for Linear Discriminant Analysis|journal = Pattern Recognit. Lett.|date = 2005-03-01|issn = 0167-8655|pages = 421–431|volume = 26|issue = 4|doi = 10.1016/j.patrec.2004.08.005|first1 = G. K.|last1 = Demir|first2 = K.|last2 = Ozmehmet|bibcode = 2005PaReL..26..421D}}</ref> Later, Aliyari ''et a''l. derived fast incremental algorithms to update the LDA features by observing the new samples.<ref name=":0" /> |
||
==Practical use== |
==Practical use== |
||
In practice, the class means and covariances are not known. They can, however, be estimated from the training set. Either the [[maximum likelihood estimation|maximum likelihood estimate]] or the [[maximum a posteriori]] estimate may be used in place of the exact value in the above equations. Although the estimates of the covariance may be considered optimal in some sense, this does not mean that the resulting discriminant obtained by substituting these values is optimal in any sense, even if the assumption of normally distributed classes is correct. |
In practice, the class means and covariances are not known. They can, however, be estimated from the training set. Either the [[maximum likelihood estimation|maximum likelihood estimate]] or the [[maximum a posteriori]] estimate may be used in place of the exact value in the above equations. Although the estimates of the covariance may be considered optimal in some sense, this does not mean that the resulting discriminant obtained by substituting these values is optimal in any sense, even if the assumption of normally distributed classes is correct. |
||
Another complication in applying LDA and Fisher's discriminant to real data occurs when the number of measurements of each sample (i.e., the dimensionality of each data vector) exceeds the number of samples in each class.<ref name="Martinez:2001" /> In this case, the covariance estimates do not have full rank, and so cannot be inverted. There are a number of ways to deal with this. One is to use a [[pseudo inverse]] instead of the usual matrix inverse in the above formulae. However, better numeric stability may be achieved by first projecting the problem onto the subspace spanned by <math> \Sigma_b </math>.<ref>{{cite journal | last1 = Yu | first1 = H. | last2 = Yang | first2 = J. | year = 2001 | title = A direct LDA algorithm for high-dimensional data — with application to face recognition | journal = Pattern Recognition | volume = 34 | issue = 10| pages = 2067–2069 | doi=10.1016/s0031-3203(00)00162-x| citeseerx = 10.1.1.70.3507 }}</ref> |
Another complication in applying LDA and Fisher's discriminant to real data occurs when the number of measurements of each sample (i.e., the dimensionality of each data vector) exceeds the number of samples in each class.<ref name="Martinez:2001" /> In this case, the covariance estimates do not have full rank, and so cannot be inverted. There are a number of ways to deal with this. One is to use a [[pseudo inverse]] instead of the usual matrix inverse in the above formulae. However, better numeric stability may be achieved by first projecting the problem onto the subspace spanned by <math> \Sigma_b </math>.<ref>{{cite journal | last1 = Yu | first1 = H. | last2 = Yang | first2 = J. | year = 2001 | title = A direct LDA algorithm for high-dimensional data — with application to face recognition | journal = Pattern Recognition | volume = 34 | issue = 10| pages = 2067–2069 | doi=10.1016/s0031-3203(00)00162-x| bibcode = 2001PatRe..34.2067Y | citeseerx = 10.1.1.70.3507 }}</ref> |
||
Another strategy to deal with small sample size is to use a [[shrinkage estimator]] of the covariance matrix, which |
Another strategy to deal with small sample size is to use a [[shrinkage estimator]] of the covariance matrix, which |
||
can be expressed mathematically as |
can be expressed mathematically as |
||
Line 136: | Line 146: | ||
===Bankruptcy prediction=== |
===Bankruptcy prediction=== |
||
In [[bankruptcy prediction]] based on accounting ratios and other financial variables, linear discriminant analysis was the first statistical method applied to systematically explain which firms entered bankruptcy vs. survived. Despite limitations including known nonconformance of accounting ratios to the normal distribution assumptions of LDA, [[Edward Altman]]'s [[Z-Score Financial Analysis Tool|1968 model]] |
In [[bankruptcy prediction]] based on accounting ratios and other financial variables, linear discriminant analysis was the first statistical method applied to systematically explain which firms entered bankruptcy vs. survived. Despite limitations including known nonconformance of accounting ratios to the normal distribution assumptions of LDA, [[Edward Altman]]'s [[Z-Score Financial Analysis Tool|1968 model]]<ref >{{Cite journal |
||
|last=Altman |first=Edward I. |author-link=Edward I. Altman |
|||
|title=Financial Ratios, Discriminant Analysis and the Prediction of Corporate Bankruptcy |
|||
|journal=[[The Journal of Finance]] |
|||
|volume=23 |issue=4 |
|||
|year=1968 |
|||
|pages=589–609 |
|||
|doi=10.2307/2978933 |
|||
|jstor=2978933 }}</ref> is still a leading model in practical applications.<ref >{{Cite web |
|||
|last1=Agarwal |first1=Vineet |
|||
|last2=Taffler |first2=Richard |
|||
|year=2005 |
|||
|title=Twenty-five years of z-scores in the UK: do they really work? |
|||
|url=https://efmaefm.org/0efmameetings/EFMA%20ANNUAL%20MEETINGS/2006-Madrid/papers/932609_full.pdf |
|||
}}</ref><ref >{{Cite journal |
|||
|last1=Agarwal |first1=Vineet |
|||
|last2=Taffler |first2=Richard |
|||
|year=2007 |
|||
|title=Twenty-Five Years of the Taffler Z-Score Model: Does It Really Have Predictive Ability? |
|||
|journal=Accounting and Business Research |
|||
|volume=37 |
|||
|issue=4 |
|||
|pages=285–300 |
|||
|doi=10.1080/00014788.2007.9663313 |
|||
}}</ref><ref >{{Cite journal |
|||
|first1=Patrick |last1=Bimpong |
|||
|first2=Ishmael |last2=Arhin |
|||
|first3=Thomas hezkeal Khela |last3=Nan |
|||
|first4=Edward |last4=Danso |
|||
|first5=Pious |last5=Opoku |
|||
|first6=Arthur |last6=Benedict |
|||
|first7=Grace |last7=Tettey |
|||
|display-authors=1 |
|||
|title=Assessing Predictive Power and Earnings Manipulations. Applied Study on Listed Consumer Goods and Service Companies in Ghana Using 3 Z-Score Models |
|||
|journal=Expert Journal of Finance |
|||
|volume=8 |issue=1 |
|||
|pages=1–26 |
|||
|year=2020 |
|||
|url=https://finance.expertjournals.com/23597712-801/ |
|||
}}</ref> |
|||
===Face recognition=== |
===Face recognition=== |
||
Line 144: | Line 193: | ||
In [[marketing]], discriminant analysis was once often used to determine the factors which distinguish different types of customers and/or products on the basis of surveys or other forms of collected data. [[Logistic regression]] or other methods are now more commonly used. The use of discriminant analysis in marketing can be described by the following steps: |
In [[marketing]], discriminant analysis was once often used to determine the factors which distinguish different types of customers and/or products on the basis of surveys or other forms of collected data. [[Logistic regression]] or other methods are now more commonly used. The use of discriminant analysis in marketing can be described by the following steps: |
||
#Formulate the problem and gather data—Identify the [[Social salience|salient]] attributes consumers use to evaluate products in this category—Use [[quantitative marketing research]] techniques (such as [[statistical survey|surveys]]) to collect data from a sample of potential customers concerning their ratings of all the product attributes. The data collection stage is usually done by marketing research professionals. Survey questions ask the respondent to rate a product from one to five (or 1 to 7, or 1 to 10) on a range of attributes chosen by the researcher. Anywhere from five to twenty attributes are chosen. They could include things like: ease of use, weight, accuracy, durability, colourfulness, price, or size. The attributes chosen will vary depending on the product being studied. The same question is asked about all the products in the study. The data for multiple products is codified and input into a statistical program such as [[R language|R]], [[SPSS]] or [[SAS programming language|SAS]]. (This step is the same as in Factor analysis). |
#Formulate the problem and gather data—Identify the [[Social salience|salient]] attributes consumers use to evaluate products in this category—Use [[quantitative marketing research]] techniques (such as [[statistical survey|surveys]]) to collect data from a sample of potential customers concerning their ratings of all the product attributes. The data collection stage is usually done by marketing research professionals. Survey questions ask the respondent to rate a product from one to five (or 1 to 7, or 1 to 10) on a range of attributes chosen by the researcher. Anywhere from five to twenty attributes are chosen. They could include things like: ease of use, weight, accuracy, durability, colourfulness, price, or size. The attributes chosen will vary depending on the product being studied. The same question is asked about all the products in the study. The data for multiple products is codified and input into a statistical program such as [[R language|R]], [[SPSS]] or [[SAS programming language|SAS]]. (This step is the same as in Factor analysis). |
||
#Estimate the Discriminant Function Coefficients and determine the statistical significance and validity—Choose the appropriate discriminant analysis method. The direct method involves estimating the discriminant function so that all the predictors are assessed simultaneously. The stepwise method enters the predictors sequentially. The two-group method should be used when the dependent variable has two categories or states. The multiple discriminant method is used when the dependent variable has three or more categorical states. Use [[Wilks' lambda distribution|Wilks's Lambda]] to test for significance in SPSS or F stat in SAS. The most common method used to test validity is to split the sample into an estimation or analysis sample, and a validation or holdout sample. The estimation sample is used in constructing the discriminant function. The validation sample is used to construct a classification matrix which contains the number of correctly classified and incorrectly classified cases. The percentage of correctly classified cases is called the ''hit ratio''. |
#Estimate the Discriminant Function Coefficients and determine the statistical significance and validity—Choose the appropriate discriminant analysis method. The direct method involves estimating the discriminant function so that all the predictors are assessed simultaneously. The [[Stepwise regression|stepwise method]] enters the predictors sequentially. The two-group method should be used when the dependent variable has two categories or states. The multiple discriminant method is used when the dependent variable has three or more categorical states. Use [[Wilks' lambda distribution|Wilks's Lambda]] to test for significance in SPSS or F stat in SAS. The most common method used to test validity is to split the sample into an estimation or analysis sample, and a validation or holdout sample. The estimation sample is used in constructing the discriminant function. The validation sample is used to construct a classification matrix which contains the number of correctly classified and incorrectly classified cases. The percentage of correctly classified cases is called the ''hit ratio''. |
||
#Plot the results on a two dimensional map, define the dimensions, and interpret the results. The statistical program (or a related module) will map the results. The map will plot each product (usually in two-dimensional space). The distance of products to each other indicate either how different they are. The dimensions must be labelled by the researcher. This requires subjective judgement and is often very challenging. See [[perceptual mapping]]. |
#Plot the results on a two dimensional map, define the dimensions, and interpret the results. The statistical program (or a related module) will map the results. The map will plot each product (usually in two-dimensional space). The distance of products to each other indicate either how different they are. The dimensions must be labelled by the researcher. This requires subjective judgement and is often very challenging. See [[perceptual mapping]]. |
||
===Biomedical studies=== |
===Biomedical studies=== |
||
⚫ | The main application of discriminant analysis in medicine is the assessment of severity state of a patient and prognosis of disease outcome. For example, during retrospective analysis, patients are divided into groups according to severity of disease – mild, moderate and severe form. Then results of clinical and laboratory analyses are studied |
||
⚫ | The main application of discriminant analysis in medicine is the assessment of severity state of a patient and prognosis of disease outcome. For example, during retrospective analysis, patients are divided into groups according to severity of disease – mild, moderate, and severe form. Then results of clinical and laboratory analyses are studied to reveal statistically different variables in these groups. Using these variables, discriminant functions are built to classify disease severity in future patients. Additionally, Linear Discriminant Analysis (LDA) can help select more discriminative samples for data augmentation, improving classification performance.<ref>{{cite journal | last1 = Moradi | first1 = M | last2 = Demirel | first2 = H | year = 2024 | title = Alzheimer’s disease classification using 3D conditional progressive GAN-and LDA-based data selection | journal = Signal, Image and Video Processing | volume = 18 | issue = 2 | pages = 1847–1861 }}</ref> |
||
⚫ | In biology, similar principles are used in order to classify and define groups of different biological objects, for example, to define phage types of Salmonella enteritidis based on Fourier transform infrared spectra,<ref>{{cite journal | last1 = Preisner | first1 = O | last2 = Guiomar | first2 = R | last3 = Machado | first3 = J | last4 = Menezes | first4 = JC | last5 = Lopes | first5 = JA | year = 2010 | title = Application of Fourier transform infrared spectroscopy and chemometrics for differentiation of Salmonella enterica serovar Enteritidis phage types | journal = Appl Environ Microbiol | volume = 76 | issue = 11| pages = 3538–3544 | doi=10.1128/aem.01589-09| pmid = 20363777 | pmc = 2876429 }}</ref> to detect animal source of Escherichia coli studying its virulence factors<ref>{{cite journal | last1 = David | first1 = DE | last2 = Lynne | first2 = AM | last3 = Han | first3 = J | last4 = Foley | first4 = SL | year = 2010 | title = Evaluation of virulence factor profiling in the characterization of veterinary Escherichia coli isolates | journal = Appl Environ Microbiol | volume = 76 | issue = 22| pages = 7509–7513 | doi=10.1128/aem.00726-10| pmid = 20889790 | pmc = 2976202 }}</ref> etc. |
||
⚫ | In biology, similar principles are used in order to classify and define groups of different biological objects, for example, to define phage types of Salmonella enteritidis based on Fourier transform infrared spectra,<ref>{{cite journal | last1 = Preisner | first1 = O | last2 = Guiomar | first2 = R | last3 = Machado | first3 = J | last4 = Menezes | first4 = JC | last5 = Lopes | first5 = JA | year = 2010 | title = Application of Fourier transform infrared spectroscopy and chemometrics for differentiation of Salmonella enterica serovar Enteritidis phage types | journal = Appl Environ Microbiol | volume = 76 | issue = 11| pages = 3538–3544 | doi=10.1128/aem.01589-09| pmid = 20363777 | pmc = 2876429 | bibcode = 2010ApEnM..76.3538P }}</ref> to detect animal source of ''Escherichia coli'' studying its virulence factors<ref>{{cite journal | last1 = David | first1 = DE | last2 = Lynne | first2 = AM | last3 = Han | first3 = J | last4 = Foley | first4 = SL | year = 2010 | title = Evaluation of virulence factor profiling in the characterization of veterinary Escherichia coli isolates | journal = Appl Environ Microbiol | volume = 76 | issue = 22| pages = 7509–7513 | doi=10.1128/aem.00726-10| pmid = 20889790 | pmc = 2976202 | bibcode = 2010ApEnM..76.7509D }}</ref> etc. |
||
===Earth science=== |
===Earth science=== |
||
This method can be used to separate the alteration zones. For example, when different data from various zones are available, discriminant analysis can find the pattern within the data and classify it effectively.<ref>{{cite journal | last1 = Tahmasebi | first1 = P. | last2 = Hezarkhani | first2 = A. | last3 = Mortazavi | first3 = M. | year = 2010 | title = Application of discriminant analysis for alteration separation; sungun copper deposit, East Azerbaijan, Iran. Australian | url = http://ajbasweb.com/old/ajbas/2010/564-576.pdf | journal = Journal of Basic and Applied Sciences | volume = 6 | issue = 4| pages = 564–576 }}</ref> |
This method can be used to {{clarify|date=May 2021 |reason=separate what, where?|text=separate the alteration zones}}. For example, when different data from various zones are available, discriminant analysis can find the pattern within the data and classify it effectively.<ref>{{cite journal | last1 = Tahmasebi | first1 = P. | last2 = Hezarkhani | first2 = A. | last3 = Mortazavi | first3 = M. | year = 2010 | title = Application of discriminant analysis for alteration separation; sungun copper deposit, East Azerbaijan, Iran. Australian | url = http://ajbasweb.com/old/ajbas/2010/564-576.pdf | journal = Journal of Basic and Applied Sciences | volume = 6 | issue = 4| pages = 564–576 }}</ref> |
||
==Comparison to logistic regression== |
==Comparison to logistic regression== |
||
Discriminant function analysis is very similar to [[logistic regression]], and both can be used to answer the same research questions.<ref name="green"/> Logistic regression does not have as many assumptions and restrictions as discriminant analysis. However, when discriminant analysis’ assumptions are met, it is more powerful than logistic regression.<ref>{{cite book|author1=Trevor Hastie|author2=Robert Tibshirani|author3=Jerome Friedman|title=The Elements of Statistical Learning. Data Mining, Inference, and Prediction|edition=second|publisher=Springer|page=128}}</ref> Unlike logistic regression, discriminant analysis can be used with small sample sizes. It has been shown that when sample sizes are equal, and homogeneity of variance/covariance holds, discriminant analysis is more accurate.<ref name="buy"/> Despite all these advantages, logistic regression has none-the-less become the common choice, since the assumptions of discriminant analysis are rarely met.<ref name="cohen"/><ref name="buy"/> |
Discriminant function analysis is very similar to [[logistic regression]], and both can be used to answer the same research questions.<ref name="green"/> Logistic regression does not have as many assumptions and restrictions as discriminant analysis. However, when discriminant analysis’ assumptions are met, it is more powerful than logistic regression.<ref>{{cite book|author1=Trevor Hastie|author2=Robert Tibshirani|author3=Jerome Friedman|title=The Elements of Statistical Learning. Data Mining, Inference, and Prediction|edition=second|publisher=Springer|page=128}}</ref> Unlike logistic regression, discriminant analysis can be used with small sample sizes. It has been shown that when sample sizes are equal, and homogeneity of variance/covariance holds, discriminant analysis is more accurate.<ref name="buy"/> Despite all these advantages, logistic regression has none-the-less become the common choice, since the assumptions of discriminant analysis are rarely met.<ref name="cohen"/><ref name="buy"/> |
||
==Linear discriminant in high |
==Linear discriminant in high dimensions== |
||
Geometric |
Geometric anomalies in higher dimensions lead to the well-known [[curse of dimensionality]]. Nevertheless, proper utilization of [[concentration of measure]] phenomena can make computation easier.<ref>Kainen P.C. (1997) [https://web.archive.org/web/20190226172352/http://pdfs.semanticscholar.org/708f/8e0a95ba5977072651c0681f3c7b8f09eca3.pdf Utilizing geometric anomalies of high dimension: When complexity makes computation easier]. In: Kárný M., Warwick K. (eds) Computer Intensive Methods in Control and Signal Processing: The Curse of Dimensionality, Springer, 1997, pp. 282–294.</ref> An important case of these [[Curse of dimensionality#Blessing of dimensionality|''blessing of dimensionality'']] phenomena was highlighted by Donoho and Tanner: if a sample is essentially high-dimensional then each point can be separated from the rest of the sample by linear inequality, with high probability, even for exponentially large samples.<ref>Donoho, D., Tanner, J. (2009) [https://arxiv.org/abs/0906.2530 Observed universality of phase transitions in high-dimensional geometry, with implications for modern data analysis and signal processing], Phil. Trans. R. Soc. A 367, 4273–4293.</ref> These linear inequalities can be selected in the standard (Fisher's) form of the linear discriminant for a rich family of probability distribution.<ref>{{cite journal |last1= Gorban| first1= Alexander N.|last2= Golubkov |first2 = Alexander |last3= Grechuck|first3 = Bogdan |last4= Mirkes|first4 = Evgeny M.|last5= Tyukin |first5 = Ivan Y. | year= 2018 |title= Correction of AI systems by linear discriminants: Probabilistic foundations|journal= Information Sciences |volume=466|pages= 303–322|doi= 10.1016/j.ins.2018.07.040| arxiv= 1811.05321| s2cid= 52876539}}</ref> In particular, such theorems are proven for [[Logarithmically concave measure|log-concave]] distributions including [[Multivariate normal distribution|multidimensional normal distribution]] (the proof is based on the concentration inequalities for log-concave measures<ref>Guédon, O., Milman, E. (2011) [https://arxiv.org/abs/1011.0943 Interpolating thin-shell and sharp large-deviation estimates for isotropic log-concave measures], Geom. Funct. Anal. 21 (5), 1043–1068.</ref>) and for product measures on a multidimensional cube (this is proven using [[Talagrand's concentration inequality]] for product probability spaces). Data separability by classical linear discriminants simplifies the problem of error correction for [[artificial intelligence]] systems in high dimension.<ref name=GMT2019>{{cite journal |last1= Gorban|first1= Alexander N.|last2= Makarov|first2= Valeri A.|last3= Tyukin |first3= Ivan Y.|date= July 2019|title= The unreasonable effectiveness of small neural ensembles in high-dimensional brain|journal= Physics of Life Reviews|volume= 29 |pages= 55–88|doi= 10.1016/j.plrev.2018.09.005|doi-access=free|arxiv= 1809.07656| pmid= 30366739|bibcode= 2019PhLRv..29...55G}}</ref> |
||
==See also== |
==See also== |
||
Line 183: | Line 233: | ||
|year=2000 |isbn=978-0-471-05669-0 |mr=1802993}} |
|year=2000 |isbn=978-0-471-05669-0 |mr=1802993}} |
||
*{{cite book |title=Logistic Regression Models |first1=J. M. |last1=Hilbe |publisher=Chapman & Hall/CRC Press |isbn=978-1-4200-7575-5 |year=2009}} |
*{{cite book |title=Logistic Regression Models |first1=J. M. |last1=Hilbe |publisher=Chapman & Hall/CRC Press |isbn=978-1-4200-7575-5 |year=2009}} |
||
*{{cite book |author=Mika, S.|title=Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No.98TH8468) |
*{{cite book |author=Mika, S.|date=1999 |chapter=Fisher discriminant analysis with kernels |title=Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No.98TH8468) |year=1999 |pages=41–48 |doi=10.1109/NNSP.1999.788121 |display-authors=etal|citeseerx=10.1.1.35.9904 |isbn=978-0-7803-5673-3 |s2cid=8473401 }} |
||
*{{cite journal | last1 = McFarland | first1 = H. Richard | last2 = Donald | first2 = St. P. Richards | year = 2001 | title = Exact Misclassification Probabilities for Plug-In Normal Quadratic Discriminant Functions. I. The Equal-Means Case | journal = Journal of Multivariate Analysis | volume = 77 | issue = 1| pages = 21–53 | doi=10.1006/jmva.2000.1924}} |
*{{cite journal | last1 = McFarland | first1 = H. Richard | last2 = Donald | first2 = St. P. Richards | year = 2001 | title = Exact Misclassification Probabilities for Plug-In Normal Quadratic Discriminant Functions. I. The Equal-Means Case | journal = Journal of Multivariate Analysis | volume = 77 | issue = 1| pages = 21–53 | doi=10.1006/jmva.2000.1924| doi-access = free }} |
||
*{{cite journal | last1 = McFarland | first1 = H. Richard | last2 = Donald | first2 = St. P. Richards | year = 2002 | title = Exact Misclassification Probabilities for Plug-In Normal Quadratic Discriminant Functions. II. The Heterogeneous Case | journal = Journal of Multivariate Analysis | volume = 82 | issue = 2| pages = 299–330 | doi=10.1006/jmva.2001.2034}} |
*{{cite journal | last1 = McFarland | first1 = H. Richard | last2 = Donald | first2 = St. P. Richards | year = 2002 | title = Exact Misclassification Probabilities for Plug-In Normal Quadratic Discriminant Functions. II. The Heterogeneous Case | journal = Journal of Multivariate Analysis | volume = 82 | issue = 2| pages = 299–330 | doi=10.1006/jmva.2001.2034| doi-access = free }} |
||
*{{cite journal | last1 = Haghighat | first1 = M. | last2 = Abdel-Mottaleb | first2 = M. | last3 = Alhalabi | first3 = W. | year = 2016 | title = Discriminant Correlation Analysis: Real-Time Feature Level Fusion for Multimodal Biometric Recognition | journal = IEEE Transactions on Information Forensics and Security | volume = 11 | issue = 9| pages = 1984–1996 | doi=10.1109/TIFS.2016.2569061| s2cid = 15624506 | url = https://zenodo.org/record/889881 }} |
*{{cite journal | last1 = Haghighat | first1 = M. | last2 = Abdel-Mottaleb | first2 = M. | last3 = Alhalabi | first3 = W. | year = 2016 | title = Discriminant Correlation Analysis: Real-Time Feature Level Fusion for Multimodal Biometric Recognition | journal = IEEE Transactions on Information Forensics and Security | volume = 11 | issue = 9| pages = 1984–1996 | doi=10.1109/TIFS.2016.2569061| s2cid = 15624506 | url = https://zenodo.org/record/889881 }} |
||
==External links== |
==External links== |
||
{{ |
{{Wikiversity|Discriminant function analysis}} |
||
* [https://github.com/mhaghighat/dcaFuse Discriminant Correlation Analysis (DCA) of the Haghighat article (see above)] |
* [https://github.com/mhaghighat/dcaFuse Discriminant Correlation Analysis (DCA) of the Haghighat article (see above)] |
||
* [http://www.alglib.net/dataanalysis/lineardiscriminantanalysis.php ALGLIB] contains open-source LDA implementation in C# / C++ / Pascal / VBA. |
* [http://www.alglib.net/dataanalysis/lineardiscriminantanalysis.php ALGLIB] contains open-source LDA implementation in C# / C++ / Pascal / VBA. |
||
Line 198: | Line 248: | ||
* [https://web.archive.org/web/20080312065328/http://www2.chass.ncsu.edu/garson/pA765/discrim.htm Course notes, Discriminant function analysis by G. David Garson, NC State University] |
* [https://web.archive.org/web/20080312065328/http://www2.chass.ncsu.edu/garson/pA765/discrim.htm Course notes, Discriminant function analysis by G. David Garson, NC State University] |
||
* [http://people.revoledu.com/kardi/tutorial/LDA/ Discriminant analysis tutorial in Microsoft Excel by Kardi Teknomo] |
* [http://people.revoledu.com/kardi/tutorial/LDA/ Discriminant analysis tutorial in Microsoft Excel by Kardi Teknomo] |
||
* [http://www.psychstat.missouristate.edu/multibook/mlt03m.html Course notes, Discriminant function analysis by David W. Stockburger, Missouri State University] |
* [http://www.psychstat.missouristate.edu/multibook/mlt03m.html Course notes, Discriminant function analysis by David W. Stockburger, Missouri State University] {{Webarchive|url=https://web.archive.org/web/20160303223657/http://www.psychstat.missouristate.edu/multibook/mlt03m.html |date=2016-03-03 }} |
||
* [http://userwww.sfsu.edu/~efc/classes/biol710/discrim/discrim.pdf Discriminant function analysis (DA) by John Poulsen and Aaron French, San Francisco State University] |
* [http://userwww.sfsu.edu/~efc/classes/biol710/discrim/discrim.pdf Discriminant function analysis (DA) by John Poulsen and Aaron French, San Francisco State University] |
||
{{statistics|analysis|collapsed}} |
{{statistics|analysis|collapsed}} |
||
{{Authority control}} |
|||
{{DEFAULTSORT:Linear Discriminant Analysis}} |
{{DEFAULTSORT:Linear Discriminant Analysis}} |
Latest revision as of 20:50, 31 July 2024
Linear discriminant analysis (LDA), normal discriminant analysis (NDA), or discriminant function analysis is a generalization of Fisher's linear discriminant, a method used in statistics and other fields, to find a linear combination of features that characterizes or separates two or more classes of objects or events. The resulting combination may be used as a linear classifier, or, more commonly, for dimensionality reduction before later classification.
LDA is closely related to analysis of variance (ANOVA) and regression analysis, which also attempt to express one dependent variable as a linear combination of other features or measurements.[2][3] However, ANOVA uses categorical independent variables and a continuous dependent variable, whereas discriminant analysis has continuous independent variables and a categorical dependent variable (i.e. the class label).[4] Logistic regression and probit regression are more similar to LDA than ANOVA is, as they also explain a categorical variable by the values of continuous independent variables. These other methods are preferable in applications where it is not reasonable to assume that the independent variables are normally distributed, which is a fundamental assumption of the LDA method.
LDA is also closely related to principal component analysis (PCA) and factor analysis in that they both look for linear combinations of variables which best explain the data.[5] LDA explicitly attempts to model the difference between the classes of data. PCA, in contrast, does not take into account any difference in class, and factor analysis builds the feature combinations based on differences rather than similarities. Discriminant analysis is also different from factor analysis in that it is not an interdependence technique: a distinction between independent variables and dependent variables (also called criterion variables) must be made.
LDA works when the measurements made on independent variables for each observation are continuous quantities. When dealing with categorical independent variables, the equivalent technique is discriminant correspondence analysis.[6][7]
Discriminant analysis is used when groups are known a priori (unlike in cluster analysis). Each case must have a score on one or more quantitative predictor measures, and a score on a group measure.[8] In simple terms, discriminant function analysis is classification - the act of distributing things into groups, classes or categories of the same type.
History
[edit]The original dichotomous discriminant analysis was developed by Sir Ronald Fisher in 1936.[9] It is different from an ANOVA or MANOVA, which is used to predict one (ANOVA) or multiple (MANOVA) continuous dependent variables by one or more independent categorical variables. Discriminant function analysis is useful in determining whether a set of variables is effective in predicting category membership.[10]
LDA for two classes
[edit]Consider a set of observations (also called features, attributes, variables or measurements) for each sample of an object or event with known class . This set of samples is called the training set in a supervised learning context. The classification problem is then to find a good predictor for the class of any sample of the same distribution (not necessarily from the training set) given only an observation .[11]: 338
LDA approaches the problem by assuming that the conditional probability density functions and are both the normal distribution with mean and covariance parameters and , respectively. Under this assumption, the Bayes-optimal solution is to predict points as being from the second class if the log of the likelihood ratios is bigger than some threshold T, so that:
Without any further assumptions, the resulting classifier is referred to as quadratic discriminant analysis (QDA).
LDA instead makes the additional simplifying homoscedasticity assumption (i.e. that the class covariances are identical, so ) and that the covariances have full rank. In this case, several terms cancel:
- because is Hermitian
and the above decision criterion becomes a threshold on the dot product
for some threshold constant c, where
This means that the criterion of an input being in a class is purely a function of this linear combination of the known observations.
It is often useful to see this conclusion in geometrical terms: the criterion of an input being in a class is purely a function of projection of multidimensional-space point onto vector (thus, we only consider its direction). In other words, the observation belongs to if corresponding is located on a certain side of a hyperplane perpendicular to . The location of the plane is defined by the threshold .
Assumptions
[edit]The assumptions of discriminant analysis are the same as those for MANOVA. The analysis is quite sensitive to outliers and the size of the smallest group must be larger than the number of predictor variables.[8]
- Multivariate normality: Independent variables are normal for each level of the grouping variable.[10][8]
- Homogeneity of variance/covariance (homoscedasticity): Variances among group variables are the same across levels of predictors. Can be tested with Box's M statistic.[10] It has been suggested, however, that linear discriminant analysis be used when covariances are equal, and that quadratic discriminant analysis may be used when covariances are not equal.[8]
- Independence: Participants are assumed to be randomly sampled, and a participant's score on one variable is assumed to be independent of scores on that variable for all other participants.[10][8]
It has been suggested that discriminant analysis is relatively robust to slight violations of these assumptions,[12] and it has also been shown that discriminant analysis may still be reliable when using dichotomous variables (where multivariate normality is often violated).[13]
Discriminant functions
[edit]Discriminant analysis works by creating one or more linear combinations of predictors, creating a new latent variable for each function. These functions are called discriminant functions. The number of functions possible is either where = number of groups, or (the number of predictors), whichever is smaller. The first function created maximizes the differences between groups on that function. The second function maximizes differences on that function, but also must not be correlated with the previous function. This continues with subsequent functions with the requirement that the new function not be correlated with any of the previous functions.
Given group , with sets of sample space, there is a discriminant rule such that if , then . Discriminant analysis then, finds “good” regions of to minimize classification error, therefore leading to a high percent correct classified in the classification table.[14]
Each function is given a discriminant score[clarification needed] to determine how well it predicts group placement.
- Structure Correlation Coefficients: The correlation between each predictor and the discriminant score of each function. This is a zero-order correlation (i.e., not corrected for the other predictors).[15]
- Standardized Coefficients: Each predictor's weight in the linear combination that is the discriminant function. Like in a regression equation, these coefficients are partial (i.e., corrected for the other predictors). Indicates the unique contribution of each predictor in predicting group assignment.
- Functions at Group Centroids: Mean discriminant scores for each grouping variable are given for each function. The farther apart the means are, the less error there will be in classification.
Discrimination rules
[edit]- Maximum likelihood: Assigns to the group that maximizes population (group) density.[16]
- Bayes Discriminant Rule: Assigns to the group that maximizes , where πi represents the prior probability of that classification, and represents the population density.[16]
- Fisher's linear discriminant rule: Maximizes the ratio between SSbetween and SSwithin, and finds a linear combination of the predictors to predict group.[16]
Eigenvalues
[edit]An eigenvalue in discriminant analysis is the characteristic root of each function.[clarification needed] It is an indication of how well that function differentiates the groups, where the larger the eigenvalue, the better the function differentiates.[8] This however, should be interpreted with caution, as eigenvalues have no upper limit.[10][8] The eigenvalue can be viewed as a ratio of SSbetween and SSwithin as in ANOVA when the dependent variable is the discriminant function, and the groups are the levels of the IV[clarification needed].[10] This means that the largest eigenvalue is associated with the first function, the second largest with the second, etc..
Effect size
[edit]Some suggest the use of eigenvalues as effect size measures, however, this is generally not supported.[10] Instead, the canonical correlation is the preferred measure of effect size. It is similar to the eigenvalue, but is the square root of the ratio of SSbetween and SStotal. It is the correlation between groups and the function.[10] Another popular measure of effect size is the percent of variance[clarification needed] for each function. This is calculated by: (λx/Σλi) X 100 where λx is the eigenvalue for the function and Σλi is the sum of all eigenvalues. This tells us how strong the prediction is for that particular function compared to the others.[10] Percent correctly classified can also be analyzed as an effect size. The kappa value can describe this while correcting for chance agreement.[10]Kappa normalizes across all categorizes rather than biased by a significantly good or poorly performing classes.[clarification needed][17]
Canonical discriminant analysis for k classes
[edit]Canonical discriminant analysis (CDA) finds axes (k − 1 canonical coordinates, k being the number of classes) that best separate the categories. These linear functions are uncorrelated and define, in effect, an optimal k − 1 space through the n-dimensional cloud of data that best separates (the projections in that space of) the k groups. See “Multiclass LDA” for details below.
Fisher's linear discriminant
[edit]The terms Fisher's linear discriminant and LDA are often used interchangeably, although Fisher's original article[2] actually describes a slightly different discriminant, which does not make some of the assumptions of LDA such as normally distributed classes or equal class covariances.
Suppose two classes of observations have means and covariances . Then the linear combination of features will have means and variances for . Fisher defined the separation between these two distributions to be the ratio of the variance between the classes to the variance within the classes:
This measure is, in some sense, a measure of the signal-to-noise ratio for the class labelling. It can be shown that the maximum separation occurs when
When the assumptions of LDA are satisfied, the above equation is equivalent to LDA.
Be sure to note that the vector is the normal to the discriminant hyperplane. As an example, in a two dimensional problem, the line that best divides the two groups is perpendicular to .
Generally, the data points to be discriminated are projected onto ; then the threshold that best separates the data is chosen from analysis of the one-dimensional distribution. There is no general rule for the threshold. However, if projections of points from both classes exhibit approximately the same distributions, a good choice would be the hyperplane between projections of the two means, and . In this case the parameter c in threshold condition can be found explicitly:
- .
Otsu's method is related to Fisher's linear discriminant, and was created to binarize the histogram of pixels in a grayscale image by optimally picking the black/white threshold that minimizes intra-class variance and maximizes inter-class variance within/between grayscales assigned to black and white pixel classes.
Multiclass LDA
[edit]In the case where there are more than two classes, the analysis used in the derivation of the Fisher discriminant can be extended to find a subspace which appears to contain all of the class variability.[18] This generalization is due to C. R. Rao.[19] Suppose that each of C classes has a mean and the same covariance . Then the scatter between class variability may be defined by the sample covariance of the class means
where is the mean of the class means. The class separation in a direction in this case will be given by
This means that when is an eigenvector of the separation will be equal to the corresponding eigenvalue.
If is diagonalizable, the variability between features will be contained in the subspace spanned by the eigenvectors corresponding to the C − 1 largest eigenvalues (since is of rank C − 1 at most). These eigenvectors are primarily used in feature reduction, as in PCA. The eigenvectors corresponding to the smaller eigenvalues will tend to be very sensitive to the exact choice of training data, and it is often necessary to use regularisation as described in the next section.
If classification is required, instead of dimension reduction, there are a number of alternative techniques available. For instance, the classes may be partitioned, and a standard Fisher discriminant or LDA used to classify each partition. A common example of this is "one against the rest" where the points from one class are put in one group, and everything else in the other, and then LDA applied. This will result in C classifiers, whose results are combined. Another common method is pairwise classification, where a new classifier is created for each pair of classes (giving C(C − 1)/2 classifiers in total), with the individual classifiers combined to produce a final classification.
Incremental LDA
[edit]The typical implementation of the LDA technique requires that all the samples are available in advance. However, there are situations where the entire data set is not available and the input data are observed as a stream. In this case, it is desirable for the LDA feature extraction to have the ability to update the computed LDA features by observing the new samples without running the algorithm on the whole data set. For example, in many real-time applications such as mobile robotics or on-line face recognition, it is important to update the extracted LDA features as soon as new observations are available. An LDA feature extraction technique that can update the LDA features by simply observing new samples is an incremental LDA algorithm, and this idea has been extensively studied over the last two decades.[20] Chatterjee and Roychowdhury proposed an incremental self-organized LDA algorithm for updating the LDA features.[21] In other work, Demir and Ozmehmet proposed online local learning algorithms for updating LDA features incrementally using error-correcting and the Hebbian learning rules.[22] Later, Aliyari et al. derived fast incremental algorithms to update the LDA features by observing the new samples.[20]
Practical use
[edit]In practice, the class means and covariances are not known. They can, however, be estimated from the training set. Either the maximum likelihood estimate or the maximum a posteriori estimate may be used in place of the exact value in the above equations. Although the estimates of the covariance may be considered optimal in some sense, this does not mean that the resulting discriminant obtained by substituting these values is optimal in any sense, even if the assumption of normally distributed classes is correct.
Another complication in applying LDA and Fisher's discriminant to real data occurs when the number of measurements of each sample (i.e., the dimensionality of each data vector) exceeds the number of samples in each class.[5] In this case, the covariance estimates do not have full rank, and so cannot be inverted. There are a number of ways to deal with this. One is to use a pseudo inverse instead of the usual matrix inverse in the above formulae. However, better numeric stability may be achieved by first projecting the problem onto the subspace spanned by .[23] Another strategy to deal with small sample size is to use a shrinkage estimator of the covariance matrix, which can be expressed mathematically as
where is the identity matrix, and is the shrinkage intensity or regularisation parameter. This leads to the framework of regularized discriminant analysis[24] or shrinkage discriminant analysis.[25]
Also, in many practical cases linear discriminants are not suitable. LDA and Fisher's discriminant can be extended for use in non-linear classification via the kernel trick. Here, the original observations are effectively mapped into a higher dimensional non-linear space. Linear classification in this non-linear space is then equivalent to non-linear classification in the original space. The most commonly used example of this is the kernel Fisher discriminant.
LDA can be generalized to multiple discriminant analysis, where c becomes a categorical variable with N possible states, instead of only two. Analogously, if the class-conditional densities are normal with shared covariances, the sufficient statistic for are the values of N projections, which are the subspace spanned by the N means, affine projected by the inverse covariance matrix. These projections can be found by solving a generalized eigenvalue problem, where the numerator is the covariance matrix formed by treating the means as the samples, and the denominator is the shared covariance matrix. See “Multiclass LDA” above for details.
Applications
[edit]In addition to the examples given below, LDA is applied in positioning and product management.
Bankruptcy prediction
[edit]In bankruptcy prediction based on accounting ratios and other financial variables, linear discriminant analysis was the first statistical method applied to systematically explain which firms entered bankruptcy vs. survived. Despite limitations including known nonconformance of accounting ratios to the normal distribution assumptions of LDA, Edward Altman's 1968 model[26] is still a leading model in practical applications.[27][28][29]
Face recognition
[edit]In computerised face recognition, each face is represented by a large number of pixel values. Linear discriminant analysis is primarily used here to reduce the number of features to a more manageable number before classification. Each of the new dimensions is a linear combination of pixel values, which form a template. The linear combinations obtained using Fisher's linear discriminant are called Fisher faces, while those obtained using the related principal component analysis are called eigenfaces.
Marketing
[edit]In marketing, discriminant analysis was once often used to determine the factors which distinguish different types of customers and/or products on the basis of surveys or other forms of collected data. Logistic regression or other methods are now more commonly used. The use of discriminant analysis in marketing can be described by the following steps:
- Formulate the problem and gather data—Identify the salient attributes consumers use to evaluate products in this category—Use quantitative marketing research techniques (such as surveys) to collect data from a sample of potential customers concerning their ratings of all the product attributes. The data collection stage is usually done by marketing research professionals. Survey questions ask the respondent to rate a product from one to five (or 1 to 7, or 1 to 10) on a range of attributes chosen by the researcher. Anywhere from five to twenty attributes are chosen. They could include things like: ease of use, weight, accuracy, durability, colourfulness, price, or size. The attributes chosen will vary depending on the product being studied. The same question is asked about all the products in the study. The data for multiple products is codified and input into a statistical program such as R, SPSS or SAS. (This step is the same as in Factor analysis).
- Estimate the Discriminant Function Coefficients and determine the statistical significance and validity—Choose the appropriate discriminant analysis method. The direct method involves estimating the discriminant function so that all the predictors are assessed simultaneously. The stepwise method enters the predictors sequentially. The two-group method should be used when the dependent variable has two categories or states. The multiple discriminant method is used when the dependent variable has three or more categorical states. Use Wilks's Lambda to test for significance in SPSS or F stat in SAS. The most common method used to test validity is to split the sample into an estimation or analysis sample, and a validation or holdout sample. The estimation sample is used in constructing the discriminant function. The validation sample is used to construct a classification matrix which contains the number of correctly classified and incorrectly classified cases. The percentage of correctly classified cases is called the hit ratio.
- Plot the results on a two dimensional map, define the dimensions, and interpret the results. The statistical program (or a related module) will map the results. The map will plot each product (usually in two-dimensional space). The distance of products to each other indicate either how different they are. The dimensions must be labelled by the researcher. This requires subjective judgement and is often very challenging. See perceptual mapping.
Biomedical studies
[edit]The main application of discriminant analysis in medicine is the assessment of severity state of a patient and prognosis of disease outcome. For example, during retrospective analysis, patients are divided into groups according to severity of disease – mild, moderate, and severe form. Then results of clinical and laboratory analyses are studied to reveal statistically different variables in these groups. Using these variables, discriminant functions are built to classify disease severity in future patients. Additionally, Linear Discriminant Analysis (LDA) can help select more discriminative samples for data augmentation, improving classification performance.[30]
In biology, similar principles are used in order to classify and define groups of different biological objects, for example, to define phage types of Salmonella enteritidis based on Fourier transform infrared spectra,[31] to detect animal source of Escherichia coli studying its virulence factors[32] etc.
Earth science
[edit]This method can be used to separate the alteration zones[clarification needed]. For example, when different data from various zones are available, discriminant analysis can find the pattern within the data and classify it effectively.[33]
Comparison to logistic regression
[edit]Discriminant function analysis is very similar to logistic regression, and both can be used to answer the same research questions.[10] Logistic regression does not have as many assumptions and restrictions as discriminant analysis. However, when discriminant analysis’ assumptions are met, it is more powerful than logistic regression.[34] Unlike logistic regression, discriminant analysis can be used with small sample sizes. It has been shown that when sample sizes are equal, and homogeneity of variance/covariance holds, discriminant analysis is more accurate.[8] Despite all these advantages, logistic regression has none-the-less become the common choice, since the assumptions of discriminant analysis are rarely met.[9][8]
Linear discriminant in high dimensions
[edit]Geometric anomalies in higher dimensions lead to the well-known curse of dimensionality. Nevertheless, proper utilization of concentration of measure phenomena can make computation easier.[35] An important case of these blessing of dimensionality phenomena was highlighted by Donoho and Tanner: if a sample is essentially high-dimensional then each point can be separated from the rest of the sample by linear inequality, with high probability, even for exponentially large samples.[36] These linear inequalities can be selected in the standard (Fisher's) form of the linear discriminant for a rich family of probability distribution.[37] In particular, such theorems are proven for log-concave distributions including multidimensional normal distribution (the proof is based on the concentration inequalities for log-concave measures[38]) and for product measures on a multidimensional cube (this is proven using Talagrand's concentration inequality for product probability spaces). Data separability by classical linear discriminants simplifies the problem of error correction for artificial intelligence systems in high dimension.[39]
See also
[edit]- Data mining
- Decision tree learning
- Factor analysis
- Kernel Fisher discriminant analysis
- Logit (for logistic regression)
- Linear regression
- Multiple discriminant analysis
- Multidimensional scaling
- Pattern recognition
- Preference regression
- Quadratic classifier
- Statistical classification
References
[edit]- ^ Holtel, Frederik (2023-02-20). "Linear Discriminant Analysis (LDA) Can Be So Easy". Medium. Retrieved 2024-05-18.
- ^ a b Fisher, R. A. (1936). "The Use of Multiple Measurements in Taxonomic Problems" (PDF). Annals of Eugenics. 7 (2): 179–188. doi:10.1111/j.1469-1809.1936.tb02137.x. hdl:2440/15227.
- ^ McLachlan, G. J. (2004). Discriminant Analysis and Statistical Pattern Recognition. Wiley Interscience. ISBN 978-0-471-69115-0. MR 1190469.
- ^ Analyzing Quantitative Data: An Introduction for Social Researchers, Debra Wetcher-Hendricks, p.288
- ^ a b Martinez, A. M.; Kak, A. C. (2001). "PCA versus LDA" (PDF). IEEE Transactions on Pattern Analysis and Machine Intelligence. 23 (2): 228–233. doi:10.1109/34.908974. Archived from the original (PDF) on 2008-10-11. Retrieved 2010-06-30.
- ^ Abdi, H. (2007) "Discriminant correspondence analysis." In: N.J. Salkind (Ed.): Encyclopedia of Measurement and Statistic. Thousand Oaks (CA): Sage. pp. 270–275.
- ^ Perriere, G.; Thioulouse, J. (2003). "Use of Correspondence Discriminant Analysis to predict the subcellular location of bacterial proteins". Computer Methods and Programs in Biomedicine. 70 (2): 99–105. doi:10.1016/s0169-2607(02)00011-1. PMID 12507786.
- ^ a b c d e f g h i Büyüköztürk, Ş. & Çokluk-Bökeoğlu, Ö. (2008). Discriminant function analysis: Concept and application. Egitim Arastirmalari - Eurasian Journal of Educational Research, 33, 73-92.
- ^ a b Cohen et al. Applied Multiple Regression/Correlation Analysis for the Behavioural Sciences 3rd ed. (2003). Taylor & Francis Group.
- ^ a b c d e f g h i j k Hansen, John (2005). "Using SPSS for Windows and Macintosh: Analyzing and Understanding Data". The American Statistician. 59: 113. doi:10.1198/tas.2005.s139.
- ^ Venables, W. N.; Ripley, B. D. (2002). Modern Applied Statistics with S (4th ed.). Springer Verlag. ISBN 978-0-387-95457-8.
- ^ Lachenbruch, P. A. (1975). Discriminant analysis. NY: Hafner
- ^ Klecka, William R. (1980). Discriminant analysis. Quantitative Applications in the Social Sciences Series, No. 19. Thousand Oaks, CA: Sage Publications.
- ^ Hardle, W., Simar, L. (2007). Applied Multivariate Statistical Analysis. Springer Berlin Heidelberg. pp. 289–303.
- ^ Garson, G. D. (2008). Discriminant function analysis. https://web.archive.org/web/20080312065328/http://www2.chass.ncsu.edu/garson/pA765/discrim.htm.
- ^ a b c Hardle, W., Simar, L. (2007). Applied Multivariate Statistical Analysis. Springer Berlin Heidelberg. pp. 289-303.
- ^ Israel, Steven A. (June 2006). "Performance Metrics: How and When". Geocarto International. 21 (2): 23–32. Bibcode:2006GeoIn..21...23I. doi:10.1080/10106040608542380. ISSN 1010-6049. S2CID 122376081.
- ^ Garson, G. D. (2008). Discriminant function analysis. "PA 765: Discriminant Function Analysis". Archived from the original on 2008-03-12. Retrieved 2008-03-04. .
- ^ Rao, R. C. (1948). "The utilization of multiple measurements in problems of biological classification". Journal of the Royal Statistical Society, Series B. 10 (2): 159–203. doi:10.1111/j.2517-6161.1948.tb00008.x. JSTOR 2983775.
- ^ a b Aliyari Ghassabeh, Youness; Rudzicz, Frank; Moghaddam, Hamid Abrishami (2015-06-01). "Fast incremental LDA feature extraction". Pattern Recognition. 48 (6): 1999–2012. Bibcode:2015PatRe..48.1999A. doi:10.1016/j.patcog.2014.12.012.
- ^ Chatterjee, C.; Roychowdhury, V.P. (1997-05-01). "On self-organizing algorithms and networks for class-separability features". IEEE Transactions on Neural Networks. 8 (3): 663–678. doi:10.1109/72.572105. ISSN 1045-9227. PMID 18255669.
- ^ Demir, G. K.; Ozmehmet, K. (2005-03-01). "Online Local Learning Algorithms for Linear Discriminant Analysis". Pattern Recognit. Lett. 26 (4): 421–431. Bibcode:2005PaReL..26..421D. doi:10.1016/j.patrec.2004.08.005. ISSN 0167-8655.
- ^ Yu, H.; Yang, J. (2001). "A direct LDA algorithm for high-dimensional data — with application to face recognition". Pattern Recognition. 34 (10): 2067–2069. Bibcode:2001PatRe..34.2067Y. CiteSeerX 10.1.1.70.3507. doi:10.1016/s0031-3203(00)00162-x.
- ^ Friedman, J. H. (1989). "Regularized Discriminant Analysis" (PDF). Journal of the American Statistical Association. 84 (405): 165–175. CiteSeerX 10.1.1.382.2682. doi:10.2307/2289860. JSTOR 2289860. MR 0999675.
- ^ Ahdesmäki, M.; Strimmer, K. (2010). "Feature selection in omics prediction problems using cat scores and false nondiscovery rate control". Annals of Applied Statistics. 4 (1): 503–519. arXiv:0903.2003. doi:10.1214/09-aoas277. S2CID 2508935.
- ^ Altman, Edward I. (1968). "Financial Ratios, Discriminant Analysis and the Prediction of Corporate Bankruptcy". The Journal of Finance. 23 (4): 589–609. doi:10.2307/2978933. JSTOR 2978933.
- ^ Agarwal, Vineet; Taffler, Richard (2005). "Twenty-five years of z-scores in the UK: do they really work?" (PDF).
- ^ Agarwal, Vineet; Taffler, Richard (2007). "Twenty-Five Years of the Taffler Z-Score Model: Does It Really Have Predictive Ability?". Accounting and Business Research. 37 (4): 285–300. doi:10.1080/00014788.2007.9663313.
- ^ Bimpong, Patrick; et al. (2020). "Assessing Predictive Power and Earnings Manipulations. Applied Study on Listed Consumer Goods and Service Companies in Ghana Using 3 Z-Score Models". Expert Journal of Finance. 8 (1): 1–26.
- ^ Moradi, M; Demirel, H (2024). "Alzheimer's disease classification using 3D conditional progressive GAN-and LDA-based data selection". Signal, Image and Video Processing. 18 (2): 1847–1861.
- ^ Preisner, O; Guiomar, R; Machado, J; Menezes, JC; Lopes, JA (2010). "Application of Fourier transform infrared spectroscopy and chemometrics for differentiation of Salmonella enterica serovar Enteritidis phage types". Appl Environ Microbiol. 76 (11): 3538–3544. Bibcode:2010ApEnM..76.3538P. doi:10.1128/aem.01589-09. PMC 2876429. PMID 20363777.
- ^ David, DE; Lynne, AM; Han, J; Foley, SL (2010). "Evaluation of virulence factor profiling in the characterization of veterinary Escherichia coli isolates". Appl Environ Microbiol. 76 (22): 7509–7513. Bibcode:2010ApEnM..76.7509D. doi:10.1128/aem.00726-10. PMC 2976202. PMID 20889790.
- ^ Tahmasebi, P.; Hezarkhani, A.; Mortazavi, M. (2010). "Application of discriminant analysis for alteration separation; sungun copper deposit, East Azerbaijan, Iran. Australian" (PDF). Journal of Basic and Applied Sciences. 6 (4): 564–576.
- ^ Trevor Hastie; Robert Tibshirani; Jerome Friedman. The Elements of Statistical Learning. Data Mining, Inference, and Prediction (second ed.). Springer. p. 128.
- ^ Kainen P.C. (1997) Utilizing geometric anomalies of high dimension: When complexity makes computation easier. In: Kárný M., Warwick K. (eds) Computer Intensive Methods in Control and Signal Processing: The Curse of Dimensionality, Springer, 1997, pp. 282–294.
- ^ Donoho, D., Tanner, J. (2009) Observed universality of phase transitions in high-dimensional geometry, with implications for modern data analysis and signal processing, Phil. Trans. R. Soc. A 367, 4273–4293.
- ^ Gorban, Alexander N.; Golubkov, Alexander; Grechuck, Bogdan; Mirkes, Evgeny M.; Tyukin, Ivan Y. (2018). "Correction of AI systems by linear discriminants: Probabilistic foundations". Information Sciences. 466: 303–322. arXiv:1811.05321. doi:10.1016/j.ins.2018.07.040. S2CID 52876539.
- ^ Guédon, O., Milman, E. (2011) Interpolating thin-shell and sharp large-deviation estimates for isotropic log-concave measures, Geom. Funct. Anal. 21 (5), 1043–1068.
- ^ Gorban, Alexander N.; Makarov, Valeri A.; Tyukin, Ivan Y. (July 2019). "The unreasonable effectiveness of small neural ensembles in high-dimensional brain". Physics of Life Reviews. 29: 55–88. arXiv:1809.07656. Bibcode:2019PhLRv..29...55G. doi:10.1016/j.plrev.2018.09.005. PMID 30366739.
Further reading
[edit]- Duda, R. O.; Hart, P. E.; Stork, D. H. (2000). Pattern Classification (2nd ed.). Wiley Interscience. ISBN 978-0-471-05669-0. MR 1802993.
- Hilbe, J. M. (2009). Logistic Regression Models. Chapman & Hall/CRC Press. ISBN 978-1-4200-7575-5.
- Mika, S.; et al. (1999). "Fisher discriminant analysis with kernels". Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No.98TH8468). pp. 41–48. CiteSeerX 10.1.1.35.9904. doi:10.1109/NNSP.1999.788121. ISBN 978-0-7803-5673-3. S2CID 8473401.
{{cite book}}
: CS1 maint: date and year (link) - McFarland, H. Richard; Donald, St. P. Richards (2001). "Exact Misclassification Probabilities for Plug-In Normal Quadratic Discriminant Functions. I. The Equal-Means Case". Journal of Multivariate Analysis. 77 (1): 21–53. doi:10.1006/jmva.2000.1924.
- McFarland, H. Richard; Donald, St. P. Richards (2002). "Exact Misclassification Probabilities for Plug-In Normal Quadratic Discriminant Functions. II. The Heterogeneous Case". Journal of Multivariate Analysis. 82 (2): 299–330. doi:10.1006/jmva.2001.2034.
- Haghighat, M.; Abdel-Mottaleb, M.; Alhalabi, W. (2016). "Discriminant Correlation Analysis: Real-Time Feature Level Fusion for Multimodal Biometric Recognition". IEEE Transactions on Information Forensics and Security. 11 (9): 1984–1996. doi:10.1109/TIFS.2016.2569061. S2CID 15624506.
External links
[edit]- Discriminant Correlation Analysis (DCA) of the Haghighat article (see above)
- ALGLIB contains open-source LDA implementation in C# / C++ / Pascal / VBA.
- LDA in Python- LDA implementation in Python
- LDA tutorial using MS Excel
- Biomedical statistics. Discriminant analysis
- StatQuest: Linear Discriminant Analysis (LDA) clearly explained on YouTube
- Course notes, Discriminant function analysis by G. David Garson, NC State University
- Discriminant analysis tutorial in Microsoft Excel by Kardi Teknomo
- Course notes, Discriminant function analysis by David W. Stockburger, Missouri State University Archived 2016-03-03 at the Wayback Machine
- Discriminant function analysis (DA) by John Poulsen and Aaron French, San Francisco State University