SMHS BigDataBigSci CrossVal LDA QDA - Revision history

Dinov: /* QDA (Quadratic Discriminant Analysis) */

2020-10-19T17:14:34Z

‎QDA (Quadratic Discriminant Analysis)

Dinov: /* Big Data Science and Cross Validation - Foundation of LDA and QDA for prediction, dimensionality reduction or forecasting */

2020-10-19T16:56:12Z

‎Big Data Science and Cross Validation - Foundation of LDA and QDA for prediction, dimensionality reduction or forecasting

Dinov at 14:52, 19 October 2020

2020-10-19T14:52:52Z

Show changes

Pineaumi: /* See also */

2016-05-24T14:09:20Z

‎See also

Pineaumi: /* Appendix: R Debugging */

2016-05-16T16:35:49Z

‎Appendix: R Debugging

Pineaumi: /* k-Means Clustering (k-MC) */

2016-05-16T16:30:57Z

‎k-Means Clustering (k-MC)

Pineaumi: /* k-Nearest Neighbors algorithm */

2016-05-16T16:29:11Z

‎k-Nearest Neighbors algorithm

Pineaumi: /* k-Nearest Neighbors algorithm */

2016-05-16T16:23:04Z

‎k-Nearest Neighbors algorithm

Pineaumi: /* Summary */

2016-05-16T16:18:31Z

‎Summary

Pineaumi: /* Summary */

2016-05-16T16:17:15Z

‎Summary

@@ Line 98: / Line 98: @@
 ==QDA (Quadratic Discriminant Analysis)==
   predfun.qda = function(train.x, train.y, test.x, test.y, neg)
+  {

← Older revision		Revision as of 16:56, 19 October 2020
Line 38:		Line 38:

	But, in the more general, <u>quadratic case</u> of QDA, there are no assumptions on the covariance matrices \(Σ_k\) of the Gaussians, leading to quadratic decision surfaces.		But, in the more general, <u>quadratic case</u> of QDA, there are no assumptions on the covariance matrices \(Σ_k\) of the Gaussians, leading to quadratic decision surfaces.
		+
		+	=== Mathematical formulation===
		+	Fisher's [http://en.wikipedia.org/wiki/Linear_discriminant_analysis Linear discriminant analysis] is a technique aiming to identify a ''linear combination of features'' characterizing, or separating, two or more groups of objects. The linear combination can be used to predict, or linearly classify, heterogeneous datasets.
		+
		+	LDA is related to [http://www.socr.umich.edu/people/dinov/courses/DSPA_notes/05_DimensionalityReduction.html principal component analysis, independent component analysis, and factor analysis]. However, LDA explicitly attempts to model the difference between the binary or multinomial classes, where LDA works with continuous independent variables. Discriminant correspondence analysis represents the analogue when dealing with categorical independent variables.
		+
		+	For simplicity, assume we are looking for a binary classification of an outcome variable,
		+	\(\mathcal{Y}=\{0, 1\}\). The ''decision boundary of the Bayes classifier'', representing a linear hyperplance separating the objects in the two possible outcome labels/classes, is defined as
		+	\(D(h^*)=\{x: P(Y=1\|X=x)=P(Y=0\|X=x)\}\).
		+
		+	To explicitly derive the Bayes classifier's decision boundary, we will denote the likelihood function by
		+	\( P(X=x\|Y=y) = f_y(x) \) and the prior probability \( P(Y=y) = \pi_y \).
		+
		+	LDA assumes that both outcome classes have [http://en.wikipedia.org/wiki/Multivariate_normal_distribution multivariate normal (Gaussian) distributions] and share the same variance-covariance matrix \(\Sigma\).
		+
		+	Then, \( P(X=x\|Y=y) = f_y(x) = \frac{1}{ (2\pi)^{d/2}\|\Sigma\|^{1/2} }\exp\left( -\frac{1}{2} (x - \mu_k)^\top \Sigma^{-1} (x - \mu_k) \right)\), and we can derive the Bayes classifier decision boundary by
		+	setting \( P(Y=1\|X=x) = P(Y=0\|X=x) \) and expanding this equation as follows:
		+
		+	:: \(Pr(Y=1\|X=x)=Pr(Y=0\|X=x)\)
		+	:: \( \frac{Pr(X=x\|Y=1)Pr(Y=1)}{Pr(X=x)}=\frac{Pr(X=x\|Y=0)Pr(Y=0)}{Pr(X=x)}\) (Bayes' rule)
		+	:: \( Pr(X=x\|Y=1)Pr(Y=1)=Pr(X=x\|Y=0)Pr(Y=0)\) (canceling the denominators)
		+	:: \( f_1(x)\pi_1=f_0(x)\pi_0\)
		+	:: \( \frac{1}{ (2\pi)^{d/2}\|\Sigma\|^{1/2} }\exp\left( -\frac{1}{2} (x - \mu_1)^\top \Sigma^{-1} (x - \mu_1) \right)\pi_1=\frac{1}{ (2\pi)^{d/2}\|\Sigma\|^{1/2} }\exp\left( -\frac{1}{2} (x - \mu_0)^\top \Sigma^{-1} (x - \mu_0) \right)\pi_0\)
		+	:: \( \exp\left( -\frac{1}{2} (x - \mu_1)^\top \Sigma^{-1} (x - \mu_1) \right)\pi_1=\exp\left( -\frac{1}{2} (x - \mu_0)^\top \Sigma^{-1} (x - \mu_0) \right)\pi_0\)
		+	:: \( -\frac{1}{2} (x - \mu_1)^\top \Sigma^{-1} (x - \mu_1) + \log(\pi_1)=-\frac{1}{2} (x - \mu_0)^\top \Sigma^{-1} (x - \mu_0) +\log(\pi_0)\) (log-transform to both sides)
		+	:: \( \log(\frac{\pi_1}{\pi_0})-\frac{1}{2}\left( x^\top\Sigma^{-1}x + \mu_1^\top\Sigma^{-1}\mu_1 - 2x^\top\Sigma^{-1}\mu_1 - x^\top\Sigma^{-1}x - \mu_0^\top\Sigma^{-1}\mu_0 + 2x^\top\Sigma^{-1}\mu_0 \right)=0\) (expand)
		+	:: \( \log(\frac{\pi_1}{\pi_0})-\frac{1}{2}\left( \mu_1^\top\Sigma^{-1}
		+	\mu_1-\mu_0^\top\Sigma^{-1}\mu_0 - 2x^\top\Sigma^{-1}(\mu_1-\mu_0) \right)=0.\) (cancel and factoring the terms).
		+
		+	Therefore, the Bayes's classifier's decision boundary \(D(h^*)\) is linear \(ax+b=0\) in terms of hte argument \(x\).
		+
		+	The 2-class LDA classification formulation can be generalized to multinomial classification using \(k \ge 2\) classes. For two class indices, the corresponding \(m\)-class to \(n\)-class (Bayes classifier's decision) boundary will be
		+	\( D(h^*)=\log(\frac{\pi_m}{\pi_n})-\frac{1}{2}\left( \mu_m^\top\Sigma^{-1}
		+	\mu_m-\mu_n^\top\Sigma^{-1}\mu_n - 2x^\top\Sigma^{-1}(\mu_m-\mu_n) \right)=0\).
		+
		+	In the special case when both classes have the same number of samples, the resulting decision boundary would represent a linear hyperplane exactly halfway between the mean centers of the sample points in the pair of classes.
		+
		+	Although the assumption under LDA may not hold true for most real-world data, it nevertheless usually performs quite well in practice, where it often provides near-optimal classifications. For instance, the Z-Score credit risk model that was designed by Edward Altman in 1968 and [http://pages.stern.nyu.edu/~ealtman/Zscores.pdf revisited in 2000], is essentially a weighted LDA. This model has demonstrated a 85-90% success rate in predicting bankruptcy, and for this reason it is still in use today.
		+
		+	LDA assumptions include:
		+
		+	* The data in each class has a Gaussian distribution,
		+	* The mean rather than the variance is the discriminating factor, and
		+	* In certain situations, LDA may over-fit the training data.

	==LDA (Linear Discriminant Analysis)==		==LDA (Linear Discriminant Analysis)==

@@ Line 303: / Line 303: @@
 * [[SMHS_BigDataBigSci_SEM| Structural Equation Modeling (SEM)]]
 * [[SMHS_BigDataBigSci_GCM| Growth Curve Modeling (GCM)]]
-* [[SMHS_BigDataBigSci_GEE| Generalized Estimating Equation (GEE) Modeling]]
+* [[SMHS_BigDataBigSci_GCM| Generalized Estimating Equation (GEE) Modeling]]
 * [[SMHS_BigDataBigSci|Back to Big Data Science]]

@@ Line 202: / Line 202: @@
 '''Example''': compute sum of squared error SS
-&#35;&#35; compute sum of squares
+ &#35;&#35; compute sum of squares
   SS<-function(mu,x) {  d<-x-mu; d2<-d^2; ss<-sum(d2);  ss }
   set.seed(100);  x<-rnorm(100); SS(1,x)
-&#35;&#35; to debug
+ &#35;&#35; to debug
   debug(SS); SS(1,x)
   debugging in: SS(1, x) debug: {
@@ Line 261: / Line 261: @@
   <i>In log(z) : NaNs produced</font></i>
-But, we can also label g and h for debugging when we debug ''f''
+But, we can also label ''g'' and ''h'' for debugging when we debug ''f''
   f(-1)

@@ Line 160: / Line 160: @@
   summary(knn_model)
   attributes(knn_model)
-−
   &#35;cross-validation
   knn_model.cv <- knn.cv(train= input.train, cl=as.factor(output.train), k=2)

@@ Line 67: / Line 67: @@
 k-Nearest Neighbors algorithm (''k''-NN) is a non-parametric method for either classification or regression, where the <u>input</u> consists of the ''k'' closest '''training examples''' in the feature space, but the <u>output</u> depends on whether ''k''-NN is used for classification or regression:
-*In k-NN classification, the output is a class membership (labels). Objects in the testing data are classified by a majority vote of their neighbors. Each object is assigned to a class that is most common among its k nearest neighbors (k is always a small positive integer). When k=1, then an object is assigned to the class of its single nearest neighbor.
+*In ''k''-NN '''classification''', the output is a class membership (labels). Objects in the testing data are classified by a majority vote of their neighbors. Each object is assigned to a class that is most common among its ''k'' nearest neighbors (''k'' is always a small positive integer). When ''k''=1, then an object is assigned to the class of its single nearest neighbor.
-*In k-NN regression, the output is the property value for the object representing the average of the values of its k nearest neighbors.
+*In ''k''-NN '''regression''', the output is the property value for the object representing the average of the values of its ''k'' nearest neighbors.
 &#35;X = as.matrix(input) &nbsp;&nbsp;&nbsp; # Predictor variables X = as.matrix(input.short2)
@@ Line 78: / Line 78: @@
 <u>'''&#35;KNN (k-nearest neighbors)'''</u>
- library("class")
+     library("class")
- <span style="background-color: #32cd32">&#35;knn.fit.test <- knn(X, X, cl = Y, k=3, prob=F); predict(as.matrix(knn.fit.test), X) $\$$class </span>
+     <span style="background-color: #32cd32">&#35;knn.fit.test <- knn(X, X, cl = Y, k=3, prob=F); predict(as.matrix(knn.fit.test), X) $\$$class </span>
- <span style="background-color: #32cd32">&#35;table(knn.fit.test, Y); confusionMatrix(Y, knn.fit.test, negative="1")</span>
+     <span style="background-color: #32cd32">&#35;table(knn.fit.test, Y); confusionMatrix(Y, knn.fit.test, negative="1")</span>
- <span style="background-color: #32cd32">&#35;This can be used for polytomous variable (multiple classes)</span>
+     <span style="background-color: #32cd32">&#35;This can be used for polytomous variable (multiple classes)</span>
-−
- predfun.knn = function(train.x, train.y, test.x, test.y, neg)
+     predfun.knn = function(train.x, train.y, test.x, test.y, neg)
-−
+     {
- require("class")
+     require("class")
- knn.fit = knn(train.x, test.x, cl = train.y, prob=T) 	<span style="background-color: #32cd32"># knn is already a prediction function!!!</span>
+     knn.fit = knn(train.x, test.x, cl = train.y, prob=T) 	<span style="background-color: #32cd32"># knn is already a prediction function!!!</span>
- &#35;ynew = predict(knn.fit, test.x)$\$$class			# no need of another prediction, in this case
+     &#35;ynew = predict(knn.fit, test.x)$\$$class			# no need of another prediction, in this case
- out.knn = confusionMatrix(test.y, knn.fit, negative=neg)
+     out.knn = confusionMatrix(test.y, knn.fit, negative=neg)
- return( out.knn )
+     return( out.knn )
-−
+     }
   <span style="background-color: #32cd32">cv.out.knn = '''crossval::crossval'''(predfun.knn, X, Y, K=5, B=2, neg="1")</span>
@@ Line 108: / Line 108: @@
   sample_size <- floor(0.75 * nrow(input))
+ &#35;&#35;set the seed to make your partition reproducible
   set.seed(1234)
   input.train.ind <- sample(seq_len(nrow(input)), size = sample_size)

@@ Line 36: / Line 36: @@
 $LOR=log\Big(\frac{P(Y=k│X}{P(Y=l│X)}\Big)=0 ⇔ (\mu_k-\mu_l)^T\sum^{-1}(\mu_k-\mu_1)=\frac{1}{2}({\mu_k}^T\sum^{-1}\mu_k-{\mu_l}^T\sum^{-1}\mu_l) $
+But, in the more general, <u>quadratic case</u> of QDA, there are no assumptions on the covariance matrices $Σ_k$ of the Gaussians, leading to quadratic decision surfaces.
 ==LDA (Linear Discriminant Analysis)==

@@ Line 34: / Line 34: @@
 (the LOR=0 ↔the two probabilities are identical, i.e., same class)
-$LOR=log\Big(\frac{P(Y=k│X}{P(Y=l│X)}\Big)=0 ⇔ (\mu_k-\mu_l)^T\sum^{-1} $
+$LOR=log\Big(\frac{P(Y=k│X}{P(Y=l│X)}\Big)=0 ⇔ (\mu_k-\mu_l)^T\sum^{-1}(\mu_k-\mu_1)=\frac{1}{2}({\mu_k}^T\sum^{-1}\mu_k-{\mu_l}^T\sum^{-1}\mu_l) $
-LOR=log((P(Y=k│X))/(P(Y=l│X) ))=0  □(⇔┬ )  (μ_k-μ_l )^T Σ^(-1) (μ_k-μ_l )=1/2 (〖μ_k〗^T Σ^(-1) μ_k-〖μ_l〗^T Σ^(-1) μ_l )  <mark> FIX FORMULAS!! </mark>
+〖μ_k〗^T Σ^(-1)