SMHS BigDataBigSci CrossVal - Revision history

Pineaumi: /* See also */

2016-05-24T14:08:48Z

‎See also

Dinov: /* Example (Linear Regression) */

2016-05-21T16:58:45Z

‎Example (Linear Regression)

Dinov: /* Example (Linear Regression) */

2016-05-21T16:56:22Z

‎Example (Linear Regression)

Pineaumi: /* Overfitting */

2016-05-20T13:25:22Z

‎Overfitting

Pineaumi: /* Questions */

2016-05-20T13:24:50Z

‎Questions

Pineaumi: /* Overfitting */

2016-05-16T17:18:37Z

‎Overfitting

Pineaumi: /* Overfitting */

2016-05-16T17:16:57Z

‎Overfitting

Pineaumi: /* Overfitting */

2016-05-16T17:14:47Z

‎Overfitting

Pineaumi: /* Overfitting */

2016-05-16T17:12:59Z

‎Overfitting

Pineaumi: /* Alternative predictor functions */

2016-05-16T14:43:16Z

‎Alternative predictor functions

@@ Line 531: / Line 531: @@
 * [[SMHS_BigDataBigSci_SEM| Structural Equation Modeling (SEM)]]
 * [[SMHS_BigDataBigSci_GCM| Growth Curve Modeling (GCM)]]
-* [[SMHS_BigDataBigSci_GEE| Generalized Estimating Equation (GEE) Modeling]]
+* [[SMHS_BigDataBigSci_GCM| Generalized Estimating Equation (GEE) Modeling]]
 * [[SMHS_BigDataBigSci|Back to Big Data Science]]

@@ Line 111: / Line 111: @@
 $$
-\begin{equation} MSE=\frac{1}{n}\sum_{i=1}^{n} \Bigg(y_i-\underbrace{(\alpha_1+x_{i,1}\beta_1+x_{i,2}\beta_2+\cdots+x_{i,k}\beta_k)}_{(\text{predicted value, } \hat{y_i} \text{, at }\{x_{i,1,}\cdots,x_{i,k}\})} \Bigg)^2, vs.
+\begin{equation} MSE=\frac{1}{n}\sum_{i=1}^{n} \Bigg(y_i-\underbrace{(\alpha_1+x_{i,1}\beta_1+x_{i,2}\beta_2+\cdots+x_{i,k}\beta_k)}_{(\text{predicted value, } \hat{y_i} \text{, at }\{x_{i,1,}\cdots,x_{i,k}\})} \Bigg)^2
 \end{equation}
 $$
+<center>vs.</center>
-\begin{equation} RMSE=\sqrt{\frac{1}{n}\sum_{i=1}^{n} \Bigg(y_i-\frac{(\alpha_1+x_{i,1}\beta_1+x_{i,2}\beta_2+\cdots+x_{i,k}\beta_k)}{(predicted value, \hat y_i, at \{x_{i,1,}\cdots,x_{i,k}\})} \Bigg)^2 . }
+$$
 \end{equation}
+$$
 The expected value of the MSE (over the distribution of training sets) for the <u>'''training set'''</u> is $\frac{(n-k-1)}{(n + k + 1)}  × E,$ where $E$ is the expected value of the MSE for the <u>'''testing'''</u>/<u>'''validation'''</u> data. Therefore, fitting a model and computing the MSE on the training set, we will get an over optimistic evaluation assessment (smaller RMSE) of how well the model may fit another dataset. This bias represents ''in-sample'' estimate of the fit, whereas we are interested in the cross-validation estimate as an ''out-of-sample'' estimate.

@@ Line 100: / Line 100: @@
+$$
+\begin{array}{lcl}
-<mark>PLEASE FIX EQUATION BELOW, VERTICAL LINE NEEDED!!!
+  y_1=\alpha_1+x_{1,1}\beta_1+x_{1,2}\beta_2+...+x_{1,k}\beta_k\\
-<center>
+  y_2=\alpha_2+x_{2,1}\beta_1+x_{2,2}\beta_2+...+x_{2,k}\beta_k \\
-$y_1=\alpha_1+x_{1,1}\beta_1+x_{1,2}\beta_2+...+x_{1,k}\beta_k$
+  ...\\
+  y_n=\alpha_n+x_{n,1}\beta_1+x_{n,2}\beta_2+...+x_{n,k}\beta_k
-$y_2=\alpha_2+x_{2,1}\beta_1+x_{2,2}\beta_2+...+x_{2,k}\beta_k$
+\end{array}$$
 The model fit may be evaluated using the mean squared error (MSE). The MSE for a given value of the parameters ''α'' and ''β'' on the observed training data $\{x_i,y_i\}_{1 ≤ i ≤ n}$.
-<mark> PLEASE FIX FORMULAs BELOW, SUM VALUE PLACEMENT AND DENOMINATOR !! </mark>
+$$
+\begin{equation} MSE=\frac{1}{n}\sum_{i=1}^{n} \Bigg(y_i-\underbrace{(\alpha_1+x_{i,1}\beta_1+x_{i,2}\beta_2+\cdots+x_{i,k}\beta_k)}_{(\text{predicted value, } \hat{y_i} \text{, at }\{x_{i,1,}\cdots,x_{i,k}\})} \Bigg)^2, vs.
 \end{equation}
+$$
 \begin{equation} RMSE=\sqrt{\frac{1}{n}\sum_{i=1}^{n} \Bigg(y_i-\frac{(\alpha_1+x_{i,1}\beta_1+x_{i,2}\beta_2+\cdots+x_{i,k}\beta_k)}{(predicted value, \hat y_i, at \{x_{i,1,}\cdots,x_{i,k}\})} \Bigg)^2 . }

@@ Line 27: / Line 27: @@
 (Figure from http://xkcd.com/1122/)
-[[Image:SMHS BigDataBigSci_CrossVal2.png|400px]]
+<center>[[Image:SMHS BigDataBigSci_CrossVal2.png|400px]]</center>
 '''Example (Google Flu Trends):''' A March 14, 2014 article in Science (''DOI: 10.1126/science.1248506''), identified problems in Google Flu Trends (http://www.google.org/flutrends/about/#US), DOI  10.1371/journal.pone.0023610, which may be attributed in part to overfitting.   In February 2013, Nature reported that GFT was predicting more than double the proportion of doctor visits for influenza-like illness (ILI) than the Centers for Disease Control and Prevention (CDC), despite the fact that GFT was built to predict CDC reports.
@@ Line 51: / Line 51: @@
 *Increase the iteration index and repeat the process (termination criteria can involve a fixed number, or a desired (mean?) variability or error-rate).
-[[Image:SMHS BigDataBigSci_CrossVal3.png|400px]]
+<center>[[Image:SMHS BigDataBigSci_CrossVal3.png|400px]]</center>
 The validation results at each iteration are averaged, to reduce noise/variability, and reported.

@@ Line 5: / Line 5: @@
 * Can we do “pretend” validations that closely mimic reality?
-[[Image:SMHS_BigDataBigSci_CrossVal1.png|250px]]
+<center>[[Image:SMHS_BigDataBigSci_CrossVal1.png|250px]]</center>
 ''Validation'' is the scientific process of determining the degree of accuracy of a mathematical, analytic or computational model as a representation of the real world based on the intended model use. There are various challenges with using observed experimental data for model validation:

@@ Line 49: / Line 49: @@
 *Validating the analysis on the testing subset
-*Increase the iteration index and repeat the process (termination criteria can involve a fixed number), or a desired (mean?) variability or error-rate.
+*Increase the iteration index and repeat the process (termination criteria can involve a fixed number, or a desired (mean?) variability or error-rate).
 [[Image:SMHS BigDataBigSci_CrossVal3.png|400px]]

@@ Line 45: / Line 45: @@
 *Randomly partitioning a sample of data into 2 complementary subsets (training + testing),
-*Performing the analysis on the training subset,
+*Performing the analysis on the training subset
 *Validating the analysis on the testing subset

@@ Line 55: / Line 55: @@
 The validation results at each iteration are averaged, to reduce noise/variability, and reported.
-Cross-validation guards against testing hypotheses suggested by the data themselves (aka as "Type III errors", False-Suggestion) in cases when new observations are hard to obtain (due to costs, reliability, time or other constraints).
+Cross-validation guards against testing hypotheses suggested by the data themselves (aka: "Type III errors", False-Suggestion) in cases when new observations are hard to obtain (due to costs, reliability, time or other constraints).
 Cross-validation is different from ''conventional-validation'' (e.g. 80%-20% partitioning the data set into training and testing subsets) as the in the conventional validation, the error (e.g. Root Mean Square Error, RMSE) on the training data is not a useful estimator of model performance, as it does not generalize across multiple samples. Errors of the conventional-valuation based on the results on the test data do not assess model performance, in general. A more fair way to properly estimate model prediction performance is to use cross-validation, which combines (averages) prediction errors or measures of fit to correct for the stochastic nature of training and testing data partitions and generate a more accurate and robust estimate of real model performance.

@@ Line 472: / Line 472: @@
   ynew <- predict(lm.logit, as.data.frame(X)); plot(ynew)
   ynew2 <- ifelse(exp(ynew)<0.5, 0, 1); plot(ynew2)
-−
   '''predfun.logit = function(train.x, train.y, test.x, test.y, neg)'''
-  { lm.logit <- glm(train.y ~ ., data = train.x, family = "binomial")
+  {      lm.logit <- glm(train.y ~ ., data = train.x, family = "binomial")
-   ynew = predict(lm.logit, test.x )
+        ynew = predict(lm.logit, test.x )
-   &#35;compute TP, FP, TN, FN
+        &#35;compute TP, FP, TN, FN
-   <span style="background-color: #32cd32">ynew2</span> <- ifelse(exp(ynew)<0.5, 0, 1)
+        <span style="background-color: #32cd32">ynew2</span> <- ifelse(exp(ynew)<0.5, 0, 1)
-   out = confusionMatrix(test.y, ynew2, negative=neg)	# Binary outcome, we can use confusionMatrix
+        out = confusionMatrix(test.y, ynew2, negative=neg)	# Binary outcome, we can use confusionMatrix
-   return( out )
+        return( out )
+  }
@@ Line 485: / Line 485: @@
   input.short <- input[, which(names(input) %in% c("R_fusiform_gyrus_Volume",
-     "R_fusiform_gyrus_ShapeIndex", "R_fusiform_gyrus_Curvedness",
+        "R_fusiform_gyrus_ShapeIndex", "R_fusiform_gyrus_Curvedness",
-     "Sex",  "Weight", "Age" , "chr12_rs34637584_GT", "chr17_rs11868035_GT",
+        "Sex",  "Weight", "Age" , "chr12_rs34637584_GT", "chr17_rs11868035_GT",
-     "UPDRS_Part_I_Summary_Score_Baseline", "UPDRS_Part_I_Summary_Score_Month_03",
+        "UPDRS_Part_I_Summary_Score_Baseline", "UPDRS_Part_I_Summary_Score_Month_03",
-     "UPDRS_Part_II_Patient_Questionnaire_Summary_Score_Baseline",
+        "UPDRS_Part_II_Patient_Questionnaire_Summary_Score_Baseline",
-     "UPDRS_Part_III_Summary_Score_Baseline",
+        "UPDRS_Part_III_Summary_Score_Baseline",
-     "X_Assessment_Non.Motor_Epworth_Sleepiness_Scale_Summary_Score_Baseline"
+        "X_Assessment_Non.Motor_Epworth_Sleepiness_Scale_Summary_Score_Baseline"
     ))]
   X = as.matrix(input.short)
@@ Line 509: / Line 509: @@
       return( out.qda )
+  }
-−
-<mark>cv.out.qda = crossval::crossval(predfun.qda,</mark> as.data.frame(input.short), <mark>as.factor(Y), K=5, B=20, neg="1")</mark>
+ <mark>cv.out.qda = '''crossval::crossval'''(predfun.qda,</mark> as.data.frame(input.short),   <mark>as.factor(Y), K=5, B=20, neg="1")</mark>
   diagnosticErrors(cv.out.lda$\$$stat); diagnosticErrors(cv.out.qda$\$$stat);
@@ Line 519: / Line 518: @@
   input.short2 <- input[, which(names(input) %in%)"R_fusiform_gyrus_Volume",
-     "Sex",  "Weight", "Age" , "chr17_rs11868035_GT",
+        "Sex",  "Weight", "Age" , "chr17_rs11868035_GT",
-     "UPDRS_Part_I_Summary_Score_Baseline",
+        "UPDRS_Part_I_Summary_Score_Baseline",
-     "UPDRS_Part_II_Patient_Questionnaire_Summary_Score_Baseline",
+        "UPDRS_Part_II_Patient_Questionnaire_Summary_Score_Baseline",
-     "UPDRS_Part_III_Summary_Score_Baseline",
+        "UPDRS_Part_III_Summary_Score_Baseline",
-     "X_Assessment_Non.Motor_Epworth_Sleepiness_Scale_Summary_Score_Baseline"
+        "X_Assessment_Non.Motor_Epworth_Sleepiness_Scale_Summary_Score_Baseline"
- ))]
+     ))]
   X = as.matrix(input.short2)
   cv.out.qda = crossval::crossval(predfun.qda, as.data.frame(X), as.numeric(Y), K=5, B=2, neg="1")