AP Statistics Curriculum 2007 GLM MultLin - Revision history

Tdlee: Text replacement - "{{translate|pageName=http://wiki.stat.ucla.edu/socr/" to ""{{translate|pageName=http://wiki.socr.umich.edu/"

2020-03-03T16:35:37Z

Text replacement - "{{translate|pageName=http://wiki.stat.ucla.edu/socr/" to ""{{translate|pageName=http://wiki.socr.umich.edu/"

Jslavine: /* Parameter Estimation in Multilinear Regression */

2014-10-03T22:42:44Z

‎Parameter Estimation in Multilinear Regression

IvoDinov: /* Categorical Variables in Multiple Regression */

2013-05-02T00:02:23Z

‎Categorical Variables in Multiple Regression

IvoDinov: /* Categorical Variables in Multiple Regression */

2013-05-02T00:01:10Z

‎Categorical Variables in Multiple Regression

IvoDinov: /* Categorical Variables in Multiple Regression */

2013-05-01T23:59:40Z

‎Categorical Variables in Multiple Regression

IvoDinov at 22:57, 1 May 2013

2013-05-01T22:57:54Z

IvoDinov: /* Examples */

2011-03-04T16:51:50Z

‎Examples

Jenny at 20:56, 28 June 2010

2010-06-28T20:56:03Z

Jenny at 17:35, 28 June 2010

2010-06-28T17:35:23Z

IvoDinov: added a link to the Problems set

2009-10-26T05:22:25Z

added a link to the Problems set

← Older revision		Revision as of 16:35, 3 March 2020
Line 238:		Line 238:
	* SOCR Home page: http://www.socr.ucla.edu		* SOCR Home page: http://www.socr.ucla.edu

−	{{translate\|pageName=http://wiki.~~stat~~.~~ucla~~.edu~~/socr~~/index.php?title=AP_Statistics_Curriculum_2007_GML_MultLin}}	+	"{{translate\|pageName=http://wiki.socr.umich.edu/index.php?title=AP_Statistics_Curriculum_2007_GML_MultLin}}

@@ Line 38: / Line 38: @@
 :<math>\frac{(n-p-1)\hat\sigma^2}{\sigma^2} \sim \chi_{n-p-1}^2</math>
-* '''Interval Estimates''': The <math>100(1-\alpha)% </math> [[AP_Statistics_Curriculum_2007_Estim_S_Mean#Interval_Estimation_of_a_Population_Mean | confidence interval]] for the parameter, <math>\beta_i </math>, is computed as follows:
+* '''Interval Estimates''': The <math>100(1-\alpha)\% </math> [[AP_Statistics_Curriculum_2007_Estim_S_Mean#Interval_Estimation_of_a_Population_Mean | confidence interval]] for the parameter, <math>\beta_i </math>, is computed as follows:
 :<math> {\widehat \beta_i  \pm t_{\frac{\alpha }{2},n - p - 1} \hat \sigma \sqrt {(\mathbf{X}^T \mathbf{X})_{ii}^{ - 1} } }  </math>,

@@ Line 165: / Line 165: @@
 An independent contrast is a contrast that is not a linear combination of any other set of contrasts. Any set of independent contrasts would work equally well if the end result was the simultaneous test of the five contrasts, as in an ANOVA. One of the many possible examples is presented below.
 <center>
@@ Line 191: / Line 185: @@
 This solution would not be ideal, however, because there is considerable information available by setting the contrasts to test specific hypotheses. The levels of the categorical variable generally dictate the structure of the contrasts. In the example study, it makes sense to contrast the two control groups (1 and 2) with the other four experimental groups (3 and 4). Any two numbers would work, one assigned to groups 1 and 2 and the others assigned to the other four groups, but it is conventional to have the sum of the contrasts equal to zero. One contrast that meets this criterion would be (-2, -2, 1, 1).
 Generally it is easiest to set up contrasts within subgroups of the first contrast. For example, a second contrast might test whether there are differences between the two control groups. This contrast would appear as (1, -1, 0, 0). As can be seen, this would be a contrast within the experimental treatment groups. Within the non-drug treatment, a contrast comparing Group 3 with Group 4 might be appropriate (0, 0, 1, -1). Combined, the contrasts are given in the following table.
- 	 	Dummy Coded Variables
 <center>
 {| class="wikitable" style="text-align:center; width:75%" border="1"

@@ Line 100: / Line 100: @@
 Adding variables to a linear regression model will always increase the (unadjusted) \(R^2\) value. If additional predictor variables are correlated with the predictor variables already in the model, then the combined results may be difficult to predict. In some cases, the combined result will only slightly improve the prediction, whereas in other cases, a much better prediction may be obtained by combining two correlated variables.
-If the additional predictor variables are uncorrelated (their correlation is trivial) with the predictor variables already in the model, then the result of adding additional variables to the regression model is easy to predict. Thus, the \(R^2\) change will be equal to the correlation coefficient squared between the added variable and predicted variable. In this case it makes no difference what order the predictor variables are entered into the prediction model. For example, if \(X_1\) and \(X_2\) were uncorrelated \(\rho_{12} = 0\) and \(r^2_{1y) = 0.4\) and \(r_2_{2y} = 0.5\), then \(R^2\) for \(X_1\) and \(X_2\) would equal \(0.4 + 0.5 = 0.9\). The value for \(R^2\) change for \(X_2\), given that X1 is included in the model, would be 0.5. The value for \(R^2\) change for \(X_2\) given no variable was in the model would be 0.5. It would make no difference at what stage \(X_2\) was entered into the model, the value for \(R^2\) change would always be 0.5. Similarly, the \(R^2\) change value for \(X_1\) would always be 0.4. Because of this relationship, uncorrelated predictor variables will be preferred, when possible.
+If the additional predictor variables are uncorrelated (their correlation is trivial) with the predictor variables already in the model, then the result of adding additional variables to the regression model is easy to predict. Thus, the \(R^2\) change will be equal to the correlation coefficient squared between the added variable and predicted variable. In this case it makes no difference what order the predictor variables are entered into the prediction model. For example, if \(X_1\) and \(X_2\) were uncorrelated \(\rho_{12} = 0\) and \(\rho^2_{1y} = 0.4\) and \(\rho^2_{2y} = 0.5\), then \(R^2\) for \(X_1\) and \(X_2\) would equal \(0.4 + 0.5 = 0.9\). The value for \(R^2\) change for \(X_2\), given that X1 is included in the model, would be 0.5. The value for \(R^2\) change for \(X_2\) given no variable was in the model would be 0.5. It would make no difference at what stage \(X_2\) was entered into the model, the value for \(R^2\) change would always be 0.5. Similarly, the \(R^2\) change value for \(X_1\) would always be 0.4. Because of this relationship, uncorrelated predictor variables will be preferred, when possible.
 Look at the [[SOCR_Data_AD_BiomedBigMetadata| Modeling and Analysis of Clinical, Genetic and Imaging Data of Alzheimer’s Disease dataset]]. It is fairly clear that 	'''DX_Conversion''' could be directly entered into a regression model predicting '''MMSE''', because it is dichotomous. The problem is how to deal with the two categorical predictor variables with more than two levels (e.g., '''GDTOTAL''').

← Older revision		Revision as of 23:59, 1 May 2013
Line 91:		Line 91:

	===Categorical Variables in Multiple Regression===		===Categorical Variables in Multiple Regression===
		+
		+	When using categorical variables with more than two levels in a multiple regression modeling, we need to make sure the results are correctly interpreted. For categorical variables with more than 2 levels, a number of separate, dichotomous variables need to be created – this is called “dummy coding” or “dummy variables”.
		+
		+	Dichotomous categorical predictor variables, variables with two levels, may be directly entered as predictor or predicted variables in a multiple regression model. Their use in multiple regression is a direct extension of their use in simple linear regression. The interpretation of their regression weights depends upon how the variables are coded. If dichotomous variables are coded as 0 and 1, their regression weights are added or subtracted to the predicted value of the response variable, Y, depending upon whether it is positive or negative.
		+
		+	If a dichotomous variable is coded as -1 and 1, then if the regression weight is positive, it is subtracted from the group coded as -1 and added to the group coded as 1. If a regression weight is negative, then addition and subtraction is reversed. Dichotomous variables are also included in hypothesis tests for R2 change like any other quantitative variable.
		+
		+	Adding variables to a linear regression model will always increase the (unadjusted) \(R^2\) value. If additional predictor variables are correlated with the predictor variables already in the model, then the combined results may be difficult to predict. In some cases, the combined result will only slightly improve the prediction, whereas in other cases, a much better prediction may be obtained by combining two correlated variables.
		+
		+	If the additional predictor variables are uncorrelated (their correlation is trivial) with the predictor variables already in the model, then the result of adding additional variables to the regression model is easy to predict. Thus, the \(R^2\) change will be equal to the correlation coefficient squared between the added variable and predicted variable. In this case it makes no difference what order the predictor variables are entered into the prediction model. For example, if \(X_1\) and \(X_2\) were uncorrelated \(\rho_{12} = 0\) and \(r^2_{1y) = 0.4\) and \(r_2_{2y} = 0.5\), then \(R^2\) for \(X_1\) and \(X_2\) would equal \(0.4 + 0.5 = 0.9\). The value for \(R^2\) change for \(X_2\), given that X1 is included in the model, would be 0.5. The value for \(R^2\) change for \(X_2\) given no variable was in the model would be 0.5. It would make no difference at what stage \(X_2\) was entered into the model, the value for \(R^2\) change would always be 0.5. Similarly, the \(R^2\) change value for \(X_1\) would always be 0.4. Because of this relationship, uncorrelated predictor variables will be preferred, when possible.
		+
		+	Look at the [[SOCR_Data_AD_BiomedBigMetadata\| Modeling and Analysis of Clinical, Genetic and Imaging Data of Alzheimer’s Disease dataset]]. It is fairly clear that '''DX_Conversion''' could be directly entered into a regression model predicting '''MMSE''', because it is dichotomous. The problem is how to deal with the two categorical predictor variables with more than two levels (e.g., '''GDTOTAL''').
		+
		+	'''Dummy Coding''' refers to making many dichotomous variables out of one multilevel categorical variable. Because categorical predictor variables cannot be entered directly into a regression model and be meaningfully interpreted, some other method of dealing with information of this type must be developed. In general, a categorical variable with k levels will be transformed into k-1 variables each with two levels. For example, if a categorical variable had 4 levels, then 3 dichotomous (dummy) variables could be constructed that would contain the same information as the single categorical variable. Dichotomous variables have the advantage that they can be directly entered into the regression model. The process of creating dichotomous variables from categorical variables is called dummy coding.
		+
		+	Depending upon how the dichotomous variables are constructed, additional information can be gleaned from the analysis. In addition, careful construction will result in uncorrelated dichotomous variables. These variables have the advantage of simplicity of interpretation and are preferred to correlated predictor variables.
		+
		+	For example, when using dummy coding with three levels, the simplest case of dummy coding is when the categorical variable has three levels and is converted to two dichotomous variables. School in the example data has three levels, 1=Math, 2=Biology, and 3=Engineering. This variable could be dummy coded into two variables, one called Math and one called Biology. If School = 1, then Math would be coded with a 1 and Biology with a 0. If School=2, then Math would be coded with a 0 and Biology would be coded with a 1. If School=3, then both Math and Biology would be coded with a 0. The dummy coding is represented below.
		+
		+	<center>
		+	{\| class="wikitable" style="text-align:center; width:75%" border="1"
		+	\|+ Dummy Coded Variables
		+	\|-
		+	\| colspan=2\|Subject\|\|Math\|\|Biology
		+	\|-
		+	\| Math\|\|1\|\|1\|\|0
		+	\|-
		+	\| Biology\|\|2\|\|0\|\|1
		+	\|-
		+	\| Engineering\|\|3\|\|0\|\|0
		+	\|}</center>
		+
		+	See the [[SOCR_EduMaterials_AnalysisActivities_MLR \|SOCR Regression analysis]] and compare it against [[SOCR_EduMaterials_AnalysisActivities_ANOVA_1\|SOCR ANOVA]].
		+
		+	Dummy Coding into Independent Variables, and the selection of an appropriate set of dummy codes, will result in new variables that are uncorrelated or independent of each other. In the case when the categorical variable has three levels this can be accomplished by creating a new variable where one level of the categorical variable is assigned the value of -2 and the other levels are assigned the value of 1. The signs are arbitrary and may be reversed, that is, values of 2 and -1 would work equally well. The second variable created as a dummy code will have the level of the categorical variable coded as -2 given the value of 0 and the other values recoded as 1 and -1. In all cases the sum of the dummy coded variable will be zero.
		+
		+	We can directly interpret each of the new dummy-coded variables, called a contrast, and compare levels coded with a positive number to levels coded with a negative number. Levels coded with a zero are not included in the interpretation.
		+
		+	For example, School in the example data has three levels, 1=Math, 2=Biology, and 3=Engineering. This variable could be dummy coded into two variables, one called Engineering (comparing the Engineering School with the other two Schools) and one called Math_vs_Bio (comparing Math versus Biology schools). The Engineering contrast would create a variable where all members of the Engineering Department would be given a value of -2 and all members of the other two Schools would be given a value of 1. The Math_vs_Bio contrast would assign a value of 0 to members of the Engineering Department, 1 divided by the number of members of the Math Department to member of the Math Department, and -1 divided by the number of members of the Biology Department to members of the Biology Department. The Math_vs_Bio variable could be coded as 1 and -1 for Math and Biology respectively, but the recoded variable would no longer be uncorrelated with the first dummy coded variable (Engineering). In most practical applications, it makes little difference whether the variables are correlated or not, so the simpler 1 and -1 coding is generally preferred. The contrasts are summarized in the following table.
		+
		+	<center>
		+	{\| class="wikitable" style="text-align:center; width:75%" border="1"
		+	\|+ Dummy Coded Variables
		+	\|-
		+	\| colspan=2\|School\|\|Engineering\|\|Math_vs_Bio
		+	\|-
		+	\| Math\|\|1\|\|1\|\|1/N1 = 1/12= 0.0833
		+	\|-
		+	\| Biology\|\|2\|\|1\|\|-1/N2 = -1/7 = -0.1429
		+	\|-
		+	\| Engineering\|\|3\|\|-2\|\|0
		+	\|}</center>
		+
		+	Note that the correlation coefficient between the two contrasts is zero. The correlation between the Engineering contrast and Salary is -.585 with a squared correlation coefficient of .342. This correlation coefficient has a significance level of .001. The correlation coefficient between the Math_vs_Bio contrast and Salary is -.150 with a squared value of .023.
		+
		+	Generate the corresponding [[AP_Statistics_Curriculum_2007_ANOVA_1Way#ANOVA_Table\|SOCR ANOVA table]].
		+
		+	Show that the significance level is identical to the value when each contrast was entered last into the regression model. In this case the Engineering contrast was significant and the Math_vs_Bio contrast was not. The interpretation of these results would be that the Engineering Department was paid significantly more than the Math and Biology Schools, but that no significant differences in salary were found between the Math and Biology Schools.
		+	If a categorical variable had four levels, three dummy coded contrasts would be necessary to use the categorical variable in a regression analysis. For example, suppose that a researcher at a pain center did a study with 4 groups of four patients each (N is being deliberately kept small). The dependent measure is subjective experience of pain. The 4 groups consisted of 4 different treatment conditions.
		+
		+	<center>
		+	{\| class="wikitable" style="text-align:center; width:75%" border="1"
		+	! Group\|\|Treatment
		+	\|-
		+	\|1\|\|None
		+	\|-
		+	\|2\|\|Placebo
		+	\|-
		+	\|3\|\|Psychotherapy
		+	\|-
		+	\|4\|\|Acupuncture
		+	\|}</center>
		+
		+	An independent contrast is a contrast that is not a linear combination of any other set of contrasts. Any set of independent contrasts would work equally well if the end result was the simultaneous test of the five contrasts, as in an ANOVA. One of the many possible examples is presented below.
		+	Dummy Coded Variables
		+	Group C1 C2 C3 C4 C5
		+	None 1 0 0 0 0 0
		+	Placebo 2 1 0 0 0 0
		+	Psychotherapy 3 0 1 0 0 0
		+	Acupuncture 4 0 0 1 0 0
		+
		+	<center>
		+	{\| class="wikitable" style="text-align:center; width:75%" border="1"
		+	\|+ Dummy Coded Variables
		+	\|-
		+	\| colspan=2\|Group\|\|C1\|\|C2\|\|C3\|\|C4
		+	\|-
		+	\| None\|\|1\|\|0\|\|0\|\|0\|\|0
		+	\|-
		+	\| Placebo\|\|2\|\|1\|\|0\|\|0\|\|0
		+	\|-
		+	\| Psychotherapy\|\|3\|\|0\|\|1\|\|0\|\|0
		+	\|-
		+	\| Acupuncture\|\|4\|\|0\|\|0\|\|1\|\|0
		+	\|}</center>
		+
		+	Application of this dummy coding in a regression model entering all contrasts in a single block would result in an ANOVA table similar to the one obtained using Means, ANOVA, or General Linear Model.
		+
		+	This solution would not be ideal, however, because there is considerable information available by setting the contrasts to test specific hypotheses. The levels of the categorical variable generally dictate the structure of the contrasts. In the example study, it makes sense to contrast the two control groups (1 and 2) with the other four experimental groups (3 and 4). Any two numbers would work, one assigned to groups 1 and 2 and the others assigned to the other four groups, but it is conventional to have the sum of the contrasts equal to zero. One contrast that meets this criterion would be (-2, -2, 1, 1).
		+	Generally it is easiest to set up contrasts within subgroups of the first contrast. For example, a second contrast might test whether there are differences between the two control groups. This contrast would appear as (1, -1, 0, 0). As can be seen, this would be a contrast within the experimental treatment groups. Within the non-drug treatment, a contrast comparing Group 3 with Group 4 might be appropriate (0, 0, 1, -1). Combined, the contrasts are given in the following table.
		+	Dummy Coded Variables
		+	Group C1 C2 C3
		+	None 1 -2 1 0
		+	Placebo 2 -2 -1 0
		+	Psychotherapy 3 1 0 1
		+	Acupuncture 4 1 0 1
		+	<center>
		+	{\| class="wikitable" style="text-align:center; width:75%" border="1"
		+	\|+ Dummy Coded Variables
		+	\|-
		+	\| colspan=2\|Group\|\|C1\|\|C2\|\|C3
		+	\|-
		+	\| None\|\|1\|\|-2\|\|1\|\|0
		+	\|-
		+	\| Placebo\|\|2\|\|-2\|\|-1\|\|0
		+	\|-
		+	\| Psychotherapy\|\|3\|\|1\|\|0\|\|1
		+	\|-
		+	\| Acupuncture\|\|4\|\|1\|\|0\|\|1
		+	\|}</center>
		+
		+	Equal sample sizes are important as the results are much cleaner when the sample sizes are presumed to be equal. However equal sample size are not common in real applications, even in well-designed experiments. Unequal sample size makes the effects no longer independent. This implies that it makes difference in hypothesis testing when the effects are added into the model, first, middle, or last. The same dummy coding that was applied to equal sample sizes will now be applied to the original data with unequal sample sizes.

	===Examples===		===Examples===

@@ Line 89: / Line 89: @@
          \frac{\rho_{XY} - \rho_{XZ}\rho_{YZ}}
               {\sqrt{1-\rho_{XZ}^2} \sqrt{1-\rho_{YZ}^2}}.</math>
 ===Examples===

@@ Line 114: / Line 114: @@
 <center>[[Image:SOCR_EBook_Dinov_GLM_MLR_021808_Fig3.jpg|500px]]
 [[Image:SOCR_EBook_Dinov_GLM_MLR_021808_Fig4.jpg|500px]]</center>
 <hr>
 ===[[EBook_Problems_GLM_MultLin|Problems]]===

@@ Line 12: / Line 12: @@
 * Multilinear vs. non-linear regression: This multilinear regression method is "linear" because the relation of the response (the dependent variable <math>Y</math>) to the independent variables is assumed to be a [http://en.wikipedia.org/wiki/Linear_function linear function] of the parameters <math>\beta_i</math>. Note that multilinear regression is a linear modeling technique '''not''' because that the graph of <math>Y = \beta_{0}+\beta x </math> is a straight line '''nor''' because <math>Y</math> is a linear function of the ''X'' variables. But the "linear" term refers to the fact that <math>Y</math> can be considered a linear function of the parameters ( <math>\beta_i</math>), even though it is not a linear function of <math>X</math>. Thus, any model like
-: <math>Y = \beta_o + \beta_1 x + \beta_2 x^2 + \varepsilon</math> <br> is still one of '''linear''' regression, that is, linear in <math>x</math> and <math>x^2</math> respectively, even though the graph on <math>x</math> by itself is not a straight line.
+: <math>Y = \beta_o + \beta_1 x + \beta_2 x^2 + \varepsilon</math> <br> is still one of the '''linear''' regression, that is, linear in <math>x</math> and <math>x^2</math> respectively, even though the graph on <math>x</math> by itself is not a straight line.
 ===Parameter Estimation in Multilinear Regression===

@@ Line 10: / Line 10: @@
 The coefficient <math>\beta_0</math> is the intercept ("constant" term) and <math>\beta_i</math>s are the respective parameters of the '' p'' independent variables. There are ''p+1'' parameters to be estimated in the multilinear regression.
-* Multilinear vs. non-linear regression: This multilinear regression method is "linear" because the relation of the response (the dependent variable <math>Y</math>) to the independent variables is assumed to be a [http://en.wikipedia.org/wiki/Linear_function linear function] of the parameters <math>\beta_i</math>. Note that multilinear regression is a linear modeling technique '''not''' because is that the graph of <math>Y = \beta_{0}+\beta x </math> is a straight line '''nor''' because <math>Y</math> is a linear function of the ''X'' variables. But the "linear" terms refers to the fact that <math>Y</math> can be considered a linear function of the parameters ( <math>\beta_i</math>), even though it is not a linear function of <math>X</math>. Thus, any model like
+* Multilinear vs. non-linear regression: This multilinear regression method is "linear" because the relation of the response (the dependent variable <math>Y</math>) to the independent variables is assumed to be a [http://en.wikipedia.org/wiki/Linear_function linear function] of the parameters <math>\beta_i</math>. Note that multilinear regression is a linear modeling technique '''not''' because that the graph of <math>Y = \beta_{0}+\beta x </math> is a straight line '''nor''' because <math>Y</math> is a linear function of the ''X'' variables. But the "linear" term refers to the fact that <math>Y</math> can be considered a linear function of the parameters ( <math>\beta_i</math>), even though it is not a linear function of <math>X</math>. Thus, any model like
 : <math>Y = \beta_o + \beta_1 x + \beta_2 x^2 + \varepsilon</math> <br> is still one of '''linear''' regression, that is, linear in <math>x</math> and <math>x^2</math> respectively, even though the graph on <math>x</math> by itself is not a straight line.

@@ Line 116: / Line 116: @@
 <hr>
-===References===
+===[[EBook_Problems_GLM_MultLin|Problems]]===
 <hr>