View source for AP Statistics Curriculum 2007 GLM MultLin

==[[AP_Statistics_Curriculum_2007 | General Advance-Placement (AP) Statistics Curriculum]] - Multiple Linear Regression ==

In the [[AP_Statistics_Curriculum_2007#Chapter_X:_Correlation_and_Regression | previous sections]], we saw how to study the relations in bivariate designs. Now we extend that to any finite number of variables (multivariate case).

=== Multiple Linear Regression ===
We are interested in determining the '''linear regression''', as a model, of the relationship between one '''dependent''' variable ''Y'' and many '''independent''' variables ''X''<sub>''i''</sub>, ''i'' = 1, ..., ''p''. The multilinear regression model can be written as

: <math>Y = \beta_0  + \beta_1 X_1 + \beta_2 X_2 +  \cdots +\beta_p X_p + \varepsilon</math>, where <math>\varepsilon</math> is the error term.

The coefficient <math>\beta_0</math> is the intercept ("constant" term) and <math>\beta_i</math>s are the respective parameters of the '' p'' independent variables. There are ''p+1'' parameters to be estimated in the multilinear regression.

* Multilinear vs. non-linear regression: This multilinear regression method is "linear" because the relation of the response (the dependent variable <math>Y</math>) to the independent variables is assumed to be a [http://en.wikipedia.org/wiki/Linear_function linear function] of the parameters <math>\beta_i</math>. Note that multilinear regression is a linear modeling technique '''not''' because that the graph of <math>Y = \beta_{0}+\beta x </math> is a straight line '''nor''' because <math>Y</math> is a linear function of the ''X'' variables. But the "linear" term refers to the fact that <math>Y</math> can be considered a linear function of the parameters ( <math>\beta_i</math>), even though it is not a linear function of <math>X</math>. Thus, any model like

: <math>Y = \beta_o + \beta_1 x + \beta_2 x^2 + \varepsilon</math> <br> is still one of the '''linear''' regression, that is, linear in <math>x</math> and <math>x^2</math> respectively, even though the graph on <math>x</math> by itself is not a straight line. 

===Parameter Estimation in Multilinear Regression===
A multilinear regression with ''p'' coefficients and the regression intercept β<sub>0</sub> and ''n'' data points (sample size), with <math>n\geq (p+1) </math> allows construction of the following vectors and matrix with associated standard errors:

:<math> \begin{bmatrix} y_{1} \\ y_{2} \\ \vdots \\ y_{n} \end{bmatrix} = \begin{bmatrix} 1 & x_{11} & x_{12} & \dots & x_{1p} \\ 1 & x_{21} & x_{22} & \dots & x_{2p} \\ \vdots & \vdots & \vdots & & \vdots \\ 1 & x_{n1} & x_{n2} & \dots & x_{np} \end{bmatrix} \begin{bmatrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_p \end{bmatrix} + \begin{bmatrix} \varepsilon_1\\ \varepsilon_2\\ \vdots\\ \varepsilon_n \end{bmatrix} </math>

or, in '''vector-matrix notation''' 

:<math> \ y =  \mathbf{X}\cdot\beta + \varepsilon.\, </math>
Each data point can be given as <math>(\vec x_i, y_i)</math>, <math>i=1,2,\dots,n.</math>. For n = p, standard errors of the parameter estimates could not be calculated. For n less than p, parameters could not be calculated. 

* '''Point Estimates''': The estimated values of the parameters <math>\beta_i</math> are given as 
:<math>\widehat{\beta} </math><math>=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T {\vec y}</math>

* '''Residuals''': The residuals, representing the difference between the observations and the model's predictions, are required to analyse the regression and are given by:

:<math>\hat\vec\varepsilon = \vec y - \mathbf{X} \hat\beta\,</math>

The standard deviation, <math>\hat \sigma </math> for the model is determined from

:<math> {\hat \sigma  = \sqrt{ \frac {\hat\vec\varepsilon^T \hat\vec\varepsilon} {n-p-1}} = \sqrt {\frac{{  \vec y^T  \vec y -   \hat\vec\beta^T \mathbf{X}^T   \vec y}}{{n - p - 1}}} }  </math>

The variance in the errors is Chi-square distributed:
:<math>\frac{(n-p-1)\hat\sigma^2}{\sigma^2} \sim \chi_{n-p-1}^2</math>

* '''Interval Estimates''': The <math>100(1-\alpha)% </math> [[AP_Statistics_Curriculum_2007_Estim_S_Mean#Interval_Estimation_of_a_Population_Mean | confidence interval]] for the parameter, <math>\beta_i </math>, is computed as follows:

:<math> {\widehat \beta_i  \pm t_{\frac{\alpha }{2},n - p - 1} \hat \sigma \sqrt {(\mathbf{X}^T \mathbf{X})_{ii}^{ - 1} } }  </math>,

where ''t'' follows the [[AP_Statistics_Curriculum_2007_StudentsT | Student's t-distribution]] with <math>n-p-1</math> degrees of freedom and <math> (\mathbf{X}^T \mathbf{X})_{ii}^{ - 1}</math> denotes the value located in the <math>i^{th}</math> row and column of the matrix.

The '''regression sum of squares''' (or sum of squared residuals) ''SSR'' (also commonly called ''RSS'') is given by:

:<math> {\mathit{SSR} = \sum {\left( {\hat y_i  - \bar y} \right)^2 }  =   \hat\beta^T \mathbf{X}^T   \vec y - \frac{1}{n}\left( {  \vec y^T   \vec u  \vec u^T   \vec y} \right)}  </math>,

where <math> \bar y = \frac{1}{n} \sum y_i</math> and <math>  \vec u </math> is an ''n'' by 1 unit vector (i.e. each element is 1). Note that the terms <math>y^T u</math> and <math>u^T y</math> are both equivalent to <math>\sum y_i</math>, and so the term <math>\frac{1}{n} y^T u u^T y</math> is equivalent to <math>\frac{1}{n}\left(\sum y_i\right)^2</math>.

The '''error''' (or explained)''' sum of squares''' (''ESS'') is given by:

:<math> {\mathit{ESS} = \sum {\left( {y_i  - \hat y_i } \right)^2 }  =   \vec y^T   \vec y -   \hat\beta^T \mathbf{X}^T   \vec y}. </math>

The '''total sum of squares''' ('''TSS''') is given by

:<math> {\mathit{TSS} = \sum {\left( {y_i  - \bar y} \right)^2 }  =   \vec y^T   \vec y - \frac{1}{n}\left( {  \vec y^T   \vec u  \vec u^T   \vec y} \right) = \mathit{SSR}+ \mathit{ESS}}. </math>

===Partial Correlations===
For a given linear model
<center> <math>Y = \beta_0  + \beta_1 X_1 + \beta_2 X_2 +  \cdots +\beta_p X_p + \varepsilon</math></center>
the partial correlation between <math>X_1</math> and ''Y'' given a set of ''p-1'' controlling variables <math>Z = \{X_2, X_3, \cdots, X_p\}</math>, denoted by  <math>\rho_{YX_1|Z}</math>, is the correlation between the residuals ''R''<sub>''X''</sub> and ''R''<sub>''Y''</sub> resulting from the linear regression of ''X'' with '''Z''' and that of ''Y'' with '''Z''', respectively. The first-order partial correlation is just the difference between a correlation and the product of the removable correlations divided by the product of the coefficients of alienation of the removable correlations.

* Partial correlation coefficients for three variables is calculated from the pairwise simple correlations. 
: If, <math>Y = \beta_0  + \beta_1 X_1 + \beta_2 X_2 + \varepsilon</math>, 
: then the partial correlation between <math>Y</math> and <math>X_2</math>, adjusting for <math>X_1</math> is:
: <math>\rho_{YX_2|X_1} = \frac{\rho_{YX_2} - \rho_{YX_1}\times \rho_{X_2X_1}}{\sqrt{1- \rho_{YX_1}^2}\sqrt{1-\rho_{X_2X_1}^2}}</math>

* In general, the sample partial correlation is
:<math>\hat{\rho}_{XY\cdot\mathbf{Z}}=\frac{N\sum_{i=1}^N r_{X,i}r_{Y,i}-\sum_{i=1}^N r_{X,i}\sum r_{Y,i}}
{\sqrt{N\sum_{i=1}^N r_{X,i}^2-\left(\sum_{i=1}^N r_{X,i}\right)^2}~\sqrt{N\sum_{i=1}^N r_{Y,i}^2-\left(\sum_{i=1}^N r_{Y,i}\right)^2}},</math> where the [http://en.wikipedia.org/wiki/Partial_correlation residuals] <math>r_{X,i}</math> and <math>r_{X,i}</math> are given by:
:: <math>r_{X,i} = x_i - \langle\mathbf{w}_X^*,\mathbf{z}_i \rangle</math>
:: <math>r_{Y,i} = y_i - \langle\mathbf{w}_Y^*,\mathbf{z}_i \rangle</math>,
:: with <math>x_i</math>, <math>y_i</math> and <math>z_i</math> denoting the random (IID) samples of some joint probability distribution over X, Y and Z.

====Computing the partial correlations====
The ''n''<sup>th</sup>-order partial correlation (|'''Z'''| = ''n'') can be computed from three (''n'' - 1)<sup>th</sup>-order partial correlations. The ''0''<sup>th</sup>-order partial correlation <math>\rho_{YX|\empty}</math> is defined to be the regular [http://en.wikipedia.org/wiki/Correlation correlation coefficient] <math>\rho_{YX}</math>.

For any <math>Z_0 \in \mathbf{Z}</math>:
:<math>\rho_{XY| \mathbf{Z} } =
        \frac{\rho_{XY| \mathbf{Z}\setminus\{Z_0\}} - \rho_{XZ_0| \mathbf{Z}\setminus\{Z_0\}}\rho_{YZ_0 | \mathbf{Z}\setminus\{Z_0\}}}
             {\sqrt{1-\rho_{XZ_0 |\mathbf{Z}\setminus\{Z_0\}}^2} \sqrt{1-\rho_{YZ_0 | \mathbf{Z}\setminus\{Z_0\}}^2}}.</math>

Implementing this computation recursively yields an exponential time [http://en.wikipedia.org/wiki/Computational_complexity_theory complexity]. 

Note in the case where Z is a single variable, this reduces to:
:<math>\rho_{XY | Z } =
        \frac{\rho_{XY} - \rho_{XZ}\rho_{YZ}}
             {\sqrt{1-\rho_{XZ}^2} \sqrt{1-\rho_{YZ}^2}}.</math>

===Examples===

We now demonstrate the use of [[SOCR_EduMaterials_AnalysisActivities_MLR | SOCR Multilinear regression applet]] to analyze multivariate data.

====Earthquake Modeling====
This is an example where the relation between variables may not be linear or explanatory. In the [[AP_Statistics_Curriculum_2007_GLM_Regress | simple linear regression case]], we were able to compute by hand some (simple) examples. Such calculations are much more involved in the multilinear regression situations. Thus we demonstrate multilinear regression only using the [http://socr.ucla.edu/htmls/SOCR_Analyses.html SOCR Multiple Regression Analysis Applet].

Use the SOCR California Earthquake dataset to investigate whether Earthquake magnitude (dependent variable) can be predicted by knowing the longitude, latitude, distance and depth of the quake. Clearly, we do not expect these predictors to have a strong effect on the earthquake magnitude, so we expect the coefficient parameters not to be significantly distinct from zero (null hypothesis). SOCR Multilinear regression applet reports this model:

:  <math>Magnitude = \beta_o + \beta_1\times Close+ \beta_2\times Depth+ \beta_3\times Longitude+ \beta_4\times Latitude + \varepsilon.</math>

:  <math>Magnitude = 2.320 + 0.001\times Close -0.003\times Depth -0.035\times Longitude -0.028\times Latitude + \varepsilon.</math>

<center>[[Image:SOCR_EBook_Dinov_GLM_MLR_021808_Fig1.jpg|500px]]
[[Image:SOCR_EBook_Dinov_GLM_MLR_021808_Fig2.jpg|500px]]</center>

====Multilinear Regression on Consumer Price Index====
Using the [[SOCR_Data_Dinov_021808_ConsumerPriceIndex | SOCR Consumer Price Index Dataset]] we can explore the relationship between the prices of various products and commodities. For example, regressing '''Gasoline''' on the following three predictor prices: '''Orange Juice''', '''Fuel''' and '''Electricity''' illustrates significant effects of all these variables as significant explanatory prices (at <math>\alpha=0.05</math>) for the cost of ''Gasoline'' between 1981 and 2006.

: <math>Gasoline = 0.083 -0.190\times Orange +0.793\times Fuel +0 .013\times Electricity
</math>

<center>[[Image:SOCR_EBook_Dinov_GLM_MLR_021808_Fig3.jpg|500px]]
[[Image:SOCR_EBook_Dinov_GLM_MLR_021808_Fig4.jpg|500px]]</center>

====2011 Best Jobs in the US====
Repeat the multiliniear regression analysis using hte [[SOCR_Data_2011_US_JobsRanking| Ranking Dataset of the Best and Worst USA Jobs for 2011]].

<hr>

===[[EBook_Problems_GLM_MultLin|Problems]]===

<hr>
* SOCR Home page: http://www.socr.ucla.edu

{{translate|pageName=http://wiki.stat.ucla.edu/socr/index.php?title=AP_Statistics_Curriculum_2007_GML_MultLin}}