AP Statistics Curriculum 2007 Estim MOM MLE - Revision history

Tdlee: Text replacement - "{{translate|pageName=http://wiki.stat.ucla.edu/socr/" to ""{{translate|pageName=http://wiki.socr.umich.edu/"

2020-03-03T20:23:54Z

Text replacement - "{{translate|pageName=http://wiki.stat.ucla.edu/socr/" to ""{{translate|pageName=http://wiki.socr.umich.edu/"

IvoDinov: /* MOM Beta Distribution Example */

2011-05-18T17:03:03Z

‎MOM Beta Distribution Example

IvoDinov: /* MOM Beta Distribution Example */

2011-05-18T15:58:11Z

‎MOM Beta Distribution Example

IvoDinov: /* Method of Moments (MOM) Estimation */ added MOM Beta Distribution Example section

2011-05-18T15:56:50Z

‎Method of Moments (MOM) Estimation: added MOM Beta Distribution Example section

Jenny: /* Maximum Likelihood Estimation (MLE) */

2010-06-28T20:27:47Z

‎Maximum Likelihood Estimation (MLE)

Jenny: /* Method of Moments (MOM) Estimation */

2010-06-28T20:26:38Z

‎Method of Moments (MOM) Estimation

Jenny at 17:11, 28 June 2010

2010-06-28T17:11:13Z

IvoDinov: added a link to the Problems set

2009-10-26T05:06:55Z

added a link to the Problems set

IvoDinov: /* References */

2008-12-08T19:33:49Z

‎References

IvoDinov: /* References */ typo

2008-12-08T19:33:10Z

‎References: typo

← Older revision		Revision as of 20:23, 3 March 2020
Line 133:		Line 133:
	* SOCR Home page: http://www.socr.ucla.edu		* SOCR Home page: http://www.socr.ucla.edu

−	{{translate\|pageName=http://wiki.~~stat~~.~~ucla~~.edu~~/socr~~/index.php?title=AP_Statistics_Curriculum_2007_Estim_MOM_MLE}}	+	"{{translate\|pageName=http://wiki.socr.umich.edu/index.php?title=AP_Statistics_Curriculum_2007_Estim_MOM_MLE}}

@@ Line 39: / Line 39: @@
 The sample-mean and sample-variance are:
 : sample-mean: <math>\overline{x}=0.251</math>
-: sample-mean: <math>s^2=0.16187</math>
+: sample-variance: <math>s^2=0.16187</math>
 Solve the 2 equations (for the first two moments according to the MOM parameter estimation protocol) for the unknown parameters (<math>\alpha</math> and <math>\beta</math>):

@@ Line 41: / Line 41: @@
 : sample-mean: <math>s^2=0.16187</math>
-Solve the 2 equations for the unknown parameters (<math>\alpha</math> and <math>\beta</math>):
+Solve the 2 equations (for the first two moments according to the MOM parameter estimation protocol) for the unknown parameters (<math>\alpha</math> and <math>\beta</math>):
 : <math>\overline{x}=\mu</math> and
 : <math>s^2=\sigma^2</math>.

@@ Line 24: / Line 24: @@
 <center>[[Image:SOCR_EBook_Dinov_Estimates_MOM_MLE_032808_Fig1.jpg|400px]]</center>
 More information about the [http://en.wikipedia.org/wiki/Method_of_moments_%28statistics%29 method of moments for parameter (point) estimation may be found here].
 ===Maximum Likelihood Estimation (MLE)===

@@ Line 41: / Line 41: @@
 :<math>\hat{\theta} = \arg \max_{\theta}{L (\theta)}.</math>
-The outcome of a maximum likelihood analysis is the maximum likelihood estimate <math>\hat{\theta}</math>. One typically assumes that the observed data are independent and identically distributed (IID) with unknown parameters (<math>\theta</math>). This considerably simplifies the problem because the likelihood can then be written as a product of ''n'' univariate probability densities:
+The outcome of a maximum likelihood analysis is the maximum likelihood estimate <math>\hat{\theta}</math>. One typically assumes that the observed data are independent and identically distributed (IID) with unknown parameters (<math>\theta</math>). This considerably simplifies the problem because the likelihood that it can then be written as a product of ''n'' univariate probability densities:
 :<math>\mathcal{L}(\theta) = \prod_{i=1}^n f_{\theta}(x_i \mid \theta)</math>

@@ Line 8: / Line 8: @@
 ===Method of Moments (MOM) Estimation===
-Parameter estimation using the method of moments is both intuitive and easy to calculate. The idea is to use the sample data to calculate some [[AP_Statistics_Curriculum_2007_Distrib_MeanVar#Higher_Moments |sample moments]] and then set these equal to their corresponding population counterparts. Typically the latter involve the parameter(s) that we are interested in estimating and thus we obtain a computationally tractable protocol for their estimation. Summarizing the MOM:
+Parameter estimation using the method of moments is both intuitive and easy to calculate. The idea is to use the sample data to calculate some [[AP_Statistics_Curriculum_2007_Distrib_MeanVar#Higher_Moments |sample moments]] and then set these equal to their corresponding population counterparts. Typically the latter involves the parameter(s) that we are interested in estimating and thus we obtain a computationally tractable protocol for their estimation. Summarizing the MOM:
 * First: Determine the k parameters of interest and the specific (model) distribution for this process;
 * Second: Compute the first k (or more) [[AP_Statistics_Curriculum_2007_Distrib_MeanVar#Higher_Moments |sample-moments]];
-* Third: Set the [[AP_Statistics_Curriculum_2007_Distrib_MeanVar#Higher_Moments |sample-moments]] equal to the population moments and solve a (linear or non-linear) system of k equations with k unknowns.
+* Third: Set the [[AP_Statistics_Curriculum_2007_Distrib_MeanVar#Higher_Moments |sample-moments]] equal to the population moments and solve for a (linear or non-linear) system of k equations with k unknowns.
 ====MOM Proportion Example====
-Let's look at the motivational problem we discussed above. We want to flip a coin 8 times, observe the number of heads (successes) in the outcomes and use that to infer the true (unknown) probability of a Head (''P(H)=?'') for this specific coin.
+Let's look at the motivational problem we discussed above. We want to flip a coin 8 times, observe the number of heads (successes) in the outcomes and use that to infer the true (unknown) probability of a Head (''P(H)=?'') of this specific coin.
 * Hypothetical solution: Suppose we observe the following sequence of outcomes ''{T,H,T,H,H,T,H,H}''. Using the MOM protocol we obtain:

@@ Line 20: / Line 20: @@
 ** The first [[AP_Statistics_Curriculum_2007_Distrib_MeanVar#Higher_Moments |sample-moment]] for a [[AP_Statistics_Curriculum_2007_Distrib_Binomial#Bernoulli_process |Bernoulli process]] is p=E(Y). Therefore, if the [[EBook#Random_Variables |random variable]] ''X = {# H’s}'' is [[AP_Statistics_Curriculum_2007_Distrib_Binomial |Binomially distributed]], then the expected value of X is ''E(X)=np=8p''. And the sample value of X is ''Sample#H’s = 5''. Equating the first sample moments yields <math>8p \approx 5</math>. Hence, we would estimate the unknown <math>p=P(H) \approx MOM(p)=\hat{p}={5\over 8}</math>.
-* Experimental Solution: We can also use [http://socr.ucla.edu/htmls/SOCR_Experiments.html SOCR Experiments] to demonstrate the MOM estimation technique. You can refer to the [[SOCR_EduMaterials_Activities_CoinSampleExperiment | SOCR Coin Sample Experiment]] for more information of this SOCR applet. The The figure below illustrates flipping a coin 8 times and observing 5 Heads.  This is a [[AP_Statistics_Curriculum_2007_Distrib_Binomial | Binomial(n=8, p=0.65)]] distribution. However, let's pretend for a minute that we did '''not''' know the actual ''p=P(H)'' value! So we have a good approximation <math>0.65=p=P(H) \approx MOM(p)=\hat{p}={5\over 8}=0.625</math>. Of course, if we run this experiment again, our MOM estimate for ''p'' would change!
+* Experimental Solution: We can also use [http://socr.ucla.edu/htmls/SOCR_Experiments.html SOCR Experiments] to demonstrate the MOM estimation technique. You can refer to the [[SOCR_EduMaterials_Activities_CoinSampleExperiment | SOCR Coin Sample Experiment]] for more information of this SOCR applet. The figure below illustrates flipping a coin 8 times and observing 5 Heads.  This is a [[AP_Statistics_Curriculum_2007_Distrib_Binomial | Binomial(n=8, p=0.65)]] distribution. However, let's pretend for a minute that we did '''not''' know the actual ''p=P(H)'' value! So we have a good approximation <math>0.65=p=P(H) \approx MOM(p)=\hat{p}={5\over 8}=0.625</math>. Of course, if we run this experiment again, our MOM estimate for ''p'' would change!
 <center>[[Image:SOCR_EBook_Dinov_Estimates_MOM_MLE_032808_Fig1.jpg|400px]]</center>
@@ Line 41: / Line 41: @@
 :<math>\hat{\theta} = \arg \max_{\theta}{L (\theta)}.</math>
-The outcome of a maximum likelihood analysis is the maximum likelihood estimate <math>\hat{\theta}</math>. One typically assumes that the observed data comes are independent and identically distributed (IID) with unknown parameters (<math>\theta</math>). This considerably simplifies the problem because the likelihood can then be written as a product of ''n'' univariate probability densities:
+The outcome of a maximum likelihood analysis is the maximum likelihood estimate <math>\hat{\theta}</math>. One typically assumes that the observed data are independent and identically distributed (IID) with unknown parameters (<math>\theta</math>). This considerably simplifies the problem because the likelihood can then be written as a product of ''n'' univariate probability densities:
 :<math>\mathcal{L}(\theta) = \prod_{i=1}^n f_{\theta}(x_i \mid \theta)</math>
@@ Line 51: / Line 51: @@
 The maximum of this expression can then be found numerically using various optimization algorithms.
-Note that the maximum likelihood estimator may not be unique, or is guaranteed to exist.
+Note that the maximum likelihood estimator may not be unique, or guaranteed to exist.
 ====MLE Proportion Example====
-Let's look again at the motivational problem of flipping a coin 8 times, observing the number of heads (successes) in the outcomes and using this inferring (based on MLE) the true (unknown) probability of a Head (''P(H)=?'') for this specific coin.
+Let's look again at the motivational problem of flipping a coin 8 times, observing the number of heads (successes) in the outcomes and using this to infer (based on MLE) the true (unknown) probability of a Head (''P(H)=?'') for this specific coin.
 Suppose again we observe the same sequence of 8 outcomes ''{T,H,T,H,H,T,H,H}''. Using the MLE protocol we obtain:
@@ Line 67: / Line 67: @@
 Suppose we have observed IID {<math>x_1, \cdots, x_n</math>}={<math>0.5, 0.3, 0.6, 0.1, 0.2</math>}, subjects' weights, coming from <math>N(\mu, \sigma^2=1)</math>, with marginal density function <math>f(x|\mu)</math>. Then the joint density is <math>f(x_1, \cdots,x_n| \mu) =f(x_1|\mu)\times \cdots \times f(x_n|\mu)</math> and the likelihood function <math>L(\mu) = f(x_1, \cdots,x_n|\mu) </math>.
-Of course, we are trying to estimate the average weight for a subject form this population -- i.e., we are trying to find the <math>MLE(\mu)</math>.
+Of course, we are trying to estimate the average weight for a subject from this population -- i.e., we are trying to find the <math>MLE(\mu)</math>.
 : <math>L(\mu) = e^{-{(0.5-\mu)^2+(0.3-\mu)^2+(0.6-\mu)^2+(0.1-\mu)^2+(0.2-\mu)^2\over 2}}</math>
@@ Line 73: / Line 73: @@
 : <math>0={d\ln{L(\mu)} \over d\mu} = (0.5-\mu)+(0.3-\mu)+(0.6-\mu)+(0.1-\mu)+(0.2-\mu) = -5\mu +1.7</math>
 : Thus, <math>\hat{\mu}=0.34</math>
-: Validate the this value indeed maximizes the log-likelihood function, i.e., <math>{d^2 \ln{L(\mu)} \over d\mu^2}(\mu=\hat{\mu}) <0</math>.
+: Validate that this value indeed maximizes the log-likelihood function, i.e., <math>{d^2 \ln{L(\mu)} \over d\mu^2}(\mu=\hat{\mu}) <0</math>.
 * How does this estimate, <math>\hat{\mu}=0.34</math>, compare to the ''sample average'' of the 5 observations?

@@ Line 92: / Line 92: @@
 ===Parameter Estimation Examples===
 The [http://socr.ucla.edu/htmls/SOCR_Modeler.html SOCR Modeler] and the corresponding [[SOCR_EduMaterials_ModelerActivities | SOCR Modeler Activities]] provide a number of interesting examples of parameter (point) estimation in terms of fitting ''best'' models to observed data.
 <hr>

@@ Line 105: / Line 105: @@
 ** http://code.google.com/p/socr/source/browse/#svn/trunk/SOCR2.0/src/edu/ucla/stat/SOCR
 ** http://code.google.com/p/socr/source/browse/trunk/SOCR2.0/src/edu/ucla/stat/SOCR/modeler/RiceFit_Modeler.java
 <hr>

@@ Line 98: / Line 98: @@
 * [http://www.stat.ucla.edu/%7Edinov/courses_students.dir/07/Winter/PN284.dir/NITP_PN_M284_Inference2.pdf Lecture notes on Statistical Methods in Neuroimaging]
 * [http://repositories.cdlib.org/socr/EM_MM  Notes on parameter estimation, expectation maximization and mixture modeling] and the corresponding [[SOCR_EduMaterials_Activities_2D_PointSegmentation_EM_Mixture | Java applets and activities]].
-* [http://socr.ucla.edu/htmls/SOCR_Modeler.html Web-based executable Applet): http://socr.ucla.edu/htmls/SOCR_Modeler.html
+* [http://socr.ucla.edu/htmls/SOCR_Modeler.html Web-based executable Applet]: http://socr.ucla.edu/htmls/SOCR_Modeler.html
 * [[SOCR_EduMaterials_ModelerActivities | SOCR EM Wiki activities]]
 * [http://www.socr.ucla.edu/docs/edu/ucla/stat/SOCR/modeler/RiceFit_Modeler.html HTML JavaDocs]