SMHS CorrectionMultipleTesting - Revision history

Dinov: /* References */

2016-02-03T14:36:53Z

‎References

Dinov: /* FDR (false discovery rate) */

2014-12-08T17:52:53Z

‎FDR (false discovery rate)

Dinov: /* Family-Wise Error Rate (FWER) */

2014-12-08T17:51:53Z

‎Family-Wise Error Rate (FWER)

Dinov: /* Motivation */

2014-12-08T17:51:12Z

‎Motivation

Dinov: /* Family-Wise Error Rate (FWER) */

2014-09-18T17:56:04Z

‎Family-Wise Error Rate (FWER)

Dinov: /* Family-Wise Error Rate (FWER) */

2014-09-03T14:06:00Z

‎Family-Wise Error Rate (FWER)

Dinov: /* Family-Wise Error Rate (FWER) */

2014-09-03T14:05:33Z

‎Family-Wise Error Rate (FWER)

Dinov: /* Motivation */

2014-09-03T14:00:31Z

‎Motivation

Liyufang: /* References */

2014-08-29T17:02:38Z

‎References

Dinov: /* FDR (false discovery rate) */

2014-08-27T14:30:33Z

‎FDR (false discovery rate)

@@ Line 103: / Line 103: @@
 ===References===
 * [http://en.wikipedia.org/wiki/Multiple_comparisons_problem  Multiple Comparison Problem Wikipedia]
 * [http://en.wikipedia.org/wiki/Familywise_error_rate  FWER Wikipedia]

@@ Line 71: / Line 71: @@
 * Benjamini-Hochberg procedure controls the false discovery (at least α). For a given α, find the largest k such that \(p_((k))≤k/m α\); then reject all \(H_{(i)}\) for \(i=1,…,k\). This method works when the m tests are independent as well as with some cases of dependence: \(E(Q)≤\frac{m_0}{m}α≤α\).
-* Benjamini-Hochberg-Yekutieli procedure controls the FDR under positive dependence assumptions. It modifies the BH procedure: \(p_{(k)}≤\frac{k}{m*c(m)}α\), if the tests are independent or positively correlated we choose \(c(m)=1\) and choose \(c(m)=\sum_{i=1}^m{1/i}\) with arbitrary dependent tests, when the tests are negatively correlated, c(m) can be approximated with \(\sum_{i=1}^m{1/i} \approx ln(m)+ \gamma\), where \(\gamma\) is the [http://en.wikipedia.org/wiki/Euler%E2%80%93Mascheroni_constant Euler-Mascheroni constant].
+* Benjamini-Hochberg-Yekutieli procedure controls the FDR under positive dependence assumptions. It modifies the BH procedure: \(p_{(k)}≤\frac{k}{m*c(m)}α\), if the tests are independent or positively correlated we choose \(c(m)=1\) and choose \(c(m)=\sum_{i=1}^m{1/i}\) with arbitrary dependent tests, when the tests are negatively correlated, c(m) can be approximated with \(\sum_{i=1}^m{\frac{1}{i}} \approx \ln(m)+ \gamma\), where \(\gamma\) is the [http://en.wikipedia.org/wiki/Euler%E2%80%93Mascheroni_constant Euler-Mascheroni constant].
 ====Example====

@@ Line 19: / Line 19: @@
 ===Theory===
 ====Family-Wise Error Rate (FWER)====
-The probability of making the type I error among all the hypotheses when performing multiple hypothesis tests. FWER exerts a more stringent control over false discovery compared to false discovery rate controlling procedure. Suppose we did simultaneous tests on m hypotheses denoted by \(H_1,H_2,…,H_m\) with corresponding p-values \(p_1,p_2,…,p_m\). Let \(I_0\) be the subset of the true null hypotheses with \(m_0\). Our aim is to achieve an ''overall type I error rate of α'' from this cumulative multiple testing inference.  \(FWER=Pr⁡(V≥1)=1-Pr⁡(V=0)\). By assuming \(FWER≤α\), the probability of making even one type I error in the family is controlled at level α.
+FWER is the probability of making the type I error among all the hypotheses when performing multiple hypothesis tests. FWER exerts a more stringent control over false discovery compared to false discovery rate controlling procedure. Suppose we did simultaneous tests on m hypotheses denoted by \(H_1,H_2,…,H_m\) with corresponding p-values \(p_1,p_2,…,p_m\). Let \(I_0\) be the subset of the true null hypotheses with \(m_0\). Our aim is to achieve an ''overall type I error rate of α'' from this cumulative multiple testing inference.  \(FWER=Pr⁡(V≥1)=1-Pr⁡(V=0)\). By assuming \(FWER≤α\), the probability of making even one type I error in the family is controlled at level α.
 <center>

@@ Line 13: / Line 13: @@
 * Suppose we investigate the safety of a drug in terms of the occurrences of different types of side effects. As more types of side effects are considered, it becomes more likely that the new drug will appear to be less safe than existing drugs in terms of at least one side effect.
-The number of comparisons increases in these examples which leads to conclusions that the groups being compared do differ in terms of at least one attribute. Our confidence that a result will generalize to independent data should generally be weaker if it is observed as part of an analysis that involves multiple comparisons, rather than an analysis that involves only a single comparison. If a test is performed at the \(\alpha=0.05\) level, there is only a 5% chance of incorrectly rejecting the null hypothesis if the null hypothesis is true. However, for 100 tests where all null hypotheses are true, the expected number of incorrect rejections is 5. If the tests are independent, the probability of at least one incorrect rejection is 99.4% (as \(P(\ge 1)=1-P(0) = 1-0.95^{100}=0.9940795\). These errors are called false positives or Type I errors.
+: The number of comparisons increases in these examples which leads to conclusions that the groups being compared do differ in terms of at least one attribute. Our confidence that a result will generalize to independent data should generally be weaker if it is observed as part of an analysis that involves multiple comparisons, rather than an analysis that involves only a single comparison. If a test is performed at the \(\alpha=0.05\) level, there is only a 5% chance of incorrectly rejecting the null hypothesis if the null hypothesis is true. However, for 100 tests where all null hypotheses are true, the expected number of incorrect rejections is 5. If the tests are independent, the probability of at least one incorrect rejection is 99.4% (as \(P(\ge 1)=1-P(0) = 1-0.95^{100}=0.9940795\). These errors are called false positives or Type I errors. Try the [http://socr.ucla.edu/htmls/dist/Binomial_Distribution.html SOCR Java Binomial(100,0.05) calculator].
-So what can we do to adjust for multiple testing? How can we keep the prescribed family wise error rate of α in an analysis involving more than one comparison? Apparently, the error rate for each comparison, it must be more stringent than α. Multiple testing correction would be the way to go and we are going to introduce some commonly used methods for adjusting for this type of error in multiple testing.
+: So, what can we do to adjust for multiple testing? How can we keep the prescribed family wise error rate of α in an analysis involving more than one comparison? Apparently, the error rate for each comparison, it must be more stringent than α. Multiple testing correction would be the way to go and we are going to introduce some commonly used methods for adjusting for this type of error in multiple testing.
 ===Theory===

@@ Line 30: / Line 30: @@
 | Declared non-significant||U(number of true negatives)||T(number of false negatives)||m-R
 |-
-| Total	|| m_0(number of true null hypotheses)||m-m_0(number of true alternatives)||m
+| Total	|| $m_0$(number of true null hypotheses)||m-$m_0$(number of true alternatives)||m
 |}
 </center>

@@ Line 37: / Line 37: @@
 * A procedure controls the FWER in the strong sense if the FWER control at level α is guaranteed for any configuration of true and non-true null hypotheses.
-Controlling FWER:
+====Controlling FWER====
 * '''Bonferrroni Procedure''': states that rejecting all \(p_i≤\frac{α}{m}\) will control that \(FWER≤α\) which is proved through Boole’s Inequality: \(FWER=Pr⁡(\cup_i{p_i≤\frac{α}{m}}) \leq \sum_i{Pr⁡(p_i≤\frac{α}{m})} ≤ m_0 \frac{α}{m} ≤ m \frac{α}{m} = α\). This is the simplest and most conservative method to control FWER though it can be (very) conservative if there are a large number of tests and/or the test statistics are positively correlated. It controls the probability of false positives only.

@@ Line 38: / Line 38: @@
 Controlling FWER:
-* Bonferrroni: states that rejecting all \(p_i≤α/m\) will control that \(FWER≤α\) which is proved through Boole’s Inequality: \(FWER=Pr⁡(\Cup_i{p_i≤α/m}) \leq \sum_i{Pr⁡(p_i≤α/m)} ≤ m_0 α/m ≤ m α/m = α\). This is the simplest and most conservative method to control FWER though it can be (very) conservative if there are a large number of tests and/or the test statistics are positively correlated. It controls the probability of false positives only.
+* '''Bonferrroni Procedure''': states that rejecting all \(p_i≤\frac{α}{m}\) will control that \(FWER≤α\) which is proved through Boole’s Inequality: \(FWER=Pr⁡(\cup_i{p_i≤\frac{α}{m}}) \leq \sum_i{Pr⁡(p_i≤\frac{α}{m})} ≤ m_0 \frac{α}{m} ≤ m \frac{α}{m} = α\). This is the simplest and most conservative method to control FWER though it can be (very) conservative if there are a large number of tests and/or the test statistics are positively correlated. It controls the probability of false positives only.
-* Tukey’s procedure is only applicable for pairwise comparisons. It assumes independence of the observations being tested as well as equal variance across observations. The procedure calculates for each pair the standardized range statistics: \( \frac{Y_A-Y_B}{SE}\), where \(Y_A\) is the larger of the two means being compared and \(Y_B\) is the smaller one and SE is the standard error of the data.
+* '''Tukey’s procedure''' is only applicable for pairwise comparisons. It assumes independence of the observations being tested as well as equal variance across observations. The procedure calculates for each pair the standardized range statistics: \( \frac{Y_A-Y_B}{SE}\), where \(Y_A\) is the larger of the two means being compared and \(Y_B\) is the smaller one and SE is the standard error of the data.
-* The Sidak procedure works for independent tests where each hypothesis test has \(α_SID=1-(1-α)^{1/m}\). This is a more powerful method than Bonferroni but the gain is small.
+* The '''Sidak procedure''' works for independent tests where each hypothesis test has \(α_{SID}=1-(1-α)^{\frac{1}{m}}\). This is a more powerful method than Bonferroni but the gain is small.
-* Holm’s step-down procedure starts by ordering the p values from lowest to highest as  \(p_{(1)},p_{(2)},…,p_{(m)} \) with corresponding hypotheses  \(H_{(1)},H_{(2)},…,H_{(m)}\). Suppose R is the smallest k such that \(p_{(k)}>α/(m+1-k)\). Reject the null hypotheses \(H_{(1)},H_{(2)},…,H_{(m)}\), if \(R=1\) then none of the hypotheses are rejected. This method is uniformly better than Bonferroni’s and it is based on Bonferroni with no restriction on the joint distribution of the test statistics.
+* '''Holm’s step-down procedure''' starts by ordering the p values from lowest to highest as  \(p_{(1)},p_{(2)},…,p_{(m)} \) with corresponding hypotheses  \(H_{(1)},H_{(2)},…,H_{(m)}\). Suppose R is the smallest k such that \(p_{(k)}>α/(m+1-k)\). Reject the null hypotheses \(H_{(1)},H_{(2)},…,H_{(m)}\), if \(R=1\) then none of the hypotheses are rejected. This method is uniformly better than Bonferroni’s and it is based on Bonferroni with no restriction on the joint distribution of the test statistics.
-* Hochberg’s step-up procedure: starts by ordering the p values from lowest to highest as  \(p_{(1)},p_{(2)},…,p_{(m)} \)  with corresponding hypotheses  \(H_{(1)},H_{(2)},…,H_{(m)}\). For a given α, let \(R\) be the largest k such that \(p_{(k)}≤α/(m+1-k)\). Reject the null hypotheses \(H_{(1)},H_{(2)},…,H_{(R)}\) only, and none of the \(H_{(R+1)},…,H_{(m)}\). It is more powerful than Holm’s, however, it is based on the Simes test so it holds only under independence (and also under some form of positive dependence).
+* '''Hochberg’s step-up procedure''': starts by ordering the p values from lowest to highest as  \(p_{(1)},p_{(2)},…,p_{(m)} \)  with corresponding hypotheses  \(H_{(1)},H_{(2)},…,H_{(m)}\). For a given α, let \(R\) be the largest k such that \(p_{(k)}≤α/(m+1-k)\). Reject the null hypotheses \(H_{(1)},H_{(2)},…,H_{(R)}\) only, and none of the \(H_{(R+1)},…,H_{(m)}\). It is more powerful than Holm’s, however, it is based on the Simes test so it holds only under independence (and also under some form of positive dependence).
 ====FDR (false discovery rate)====

@@ Line 60: / Line 60: @@
 | Declared non-significant||U(number of true negatives)||T(number of false negatives)||m-R
 |-
-| Total	|| m_0(number of true null hypotheses)||\(m-m_0\)(number of true alternatives)||m
+| Total	|| $m_0$(number of true null hypotheses)||$m-m_0$ (number of true alternatives)||m
 |}
 </center>
-Define Q as the proportion of false discoveries among the discoveries (Q=V/R) then FDR is defined as \(FDR=Q_e=E[Q]=E[V/(V+S)]=E[V/R]\),where V/R is defined to be 0 when R=0. Our aim is to keep FDR below the threshold α (or q). And q-value is defined as FDR analogue of the p-value, the q-value of individual hypothesis test is the minimum FDR at which the test may be called significant.
+Define $Q$ as the proportion of false discoveries among the discoveries $Q=\frac{V}{R}$, then FDR is defined as $FDR=Q_e=E[Q]=E[\frac{V}{V+S}]=E[\frac{V}{R}]$, where $\frac{V}{R} is defined to be 0 when R=0. Our aim is to keep FDR below the threshold α (or q). And q-value is defined as FDR analogue of the p-value, the q-value of individual hypothesis test is the minimum FDR at which the test may be called significant.
 Controlling procedures of FDR: