SMHS ReliabilityValidity - Revision history

Clgalla: /* Theory */

2014-09-08T15:34:29Z

‎Theory

Clgalla: /* Theory */

2014-09-08T15:23:56Z

‎Theory

Clgalla: /* Theory */

2014-09-08T15:07:30Z

‎Theory

Clgalla: /* Theory */

2014-09-08T15:06:56Z

‎Theory

Dinov: /* Software */

2014-09-06T19:17:23Z

‎Software

Liyufang: /* References */

2014-08-29T17:29:42Z

‎References

Clgalla: /* Applications */

2014-08-29T14:25:53Z

‎Applications

Mwolvie: /* References */

2014-08-14T14:25:19Z

‎References

Mwolvie: /* Scientific Methods for Health Sciences - Measurement Reliability and Validity */

2014-08-14T14:24:04Z

‎Scientific Methods for Health Sciences - Measurement Reliability and Validity

Mwolvie: /* Scientific Methods for Health Sciences - Measurement Reliability and Validity */

2014-08-14T14:21:02Z

‎Scientific Methods for Health Sciences - Measurement Reliability and Validity

@@ Line 30: / Line 30: @@
 .4) Kappa statistic: Answers the question of ‘How much better is the agreement between observers than would be expected by chance alone?’
-$Kappa=\frac{(\%\, agreement\, observed)-(\%\,agreement\,expected\,by\, chance\, alone)} {100\%-(\%\, agreement\, expected\, by\, chance\, alone)}$
+$Kappa=\frac{(\% \, agreement\, observed)-(\% \,agreement\,expected\,by\, chance\, alone)} {100\%-(\% \, agreement\, expected\, by\, chance\, alone)}$
 $Percent\,agreement=\frac{number\,in\,cells\,that\,'agree'}{Total\,number\, readings}*100$
@@ Line 61: / Line 61: @@
 </center>
 *Expected agreement by chance $=\frac{101+151}{500}=50.4%.$
-*Observed agreement = 82%; expected agreement based on chance = 50.4%. $Kappa=\frac{(%\, agreement\,observed)-(\%\,agreement\, expected\,by\,chance\, alone)} {100\%-(\%\,agreement\, expected\,by\,chance\,alone))=\frac{82\%-50.4\%}{100\%-50.4\%}=63.71\%$
+*Observed agreement = 82%; expected agreement based on chance = 50.4%.
 Interpretation of Kappa: > 0.75 excellent; 0.4 – 0.75 intermediate to good; < 0.40 poor reliability.

@@ Line 30: / Line 30: @@
 .4) Kappa statistic: Answers the question of ‘How much better is the agreement between observers than would be expected by chance alone?’
-$Kappa=\frac{(%\, agreement\, observed)-(%\, agreement\, expected\, by\, chance\, alone)} {100%-(%\, agreement\, expected\, by\, chance\, alone)}$
+$Kappa=\frac{(\%\, agreement\, observed)-(\%\,agreement\,expected\,by\, chance\, alone)} {100\%-(\%\, agreement\, expected\, by\, chance\, alone)}$
-$Percent\, agreement=(number\, in\, cells\, that\, 'agree')/(Total\, number\, readings)*100$
+$Percent\,agreement=\frac{number\,in\,cells\,that\,'agree'}{Total\,number\, readings}*100$
 Calculation of Kappa:
 <center>
@@ Line 46: / Line 46: @@
 |}
 </center>
-*Percent agreement: (1) proportion of tests reader 1 rate as positive $=\frac{180+50}{500}=46%$; proportion of tests reader 1 rate as negative =54%; if the results from reader 1 and reader 2 are independent, then reader 1 should have 46% positives regardless of reader 2’s scores.
+*Percent agreement: (1) proportion of tests reader 1 rate as positive $=\frac{180+50}{500}=46\%$; proportion of tests reader 1 rate as negative =54%; if the results from reader 1 and reader 2 are independent, then reader 1 should have 46% positives regardless of reader 2’s scores.
 *Expected agreement based on chance alone: for 220 times reader 2 is positive, we expect reader 1 will be positive 46% of the time; for 280 times reader 2 is negative, we expect reader 1 will be negative 54% of the time:
 <center>
@@ Line 61: / Line 61: @@
 </center>
 *Expected agreement by chance $=\frac{101+151}{500}=50.4%.$
-*Observed agreement = 82%; expected agreement based on chance = 50.4%. $Kappa=\frac{(%\, agreement\, observed)-(%\, agreement\, expected\, by \,chance\, alone)} {100%-(%\,agreement\, expected\, by\, chance\, alone))=\frac{82%-50.4%}{100%-50.4%}=63.71%$
+*Observed agreement = 82%; expected agreement based on chance = 50.4%. $Kappa=\frac{(%\, agreement\,observed)-(\%\,agreement\, expected\,by\,chance\, alone)} {100\%-(\%\,agreement\, expected\,by\,chance\,alone))=\frac{82\%-50.4\%}{100\%-50.4\%}=63.71\%$
 Interpretation of Kappa: > 0.75 excellent; 0.4 – 0.75 intermediate to good; < 0.40 poor reliability.

@@ Line 32: / Line 32: @@
 $Kappa=\frac{(%\, agreement\, observed)-(%\, agreement\, expected\, by\, chance\, alone)} {100%-(%\, agreement\, expected\, by\, chance\, alone)}$
-$Percent\, agreement=(number\, in\, cells\, that\, 'agree')/(Total\, number\, readings)*100
+$Percent\, agreement=(number\, in\, cells\, that\, 'agree')/(Total\, number\, readings)*100$
 Calculation of Kappa:
 <center>

@@ Line 30: / Line 30: @@
 .4) Kappa statistic: Answers the question of ‘How much better is the agreement between observers than would be expected by chance alone?’
-Kappa=((% agreement observed)-(% agreement expected by chance alone))/(100%-(% agreement expected by chance alone))
+$Kappa=\frac{(%\, agreement\, observed)-(%\, agreement\, expected\, by\, chance\, alone)} {100%-(%\, agreement\, expected\, by\, chance\, alone)}$
 Percent agreement=(number in cells that 'agree')/(Total number readings)*100
 Calculation of Kappa:
 <center>
@@ Line 45: / Line 46: @@
 |}
 </center>
-*Percent agreement: (1) proportion of tests reader 1 rate as positive =(180+50)/500=46%; proportion of tests reader 1 rate as negative =54%; if the results from reader 1 and reader 2 are independent, then reader 1 should have 46% positives regardless of reader 2’s scores.
+*Percent agreement: (1) proportion of tests reader 1 rate as positive $=\frac{180+50}{500}=46%$; proportion of tests reader 1 rate as negative =54%; if the results from reader 1 and reader 2 are independent, then reader 1 should have 46% positives regardless of reader 2’s scores.
 *Expected agreement based on chance alone: for 220 times reader 2 is positive, we expect reader 1 will be positive 46% of the time; for 280 times reader 2 is negative, we expect reader 1 will be negative 54% of the time:
 <center>
@@ Line 59: / Line 60: @@
 |}
 </center>
-*Expected agreement by chance =(101+151)/500=50.4%.
+*Expected agreement by chance $=\frac{101+151}{500}=50.4%.$
-*Observed agreement = 82%; expected agreement based on chance = 50.4%. Kappa=((% agreement observed)-(% agreement expected by chance alone))/(100%-(% agreement expected by chance alone))=(82%-50.4%)/(100%-50.4%)=63.71%
+*Observed agreement = 82%; expected agreement based on chance = 50.4%. $Kappa=\frac{(%\, agreement\, observed)-(%\, agreement\, expected\, by \,chance\, alone)} {100%-(%\,agreement\, expected\, by\, chance\, alone))=\frac{82%-50.4%}{100%-50.4%}=63.71%$
 Interpretation of Kappa: > 0.75 excellent; 0.4 – 0.75 intermediate to good; < 0.40 poor reliability.

@@ Line 72: / Line 72: @@
 ===Software===
-none
 ===Problems===

@@ Line 125: / Line 125: @@
 ===References===
-Statistical inference / George Casella, Roger L. Berger.  http://mirlyn.lib.umich.edu/Record/004199238
+*[https://www.uni.edu/chfasoa/reliabilityandvalidity.htm  Exploring Reliability in Academic Assessment]
+*[http://ocw.jhsph.edu/courses/hsre/PDFs/HSRE_lect7_weiner.pdf  Measurements: Reliability and Validity Measures]
-Sampling / Steven K. Thompson.  http://mirlyn.lib.umich.edu/Record/004232056
+*[http://en.wikipedia.org/wiki/Validity_(statistics)  Validity (statistics) Wikipedia]

@@ Line 65: / Line 65: @@
 ===Applications===
-.1) This article (http://www.socialresearchmethods.net/kb/constval.php) presents a comprehensive introduction to measurements, validity and reliability and illustrate the concepts and application with examples. It is very well developed and would be a great introduction to the material we are going to cover in this lecture.
+[http://www.socialresearchmethods.net/kb/constval.php  This article] presents a comprehensive introduction to measurements, validity and reliability and illustrate the concepts and application with examples. It is very well developed and would be a great introduction to the material we are going to cover in this lecture.
-.2) This article (http://www.egadconnection.org/Reliability%20and%20validity.pdf) presents a general introduction to reliability, validity and generalizability and studied on various problems with measurement. It gives comprehensive analysis of reliability and validity with definitions, different ways to measure reliability and validity as well as problems associated with these characteristics. This would be a great start to get to know measurement reliability and validity.
+[http://www.egadconnection.org/Reliability%20and%20validity.pdf  This article] presents a general introduction to reliability, validity and generalizability and studied on various problems with measurement. It gives comprehensive analysis of reliability and validity with definitions, different ways to measure reliability and validity as well as problems associated with these characteristics. This would be a great start to get to know measurement reliability and validity.
-.3) This article (http://psycnet.apa.org/psycinfo/1995-00092-001) assessed the reliability and validity of the Childhood Trauma Questionnaire (CTQ), a retrospective measure of child abuse and neglect. 286 drug- or alcohol-dependent patients (aged 24–68 years) were given the CTQ as part of a larger test battery, and 40 of these patients were given the questionnaire again after an interval of 2–6 months. 68 Ss were also given the Childhood Trauma Interview. Principal-components analysis of responses on the CTQ yielded 4 rotated orthogonal factors: physical and emotional abuse, emotional neglect, sexual abuse, and physical neglect. The CTQ demonstrated high internal consistency and good test-retest reliability over an interval of 2–6 months. The CTQ also demonstrated convergence with the Childhood Trauma Interview indicating that Ss' reports of child abuse and neglect based on the CTQ were highly stable, both over time and across types of instruments.
+[http://psycnet.apa.org/psycinfo/1995-00092-001 This article] assessed the reliability and validity of the Childhood Trauma Questionnaire (CTQ), a retrospective measure of child abuse and neglect. 286drug- or alcohol-dependent patients (aged 24–68 years) were given the CTQ as part of a larger test battery, and 40 of these patients were given the questionnaire again after an interval of 2–6 months. 68 Ss were also given the Childhood Trauma Interview. Principal-components analysis of responses on the CTQ yielded 4 rotated orthogonal factors: physical and emotional abuse, emotional neglect, sexual abuse, and physical neglect. The CTQ demonstrated high internal consistency and good test-retest reliability over an interval of 2–6 months. The CTQ also demonstrated convergence with the Childhood Trauma Interview indicating that Ss' reports of child abuse and neglect based on the CTQ were highly stable, both over time and across types of instruments.
 ===Software===

@@ Line 45: / Line 45: @@
 |}
 </center>
+*Percent agreement: (1) proportion of tests reader 1 rate as positive =(180+50)/500=46%; proportion of tests reader 1 rate as negative =54%; if the results from reader 1 and reader 2 are independent, then reader 1 should have 46% positives regardless of reader 2’s scores.
+*Expected agreement based on chance alone: for 220 times reader 2 is positive, we expect reader 1 will be positive 46% of the time; for 280 times reader 2 is negative, we expect reader 1 will be negative 54% of the time:
-	Percent agreement: (1) proportion of tests reader 1 rate as positive =(180+50)/500=46%; proportion of tests reader 1 rate as negative =54%; if the results from reader 1 and reader 2 are independent, then reader 1 should have 46% positives regardless of reader 2’s scores.
+<center>
-	Expected agreement based on chance alone: for 220 times reader 2 is positive, we expect reader 1 will be positive 46% of the time; for 280 times reader 2 is negative, we expect reader 1 will be negative 54% of the time:
+{|class="wikitable" style="text-align:center; width:75%" border="1"
+|-
-	Reader 1
+| ||colspan=3|Reader 1
-Reader 2		Positive	Negative
+|-
-	Positive	101	119
+|rowspan=3|Reader 2|| ||Positive||Negative
-	Negative	129	151
+|-
+|Positive||101||119
-Expected agreement by chance =(101+151)/500=50.4%.
+|-
-	Observed agreement = 82%; expected agreement based on chance = 50.4%. Kappa=((% agreement observed)-(% agreement expected by chance alone))/(100%-(% agreement expected by chance alone))=(82%-50.4%)/(100%-50.4%)=63.71%
+|Negative||129||151
 Interpretation of Kappa: > 0.75 excellent; 0.4 – 0.75 intermediate to good; < 0.40 poor reliability.
-) Applications
+===Applications===
 .1) This article (http://www.socialresearchmethods.net/kb/constval.php) presents a comprehensive introduction to measurements, validity and reliability and illustrate the concepts and application with examples. It is very well developed and would be a great introduction to the material we are going to cover in this lecture.
@@ Line 67: / Line 71: @@
 .3) This article (http://psycnet.apa.org/psycinfo/1995-00092-001) assessed the reliability and validity of the Childhood Trauma Questionnaire (CTQ), a retrospective measure of child abuse and neglect. 286 drug- or alcohol-dependent patients (aged 24–68 years) were given the CTQ as part of a larger test battery, and 40 of these patients were given the questionnaire again after an interval of 2–6 months. 68 Ss were also given the Childhood Trauma Interview. Principal-components analysis of responses on the CTQ yielded 4 rotated orthogonal factors: physical and emotional abuse, emotional neglect, sexual abuse, and physical neglect. The CTQ demonstrated high internal consistency and good test-retest reliability over an interval of 2–6 months. The CTQ also demonstrated convergence with the Childhood Trauma Interview indicating that Ss' reports of child abuse and neglect based on the CTQ were highly stable, both over time and across types of instruments.
-) Software
+===Software===
 none
-) Problems
+===Problems===
 .1) In public health practice, optimizing the validity of tests is important in order to:
 a. reduce health care costs
 b. reduce unnecessary stress for patients
 c. Be able to identify opportunities for intervention early in the course of disease
 d. all of the above
 .2) The Kappa statistic is used to measure ___ of a test?
 a. sensitivity
 b. reliability
 c. positive predictive value
 d. specificity
 .3) Randomization of treatment groups ensures the study’s external validity.
 a. True
 b. False
 .4) In a study investigating whether a new serum-based screening test for pancreatic cancer allowed for earlier detection than traditional tests, the researchers found that those who enrolled in the study were more likely to have a family history of pancreatic cancer than those who did not. This characteristic of the study population affects the study’s:
 a. Internal validity
 b. External validity
 c. Both
 d. Neither
 .5) As sample size increases:
 a. Sampling variability increases and the chance of selecting an unrepresentative sample increases
 b. Sampling variability increases and the chance of selecting an unrepresentative sample decreases
 c. Sampling variability decreases and the chance of selecting an unrepresentative sample decreases
 d. Sampling variability decreases and the chance of selecting an unrepresentative sample increases
-) References
+===References===
 http://mirlyn.lib.umich.edu/Record/004199238
 http://mirlyn.lib.umich.edu/Record/004232056
 http://mirlyn.lib.umich.edu/Record/004133572