SMHS ExpObsStudies
Scientific Methods for Health Sciences - Experiments vs. Observational Studies
IV. HS 850: Fundamentals
Experimental vs. Observational Studies
1) Overview: In an experiment investigators apply treatment to experimental units and then proceed to observe the effect of the treatments on the experimental units. It’s an ordered procedure carried out with the goal of verifying, refuting, or establishing the validity of a hypothesis. This lecture will present a general introduction to the field of experiment and different types of experiment that we may apply in researches later. We are also going to talk about observational study, which draws inferences about the possible effect of a treatment on subjects where the assignment of subjects into a treated group versus a control group outside the control of the investigators. A general comparison between these two types of studies will be presented thereafter.
2) Motivation: Experimental and observational studies are among the most commonly applied studies in researches. Consider a simple example of experimental study. Suppose we enrolled 200 women aged 30 who aren’t smokers and assign half of them to smoking treatment with one pack per day and the other half to no smoking treatment and kept this status for 10 years. Then the lung capacity of all the women are measured and the data is further analyzed and interpreted. Here we have an experimental study. For the other study, we find 200 women aged 30, of whom 100 are smoke free and the other half having been smoking one pack per day for 10 years and the lung capacity of those women are measured. Then the data is further analyzed and interpreted. This would be an easy example of observational study. The difference can be easily drawn from the comparison between these two studies: the assignment of subjects into a treated group versus a control group is outside the control of the investigators where in an experimental study, each subject is randomly assigned to a treated group or a control group. So, what characteristics would define experimental and observational studies in general and what would be the major difference between these two types of studies?
3) Theory
3.1) Experimental study: an empirical method that arbitrates between models or hypothesis and used to test existing theories or new hypotheses in order to support them or disprove them. Controlled experiments provide researches with insight into the causal relationship by demonstrating what outcome occurs when a particular factor is manipulated. Experiments may vary from personal and informal natural comparisons, to highly controlled ones.
- Types of experimental studies:
- Controlled experiments: compare the results obtained form experimental samples against control samples, which are practically identical to the experimental sample except for the one whose effect is being tested.
- Natural experiments: rely solely on observations of the variables of the system under study, rather than manipulation of just one or a few variables as occurs in controlled experiments.
- Field experiments: named to draw a contrast with laboratory experiments, which enforce scientific control by testing a hypothesis in the artificial and highly controlled setting of a laboratory. It is often used in social sciences, and especially in economic analyses o education.
3.2) Observational Study: it draws inferences about the possible effect of a treatment on subjects where the assignment of subjects into a treated group versus a control group is outside the control of the investigator.
- Types of observational studies:
- Case-control study: originally developed in epidemiology, where two existing groups differing in outcome are identified and compared on the basis of some supposed causal attribute.
- Cross-sectional study: involves data collection from a population, or a representative subset, at one specific point in time.
- Longitudinal study: correlational research study that involves repeated observations of the same variables over long periods of time.
- Cohort study (panel study): a particular form of longitudinal study where a group of patients is closely monitored over a span of time.
- Ecological study: an observational study in which at least one variable is measured at the group level.
- Degree of usefulness and reliability: though observational studies can’t be used as reliable sources to make statements of fact about the ‘safety, efficacy, or effectiveness’ of a practice, they can still be used for: (1) provide information on real world use and practice; (2) detect signals about the benefits and risk of use in the general population; (3) help formulate hypotheses to be tested in subsequent experiments; (4) provide part of the community-level data needed to design more informative pragmatic clinical trials; (5) inform clinical practice.
- Bias and compensating methods: when a randomized experiment cannot be carried out, the alternative line of investigation suffers from the problem that the decision of which subjects receive the treatment is not entirely random and thus is a potential source of bias. A major challenge in conducting observational studies is to draw inferences that are acceptably free from influences by overt biases, as well as to assess the influence of potential hidden biases. An observer of an uncontrolled experiment records potential factors and the data output: the goal is to determine the effects of the factors. Sometimes the recorded factors may not be directly causing the differences in the output. Also, recorded or unrecorded factors may be correlated which may yield incorrect conclusions. Finally, as the number of recorded factors increases, the likelihood increases that at least one of the recorded factors will be highly correlated with the data output simply by chance.
3.3) Comparisons between experimental and observational studies:
- An observational study is used when it is impractical, cost-prohibitive, or inefficient to fit a physical or social system into a laboratory setting, to completely control confounding factors, or to apply random assignment. It can also be used when confounding factors are either limited or known well enough to analyze the data in light of them. In order for an observational science to be valid, confounding factors must be known and accounted for.
- Fundamentally, however, observational studies are not experiments. In addition, observational studies often involve variables that are difficult to quantify or control. Observational studies are limited because they lack the statistical properties of randomized experiments. In a randomized experiment, the method of randomization specified in the experimental protocol guides the statistical analysis, which is usually specified also by the experimental protocol.
- A particular problem with observational studies involving human subjects is the great difficulty attaining fair comparisons between treatments because such studies are prone to selection bias, and groups receiving different treatments (exposures) may differ greatly according to their covariates. In contrast, the randomization ensures that the experimental groups have mean values that are close, due to the CLT or Markov’s inequality. With inadequate randomization or low sample size, the systematic variation in covariates between the treatment groups (or exposure groups) makes it difficult to separate the effect of the treatment (exposure) from the effects of the other covariates, most of which have not been measured.
- To avoid conditions that render an experiment far less useful, physicians conducting medical trials will quantify and randomize the covariates that can be identified. Researchers attempt to reduce the biases of observational studies with complicated statistical methods such as propensity score matching methods, which require large populations of subjects and extensive information on covariates. Outcomes are also quantified when possible and not based on a subject's or a professional observer's opinion. In this way, the design of an observational study can render the results more objective and therefore, more convincing.
4) Applications
4.1) This article (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC534936/) titled Observational Versus Experimental Studies: What’s the Evidence for a Hierarchy presents information that contradicts and discourages such a rigid approach to evaluating the quality of research design. It argued that the popular belief that randomized, controlled trials inherently produce gold standard results, and that all observational studies are inferior, does a disservice to patient care, clinical investigation, and education of health care professionals. It proposed that a more balanced strategy evolves, new claims of methodological authority may be just as problematic as the traditional claims of medical authority that have been criticized by proponents of evidence-based medicine.
4.2) This article (http://www.nber.org/papers/w13516) titled Observational Learning: Evidence From A Randomized Natural Field Experiment presents results about the effects of observing others’ choices, called observational learning, on individuals' behavior and subjective well-being in the context of restaurant dining from a randomized natural field experiment. In the paper, they conducted experiment to distinguish observational learning effect from saliency effect (because observing others' choices also makes these choices more salient) and found that depending on specifications, the demand for the top 5 dishes was increased by an average of about 13 to 18 percent when these popularity rankings were revealed to the customers; in contrast, being merely mentioned as some sample dishes did not significantly boost their demand. Plus, consistent with theoretical predictions, some modest evidence that observational learning effect was stronger among infrequent customers. Finally, it argued that customers' subjective dining experiences were improved when presented with the information about the top choices by other consumers, but not when presented with the names of some sample dishes.
5) Software not applicable …
6) Problems
Problem 6.1 – 6.5 are based on the following article:
You want to study if exercise decreases your risk of having a cardiovascular event. One of your friends says that they have access to data of 5,000 US adults (aged 65-85) who were surveyed once in 2010 about their current exercise patterns and past cardiovascular events. You look at their data and find that people with cardiovascular events reported higher levels of exercise than those without cardiovascular events. This is counter to your expectation since you know that other studies have shown that exercise should put you at a lower risk of an event.
6.1) What could be a plausible explanation(s) for this unexpected finding?
6.2) What kind of study is this? Experimental or Observational?
6.3) What feature of a cohort study design might improve your ability to look at this association?
6.4) Now being impressed by cohort studies, you decide to recruit the current student body of UM SPH to prospectively study exercise as a risk factor for many diseases. One of your friends asks if they can use your data to study exercise as a risk factor for MERRF syndrome, an extremely rare disease of the mitochondria with an estimated prevalence of 9 per 1,000,000. Why or why not might this be a good idea?
6.5) Your real interest is maternal fitness level during pregnancy and cardiovascular disease of the offspring in adulthood. What might be the challenges that you will face in conducting a cohort study to answer this question?
Problem 6.6 – 6.10 are based on the following article:
Abstract from Appel et al.
BACKGROUND: It is known that obesity, sodium intake, and alcohol consumption factors influence blood pressure. In this clinical trial, Dietary Approaches to Stop Hypertension, we assessed the effects of dietary patterns on blood pressure.
METHODS: We enrolled 459 adults with systolic blood pressures of less than 160 mm Hg and diastolic blood pressures of 80 to 95 mm Hg. For three weeks, the subjects were fed a control diet that was low in fruits, vegetables, and dairy products, with a fat content typical of the average diet in the United States. They were then randomly assigned to receive for eight weeks the control diet, a diet rich in fruits and vegetables, or a "combination" diet rich in fruits, vegetables, and low-fat dairy products and with reduced saturated and total fat. Sodium intake and body weight were maintained at constant levels.
RESULTS: At base line, the mean (+/-SD) systolic and diastolic blood pressures were 131.3+/-10.8 mm Hg and 84.7+/-4.7 mm Hg, respectively. The combination diet reduced systolic and diastolic blood pressure by 5.5 and 3.0 mm Hg more, respectively, than the control diet (P<0.001 for each); the fruits-and-vegetables diet reduced systolic blood pressure by 2.8 mm Hg more (P<0.001) and diastolic blood pressure by 1.1 mm Hg more than the control diet (P=0.07). Among the 133 subjects with hypertension (systolic pressure, > or =140 mm Hg; diastolic pressure, > or =90 mm Hg; or both), the combination diet reduced systolic and diastolic blood pressure by 11.4 and 5.5 mm Hg more, respectively, than the control diet (P<0.001 for each); among the 326 subjects without hypertension, the corresponding reductions were 3.5 mm Hg (P<0.001) and 2.1 mm Hg (P=0.003).
CONCLUSIONS: A diet rich in fruits, vegetables, and low-fat dairy foods and with reduced saturated and total fat can substantially lower blood pressure. This diet offers an additional nutritional approach to preventing and treating hypertension.
6.6) What type of study is this?
6.7) What was the purpose of the trial?
6.8) Losing weight causes blood pressure to drop. Why do you think the investigators made an effort to keep participants’ weight stable in this trial?
6.9) What was the purpose of randomization in this study (i.e. why is randomization done)?
6.10) How would you evaluate whether randomization “worked” or not?
6.11) Did randomization work?
7) References http://mirlyn.lib.umich.edu/Record/004199238 http://mirlyn.lib.umich.edu/Record/004232056 http://mirlyn.lib.umich.edu/Record/004133572
Answer:
6.1) Since this was cross-sectional data, there is likely confusion between the timing of the relationship between exercise and cardiovascular events. In fact, the exercise data is said to be after the clinical events. Thus it is possible that people who survive a cardiovascular event are more motivated to exercise to prevent recurrence than those who had no event. It is also possible that those people who are reporting a past cardiovascular event are the ones who survived and those who were less healthy did not survive the event and were thus not included in the study.
6.2) It’s a Cohort Study so it’s an observational study.
6.3) Since a cohort study looks first at exposure and then follows people for disease, you are able to determine temporality and assess incident disease.
6.4) This is not a good idea since there is no way that you would observe enough cases (if any) to look at this question given how rare disease the disease is.
6.5) The exposure and the outcome are very far apart so it would be challenging to prospectively follow-up from prenatal exposures to middle age exposures (40+ years). A retrospective design would be useful but it would be challenging to obtain accurate data on maternal exercise information.
6.6) This is a Randomized Controlled Trial, so it’s an experimental Study.
6.7) The purpose of the trial was to compare the impacts of three dietary patterns on blood pressure in adults. The diets investigated were: (1) a “combination diet” that is high in fruit, vegetables and low-fat dairy, and low in saturated and total fat, (2) a diet high in fruit and vegetables, and (3) a typical American control diet.
6.8) The researchers tried to keep participants’ weight stable so they could isolate the effect of the dietary composition on blood pressure independent of overall caloric intake. If calorie intake changed during the trial, it would be impossible to tell whether changes in blood pressure were due to dietary composition or change in overall calorie intake.
6.9) Randomization is done to ensure that the characteristics of each study group are similar. It is important that as many personal characteristics as possible are similar between the alternate diet and control groups so that any differences observed in the group outcomes can be attributed to the dietary intervention rather than other factors like age, BMI, or income.
6.10) You would review Table 2 to see if the characteristics of participants are similar in all three groups.
6.11) While no formal statistical tests are run to evaluate whether the characteristics are significantly different between groups, the groups look pretty similar on key characteristics. Ideally, statistical tests would have been conducted to compare the characteristics of the two groups. For continuous variables this might have been an ANOVA and for categorical a Chi-squared test would be performed.
- SOCR Home page: http://www.socr.umich.edu
Translate this page:
