Ranking Evidence in Human Scientific Research
Time to Read: ~10 Minutes
Take Home Points:
- The foundation of science is evidence.
- Some evidence is valued more than others by scientists.
- The strength of evidence increases as one moves from observational studies to experimental studies and then to summary studies.
- in vitro and animal studies vary in their value based on how applicable they are to humans.
- Just because a study type is of high value, doesn't mean the study itself is of high value; the quality of the study's design and interpretation must still be assessed.
Science & Evidence
Scientific research is a search for knowledge that brings us closer to one or more truths about how the world works. In its most basic form, scientific research (and the scientific method) begin and end with evidence. Evidence is a body of information that points to a belief, theory, or proposition. In short, we gain more confidence in the "truth" of something as more evidence appears to support that proposed truth.
The scientific method begins with evidence in the form of observations or results from prior investigations. This evidence is used to ask new questions and gather new evidence. Over time when a large amount of evidence is gathered a clearer picture will start to emerge. If the collective body of evidence supports a particular viewpoint or phenomenon, scientists will move closer to calling that viewpoint or phenomenon true or factual.
However, just having a large amount of evidence is not enough. All evidence is not considered equal in the eyes of science. The more measurable and repeatable a piece of evidence is, the more valid it will generally be, and the more impact it will have on defining a possible truth. Evidence is therefore commonly ranked in a hierarchy from weakest to strongest as seen here:
The specific types of evidence listed in the pyramid above aren't always completely distinct entities; there is some overlap (discussed below). Therefore, categorizing the evidence as one of three types: Observational Studies, Experimental Studies, and Summaries can be useful.
Observational Studies
Case Reports & Anecdotes: These are the weakest type of evidence. They are generally observations made without any intentioned investigation. For example, you notice you slept better after not having diary in your diet the day before. Another example is a doctor who notices a patient now has lower blood pressure after taking a drug to improve bone density. Did the lack of dairy really affect sleep? Did the bone density drug really reduce the blood pressure? These types of observations may indicate a significant finding, but they just as well might not. More evidence is always needed.
Cross-Sectional & Longitudinal Studies: For cross-sectional studies, one or more groups are examined at one point in time. For example, examining cholesterol levels in women of three different age groups. There are no variables (factors) being actively manipulated here, nor are other variables being controlled for (excluded from influence), so these types of studies are just descriptive in nature. Longitudinal studies are cross-sectional studies that look at several time points.
Case-Control Studies: In this type of study, two or more groups that are found to have a different outcome or behavior are investigated for potential factors causing the difference. Here again, the study is descriptive in nature; no causative testing is undertaken. An example would be examining a large pool of individuals and dividing them into those who have lung cancer and those who do not. You could then compare factors present at that time (biological and environmental) in the hopes of seeing a difference that might have caused the cancer.
Cohort Studies: Technically these are longitudinal studies, but ones in which all the individuals share one or more common events or traits. For example, comparing individuals of different demographics that all received a particular drug or were all living in a particular town during a common period of time where lead exposure was high.
Experimental Studies
Non-Randomized Control Trials: Here we have an experimental investigation where individuals are assigned to groups with one group getting a treatment and the other getting no treatment. However, non-randomized means the individuals are not placed into these groups randomly. Either the researchers or the individuals themselves are selecting their group. This can lead to bias in the resulting study's findings. Imagine if the groups ended up with mostly women in one group and men in another group. Or with one group being mostly adults over the age of 70. In such situations these non-randomized variables of sex/gender and age might obscure the effect of the drug being tested.
Randomized Control Trials: In these studies all participants are randomized into groups, to minimize potential effects of uncontrolled variables. The group placement is also ideally blinded so that the individuals and researchers do not know who is in which group until the experiment is completed. Knowing which group you are in (as a participant) or which group participants are in (as a researcher) can generate bias in the study.
Summaries
Systematic Reviews: These are just as they sound, thoroughly planned out reviews of the available evidence in support of a clearly formulated question or theory. For example, a systematic review of the evidence that resistance training improves bone mass in adults over the age of 50 may pull all available studies, rank their quality, remove ones of lower quality and attempt to quantify a consensus based on the remaining studies. The approach taken to review the available evidence must be clear and repeatable by other researchers.
Meta-Analyses: Generally recognized as the most powerful type of research study, these are statistical analyses of data pooled from previously published studies that all ask the same question.
in vitro & Animal Studies
You will notice that in the above figure in vitro & animal studies are not part of the hierarchy. As depicted, the hierarchy is for human research and these types of studies, not being performed in humans, don't cleanly fit into the pyramid. I've placed them towards the bottom, though they can be both observational and experimental in nature. The lower placing is given because the behavior of cells (in vitro) and animals might not always be relevant to how things work in humans. However, sometimes an animal study will fall higher up on the list, particularly if the animals used were primates.
The Big Picture
Systematic Reviews and Meta-Analyses are, themselves, often seen as the "big picture" answer of a particular topic or question. However, they are still a form of evidence compiled by human researchers and therefore subject to human mistakes and errors. They are still studies with constraints and assumptions, ones that other researchers might not agree with. It is therefore crucial to remember that although some evidence is generally more valuable than others, any study, no matter where it falls on the hierarchy, could have low value if conducted poorly. So, when determining what overall theme or truth might be suggested by the evidence, keep the big picture in mind, but be mindful of the details.