There seems to be a fairly common view that virtually any idea or concept can be supported with “evidence”. I have had a couple of recent comments that highlight this view.
One of the recent comments was;
“All scientific research is based on controlled studies, all of which can be arranged to “prove” any outcome that you want. I can setup a viable study to have it demonstrate that protein is bad and whole grains is good, even if that is not the case.”
I think that these types of statements are used by people who feel that their view on a subject is challenged. They use this position as some type of argument.
Is this true, can we prove anything we want and find research to back up anything? On the surface there may be some truth to these statements. However, when you look behind the curtain you really CAN’T prove everything or find research to back up anything, IF, and this is a big IF, you play by the “rules”. Let me explain.
When it comes to research and doing studies there are many variables that will determine the internal and external validity of it. Basically, the type of research and how the studies are done will determine how well it can demonstrate a CAUSE and effect relationship. There are many aspects that help us determine the “quality” of the research and therefore, how much emphasis we can put on it. Some of the variables include (this is not an exhaustive list);
- Sample size
- Sample type
- Randomization
- Length of intervention
- Blinded or not
- Type of statistical analysis
- How well variables were controlled (i.e., metabolic ward versus free-living)
- Type of study (randomized controlled trials, cohort, case-controlled, cross-section survey, etc)
These variables will determine if the study actually demonstrates some kind of effect and if we can determine WHAT actually CAUSED the effect.
Before making any conclusive judgments, it is generally recommended that a similar effect has been produced more than once.
These would be the “rules” of research and if they are followed it is NOT likely that anything can be supported. I think anyone familiar with the many aspect of research would agree. You just CAN’T prove anything, just like you CAN’T say that left is right, up is down, or that researchers always follow the rules.
This last point also relates to the view that “anything” can be supported or there is more uncertainty than there is on a topic. Researchers are human, therefore, like us mortals, they to do NOT always follow the rules. This means that the conclusions from a study may actually have little or no validity. This can be due to weaknesses in the above “rules”. However, the problem can actually be with how the authors state the results in the conclusion of the paper which is the part that is included in the abstract, which often translates into a misleading title of a paper (due to the the peer-review process this should really not happen, but it does). These latter points are what are often publicized and contribute to the confusion. However, if someone were to look at the meat of the paper, the boring information explaining the “rules” of the study and the actual results, a different conclusion may be more likely or at least a less “definitive” conclusion would be warranted. I am not saying that this happens all the time, but it happens enough to muddy the waters and leads to the impression that “anything” can be supported with evidence.
This problem also stems from biases (often we are not conscious of) and a lack or lapse of critical thinking (it is very demanding) that happens to researchers, writers, and practitioners. We WANT to believe certain things, therefore we can easily focus on what seems to support our current beliefs and miss the information that may challenge our current beliefs.
On a superficial level, there is certainly an impression that most anything can be supported with evidence. But, when a deeper perspective is taken, this view loses its validity. I am NOT saying that things are black and white, there is definitely a lot of grey. But, the grey area is often more like nuances of a topic rather than a complete divergence from what the “quality” evidence actually demonstrates. With that said, I don’t think that the argument that anything can be supported with evidence or any study can be manipulated to show any result are valid IF the rules are followed. When the rules are followed and we become aware of our biases and implement critical thinking skills there is often a rather small range of possible conclusions that can be supported by evidence.