Are you confused by conflicting results from research studies? An analogy may help you understand the situation and how we interpret studies.
Trying to see the whole picture
Each study is a bit like a small photo of one part of an enormous landscape. Imagine a landscape that might have mountains, a stream, a forest, some meadow, and many varieties of plants and wildlife. Now imagine an array of photos showing the whole landscape, but each photo shows only one small piece. Each photo is like an individual study.
With research, we’re trying to figure out what the whole picture looks like for each therapy. If we turn all those photos face-down on a big table, then we see none of the landscape. Each study that we review turns one photo face-up. A meta-analysisa statistical analysis that combines the results of two or more research studies; the results of smaller research studies addressing the same or similar questions can be analyzed as though they are one bigger, more powerful study turns several photos over at once. If we’re asking about a therapy’s effectiveness for cancer as a whole, we’re asking broad questions: Are there any mountains here? Any trees? Each study for a specific cancer type is somewhat like asking “Are there any fir trees (breast cancer) in this picture?” “How about birch trees (prostate cancer)?” And so on.
We can summarize only the research that has been done so far, and because we still have a lot of gaps, we still have a lot of questions.
Some photos/studies are much larger than others. With some, we get to see more of the whole picture than with others. Some photos/studies are poor quality, and we can’t see much detail. Some are so poor that we toss them out, as they don’t add anything to the whole picture.
We’re trying to figure out the landscape picture for each therapy we review. However, none of the photo arrays is complete. Sometimes huge gaps are in the array. Each of the small photos/studies we need is expensive to create, and many simply have not been completed or even begun yet. Some therapies have much better funding, and so we have more of the small photos/studies to look at for those. Some types of cancer also have better funding, and so we may have many more photos/studies on breast cancer or prostate cancer compared to kidney cancer or sarcoma, for instance.
Sometimes we find so few photos/studies that we simply cannot draw any conclusions about what we’re seeing. While we’re looking at an incomplete picture, we’re not always able to resolve conflicts or contradictions. We can summarize only the research that has been done so far, and because we still have a lot of gaps, we still have a lot of questions.
Help from experts
We have one more tool to help guide you, however. We may not have photos of a particular landscape/therapy, but we know some people who’ve been there and can tell us about it. These are the integrative oncology experts that we refer to in our reviews. Some integrative oncologists and other integrative oncology professionals have published books drawing from both research and decades of experience treating people with cancer. In the absence of good photos/studies, these books are another source to answer questions about effectiveness and safety. These books also package several therapies together as a program, which most research studies do not do.
Making sense of the pieces we can see
With more/larger/better studies all finding similar results, we’re able to answer questions about effectiveness and safety with more confidence, moving up from insufficient to weak to preliminary to modest to good to strong assessments. But we might also downgrade an assessment: if we have several smaller studies all showing a positive effect, we’d assess the evidence as “preliminary” or even “modest.” Then if a bigger, better study finds an opposite effect, we’d have to change our assessment to “insufficient.” New evidence can contradict old evidence, and we need to look not only at individual studies, but at all the studies so far, also considering their size and quality. This is the reality of scientific research, and we’ll do our best to interpret it for you.
New evidence can contradict old evidence, and we need to look not only at individual studies, but at all the studies so far, also considering their size and quality.
Science doesn’t often provide simple, clear-cut answers. Winston Churchill once said: “[D]emocracy is the worst form of government except for all those other forms that have been tried from time to time.” 1The Worst Form of Government. International Churchill Society. February 25, 2016. Viewed September 14, 2021. Scientific research is a little like that. It’s not perfect, but it’s the best system we have for distinguishing fact from opinion, and we get a little closer to the facts with each new discovery.
Related posts
References
Ms. Hepp is a researcher and communicator who has been writing and editing educational content on varied health topics for more than 20 years. She serves as lead researcher and writer for CancerChoices and also served as the first program manager. Her graduate work in research and cognitive psychology, her master’s degree in instructional design, and her certificate in web design have all guided her in writing and presenting information for a wide variety of audiences and uses. Nancy’s service as faculty development coordinator in the Department of Family Medicine at Wright State University also provided experience in medical research, plus insights into medical education and medical care from the professional’s perspective.
Follow this blog
Subscribe to be notified of new posts