Why Randomized Controlled Trials Don’t Always Tell the Real-World Story

We have found some common guidelines that help us determine whether studies and websites can be trusted. We share what we find valuable in a series of blog posts. Andy Jackson, ND, and Miki Scheidel reviewed and provided input on this post.

Why RCTs are considered good research

As mentioned in a previous post in this series ›, randomized controlled trials (RCTs) are promoted as the “gold standard” of clinical trials. They are very good at some key aspects of medical research.

First, RCTs allow researchers to isolate the effect of a single intervention, such as a drug. Everything else reasonably known to affect treatment is controlled, usually through randomization.

Second, they reduce the bias we see in observational studies in which participants choose which treatments to take or lifestyle behaviors to follow. Many factors can be involved in why a person may choose one treatment over another, including cost, distance to treatment, insurance coverage, recommendations from family and friends, past experience, and on and on. If the “treatment” being studied is a behavior change and not a drug, then a different set of preferences may be in play. 

Example: Researchers look for a connection between the practice of yoga and fatigue in an observational study. They ask people whether they practice yoga or not, and then they compare the level of fatigue between people who do and people who don’t. But people who practice yoga may have many other differences compared to people who do not practice yoga. Perhaps people who practice yoga tend to have more education, different types of jobs, higher family income, younger or older age, less disability, less illness, less use of tobacco or alcohol, less use of medications, and on and on. Some of these differences could affect the level of fatigue between the two groups. In a large enough randomized study, we randomly assign people either to practice yoga or not (and we provide all the support and infrastructure they need). Any differences between groups are assumed to balance out, and we observe how the practice of yoga affects fatigue levels. We can have some confidence that the difference between groups was due to the practice of yoga and not something else.

What RCTs aren’t as good at

RCTs are not always the best approach to get definitive answers to clinical questions. Studying an intervention or exposure using an RCT design is not always feasible, for one. 

Example 1: Not many RCTs have studied cannabis use so far because until very recently the drug was illegal in most places. It still is in many places. Such studies may still not be eligible for grant funding or for approval from an institutional review board (IRB) that decides whether human subjects can be used in experiments. Without funding, and definitely without IRB approval, very few quality studies happen.

Example 2: Researchers want to find out if metformin contributes to cancer survival among people with diabetes. But designing a study to randomly assign diabetic people to receive either metformin or a placebo would be a huge problem. Withholding a needed treatment for a disease is unethical, even for the goal of studying its effects on another disease. The best researchers could do is compare metformin to other diabetes medications. But not all diabetes medications are equally effective at controlling high blood sugar, and so finding a comparable control group would be a real challenge.

RCTs are also not very good at determining how a treatment will work in the messy “real world” situations in which most of us live, and most medical professionals practice. RCTs typically have exclusion criteria—a list of conditions that will keep someone from participating in the study—and these can be quite long. In studies, people may be excluded if they have any medical conditions that are related to cancer or that might interfere with the treatment effect—or that may cause people not to finish the study. This might include diabetes, obesity, liver disease, cardiovascular disease, and many more conditions. Pregnant women are almost always excluded. People undergoing other treatments are often excluded, as are people whose cancer is too advanced, who are over or under a certain age, who are taking certain supplements, and so on. But these are the very people that health professionals are trying to treat every day. 

RCTs try to control as many variables as possible, but in doing so they may create conditions that are so artificial that the study results may not apply in a clinical setting. In the real world, outcomes may be different. That’s why observational studies—what clinicians see in practice—matter. In some cases, observational studies that come closer to real-life situations may provide more valuable information for clinical use than highly controlled studies that are considered the “gold standard” by researchers. 

Hybrid and individualized research approaches

We think there’s a place for both types of studies, and many researchers agree. Researchers have developed a rating system to evaluate the strength of study designs and the rigor of their conclusions. GRADE (Grading of Recommendations Assessment, Development, and Evaluation) is a formal, scientific rating system. Similar to the hierarchy of evidence in our previous post ›, GRADE prioritizes randomized controlled trials but also considers information from observational studies when necessary. 

Finally, as we mentioned in Understanding Research Studies ›, research studies judge the success of treatments and therapies at a population level, not an individual level. Studies of groups are not designed to identify a specific, optimal combination of therapies, nutrition, movement, and so on, all aligned with one person’s circadian rhythms, disease history, and much more. Research studies, whether for conventional or complementary therapies or for behaviors and lifestyle practices, ultimately only give an experienced health professional a good starting place for treatment. Depending on each person’s response, treatment may need to be adjusted, perhaps several times. We at CancerChoices think this is the true “gold standard” of treatment, drawing heavily from quality research studies, but not limited to them. 

Bottom line: using research in your own treatment plan

For people with cancer, research on cancer therapies is a necessary starting point for a collaborative approach between you and your healthcare team. Carefully selecting therapies based on evidence, experience, and individual characteristics can lead to an individualized treatment plan appropriate for your situation. Monitoring your response and adjusting as needed are core parts of a successful plan.

We provide handbooks on specific topics, based on our reviews of complementary therapies and self-care practices.

Helpful link

Other posts in this series