healthcare interventions Archives - Health Tips From The Professor

You Need To Be Your Health Advocate

Author: Dr. Stephen Chaney

I taught medical students for 40 years. During that time, I did my best to emphasize the importance of basing their practice on “evidence-based medicine”. So, it broke my heart when I saw recent headlines claiming that more than 90% of healthcare interventions (drugs and medical procedures) were not based on high-quality evidence.

Even worse, the headlines claimed that the harm caused by healthcare interventions was not adequately investigated and may, therefore, be under-reported.

When I saw these headlines, I knew I had to investigate further to see if the claims were true and report what I found to you, my readers.

I would not have been surprised by headlines claiming that some healthcare interventions were based on low-quality evidence. For example:

Hormone replacement therapy was widely prescribed to manage menopause symptoms until it was discovered to increase the risk of breast cancer. Since then, the hormones used have been reformulated, it is only recommended for severe menopause symptoms, and only for the shortest possible time.

Antiarrhythmic drugs were widely prescribed to reduce mortality from heart attacks until a placebo-controlled trial showed they actually increased mortality.

A drug called oseltamivir was widely prescribed for the flu until a systematic review of clinical studies showed it was ineffective.

But I, like many of my colleagues, assumed that these cases were rare. However, recent reviews have called this assumption into question. But most of those reviews had a small sample size or did not adequately evaluate the quality of the studies included in the review.

The study (J Howick et al, Journal of Clinical Epidemiology, 148: 160-169, 2022) behind these headlines was designed to avoid those limitations and provide a more accurate estimate of the percentage of clinical interventions that are based on high-quality evidence.

It evaluated 1,567 healthcare interventions that had been studied in Cochrane Reviews, which are considered the gold-standard of evidence-based medicine (I will describe Cochrane Reviews in more detail below, so you can appreciate why they are considered the gold standard).

What Is A Cochrane Review?

At this point you are probably wondering what the Cochrane Review is and why it is considered the gold standard of evidence-based medicine. I have covered this in previous articles. But I am not expecting you to remember it (I never told you there would be a quiz). So, I will repeat the information here.

The Cochrane Collaboration consists of 30,000 volunteer scientific experts from across the globe whose sole mission is to analyze the scientific literature and publish reviews of health claims so that health professionals, patients, and policy makers can make evidence-based choices about health interventions.

The Cochrane Collaboration reviews all the relevant studies on a topic, exclude those that are biased or weak, and make their recommendations based on only the strongest studies. They use a systematic approach called GRADE (Grading of Recommendations, Assessment, Development, and Evaluation) that has been endorsed by over 100 organizations worldwide to assess the quality of the studies. The scientists writing Cochrane Reviews are trained in how to use the GRADE evaluation system before they are allowed to write a review.

In one sense, Cochrane reviews are what is called a “meta-analysis”, in which data from numerous studies are grouped together so that a statistically significant conclusion can be reached. However, Cochrane Collaboration reviews differ from most meta-analyses found in the scientific literature in a very significant way.

Many published meta-analyses simply report “statistically significant” conclusions. However, statistics can be misleading. As Mark Twain said: “There are lies. There are damn lies. And then there are statistics”.

The problem is that the authors of most meta-analyses group studies together without considering the quality of studies included in their analysis. This creates a “Garbage In – Garbage Out” effect. If the quality of individual studies is low, the quality of the meta-analysis will also be low. Simply put, the conclusions from some published meta-analyses are not worth the paper they are written on.

The Cochrane Collaboration also reports statistically significant conclusions from their meta-analyses. However, they also carefully consider the quality of each individual study in their analysis based on the GRADE system. They look at possible sources of bias. They look at the design and size of the studies. Finally, they ask whether the conclusions are consistent from one study to the next. They clearly define the quality of evidence that backs up each of their conclusions as follows:

High-quality evidence.Further research is unlikely to change their conclusion. This is generally reserved for conclusions backed by multiple high-quality studies that have all come to the same conclusion.

Moderate-quality evidence.This conclusion is likely to be true, but further research could have an impact on it.

Low-quality evidence.Further research is needed and could alter the conclusion. They are not judging whether the conclusion is true or false. They are simply saying more research is needed to reach a definite conclusion.

Now perhaps you understand why Cochrane Reviews are considered the gold standard of evidence-based medicine.

How Was This Study Done?

The authors started with 6928 reviews that compared a healthcare intervention with either a placebo or no intervention between January 1, 2008, and March 5, 2021. They then randomly selected 1,567 reviews for this study. They asked the following 3 questions for each Cochrane Review:

Was the evidence for a positive outcome high-quality, as rated by the GRADE system?

Were the results statistically significant?

Did the review authors consider the intervention to be effective?

Is Your Doctor’s Advice Based On Good Science?

As I said earlier, the results were unnerving to say the least. When the authors applied their 3 criteria to the 1,567 Cochrane Reviews they found:

Only 10% of the medical interventions (drugs and medical procedures) were supported by high-quality evidence.

- In other words, 90% of the time the evidence wasn’t good enough to determine whether the intervention worked or not.

Only 6.8% of the interventions studied had a positive, statistically significant outcome.

Only 5.6% of the medical interventions studied were judged to be effective by the Cochrane Review authors.

The harm of medical interventions was poorly studied. Only 36.8% of clinical interventions were evaluated for potential harms of the intervention, and most of those data were of low quality.

- Of the Cochrane Reviews that evaluated potential harms, there was evidence of statistically significant harm in 22% of the interventions.

The authors concluded,

“Using rigorous methods for judging quality of evidence, more than 9 in 10 healthcare interventions studied within Cochrane Reviews do not have high-quality evidence to support their effectiveness and safety. This probably can be remedied by high-quality studies in priority areas.”

“Potential harms of healthcare interventions were measured more rarely than benefits…These studies should measure harms as rigorously as benefits.”

“Practitioners and the public should be aware that most frequently used interventions are not supported by high-quality evidence.”

Putting This Study Into Perspective

I should start by saying that this study does not reflect poorly on your doctor. They have your best interest in mind, and they are doing their best to keep up with a constantly changing medical landscape.

In most cases, the advice your doctor gives you is based on clinical guidelines issued by medical societies and government agencies. This study is an indictment of those agencies for not evaluating the quality of the clinical studies used to formulate their clinical guidelines.

With that out of the way, it is fair to ask whether criteria these authors used were too strict. And, in fact, the authors gave this quite a bit of thought. Here are some of the questions the authors asked.

Were the authors of the Cochrane Reviews biased in their evaluation? While you can never eliminate the possibility of bias:

- The Cochrane Collaboration puts a great deal of effort into training reviewers in how to use the GRADE system without bias.

- If the opinions of the review authors were removed as a criterion, it would have a minimal impact on the outcome of this study. As reported above, only 6.8% of healthcare interventions had a positive outcome that was supported by high-quality data.

2) Is the GRADE system for evaluating the quality of clinical studies too stringent? The authors considered this possibility, but:

- Prior to GRADE individual meta-analyses used different methods to evaluate the quality of clinical studies, so it was difficult to compare the conclusions of these meta-analyses.

- The GRADE system was designed, evaluated, and accepted by top experts around the world to unify how the quality of clinical studies is evaluated.

- This study found that only 30% of the healthcare interventions were supported by even moderate quality evidence according to GRADE. In other words, even when less stringent standards are used, a high percentage of healthcare interventions may be ineffective.

3) Does relying solely on Cochrane Reviews underrepresent the percentage of healthcare interventions based on high-quality evidence?

- Cochrane Reviews are primarily undertaken for interventions that are controversial and/or a least one major study suggests the intervention may be ineffective or harmful. Since Cochrane Reviews are less likely to have been conducted on well-established, non-controversial healthcare interventions, it is possible that this study underrepresented the percentage of healthcare interventions backed by high-quality studies.

- However, a recent study that did not use Cochrane Reviews or the GRADE system concluded that only 22% of healthcare interventions were likely to be beneficial. Once again, even when less stringent standards are used, a high percentage of healthcare interventions may be ineffective.

However, the authors did point out that there may be situations in which high-quality evidence is not needed to recommend a particular healthcare intervention. For example, when inaction leads to dire consequences and there are no other treatment options, a healthcare intervention supported by moderate or low-quality evidence might be preferable to no action at all.

You Need To Be Your Health Care Advocate

You are probably wondering what this study means for you. Unfortunately, the authors of this study did not provide a list of healthcare interventions that were not supported by high-quality evidence. So, I can’t provide you with a list of interventions to avoid.

At one point, the authors of this study said, “Patients, doctors, and policy makers should consider the lack of high-quality evidence supporting the benefits and harms of many interventions in their decision-making.” However,

Doctors are very busy. They don’t have time to read and evaluate the quality of clinical studies. They rely on the clinical guidelines issued by policy makers (medical societies and government agencies).

Policy makers don’t like to admit they were wrong and are very slow to revise their clinical guidelines.

That means you must be the advocate for your health. I’m not suggesting that you question every recommendation your doctor makes. However, you should research major healthcare interventions and discuss the pros and cons with your physician.

Dr. Google can be wildly inaccurate, but it is a place to start. You can look up the side effects of drugs your doctor is recommending, downsides of medical treatments they are recommending, and/or other treatment options for your medical condition. I like to focus on reliable sites such as the Cleveland Clinic, Mayo Clinic, and the NIH. WebMD is often, but not always, a reliable source.

Ask other health professionals about alternative approaches and/or other doctors they may recommend for your condition. Ask friends who have had the same condition what medical interventions worked for them and if they have other doctors they recommend.

Then discuss some of these with your doctor. He or she should be willing to discuss the pros and cons of their recommendations and alternate approaches. If not, ask for a second opinion or consult other doctors.

Of course, you should always be open to the possibility that no other good options exist, and some intervention is essential. I’m just suggesting you evaluate your options fully and discuss them with your doctor before starting any major healthcare intervention.

The Bottom Line

A recent study evaluated quality of clinical studies supporting many common healthcare interventions (drugs and medical treatments). The study found that:

Only 5.6% of healthcare interventions studied were supported by high-quality evidence.

This study is not an outlier. Previous studies have come to similar conclusions.

The evidence of harms caused by healthcare interventions has not been adequately studied and could be as high as 22%.

As someone who taught the importance of evidence-based medicine to medical students for 40 years, I was appalled by this finding. And as patients trying to navigate the medical system, you should be appalled as well.

The authors of this study said, “Patients, doctors, and policy makers should consider the lack of high-quality evidence supporting the benefits and harms of many interventions in their decision-making.” However,

Doctors are very busy. They don’t have time to read and evaluate the quality of clinical studies. They rely on the clinical guidelines issued by policy makers (medical societies and government agencies).

Policy makers don’t like to admit they were wrong and are very slow to revise their clinical guidelines.

For more details on this study and my discussion of how you can research major healthcare interventions recommended by your doctor, read the article above.

These statements have not been evaluated by the Food and Drug Administration. This information is not intended to diagnose, treat, cure, or prevent any disease.