Do They “Cherry Pick” Scientific Studies?
Author: Dr. Stephen Chaney
When we buy a food supplement from a company we assume that it will provide a benefit. We are trusting that company to be honest in their product claims. But, are there any honest nutritional supplement companies?
- What if they were lying to us?
- What if they had no clinical studies done with their product?
- What if they were just quoting studies done with ingredients found in their product?
- What if they were “cherry picking” the studies they listed to support the claims they wanted to make?
Unfortunately, that happens far too often in the nutraceutical industry. As an example, I came across an article in a recent issue of www.nutraingredients.com about a FDA warning letter (http://www.fda.gov/ICECI/EnforcementActions/WarningLetters/2016/ucm518533.htm) to a noni juice company. In case you are wondering, noni fruit is the latest in a long line of “magical fruits” that is going to cure everything that ails you.
The thing that brought this company to the FDA’s attention in the first place was the health claims the company made on their website. The company claimed or implied that their product would cure cancer, cure gout, cure arthritis, lower cholesterol, and help fight infections. Claims like that always invite FDA scrutiny.
What caught my attention, however, was the quote by an attorney specializing in FDA compliance issues that the studies cited on their website were “cherry picked” to support their claims. He said that the studies they cited “…do not meet the standards of third party literature…You have to include a full range [of published studies], and not just cherry pick the positive studies. It has to be a balanced presentation. It looks like they just did a literature search on noni and included only the positive studies.”
That statement caught my attention because it doesn’t just apply to just this one company. It is a practice that is common in the nutraceutical industry. Many supplement companies cherry pick studies from third party literature. They list only the studies that support their product claims and ignore the rest. That is misleading because it implies a level of proof for their product claims that does not exist. It is fundamentally dishonest. These are certainly not honest nutritional supplement companies.
Using Borrowed Science
The noni juice company cited in the FDA letter had no clinical studies to support their claims. Instead they quoted studies done with ingredients found in their product. This is what I call “borrowed science.”
I call this “borrowed science” because the studies were not actually done with their products. They were simply trying to “borrow” results done with individual ingredients and pretend that they applied them to their product.
Let me be clear. Third party studies done with ingredients found in a company’s product are of little value in predicting whether that product will provide any benefit to you. To claim otherwise is dishonest. Again, these are not honest nutritional supplement companies.
There are several reasons this is true.
- In many cases, the amount of that ingredient provided by the supplement does not match the amount actually used in the clinical study they quote. The ingredient may or may not be effective at the dose provided in the supplement.
- More importantly, a supplement usually contains multiple other components that may influence how a single ingredient works in your body. The other components may enhance the effectiveness of the ingredient in question, or they may inhibit it.
- Without clinical trials done with their product, companies actually have no idea whether their product works or not.
Unfortunately, I see this practice all too frequently in the nutraceutical industry. Clinical trials are expensive. It’s cheaper and easier to search the literature for published studies you can “borrow” to support your product.
Honest Nutritional Supplement Companies Do Not“Cherry Pick” Studies
Even worse, many companies cherry pick studies from the literature to support the product claims they want to make.
To understand what that statement means you need to know a little bit about the scientific method. Most scientists design their experiments to disprove what other scientists have published. This is a self-correcting process that is a strength of the scientific method.
However, it also means that you will find articles in the literature supporting and refuting the benefits of almost every nutraceutical ingredient. The scientific community waits until enough studies have accumulated and then relies on the weight of evidence before drawing any conclusions.
Unfortunately, unscrupulous supplement companies decide first on what claims they want to make and quote only the studies that support those claims. This is what is referred to as “cherry picking” the studies.
The Dietary Supplement Health and Education Act of 1994 (otherwise known as DSHEA) is very clear about that. Section 5 of DSHEA states “…scientific journal articles, books and other publications can be used in the sale of dietary supplements provided…[they] are presented with other materials to create a balanced view of the scientific information…”
In plain words this legalese simply means that you can’t cherry pick studies. You can’t select only the studies that support your product claims and ignore those that don’t. Honest nutritional supplement companies would not use these deceitful practices.
However, this is a practice that I see all too often in the nutraceutical industry. It is dishonest. It is disgraceful
Are There Any Honest Nutritional Supplement Companies?
The bad news is that there are lots of supplement companies that do no clinical studies of their own. Instead they rely on borrowed science from studies that really do not provide proof that their products are either safe or effective. Even worse, many of those companies cherry pick only the studies that support their product claims and ignore studies that do not. This is a practice I regard as clearly dishonest. Those are companies I would avoid.
The good news is that there are a few companies that actually support clinical studies on their key products and publish those studies in peer reviewed scientific journals. Those are companies worthy of your consideration.
There are other things to take into account in selecting the best of the best – things like the number of studies and the quality of the studies. However, that’s a topic for another day.
Many Blogs Cherry Pick As Well
I can’t leave this topic without pointing out that many popular health and nutrition blogs, including those written by some well-known doctors, do exactly the same thing.
The pressures that lead to this behavior are obvious. The very popularity of these blogs depends on them being sensational week after week.
Unfortunately, true science is rarely sensational. It’s usually pretty wishy-washy. If you do a complete search of the literature, you usually find articles that are both for and against any point of view you wish to express. Occasionally, enough evidence accumulates on one side of an issue that scientists are willing to come to a definitive conclusion, but that conclusion is hardly ever sensational.
The only way that the authors of these popular blogs can make sensational claims each week is to cherry pick only the studies that support their point of view and ignore everything else.
Unfortunately, the average reader doesn’t realize this. They see the list of references supporting the claims and believe what they read. Then these bizarre claims get reposted over and over until the general public actually starts believing that they are true.
It really is a shame that DSHEA doesn’t apply to blogs. If it did, they wouldn’t be nearly as sensational, but they would be much more accurate. They would have to report on the whole body of scientific literature, rather than cherry picking just the studies that support their point of view.
In conclusion, there are some honest nutritional supplement companies, but be sure the company you choose to believe is citing studies on their actual products and not just ingredients in their products. Also, watch out for “cherry picking.”
The Bottom Line
- The FDA recently sent a warning letter to a noni juice company for making unsupported health claims for their product. The company was claiming their product could cure things like cancer, gout and arthritis. Whenever a company makes claims like that, they can expect to draw the attention of the FDA.
- An outside attorney specializing in FDA compliance pointed out that the company also had no good evidence to support their product claims. The company had done no clinical studies on the products. Instead they had “borrowed” the results of third party studies done with ingredients found in their product. Even worse, they had cherry picked only the studies that supported their product claims and ignored the studies that did not.
- Third party studies done with ingredients found in a company’s product are often worthless in predicting whether that product will provide any benefit to you. I discuss the reasons for that in the article above.
- Cherry picking only the studies that support a company’s product claims runs afoul of the Dietary Supplement Health and Education Act of 1994 (DSHEA) requirement that companies provide a balanced view of the scientific literature relating to their products. It is also misleading and dishonest.
- Unfortunately, the practice of using “borrowed science” from third party studies and cherry picking only the studies that support their product claims is common in the nutraceutical industry. Supplement companies that rely on this kind of evidence to support their product claims are dishonest and should be avoided.
- For products you can trust, choose companies that support clinical studies on their key products and published those studies in peer-reviewed journals. You should also look at the number and quality of studies, but that is a topic for another day.
These statements have not been evaluated by the Food and Drug Administration. This information is not intended to diagnose, treat, cure or prevent any disease.