How To Game The Peer Review Process
Author: Dr. Stephen Chaney
Peer review is supposed to assure that clinical studies are well designed, well executed, and correctly interpreted before they are accepted for publication. That is why I frequently advise you, my readers, to look for clinical studies on their nutritional products that are published in peer reviewed scientific journals as a criteria for choosing a supplement company that you can trust.
But, can the system be gamed? Sadly, the answer is yes. There are journals that only pay lip service to the peer review process. Earlier this year an investigative reporter set out to prove just how easy it is to game the peer review process. It is a fascinating story of how a journalist created the fake chocolate study.
Creating The Fake Chocolate Study Hoax
The fake chocolate study was a hoax put together by John Bohannon, an investigative journalist and correspondent to Science (a very well respected scientific journal) to test the peer review system. The study was real, but it was seriously flawed. For example, it had only 16 subjects, there was no effort made to determine what the subjects were eating other than chocolate, and the conclusions were not supported by the data. In short, it was a very bad study—one that would have been rejected by any reputable journal.
For the purposes of the test he called himself Dr. Johannes Bohannon from the “Institute of Diet and Health”, a nonexistent entity that consisted of nothing more than a fake website he set up. He then wrote up the study and titled it “Chocolate with High Cocoa Content as a Weight Loss Accelerator” with the conclusion (not supported by the data) that “Long-term weight loss, however, seems to occur easier and more successfully by adding chocolate. The effect of chocolate, the so-called ‘weight loss turbo’, seems to go hand in hand with personal well-being, which was significantly higher than in the control groups.”
Journals Take The Bait
In March 2015 he submitted the article to 20 online journals. Several accepted it within 24 hours. He chose to publish it in the “International Archives of Medicine.” His paper was published online without any revisions a mere two weeks later. [Note: You should not assume the fact that several out of just 20 journals accepted his paper as in indication that a significant percentage of journals accept sub-standard papers without serious peer review. He had, in fact, done previous research for Science magazine identifying those journals most likely to accept flawed studies. It was those types of journals he sent his study to.]
John Bohannon was later quoted as saying “Editors of reputable journals reject [these kinds of studies] out of hand without even sending them to peer reviewers. But there are plenty of journals that care more about money than reputation.” [It costs $650 to have an article published in the International Archives of Medicine.]
The Media Fans The Flames
If this study had just been published in an obscure journal and had been ignored, that would have been bad enough. But the story gets even worse. He then created a press release that he sent to news outlets. The press release made some pretty outrageous statements and even contained a link to an unrelated music video. However, the study made news headlines in more than 20 countries in half a dozen languages. For example, headlines from the Daily Express in England blared: “Chocolate Accelerates Weight Loss: Research Claims it Lowers Cholesterol and Aids Sleep.”
John Bohannon’s take was: “The key is to exploit journalist’s incredible laziness. If you lay out the information just right, you can shape the story that emerges in the media almost like you were writing those stories yourself. In fact, that’s literally what you are doing, since many reporters just copied and pasted our text.”
For the complete story of how John Bohannon pulled off this hoax, read his blog post about the “fake study.”
The Bottom Line
- An investigative reporter for Science magazine demonstrated recently just how easy it is to get a fake study published in a peer-reviewed journal.
- He created a completely bogus study about chocolate aiding weight loss.
- He submitted it to several journals that he had previously identified as having substandard peer review processes.
- Several journals accepted it within 24 hours without any peer review process. One published it two weeks later without any revisions.
- The story of a new “study” showing that chocolate aids weight loss was picked up by the new media and made the headlines in more than 20 countries in half a dozen languages.
- The vast majority of journals have a very exacting peer review process, which means that most published clinical studies have been thoroughly reviewed and edited prior to publication. For the most part the peer review process works exactly as it is supposed to.
- However, this hoax shows just how easy it would be for an unethical supplement company to subvert the peer review process and publish bogus studies to support their product claims. That is why when you are choosing a supplement company it is important to choose one with a reputation for scientific integrity.
- This hoax also shows just how easy it is for bogus information to be picked up by the media and make it into the headlines. You simply cannot believe everything you read in the press, see on the TV and view online. That is why I created “Health Tips From The Professor.” I wanted to create a place where you can go for accurate information.
These statements have not been evaluated by the Food and Drug Administration. This information is not intended to diagnose, treat, cure or prevent any disease.