The research paper: How science gets from the laboratory to the public
READ MORE
Why should I believe a scientist?
While the pandemic exacerbated mistrust in scientists, confidence in research has been slowly eroding for decades. This series of stories peels back the curtain to give you a behind-the-scenes look at the people and processes behind scientific research, with the hope that understanding how science works will help you decide when to be a skeptic (and how to become a better one).
Expand All
Confidence in science has plummeted. Here’s why we might want to rethink that.
Scientists at work don’t act much like the scientists you see on TV
The research paper: How science gets from the laboratory to the public
Scientists need money to pursue research. But taking money is not without risk.
A basic guide to reading and vetting scientific articles
Without scientific articles, researchers might spend their whole careers siloed in their labs, performing experiments and making discoveries that no one ever knows about.
“It doesn’t really have any value if nobody knows the results,” said Mohamed Noor, professor of biology and interim dean of Arts & Sciences at Duke University.
Researchers disseminate their findings in a standardized and dependable way via scientific papers, which allows scientists to build on the work of their peers.
Publishing papers is so central to the scientific process that there is a well-known aphorism in academia: publish or perish.
Peer Review
Before a scientific paper can be published in a journal, it must be thoroughly examined and questioned in a process called peer review. During that process — which can take weeks to months — journal editors reach out to a handful of subject experts and ask for their unpaid feedback on the paper in question.
Those experts, usually other academics in the same field, look at the paper and consider whether the writing is clear, the methods are rigorous and the findings meaningfully add to the field, among other things.
Crucially, peer reviewers evaluate the study’s reproducibility, or whether other scientists can perform the same experiment and get the same results. This allows researchers to vet each other’s work and build on past findings after the paper is published.
The reviewers tell the editors whether the paper should be accepted into the journal and typically include a list of ways the paper could be improved.
Even though scientific research is thoroughly reviewed, mistakes — and even falsified results — can seep through.
Duke University retracted several scientific articles and paid the U.S government millions of dollars in penalties and to repay grants after a biologist falsified data in several federal grant applications.
In fact, there is an entire publication dedicated to reporting daily instances of scientific paper retractions.
One of the most famous errors was a scientific article published in the Lancet, a highly prestigious medical journal, that claimed to have found a link between vaccines and autism. The study is widely believed to have set off the modern day anti-vaccination movement and caused the MMR vaccination rates to plummet.
The study has been repeatedly proven wrong in subsequent studies and was retracted by the journal in 2010 after it was revealed the researcher manipulated his data and accepted funding from a personal injury lawyer to publish the study.
Will Vizuete, a climate researcher at UNC-Chapel Hill, said the public shouldn’t necessarily accept the results of each individual study as fact.
Disagreement about how to interpret data or the best way to perform an experiment is not only common among scientists, but an important part of the scientific process.
Scientists often clash with one another and will sometimes go to extreme lengths to prove each other wrong. In the 1980s, gastroenterologist Barry Marshall was so intent on showing that doctors were treating stomach ulcers incorrectly that he drank a concoction of beef broth and bacteria to prove he could treat his self-induced ulcers with antibiotics (he was later awarded a Nobel Prize for his work).
Scientific consensus, which researchers reach through countless studies and peer-reviewed papers, is what you can rely on.
For example, while one study might question the veracity of climate change, 97% of climate scientists agree that humans are causing global warming.
“You can’t trust one paper in its own right,” Vizuete said. “It’s the collection of things that matter.”
Not all journals are equal
Some publications present themselves as reputable journals but do very little, if any, vetting before publishing a scientific paper.
Noor said scientists have submitted bogus articles to these journals to expose their lax publishing standards. About four years ago, a biologist wrote a paper that recounted an episode of “Star Trek” in scientific jargon and submitted it to some of these journals.
The American Research Journal of Biosciences published it, despite the pseudonymous author, Dr. Lewis Zimmerman (a “Star Trek” character), thanking the United Federation of Planets at the end.
“They might have some science that is legit,” Noor said. “But they accepted a paper that just recounted the scenes in this episode of ‘Star Trek.’”
Noor, a biologist himself, declined to say who pulled the prank.
“It was somebody, I can’t imagine who, who was a ‘Star Trek’ fan and a biologist,” Noor said against a backdrop of “Star Trek” posters and figurines.
An easy (and somewhat blunt) way to gauge the legitimacy of a journal is to Google its impact factor, which tells you how often the average article is cited by other scientists. The most prestigious scientific journals, like Nature or Science will have an impact factor in the 60s. A journal like the one that published the “Star Trek” article will typically have an impact factor less than one.
Imperfect system
The peer review process is not immune to human biases.
“It’s every bit as competitive and political — and in some ways, even more so — than any other sort of human endeavor,” said Paul Brinkman, a science historian at NC State University.
Michael Dickey, an engineering professor at NC State, said papers authored by well-known scientists may have an advantage over those written by lesser-known researchers. Journals may be incentivized to publish articles by scientists that they know will have a wide readership, he said.
He experienced that first-hand during a postdoctoral fellowship at Harvard University, where he worked under one of the most renowned chemists in the world. The papers he submitted with his adviser’s name attached were accepted into highly prestigious journals — journals he isn’t sure would have accepted the paper a year later when the affiliation was N.C. State instead of Harvard.
“If I took his name off the paper — I’m just guessing, but I don’t think that they would have gone through,” he said. “There’s no doubt in my mind there’s some human bias.”
Some journals have tried to make the process double-blind, which conceals the identity of both the paper’s authors and the reviewer.
But Brinkman said scientific fields are so hyper-specialized that it’s obvious who wrote a paper (how many researchers really study neuroethology in midshipman fish).
“I get asked to review the same authors all the time,” Brinkman said. “Sometimes you can tell by the phrases or words someone uses — I immediately know it must be my friend.”
Despite its flaws, Brinkman still insists peer review is the best strategy for vetting the quality of scientific research.
“Can you think of a better way to take people’s perspective on publications and to decide which ones should go forward and which ones should not?” he said. “I can’t think of a better way to do it.”
This story was originally published November 9, 2022 at 6:00 AM with the headline "The research paper: How science gets from the laboratory to the public."
