Peer review is the system we have in place to prevent duff papers slipping through the net. It’s not perfect, and recently it has come under increased scrutiny as more and more papers’ findings (mainly in psychology) have not been reproducible when the same steps are followed.
But a new study in France has revealed one serious issue that could be making it less reliable over time: only about 20% of scientists are taking the time to review their peers’ work, and the number of peer-reviewed papers has shot up 204% since 1990.
“We showed that a small portion of the scientific community is carrying a disproportionate load of the peer review,” concluded the researchers from Paris’ Descartes University. “This inequality may be the root of a potentially unmanageable burden. These ‘peer-review heroes’ may be overworked, with risk of downgraded peer-review standards.”
Using a mathematical model, the researchers examined all the biomedical and life science papers listed in MEDLINE between 1990 (372,589) and 2015 (1,134,686). They made estimates as to how many rounds of review the papers would have gone through, and how many hours it would take, and then compared this to the number of researchers available for peer review work. The paper assumes 25% of papers will be rejected and that 90% of papers would go through a second round of reviews, at around four to five hours reviewing a paper. That means that 2015’s papers added up to a whopping 63.4 million hours of review work.
READ NEXT: The elephant in the lab. How much science is fabricated?
And that work is really unevenly distributed. Last year, around 1.8 million researchers did the lion’s share of the peer review work, while around 4.6 million contributed very little. The team’s full estimates were that 70% dedicated 1% or less of their research time to peer review, while 5% dedicated over 13% of their schedules to keeping the system afloat.
The good news (kind of) is that this model is sustainable. It’s just a bit unfair to the people doing the heavy lifting. “The current peer-review system is sustainable in terms of volume but the distribution of the peer-review effort is substantially imbalanced across the scientific community,” the researchers write.
Despite this, they aren’t in favour of ditching peer review, despite its imperfections: chiefly because there’s nothing that’s proven to be better.
“The evidence base for alternative peer-review systems is still sparse. An evidence-based approach to study peer review, combining computer modelling, experimental studies and sharing of data from journals and publishers, should be encouraged.”
To paraphrase Churchill, peer review might be the worst method of scientific scrutiny, apart from all the others.
“Improvements in peer review will come in response to evidence,” the researchers conclude.
Images: Nic McPhee and Nic McPhee used under Creative Commons
Disclaimer: Some pages on this site may include an affiliate link. This does not effect our editorial in any way.
