The Cost of Peer Review

(Update added April 20, 2015, see the end of this post.)

I have been procrastinating for while about setting up this blog, but recent events have pushed me over the bump, so here it is. In the past week about 145 editors have resigned (including yours truly) from the editorial membership board of the journal Scientific Reports, for a cause supported now by over 550 signed supporters and counting. The editorial membership board of SciRep has thousands of  members.  Sorry for the long post, but the issues are serious.

On March 24, 2015, Scientific Reports (SciRep from here on),  an open access journal owned by NPG/Palgrave Macmillan has sent out a letter to its editorial board members, alerting us about an experiment in peer review they were conducting. See here for additional info. In a nutshell, SciRep has launched a small-scale experiment, only in biology, which would secure fast peer review for those authors who are willing to pay for it.  It guarantees to return the reviews to the authors within three weeks,  or their money back. To accomplish this, SciRep employs the services of a 3rd party, Research Square, using their peer review system Rubriq. More details are revealed here. Currently, fast-track service seems to be set at \$750 a pop which would be in addition to the \$1,495 fee for publishing the paper. It is not clear how much of that goes to the reviewers, I suspect a small fraction (\$100 ?) as the rest goes to the for-profit parts of the setup.

This has led to an outcry from the part of the editors, for reasons outlined in an open letter that can be found here: . This letter is also a letter of resignation by those editors, effective immediately. If you are an editor for this journal, please read this letter and consider the reasons for it and the arguments below; hopefully you’ll sign it too. Even if you are not an editor, as a scientist, please consider supporting the cause. As I will try to outline below, the issues are larger than this particular problem, and we all need to be aware of them.

At first, this does not seem to be a bad idea: paying a reviewer for her/his work. Refereeing is hard and time-consuming and we do it for free because we feel it is our responsibility to keep up the quality of science. However, the SciRep trial is about something else: a referee gets paid if she/he returns the report within a short time frame (3 weeks) and only for those authors who opt to pay for it! And, for now, only within biology. Here are only two of the issues that this pay-for fast-track review introduces, others are raised in the open letter: 1) Since the monetary incentive is visibly only tied to the return time, this will put a pressure on referees that may lead to erroneous or superficial reviews. I frankly would want to have a better quality review than a non-informative fast one. Good quality review takes time because research is hard and good research is even harder, and thus judging difficult things will be highly non-trivial. It often takes years before someone is able to make progress on a difficult problem. Are we saying that it can be reviewed for correctness within 3 weeks by someone who has several other papers to write, referee, handle editorially, teach classes, mentor students, write grants and write reports? So maybe the goal is not to make sure that reviews are good, only to  be fast; you might wonder why. 2) This sets up a two-tier system. The Haves will get quick and superficial reviews and publish before the Havenots. Just to be the Devil’s advocate, consider the extreme case  of a conference where someone talks about their latest results that have just been submitted and are under review. Hypothetically, someone else at the conference calls up their students to use their own data and simulations and publish the same idea before the original author, using pay-for fast-track. The key word here is publish (because people can always post to archives). This incentive system, in general, will create advantages and advances for some people based not on scientific excellence, but on other criteria (on funding availability). It will definitively disadvantage the younger generation that normally has less funding; or folks from poorer countries.

The journal has responded to the questions raised in the (now) open letter, the response can be read here. To questions about the qualifications of the peer review coordinators at Rubriq the reply was a brief “minimum a PhD degree in biomedical sciences”. The answer was clearly unsatisfactory and the letter went open. There is wide range of variation in expertise within scientists with a PhD. The scientific hierarchy is very large, ranging from the freshly graduated, still green-eared PhDs (no offense, we all were there at some point) to members of the National Academy of Sciences, Nobel Prize winners, etc. Why not try to enlist the biggest and most respected names in a research area who genuinely care about the dissemination of good science? Why eliminate expert based peer-review? A fresh PhD does not make someone an expert, believe it or not; it is only the beginning of expertise.  Why not pay referees from the overall profits of the journal, uniformly or by some commonly agreed (with expert-editors) scheme that does not discriminate based on criterias other than excellence? The answer lies within the bigger picture.

The bigger picture

Publishing is a big business and it operates as big businesses do. Big mergers generate bigger sticks for their owners to help increase their control of the market, and of course, reduce competition. See this  report  on a recent “consolidation” step  between Macmillan Science and Education and Springer Science+Business Media. The article also reports on the five largest publishers, the number of journals they own and some estimates of their revenues: “Elsevier reported £2.1 billion  revenues in 2013 (€2.7 billion), while its parent company, Reed Elsevier, made £6.6 billion, but only £1.1 billion of that is from science journals, Aspesi estimates.” Once there is little or weak competition, the prices will go up, simply because they can. In old times the costs of publishing were real, as someone had to actually typeset the article. New technologies, including the Internet have made the real costs of publishing drop drastically and thus one would have expected a drop in publishing costs as well. Instead, as Donald Knuth nicely shows it in an open letter written 12 years ago, the per-page costs of published material has increased (doubled by 2003)!

Publishing is also based on “sound” business fundamentals: someone does the hard part of the work for free while others reap its benefits. In particular, as a scientist you invest serious amounts of time and effort to generate an intellectual property. To disseminate your results, you typeset your own paper using typesetting software following the guidelines imposed by the journal; you also generate figures and art-work to make your paper visually more appealing, again following journal guidelines; you are also often asked to write pilot texts and additional material to better advertise your own work through the journal and following journal guidelines; then you spend significant effort on reviewing similar type of work by other scientists, following journal guidelines; you also act as an editor having the responsibility to choose the referees for the paper that you are handling for the journal, again following the guidelines. And you do all of this for free!  Yep. Pro bono, Gratis, Free. If this does not sound ridiculous to you I invite you to read this writing by the one and only Scott Aaronson. Yeah, folks, as Scott puts it: it is time to realize that you’d been had.

To increase their profits, big publishers have simply increased subscription prices to the point that university libraries could no longer fork them out. They have also resorted to various sneaky tricks from bundling to legal bullying as described in a post by John Baez here. The community of scientists then formed several organizations to combat this, such as the Budapest Open Access Initiative. This is how open access journals have come into existence, such as PLoS journals and SciRep to name a few. You’d think that this resolves the problem… Nah.

So let’s see.  Imagine you are an ambitious, energetic fellow with a strong desire for eternal riches, at the helm of an open access journal. An open access journal is nothing but a queuing system: a manuscript pops in, it is processed for a time $T$, then it pops out as a publication; you collect a fee, with a net profit of $X$ per paper. Great, after $N$ papers in a given period of time, let’s say a year, you collect a $P=NX$ profit. Now what could you do to increase $P$? You can raise the fee per paper but that is already high enough so that a further increase would limit the number of submissions (would also be more expensive than the competitors). You need to grow $N$. Assuming a plentiful demand/submission, your only bottleneck is $T$. What goes into the time lag $T$? At least two things: a) initial format check and b) peer review time. In the beginning, your staff could handle a) efficiently, the turn-around time was a few days. But since the submissions have sky-rocketed, it went up to few weeks for every submission and re-submission. You’d need to hire more staff, but that would reduce your profits! A fellow editor Gyorgy Korniss and others have already noticed this increase in quality check times. Clearly, b) is the option to change. But how can you reduce the refereeing time? For one, make it easier for a paper to go through: reduce the criteria for rejections. Among these, the most important one is technical correctness, so keep that one, remove all others. It does not matter, if it is ultimately an irrelevant and technically insignificant result, as long as it is correct. I got very surprised the first time when the journal pushed back on me for rejecting papers written at a graduate student homework level. Other colleagues had the same experience. Next, in your standard letter to reviewers you could request that the referee returns his/her reports within one week. Of course, that is unreasonable and will not work.  What else? Light-bulb moment: queues are serial, we need to parallelize! Take the topic in which you received the most submissions, and outsource it to some 3rd party. But if you were to pay for that from the existing income you might be lowering your profits. Another light-bulb: outsource only those papers that the authors are willing to pay for an extra fee, yet still collect the regular $X$ (at least, it depends on the agreement with the 3rd party). Isn’t this a great scheme 🙂 ?

If you look back at history, in the long term, nothing has generated more value than scientific research, and technology. No-one can pay for it, really. It saves lives, increases life expectancy, and one day it might even take us to other planets. A little respect, please.

Update, April 20, 2015

SciRep responded today in an email to the editors who have signed the open letter. In a nutshell, the letter acknowledges the issues as “philosophical issues” potentially brought on by the fast-track peer-review service. It expresses their regret that they have not discussed their plans with us in advance. It also gives a summary of the outcome of the trial, acknowledging that “for some communities speed matters and that there is a demand for a service of this type”. The letter then concludes with a paragraph filled with reiterations of commitment towards scientists and science advancement.

There is no statement in the response letter on whether the journal will proceed with implementing the fast-track peer-review service; only that this experiment was not intended to be scalable.  However, if they are focused on increasing their profits then I think it is likely that this service will be implemented in some shape or form. Both the original announcement of the trial and the most recent response letter by the journal builds around the argument that there might be a demand for such a service. Of course there would be a demand for such service! Just as there would be a demand for publishing without a review, or at most after a quality/language check!  And there is no need to do that experiment. It has been done and it is called the arXiv. With more than a million submissions by the end of 2014 and over 8000 monthly submissions it demonstrates the baseline for need of scientific publishing (papers submitted to the arxiv are not peer reviewed, only moderated).

The response letter does not address any of the potential issues raised by a fast-track service if it were to be implemented (see here as well). One can add yet another problem: fast-track service may dis-incentivize some reviewers who work on a non fast-track paper: why would someone generate a review within 3 weeks, whereas those who work on a fast-track paper will be paid for the same work, and this reviewer will not? This might actually slow down the journal’s throughput on the regular articles, lowering their profits. While some of these problems could be addressed, the biggest one to me is that the payments would be tied to the speediness of the service, not to the quality of the work by the referees or by the authors. Such services cannot improve on the quality of published science, only dilute it.

For the incident described in this blog and also based on my recent and past experiences with this journal I decided to withdraw my support from it. I will not serve nor as an editor nor as a reviewer for SciRep, at least not until it improves its practices to better serve the interests of the scientific community.

3 responses to “The Cost of Peer Review

  1. Brilliant 🙂

  2. Great piece, thanks!

  3. Thanks Yamir and Sauro!

Leave a Reply