It would seem that more and more people are putting their papers up in bioRxiv. My lab is, in all honesty, doing a lot of these papers in journal club. But I have some major problems with bioRxiv and major concerns that they have not addressed. While I fully admit that it is possible that these problems may not be problems in most practice, following the tenet of this blog (“Don’t tempt Ol’ doc Murphy, he’ll come kick our a**.” Oh wait, the other one… “If anything can go wrong it will.”), I am worried that these problems just haven’t shown up yet.
Once up, the manuscript is permanent
My largest problem is the same issue that I complained about making peer review public (q.v. olddocmurphy/peer-review-is-not-commentary) – the history of the paper process shouldn’t matter. As pointed out elegantly by Moti Ben-Ari (in his wonderful book Just a Theory), the largest thing that separates science from pseudoscience is that history shouldn’t matter for science. While it is interesting that Newton fought with Leibniz about the priority of calculus, it is irrelevant as to the validity of calculus. What matters is not the history of how the paper was made. What matters is how the science was done. However, because bioRxiv believes that the history of the paper is important, they do not actually replace papers with new versions.
I am concerned about the presence of past versions remaining on the web. While one can “withdraw” or “replace” one’s paper, the old version is not removed from bioRxiv, instead a notification is placed at the top that the paper has been withdrawn or that a newer version is now available.
I am concerned because, in my experience, papers change during the peer review process. [That’s the point of good peer review!] (For example, what if the peer reviewer finds a sign error in our paper? Let’s assume we’ve done the experiment carefully so either result is interesting and important and publishable, so the sign error corrects the result but doesn’t change its usefulness to the scientific literature.) I don’t want both versions out there. (Yes, partially because I’m embarrassed at making mistakes. I admit it. There’s a little vanity there.) But in larger part because I don’t want people accidentally finding the wrong version. What if someone doesn’t notice the “replaced” notification? What about people with incomplete English skill sets? More and more text-scrapers are appearing trying to build out of the scientific literature – do we really think those scrapers are going to correctly see the “replaced” notification? It seems crazy to me not to remove the old version.
This is clearly a general problem with the availability of preprints and the web. Because nothing is ever lost from the web, the preprints come up first. In part, because Google’s search algorithm is driven by links and references to a webpage, the preprint (which has been there longer and gathered more connections) is the first thing to pull up. My colleagues have convinced me to use PaperPile for references on collaboration papers that are being written in Google Docs. When I pull up a citation, the citation pulls up a bioRxiv preprint as often as it pulls up the final journal article. I see a similar thing when I use Google Scholar to get the citation. Google doesn’t know the difference between bioRxiv and peer-review. (Should we expect it to?)
bioRxiv is not peer reviewed
The second problem with bioRxiv is that it is not peer reviewed. In my experience, most journalists (#notalljournalists) do not understand peer review. You can tell from the articles on the recent Elon Musk paper that was put up in bioRxiv – while many articles did note that the work “had not been peer reviewed”, it didn’t stop them from writing lots of positive text about it.
As I noted before, peer review is an important check on problematic science. (It’s not perfect, but it’s better than nothing, which is what bioRxiv has.) I’ve often likened bioRxiv to SFN abstracts, but I recently found that even SFN does peer review to ensure that abstracts contain complete scientific information, such as the actual chemical name of pharmacological products. To my knowledge, BigPharma has not tried to use bioRxiv (or the new medRxiv) to side-step the peer review publication process, but I’m sure they’ll pick up on that soon. I’m waiting for the creationists to figure out this trick and the disaster it will be trying to correct the literature once they are in it.
So what to do?
Finally, there’s a big question of how should we, as readers, treat bioRxiv? Are these finished papers for us to read and treat as part of the scientific literature? We have done many bioRxiv papers in our lab journal club, discussing them as if they were papers in any other journal. (Of course, my lab’s journal club is pretty sophisticated and able to look at the methods section to determine whether we believe a result or not. I would not expect most readers to be so sophisticated. [q.v., journalists, above]. I would also point out that we do the same scrutiny on any paper in any journal that we discuss in our journal club. [One of my favorite homeworks to give to a graduate class is to ask them to be “peer reviewers” for certain published papers in famous journals that I know to have fundamental errors and to see if they catch them.])
This leads to the inevitable problem of what should we do with bioRxiv? It seems like an archive of in-preparation papers would be a useful tool, but bioRxiv doesn’t seem to know whether it is a conference proceedings (like SFN abstracts) or a journal (are these complete parts of the scientific literature?). This is a real problem on NIH BioSketches and CVs for job applications and awards. In my experience, how a bioRxiv paper is seen currently varies from reviewer to reviewer. Some reviewers treat them as journal articles and count them equivalent to peer reviewed papers. Other reviewers treat them as SFN abstracts – proof that one is working on something, but not as an actual paper. Still others cross them off the list and ignore them. (This is a real problem for applicants – putting your bioRxiv paper on your CV could help you, yes, but it could also hurt you if the reviewer thinks you’re trying to get away with padding your CV with publications that aren’t real publications.)
I’m torn. There are huge advantages to having a means of putting our papers up publicly so that we do not have to wait for someone to wade through the journal fight (q.v. oldocmurphy/papers-should-move-up-not-down). But right now, bioRxiv has some major flaws that make me unwilling (yet) to put my work up there.