It took three years for Richard Price, a PhD in philosophy, to get a paper published. The slow speed of that inspired him to start what is essentially a social network called Academia, where academics can publish their papers and have them reviewed by other experts called editors.
Now, Price wants to take the next step to surface the best papers with a score. It’s called PaperRank, and it’s a way to help academics quickly determine the quality and validity of a paper. Experts can already recommend and make comments on papers as a sort of live peer review process, but now those recommendations fit into an algorithm that helps rank the paper.
“In the journal model, the editor of the journal is a paid employee of the journal. They go and email a couple people and say, can you peer review this?” Price said. “And then they do it for free. It’s just a sniff test. It’s reading it and saying, yeah, I recommend it. What we thought was, what does peer review look like when you have a network, and that’s what we tried to build.”
The number of recommendations a paper has and the scores of the authors recommending the paper determine the paper’s rank. It’s a shot at basically distributing the credentialing process across an entire network, rather than relying on editors emailing various expects to peer review the paper before it ends up in a journal. It’s not entirely dissimilar to Google’s PageRank in terms of the mathematics, Price said, though there are some more nuanced differences.
The credentialing process is slow and generally, to get a paper widely viewed and accepted, it has to land in a major publication like Science or Nature. Editors receive papers and then have to seek out experts in their networks to get some of them to review the paper, to essentially tell whether it’s a valid scientific study or not.
“The basic thought was, right now the system collects sentiment from two people, and we thought why stop at two? Why have this cap?” Price said. “What I’m trying to do is take the flat distribution right now of two peer reviews per paper and augment it and collect more sentiment. And the way we’re getting it is by asking people who read the paper just what they thought.”
There’s always the chance that the credentialing process might turn out to not generally be accepted by the scientific community. But Academia does have 27 million registered users that upload 20,000 papers each day, and editors recommend around 3.5 papers per month. Price said that Academia editors are basically putting their reputations behind their recommendations.
There’s another benefit to improving the credentialing process and more quickly determining the best papers — it can improve the speed at which additional papers that stem from that first study emerge. That improves the speed of scientific innovation, if it works, Price said. And, indeed, there’s definitely an opportunity here to do just that, with papers emerging all the time that stem from research.
Price also said the ranking system helps solve what he calls the “reproducibility crisis,” which is an instance where the results of the experiments in a paper can’t be replicated. This happens in a large number of papers, he said, and in theory the higher the PaperRank, the higher the confidence in the experiments.
Authors will also get a ranking, which then feeds back into how strong their recommendations are. The AuthorRank, which is what it’s called, is a factor of the scores of their papers. If it’s becoming a bit obvious that there’s a lot of self-referring happening here, that’s by design: it’s essentially a giant recursive function.
So why would additional editors peer review? For starters, to get things rolling, Academia has set up about 2,000 academics to seed reviews and build up scores initially for papers. And in general, editors want to get their names out there — with the various audience sizes see the names of people who have recommended papers, Price said.
To be sure, Academia is not the only player in this space. There are other networks like Mendeley and ResearchGate, and the service has to try to get ahead of incumbent publishers like Springer and Wiley, Price said. But the startup has twice the traffic of ResearchGate and is much larger, and PaperRank is the first peer review service implemented on a network, Price said.
Academia is based in San Francisco and has raised $18 million from firms like Spark Capital, Khosla Ventures and True Ventures.