Empirical analysis tells Reviewer 2: “Go F’ Yourself”
Peer review is often the key hurdle between obtaining some data and getting it published in the scientific literature. As such, it’s often essential to keeping questionable results out of the scientific literature. But for vast numbers of scientists with solid-but-unexciting results, it can be a hurdle that raises frustrations to thermonuclear levels. So it’s no surprise that many scientists privately wish that certain reviewers would end up engaged in activities that aren’t mentionable in a largely family-friendly publication like Ars.
What was a surprise was to see a peer-reviewed publication make this wish public. Very public. As in entitling the paper “Dear Reviewer 2: Go F’ Yourself” levels of public.
Naturally, we read the paper and got in touch with its author, Iowa State’s David Peterson, to find out the details of the study. The key detail is that the title’s somewhat misleading: it’s actually the person who’s somewhat randomly assigned to the Reviewer 3 slot who’s the heartless bastard that keeps trying to torpedo the careers of other academics. For the rest, well, read on.
We have to ask: why?
Peterson laid out his case for looking at one particular reviewer in his paper, in the section helpfully entitled “Why Reviewer 2?”
The main motivation for this article is that the broader community has decided that Reviewer 2 is a monster. A Google search for “Reviewer 2” produces the interdisciplinary Facebook group “Reviewer 2 Must Be Stopped!” (which has over 9,000 members), a blog entry entitled “How Not to Be Reviewer #2,” and countless images combining almost every visual meme imaginable. In academia, it is fair to say that Reviewer 2 is the ultimate boogeyman. He is Pennywise the Clown, combined with el chupacabra, wrapped in the Blair Witch.
Put another way, Peterson wrote “Reviewer 2 is dismissive of other people’s work, lazy, belligerent, and smug.”
But that doesn’t get at the larger issue: why look at this issue at all? Peterson said that it’s more or less because he had the data anyway. He was the editor of the journal Political Behavior for four years, and Peterson had been analyzing the results of its peer review as part of a process looking for any systemic biases in outcomes based on things like the race or gender of people who tried to publish there. “So I had all the data, right? I had sort of collected it all for this other project,” he told Ars. “And then it dawned on me—honestly, after a beer or two—that I could try to test this. It’s pretty straightforward, you know? It’s a really, really straightforward statistical test.”
Make that two statistical tests. In the first, Peterson checked whether there was any systematic difference in the ratings of papers based on reviewer number. That turned up absolutely nothing. But Peterson wasn’t done. “There’s this sort of second possibility—that when academics… when they get mad at reviews, really, it’s the negative outlier that we hate, right? And so maybe I could try to capture that idea that Reviewer 2 is the reviewer number likely to matter for being one category on average lower than the mean of the other reviewers.”
He did the statistics to check if any reviewer frequently scored papers quite differently from his other peers. “I developed an original measure of ‘being Reviewer 2,'” Peterson wrote, before going on to say “the real problem of Reviewer 2 is that he is an outlier and that can only be seen when the manuscript is strong enough to get positive evaluations from the other reviewers. This is when Reviewer 2 crushes your hopes.”
We could have told you that
Amazingly, this turned up something. When asked whether this surprised him, Peterson’s response was “Oh, God, yeah.” But the surprise didn’t end at the fact that there was any result at all; it extended to the fact that the outlier wasn’t Reviewer 2.
It was Reviewer 3.
Those of you who are biologists will be nodding sagely as (confirmed via Dr. Beth Mole) that field has always blamed Reviewer 3. In fact, there’s an entire Downfall meme about Reviewer 3.
We asked Peterson about this, and he speculated that biologists might just be a bit more sharp when it comes to picking out the nefarious reviewer. “I think biologists had it right,”” he told Ars. “I think biologists might be a little better at this than [political scientists] are. Honestly, that’s amusing to me. And I’m not sure why different disciplines would choose different numbers to make the devil.”
He suggests it might have something to do with how reviewers are chosen. Reviewers for Political Behavior end up in the dreaded 2 slot largely by self selection. Knowing many potential reviewers would say no for various reasons, Peterson said he would send requests out to more people than he needed. Anyone who said yes would simply get assigned a reviewer number based on the order in which they replied. Other journals might handle that differently.
What stood out to Peterson was the fact that, at least among political scientists, Reviewer 3 is the problem, yet the community has managed to shift the blame to someone else. “Not only is Reviewer 3 the bad actor, but Reviewer 3’s crafty enough that they get Reviewer 2 blamed,” he told Ars. “Which kind of tickled me to no end, frankly.”
How do you get this published?
In the paper, Peterson skips the normal academic language to evaluate this: “This seems like it is the ultimate jerk move.” Language like that, the references to el chupacabra, and the title itself are all pretty unusual in the academic literature. But Peterson got it published without abusing the fact that he was an editor. Part of this is due to the fact that, at its heart, this is a quantitative analysis of human behavior, the sort of study that’s handled by a lot of journals.
Still, that didn’t make publishing it easy. “This was not the first journal I submitted it to,” he admitted. Part of the problem, it seemed, was that some of his reviewers had somehow managed to remain oblivious to the whole concept of a reviewer from hell. “I kept getting reviewers who had never heard of a Reviewer 2 idea,” he said. “So the basic idea that there is this jerk out there was totally foreign to them, and so they didn’t understand why anyone would ever think this was an interesting question. Which amazed me. But yeah.”
Eventually, he had a chat with the people who would serve as editors in the journal where it was published. “I’ve known the editors of Social Science Quarterly for a long time and had a conversation with them before I submitted it, to make sure that they were going to recognize it for what it was,” Peterson said.
Even so, it wasn’t necessarily easy for them to translate that into getting the paper accepted. “I believe that the editors were careful in their selection of reviewers,” Peterson acknowledged.
The other hitch he had with editing is the title, which combines an obscenity with blaming the wrong reviewer—the latter of which almost got changed to Reviewer 3 by a copyeditor. “When I sent it to other journals. The title was ‘Is Reviewer 2 really Reviewer 2?’ And that’s probably a better title, but I kinda like this one more myself.”
Social Science Quarterly, 2020. DOI: 10.1111/ssqu.12824 (About DOIs).
https://arstechnica.com/?p=1687779