Referees Who Move Goalposts Make Lousy Peers

My latest paper, The Arecibo Galaxy Environment Survey VII : A Dense Filament With Extremely Long HI Streams, has finally been accepted for publication. This has consumed, in one way or another, about the last year or so of my research time. So I'm going to indulge myself with not one, not two, but three entire blog posts. In this first one I'll look at the peer review process, which for this particular paper was slightly less fun than being given an enema by a bear. In the next post I'll describe how we went about searching for galaxies using Blender, and in the third, thrilling conclusion, I'll say something about what it is we actually learned.

I imagine that giving an angry bear an enema would also be unpleasant.

Peer review does not mean writing articles in the Guardian about what you think of Jeffrey Archer, unless you're a journalist. In science, the idea behind peer review is simple : someone writes a paper, and someone else (usually anonymous and selected by a respected academic journal) checks to make sure it makes sense. In this way, scientific literature is pretty well-defended against giant space bananas and moon landing conspiracy theorists. No-one think's it's perfect - a perfectly objective system is probably impossible - but it's the only system we've got for checking that everyone else is basically sane.

In this case it took over seventh months from first pushing the big scary, "WARNING : THIS WILL SUBMIT THE ARTICLE ! ARE YOU ABSOLUTELY, POSITIVELY, DAMN WELL BLEEDIN' SURE YOU REALLY REALLY WANT TO DO THIS ?!?!?!" button to having someone say, "This looks decent, let's publish it." Almost as long as creating a baby, though less icky, and as you can imagine I'm none too happy about that.

Although now I come to think about it, if it's a choice between one or the other, I'll take the publication, thanks.


I've never had to referee a paper, mercifully. I don't imagine it would be much fun. Several weeks trying to make sense of other people's wranglings with no time for your own research can't be much of a hoot. But I have received several referee's reports on my own work. Typically, this results in my mental processes going something like this :

First reading
"Add error bars in figure 6 ? What a JERK ! No-one else puts error bars on this plot. This guy is clearly a complete amateur."
Two hours later
"Huh, errors aren't so hard to calculate after all, maybe I should add them in instead of insulting the referee's mother at the next submission..."
The next day
"Hey ! With error bars this figure looks waaay better !"

Which is precisely why this post has been in draft form for several months, but I'm not going to even try claiming an unbiased viewpoint.

Referee's comments come in all varieties, from the extremely helpful (which are probably the majority, if I'm honest) to the extremely unhelpful (more on them later). A few are just downright strange. One I've seen every single time is, "shorten the text." This is like saying, "There are too many notes, that's all. Just cut a few and it'll be perfect.". It doesn't make any sense. Tell me which bits you think are too long, or I'm just not going to do anything*.

* Actually what I'm going to do is cut out one paragraph just so I can say I've done something. What I'd like to do is remove the final word from every sentence.

What's almost worse and just as common is that the referee will simultaneously state that the paper is too long and require major additions. Referees, please don't do this. It makes you sound like a crazy person. You have to at least suggest where the cuts should be made, or it makes no sense. I can't possibly make it any shorter if you're telling me to add stuff.

In like vein, the weirdest, most incomprehensible suggestion I've received so far was that I should add "supporting figures to the paper" (there wasn't any context to this). What, you mean like a picture of a giraffe or something ? A self-portrait in crayon ? Maybe a drawing of Atlas, he's a supporting figure, after all. Love to help, but I'm not psychic ! Of course, the assume-everyone-is-psychic factor affects us all, but not usually to that degree.

Fortunately, this particular mystery was solved in consultation with a friend, who brilliantly realised :
"It means there aren't enough characters. You need to flesh out your story with supporting figures that the referees can relate with. Maybe add a downtrodden scientific paper reviewer, who started out with the bright-eyed idea of changing the world through peer reviewed science, but instead met paper after paper of nonsense and flim-flam, and now has no joy left when a paper of some real worth comes through."

Much more annoying are criticisms of the second draft that could have been made at the first submission. Now obviously there's nothing wrong with saying, :
"Oh and I forgot, you need to correct the spelling of GALAXIE on page 2 and make the caption bigger."
No-one can spot every typo at the first reading. But if you're going to suggest :
"You should replot all your figures so that they actually show something completely different, because although I didn't bother to mention it last time, I think they're all wrong. And also I don't think you wrote your code in C, so please do that."
... then that's fine too - it's the referee's job to do this - but don't suggest this at the second draft when you could do so at the first. This is unfair and unprofessional - it's moving the goalposts, which is traditionally frowned upon. Especially if it's done by a referee. Also, it really wastes a huge amount of time - I could have made the changes months ago and have the paper ready for publication already.

Perhaps the most annoying of all is to repeat verbatim criticisms of the first draft that the author quite clearly, unmistakably and - above all - directly addressed in the second draft. As in :
Referee : "You need to say how many galaxies are in your sample."
Author : "There are 5 galaxies in this sample."
Referee  : "You need to say how many galaxies are in your sample."
That's not a real example, but some comments really are every bit as inane as that. They're even worse when they're longer comments that took correspondingly longer to address. I can only assume in these cases that the referee didn't actually read the author's response - which makes it a no-win situation. If you're not going to even read my response, why should I bother making corrections at all ?

The worst example of this was a referee who insisted that there were better data processing techniques we could use. Great ! It was obvious to anyone that the method we were using, while an accepted and well-tested approach, might not be ideal. So in my response I asked the referee to provide a reference so that I could try this new approach for myself. Unfortunately in the second report the reviewer didn't answer my question at all, but instead did a copy+paste job of their previous comment.

Sigh.

But we all have bad days. In my response to the second report, I again asked (twice this time, just to make sure) for a reference to these shiny new methods. In the third report, the referee flat-out ignored this altogether, merely stating simply that our measurements must be wrong - and worse, that not only were the numbers wrong, but all of the major results were also wrong. Even though they hadn't raised any previous objections, and simultaneously stated that the paper was now "much improved".

In this case this rather brusque dismissal, at the third iteration of the paper, was particularly galling since I'd spent a very, very long time making sure those measurements were as good as I could damn well make them. The end result was quite nice, agreed with previous measurements (where available) and wasn't particularly controversial. All lines of inquiry pointed to the same result. To have the referee then suddenly dismiss them - after not raising any previous objections - without giving a clear reason why was simply too much, so we asked for a second referee.

Now, as an aside on the much-vaunted objectivity of science, it may interest readers that referee 2.0 had no qualms whatever with the data analysis, which was vindicating. Whereas referee 1.0 was perfectly happy with the introduction, referee 2.0 thought parts of it were incomplete and misleading, and provided a list of some genuinely interesting papers in support of this. Which is great, and I happily made the requested changes (although I'm not sure I agree with the referee's sentiments, a happy compromise was easily reached)... but, it does, of course, expose a basic flawed truth of peer review : not all scientists agree with each other. But referee 2.0 provided justification for their assertions, whereas referee 1.0 didn't.


Make no mistake : I support the peer-review process. And philosophically, I don't like the notion that if you disagree with the reviewer, you should find another reviewer - but let's not go nuts. Sometimes, just like everyone else, reviewers are just jerks. Peer review may not be the perfect, objective solution we'd all like it to be - but it's a damn sight more objective than not doing it at all.

In short, reviewers :

• Specify what it is you want the authors to do. Authors aren't psychic. They'd like to be, but they're not. This means vague instructions really aren't helpful. Asking the author to "do this bit again, but better" (that's a real example, albeit paraphrased) won't work !

• Don't make non-constructive criticisms. Ask questions rather than stating that the authors have missed something obvious - it at least gives the impression of benefit of the doubt. There's no point whinging that the paper has lots of typos - send them a list instead*. As above, vague, hand-waving instructions help no-one.

• Don't move the goalposts. Spend longer on the first draft so that the authors will be able to address as much as possible at stage one. Telling them that they need minor revisions at the first draft, then saying major revisions at the second draft (after the previous comments have been addressed) is misleading, unprofessional and delays publication. And if you do find a major error at the second draft, have the common courtesy to apologise for not spotting it sooner !

• Demand rigour, but don't be anal about it. Life's too short to worry about half-spaces, or, more importantly, the precise meaning of a word. Insisting that the author uses a long-winded phrase (e.g. "galaxies in the overdensity between redshift 0.02 and 0.03") just in case a commonly-used word (e.g. "population" - seriously, referee 1.0, WTF have you got against the word "population" ?) is misunderstood helps no-one.

• Check that the author hasn't already addressed your point in the paper or their response. There's really no reason for the author to bother if you're not going to read what they write - it makes the whole process meaningless and unworkable.

• Remember that not everyone is a native English speaker. Many of my non-Anglo/American colleagues report that referees can be very unhelpful when it should be obvious enough that the author might not speak English as their first language !** If you suspect this is the case, then criticising their writing flaws is simply rude. Don't do it. Instead, give them a list of corrections.

• If you can't justify your assertions, don't make them. The same applies to the authors, of course. Unless you can point out why something is wrong - and, if at all possible, how to correct it - don't say what you think is wrong. If the authors have justified what they've done, you may disagree with them - but unless you can prove that they're wrong, that should be the end of it. You've got to allow people to publish things you disagree with.

* And no, they can't be "easily avoided." If you've read your own work a dozen times it becomes literally impossible to spot any more typos. Fresh eyes are needed, and it makes a lot of sense to me that those eyes should be provided by the referee/editor. To say, "please check more carefully in future" is condescending.
**Helpful hint : if the author's name is Pierre von Hindenberg, chances are they're not a native English speaker - and with a name like that, they're probably way more awesome than you will ever be.