The first problem with a double-blind system is that for many geology and planetary applications, the anonymity of the authors is laughably fragile. Suppose, for example, that I were to write a paper using SHRIMP or laser ICPMS results (those are the instruments that consume 90% of my time). Assuming my methods section adequately describes these unique systems, then the anonymous reviewers will be able to determine which research group I am a part of. Combining that with the subject matter and/or field area will give them a pretty good idea of who the authors are likely to be.
For planetary science, the situation is even worse. Suppose a reviewer gets a paper on recent UV spectrometry results from Jupiter. The only UV spectrometer within 500 million km of Jupiter is on the New Horizons spacecraft, and the science team for that mission is posted on the internet. Double blind quickly becomes single blind if the reviewers decide they want to know whose paper they have. At least with a fully attributable system, everybody has the same level of transparency.
Finally, the most common complaint about “bad” reviews is not the amount of criticism. Rather, it is generally the lack of science on which that criticism is based. Most people get over fair but harsh reviews; rather it is the inane, mean-spirited (and always anonymous) reviews that are really frustrating. If reviews were signed, and included in supplementary material archives, then the cost to one’s reputation for writing such reviews would be substantially higher than it would be under a partially or totally blind system.
Of course, it’s easy for pudknockers like
After all, the powers-that-be have spent considerable time and effort schmoozing in smoke-filled rooms in order to make the connections and handshakes necessary to game the review process. It is only natural for them to want to preserve the advantage bought with Cuban tobacco and intellectual kickbacks. But there is a change that would open the process without making life difficult for the kingpins of the scientific establishment.
Department heads could require tenure applicants to include a collection of signed reviews as part of their tenure portfolio. This would illuminate the applicant’s standing in his or her field of research, and more importantly, it would cost the Haupt-professors nothing- the work and risk would be borne by the associate professors and lecturers. And it would actually make life easier for editors, as many fields are currently suffering from a shortage of willing reviewers. Of course, the follow-on effect would be to decrease the proportion of anonymous reviews, in a way that benefits science as a whole. So everybody wins.
Yeah, I'd go for that.
ReplyDelete(And can I just say how much I hate the Blogger/Google login alliance? I have to log out of my calendar and email every time I want to preserve my pseudonymity on someone's blog. Grah. Agh.)
That's a good idea - although as you say, it won't put much of a crimp in the stride of the gate-keepers.
ReplyDeleteIt's also a good idea to make the reviews (and the authors' response) available somehow - it might make people pause a bit before sharpening their hatchets, although their aim is of course to stop publication anyway...
Yami, I hear your pain. I set a second email up for blog stuff, but it turns out I need to use my old one to change any settings or do anything other than post anyway.
ReplyDeleteCJ, gatekeepers are fine as long as they judge science, and not connectivity or factional alliance. But what we little people can do to change their behavoiur is tricky.
I like the system adopted by several EGU journals - they publish the original manuscript and the reviews, signed or not. Anyone else outside the nominated reviewers can comment too but I think they do not get the option of anonymity (although if they really wanted to they could presumably make up a sock puppet). This review process stays up on the web whether or not the paper is eventually accepted.
ReplyDeleteThis doesn't necessarily stop the most excessive behaviour - witness ref 2 on this hockey stick paper - but at least in that case the editor had to publicly distance himself from the reviewer. And everyone can see the exchanges and make an educated guess as to the person in question...
If there was a suitable EGU journal for my multiply-rejected paper, I'm confident that it would have been published, and moreover the review discussion might have been useful to others.
As for making junior staff sign their reviews - I don't think so.
Lemming,
ReplyDeleteA great idea - I'm all for scientists being open and putting their names and reputations behind their work.
You must be acutely aware, then, that your own credibility is only indirectly on the line in making the suggestion since you're doing it under a pseudonym. While it certainly doesn't devalue your argument, it's not exactly leading by example.
For a bit of cross-pollination, here's are some pretty thorough discussions from another field of the review system and its faults.
ReplyDeleteAs for being powerless to change the system, well... perhaps it starts with becoming a conference chair and adjusting the system for that conference, and then trying to make your changes catch on...
Ron,
ReplyDeleteMy full name is in the disclaimer on the right- anyone who actually comes to the site instead of reading feeds can see it. I got outed 9 months ago- in front of my lab manager by the former director, no less.
I use the lemming handle for continuity purposes.
Double blind reviewing is totally untenable. If you are doing field work in a unique location and/or you've given a talk at a conference, anyone can find you on a google.
ReplyDeleteI agree with the non-blind reviewing proposal from the author perspective, but what about from the perspective of the young reviewer. What if I, as a lowly postdoc or assistant professor, write a critical review of a paper for a big shot and I sign the review. How likely is the big shot going to look upon me kindly when we meet at a conference or I apply for a job or he gets my paper to review?
Is there a good story to the outing?
ReplyDeleteSW:
ReplyDeleteIf you are fair to prof. Big Shot, hopefully your willingness to point out his faults will impress everyone else more than they upset prof. B.S.
James, it was a non-event. After a week, everyone who was in the lab at the time decided that they had better things to do than reading this blog.
D'oh! I guess you don't want me reviewing your papers - I obviously didn't do a very thorough job of reading your site. (Though, in my defense, I usually read it in my feed reader, where the disclaimer doesn't show up.) Nonetheless, mea culpa.
ReplyDelete