Peer review (originally posted 3-14-2009)   no comments

Posted at 6:23 pm in journals

I absolutely believe in the double-blind peer-review system for advancing scholarship. I have experienced the system from every which way possible, and I have absolute faith as a consumer of research (not a reader, but one who uses scholarship to advance my own scholarship) that peer review has led me to valid data. I am writing a paper now, for instance, in which I am relying on a paper by Maria Lopez-Huertas; I know I can count on the validity of her data to inform my own data analysis.

Still and all, the system has its quirks. I was astonished and dismayed to discover that JASIST was no longer using double-blind review. I discovered this, to my dismay, when I was sent a paper for review that turned out to be a paper by one of my own students that I had given a less than wonderful grade. Of course, I’d have recognized the paper anyway; but I was appalled to receive it unblinded, as it were, for review.

Most of the journals I read for maintain double-blind review and I appreciate it. Of course, there is often a moment when one thinks one knows who the author is, but the polite thing to do is put that thought out of your head and proceed as though you didn’t know (and who knows, you might not).

Reviewing in the knowledge organization domain also has its own domain-centric characteristics. For one thing, we are a small domain with a lot of ongoing work. Every year there are regional conferences and every other year there is an international conference, so there is an almost constant demand for 60 or so referees to be reading. I have two really terrific referees, both of whom return papers to me at once (usually overnight, but occasionally within a couple of hours). I figured out they both are simply reading them as they arrive in the email and therefore getting them out of the way. I have adopted that practice as well, and I’m much relieved not to have an inbox full of papers for review. I recommend this approach highly.

I’m always irritated when referees turn me down; I figure, we’re all in this thing together and we all have to play our parts, whether the dog is sick or not. But, it happens.

Papers for Knowledge Organization are sent to three referees. Most reply within a month, although in rare cases I have to chase after a reader. In some cases I never hear from the person. I enjoy getting diverse reports (a hates it, b loves it, and c thinks it needs work) because usually it gives me a fair amount of leeway for advising the authors. Sometimes referees get too wrapped up in grammar and punctuation. I figure that is the editor’s job–a referee should comment on the originality of the research, its appropriateness for KO, the rigor of the methodology and accuracy of the results, and applicability of the conclusions. Referees also ought to check the references, not necessarily for formatting, but for the presence or absence of material that ought to be cited. After all, this is how a domain acquires cohesion. This is the gatekeeping function that constantly checks the intension of the domain.

Written by lazykoblog on November 17th, 2010

Tagged with , ,