lazykoblog   occasional musings about knowledge organization

Refereeing is about community. We use peer-review as a gate-keeping device. But too often these days that seems to mean slamming the gate shut instead of standing at the gate with a welcome kit.

Too many referees for journal articles, conference papers, etc., are playing AMERICAN IDOL with the role. You know what, your job is not to refuse everything egotistically. Your job even is not to refuse things that you think are flawed.

Your job is to tell people who have legitimate contributions how to make them fit into the domain. Because ultimately our collective job is to secure and promote the domain. If you refuse every abstract or manuscript you see you will shut the door so firmly the domain will die.

So the next time you are tempted to say “NO NO NEVER” to a proposal, stop to think how you would feel if you were the author. What will you have learned from that response? Never to come back, likely.

Now, I have had recently an instance of reviewing the same text for a journal and a couple of conferences. That’s a no-no. Don’t submit the same thing in multiple venues. And if you can’t think of a second way of phrasing your point, it probably isn’t worth making it twice.

But aside from that, the rest of us need to get focused on how to build our domain. Saying “that is not KO” is not helpful. Saying “this is interesting but it needs to cite X and Y ” is helpful.

I absolutely believe in the double-blind peer-review system for advancing scholarship. I have experienced the system from every which way possible, and I have absolute faith as a consumer of research (not a reader, but one who uses scholarship to advance my own scholarship) that peer review has led me to valid data. I am writing a paper now, for instance, in which I am relying on a paper by Maria Lopez-Huertas; I know I can count on the validity of her data to inform my own data analysis.

Still and all, the system has its quirks. I was astonished and dismayed to discover that JASIST was no longer using double-blind review. I discovered this, to my dismay, when I was sent a paper for review that turned out to be a paper by one of my own students that I had given a less than wonderful grade. Of course, I’d have recognized the paper anyway; but I was appalled to receive it unblinded, as it were, for review.

Most of the journals I read for maintain double-blind review and I appreciate it. Of course, there is often a moment when one thinks one knows who the author is, but the polite thing to do is put that thought out of your head and proceed as though you didn’t know (and who knows, you might not).

Reviewing in the knowledge organization domain also has its own domain-centric characteristics. For one thing, we are a small domain with a lot of ongoing work. Every year there are regional conferences and every other year there is an international conference, so there is an almost constant demand for 60 or so referees to be reading. I have two really terrific referees, both of whom return papers to me at once (usually overnight, but occasionally within a couple of hours). I figured out they both are simply reading them as they arrive in the email and therefore getting them out of the way. I have adopted that practice as well, and I’m much relieved not to have an inbox full of papers for review. I recommend this approach highly.

I’m always irritated when referees turn me down; I figure, we’re all in this thing together and we all have to play our parts, whether the dog is sick or not. But, it happens.

Papers for Knowledge Organization are sent to three referees. Most reply within a month, although in rare cases I have to chase after a reader. In some cases I never hear from the person. I enjoy getting diverse reports (a hates it, b loves it, and c thinks it needs work) because usually it gives me a fair amount of leeway for advising the authors. Sometimes referees get too wrapped up in grammar and punctuation. I figure that is the editor’s job–a referee should comment on the originality of the research, its appropriateness for KO, the rigor of the methodology and accuracy of the results, and applicability of the conclusions. Referees also ought to check the references, not necessarily for formatting, but for the presence or absence of material that ought to be cited. After all, this is how a domain acquires cohesion. This is the gatekeeping function that constantly checks the intension of the domain.