lazykoblog   occasional musings about knowledge organization

All of us in this business know how bruising it can be to get referee reports on our research papers. Even when the referees are gentle and helpful it still is embarrassing to discover the errors we’ve made in our texts. On the other hand, it’s a good thing they were found before publication. Of course, in double-blind peer review we are relying on the system for gate-keeping to make certain publications are original and rigorous and also that the research is replicable. That’s really the role of the referee.

Unfortunately, often referees produce really unpleasant reports. I know many of us have experienced this because we talk about it all the time at conferences and in the hallways at school. I really don’t know why referees find it necessary to be so rude, but it happens for sure. When I can, as editor, I edit the referee reports making healthy use of ellipses to leave out vitriol and just give the authors the critical commentary needed to revise the paper.

All of this is prelude, of course, for the most outrageous peer review I’ve ever received. Of course, I do not know who the referee was, although I suspect when I get done here, whoever it was, will recognize the points I’m going to make. Oh well, that’s why this is a blog ….

Clearly, whoever it was either knew who I was to begin with or sussed it out from the methodology and context. Let’s face it, in a small field like knowledge organization we all know each other well enough to recognize each other’s writing. And often we have the situation of being one of only two or three authors in an area, which leads to a lot of self-citation in order to cite the most relevant prior research. Be that as it may, whoever it was reads my editorials in Knowledge Organization–the ones that are simple bibliometric analyses. I have limited space of course, and I try to focus the editorial as commentary on the evolution of the domain. But I also deliberately state that I am just giving a few simple metrics. I always post my data files here, and encourage others to download those files and take the research further. So far as I know, nobody has ever done that.

But, please–here is what this anonymous blind reviewer wrote: “Two or three observations and we move on … or maybe there is really nothing more that can be said? … Are we supposed to do the analysis ourselves?”

Umm, yes, that would be the idea!

Okay, I took the point and extended the analysis into excruciating detail in the final version of the manuscript. But, yes, the idea of science is that we all work on the same problem sets to try to advance understanding! Please!

But, the title of this post comes from the best, which I’ve saved for last. I had quoted a paper that I believe will be seen as seminal by Joe Tennis (you know, author on subject ontogeny, knowledge organization domain analysis, ethics, etc., and current president of ISKO?). Not wanting to copy wholesale (because, that’s not what we do in science, is it?) I made reference to the article and cited one or two points. Here is how this referee indicated more detail would be useful: “but most of us … will
not necessarily have time to read Tennis.”

!!!

Well, I want to reply, “you’d better find the time!” For goodness sakes, people, this is why we have text referencing. If you haven’t read the paper being referred to, you are supposed to go do that!

Sigh.

I absolutely believe in the double-blind peer-review system for advancing scholarship. I have experienced the system from every which way possible, and I have absolute faith as a consumer of research (not a reader, but one who uses scholarship to advance my own scholarship) that peer review has led me to valid data. I am writing a paper now, for instance, in which I am relying on a paper by Maria Lopez-Huertas; I know I can count on the validity of her data to inform my own data analysis.

Still and all, the system has its quirks. I was astonished and dismayed to discover that JASIST was no longer using double-blind review. I discovered this, to my dismay, when I was sent a paper for review that turned out to be a paper by one of my own students that I had given a less than wonderful grade. Of course, I’d have recognized the paper anyway; but I was appalled to receive it unblinded, as it were, for review.

Most of the journals I read for maintain double-blind review and I appreciate it. Of course, there is often a moment when one thinks one knows who the author is, but the polite thing to do is put that thought out of your head and proceed as though you didn’t know (and who knows, you might not).

Reviewing in the knowledge organization domain also has its own domain-centric characteristics. For one thing, we are a small domain with a lot of ongoing work. Every year there are regional conferences and every other year there is an international conference, so there is an almost constant demand for 60 or so referees to be reading. I have two really terrific referees, both of whom return papers to me at once (usually overnight, but occasionally within a couple of hours). I figured out they both are simply reading them as they arrive in the email and therefore getting them out of the way. I have adopted that practice as well, and I’m much relieved not to have an inbox full of papers for review. I recommend this approach highly.

I’m always irritated when referees turn me down; I figure, we’re all in this thing together and we all have to play our parts, whether the dog is sick or not. But, it happens.

Papers for Knowledge Organization are sent to three referees. Most reply within a month, although in rare cases I have to chase after a reader. In some cases I never hear from the person. I enjoy getting diverse reports (a hates it, b loves it, and c thinks it needs work) because usually it gives me a fair amount of leeway for advising the authors. Sometimes referees get too wrapped up in grammar and punctuation. I figure that is the editor’s job–a referee should comment on the originality of the research, its appropriateness for KO, the rigor of the methodology and accuracy of the results, and applicability of the conclusions. Referees also ought to check the references, not necessarily for formatting, but for the presence or absence of material that ought to be cited. After all, this is how a domain acquires cohesion. This is the gatekeeping function that constantly checks the intension of the domain.