The team at Scholastica has been talking with editors, authors, and reviewers about their experiences with peer review, and this sort of primary and interactive data gathering is ideal for product development. At the same time, we have also been doing a version of archive research (where you utilize records of past events, usually created for purposes unrelated to your study) using comments from a wide range of forums concerning peer review.

One of our chief questions is to understand how our users understand the value of the peer review system as it exists, as well as where they feel sincere frustration or need for improvement. There is a treasure trove of data in the comments of news articles, blog posts, and professional association forums, and we thought we’d share some of the gems we’ve found.

Comments on an article about comments might be too meta for some, but please do comment and let us know what you think the benefits (and frustrations) with the peer review process are.

David Hone: “I think after having written this that the major issue os that of timing, if we could fix that then the others become far less relevant (a poor review or editorial process that means you get rejected is not so bad if it happens in 3 weeks and you can resubmit fast elsewhere).”
http://archosaurmusings.wordpress.com/2009/03/09/problems-with-peer-review/

TheBrummell - “He’s also got a story about seeing a manuscript for review three times for three different journals. He showed me the last review – it included a long, but fairly polite, rant about not just shopping a flawed manuscript around until acceptance.”
http://blog.everydayscientist.com/?p=544

Chris: “My own experience with peer review has been that it is all but worthless, except in one respect (and this may actually be why it’s worthless): it causes researchers to self-edit, and it’s not uncommon for researchers to anticipate reviewers’ objections and run additional studies.

Anon: “reviewers are not anonymous to editors. If they do unprofessional work or gate-keep or self-promote it is in my experience fairly obvious and it taints the reputation of the reviewer. Editors talk - not least to each other - and reviewers can get bad reputations.”

Roland Krause: “I have no problem criticizing the work of colleague in a closed system and welcome negative comments myself – delays and poor reviews written by dimwits on their commute are more of a concern.”
http://suicyte.wordpress.com/2008/04/13/how-to-deal-with-bad-peer-review/

avalongod on “open” peer review: “The weakness I could see is that this approach could unwittingly further the status quo and discourage novel ways or thinking or critiques of popular (and dogmatic) views. For instance, lets say position X is held by 90% of scholars in a given field. Researcher Y finds evidence against position X. During an “open” peer review, scholars who hold position X could essentially “pig pile” on Researcher Y’s paper, commenting negatively en masse.“
http://chronicle.com/article/Leading-Humanities-Journal/123696/

DrVector on late and excessively negative reviews: “I am coming to the view that when people abuse the process that badly, they should get called on it, publicly, in blogs and before the relevant professional societies. In my view this would enhance rather than erode collegiality; what destroys collegiality is letting people get away with this blatantly unethical behavior repeatedly.”
http://archosaurmusings.wordpress.com/2009/03/09/problems-with-peer-review/