Most recent
Comments on the open review process|
Simon Benjamin
Operator posted 10:19 02/07/09 |
 Opening comments
This thread is intended for a general discussion of the concept of open review, that is, the process of reviewing papers in a publicly viewable forum rather than the traditional method of private review mediated via the journal editor.
|
|
Simon Benjamin
Operator posted 10:22 02/07/09 |
 from Rodney Van Meter
The following comments were made by Rodney Van Meter after his experience of completing a review on quantalk (specifically, a review in this thread: http://quantalk.org/132).
Meta-comments on the open review process: The open review process appears to have (no surprise) pluses and minuses. On the whole, I would say it resulted in, from me, a slightly better review than an anonymous one would have, for a couple of reasons: 1) You sit up straighter when you know someone is watching. My reviews are often long, and I always try to be careful, but the mental nudge that says, "Hey, others will read this review and map it to you," seems to be good. 2) I felt free to ask the opinion of a couple of other people on technical details I was unsure about. Since the paper is on the arXiv and my review will be public, there is no anonymity problem. On the other hand, there are tradeoffs: 1) Knowing that others will read the review may force opinion on the paper toward the middle; I don't want to be viewed as unnecessarily harsh, so I tone down some criticism a bit that could be more direct. On the other hand, if I think the paper is revolutionary, I might be reluctant to say so, out of fear that it was both revolutionary and wrong, which would reflect poorly on my judgment. 2) The extra time put into this review, or a simple reluctance to have one's work scrutinized so publicly, may reduce the number of reviewers willing to do the work. (Likewise, this public hazing may discourage some authors either before or after the fact.) Some people will say that the "tradeoffs" are actually wins for science: fewer, higher-quality papers and reviews, if fear of public humiliation is a driving factor. The personal costs might be non-zero, but if Science benefits in the long run, it may be acceptable. Speaking more generally, there are a couple of other pluses: 1) Someone should "have your back" as both author and reviewer; mis-review a paper, and someone likely will take exception to the review, either to the benefit or detriment of the paper, but hopefully in service of good science rather than friendship. 2) If managed properly, reviewer time and effort might be reduced by comparing to prior open reviews, rather than pointing out the same flaws. Of course, there is the direct risk that reading the first review before reading the paper can bias the second reviewer. Although this is no doubt just a first step toward "Science 2.0" (along with e.g. the "video abstracts" on Quantiki), on the whole I think I am in favor. However, I think it will be a long time before this method is the *preferred* one among a large community, let alone the *required* method. |
|
Simon Benjamin
Operator posted 13:03 02/07/09 |
re: from Rodney Van Meter
Having quoted Rodney's comments in the above post, I'll now answer a couple of them.
So, thanks for your comments Rod. As you'd guess, as one of the creators of this site I agree with the plusses... and I'd like to say a word or two about the minuses. First off, it is certainly true that people may feel uncomfortable having their work critiqued in public. I think that generally, an author should be prepared to stand by his/her work in a public forum and answer comments/criticisms -- if the author feels uncomfortable about that, what does it say about the quality of the work? From the point of view of maximizing the chance of publication, it may be in the author's interests to opt for a conventional review process because there is a 'reset switch': if I submit to journal A, and the referees hate the paper, then rather than trying to debate with them I can instead submit to journal B and (very probably) get entirely new reviewers. By opting for an open review, the authors take the risk that negative reviews will permanently colour the impressions of the paper. But in practice, I think this is will principally occur when there is some real flaw in the paper, which the authors are unwilling/unable to correct -- otherwise, they should be able to either defend their work (countering the referee) or fix the problem. Perhaps this is an idealized view, but I do believe that open peer review can improve the standards of review and ultimately of published science. |
|
Christoph Pacher
posted 18:29 09/07/09 |
Open Review is a good start
I think that open review has important pros:
* Since the whole community can read (the paper and) the reports, the reports will become more objective. * The referees directly get credit for their time consuming work (or their bad reports), and everybody can see the contribution of their work to the final paper. The system would even be better if the overall number of papers that are written would be reduced because then more readers/referees per article would have time to provide feedback. Therefore one could even argue that it is necessary to reduce the overall number of papers that are written to really benefit from open review. One con could be that * it is very hard to argue against papers from well known scientists. The referees might suffer in their future work... --- For me the next logical step would be to try to switch to community written papers: * An initial version of a paper is presented (the exact scope of the paper and what is outside is clearly (!) stated in its introduction) * There is a mechanism that allows readers to change/improve in an open manner the paper * To cite papers that are in progress versioning has to be used. * There could be some mechanism that allows to declare a paper as finished (but it should be possible to correct errors at every time). For open changes there have to be rules defining which contributions will be included and which not. E.g. the original authors can decide on the acceptance but they must provide good arguments if they do not accept a contribution. Or a wikipedia like approach is chosen. Since more and more scientists are measured by their productivity (in terms of papers, impact factors...) a measure for the importance of changes must be introduced to benefit their authors. This could be done if every author of a paper is assigned a "personal percentage of contribution" (The author of a paper solely written by him gets 100%, a paper by three authors could have individual contributions of 50%,30%,20%. I think some journals already try this.). Then authors of significant improvements (say more than 2% contribution to the paper) could simply become authors of the paper. The original authors might restrict the scope of the paper if the changes become too numerous. In fact referees could become authors. Finally this would allow for a better measure of the productivity of a scientist. The scientific community might learn a lot from the open source software community and the wikipedia community to bring this to a success. Finally I think and hope the overall number of published papers could be tremendously reduced. At the same time the quality would be much improved. |
|
Jennifer Vaux
posted 17:45 21/07/09 |
Open Review Preocess
This is a great process. http://www.wesrch.com has a very similar ideas about Science 2.0 and the generalized move towards the open review preocess amd research/paper sharing. weSRCH.com is a professional networking site for individuals in the green tech, medical, and high tech industries. It fosters innovation through publications, conferences, surveys, industry news, press releases, and member collaboration. To be able to post your paper you will first have to become a member, but becoming a member is quick, easy and FREE. Members can publish papers and research findings for others to comment on, review and discuss. As a member you establish instant priority & precedence for your ideas, thoughts, and IP – no waiting to publish for 6 months, fearing all the time that a peer with early access may scoop you. Your claim to priority is instantaneous and then peers can review it.
Keep up the good work, and stop by weSRCH sometime. |