Monday, May 18, 2009

Crowdsourcing Peer Review in the Age of Blogging

That the internet is causing paradigm shifts in a number of areas, not the least of which is news publishing is beyond dispute. Much less has been said about its effect, if any, on the hallowed and time-worn process of academic peer review.

One of the cornerstones of modern scholarship, whether it be in the sciences, humanities or the arts -- or anywhere else -- is the peer-review process. Peer review is a filtering step in the process of scholarly research generation and dissemination. It is intended to ensure that the work that is finally published meets certain minimum standards and thereby does not take up valuable journal space, waste the reader's time and until and unless refuted appears to contribute to a corpus of research in some specific field. It is also intended to ensure that a researcher gets a fair hearing and is not prevented from disseminating research because of some gatekeeper's idiosyncratic beliefs. Typically, a scholar sends in a manuscript for review to a journal's editor who in turn sends them on to other scholars who are familiar with the specific area of scholarship addressed in the manuscript. The reviewers read the article, write up their reviews, including comments, suggestions and recommendations and send them on to the editor, who in turn passes them on to the manuscript's author. The author then chooses to address the comments and incorporate suggestions and sends them back to the editor. This process may involve several rounds before a submission is accepted or rejected for publication. Of note is the fact that both author and reviewers are anonymous with respect to each other, in the interest of ensuring some degree of neutrality.

Peer review has many critics -- and valid criticisms -- but it has worked reasonably well, and there haven't been any alternatives that can be deployed more conveniently, and so it lives on. One major problem with peer review is the large amount of time that elapses between the time a manuscript is submitted for review and is accepted for publication. Even more time elapses before it actually appears in print -- especially in the most prestigious journals -- given the large number of article submissions and the limited size of each issue. The situation was much worse before the widespread availability of email, but there yet remains much dissatisfaction with the process among researchers; like a bad marriage, it's a nuisance they have felt compelled to live with for all the benefits it offers.

One step in the article publication process is beginning to be sped up by the appearance of electronic or online journals. Online journals have at least three advantages over their 'dead tree' counterparts made from wood pulp: there is no practical limit to the size of each 'issue'; the step involved in typesetting an issue is eliminated; and most importantly, there is no need to wait for each issue 'cycle' -- an article may be 'published' on the internet the day it is accepted and proofed.

None of this has reduced the time taken up by the traditional peer review process or in any way eliminated or reduced its shortcomings.

Now comes an experiment to test the efficacies (and limitations) of having a scholarly book 'peer-reviewed' by anonymous blog comments. Noah Wardrip-Fruin, an assistant professor of communication at the University of California at San Diego, decided to test the efficacy of having a scholarly work 'peer-reviewed' through anonymous blog comments. Wardrip-Fruin was either a fool or a brave man, for leaving a blog wide open to the world is inviting a veritable deluge into your life. Anyway, the experiment is done and the results are out. The results are mixed but interesting. Blog-based review can in fact add in valuable ways to the peer-review process.

I am a little rushed for time right now, so will continue this discussion a bit later. Stay tuned. (to be continued)


Blogging had already changed how I worked as a scholar and creator of digital media. Reading blogs started out as a way to keep up with the field between conferences, and I soon realized that blogs also contain raw research, early results, and other useful information that never gets presented at conferences. (emphasis mine)
So one significant gain from using the internet -- blogs, specifically -- for the early dissemination of raw, unreviewed result is a speeding up of research because scientific research is not a solitary pursuit (as is often imagined) but occurs in a social context. Many researchers are comfortable with reading unreviewed research, because as responsible and knowledgeable scientists, they know enough to tell good science from bad. Integrating blogs into research is particularly useful in academic disciplines where significant work is done outside of academia:
Given that ours is a field in which major expertise is located outside the academy (like many other fields, from 1950s cinema to Civil War history) the Grand Text Auto community has been invaluable for my work.
Compared to traditional peer review,
... the blog-based review form not only brings in more voices (which may identify more potential issues), and not only provides some “review of the reviews” (with reviewers weighing in on the issues raised by others), but is also, crucially, a conversation (my proposals for a quick fix to the discussion of one example helped unearth the breadth and seriousness of the larger issues with the section).
In traditional peer review, on reviewer might make a point, but the author may not understand how widely such a view is shared. Blogs, however, allow other reviewers to refute or extend an observation made by one poster, thereby giving weight and adding substance to an opinion. On the down side,
... the flow of blog conversation is mercilessly driven by time. While it is possible to try to pick up threads of conversation after they have been quiet for a few days, the results are generally much less successful than when one responds within a day or, better yet, an hour. I hadn’t anticipated or planned for this.
Turning on a blog-review will have the author on a treadmill that never stops running, a firehouse that ceases to spew forth torrents of commentary, many of which she may never be able to read at all, leave alone respond to. There is another, perhaps more troubling problem with online reviews:
One concern expressed repeatedly about the blog-based review form — by blog commenters, outside observers, and myself — is that its organization around individual sections might contribute to a “forest for the trees” phenomenon. While individual sections and their topics are important to a book, it is really by the wider argument and project that most books are judged.
This is in contrast to traditional anonymous, double-blind review where reviewers tend to provide 'big picture' commentary regarding an entire manuscript rather than mere sections, or chapters. The positive aspect of this 'problem', however, is:
The blog-based reviewers offered almost no remarks comparing chapters to one another — perhaps because they experienced the manuscript more as sections than chapters. Still, as I will discuss below, they also offered much more detailed section-specific commentary, much of it quite useful, than it would be possible to expect from press-solicited anonymous reviews.
The most significant benefit of online-reviews is the diversity of perspectives that it is possible to collect from a wide readership. In the traditional review process, it is not feasible to send out a manuscript for review to more than a few reviewers due to large number of manuscripts that need review.
When only a few reviewers look at each manuscript, each will have some areas of relevant expertise. But for many manuscripts, especially interdisciplinary ones, there will be many topics discussed for which none of the reviewers possess particular expertise. There’s no real way around this.
This happened repeatedly, with humanities and social science scholars, digital media designers, computer science researchers, game players, and others offering helpful thoughts and information about everything from dialogue system interface specifics to statistical AI models.
Crowdsourcing "peer-reviews" through blogs certainly has a future. Work is needed to make the task manageable, and even if a 'lite' version were made possible, then it would make an excellent augmentation to the traditional peer-review process -- not eliminating it, but making it much more useful and usable.

No comments:

Post a Comment