Academia tests crowd-sourcing

Journals are slowly moving toward open-online-peer reviews for scholarly work

The word "medieval" is often used, and perhaps appropriately so, by scholars and academics when describing the world of academia. 

Any grad student can surely list off names of several resentful scholars they know: scholars who spent a near decade slogging through doctoral research only to end up teaching equally resentful Freshmen part-time for some $2,000 a month. 

On paper, a Ph.D. life-plan sounds ludicrous: go to college for four years; then sign up for an additional six to 10 years. Upon that, if you’re really lucky and really smart, you might end up lucky enough to find a job making, oh some $45,000 a year. 

To top it off, to achieve tenure, professors must have their work appear in elite publications. That process requires submitting work to an elite panel of specialists in a double-blind process where neither the reviewers nor the author know who the other is.

The process can take months, or longer, despite the fact that the world is serioulsy moving away from long publication cycles.

But movements are afoot to bring the peer-review process to the digital age by allowing a crowd-sourcing process to the evaluation and assessment of work.

This fall, the prestigious Shakespeare Quarterly, founded in 1950 by the Shakespeare Association of America and published by the Folger Institute, will use an open-review process for a special issue, Shakespeare and New Media. 

Four essays, not yet accepted for publication, were posted on the journal’s Web site, and experts were invited to review and comment on them via MediaCommons, an academic digital network. 

Unlike the traditional peer-review process, anonymity was left at the doorstep, and other registered users were allowed to comment on the essays and participate in the scholarly discourse as well.  By the process’s end, a total of 41 people had discussed the essays in more than 350 comments.  The dissected, reviewed, and revised essays were then reviewed and published by SQ.  The experimental issue will be released on September 17, 2010, but transcripts are available on the journal’s Web site.

The experimental SQ issue has paved the way for other publications who wish to implement the same open peer review process, such as Postmedieval.  It has also led to broader public discussion of the traditional peer review process, its benefits, and its disadvantages.

One of the obvious problems with this process, however, is the potential for scholarship to devolve into a popularity contest.  As Harvard sociologist Michele Lamont points out in her book, How Professors Think: Inside the Curious World of Academic Judgement, “Knowledge is not democratic,” as quoted by The New York Times.  Neither is it owned by one club.  Another glaring problem with the open online peer review process is the fact that without the security of a double-blind system, there is nothing to protect journals from cronyism. 

The benefits of open-source knowledge

Among the benefits of an open-online-peer-review process is the fact that knowledge is not relegated to an exclusive core group of people who get to set the standard for what passes as scholarship.  In an open, online setting, anyone can post a comment and reviewers are held accountable for their statements.  The balance of power is not so much tipped as redistributed.

But Theodora Bloom, who has a Ph.D. in developmental biology and is an editor for BioMed Central, highlights the fact that one of the key advantages of an open online peer review process is the fact that scholarship doesn’t have to end at publication.  “Use of the Internet also means that peer review need no longer be a one-step process: preprint servers and online comment and reviewing systems allow an article to accrue commentary and analysis after it is published,” she states in her article, “Systems: Online Frontiers of the Peer-Reviewed Literature,” published by Nature. 

This is probably the most innovative facet of online-peer review.  Knowledge is not cemented in place by one person—or by one group of people.  It is ongoing, shifting, and evolving.  Bloom likens this continuous editorial process to updating software.

“Version 1 is modified through versions 1.1, 1.2 and so on, until a new version 2 is released. If some data have been wholly reanalyzed, or indeed if a research article describes a software tool rather than a data set, is it not reasonable to publish a second article describing progress at some point, or a new version of the first article?” 

Education and scholarship are not simply adapting to digital media, though—they are being inaugurated into the age of social media.  Could we be on the brink of education’s future as user-generated content?

(Image source: bc.net)