Why and When Peer Grading is Effective for Open and Online Learning

Is peer grading an effective assessment method for open and online learning? What about in MOOCs where student feedback may be the only means of determining a pass or fail in a course? This posts examine peer grading and suggests what conditions must be present in order for peer grading to be effective.

Keeping Score for the TeamAfter I wrote the outline for this post I came across this essay, by history professor Jonathon Rees, Why Peer Grading Can’t Work. The title was in stark contrast to my views on peer grading, but I  incorporated Rees’ argument here as it is worth consideration. Rees is also author of a blog I follow, More or Less Bunk where he writes about current issues within Higher Education often with a slice of sarcasm. Our views on online learning couldn’t be more dissimilar, yet I appreciate Professor Rees’ perspective and enjoy reading his posts. In his essay Rees shares his views on peer grading, which he experienced while taking a world history course as a student through Coursera. Rees’ central argument is that peer grading can’t and won’t be effective in grading written work produced within MOOCs, as the majority of students-as-graders are not able to provide quality feedback that can help students develop their writing and critical thinking skills.

In this post I’ll examine the conditions that need to be present for peer grading to work,   factors that can sabotage the process, and I’ll address the points put forth in Rees’ essay.  I’ll also explore briefly what is at the root of the differing views on peer grading, which I suggest is based on differing perspectives on learning philosophies [which I wrote about it in my last post, The Tale of Two MOOCs: Divided by Pedagogy].

Conflicting Views on How People Learn
At the root of the dissension over peer grading is the conflicting view on how people learn. One of Rees’ comments within the essay “Professors in the trenches tend to hold their monopoly on evaluating their students’ work dearly, since it helps them control the classroom better by reinforcing their power and expertise,” supports a cognitive and instructor-focused learning orientation. The concept of peer review, which leaves for the most part the instructor out of the equation, aligns with the social constructivist learning orientation. There is strong support in constructivist theories for the peer review which is grounded in student-centered learning where students learn as much from the review process itself as from the final grade on an assignment.

A paper on peer review published in 2007 described how the idea of peer review is embedded in the philosophies of learning theorists. The authors call out Vygotsky and his beliefs that learning occurs in, and is mediated by, social interaction. Authors do present the downsides of the peer review process, though at the conclusion of their research they determined that students involved in peer review perform better academically than peers graded only by their instructors (Lu & Bol, 2007).

8220513675_3c7aaea7b2

Peer Grading @ Coursera
When developing a course there are numerous assessment methods instructors can choose from, yet the choice of the tool and method depends upon the learning conditions within the course that include—the planned learning outcomes and the learning environment. An analysis of these conditions determine the best assessment strategy [which may include several methods in one course] for the course and its objectives. Peer grading worked well in the Digital Cultures I completed recently with Coursera in consideration of the learning context—the environment, topic and goals of the course. Also the course was not-for-credit and only five weeks in length.

In Rees’ history course, peer grading was used to evaluate essays, which appeared to be the primary method for assessment. Given the topic of Rees course, essays could be considered an appropriate assessment mechanism given the number of students and the availability of the software available that can facilitate peer grading [see link at the end of this post on Calibrated Peer Review]. Rees admitted the guidelines were clearly outlined as to how to grade, and that the grades he received were accurate, yet it was the quality of comments that he felt was lacking,

For me at least, the primary problem with peer grading lay in the comments. While I received five comments on my first essay, for every subsequent essay I received number grades with no comments from a minimum of two peers and as many as four…Every time I did get a comment, no peer ever wrote more than three sentences. And why should they? Comments were anonymous so the hardest part of the evaluative obligation lacked adequate incentive and accountability. (Rees, 2013)

What can Go Wrong: The Loafers and Others
What Rees shares here is a good example of what can go wrong with peer grading in anonymity in a massive course. Though the algorithms within a peer grading system may work, the problem lies in the uncontrollable learning conditions inherent to a massive course. One example is outlined in Lu & Bol’s paper I quoted earlier, which is the phenomenon of social loafing or hide-in-the crowd behaviours associated with anonymity. Students that fell into this group were physically and cognitively lazy, not contributing to the process as required. This phenomenon was referenced in several other research studies within the paper. I suggest another group be added to the mix besides the loafers— students that cannot provide feedback due to the lack of necessary skills, whether it be education background or language.

iStock_000019698408XSmallWhen Peer Grading is Effective
Peer grading has tremendous value for a variety of learning situations in higher education, though it requires a specific set of learning conditions to be present in order for it to work as intended. Listed below are a list of conditions needed to ensure that peer grading  is effective: 

1) When learners are at a similar skill level.
2) When assignments are low stakes [i.e. when a course is taken for professional development of personal interest as was the Digital Cultures course].
3) Where credit is not granted.
4) When learners are mature, self-directed and motivated.
5) When learners have experience in learning and navigating within a networked setting [if the review is completed in an open and online setting].
6)  Learners have a developed set of communication skills.

The break down in peer grading occurs when the learning environment cannot provide the conditions as mentioned above. Also, there are other factors that can sabotage its effectiveness, including an assignment that requires a high level of critical thinking skills, or when there are students in the mix that are non-participative, or have intentions that don’t align with the course. In my Coursera experience for example, with Digital Cultures, one of the artifacts I was to evaluate in the peer grading process [which was a website] was a marketing pitch. This happened to at least one other student, according to the course Twitter stream.

Closing
Peer grading has great value. It has proven to be effective in variety of education settings. It can work well in MOOCs that are not for credit, when the assignment lends itself to a peer review, such as the digital artifact we graded in our Digital Cultures course. It can also be very effective in small, closed online classes where students are at similar skill level and receive instruction and guidance in how to grade within the process. Yet there are times when it won’t work, this is where I agree with Professor Rees, the situations where students do need detailed and constructive feedback from an instructor, or mentor that is qualified. Furthermore, there are many students that need remedial support in writing and communications skills, some require support in how to learn online, and how to be responsible for their own learning.

Further fine tuning is needed to address some of these issues within MOOCs. I see the opportunity for two or three tracks within a MOOC for students wanting to participate at varying levels, which would address some of the issues with peer grading by addressing some of the required learning conditions as mentioned. Another suggestion is to offer resources for skill development for students requiring help with their writing skills within the course itself, i.e. links to Purdue’s writing center.  Perhaps, addressing students that receive a grade below a certain level in the peer review after the course closes with suggestions for additional writing skill development would be helpful. There are many options still to be explored. Time will tell.

Update: Response post from Professor Rees on More or Less Bunk, here.

References:

14 thoughts on “Why and When Peer Grading is Effective for Open and Online Learning

  1. Pingback: Peer Editing Comment on Experts Blog | Alexandra's Blog

  2. Dear Debbie,

    Thank you for sharing such an interesting topic. As a student myself and in an International School, it can sometimes be difficult to use peer editing in an effective way. Language barriers and lack of detailed comments makes it hard to understand why peer editing would be useful, but after reading this post I have noticed just how effective it can be. It’s interesting to see what my peers think of my work rather than the teacher and I think that peer editing works even more effectively when anonymous. I find that many times when doing peer editing in class students are afraid to state their honest opinions because of friendship reasons. People don’t like to seem mean by giving criticism because they think the others would view it as offensive or disrespectful, so in the end they don’t really help. On the other hand, when peer editing anonymously I mostly get useful and honest feedback which I can use to develop and improve my work.

    I believe that sharing your work and trying to help other students in the classroom area is a very important aspect in teaching because in the end we are learning from each other. There are some that will always be stronger in a certain subject than others, but that exactly why peer editing is such a great resource to use. Those who are not as great in the subject receive help from someone they can connect with and understand better than with a teacher.

    Overall, I really enjoyed reading the piece because I thought it brought a fresh idea on a subject which people often don’t think is important but in the end really is. Peer editing isn’t always looked upon as a great value, and in some circumstances it really isn’t. The next step to think about is how we can make is a great value everywhere. What can we do to make peer editing even more effective, or is there even another way to make the process more productive.

    Thank you for sharing your thoughts on the subject and I hope to hear back from you,

    Alexandra Ehrensperger

    • Dear Alexandra,

      Your comments highlight another aspect of peer grading which I did not address in the article, but is a key factor that affects peer grading effectiveness, and that is the consideration of cultural differences. Providing feedback as you identified can be a sensitive endeavor, more so in some cultures that value group vs individual efforts, or have different values when it comes to education. Of course language barriers are another critical factor as you mentioned.

      Yet peer grading and review can be very effective and so valuable when facilitated carefully and purposefully by the teacher. Anonymous feedback of course helps this, but so can the teacher by providing students with instruction and practice in peer grading. Research has shown that students become very effective at grading other students work with instruction and practice. I might suggest in the classroom that first the teacher introduce the idea of peer editing, or review of other students work, when a piece of work is in ‘draft’ format. Guidelines that are specific for how to peer edit (review) are critical in helping students be successful with this exercise. For example instructions might include, ‘provide constructive feedback that states two or three positive elements of the work, and two or three improvements that the student can make to improve the essay’

      The benefits of peer review and grading and are many – which makes it such a valuable exercise.

      Thank you Alexandra for taking the time to comment and share your thoughts!
      Debbie

  3. From my experience with Calibrated Peer Review in face-to-face courses, I see the main value of peer review to derive directly from the process, rather than from the comments students receive as a product. Most of the learning happens after students have submitted their own text, when they read and critique first the standardized calibration samples, then the work of other students, and finally their own submission.

    • This is a good point – perhaps we should look at peer grading from a different perspective, the value of the process itself, and thus structure the exercise for students that will facilitate this aspect of the exercise. Thanks for commenting and sharing your experience.

  4. A very interesting article! congratulations!

    I’d like to introduce another approach to the analysis. I think that we should consider connectivism, not constructivism, as the pedagogical paradigm behind the learning experiences in most peer-evaluated activities.

    Best, guadalupe

  5. Pingback: Bang your head (Peer grading edition). | More or Less Bunk

  6. Pingback: Peer Grading: A Student Perspective from an Open and Online Course | online learning insights

  7. Debbie,

    Very interesting thoughts here. My experience with peer grading of essays in coursera is not unlike that of my undergraduate and graduate programs as a student – some folks were thoughtful and provided insightful comments and critiques, some folks gave a point value with no comments as though they were rushing through, and some folks clearly misunderstood the material in writing comments that were irrelevant. Although I always write some comment in my coursera peer reviews, I assume that my thoughts are variously judged somewhere between helpful and useless (though of course I believe that all of my comments are insightful :) ).

    A couple of things come to mind about this process:

    In academic publishing, peer review is considered the gold standard. Student peer review prepares folks for this standard early on. Goran’s comments on how students receive criticism as an assault on oneself is a serious problem that needs to be addressed. I now use props from my own experience – the well-deserved and heavily red inked paper I got back from the instructor for the first assignment in my PhD program years ago. etc.

    In the digital culture MOOC you reference, there seemed an attempt to begin breaking down the massiveness of the student body. For myself, this issue was addressed through topical study forums, Twitter feeds, and blogs where I met some folks that I continue to engage with after the course is over. I was impressed that the massive issue seemed a real concern for the instructors of the MOOC in developing solutions for future offerings. I am currently enrolled in an Innovations coursera offering that seems to be dealing with the massiveness further. The course is offering two tracks – the standard and a second where the student must work in a group and produce a group project.

    I am hopeful that the group approach option might work for the peer review as well. That is, one would select or be assigned to a known group of peers who would form a study/critique group to remain with for the entire course. Such an approach might prove workable as a second track where the commitment is a bit higher.

    As a general statement, I am impressed that the coursera folks seem to be taking quite seriously the need to innovate and refine the program.

    • Hi Robert, You bring up another valid point in support of peer grading, introducing the student to the peer review process and preparing the student to deal constructive feedback. Now that you mention it, the Digital Cultures instructors included a fairly robust section on ‘how to receive feedback’ – geared to students that are new to the process.

      I am very interested in the format for the innovations course with the two tracks for learners. This is a unique approach for Coursera and I look forward to hearing how it goes.

      Thanks for your comments Robert. As always, very interesting.
      Debbie

    • Robert, I do hope you will share a bit more about your experience with the Robert, I am interested in your experience with the Innovation course offered on the Coursera platform – in particular, how the instructors managed/guided group formation. Enabling the formation of stable groups would be useful for peer grading, but also for other forms of group work; if they have discovered elements of a mechanism to achieve that (in any MOOC platform) that is of potentially great significance.
      Did they suggest group formation at the beginning of the course? Or did they allow participants some time in the course to get a sense of the level of work, and content, and whatever other info that might inform the participants decision to get into a group?

  8. Debbie,
    Your post is insightful and point to a serious problem for MOOCs. If large groups attend a course you cannot expect all participants to fulfill the conditions you list.
    When I was teaching a net-based course ten years ago I spent some time writing to the students how they could improve the usefulness of their comments.
    • Many students (as all of us) tend to receive criticism as an assault on ourself. It is important to regard all suggestions as help to improvement instead.
    • Start with the good points of the work you are discussing and then suggest how to make it even better
    • Self-assesment can be a good start
    • As in all teaching clear criteria for assessment is very useful and helpful for all
    • Discussing why some parts are good and how other can be improved is educational for all

    No answers but maybe some ideas that can stimulate thinking for how to improve peer grading in smaller groups than MOOCs.
    Göran

    • HI Göran, Theses are excellent points to consider and to incorporate into the peer grading process. It is true isn’t it, that many people are uncomfortable with receiving constructive feedback, yet as you said, this is the instructors role to describe how to receive feedback. I don’t know how it was done in the world history course, but as I mentioned when replying to Robert, that the Digital Cultures course did included a section on this. Of course the challenge is knowing whether students read it or not.

      Yes further discussion is needed, as you mentioned. Educators need to keep the conversation going.
      Thanks for taking the time to comment! Debbie

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s