News of the Week: Robo-Grading Debate, MOOCs Promoting Peer Collaboration & New Ed-Tech Tool

In this ‘Need-to-Know’ blog post series I share noteworthy stories that speak of need-to-know developments within higher education and K-12 that have the potential to influence, challenge and/or transform the traditional model of education.

Stanford University launched new MOOC platform NovoEd this week.

In this post I’ve included the key developments of this past week that will keep readers in-the-know on education news. Another new MOOC platform, NovoEd launched by Stanford this week offers challenging courses and takes a unique approach to team projects and peer grading, and the machine grading of essays—the debate continues and is an issue that prevents one school from joining edX. Also, I’ll introduce a new tool that bring interactivity to online learning.

1)  Machine Grading Generates Petitions, Debates and a Message
The NYT story, Essay-Grading Software Offers Professor a Break continues to generate serious and heated debates. This particular article has received almost 1,000 comments, many from students, parents and teachers vehemently opposed to machine grading. [Background for readers not familiar with machine grading: a software program is programmed to provide a grade on student essays based upon factors such as essay length, grammar, sentence length, etc. However it cannot provide comments on tone, logic, development of main idea or thesis, etc.]

Online Petitions: This week I came across a site launched by a group of educators, Professionals Against Machine Scoring Of Student Essays In High-Stakes Assessment that has collected over 3700 signatures of individuals opposing machine grading. The groups’ mission “to eliminate computer scoring of essays used in any decision that might affect a person’s life or livelihood and should be discontinued for all large-scale assessment purposes.”

College Rejects edX – machine grading a factor: The use of machine grading by edX was of serious concern to Amherst College a [top-rated] liberal arts college that had been considering joining the edX consortium. This week Amherst announced it has decided not to partner with edX, citing several reasons, and computer-grading software was one of the major concerns.

They [edX representatives] came in and they said, ‘Here’s a machine grader that can grade just as perceptively as you, but by the way, even though it can replace your labor, it’s not going to take your job,’ ” Sitze [professor at Amherst] said. “I found that funny and I think other people may have realized at that point that there was not a good fit.”  (Inside Higher Ed, Rivard )

I admire Amherst for the in-depth process administrators and faculty appeared to follow to determine whether to join in on the MOOC parade. In the end, faculty voted to move more class material and classes online and to create ways to incorporate technology in the classroom rather than join edX, which sounds like a rational decision. Reading the background of how the school came to this decision, it does make me wonder what process other higher education institutions do [or don’t] follow when considering what to do about MOOCs. Hmmm.

2)  MOOC platform NovoEd: Good courses but potential challenges with peer grading System
Another MOOC platform launched this week NovoEd [formerly Venture Lab] and seeks to differentiate itself from other MOOC providers by promoting peer collaboration. I am both intrigued and impressed by the line-up of classes NovoEd offers. The course Mobile Health without Borders for example, will operate more like a conference than a course. Its focus is on global health challenges, and students will work in teams on small group assignments with the primary goal to “help you prepare for the Health Innovation Challenge, an opportunity to work with a global multi-disciplinary team and world-class mentors to design a solution to a health challenge you care about.”

There are eight courses in total, including Hippocrates Challenge, Technology Entrepreneurship and more. It really is a tremendous opportunity for interested individuals to participate in such courses with faculty from an excellent school such as Stanford.

Screen Shot 2013-04-20 at 12.43.54 PM
‘Designing a New Learning Environment’ course offered through NovoEd

Though the primary challenge I see with the platform is the team work assignments and the respective peer grading process. Here are some of the issues—how effective and inclusive can groups be when working in large teams assembled by algorithms? This platform apparently has software which separates learners into groups based on certain criteria. These are not self-selecting groups, which usually is how it goes in other MOOCs.

Differing Views of Groups vs Individuals
Another factor is the potential impact that cultural differences will have on teamwork. Though diversity in groups is a positive in terms of the multiple perspectives put forth—the problem I anticipate is NovoEd’s sophisticated peer grading program, where group members grade one another on each individual’s participation and contribution to team assignments.  Venture Lab [before becoming NovoEd] named and described this process as a “Reputation System’ for rating peers (evaluations, forum posts, team contribution)” [Stanford Venture Lab].

I believe this process of grading individual team members undermines the purpose and value of teamwork. Rather than working together to sort out differences during the process of working on an assignment, the system supports addressing the issue not in real-time, but after the fact through [anonymous] grading.

Furthermore, the idea of assigning grades to an individual’s work on a group project is a reflection of the North American value system, which values individual contributions over team. Other countries view teamwork as a collective effort, and the idea of grading individuals within the team is quite extraordinary. Professor Geert Hofstede created a well-known framework centered on four dimensions [individualism versus collectivism is one dimension] for analyzing how countries values affect workplace interactions and productivity.  I see these dimensions playing a role in the projects put forth by NovoEd. You can find out more from this website and even compare different countries rankings of its values.

Several of my peers on Google+ completed one of the first courses on Venture Lab, and have positive feedback about it, as well as some constructive. Overall it appears NovoEd has a tremendous and worthy platform and selection of courses. I look forward to reading about the results.

New Ed-Tech Tool to Support Interaction in Online Courses


This platform looks like its worthy of investigating further – as it provides easy way to build interactive content into online courses: “Smart Sparrow is an Australian ed-tech start-up pioneering adaptive and personalized learning technology. It was founded by Dr Dror Ben Naim who led a research group in the field of Intelligent Tutoring Systems and Educational Data Mining at the University of New South Wales in Sydney resulting in the development of the Adaptive e-Learning Platform”.

Have a great week!

Peer Grading: A Student Perspective in an Open and Online Course

In this post I share my peer grading experience as a student in the e-learning and digital cultures course [edcmooc] offered through Coursera. I’ll provide readers a window into the student experience how it works, guidelines provided by the instructors and assignment criteria. I’ll also share the assignment I submitted for this course and share the results—grades and comments provided by four students that evaluated my digital artefact.

Holding Blank Score CardsMy last post delved into peer grading, the pedagogy and the learning theories behind the process of peer grading. I thought readers may find it useful to view the experience from the inside, viewing the process as a student would.

Description of Assignment: A Digital Artefact
Within the five-week course, topics included, a) what it means to be human in a digital world, b) utopian and dystopian views of our world past, present and future, and c) how learning is influenced by technology in today’s digital culture. There was one assignment for the course, an artefact [artifact spelled the British way is with an ‘e’], a digital presentation representative of two or more concepts from the course, as described below:

Screen Shot 2013-03-05 at 6.21.47 PM
Description of assignment for #edcmooc from the course web site. Below this introduction on the page within the course website, were further detailed directions and guidelines, including how long the assignment should be, suggestions for platforms to use, i.e. Voicethread, Pixton, Prezi, etc. possible topics, and assignment criteria which in turn was used for grading purposes.

I learned far more than I expected from the process of completing the assignment, and from the peer grading exercise itself. It was engaging, quite enjoyable, and if we use the activity on the social networks as any indication, numerous students appeared to feel the same as I did. Discussions on Twitter @#edcmooc were prolific and are still going strong.

Screen Shot 2013-03-07 at 8.07.18 PM
Screen Shot of Twitter conversation in #edcmooc happening after the evaluation.

Student Enthusiasm for Peer Grading
Students appeared highly engaged, excited about the results of the assessments, theirs and others during the three-day evaluation period. Students shared on the courses’ Facebook page, they Tweeted, they discussed, and posted questions seeking advice about grading. Peer grading seemed to be taken quite seriously by active students.

Assignment Criteria
Often neglected in online courses are clear and specific descriptions provided about class assignments, the why, the how and the purpose. In my experience working with faculty in designing online courses, writing the narrative to cover these points requires time and attention to detail, but is well worth the time it takes, and instructors in #edcmooc followed these principles to a tee. One example is the assignment criteria:

“These are the elements peer markers will be asked to consider as they engage with your artefact. You should make sure you know how your work will be judged by reading these criteria carefully before you begin.

  1. The artefact addresses one or more themes for the course
  2. The artefact suggests that the author understands at least one key concept from the course
  3. The artefact has something to say about digital education
  4. The choice of media is appropriate for the message
  5. The artefact stimulates a reaction in you, as its audience, e.g. emotion, thinking, action” [Coursera, e-learning and digital cultures]

The GradingHow it Worked
The instructions provided on how to grade were thorough, and once I started the process of grading, the system guided me through following the assignment criteria closely. From the course website, with use of screen shots:

What you have to do
“When you have submitted your own artefact, the system will give you access to three other artefacts created by your peers, on which we ask you to provide feedback, and to offer your evaluation. This feedback will take the form of numbers and comments. It will involve the following steps for each”. Following this paragraph where further descriptions on how to make comments, (provide reasons), how to give and receive feedback, and encouraged further discussion and sharing on social media platforms after the close date of the assignment and included links for further reading on the peer review process and critical thinking.

Screen Shot 2013-03-02 at 3.50.45 PM
Screen shot: The first step we had to do in grading was, upon reviewing the artefact, was to give feedback according the criteria for this assignment.
Screen Shot 2013-03-02 at 3.50.53 PM
Screen Shot: The next step was assignment a grade following the above scale.
Screen Shot 2013-03-02 at 3.51.01 PM
Screen Shot: In the final step we were given the opportunity to write a synopsis,

My Artefact and Peer Feedback
My artefact which I submitted for grading focused on the theme of ‘being human in a digital world’ and included the concepts discussed in the coursehumanism, posthumanism and transhumanism specifically. I used the platform of Pinterest [which I joined some time ago, but didn’t use until this assignment]. I was pleasantly surprised at how effective this tool was; I was able include a fair bit of text to describe and summarize the concepts with image files, or embedded YouTube clips.

Screen Shot 2013-03-10 at 6.45.32 PM
Screen shot of my Digital Artefact on Pinterest. Click the image to view the board.

Peer Feedback
The quality of feedback I received from the student graders was overall very good. I was impressed with the comments, the insight and depth given the assignment criteria.  Also of note what how peer #2 mentioned reviewing the board helped him or her to ‘conceptualize the concepts’. This is an example of how peer grading can enhance learning for students.

Screen Shot 2013-03-03 at 10.46.38 PM
Screen Shot: Peer Feedback
Screen Shot 2013-03-10 at 6.56.37 PM
Screen shot: Final score out of 2.
Screen Shot 2013-03-03 at 10.46.55 PM
Screen shot: Final comments

I wrote in a previous post, A Tale of Two MOOCs @ Coursera, how the e-Learning and digital cultures format was an excellent example of a connectivist learning environment; a student focused learning community where students learn through making connections within a network. The digital artefact assignment and peer grading method were excellent choices in keeping with the connectivist coursestudents not only made connections within social networks throughout the course, but the peer review process served as a means to further conceptualize learning, expand personal connections beyond the class network, and prompted students to share their work with peers after the formal grading process using their real identities. The value of peer grading in this course went far beyond the grade and feedback each student received on his or her assignment; it created opportunities for learning that traditional grading could never provide. It was a brilliant fit for this course.

Links to #edmooc discussions and Final assignment Sharing

Peer Grading in Online Classes: Does it Work?

Is student grading good enough to use? A loaded question – and though it is context-dependent, the answer is yes. Yesterday I peer-graded six student midterm exams in the Introduction to Sociology course I’m taking through Coursera, along with 30,000 other enrolled students. In this learning environment, also known as a MOOC, peer grading is the only option, though it did motivate me to research its effectiveness and applicability to online courses for credit (the context for this post). Intrigued, I could see the potential in light of the online program at my workplace, not only from a time-saving standpoint for the instructor, but also for the potential in enhancing the student learning experience.

Enhanced Learning
I see the value of peer grading for what the student gets out of it, more than for the time it saves the instructor. I experienced this value first hand as mentioned above. The exam in question consisted of two short answer questions each requiring 250 words, and an essay response requiring 750 words. Though tedious [90 minutes of my undivided attention], I was far more familiar with the course concepts after the exercise, it was well worth the time.

Several of my classmates reported the same phenomenon via the discussion board – the deeper learning experienced while grading their peers. Below are snippets from select posts on the discussion board from the Coursera Introduction to Sociology course:

“…I actually enjoyed the peer review part mostly because of looking at different answers, giving me more perspective from an group of people all from different cultures. Not only that, but I felt I was able to be unbiased while grading my own work ….” Greg

I’ve learned a lot through the peer assessment, and even though maybe the scores will not be perfect, everybody who goes through that process will have now a better and more complete understanding of the first half of the course.” Horacio

For what it’s worth, one of the essays I corrected didn’t seem too good on first reading but when I checked it against the rubric, lo and behold, most of what was required was there.” Ron

“In my experience as an educator for over 25 years, it’s often the case that we think our grades should be higher than they are. ….the rubric was great, very thorough and complete…” Kendall

How accurate is peer grading? Another interesting question, and it depends upon the perspective. The viewpoint that most educators are familiar with, is where the instructor’s grade is the benchmark – his or her grading is the standard for measuring accuracy.

This is the method used in a research study by two biology professors at the University of Washington which determined that on a per-question basis, students were more generous in grading, actually 25% more, “0.27 points—roughly a quarter point on each 2-point question”. However, despite the differing grades, authors support peer grading and suggest further research be done to examine its value and the role peer grading can play in enhancing student learning. (Freeman & Parks, 2010).

How can student grading be effective within an online environment? Effectiveness depends upon the thoroughness and specificity of the rubric. A grading tool that guides the student through assessment of short answer and essay questions is critical. Below is an example from the research study mentioned earlier [a biology course was used for the  study]. There may be five or six of these questions for each point within a given question.

Sample answer: If the two species mate on different fruits,
then no gene flow occurs and they are reproductively isolated.


  • Full credit (2 pts.): Clear articulation of logic that mating on different fruits reduces or eliminates gene flow—a Prerequisite for speciation to occur.
  •  Partial credit (1 pt.): Missing or muddy logic with respect to connection between location of mating and gene flow, or no explanation of why reductions in gene flow are Important.
  • No credit (0 pts.): both components required for full credit missing; no answer; or answer is unintelligible.  (Freeman & Parks, 2012)

To begin peer grading in an online course consider the following:

  • Create your own rubric that provides standards for each point the question is worth. For instance if a given questions is worth 6 points, 6 statements will need to be developed, similar to the one above.
  • Create detailed instructions for students that clearly outline how peer grading will work.
  • Set-up the process so that students grade a minimum of 3 peers’ assignments/exams and self-grade his or her own.
  • Average the peer scores and include the student’s self-graded assignment.

Though creating a peer review exercise is time consuming at the outset, rewards are tremendous.  First, for the potential time saved by the course instructor in grading, and second, [perhaps the most important] is the value that peer grading provides for the student.

Freeman, S. & Parks, C. How accurate is peer grading (2010). CBE—Life Sciences Education. Vol. 9, 482–488.

Peer Review, Peer Grading, JaZahn
UCLA’s Calibrated Peer Review, Eric Mazur.  This is a software program/platform that can handle and support peer grading for large classes and/or institutions that plan to implement peer grading in several classes.