Giving Feedback to Students: Instructor vs. Machine

“edX, a nonprofit enterprise founded by Harvard and the Massachusetts Institute of Technology, will release automated software that uses artificial intelligence to grade student essays and short written answers.”  John Markoff, New York Times

T-Pain's Singing Teacher

There has been much discussion this week among educators about the idea of robo-grading, or machine grading, prompted by the New York Times article Essay Grading Software Gives Professors a Break of which the quote above is an excerpt. To date over 1,000 comments posted to the article, most vehemently opposing the idea of automated grading. Quite by coincidence, I posted an article on this blog, Four Reasons Why we Need Instructor Feedback in Online Courses that emphasizes the value of instructor feedback specifically in online courses—and I stressed why MOOCs won’t cut it. 

My argument is that undergraduate students need constructive and specific feedback to develop their writing and critical thinking skills, and a massive course such as a MOOC cannot provide it. My view contrasts starkly with the president of edX, Dr. Agarwal.  Agarwal is convinced that students can learn from, and develop writing skills in a MOOC setting with feedback via automated grading.  It’s the immediate feedback that is useful states Agarwal, and that students are able to “take tests and write essays over and over and improve the quality of their answers” (Markoff, 2013). Hmmm—while I do agree that immediate feedback supports the conditions required for learning, I don’t see students being motivated to rewrite an essay again and again.

How Does Automated Grading Affect Student Motivation?

In response to the NYT article, Elijah Mayfield, founder of LightSIDE Labs, developed a computer program that uses “machine learning to automatically assess written text“.  Mayfield wrote a post for e-Literate discounting the claims outlined in the NYT article which generated over 50 comments, mostly from university professors opposing the robo-grader concept. I have minimal experience with machine grading, and my comments to Mayfield’s post took a different (perhaps less informed) approach, focusing more on the conditions of learning. The concerns I have focus on students perception and their willingness to consider automated grading as valuable. Also its effect on student motivation, thus potential learning. Two of my recent posts, here and here, reference research studies that support explanatory and constructive feedback from instructors.

Below is the comment I posted in response to Mayfield’s post Six Way the edX Announcement Gets Automated Essay Grading Wrong on e-Literate.

Thank you Elijah for this in depth post. Questions I have-how do students perceive machine grading? And how much research has been done on the impact on learning performance and motivation?

I wonder what the implications are (or will be) on students’ motivation, and quality of their effort and work? Students spend time on writing essays, some more than others, yet for students to know that a real person will not be reading their essay, could impact many processes. My teenagers have been exposed to automated grading periodically at their high school and they both strongly dislike it (despise it is a more fitting term). They discount its value completely. I predict that teenagers and young college students will not be receptive to this type of grading. Why should they spend hours researching, writing and re-writing an essay when they know no one ( a real person) will even read it? Even more so in a MOOC that is not for credit, why on earth would you write an essay for an automated grader?

For large-scale classes, as you discuss in your post, peer grading would be a far more valuable exercise and learning experience for students than machine grading. Two studies I have read show that there is 20 to 25% grade inflation with peer grading, but the learning for sides, peer and student is far more meaningful in my opinion.

I am all for technological advancements, yet at some point are we not going too far, and when will that be? (A rhetorical question). However, I do look forward to reading further and learning more about this method. Thank you for the thought-provoking post. Debbie

Response from Elijah Mayfield:

Debbie – There are mixed results in the literature, but most of all they point to a negative impression from students if they’re working purely alone, even if writing skill does go up. However, if automated technology is being used in a collaborative setting, scaffolding the interaction, we see almost the opposite effect – compared to a control it increases student satisfaction with the learning experience, and their own self-efficacy, even if the learning gains on top of that collaborative process are modest…

Mayfield’s response is fair and honest, and I appreciate his willingness to engage in discussion with readers that commented and expressed skepticism, if not criticism of his program. I encourage readers that are interested in learning more about the topic to read the post and the discussion that follows it.

Let’s Think about This More…

I want to learn more about the idea of machine grading, and am eager to review feedback from students after edX implements its grading software that Agarwal speaks of in the NYT article. Though I remain skeptical—I’m keeping my mind open. As mentioned, I am most concerned about its implications on student motivation, and the potential long-term effects on learning should machine grading become the norm. There is an emotional side to this story, the idea of students making personal connections and feel that their writing is of value when writing to a real person. Can the joy of writing be fostered when writing for a machine?

Further Reading:

Image credit: Mike Licht,’s photostream (Flickr)

Need-to-Know News: Minerva’s Model, MOOC Students Reveal Why they Quit and More

In this ‘Need-to-Know’ blog post series I share noteworthy stories that speak of need-to-know developments within higher education and K-12 that have the potential to influence, challenge and/or transform the traditional model of education.

Screen Shot 2012-04-11 at 9.01.58 PMNot to disappoint, last week the education sector provided numerous stories worthy of review—I’ll highlight the key themes for readers in this post. One development worth watching is Minerva, a want-to-be Ivy League Online School that is moving forward with a new model for learning. And with MOOCs, never a dull week—I’ll share the top stories here. For those looking for professional development opportunities, I’ve include two upcoming [online] events to consider attending.

Minerva University
I first read about Minerva in 2011 and was intrigued. At the time it sounded viable and interesting—a university that planned to offer rigorous course work through online course work and experiential learning in International locations with small cohorts of students. Yet that was before the MOOC movement, and now looking closer it appears to be an expensive and perhaps even elitist education for select students that must meet stringent, if not unreasonable entrance criteria.

I don’t want to discredit the program completely as it does offer a novel approach to higher education, which is what we need more of – innovative ideas for educating students. As we speak, the school is assembling an impressive team, evidenced by the recent hire, Stephen Kosslyn from Stanford, appointed as Minerva’s Founding Academic Dean. Apparently, Kosslyn’s also in recruiting mode (Rivard, 2013).  Yet there are a few red flags.  One is the selection criteria for students, which according to Minerva’s founder Ben Nelson [former executive of Snapfish], will be gifted students that will need to pass, “psychometric tests to try to find students who are self-confident leaders, and intellectually and emotionally mature”.  This strikes me as odd—how many 18 year-old teenagers do you know that are intellectually mature? Another concern, students will be traveling globally during their studies, which makes me think that some parents might be concerned about their teenager’s health, safety and well-being as they globe trot around the world, and some kids might not be able to adapt.

We’ll see what happens, this for profit venture has already raised $25 million, but apparently needs millions more.  I am a tad skeptical of the viability of Minerva, but what it does do is challenge the model of higher education, which David Brooks from the New York Times suggests is a good thing, The best part of the rise of online education is that it forces us to ask: What is a university for? (2013).


1) US MOOC platforms openness questioned.  A professor at UK’s Open University states that platforms edX and Coursera do not meet the criteria of openness as students need to register first with the platform, then enroll in courses they want to take. A valid point. Though free, Coursera and edX cannot be viewed without creating an account which is counter intuitive to the concept of open.

2) Essay Grading Software Gives Professors a Break. edX, the MOOC platform of Harvard and MIT plans to introduce essay grading software for several courses. It’s the robo-grader that will grade students written work. Automated grading perhaps has its place, but I’m not sure if MOOCs are the place given the trajectory that MOOCs are on within higher education.

3) Top ten reasons why MOOC students drop. Though an unscientific one, Open Culture conducted a survey among MOOC students, and even with 50 responses, the feedback is most insightful.

Top Ten Reasons Students Didn’t Finish MOOC:

  1. Takes too Much Time
  2. Assumes Too Much Knowledge
  3. Too Basic
  4. Lecture Fatigue
  5. Poor Course Design
  6. Clunky Community/Communication Tools
  7. Bad Peer Review & Trolls
  8. Surprised by Hidden Costs
  9. Shopping Around
  10. Want to Learn,  Nor for Credential

Professional Development

This week should bring more exciting news. Stay tuned. I’ll be posting to my Twitter stream during the Sloan Symposium this week as well. Have a great week.

Pearls of the Week: News Educators Need-to-Know

Image representing Pearltrees as depicted in C...I’ve selected the weightiest of pearls [bookmarks] to share this week –  critical need-to-know information for educators about MOOCs and a new ed tech tool launched this week Google’s Course Builder. New developments in MOOCs are gaining momentum which are raising questions about credentialing in higher ed institutions, forcing discussions about their implications, and in some instances impacting policy decisions within universities. Course Builder on the other hand, may impact online education at a different level, and though this open-source software has garnered much attention and touted as a big opportunity for online education, I’m not convinced that this will be the case.

For those new to Pearltrees, pearls are bookmarks; noteworthy articles, blog posts and resources which I’ve collected and organized into a digital content collection tool Pearltrees. Pearling is the primary method I use to build my knowledge network. Click here to learn more.

MOOC Developments

1)  Moody’s: Massive open online courses carry mixed credit implications for Higher Ed.  Moody’s Investor Service, a respected research company that advises firms on credit ratings and financial risk, released a report this week, “Shifting Ground: Technology Begins to Alter Centuries-Old Business Model for Universities”. The fact that an investment advising service is weighing in on the MOOC discussion is significant – education is big business and MOOCs are the newest disruptor. Key points:

  • “Most universities will likely gravitate to a ‘mixed’ model that combines residential learning with the new technology, some will increasingly feature online course delivery, and some colleges may choose to create a niche by remaining focused solely on the traditional residential-classroom experience.”
  • “MOOCs and related technology have the potential to transform a university’s operations, academic and social programming, and pedagogical approach.”
  • The report is behind a pay wall – the price to download the full report is $550.00. Someone is making money from the MOOC movement. Click here to read the summary and/or to buy the report or here for the full article.

2)  A First for Udacity: A U.S. University Will Accept Transfer Credit for One of Its Courses. The decision by a university within the United States to accept transfer credit from a MOOC provider [Udacity] is big news. Colorado State University Global Campus announced last week that it will give full transfer credits (three credits) to students who successfully complete Introduction to Computer Science: Building a Search Engine offered by Udacity. Key points:

  • “Several European universities, including the University of Salzburg, the University of Freiburg, the Free University of Berlin, and the Technical University of Munich, have already given credit for an earlier Udacity course.” Click here for full story.
  • Related story: Colorado State to Offer Credits for Online Class, NYT

 3) edX Announces Option Of Proctored Exam Testing Through Collaboration With Pearson VUEWow. This is big news. edX, the joint online venture between Harvard, MIT and most recently UC Berkley, is planning on providing students with the option of taking a proctored exam at the end of a course (MOOC). The exam will need to be completed at a Pearson VUE testing center. When we consider the previous story about Udacity and colleges accepting credits from MOOCs, one can see the implications for higher education. Click here to read the Pearson VUE press release.

Google’s Course Builder

1) Google Releases Open-Source Online-Education Software. Google released an open-source platform this week, Course Builder which has the ability to create and host an online course on the Google platform. Here is Google’s statement, which accompanies the YouTube video introducing Course Builder.

Course Builder is our experimental first step in the world of online education. We hope you will use it to create your own online courses, whether they’re for 10 students or 100,000 students. Course Builder contains software and instructions for presenting your course material, including student activities and assessments ….To use it effectively, you should … [have] familiarity with HTML and JavaScript.”

I am a bit puzzled by this tool –  who really is the target market for Course Builder?  It seems to me that with so many other options available, and many already in place, why would one use Course Builder? Another consideration is users must have skills equivalent of a Web Master, know HTML and JavaScript to work with it.  I read an informative blog post by Phil Hill at e-literate about Course Builder, and he suggests that Google’s strategy is to break into the market of hosting MOOCs for big universities and MOOC providers with this application. This makes sense, given that most of the business of hosting MOOCs is currently with Amazon’s Web Services infrastructure. As I’ve said before, education is big business. Click here to read Phil Hill’s post.

If there is one thing that is constant, it is change. To view my entire collection of Pearls, visit my Pearltree page, which is open to anyone.

The Next Big Disruptor – Competency-based Learning

The ‘model’ for higher education not only has to change, but will change, it’s inevitable. And, online learning won’t be the catalyst, but competency based learning will be –  how learning is assessed and degrees are granted will be the impetus for change. When speaking of ‘model’ in this context, it’s similar to a business model, where the education model is the framework for how higher-ed operates – which is, 1) how institution leaders organize people [faculty, administrators] 2) curriculum is developed and packaged  3) a place is provided [facilities, classrooms, libraries, lecture halls] to deliver education 4) and degrees are granted [based upon credit hours, (or seat-time) and assessment], all of which keeps the institution viable.

“To date, online learning has disrupted the model described above at the ‘place‘ stage [facilities i.e. classrooms, #3], causing serious waves.”

However, I predict that further disruption to the model is on the horizon –  its #4, granting degrees to students by credit hours or seat time, that’s going to shake-up the model as we know it, transform higher education to the core. In this post I’ll share what competency based learning is and what educators, both of K-12 and higher-ed will need to know about competency based education (as we’re going to be hearing much more about credentialing and assessment in the next few months).

What is Competency Based Education?
Competency-based education is fundamentally different from traditional higher education as we know it, rather than credentialing a student based upon ‘seat time’ or credit hour [duration of course] and assessment, the student is credentialed based upon demonstration of the knowledge and/or skills required to meet an established skill set (competency).

Defining competency is complex, and an educational competency even more so. Fortunately, the Organization for Economic Cooperation and Development (OECD) through PISA, (Programme for International Student Assessment) has done much work on defining competencies, and created a framework for comparing student competencies for purpose of assessment. A report completed by PISA states this,

“A competency is more than just knowledge and skills. It involves the ability to meet complex demands, by drawing upon and mobilising [mobilizing] psychosocial resources (including skills and attitudes) in a particular context.  For example, the ability to communicate effectively is a competency…” [PISA]

Furthermore, according to Center for American Progress, an independent nonpartisan educational institute which engages in research and dialogue about various issues, recently hosted a forum on competency-based education with stakeholders and experts in higher-education. In a brief published on June 7th, the concept of higher-ed accreditation is challenged,

“Unlike the credit hour, which is standardized around time, competency-based systems give “credit for learning no matter where it happens [or how long it takes]….. students would be able to build on their own skills, abilities, and knowledge, the time required to obtain a degree would be reduced, resulting in a less expensive and higher-quality education.”  (Soares, 2012)

Who’s Doing it?
It’s game-on with no barriers, as a federal law Congress passed in 2005 cleared a path for Western Governors University (WGU) to pursue “direct assessment” of student learning, allowing this college and other institutions to participate in federal aid programs without tracking credit hours.

Western Governors University
Western Governors University (Photo credit: Wikipedia)

1) Western Governors University, a fully online program that offers Bachelor and Master’s Degrees, is the leader in  competency education. The school’s curriculum is based upon the competency-based approach, and uses the online format to create self-directed learning for students with assessments that measure students mastery through demonstration. The critical element to this model is well-defined and specific description of the competency along with observable behaviours and actions.

2 ) Mozilla’s Open Badges Project, The newest program created in conjunction with the McArthur Foundation and Mozilla. The goal is to offer credit in the form of ‘badges’ to learners who demonstrate mastery of a given competency as identified by an organization that offers the education and training in a given skill or learned ability/knowledge.

3) MOOCs. The newest, and most potentially disruptive initiative in Higher Education, are Massive Open Online Courses, through projects such as Coursera and Edx (Edx is a joint open education program between Harvard and MIT).  MIT introduced the premise for ‘credentialing’ at the launch of MITx “[MITx] allows for the individual assessment of any student’s work and allows students who demonstrate their mastery of subjects to earn certificates awarded by MITx“. How Edx will proceed with credentialing is yet to be determined. But, no doubt there will be more news to come.

What Educators Need-to-Know
What does this mean for educators? What are the implications? There are several. First, the obvious, that there is much pressure on higher ed institutions from many stakeholders to justify the price of tuition and show value to students and parents. Though this is out of most educators control, below are the need-to-know or need-to-dos for educators:

  • Define competencies or learning outcomes for instruction provided, whether online or face-to-face classes (in anticipation of this ‘potential’ requirement in the future).
  • Define what students should be able to do, after a given course is complete – beyond the final assessment – the practical applications.
  • Learn how MOOCs work – take a course through Coursera or MIT Opencourseware other provider to find out how MOOCs operate. They are free – and give educators another perspective on education, as the ‘student’.
  • Be prepared for the competency-based learning discussion – being aware of how it works, and who is doing it, allows for constructive dialogue and discussion.


Why Harvard just raised the Bar [way high] for e-learning

E-learning has just been given premier status. Sooner rather than later, online learning will no longer be viewed as sub-standard, second-rate or ‘lite’ education. The heavy hitters of higher ed, the IVY Leagues are now behind online learning, open source, MOOCs, [Massive Open Online Courses] which will have cataclysmic effects for higher ed. I’m speaking of Harvard and MIT’s announcement this past week which introduced edX, a collaborative partnership between MIT and Harvard which will offer Harvard and MIT classes online for on-campus students and anyone with access to the Internet.

The edX initiative is going to accelerate a shift to a new model for education, actually it’s going to be more like a tsunami that’s going to hit campuses which is how John L. Hennessy, president of Stanford described it, according to Ken Auletta in Get Rich U. (2012). 

Why is Harvard getting ‘in’ now?
Why has Harvard (finally) thrown its hat into the ring? It’s not just because Stanford’s doing it, as is Princeton, University of Michigan and U Penn, (all of which are doing so collaboratively through a similar initiative coursera), but because they HAVE to. It’s not optional (similar to the question many campuses faced five or so years ago, should we be on Facebook – which changed to when do we get on FB) – the time has come, either acknowledge that the education model needs to change or just close your doors and crawl under a rock. With the influence of social media, 24/7 Internet access, there’s a need to respond, adapt. Furthermore, this higher education bubble we’re in is going to burst, soon. This bubble exists due in part to the cost of higher education, which according to the National Center for Public Policy and Education has increased 440% in the past five years, nearly four times the rate of inflation (Lataif, 2011).

Enough of that, let’s look at the practical reasons why Harvard [and MIT] need to change…

1) Transform: “To enhance campus-based teaching and learning“, according the edX website, which I view as a productivity issue. As much as I dislike to apply business terminology to education, it’s a necessity, universities need to innovate and embrace technology, find new ways to conduct the ‘business’ of educating.

2) Relevancy for on-campus students: Make no mistake, MIT and Harvard recognize that learning needs to transform in order to remain relevant and provide meaningful, enriched learning for students, and we’re talking about on-campus students. EdX appears to be just as much about transforming learning for and reaching students worldwide, as it is for students in traditional face-to-face classes. Watch the two-minute YouTube video at the end of this post where leaders of both institutions share their vision for edX.

3) Education Research: It appears the plan is also to conduct research into educational practices and theory, “EdX will support Harvard and MIT faculty in conducting research on teaching and learning on campus through tools that enrich classroom and laboratory experiences“, a good thing.

What it means for Online Education
We [educators] need to step-up – the bar is high – educators now more than ever need to create and deliver courses of quality and rigor. It also means capitalizing on the value of the educational experience, and showing the student how he or she will benefit from completing for-credit courses. Audit students will become a thing of the past.

The Ivy Leagues have an International reputation that speaks for itself, there will be instant name recognition, which will be associated with ‘quality education’ (whether accurate or not).  A recent example, Stanford’s  Professor Thrun offered a free online course in Artificial Intelligence last year, that drew 160,000 students from 190 countries. Thrun found this to be a life changing experience, and started Coursera, which offers a full range of courses from Ivy League professors. Granted college credit is not earned. However, there may be something in the offing for edX, as Mitx (which has now merged Harvard edX) had  planned to offer recognition of completion (certificate after testing for content mastery) at some point. We’ll have to see what happens with edX.

L. Rafael Reif, Provost of MIT, made it clear that quality and rigor will not be compromised,

This [edX] should not to be construed as MIT Lite or Harvard Lite, the content is the same”. (YouTube video).

What are the Nay Sayers Saying?
Of course there are plenty of skeptics – and I don’t discount their concerns. I feel it’s worthwhile to consider other viewpoints. Collectively some concerns include, 1) how to motivate students that aren’t intrinsically motivated [to engage with online content], 2) how to promote cross discipline learning, 3) how to get feedback from students that have dropped out [and why], 4) how to monitor progress of students (if the case of thousands of students), to name a few. One blog post I read, was quite pessimistic, suggesting that the impetuous of these schools offering online education is motivated by profit. I respect this educator’s position, though I do not agree, as it’s all about what I’ve mentioned above.

Transforming education is about moving forward, progressing and the time has come. I’ll close with one quote made by chairman of IBM when the prototype for the personal computer was introduced. It’s rather humorous now.

“I think there is a world market for maybe five computers.”
– Thomas Watson, chairman of IBM, 1943

Further Reading
Universities on the Brink, Louis Lataif,
Edx: A platform for MOOCS, and an opportunity for more Research about Teaching and Learning Online, Audrey Watters, Inside Higher Ed
About edX,
What’s the difference between a MOOC and the University of Phoenix, More or less Bunk

Photo Credit: Terrible Tsumami, , Flickr, Creative Commons