MOOC Design Tips: Maximizing the Value of Video Lectures

“Which kinds of videos lead to the best student learning outcomes in a MOOC?”
How Video Production Affects Student Engagement: An Empirical Study of MOOC Videos (Guo, Kim & Rubin, 2014)

An excellent question that design teams and instructors of MOOCs want to know—which kinds of videos lead to the best student learning outcomes in a MOOC?  According to a recent study conducted by researchers for the edX MOOC platform, this was the most pressing question posed by the course design teams working with its partner institutions. Given that most MOOCs offered through higher education institutions platforms such as edX, iVersity, or Coursera use video lectures as the primary content delivery source, it is a critical question that preoccupies many if not most MOOC instructional design teams. Adding to this need-to-know element is the fact that video production is most often the highest cost associated with MOOC production. MOOC video production can range from a few hundred dollars and run up to the thousands. This post suggests how institution can use resources effectively in the video production process with the primary goal of supporting students’ learning outcomes.

The report released by edX last week gives design teams some concrete data to examine. I’ve emphasized below the recommendations and practical application points from the paper for readers who might be part of a design team for MOOC, online course, or for those with an interest in video production for instructional videos.  There are limitations to the study outlined in the paper, though the depth of the analysis does provide data worthy of consideration. 

The report, the first of its kind according to the authors Guo, Kim & Rubin, analyzes students’ engagement* with lecture videos gathered from data extracted from over 6.9 million video watching sessions across four edX courses.  *Student engagement is defined in the study by:

1. Engagement time: the length of time that a student spends of a video. This is the same metric used by YouTube. Though researchers acknowledged the limitation of engagement assessed from this one-dimensional perspective.

2. Question/Problem Attempt:  Almost one-third of the videos across the four courses featured an assessment problem directly following the video, usually a multiple-choice question designed to check a student’s understanding of the video’s contents. “We record whether a student attempted the follow-up problem within 30 minutes after watching a video.”

Videos Types for MOOCs
Lectures are divided into two primary types for the study, [which mirrors most MOOCs]: 1) lecture videos for content delivery—presented via an instructor/professor (‘talking head’ is the term used in the paper), and 2) a tutorial/demonstration, a step-by-step problem solving walk-through, common in computer science courses, courses featuring mathematical concepts or science courses featuring lab demonstrations.

Video Production Format
For analysis purposes, researchers coded the videos examined in study using six primary video production formats, which I’ve summarized below, along with production styles not mentioned in the study.

1) Lecture-Style Video Formats:

• Instructor(s) with/without Presentation Slides: Features instructor(s) lecturing, with or without PowerPoint slide presentation slides inserted throughout with instructor ‘voice over’ while slide is displayed
• Office Setting: close-up shots of the instructor filmed at his or her office, typically instructor speaks directly to camera
• Classroom Setting: video captured from a live classroom lecture
• Production Studio Setting: instructor recorded in a studio with no audience, typically speaking to the camera

2) Tutorial/Demonstration Video Formats:

• Video Screencast: of the instructor demonstrating a concept, i.e. writing code in a text editor, or command-line prompt (in the case of computer science courses), using spreadsheet or document
• Instructor Drawing Freehand on a Digital Tablet, using a software program, which is a style popularized by Khan Academy videos (click here to view an example)

Other Formats not mentioned in the study:

• Instructor interviewing another expert or guest speaker
• Instructor delivering lecture in another setting related to the course (though not always), for example an ecologist giving lecture at the beach, an art historian in a museum, etc.
• Panel Discussion of experts on specific course-related topic

Which format to use? The primary factor that determines which format to use are the objectives of the MOOC or course, and the course content. The course design team typically selects the video formats during the course design phase when the instructional strategy is created, for example: the formats of the video are chosen, the content chosen for each, related student activities or assessments selected, etc.

The second factor determining which format to employ is the amount of resources (dollars) available for video production. This determines right off the bat which tool, program or hardware will be used for the video production. Important to note, the amount of resources invested in video production does not scale to how much students’ learn or to MOOC completion rates. For example, I completed a course on Canvas Network, Statistics in Education for Mere Mortals (my course review here). The course featured video lectures and tutorials, all created by the instructor using low-budget technology. Lectures appeared to be filmed on the instructor’s laptop using a web cam, (power point slides were added, so there was some editing). Each module featured a tutorial, a screen cast where the instructor demonstrated application of various formulas to a data set. I found the professor, Lloyd Rieber, encouraging and personable; he also delivered the content concisely in lecture videos and tutorials. Interestingly, the course completion rate was over 10%, higher than typical MOOC completion rates that are usually lower than 7%.

Key Findings of Study

• Shorter videos are more engaging. Student engagement levels drop sharply after 6 minutes
• Engagement patterns differ between the two video formats; engagement higher with the lecture style videos (‘talking head’) which researchers suggest is due to more “intimate and personal feel”
• Several MOOC instructors interviewed for study felt more comfortable with the classroom lecture format, however this format did not translate well online, even with much editing in production studio
• For tutorial/demonstrations videos, the Khan-style format where instructor draws on tablet and narrates, was found to engage students more effectively than screen casts. A contributing factor—instructors ability to situate themselves “on the same level” as student
• Video producers and edX design teams determined that pre-production planning had the largest impact on the engagement effect of the videos. Researchers used a data set within the study to test this idea

Practical Recommendations for Course Design Teams

1. Identify type and format for each video lecture using course objectives and module breakdown as a guide, and budget. Plan each lecture for the MOOC format and its potential students. Consider copyright terms for images used in videos and slides. Plan ahead by selecting appropriate images, free from copyright during the planning phase

2. Invest in pre-production planning phase. Segment course content into chunks, using six-minutes per video as a guideline. Identify purpose for each video lecture, and key content points to deliver within each.  Write script for each [lecture video format] and have instructor practice before filming—reduces filming and editing time

3. For tutorial/demonstration videos introduce motion and continuous visual flow into tutorials, along with extemporaneous speaking so that students can follow along with the instructor’s thought process. Complete basic outline of video beforehand, not full script to be read word-for-word

4. Provide more personal feel to videos. Try filming in an informal setting (such as the instructor’s  office) where he or she can make good eye contact—it often costs less and might be more effective than a professional studio. Coach instructors to use humour, personal stories and convey enthusiasm where possible

Closing Thoughts
MOOCs are here to stay, which makes studies like this one valuable for helping educators be more effective through course design. This study brings us closer to finding the answer to the question which kinds of videos lead to the best student learning outcomes in a MOOC?  Yet it’s a start, there is still much more to be done in understanding how students learn in massive courses, and how institutions can be more effective with investment of its resources for increasing student learning outcomes.

Further Reading:

• Guo, P., Kim J., & R. Rubin, How Video Production Affects Student Engagement: An Empirical Study of MOOC Videos, 2014. http://pgbovine.net/publications/
• Hibbert, M., What Makes an Online Instructional Video Compelling, 2014. EDUCAUSE
• Optimal Video Length for Student Engagement, 2013. edX Blog
• How to Make a Khan Academy Video, Khan Academy