How ‘Good’ is Your Online Course? Five Steps to Assess Course Quality

The view that online education is “just as good as” face-to-face instruction was not widely held in 2003: 42.8% of chief academic officers reported that they considered the learning outcomes for online instruction to be inferior to face-to-face instruction. The view of online quality has improved over time. However results for 2013 revealed a partial retreat in faculty perceptions of online learning providing quality learning experiences. The 2014 results indicate that the retreat continues—there’s an increase in faculty that perceive online education as inferior. — Grade Level: Tracking Online Education in the United States, 2015

quality-controlOne of the main criticisms of online courses is they are of poor quality as revealed in the annual Babson study mentioned in the opening. Positive perception of online learning by faculty has declined in 2013 and 2014 (Allen & Seaman, 2015). Face-to-face courses appear to be the hallmark for quality when it comes to higher education. Yet this doesn’t seem fitting considering the ongoing and often heated public dialogue about the quality of higher education programs with little consensus on what quality is. In this blog post I suggest that online educators can and should tackle the quality issue in their own courses, and that they do so by assessing their course holistically. A holistic approach encompasses elements such as students’ perspectives, results over a period of time, artifacts created during learning, and the instructor’s course experience.

I also review recent research on quality assessment specific to online courses. I also examine existing frameworks and rubrics for online course assessment and explain why, even if an institution follows such standards, these are starting points. I outline five-steps that instructors can follow to assess whether a course is ‘good’—an assessment for quality that considers foundational elements, student perspectives, course artifacts, student and instructor learning experiences.

What is Course Quality?
Up until a few years ago ‘quality’ in higher education was measured by a course’s content, pedagogy and learning outcomes (Bremer, 2012). This approach has changed to a process-oriented system where a combination of activities contributing to the education experience are considered. Activities that include: student needs, use of data and information for decision-making, department contributions, as well as improved learning outcomes (Thair, Garnett, & King, 2006). This holistic approach of evaluating education experiences is often applied to the development and assessment of online learning. For example, Online Learning Consortium’s Five Pillars of Quality Online Education (below) and Quality Matters (QM) rubric.

Screen Shot 2015-05-25 at 4.56.37 PM

“Five Pillars of Quality Online Education, the building blocks which provide the support for successful online learning”.

Why Assessing Quality is Difficult in Online Education
Yet there are challenges associated with setting universal quality standards for online education, and though a starting point, a thorough quality assessment requires ongoing consideration of numerous elements, some that occur over a period of time.  Key challenges with assessing quality through set standards are outlined in ‘What is online course quality‘ and include: 1) the lack of authoritative body (able and willing) to address minimum level of standards across all states with their accrediting bodies, 2) the challenge of creating a comprehensive, evaluative tool to address complexities of online courses, and 3) the implementation process itself given the significant resources that would be required to implement an institution-wide evaluation process (Thompson, n.d.).

Limitations of Quality Assessments
There are other limitations. Some assessments are inherently limiting with a prescriptive set of standards that may not fit all contexts.  Another is the tendency to establish a minimum level of quality, ‘baseline standards’ which limits innovation and creativity (Misut & Pribilova, 2015).  Most course assessments are done at a point-in-time and are unable to capture dimensions over the life of a course and post-course; dimensions that include student perceptions collected as formative feedback (mid-way through course) and end-of-course feedback surveys. Furthermore, quality assessments frequently focus on course/instructional design and fail to include learning experiences of the instructor and students.

What’s involved In a Good Course Assessment?
A holistic assessment goes beyond course design; it acknowledges the nuances that make a course unique, including input and contributions from students, developments in the field of study, and current events. Most valuable are students perceptions of their learning and of the course experience. A good course assessment considers the course over a period of time, and considers interactions between instructor and students, students and students, all of which create artifacts that can be studied and analyzed (Thompson, 2005).  Artifacts might include, emails or forum posts of student questions,  dialogue within forums, feedback from group interaction, end-of-course student surveys, LMS reports on student interaction patterns, student assignment results, and more.  Course artifacts give valuable clues to a course’s quality, more so when collected from two or more course iterations and analyzed collectively.

Screen Shot 2015-05-25 at 2.55.05 PM

Figure from paper describing the Online Course Criticism model based on the concept of educational criticism which suggests a holistic review of a course to assess quality (Thompson, 2005)

Other elements to consider:

  Student behaviours including questions asked in forums, emails, interactive patterns within LMS, interaction with resources, participation patterns within discussion forums,  social platforms designated to course, etc.   Student perceptions evaluated through questionnaires, formative course feedback, post-course questionnaires, one-on-one interactions  ♦  Knowledge creation/transfer by students evaluated through assignment analysis, course artifacts, post-course surveys  ♦  Course design as per rubric/assessment tool    Use of current technology tools and platforms    Course data and artifacts from two or more sessions analyzed and compared  ♦  Quantity and type of interaction between students and instructor

Five-Steps to Assessing Online Course Quality

1) Asses Using a Rubric or Other Tool to Consider Basic Course Elements
Assess course using the tool or framework employed by your institution e.g. Quality Matters rubric. If your institution does not have a tool in place I recommend the rubric created by California State University Chico which covers six domains. The rubric (embedded below) is free to use and download under the Creative Commons Attribution 3.0 United States License.

* Thanks to a reader’s comment – there is an updated version of the Chico rubric which is a checklist format with additional dimensions. It is similar to the Quality Matters rubric. I prefer the version embedded here — its more approachable given it’s less lengthy and rigid. Link here to the updated version.

2) Analyze Course from a Student Perspective
This is perhaps the most difficult yet useful element for improving course quality. There are a variety of ways to consider students’ perspectives, several already mentioned. Other recommendations—take an online course as a student (e.g. a MOOC) in a topic you aren’t familiar with. This provides an eye-opening view of how it feels to be an online student. Another method is to ask a colleague from another department to review your course and provide constructive feedback.

3) Assess Course Artifacts, Materials, & Feedback
Another useful exercise is analyzing course artifacts. Analyzing results from student feedback via a questionnaire midway through course is helpful. If a course is offered more than once, compare data from course iterations collectively.  Consider, is student feedback incorporated into subsequent course re-runs? What about student-generated content? All artifacts and materials associated with a course are valuable material for assessing a course’s quality.

4) Consider Level and Type of Student-to-Student and Student-to-Instructor Interactions
Interaction is critical to an online course; students that feel connected, establish themselves as individuals within an online course are likely to have higher levels of motivation and learning satisfaction over those that don’t. Consider the forums, the interactive assignments where students can participate, the social exchanges within course-associated platforms, and other places for interaction. An example of assignments that encourage student feedback and involvement, leading to high levels of engagement can be found on this online instructor’s (Laura Gibbs) course site here. Also consider the Community of Inquiry model for the types of interactions in an online course that lead to positive learning experiences.

5)  Results: Are Students Learning?
Evidence of learning  is the most important assessment dimension, yet nearly impossible for a standardized quality assessment tool to evaluate.  One could argue that before and after quizzes within a course can evaluate learning. I suggest that the instructor is able to assess at a deeper level whether or not learning occurred, can determine the level of critical thinking. This can be done only when assignments demand that students demonstrate what they know and are required to apply course concepts.  Assignments that draw out students thinking by demonstration either through dialogue or written work allow the instructor evaluate learning effectively. There’s no formula for this fifth step, this is an example of customized course evaluation. But I suggest instructors evaluate student artifacts from one course to another and to consider what students learned and how well  they articulated what they learned. There may be opportunity for revising assignments, activities or other course dimensions.

Assessing quality in online courses is complex as we’ve seen here, yet addressing quality is critical to advance the positive perception of online education for one, but more importantly to provide learning and teaching experiences that are rewarding, rich and meaningful. Quality assessment can start one course at a time, and who better to do this than the course instructor?


Does it Take More or Less Time to Facilitate and Develop an Online Course? Finally, Some Answers

How much time does it take to teach an online course? Does teaching online take more or less time than teaching face-to-face? How much time does it take instructors to develop an online course? — Instructor Time Requirements to Develop and Teach Online Courses (Freeman, 2015)

Time business conceptA study released in March of this year set out to answer these burning questions that the majority of online educators would like answers to. There’s considerable anecdotal evidence that favors both sides—it takes more time versus less time to facilitate an online course when using a face-to-face course as benchmark. The purpose of this study was to nail down the facts—to measure the perceptions of and actual time spent developing and teaching online courses. The findings are significant for institutions and educators involved in online education for several reasons. Professional development for one. The report reveals areas where survey-respondents struggled during the course development phase, and where the majority of time was spent when facilitating (the conclusions are surprising). Secondly, results may be helpful for institutions when considering compensation and work-allocation models. Institutions can use the results as benchmark, at the very least the study may act as catalyst for constructive conversations about compensation and support for online course development and facilitation. And finally, it may help online instructors gain insight into their own teaching experiences by considering the experiences of  other educators that have experience with face-to-face and online courses.

This post highlights the findings and suggests factors for educators to consider when it comes to, 1) the time spent developing online versus face-to-face courses, and 2) how much time is invested in online facilitation, and how it compares to face-to-face instructions.

Survey Details
To put the results into context—the survey gathered data from 68 instructors from a total of 165 solicited from three universities across eight academic disciplines. Each respondent had developed an average of 2.13 online courses and had experience teaching an average of 2.03 online courses, and had been teaching at the university level for an average of 14.2 years (Freeman, 2015).

1) Course Development Time: Pedagogical Learning Curve Steepest
Survey results confirmed that developing online courses is indeed more time consuming than developing face-to-face courses. Though the time required declines when the same instructor develops a second or third online course. Twenty-nine percent of respondents indicated they spend over 100 hours (median of 70 hours) to develop their (first) online course. This significant number of hours is likely due to the fact that 59% of respondents developed over 90% of the course without any assistance, which included developing content, assessments, assignments, and time associated with course design. The other 41% received course design support from instructional designer(s) and/or used ready-made content available through textbook publishers. Also significant is the technological learning curve which was found to be shorter than the pedagogical learning curve, in other words instructors required more time to determine how to implement pedagogical methods, how to create learning experiences and deliver content appropriate for the online format than they did learning about the features and nuances of the technology used to deliver the course. The learning curve is described as the time it takes to “get used to” the course elements [platform, tech features] and/or the method of teaching.

Screen Shot 2015-05-08 at 9.42.50 AM

(Freeman, 2015)

Developing a quality online course is complex due to the fact that technology adds yet another layer to course design and one that requires a unique skill set. In addition there is an interdependent relationship between technology and pedagogy specific to online courses—for instance the features of a LMS platform will determine and shape the course and the teaching methods. Using the discussion forum as an example—the flexibility of the forum feature—how easy it is to set up by the course designer for group assignments, and how it can be used by students for a group assignment whether it can facilitate the communication and collaboration that is required for the assignment will dictate how effectively the ‘method’ is executed in the course.

Online course design requires a breadth of skills that includes technical knowledge, not only familiarity with LMS features, but also outside tools including social media platforms that can enhance student learning.  Knowledge of user-focused design, or web design principles is also critical in delivering an intuitive, learning experience for students (How Five Web Design Principles Can Boost Student Learning). Second are the pedagogical methods, in other words how learning is sequenced, framed and presented to students.  This array of skills required is far beyond the scope of most faculty, who are experts in their field of study, not necessarily course design. Realistically creating an online course requires at least two or more individuals with specific skills sets working together to develop an engaging, intuitive and quality learning experience.

The onus is on institutions to provide not only professional development for faculty in course design principles and strategies, but to provide support in the technical and pedagogical aspects.

2) BIG Time Commitment Facilitating First Online Course — Levels Off After 2nd Time, But Grading Involves More Time Investment
Though respondents in the survey originally perceived that teaching online took more time than teaching face-to-face, by the third time facilitating respondents reported that it took them about the same amount of time as it did a similar face-to-face course.

There is supporting evidence to the earlier finding that teaching an online course the second and third time becomes about as time-consuming as teaching a face-to-face course the second and third time.  The factors that still remain more time-consuming for online teaching compared with face-to-face teaching, even after teaching the course three times, are Instructor-Student Interaction and Grading & Assessment, the two specific factors  that can not be prepared in advance for online courses (unlike Content Development and Pre-Semester Setup).

Sixty-nine percent of survey respondents indicated that it took ‘much more’ and ‘more’ time to facilitate an online class for the first time. Yet by the third time, it dropped to 25% in this same categories (table 4 below), which does support the learning curve theory. These findings suggest that acknowledging that more of the instructor’s time will be required the first and even the second time facilitating a course, is important for both the instructor and the institution. Though it does also suggest that professional development is needed for instructors—development focused on facilitation skills that will support skills specific to the uniqueness of online instruction. Such training can potentially reduce the learning curve for instructors, as well as reinforce the building of effective skills, best practices, and efficient use of time.

Screen Shot 2015-05-08 at 9.55.10 AM

Annotated screenshot that shows two-thirds of  respondents by the third time facilitating online indicate that it took ‘somewhat more’, ‘more’ or ‘much more’ time to grade and assess students in an online course than face-to-face (Freeman, 2015).

A startling (and significant) finding of this study is the time dedicated to grading and assessing online students. It appears that the time dedicated to grading students’ work actually increased from the first to third time of facilitating an online course (table 4). Two-thirds of the respondents indicated by the third time it took ‘somewhat more’, ‘more’ or ‘much more’ time to grade and assess students in an online course than face-to-face.  I find these results encouraging since an instructor’s feedback of students’ work is a critical component that can motivate students, deepen their knowledge and push them to think critically (Getzlaf et al., 2009). Implications are that skill development in this area are needed and will benefit not only students but can help instructors to provide feedback more efficiently. There are several technology tools and applications that can help instructors achieve efficiency and to make the most of giving feedback using online tools that deliver meaningful, quality feedback for students (Morrison, 2014). Again professional development is needed in the area of grading and assessment to support instructors in their efforts.

By no means is this study the definitive answer on the time requirements for developing and facilitating online courses, but it is an excellent starting point for conversations about ‘time’ needed to create quality online learning experiences.


How Five Web Design Principles Boost Student Learning in an Online Course

“Our team realized quickly that we needed to do a better job cross referencing material on our course site. For example if we mention syllabus, we must link to it. Some students we have learned want a great deal of guidance” MOOC instructor, Karen Head (2013)

great-designIn the quote above, without realizing it, the instructor was referring to the concept of ‘user experience’. And it’s not guidance students wanted so much as an intuitive learning experience. Creating a user-friendly course site begins with incorporating web design principles. Even the most basic of principles customized to online course design reduces barriers associated with virtual learning by minimizing distractions, highlighting concepts and making resources readily accessible. Embedding a link into the phrase ‘assignment guidelines’ for instance, when the assignment is referred to within a course page, is an example of making resources readily available (if the assignment guidelines are within the syllabus, refer students to the page number). This reduces the amount of time students spend searching and frees up time for learning.

The challenge of designing online courses is not only pedagogical, but also technical, which is the category that ‘usability’ falls under. We are at the point with online learning where pedagogy and technology are interdependent; where a well-designed, user-friendly course with a clear learning path needs to adhere to technical principles as well as pedagogical ones. Technology is a new form of pedagogy. The course site design, how content is presented, is an aspect of online pedagogy. In this post I cover five principles of web design that are essential to online course design.

Retail sites frequently adhere to best practices for web design given customers (users) are more likely to spend time and money on an attractive, intuitive website. I suggest educators use similar web design principles to support their students.

Before we examine the principles, defining user experience (UX) is in order. There are numerous definitions of user experience but the one below specific to web design, incorporates key elements of the entire experience:

“User experience (UX) is about how a person feels about using a system. User experience highlights the experiential, affective, meaningful and valuable aspects of human-computer interaction (HCI) …. It also covers a person’s perceptions of the practical aspects such as utility, ease of use and efficiency of the system. User experience is subjective in nature, because it is about an individual’s performance, feelings and thoughts about the system. User experience is dynamic, because it changes over time as the circumstances change…” All About UX

Five Principles of Web Design Applicable to Online Course Design

1. Design for the user
This seems obvious—design a course from the student’s perspective, yet it’s an atypical approach. When designing a web page for a course site, always ask ‘how will this look to the student’? Anyone involved in online course design needs to take an online course as a student. Completing at least one week of course work in a MOOC for example, gives one an entirely different perspective on course design—guaranteed. Design the course from the student’s perspective—always.

2. Consistency
For online course, consistency is probably the most under-utilized principle. Specifically in terms of how resources are titled, labeled and/or placed within the course site. I’ve taken many courses where the same resource, an article for instance, is referred to by two different names—in the syllabus it’s titled one way, and in the course site another. Confusing. Same goes for assignments, calling an assignment by slightly different names, even by one word suggests there are two assignments, not one. Another, posting the same document in two different locations within the site suggests there are two different documents. The time students spend searching, checking, comparing etc. is valuable learning time that is spent on logistics. Consistency is key.

3. White Space
Effective use of white space emphasizes key concepts, improves comprehension (up to 20%) and reduces cognitive overload (Lin, 2004). White space is the part of a web page that is left blank or unmarked. It’s the (white) space between columns, text, images, and margins on the page. This space provides visual relief to the reader and improves readability. Avoid using big blocks of text. Break it up with a graphic, or block of white space or increased line space. See examples below.


Example of text with little white space.


Same text as above but with increased line spacing. Effective use of white space improves readability.

4. Simplicity
In keeping with the idea of white space is simplicity. A cluttered page with three or more colors of font, sizes of font and images placed sporadically throughout that are of different type and size creates a chaotic-looking virtual classroom. It’s far easier to study and focus on learning in a physical classroom that is organized with minimal distractions. The same goes for an online classroom. Keep it simple, two colors of font, same size and style throughout, organized and consistent pages creates a Zen-like classroom where students can focus on course content and application of concepts. Learning is enhanced greatly.

“The way information is organized and presented to students affects not only the usability of information, but the usability of the course itself” (Young, 2014)

Screen Shot 2015-04-06 at 9.25.36 AM

Tabs grouped by category (Coursera)

5. Use Tabs Effectively
Imagine opening a file drawer that is full of file folders with inaccurate or missing labels. The same principles of file labels apply to web sites except rather than alphabetized listing, it’s an order that makes sense to the student. For example the ‘start here’ tab should be at the top of the menu not third or fourth down the list (which happens more often than you would think). Tabs should be two words (max 3 words), and with descriptive language, ‘Start Here’, ‘Week One, or ‘Student Support’ for example.  Use sub-tabs if possible, and if not group tabs into categories (screenshot right). Also avoid CAP LETTERS for titles of tabs. CAPITAL LETTERS can appear loud and abrasive on a website (there are exceptions as the screenshot above right demonstrates).

Developing an online course is a multidimensional process. Usability is one dimension often neglected; understandably so given that most educators approach online course design with little expertise in web design. Yet a little goes a long way—by implementing just the basic web design principles, educators can create an intuitive learning path that gives students the boost they need to invest more time in learning, not searching.


How (Not) to Design a MOOC: Course Design Scenarios From Four xMOOCs

designThis post examines four MOOCs completed as a student then de-briefed from a course design perspective—I share insights into what worked and what didn’t for the purpose of helping educators create better online learning experiences.

I recently completed two MOOCs on the edX platform that are part of a mini-series on education policy. The courses are great examples of how higher education institutions misuse the MOOC format by using traditional teaching methods that end up falling flat. I debrief the two MOOCs from a course design perspective and share why they were sub par, uninspiring. I also describe two other MOOCs that provided exemplary learning experiences. The two pairs of MOOCs provide instructive examples of contrasting course design approaches.

This post follows “How to Make Bad Discussion Questions Better: Using a Case Study of an edX MOOC” the first MOOC of the mini-series “Saving Schools: History, Politics, and Policy in U.S. Education”. I used actual discussion questions from this MOOC’s forums as examples of how not to write questions to foster student discussion. I rewrote the questions, providing better and best formats that would be more likely to encourage meaningful dialogue.

The second edX MOOC, “Saving Schools: History, Politics and Policy in U.S. Education: Teacher Policy” wrapped up this week (December 4). Both MOOCs followed an identical course structure that included: recorded video lectures that relied on the interview format featuring one (sometimes two) faculty member(s), two assigned readings per week (from the same source), one discussion question each week, and a final exam. This format is typical of xMOOCs; one that tries to mimic the in-class experience.

Screen Shot 2014-12-06 at 11.36.00 AM
Click to enlarge. Screen shot of instructions for the final assignment, a digital artifact, in E-learning and Digital Cultures. At the end of this post my Digital Artifact created for the course assignment

Exemplary MOOCs
The other two MOOCs used a non-traditional design approach. They took advantage of what the MOOC format could offer by acknowledging its uniqueness and providing content from a variety of sources outside the MOOC platform. They also utilized a range of assessment methods, and included social media that encouraged interaction. Both MOOCs, Introduction to Sociology and E-learning and Digital Cultures (from Coursera), inspired and promoted thought. The learner was a viewed as a contributor, not a recipient.

Screen Shot 2014-12-06 at 11.54.46 AM

Introduction video of Professor Duneier introducing his course on Coursera (2012). Duneier pulled the course from Coursera after concerns over licensing his course for other institutions use.

E-learning and Digital Cultures featured YouTube videos not lecture videos to demonstrate course concepts, along with articles, mostly from academic journals. The learning experience closely resembled a cMOOC experience (the original MOOC format developed by Downes and Siemens)—one that leverages sources on the web, shares student blogs and views students as contributors. Introduction to Sociology featured two video formats; one featuring Professor Duneier, not lecturing, but sitting in an armchair (above) talking, sharing course-related experiences. He acknowledged learners (some by name) and encouraged student interactivity. The other was live (and recorded) using Google’s Hangout platform with eight students and Duneier leading a seminar discussion.

Course Design Shortcomings of the edX MOOCs
The purpose of the following discussion about the edX MOOCs is not to criticize the course designers or faculty, but to consider the MOOCs as learning opportunities. Doing so aligns with one of the goals of edX, to use the platform to advance teaching and learning.

Learning/instructional methods: The MOOCs relied upon mostly traditional methods of instruction—lectures, multiple choice assessments. Content was instructor-centered, limited to lectures (featuring faculty member), textbook readings (from a book written by same faculty member), and articles from one source, Education Next, of which the same faculty member is editor-in-chief.

  • The edX MOOCs would benefit from inclusion of open resources, with links to outside sources showing various perspectives as well as social media platforms where students could engage live with content experts or static content. Also to share content sources, and/or their own content creations (blog posts, etc.)
  • Learning was confined to a virtually, walled classroom—inside the MOOC platform.
Target objectives

MOOCs that provide a focus and structure for students by including goals or focus questions, allow students to shape and customize their own learning accordingly

Course Objectives: There were no learning goals outlined for the MOOCs. There didn’t appear to be a focus for each week, or guiding questions to provide structure. Granted, learners should create their own learning objectives when working within a MOOC, though a stated focus or general goals for the course allows learners to establish and shape their own learning goals. E-learning and Digital Cultures provided an overview of the course which outlined the focus for each unit of study, and each week included focus questions to consider. 

Rigor: Course rigor was low. Disappointing given the institution behind the MOOC was Harvard. It’s worth noting at edX’s launch in 2012, the Provost of MIT at the time L. Rafael Reif emphasized the rigor and quality of courses on ex’s platform ”(edX courses need) not to be considered MIT Lite or Harvard Lite. It’s the same content” (MIT News).  Yet the discussion questions as outlined in my first post, the biased readings, lectures, the application activities for students did not add up to a rigorous learning experience that encouraged critical thinking. Several factors may have contributed. Suffice to say that the course design team would have benefited from someone with a high-level of expertise in effective course design principles, knowledge of learning theories and instructional methods.

Content: As mentioned the majority of the content was limited to the faculty member in the lectures, two or three chapters of a book authored by the same faculty member, and essays from the one source.

  • Biased resources did not contribute to learner’s considering multiple perspectives. Though in the second MOOC there was an effort by course facilitators to incorporate other perspectives in the discussion forums.
  • Lecture videos were long — typically 12 to 15 minutes. Research on MOOC videos suggest ideal length is 4 to 6 minutes (Guo, 2013).
  • Repetitive Content. Content from the readings were also included in the lecture, and frequently two interviews in the same lecture covered the same content.
  • Delivery methods of content were repetitive, uninspiring.
  • Content came across as telling, not interactive.

Application activities: There were few activities for learners to engage in except for discussion forums. Unfortunately the questions in the first MOOC did not encourage robust discussion, though they improved in the second course. There were two or three multiple choice questions after each video. Several questions could be considered common knowledge. I could have answered the majority of them without watching the videos.

Screen Shot 2014-09-30 at 2.20.48 PM

Screen shot of a forum discussion question from the MOOC “Saving Schools: History, Politics and Policy in U.S. Education”. A close-ended question, and one not likely to stimulate thoughtful discussion. In my previous post, “How to Make Bad Discussion Questions Better” I provide examples for more effective questions formats.

The pairs of MOOCs illustrate how varied approaches to MOOC course design significantly impacts engagement levels, perceptions and learning outcomes. The edX MOOCs examined here, typical of the majority of MOOCs, relied upon learning methods that failed to leverage the benefits of an open platform, failed to view as students as knowledge sources and contributors. Over time the MOOC format will no doubt settle into something quite different from what we’re experiencing now. A format that will find it’s purpose, engage learners and build bodies of knowledge that benefit all.

Further Reading:

How to Make Bad Discussion Questions Better for an Online Course: Case Study Using an edX MOOC

I’m enrolled as a student in the MOOC Saving Schools Mini-Course 1: History and Politics of U.S. Education on the edX platform and share in this post discussion questions used for assignment purposes from the course to illustrate what NOT to do when it comes to writing discussion questions. I use these for illustration purposes to help readers who teach online to further improve their own courses.  I share three questions from the MOOC, describe why they don’t work and include revised questions in better and best categories. I’ve incorporated guidelines from “Best Practices for Designing and Assessing Online Discussion Questions”, a webinar hosted by the Online Learning Consortium along with my experience.  In a later post I’ll review the entire course from an instructional design viewpoint, highlighting what worked and didn’t. 

First we examine (briefly) why discussions can be an effective method for learning, and second explore how to write better and best questions by looking at examples of not-so-good questions.

Discussion Questions – Two Layers of Pedagogy
It’s not uncommon for educators to believe that discussion forums are used primarily as a method to encourage student interaction. This is only partly true. There are two layers of pedagogy to the discussion method.

First layer:  Good discussion questions prompt students to evaluate course content, reflect, construct knowledge and articulate understanding through a written response. Scenarios or case studies are also effective where students are required to develop and provide a written solution demonstrating application of course concepts. Student responses are (typically) crafted individually then shared in a forum. This method encourages student to construct meaning and build knowledge by engaging with course materials, reflecting then applying concepts through written explanation.

Second layer:  Dialogue is more meaningful when students have a solid grasp of the concepts accomplished in the first layer.  Students continue to evaluate course concepts, construct knowledge, but also develop alternative perspectives, even critical thinking by engaging in discourse with classmates.  Students are exposed to others’ perspectives in this phase, ask questions, defend their own positions, evaluate alternative positions, challenge others’ positions, construct new knowledge and further develop communication skills. This is the ideal scenario. In reality what’s described will not always happen, but different levels of learning will occur depending upon the student’s motivation, confidence and trust level.

Guidelines for Developing Discussion Questions:
Discussion questions should closely align with course concepts and objectives. Below are guidelines to consider with developing questions for an online forum.

  • Frame the question as open-ended. Begin questions with how, what or why
  • Create questions that will elicit more than one answer or solution
  • Ask students to provide support for their response with examples/references, e.g. personal experience, course materials or outside sources.
  • Create questions that encourage students to voice their opinion, perspective or personal experience
  • Make specific reference to theories, diagrams, authors, and/or page numbers
  • Use words such as ‘describe’ or ‘explain’ to elicit deeper responses.
  • Review and consider the course/module objectives —ask  ‘does this discussion question support the course/module objective or focus?’ Students dislike busy work— discussion questions without a focus and purpose lead to shallow responses

Consider the above guidelines as you read the discussion questions below from the MOOC on edX. I realize that this is not perfect as the questions are out of context given you don’t have full access to the course. However the aim is to provide readers with ideas and tips for online discussion forums. Blue text highlights content from the MOOC. Following I explain why the question is ineffective—bad.  The rewritten questions follow the blue text in better and best categories.

Question One: The Challenge

Read “U.S. Students from Educated Families Lag in International Tests” by Eric A. Hanushek, Paul E. Peterson, and Ludger Woessmann from the Fall 2014 issue of EducationNext: “U.S. Students from Educated Families Lag in International Tests”

Based on the results of this study and the lecture video, are you surprised to discover that the United States has two large gaps in its education — the gap within the country and the gap internationally?

Why it’s a Bad Question: The question is closed, “are you surprised?” •  It encourages no further dialogue or application/exploration of course content • Research suggests majority of MOOC students hold at least an undergraduate degree, this question would not be a surprise to MOOC students and could suggest to MOOC learners the course is shallow, superficial

Better:  “Consider the International PISA results presented in the article and discussed in the lecture video. Identify two or more reasons using course content and/or outside sources that could account for the United States performance. Explain.”

Best:  “1) Describe the impact of PISA scores on education policy in the United States. Identify one education policy designed to raise student performance. Describe the intended outcome(s). 2) Do you agree with the policy, why or why not?”   Note: The course materials would need to provide background information, including primary sources. The questions could be more focused by providing a time range for policy, or even identifying a list of policies.

Question Two: School Boards

Read Lost at Sea by Lisa Graham Keegan and Chester E. Finn, Jr. and Steering a True Course by Sarah C. Glover, both from the Summer 2004 issue of Education Next: Lost at Sea   and  Steering a True Course.  Based on these readings and the lecture videos for this week,  discuss what you feel the role of the local school board should be in the 21st century.

Why it’s a Bad Question: Using the word ‘feel’ in questions does not encourage students to approach questions from an analytical perspective • The readings and videos in this module show only one (biased) perspective • Models a narrow point-of-view

Better:  “1) After reviewing the primary role of the school board as outlined in the materials (examples below) determine the role the school board has in the district where you reside. If there is not a school board in your district consider one from this list [provide list of 4 or more]. 2)  From your research do you think the school board is effective?  Why or why not.”

Best:  “1) After reading about the role of the school board as outlined and the other materials, and considering the poor performance of several districts within the United States as outlined in______,  do you think school boards should have a role in school districts?  Explain. 2)  What do you consider as a viable solution(s) to districts’ poor performance? Share any resources that may be of interest to other students.”

Question Three: The Progressive Movement

Read “Romancing the Child” by E.D. Hirsch, Jr. from the 2001 issue of Education Next:  “Romancing the Child”.

Based on the lecture videos, do you feel Progressive political reforms have gone too far or do they still have an important place in 21st century education? Based on the required readings (both the above article and the this week’s chapter from Saving Schools), do you feel Progressive philosophy should still play a role in 21st century education?

Why it’s a Bad Question: Questions are closed • First question is leading — “Do you think reforms have gone too far“… (it’s better not to include options) •  There is little content or resources that describes the principles of progressive education or its characteristics thus (some) students won’t have the background to respond adequately  • The content is biased and suggests that progressive reform is ‘bad’ • The essay is overtly critical of the progressive education movement, which is fine if there were resources provided to portray additional perspectives

I would eliminate this question altogether. The question lacks purpose and focus.

Alternative question:  What are examples of education reforms put forth by John Dewey in the progressive era that are evident in policy of US public schools today? Discuss.  2) Are these policies still applicable to learners’ needs? Why or why not?”

 Further Resources:

Even in Education Everything Old is New Again

“There are more people In the world than ever before, and a far greater part of them want an education. The demand cannot be met simply by building more schools and training more teachers. Education must become more efficient. To this end curricula must be revised and simplified, and textbooks and classroom techniques improved.”  (Skinner, 1958)

what's old is new again sign

It’s been over fifty years since the article “Teaching Machines” appeared in the Science Journal from which the opening quote was excerpted. Author B.F. Skinner would be pleased to read some of the headlines in the education sector this week, one in particular “College in a Box” (Kahn, 2014) which describes how textbook publishers such as Pearson have developed enhanced textbooks and put them on their online platforms such as MyMathLab. These enhanced ‘books’ feature interactive quizzes, tutorials, immediate feedback, and tutorial videos based upon students’ responses. Pearson’s new spin on the old textbook would likely meet Skinner’s definition of efficiency. Coincidently, the instructional method used for Pearson’s textbook programs is programmed instruction; a method Skinner developed and applied with his teaching machine. Skinner’s machine consisted of a program, developed to deliver a self-learning experience for the student that included presenting of content, related questions for students to answers, immediate  and corrective feedback.


Skinner’s teaching machine, a mechanical invention to automate the task of programmed instruction

The expression (also a song) comes to mind, ‘everything old is new again’.  We don’t need to look far into the education sector to find more examples of traditional, which some may consider old, instructional methods repackaged and presented as new. We see this with MOOCs offered through institution-affiliated platforms that feature recorded lectures delivered to students, multiple choice assessments and certificates awarded upon successful completion of institution-established criteria. A new twist on traditional methods.

‘Old’ Instructional Methods
In doing research recently about influential educators and educational psychologists of years past—their philosophies of education, corresponding instructional methods, influence, etc. I see glimpses of these educators’ philosophies in many new methods and education models—with Pearson as described, with Coursera’s model which according to their website uses Benjamin Bloom’s Mastery learning as its pedagogical premise, or essay-grading software that touts immediate (and formative) feedback to students as one of the most useful instructional benefits.

In this post I’ve created a photo post that illustrates the old and new concept with a selection of images on four instructional methods. Included is the corresponding scholar, an image of the method implemented in the past, and today. For description of each image, roll the cursor over the photo; text appears.

The Four Methods Illustrated: Old and New

1) Programmed Instruction: B.F. Skinner (1904—1990)
2) Experiential/occupational learning: John Dewey (1859—1952)
3) Mastery Learning: Benjamin Bloom (1913—1999)
4) Discovery learning: Jerome Bruner (1915 —  )

Closing Thoughts
In discussing the instructional methods in this post in terms of old and new, I’m more making an observation than a heavy hitting point. But, it did come to mind when writing, that if one wants different results, as many seem to want when it comes to education, that they perhaps should be trying different methods, not the same methods slightly repackaged, and then expecting different, (presumably better) results.


Three Actors that Contribute to Student Success in Online Courses: The Institution, Instructor and Student

This post examines three actors that are essential to student success in online courses: 1) the institution, 2) the instructor and, 3) the student.

Screen Shot 2014-08-19 at 10.47.37 AM

Actors Contributing to Student Success in Online Courses

What contributes to student success in a course delivered online? To consider the question from a different perspective one can pose the question this way—who is ultimately responsible when students are not successful—when they fail the course for instance? Is it the student for not having the discipline for online learning? The instructor for not providing support, or the institution for not providing services to support the online student? These are questions worthy of examining at a philosophical level, though in this post I examine select behaviours and strategies associated with the three actors involved in the process of students learning online, 1) the institution, 2) instructor and 3) the learner.

What Contributes to Student Success?
Before examining the three actors roles in the learning process it’s helpful to identify the factors contributing to student success in online environments including the skill set required. It’s also instructive to acknowledge that there is an underlying expectation that students enrolling in online courses are self-directed and capable of managing the tasks associated with online studies. Yet research and feedback from educators reveal something quite different; many students are unprepared to learn online, lack the basic skills, and are not capable of assuming responsibility for their learning. Online course work requires that students use a range of skills including accessing resources, people and content within a network, analytic and synthesis skills to distill relevant information from an abundance of information and resources (Kop, Fournier, & Mak). Though as mentioned, it’s not uncommon to find students lack some, if not many of these skills.

Not only are students often unprepared, but institutions often fail to prepare faculty and instructors for online facilitation. A starting point in boosting student success is identifying the behaviours associated with each of the three actors.

1) The Institution: Student Support Services via the Institution 
One characteristic of institutions offering successful online programs is their ability to support the unique needs of distance students through a student support services function.  As online programs evolve and mature we now have numerous programs to examine and study. Though each unique, there is a common theme—a focus on the students by acknowledging their diverse needs and challenges of studying online. Below are select examples.

Services for online students need to be customized, re-tooled from those provided to traditional students. Services should include technical support, academic advising, online community programs and clubs, library services and career planning.  Some institutions have gone further and developed programs that offer personalized academic support, SUNY Empire State College for example offers a peer tutor program. This program is unique, it’s not a subject matter coaching program, but a mentoring program where the goal is for tutors to help students identify and implement strategies that promote independence, active learning and motivation.


“Creating College Success” from Rio Salado College,  an Award Winning Program

Rio Salado College developed an orientation program “Creating College Success”. It’s a one-credit course delivered fully online. The goal of Rio Salado’s program is similar to that Empire State’s—student self-sufficiency in academic environments.  Penn State World Campus, one of the first universities to deliver online degrees has a comprehensive roster of services for virtual students. One service that all institutions should consider is offering extended hours for technical and academic help via email, phone, or instant messaging.

Western Governors University is one that offers not only academic and technical support, but wellness services through its Well Connect program where students can call a toll-free number any time of day or night for support including personal counseling, legal and debt counseling, new parent transitioning support and more.

2) The Instructor:  Course Design and Instructor Support 
There are two areas that fall under the instructor support: 1) course design, and 2) instructional support.

Course design plays a significant role in students’ potential for learning online, given that students engage with course content, instructor and peers through the course platform. The way in which course content is presented on the course site, the instructions for assignments or activities are written, even the structure and order of the tabs on the course home page (course interface) have an effect on how the students engage with the course, will potentially affect students’ learning. Professor Robin Smith, author of “Conquering the Content: A Step-by-Step Guide to Online Course Design” (2008) describes course design this way:

Design features incorporated in [the] system course development and the learning guide, will create an environment in which students are confident of their pathway, and the only challenge is the course content, not the navigation of the course or figuring out what must be done in order to complete the course…this focus on course design, will free you [instructor] up to spend the semester teaching and interacting with students rather than answering questions about course navigation or specific directions about assignments.” 

The instructor’s role in online courses will vary depending upon the nature of the course, but more importantly instructor behaviours will be a function of the level of students educational background and students’ skill level in the areas mentioned above (collaboration skills, technical, etc). To assess what level students are at when entering the course, ideally the instructor does so through involvement in discussion forums, course introductions, synchronous activities, etc. that allow the instructor to get to know students. Instructors also can do so by reviewing student work early in the course so he or she can provide detailed feedback, challenge the student, suggest external writing support as needed, etc.

The goal is that the instructors focus on challenging students academically in the course via feedback and interaction; individually and as a class. Support for technical, research, or basic academic skills should be provided by the institution, via support services. Institutions should also offer professional development courses, workshops or resources to support online instructors and faculty in course development and instruction.

3) The Student:
The student is ultimately responsible for his or her success in the learning process; it is up to him or her to leverage the resources of the institution and the support of the instructor. There is an effective tool however, a leader readiness questionnaire, that many institutions make available on its website which identifies the skills and tools students will need to be successful with their online studies. Also the concept of giving the responsibility of learning to the students, is another method to encourage success—letting students know they are ultimately responsible.

Below are links to several learner readiness questionnaires provided by various institutions, one is licensed under the creative commons share alike license which makes it available for use to anyone.

In a follow-up post I review tools and resources available on the web that support the development of the skill-set students need for online learning. Readers may also find a previous post, Five-Step Strategy for Student Success with Online Learning helpful— it outlines behaviours associated with successful outcomes for online students.

Supporting student success in online course work begins with the institution—ideally with a strategic plan that includes a system for provision of administrative services, academic counseling, and support specific to online students, as well as professional development and comprehensive resources for faculty and instructors teaching online. Yet to maximize the value of the support offered by the institution and instructor, the learner needs to own the learning, and know the responsibility for success ultimately rests with him or her.