How ‘Good’ is Your Online Course? Five Steps to Assess Course Quality

The view that online education is “just as good as” face-to-face instruction was not widely held in 2003: 42.8% of chief academic officers reported that they considered the learning outcomes for online instruction to be inferior to face-to-face instruction. The view of online quality has improved over time. However results for 2013 revealed a partial retreat in faculty perceptions of online learning providing quality learning experiences. The 2014 results indicate that the retreat continues—there’s an increase in faculty that perceive online education as inferior. — Grade Level: Tracking Online Education in the United States, 2015

quality-controlOne of the main criticisms of online courses is they are of poor quality as revealed in the annual Babson study mentioned in the opening. Positive perception of online learning by faculty has declined in 2013 and 2014 (Allen & Seaman, 2015). Face-to-face courses appear to be the hallmark for quality when it comes to higher education. Yet this doesn’t seem fitting considering the ongoing and often heated public dialogue about the quality of higher education programs with little consensus on what quality is. In this blog post I suggest that online educators can and should tackle the quality issue in their own courses, and that they do so by assessing their course holistically. A holistic approach encompasses elements such as students’ perspectives, results over a period of time, artifacts created during learning, and the instructor’s course experience.

I also review recent research on quality assessment specific to online courses. I also examine existing frameworks and rubrics for online course assessment and explain why, even if an institution follows such standards, these are starting points. I outline five-steps that instructors can follow to assess whether a course is ‘good’—an assessment for quality that considers foundational elements, student perspectives, course artifacts, student and instructor learning experiences.

What is Course Quality?
Up until a few years ago ‘quality’ in higher education was measured by a course’s content, pedagogy and learning outcomes (Bremer, 2012). This approach has changed to a process-oriented system where a combination of activities contributing to the education experience are considered. Activities that include: student needs, use of data and information for decision-making, department contributions, as well as improved learning outcomes (Thair, Garnett, & King, 2006). This holistic approach of evaluating education experiences is often applied to the development and assessment of online learning. For example, Online Learning Consortium’s Five Pillars of Quality Online Education (below) and Quality Matters (QM) rubric.

Screen Shot 2015-05-25 at 4.56.37 PM
“Five Pillars of Quality Online Education, the building blocks which provide the support for successful online learning”.  http://www.onlinelearningconsortium.org

Why Assessing Quality is Difficult in Online Education
Yet there are challenges associated with setting universal quality standards for online education, and though a starting point, a thorough quality assessment requires ongoing consideration of numerous elements, some that occur over a period of time.  Key challenges with assessing quality through set standards are outlined in ‘What is online course quality‘ and include: 1) the lack of authoritative body (able and willing) to address minimum level of standards across all states with their accrediting bodies, 2) the challenge of creating a comprehensive, evaluative tool to address complexities of online courses, and 3) the implementation process itself given the significant resources that would be required to implement an institution-wide evaluation process (Thompson, n.d.).

Limitations of Quality Assessments
There are other limitations. Some assessments are inherently limiting with a prescriptive set of standards that may not fit all contexts.  Another is the tendency to establish a minimum level of quality, ‘baseline standards’ which limits innovation and creativity (Misut & Pribilova, 2015).  Most course assessments are done at a point-in-time and are unable to capture dimensions over the life of a course and post-course; dimensions that include student perceptions collected as formative feedback (mid-way through course) and end-of-course feedback surveys. Furthermore, quality assessments frequently focus on course/instructional design and fail to include learning experiences of the instructor and students.

What’s involved In a Good Course Assessment?
A holistic assessment goes beyond course design; it acknowledges the nuances that make a course unique, including input and contributions from students, developments in the field of study, and current events. Most valuable are students perceptions of their learning and of the course experience. A good course assessment considers the course over a period of time, and considers interactions between instructor and students, students and students, all of which create artifacts that can be studied and analyzed (Thompson, 2005).  Artifacts might include, emails or forum posts of student questions,  dialogue within forums, feedback from group interaction, end-of-course student surveys, LMS reports on student interaction patterns, student assignment results, and more.  Course artifacts give valuable clues to a course’s quality, more so when collected from two or more course iterations and analyzed collectively.

Screen Shot 2015-05-25 at 2.55.05 PM
Figure from paper describing the Online Course Criticism model based on the concept of educational criticism which suggests a holistic review of a course to assess quality (Thompson, 2005)

Other elements to consider:

  Student behaviours including questions asked in forums, emails, interactive patterns within LMS, interaction with resources, participation patterns within discussion forums,  social platforms designated to course, etc.   Student perceptions evaluated through questionnaires, formative course feedback, post-course questionnaires, one-on-one interactions  ♦  Knowledge creation/transfer by students evaluated through assignment analysis, course artifacts, post-course surveys  ♦  Course design as per rubric/assessment tool    Use of current technology tools and platforms    Course data and artifacts from two or more sessions analyzed and compared  ♦  Quantity and type of interaction between students and instructor

Five-Steps to Assessing Online Course Quality

1) Asses Using a Rubric or Other Tool to Consider Basic Course Elements
Assess course using the tool or framework employed by your institution e.g. Quality Matters rubric. If your institution does not have a tool in place I recommend the rubric created by California State University Chico which covers six domains. The rubric (embedded below) is free to use and download under the Creative Commons Attribution 3.0 United States License.

* Thanks to a reader’s comment – there is an updated version of the Chico rubric which is a checklist format with additional dimensions. It is similar to the Quality Matters rubric. I prefer the version embedded here — its more approachable given it’s less lengthy and rigid. Link here to the updated version.

2) Analyze Course from a Student Perspective
This is perhaps the most difficult yet useful element for improving course quality. There are a variety of ways to consider students’ perspectives, several already mentioned. Other recommendations—take an online course as a student (e.g. a MOOC) in a topic you aren’t familiar with. This provides an eye-opening view of how it feels to be an online student. Another method is to ask a colleague from another department to review your course and provide constructive feedback.

3) Assess Course Artifacts, Materials, & Feedback
Another useful exercise is analyzing course artifacts. Analyzing results from student feedback via a questionnaire midway through course is helpful. If a course is offered more than once, compare data from course iterations collectively.  Consider, is student feedback incorporated into subsequent course re-runs? What about student-generated content? All artifacts and materials associated with a course are valuable material for assessing a course’s quality.

4) Consider Level and Type of Student-to-Student and Student-to-Instructor Interactions
Interaction is critical to an online course; students that feel connected, establish themselves as individuals within an online course are likely to have higher levels of motivation and learning satisfaction over those that don’t. Consider the forums, the interactive assignments where students can participate, the social exchanges within course-associated platforms, and other places for interaction. An example of assignments that encourage student feedback and involvement, leading to high levels of engagement can be found on this online instructor’s (Laura Gibbs) course site here. Also consider the Community of Inquiry model for the types of interactions in an online course that lead to positive learning experiences.

5)  Results: Are Students Learning?
Evidence of learning  is the most important assessment dimension, yet nearly impossible for a standardized quality assessment tool to evaluate.  One could argue that before and after quizzes within a course can evaluate learning. I suggest that the instructor is able to assess at a deeper level whether or not learning occurred, can determine the level of critical thinking. This can be done only when assignments demand that students demonstrate what they know and are required to apply course concepts.  Assignments that draw out students thinking by demonstration either through dialogue or written work allow the instructor evaluate learning effectively. There’s no formula for this fifth step, this is an example of customized course evaluation. But I suggest instructors evaluate student artifacts from one course to another and to consider what students learned and how well  they articulated what they learned. There may be opportunity for revising assignments, activities or other course dimensions.

Conclusion
Assessing quality in online courses is complex as we’ve seen here, yet addressing quality is critical to advance the positive perception of online education for one, but more importantly to provide learning and teaching experiences that are rewarding, rich and meaningful. Quality assessment can start one course at a time, and who better to do this than the course instructor?

References/Resources*

11 thoughts on “How ‘Good’ is Your Online Course? Five Steps to Assess Course Quality

  1. Hi Debbie,

    thanks for this article – it was a very useful resource I was glad to have found when I was doing research for an article about online course quality. Online courses (on Udemy, Skillshare etc.) are often created by people with zero background in education or eLearning, and getting them to take course quality seriously is somewhat challenging. I’m trying to break things down for online instructors and provide actionable step-by-step advice. Academic models don’t really apply 100% since the environment (“anybody can teach”) is very different, but they’re still a great basis to build upon.

    Thanks again,
    Gerfried

    Liked by 1 person

  2. Hi! Great post. So I am in the proccess of designing an evaluation for a blended and online course. The Chico rubric, can be embded into a Drive Form to use as an online survey? Each item should be used like an option, kind of a Likert scale?
    Thanks!
    Fede

    Like

  3. Hello Debbie – Great summary! The instructor self assessment is an interesting idea- in my experience instructors with many years of teaching experience are very confident that the teaching methods they use are the most effective. My work primarily focuses on instructors who offer On-Demand online courses which makes it a big commitment for instructors to have to go back and reshoot lectures.

    What would you say would be an ideal time for an instructor to go back and do a self assessment? Every time they receive constructive feedback? Few times a year?

    Best,
    DQ

    Like

    1. Hi David, With on-demand courses I recommend the instructor revisit is his or her course on a regular basis – two or three times per year. But during this time the instructor should be collecting feedback, data on the course and other resources related to the course to review collectively for the review time. It’s a dynamic, ongoing process. Debbie

      Like

  4. Hi Debbie. Very insightful article, thank you! We’re currently in the process of revamping our “Meta-Course” on our website, so these tips were very helpful. Cheers!

    Like

  5. I’ve been looking into different quality rubrics, and have used the ROI in the past. When I went to their site recently, under “Applying the Rubric”, they mentioned a new instrument – QOLT (http://courseredesign.csuprojects.org/wp/qolt/). It looks like it is informed by QM as well as the ROI, do you have any thoughts on it or have you used it?

    Like

    1. Hi Stephanie. Thanks for mentioning this. Yes you are right, this new instrument QOLT,(described here) http://courseredesign.csuprojects.org/wp/qolt/, looks very similar to the Quality Matters format. Here is a link for interested readers in the QOLT rubric https://drive.google.com/file/d/0BxN4M6qCVbDPOEl0d1dKWmFXOEk/view
      Upon review, I actually prefer the original rubric of Chico State’s, the one embedded in the post. Though the newer version is comprehensive, thorough, and includes some good resources, this version is more of a checklist thus lends itself to be used exclusively for narrowly-focused assessment, whereas the original rubric format is more approachable, less pages, attractive format, can be used during course development as well as for self-assessment. That being said, both are good tools to use to assess and plan for the foundational elements that need to be part of an online course.

      Thanks for commenting!

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s