Udacity & San Jose U Halt Online Course Experiment: Lack of Instructional Strategy Undermines Courses

“We want to fail fast, learn from it and move on.”
Mohammad Qayoumi, San Jose State University President (San Jose Mercury News)

sucess-600x398In his statement [above], San Jose State University’s President, Mohammad Qayoumi was referring to the dismal results from his university’s online course experiment with MOOC provider Udacity. The majority of students enrolled in one of the three courses in the pilot project failed; more students flunked the online courses than students in the same face-to-face courses at San Jose University [SJSU].

Qayoumi’s statement is unsettling. Quayomi appears to gloss over the fact the failure applies to students; students in pursuit of higher education. In his statement the university president sounds more like the CEO of a corporation than the leader of a public institution. Yet students are not consumers of a product that may be inconvenienced by a faulty design. In fairness to Qayoumi, his statement may have been taken out of context; he may have elaborated on adverse effects experienced by students. An alternative viewpoint to consider is that Qayoumi appears eager to learn from the mistakes made and move forward.

What is more pressing than Qayoumi’s comment however, is the problems within the three pilot courses as revealed by Thrun in a frank interview with EdSurge [Corcoran, 2013]. The problems were significant and numerous. Several students did not have access to computers to log onto the course site to complete their course work, there were “clerical” errors within the curriculum, poor communication with students, lack of clear expectations and lack of specific deadlines for assignments. It’s likely that the problems were exacerbated with the time crunch; Udacity and SJSU working independently [likely not collaboratively] to meet tight deadlines, resulting in a disjointed curriculum.

This pilot project between Udacity and SJSU was initiated by Governor Jerry Brown last year when Brown contacted Sebastian Thrun co-founder of Udacity, and asked for his help. The CA state governor was looking for ways to improve access to impacted classes for college students in public universities in the state of California. The deal was struck between the two where Udacity would support three classes created by professors at SJSU—Developmental Math (entry-level math, Algebra review), College Algebra and Elementary Statistics. In the remedial math class, only 29% passed, versus 80% for in the face-to-face class, and 44% versus 74% passed in the Algebra course, and 51% students passed versus 74% in the same in-class Statistics course.

What was Missing?  A Course Design Strategy
It is not only disappointing, but surprising at how basic fundamental course design errors were made in the pilot courses; errors [and omissions] which could have been avoided had there been, i) instructional design support, ii) a team approach to course development, and iii) more time allocated to course development. Granted there was a host of issues that the program faced, not to mention the skill deficiencies of the remedial students. The program’s failure is not due to one specific omission or error, however I suggest that poor course design was a significant factor.  I’ll outline why below. By doing so, I hope to provide insight that may be helpful to readers who are currently, or planning to develop their own online courses. Perhaps even institutional leaders considering similar programs to SJSU may find something of value here.

In my experience with online course design, I’ve discovered that in order for online for-credit courses to be successful, a comprehensive instructional plan and strategy are essential. An instructional strategy is the blueprint for the course development process.  In addition to a strategy, a team approach to course design and implementation is required. There are many elements to the online classroom environment that require expertise unique to this educational delivery method. Professors are the subject matter experts, but are not necessarily the experts in web page layout and design for the course site, media selection and development, use of open educational resources, etc. Nor might they have the skills in online class instruction that are required to create a sense of community, presence, and a culture of learning in the online classroom.

Lessons Learned from Udacity & San Jose University. 
Thrun provided candid responses to EdSurge when discussing the program, sharing the downfalls and mistakes made. Readers will appreciate his honesty, and willingness to share and be open about the program’s challenges. This is a positive move; it allows other institutions and educators to learn from their mistakes. Below I present four fundamental principles that course designers and educators should consider when developing their own online for-credit course in light of what was done, or not done within the SJSU/Udacity pilot. After each I include Thrun’s feedback as outlined in the EdSurge interview.

1. Analyze learner and learner contexts. This is the first phase in the process of designing curriculum, yet likely the most neglected. However this step is essential; it shapes the direction of the course, and in some cases reveals that the course should not be developed in the first place as planned. The learner analysis phase involves examining students skill level [technical and educational], cultural background, attitudes and motivations for learning, etc. Another consideration with online learning—how will students access content and instruction?, i.e. what platform will be used [learning context]. Where Udacity/SJSU went wrongthey failed to analyze the learning context – how learners would access the course. And they failed to address the level of skill required for course completion: technical and prerequisite knowledge. As an outsider looking in, it appears that delivering a remedial course to this group of students may not have been the best course of action.

“As the class progressed, Udacity also realized that many of the students simply couldn’t get to a computer regularly enough. For some students, says Thrun, “there were none in the home, [and] even in school they couldn’t get the hours needed to make progress … It was actually a big deal”.

“When students did get to the online programs, even navigating the computer systems could be daunting. One of the questions that tutors were frequently asked was how to do exponential notation on a computer”.

“…many students lacked even elementary-school-level mathematics knowledge…”

2. Establish clear and concise assignment guidelines and expectations. This is critical to student success. Students require concise, detailed and thorough instructions for all assignments, activities, forum discussions, exams, etc.  This is often the most overlooked step in online course design. Where Udacity/SJSU went wrong: they did not provide enough guidance for students; students were likely confused, disorganized and frustrated by not knowing what assignments were due, or what was expected. Time management is always a challenge for online students – without deadlines in a for-credit course, students get behind and become overwhelmed, more so towards the end of the course. 

“Then the unanticipated problems started to crop up. When the courses started, two of the three classes didn’t give students precise deadlines for assignments. “We communicated our expectations poorly,” concedes Thrun”.

3. Establish upfront the channels and avenues of communication. Create instructor presence and a sense of community where students will be more likely to feel comfortable asking for help.  Students need to know at the start of the course what academic support is available, and how to access it. Essential to successful learning in the online environment is building a sense of community where students feel connected, and know where to go for academic help and technical support. Where Udacity/SJSU went wrong: they did not communicate that support was available, which was a tremendous oversight given that one of the courses was serving remedial students.

“Initially many students were unaware of the online tutors (who are real people) who were available online to help, 12 hours a day. But over the weeks, it became clear that the tutoring services were crucial”.

4. Allow adequate time for course development. Where Udacity/SJSU went wrong: The deal between Udacity and SJSU was complex and required extensive negotiation taking up much time. The time allocated to course development was less than adequate. The courses were not completed before the program launch, but were designed while the courses were in session.

“The professors creating the curriculum for the program didn’t have much time; they were still writing curriculum when the courses began. 

We had a whole bunch of clerical mistakes. In most cases we heard about it, and fixed it on the fly. It happens in the classroom as well.”

Despite the disappointing results from the San Jose pilot there may be value in the experiment if constructive dialogue results.  However, I am concerned about the students, the students that failed in the pilot project, especially those in the remedial courses. I am also concerned that the focus in the soon-to-be published follow-up report will focus on the decision-making process of the institution, and ‘who didn’t do what’ at the leadership level rather than on the students.  Also disturbing is the apparent lack of expertise in the online learning pedagogy, course design and student support systems in this program. If this serious shortcoming is not recognized as such, online learning in general might be regarded as ineffective and/or inappropriate for college students. There are numerous successful online and blended programs in higher education institutions throughout the United States and beyond. But the results of this complicated, high-profile pilot project may hinder the dialogue about online education and its role in higher education given the lack of expertise and course design errors inherent to it. However,  I do look forward to reading the results of the forthcoming analysis and report from SJSU and hope that it provides constructive dialogue for the decision makers involved.


Posts on Instructional Design:

10 thoughts on “Udacity & San Jose U Halt Online Course Experiment: Lack of Instructional Strategy Undermines Courses

  1. I decided to take Udacity/SJSU’s ST095 Statistics, CS046 Java Programming, and PS001 Psychology this summer (not for credit). The courses are suppose to be 10 weeks long and I completed ST095 and CS046 in under 4 weeks each (still working on PS001).

    I found the courses more than acceptable and did not experience any significant difficulties. My first impression was that the courses were repetitious and if anything tedious. They seemed at times thin on content and overly repetitious. (It’s amazing how many ways they can ask the same question. My feeling was, present the material and move on to more. Don’t repeat the same thing, move on!) Not being a professional educator, my assumpion was that the instructors believe that a high level of repetition is required for today’s students to learn.

    The high failure rate of the students in the pilot study is more an indication of a lack of preparedness, motivation, and aptitude on the student’s part. The course content developed by the instructors and the technical interface provide by Udacity was more than adequate. The failure is a reflection of the failure of the K-12 system resulting in unprepared students.

    Hand-holding has not been a part of MOOC. It is unrealistic to explain every single detail to the students. If you do, you will bore the majority to death. The problem with MOOCs that I have witnessed is that because of its huge potential audience, some instructors assume a very low common background of their students. This results in a dumbed down class. The most satisfying courses are the most challenging ones.

    I just hope a successful course is not redefined as a course with a high pass rate at the loss of challenging content.


  2. Not to excuse Udacity but at least this sad situation is being played out in public which allows for some learning from mistakes to occur. The main shame, as you say is the use of already struggling students as test subjects. An honest test would have involved volunteers or more advanced students who could and would provide useful feedback. Alternately provide mentoring / tutoring accompaniment to allow for success instead of just another disappointment.

    Can we get past the notion of online being the economy-class seating section of higher education?


    1. Scott,

      You made a great point. I cannot see how they ( those casting judgment on the success of online instruction) can draw a conclusion based on the the success of students in several remedial courses. This is not a knock on those students whom are enrolled in the courses, but I think a valid test group should have been students with a minimum GPA of 3.0. Online courses, if structured correctly, can be suitable for any student; however, I think it would have been fair to set parameters as to who was allowed to take the course.

      It also seems that Udacity blew the process with poor instructional and course design. If deadlines were not established and course objectives were not explicit; how can a remedial student expected to be successful? It seems that the students did not fail, but rather the universities.


  3. TRUE: Qayoumi’s statement is unsettling.

    RE: Quayomi appears to gloss over the fact the failure applies to students; students in pursuit of higher education.
    AND: SO….
    Those students wasted their time, their efforts, their money, their chance at learning. The concept of opportunity cost comes to mind. Based on what I have read, in some cases, there seems to be some student faults, but, largely the courses were not well designed or executed. And staff responsible dropped the ball in all kinds of ways. Now, how to make it up to them? How to learn from the fiasco? What to do?

    RE: In his statement the university president sounds more like the CEO of a corporation than the leader of a public institution.
    In reality… He is both.

    RE: Yet students are not consumers of a product that may be inconvenienced by a faulty design.
    Yes, the students WERE Consumers. And YES: The product was faulty. And YES: They were inconvenienced. And far more than that. And the institution/business IS responsible to some degree. The students were treated like guinea pigs. Experimental subjects. Oh, My!

    RE: In fairness to Qayoumi, his statement may have been taken out of context; he may have elaborated on adverse effects experienced by students.
    Could not someone interview him and ask? It could be important.

    RE: An alternative viewpoint to consider is that Qayoumi appears eager to learn from the mistakes made and move forward.
    This is good. Now what is the strategy to move forward, and how will it be deployed? And how will the wasted efforts of the students be made up?

    My questions may seem silly to some, but if you look closely, you will see they are important. They could be the seeds of study, and more articles….


    1. Hi Pete
      You bring up some excellent points, and thoughts for further discussion, study and articles. One of the points you bring up is most pressing, which is using students as test subjects [or guinea pigs as you mention] in this experiment. This pilot project has been referred to more than once as an ‘experiment’. I feel it irresponsible of San Jose to decide to use remedial students in this experiment, then to let them fail. This is not only poor judgement on San Jose’s part to implement a program of this type with struggling students, but even worse – under their watch, more than half of the students failed. This is inexcusable. It is similar to conducting a drug trial, and continuing the study when the test drug makes patients ill and suffer adverse affects. Patients are allowed to stop the drug at any time – thus giving them control. How much control and support were given to San Jose students?

      Thanks Pete for your thoughtful comment . Debbie


      1. Debbie:
        Thank you for your kind reply!

        I may have spoken a bit strongly, but you see…. it is personal. A few times in my life I was used as a guinea pig in my former educational life. And well, sometimes I won, sometime s I lost. But the losses were hard, and the victories, small….

        And so, when I saw others “being used like this”, by those who had a “duty of care” and were in a position of power over the students…. And who already HAD THEIR EDUCATIONS and would suffer NO consequences. I got a bit excited.

        Now, you, are educated. You are in a position to comment, perhaps even to act. But you have already done one duty: That of reporting the facts to the world. Good job. Keep it up. Perhaps, in your studying this you may come up with some ideas and suggestions to keep such a thing from evvver happening again. Sometimes, the little things, the small acts, can have great consequences to the good. So please, since you are in the right place at the right time…. Let your little light shine. For light is needed.

        And do note there is a difference between revenge and justice. I hope you will fight for propriety, justice, fairness, and “making up for a bad thing”. Revenge at this time, will help no one.

        Take care!

        PM Laberge Ontario Canada


        1. Hi Pete,
          Yes – revenge is not the best way to impact change – it is shedding light on issues, sharing facts and expressing opinions and ideas that may impact events such as this in a positive way. Your comments are a good example of how to make a difference. My aim is to help educators help students be successful in obtaining an education.
          Thanks for your inspiration.
          (PS I see you are from Ontario, my hometown is Toronto, Canada!)


          1. Well, then Debbie, I dare you you are masking headway. Sure and slow wins the race! Ah, “Toronto the Good”. I’m from Sudbury. Take care.


  4. It is shocking that the supposed experts in online education (Udacity employees) actually did not apply the principles of sound instructional design to their courses. Perhaps, the instructional design was left to coders or professors with little experience in online education. Almost all experienced online instructors and instructional designers use the principles that you outlined to develop online course, and Udacity and SJSU neglected most of them.


    1. I agree – it is quite surprising that a company [Udacity] that delivers online education was lacking in the basic principles of online learning. Very disappointing, more so for those students that failed :(. Thanks for reading and your comment.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s