Research Questions About MOOCs

"Will MOOCs destroy academia?" asked Moshe Vardi last November in CACM. The next month, CACM published an article about how MOOCs were disrupting education. Moody's Investor Services predicts that small institutions could be "damaged" by MOOCs.  Thomas Friedman suggests that MOOCs are a "revolution" that will change higher education.  I call the threat of MOOCs destroying the university the "MOOCopalypse."

As a computing education researcher, I find MOOCs to be a fascinating and promising educational technology.  Replacing whole courses (as San Jose State University has decided to do), let alone whole universities, with a mostly untested and unproven technology seems premature. Starting from three research questions that I'm particularly interested in, here is I would like to know before we bow to our MOOC overlords:

What will be the impact of MOOCs on diversity in computing? Computing (encompassing computer science, information systems, information technology, computer engineering, and software engineering) is predominantly white or Asian and male.  The percentage of women in computing continues to decline.  Only 11.7% of Bachelor's degrees in CS went to women in 2011.

MOOCs are unlikely to make this better.  We don't know much about demographics in MOOCs, but what data we see are pretty depressing.  Tucker Balch just offered the first Coursera MOOC from Georgia Tech's College of Computing, and his demographic survey results suggest that MOOC-based computing education would be even more exclusive than what we currently have.  His completers (the small percentage of MOOC enrollees who finish the course) were 88.6% white or Asian and 91% male.

Of course, there might be ways to offer MOOCs that draw in a more diverse student body and retain them.  We should figure out those best practices before we replace existing courses with MOOCs.  Our research with high school students and Joanne Cohoon's work with undergraduate and graduate students suggests that one-on-one encouragement is the most effective way of engaging and retaining students from under-represented groups.  That's hard to do in classes with thousands of students.

Are MOOC discussions doing any good?   MOOCs use on-line discussion forums to replace in-class Q&A and discourse. We know that learning has a significant social aspect, and dialog facilitates learning.  

In the late 1990's, I was part of an NSF-funded project to increase multidisciplinary on-line discussions among undergraduate mathematics, engineering, and computing students.  It was a disaster.  We were pulling teeth to get anyone to talk to one another.   Once, we had a class of differential equation students who were going to collaborate with engineering students to help them analyze some simulation data.  40% of the mathematics students took a zero on the assignment rather than collaborate.  We changed our focus to studying why students didn't want to collaborate.  The two biggest factors were a sense of competition ("With the curve, it is better when your peers do badly") and learned helplessness, e.g., not wanting to discuss in a public forum and demonstrate a lack of knowledge ("I am confident that my answers are wrong.")

In Tucker's demographic results, 95% of his completers read in the forum, but only 33% posted.  That's not really much of a dialogue.  Is the discussion forum really facilitating learning?  There are scores associated with completing MOOCs, and recognition of top students.  Does that sense of competition inhibit use of the discussion forum for learning?

How much learning is going on in MOOCs? The standard for measuring learning from an intervention is a pre-test and post-test, e.g., to compute normalized gains.  For example, we have been working on creating summer camps in Georgia that have significant improvements pre-test to post-test on CS content knowledge.

I don't know of any studies of MOOCs that try to measure learning gains.  However, the demographic data suggests that the completers know quite a bit already when they start.  78% of the completers in edX's first circuits course already had taken a circuits course.  30% of the completers in the Udacity CS101 course got perfect scores on the final exam.  In Tucker's course, over 40% had their Master's degrees and over 10% had PhD's.  How much did these students really learn from the MOOC?  

These demographic results also suggest that the completers in MOOCs are advanced learners who have significant metacognitive skills (e.g., they "know how to learn").  Thus, they have advantages that more novice learners (like first year undergraduate or high school students) won't have.  How will MOOCs have to change in order to help less-advanced students to learn?

The bottomline is that MOOCs may do a significantly worse job at engaging a diverse range of students and at educating than face-to-face classes.  Of course, we might be able to do much better in a MOOC than in our existing classes!  It's all too new. We're still exploring the territory. We ought to know what we're getting into before we start swapping out current practice for something else.