Five Research Questions Raised by a Pre-Mortem on the 60 Minutes Segment on Code.org

Last Month, CBS "60 Minutes" had a segment on "Closing the Gender Gap in the Tech Industry" featuring an interview with Hadi Partovi of Code.org. The key claim comes at the end of the interview:

If even 1% of the girls on Code.org who are– who are learning this in middle school, elementary school and high school, if even 1% of those students decide to major in computer science in the university, that'll even the 50/50 balance among the university students…I think if we look at this in about 5, 10 years, the gender gap in university computer science will be gone.

This was a controversial segment. The founder and CEO of Girls Who Code, Reshma Saujani pushed back, saying that "60 Minutes" had ignored women’s voices (see blog post here). Partovi later apologized for his response. I was particularly appreciative of NCWIT’s data-based response that showed that efforts to diversify CS are making progress (see page here).

Here, I look at the claim from a different angle. What don’t we know about what it takes to build a gender balanced CS program, and what could we learn from this process? If the gender gap in universities did disappear in the next 5-10 years, it would be amazing. We should be watching to see how it happens!

Partovi says that "25% of all students in America have an account on Code.org." His argument is that if 1% of those students major in computer science, we’ll eliminate the gender gap. What would it take to translate all those accounts on Code.org into undergraduate CS majors? What should be watching for as we get the 1% from the 25%?

One way of figuring out what to watch is with a pre-mortem (see Wikipedia page and Harvard Business Review article). Imagine that it’s 10 years in the future, and CS is not gender-balanced. What might have gone wrong? The research questions arise from exploring what might prevent gender balance in undergraduate CS.

Partovi may well be right, and the Code.org students may go on to undergraduate CS. The people I know at Code.org do excellent work and are working hard to achieve his vision. To consider the possibility that it might not work is not meant as a criticism of Code.org. The purpose of a pre-mortem is to figure out what might go wrong and fix it. I am using a pre-mortem to identify important CS education research questions which help us understand what might lead to a more a diverse CS workforce.

Here are five research questions that explore the ways that the Code.org effort might not lead to gender balance in CS:

• Pre-mortem: Students with Code.org accounts don’t actually use the Code.org resources. Maybe they would love CS, but maybe they don’t even see it.
• How many of students with accounts on Code.org actually use any Code.org resources?. Teachers I’ve talked to have signed up their whole class with Code.org accounts. Having an account does not necessarily mean that students are doing any computer science activities. What is the actual use of Code.org curriculum, and how pervasive is it?
• Pre-mortem: Students try Code.org curriculum but don’t become CS majors A study just published in Education Researcher finds that taking Advanced Placement Mathematics classes has negligible effect on choosing any STEM major (see paper here).
• Do students using Code.org develop an "intent to persist"? For 1% of the 25% to translate into CS majors, the students have to decide that they want to continue in CS. Education researchers sometimes call this "intent to persist." There is some research on what leads students to intend to persist from undergraduate introductory CS classes into the CS major. A 2009 paper by Barker, McDowell, and Kalahar found that peer interactions (students working with each other) was the most significant predictor of intent to persist. It’s much more challenging to measure intent to persist into a CS major 7-8 years later from 10- and 11-year-olds (if the concept makes sense at that age), but it’s an important and interesting question.
• Pre-mortem: Yes, lots of kids used Code.org curriculum, but it was predominantly the boys who loved it, who went into CS, and who persisted to graduation.
• How do these answers differ across genders? It’s pretty easy to get an all-male 1% from a quarter of all school children. Do girls or boys use Code.org resources more? Is there a gender difference in how students respond to Code.org curriculum? Are the reasons girls persist the same as the ones for boys? We might find that curriculum designs that work for boys don’t also have the same impact for girls, which would prevent us from reaching gender balance.
• Pre-mortem: A diverse set of students wanted to enter University CS majors, but there wasn’t room for them, or the women were not retained to graduation.
• Will there be room enough for all those CS majors? Right now, undergraduate CS enrollment is at an all-time high in Universities (see this Blog@CACM post on the problem). Could University CS education scale to the level that Partovi predicts is coming? There are roughly 56 million children in American schools (see Dept of Ed data); 25% of that is 14 million, and 1% of that is 140,000. What might happen in Universities over the next 5-10 years so that we could welcome a diverse set of 140,000 CS majors (over some period of time) and retain them to graduation?
• Pre-mortem: The Code.org teachers started out teaching 25% of all American schoolchildren, but only for a year or two. Then they went back to whatever they taught before, and the kids found other interests a few years later so they went into non-CS majors.
• Do the Code.org teachers persist in teaching CS? To achieve gender balance, the goal is greater than just to get 1% of the 25% of American students in 2019. We will need to continue teaching American students about computer science if we want to sustain a diverse workforce. The Code.org approach takes "English teachers and science teachers and history teachers" who just "find their (Code.org) curriculum and just start teaching it." Partovi contrasts this approach with the universities who prepare CS teachers (e.g., see blog post on "What Universities Must Do to Prepare CS Teachers"), but only prepared 75 teachers last year (as mentioned in the interview). Do the Code.org teachers continue as CS teachers? As Partovi points out, it’s only been five years that Code.org has been working with schools, so we probably can’t know yet. Lijun Ni found that CS teachers having a sense of belonging to a community was an important part of retaining them as CS teachers (see paper here). Maybe if the Code.org teachers have a sense of belonging, we’ll keep them as well (or maybe longer?) than University-trained CS teachers. It’s a great research question to study.