Jack Mostow, a semi-retired computer science professor from Carnegie Mellon University, launched Project LISTEN in 1990. The goal was to use Artificial Intelligence to help students learn to read. Today, the intellectual property from Project LISTEN is used in Amira, a leading AI tutor for reading. Jack has been a research colleague for decades, and as I interviewed him, I was eager to learn about his trajectory from foundational research to a product that now is helping over a million students to learn to read.
Jack got started on Project LISTEN more than 30 years ago, after he became a new father. He previously was involved in research on automatic speech recognition at Carnegie Mellon as a Ph.D. student. He knew he wanted to apply AI to education, which was an interest that was encouraged by the field’s seminal leaders, such as Herb Simon and Raj Reddy.
The decisive moment came on a walk with his parents. Jack had a colleague who was developing an intelligent tutor to shave seconds off the work of phone operators. Jack recalls: “And I thought, well, that’s kind of not inspiring. And so I asked, ‘If we are going to get computers to teach something and help people learn something, what would be useful to help them learn?’” Jack’s mathematician father encouraged Jack to develop a math tutor. His mother, who had edited her college newspaper, held out for reading. Jack recalled, “All of a sudden, it clicked.” He had a vision for a system in which AI would listen to a child as they read aloud, and it would offer feedback to the child so they could improve. Fortunately, the National Science Foundation had a funding program at the time that was seeking to sponsor “high risk/high reward” exploratory research projects in education, and Project LISTEN fit the bill.
Automated speech recognition (ASR) was in an early state in 1990. Yet Jack realized that in this application, the computer could be supplied with the text the student was reading, and that would be “like shooting fish in a barrel” (compared to listening to anything a child might say). It was a great vision, but didn’t turn out to be so easy, he said. “It’s also harder because there’s an infinite number of ways for a child to misread the text.” Jack explored early prototypes of Project LISTEN’s Reading Tutor with his daughter. Herb Simon correctly predicted Jack’s daughter would learn to read faster than he could build the system. In fact, 20 years of productive research and development followed.
From my own experience, I know that building great educational technologies is rarely something that computer scientists can do alone, so I was interested in the team that Jack assembled. He mentioned the incredible depth of talent at CMU, and also that they worked closely with a local master reading teacher, who was an expert at training teachers to give feedback on reading.
Jack quickly found field sites in Pittsburgh to test his ideas. The first studies didn’t use AI at all, because it wasn’t clear what an AI reading tutor should do. Instead, Jack’s team conducted a “Wizard of Oz” study, where a hidden person played the role of a computer. The local teacher wrote a page of instructions for how the person should respond to students. This protocol was used to test whether the simulated system would “enable struggling readers to read and comprehend material significantly more advanced than what they could read on their own.” Jack submitted the results of the study to a prestigious AI conference, he recalls, “And when I submitted it, I thought, well, AAAI may reject it because they don’t think it’s AI. Yeah. So I was very pleasantly surprised when it won the outstanding paper award!”
With those results in hand, Jack went to work on an automated system named “Evelyn” after Evelyn Woods, famous for speed reading, and coincidentally his mother’s name. In the interview, he described many hurdles to me, such as:
- Getting the technology to run fast enough.
- Figuring out good rules for giving students feedback.
- Working out effective ways to provide visual and auditory feedback to students.
- Determining how to enable students to ask for help.
The results came together in a demo at an NSF meeting. Jack recalled:
“An NSF meeting was coming up, yet the components we were building for Evelyn didn’t even run on the same machine. The speech recognizer ran on a $50,000 HP workstation; the reading coach on a NeXT workstation. And we thought, wouldn’t it be cool if we could actually integrate them in time for this meeting? Yeah. And thanks to all the people on my team, we pulled it off. That must be a ‘demo gods’ moment! Wow!”
After this success, Jack continued with research studies for two decades. It was during this period that I met Jack frequently at conferences, and I always looked forward to learning the latest improvement to Project LISTEN. Over time, speech recognition became faster and more accurate. Jack’s team learned which student mistakes should be ignored and which should be addressed. They found ways to keep students engaged with the task, even when the computer was a bit slow. They determined new ways to provide feedback to students to better help their reading. Most importantly, they conducted larger-scale studies in schools to establish the efficacy of their tutor. It worked.
As we wrapped up the interview, I asked Jack if he had a surprising lesson that he learned through research. Jack shared a story from a period in which he was exploring implementation of the tutor in different school settings: in a classroom, in a computer lab, and with a reading specialist instead of a classroom teacher. The team learned that students spent different amounts of time with the tutor; this variation mattered for learning outcomes. But Jack did not want to set a uniform policy in the software for how much time a student should spend, because he respected teachers’ decision making. Further, teachers did not want to spend their time configuring each student’s assignment in the software. The team adopted a clever solution devised by one of the teachers: they bought kitchen timers so a teacher could give a student “a target amount of time” in a very concrete, easy-to-set way. Yet because students might not respect the target, a simple display in the software gave teachers information on how much time each student actually spent. Jack shared a few other stories in which low-tech solutions were invented by or inspired by teacher routines. These solutions led to ways to implement the reading tutor well in varied school settings.
Reflecting on this interview, I realized that today’s use of AI in Amira reflects more than intellectual property; it also reflects the philosophy of Jack’s work. While the underlying software has been updated, Amira continues to listen to students as they read aloud and it provides feedback to students when they struggle with a word. Like Project LISTEN’s breakthrough demo, Amira connects many components together to achieve a unified experience. For example, it also shows a human face pronouncing a word to give feedback to students. And these days, it offers teachers many useful reports—building on Jack’s realization that providing the right supports for the teacher was essential to success. And growing the body of research and evidence is still a core mission of the company. It’s nice to see how the intellectual footprint of Jack’s vision became part of the scale up into a commercial product.
In his semi-retirement, Jack Mostow remains engaged in related work with RoboTutor. His legacy endures not only as a specific educational use of AI, but also his emphasis on engaging reading experts, listening to teachers, conducting research to drive improvements, and moving from small demonstrations to large-scale efficacy trials in schools.
Jeremy Roschelle is Executive Director of Learning Sciences Research at Digital Promise and a Fellow of the International Society of the Learning Sciences.
Join the Discussion (0)
Become a Member or Sign In to Post a Comment