The Good Judgment Project

Government intelligence services have been in the news recently for monitoring citizens’ activities, but their charter isn’t simply to figure out what is going on now; rather, it is to try to predict what will be going on in the future. Who’s going to win the presidential election in Russia? What are the odds the government of Syria will survive the current turmoil?

Given that goal, governments try to figure out the most successful approaches to forecasting future events. That’s the reason the U.S. government’s Intelligence Advanced Research Projects Activity (IARPA) set up a "tournament" to challenge teams of forecasters to come up with their best methods. IARPA, which sponsors multiple cross-agency programs to provide the US with intel-ligence superiority, started the forecasting tournament two years ago.

Through a Broad Agency Announcement (akin to an RFP), IARPA invited teams to submit proposals for funding to be part of the tournament. Various university groups competed for the funding, and one of the grantees was the Good Judgment Project (GJP), led by Philip Tetlock and his wife Barbara Mellers; both hold appointments in the University of Pennsylvania’s De-partment of Psychology and the Wharton School of Business.

Tetlock had been somewhat responsible for sparking the idea of a tournament in the first place: "he had been on a National Academy of Sciences panel that was advising the intelligence agencies about what they could do to improve intelligence-gathering activities through social science principles," recalls GJP project manager Terry L. Murray. "To some extent, the recommendations of that group influenced IARPA to launch the Broad Agency Announcement. When they received a copy of the announcement, Phil and Barb got the idea to put together a team."

"The government chose five research groups to compete, including ours," relates Mellers. "We won the tournament in the first two years, at which point IARPA suspended the contracts with the other four teams and funded us to continue on for the next two years. We’re just at the begin-ning of year three now."

The GJP is made up of a team of about 30 researchers—faculty, graduate students, and post-docs—who manage a much larger pool of forecasters. The participating forecasters are randomly assigned to work by themselves or in teams, using various prediction methods. "Methods might include working by yourself on a survey, working with a team on a survey, or in a prediction market," says Mellers. (A prediction market is a type of forecasting ‘casino’ in which partici-pants can bet on the likelihood of particular events.)

For this year’s tournament, the GJP is running two prediction markets. One is similar to the old InTrade format, in which participants buy "futures" on possible events; the other sets market prices, and participants determine whether the prices are too high or too low.

Most forecasts take the form of probability estimates. "Do you remember that scene in Zero Dark Thirty where they go around the room and ask each person whether bin Laden is in Abbotabad?" asks Mellers. "One person says, ‘there’s a 50-percent chance he’s there,’ somebody else says 70 percent, and the hero says 95 percent." In another example, the team has been asked to provide forecasts for the number of refugees in Syria as of a given date, information that could help guide humanitarian relief efforts in that and other war zones. "It’s all probability estimates to higher-ups," says Mellers.

IARPA specified that all the forecasting was to be done online, with each project developing its own software for collecting individual participants’ forecasts and aggregating them into a single daily prediction on each question. One approach is to simply average the various probability es-timates, but an analysis of GJP’s data from the first tournament suggests that prediction accuracy can be further boosted by transforming the collective forecast away from the weighted average. Murray cautions that such a transformation doesn’t increase accuracy in all cases; "the trick is to find the transformation that will most improve accuracy in a particular situation." Results indicate that little or no transformation should be applied to the predictions of the most expert fore-casters, or to forecasts on questions with inherent unpredictability.

In addition to the role of computers in aggregating the individual forecasts, computers also support the forecasting process itself. The forecasters are located all over the world and need to be able to participate whenever it fits their schedules. "People participate either as individuals or as members of small teams," explains Murray. "If they’re on a team platform, they also have tools for interacting with their teammates, sharing information and discussing the forecast." The GJP works with three software vendors—David Wayrynen Systems Consulting, Inkling Markets, and Lumenogic—to set up the various platforms that support its efforts.

Asked if the GJP’s work influences real-world policy decisions, Mellers replies, "We certainly hope it does." Aside from the possibility the work might influence geopolitical decisions at the highest levels, the tournament offers other attractions to the competing research groups. "You win the satisfaction that you’ve come up with the cleverest way of asking people questions about uncertainty, aggregating the opinions of multiple people, and identifying who are the most accurate forecasters," says Mellers.

Murray adds, "The people who stick with it have fun. We did a Forecaster Satisfaction Survey at the end of last year, and we got comments from people saying it was their new favorite hobby. We have a group of 230 people called ‘super forecasters,’ taken from the most successful forecasters from the first two years. We had a conference with them in July, and some of them brought family members along. One of the fellows’ spouses just rolled her eyes when the discussion came around to how much time he was devoting to the project."

If being a forecaster sounds like fun, information on joining GJP (which is currently taking registrations for new forecasters) is available at https://sasupenn.qualtrics.com/SE/?SID=SV_4PdJxEZ2ncfQM7z.

Logan Kugler is a freelance technology writer based in Silicon Valley. He has written for over 60 major publications.