One of the key drivers for the decision to lock down the U.K. in late March 2020 was a computational epidemiological model developed at Imperial College, London. When the code for the model was released, it immediately attracted criticism with respect to software quality – and these criticisms then provided ammunition for those who argued the lockdown was an overreaction. So, what was the software and what are the criticisms? Are they significant? And, crucially, what can we learn from the experience?
Throughout the first half of March 2020, European nations raced to enforce lockdowns in an attempt to stem the spread of the coronavirus. By mid-March, much of Europe had closed the shutters on ordinary life. But the U.K., it seemed, was intent on business as usual: no lockdown seemed in prospect. And then, more or less overnight, the U.K. government pivoted: just 10 days later, the U.K. was also in lockdown. It became apparent that one driver for the change of heart was an epidemiological model developed by Neil Ferguson and his team at Imperial College, London. The report produced on the basis of the model made international headlines. It is surely one of the most influential and terrifying scientific papers written this century. It predicted that, unless containment measures were taken, then the virus would cause more than half a million excess deaths in the U.K. – more than the total number of U.K. fatalities in World War II. The U.K.'s health care system would be utterly overwhelmed. Given these horrifying predictions, the decision to lock down perhaps seemed inevitable.
The Ferguson model is an example of an agent-based epidemiological model: it models the spread of disease down at the level of individuals and their contacts. At the heart of such a model is a graph representing a network of social contacts, with vertices corresponding to individuals, and edges indicating a social contact, and hence a possible route for infection. This is then overlaid with a model of how the disease spreads through the network, which at its crudest might be a probability that a disease will spread from one infected individual to another individual with whom they have contact. Given such a model and an initial state of the network, it is computationally a relatively straightforward matter to simulate how infection spreads, although of course since the models are stochastic, different simulations will yield different results. The Ferguson model is much richer than this: makes a raft of assumptions about questions such how the disease spreads, whether asymptomatic individuals can infect others, how infection progresses through an individual to recovery or death, the infection fatality rate, the case fatality rate, and so on. At the time of writing, seven months into the pandemic, great uncertainty remains around many of these basic questions. Given such uncertainty, the model must be evaluated with a range of possibilities for each assumption. This highlights some fundamental difficulties with this type of modelling: First, the sheer weight of assumptions means it is hard to be confident about the results of simulation: even small changes to initial assumptions can make big differences to the results. Second, the sheer number of parameters means the space of different combinations is huge. Systematically exploring the entire parameter space is impossible.
The software base used to produce the report was about 15,000 lines of code, adapted from several decades of previous models developed within Ferguson's lab. A cleaned up version of the code (not the original) was made available on GitHub in April 2020. It quickly attracted a great deal of professional interest from the international computing community, and very shortly thereafter, the code began to attract criticism, and at the heart of these criticisms were suggestions that the software was simply of poor quality. It was, according to one U.K. newspaper report, "'totally unreliable… a buggy mess… impossible to read." The main issues raised seem to be as follows:
Of course, most research software is indeed developed without the ceremony and rigor that accompanies professional software development. Research software is not usually intended to be understood by third parties, or to be reused. It is often developed with the sole purpose of generating a set of results and is then abandoned. Ferguson's position is simply that there just wasn't time to do anything more: high-ceremony software engineering would have delayed the report. He subsequently speculated that if the U.K. had locked down just one week earlier than it did, then 20,000 lives might have been saved. If correct, then even a day's further delay would have led to significantly increased loss of life.
While many members of the scientific community support the view that a lockdown was the only realistic way to control the disease, this view is by no means universally held. A highly vocal community believes the lockdown was an overreaction, and the devastating economic and social consequences that ensued were avoidable. This group seized upon criticisms of the Ferguson model, taking them as evidence that the lockdown was based on what they claimed was poor quality science. David Davis, a U.K. Member of Parliament, queried what he called "secret and potentially flawed calculations." In a bizarre twist of fate, software engineering quality had become a political weapon.
Subsequent analysis seems to have indicated that, while the extensive criticism about relaxed software engineering practices is perhaps justified, it was not fundamentally flawed. The code is not pretty—but this does not mean it was wrong.
While we will have to wait for formal inquiries to truly understand the role the Ferguson model played in the U.K. government's decision making, and the true extent to which the various criticisms of it are substantive, we can, I believe, already identify some lessons to learn going forward, which will make us better equipped to handle an inevitable future pandemic:
Guest blogger Michael Wooldridge is a professor of computer science and head of the Department of Computer Science at the University of Oxford, U.K.
No entries found