Sagacity

The Scientific Advisory Group for Emergencies (SAGE) has attracted some mythology this year for its meetings about the pandemic response. These are like no past crisis, partly for the number of meetings, but also for the ongoing science activity, in contrast to more engineering focused responses in the past (volcanic eruption; nuclear accident; resevoir breach). Since late May much of its material has been published, to a point that it has not seen detailed scrutiny, partly because material was simply dumped on a website. Even reading through the minutes of more than 70 meetings has taken a long time, but I have an extended thread building:

https://twitter.com/Acertaintom/status/1267433437180370945

Timeline

Those meetings began with a pre-SAGE, a meeting assembled in anticipation of an emergency COBRA committee, and the minutes indicate priority unknowns to resolve. But actually, NERVTAG began meeting weeks earlier, and established the early science ahead of SAGE, indeed reviewing the minutes of SAGE meetings, there are coherent phases, some involving other committees and agencies, and others marking evolution of SAGE. Each has its trials, which is the point of crisis response, but also distinct focus and activity.

Phase 1 NERVTAG started meeting to advise on the key questions, such as transmissibility, virulence and protection. At first, a PHE test was being developed and symptom screening and travel advice were control mechanisms. This moved on to a focus on the clinical course and treatment possibilities, as well as monitoring early spread out of Hubei. At every stage they were saying how it was different to 'flu. Early January 2020

Phase 2 SAGE started modelling (SPI-M) immediately, calibrating using parameters on transmission, hospitalisation and mortality reported in China, starting from a 'flu SEIR model. They rapidly advised asymptomatic transmission risk was high and community transmission was likely, indicating a need for behavioural advice. Research and monitoring networks for clinical trials and viral genomics stood up. January - February 2020

Phase 3 SAGE strategised a reasonable worst case scenario (RWCS) and behavioural guidelines (SPI-B) to promote things like social distancing, cough etiquette and handwashing. Alongside this risks in institutions such as schools and prisons were evaluated and hospital treatment protection requested. Case surveillance in healthcare was established as early warning mechanism, because testing capacity was inadequate. February - March 2020

Phase 4 Was the shift to managing community transmission. This was the intense period of many 'changes to the science'. The virus moved faster than anticipated, overwhelming contact tracing, testing and hospital PPE capacity. New contingencies emerged from shortages of key goods to healthcare capacity and specialist equipment. An app project was developing and other systematic surveillance projects were discussed. March - April 2020

Phase 4a In lockdown, many more people were involved, now meeting on zoom, and more work packages started to emerge, including external support such as RAMP, run by the Royal Society. Ian Boyd was engaged as SAGE independent challenge function, including pastoral support for committee members. The importance of publishing evidence, membership and communicating well with the public was agreed in the minutes.

Phase 4b Formal subgroups etc emerged to tackle other issues, such as evaluation of the effectiveness of interventions: specific subgroups collected evidence on particular issues such as around children and schools; other institutions, e.g. IFoA, had formed groups to support evidence development and partnerships forged. Tensions with aspects of government priorities start to emerge, as operational issues stymie control measures.

Phase 5 Government starts to take back control, establishing separate functions for surveillance (JBC) and contact tracing (T&T). SAGE moves to meeting once per week instead of twice as it did for the first four months. Only at this point are SAGE minutes being published, but a separate science policy platform emerged led by a former science advisor, David King, while government is trying to communicate relaxation of rules. May 2020

Phase 6 SAGE shifts to focus on strategic role in respect of preparing for winter and other emerging research evidence. Receiving reports from some of the large independent groups rather than from its own subgroups and direct communication of evidence. Independent academic reports are now being published in journals rather than being sent directly, and the RAMP review system is used to triage central consideration. June - July 2020

Phase 7 Towards the end of August, problems are re-emerging as cases rise across Europe, meanwhile reopening in the autumn prompts a shift in rhythm. Meeting weekly plus ad hoc additional meetings on specific topics, including higher education or schools, and consideration of the need for greater restrictions. The political declarations about the economy and health increase separation of SAGE from government. August - September 2020

Phase 8 In October, an alternative plan emerged, with different parts of the country in tiers. A compromise between economics and epidemiology, recognising geographical differences in transmission. Meetings largely focused on receiving new science and evaluating existing restrictions. Scenario models are refined and further restrictions clearly indicated. SAGE is still perceived as the source of all government analysis. October - November 2020

Evolution

At each stage, things have been problematic but the early stages were quieter, with scientists proceeding to give advice behind the scenes, per emergency protocol. In phase 1, there were challenges to standard disease control measures and understanding how to characterise and respond proportionately. In phase 2, there were difficulties getting sufficient data to calibrate models and understand what was going on. In phase 3, surveillance was being developed in the absence of testing capacity while trying to inform the public of the strategy, then in phase 4 everything went public, with emerging concerns about ethnicity taking everyone by surprise, and control was lost. Science was happening live on the arXiv and parliament passed legislation on financial support measures and emergency restrictions on individual liberty, too shocked to challenge.

In phase 4a, problems were different as the leaders in London all caught the virus and care homes were exposed, but SAGE members started to rally to solutions, like the household survey of infections and even proposing to develop their own app based contact tracing. Meanwhile a media frenzy of speculation about SAGE's accountability and the prospect of a public inquiry had only a daily Downing St press conference for succour, as the death toll mounted and police dispersed furloughed sunbathers. Although the infection survey is now an innovation which is the envy of the world, the display of initiative was squashed as government returned to work in phase 4b, only asking SAGE subgroups to comment on proposals for the remit of the JBC. External dissent grew further with objections to government use of statistics and presentation of data, media followed up the procurement of PPE and production of ventilators, and select committees started to ask about why so many people had died before effective restrictions to limits the spread of the virus were in place.

There was considerable chaos with the Prime Minister in hospital and unprecedented restrictions, giving an impression that scientists were left to get on with it in phase 4. But centralised solution projects quickly emerged, for testing, contact tracing and then surveillance. This left SAGE without a clear role in phase 5, as COBRA handed crisis management on to several cabinet committees. Not considering operational issues on scale of testing, there was still intensive scrutiny of practical measures for control in specific settings, because of the epidemiological consequences of reopening places like schools. Meanwhile, science was taking time, and there was a gap in what could be the basis of advice as big projects scaled up. There were repeated, impotent calls for more evaluation to be done, and increasingly SAGE was not part of key discussions in phase 6. This culminated in the release of a winter RWCS outsourced to the Academy of Medical Sciences.

It might be fairer to say the members of SAGE needed a break and they probably did get one as the meetings were only weekly in the summer months. But they also started to take a more strategic view of what were likely to be challenges in the future, as well as taking a view on issues internationally. Despite much better data collection, actual analysis was of relatively low quality, meaning symptomatological understanding did not evolve beyond anosmia. By September and phase 7 SAGE started to predict exponential growth, and being rebuffed validated their 6 week forecasting tool, demonstrating higher levels of accuracy. When isolated campus outbreaks were clearly not sufficient to characterise the epidemic they did reach much more understanding with government. But it took until phase 8 for the political nature of advice in support of epidemic management to be appreciated - no longer a crisis response, it needed scientific strategy.

Criticism

There has been much ire focused on SAGE, partly because the rest of the system was invisible, and avoiding attention. But this is a symptom of one of the problems, that public communication was not planned as part of the role, and poorly managed throughout. Early briefings were occasionally planned through the Science Media Centre but were issue specific and rare. However, in March the need to communicate openly and publish material was recognised, it just took until the very end of May to do it. Even then, the capability to engage with the media was far short of demand, especially given the need to spend time doing the science. This might be the most difficult problem to resolve, that the public critics were simply not as busy.

Another feature of SAGE was its introspection on its own role without appreciation of the political nature of its mandate beyond a very narrow science based emergency. The role of SAGE is to provide advice in an emergency drawing on scientific expertise in response to questions from government departments. That this does not include economics or operational matters is completely opaque. And the suspicion that political agendas are being pushed as the use of the advice invisible also natural but not taken seriously enough. The introduction of a former science advisor in Ian Boyd, to improve internal processes, made some ground, including improving documentation, but was too little too late. And there remained a presumption that all relevant perspectives were included, even as the understanding of the disease evolved and challenged this.

Collaboration

Behind the scenes, many organisations rallied huge specialised groups to address scientific problems that were major unknowns. Some were publicly reported at the time, although never really described, such as the modelling support convened by the Royal Society (RAMP). Others included the Alan Turing Institute supporting data science, including helping with the signal processing problems which supported the contact tracing app development. Many learned societies and professional bodies took an interest, with the actuaries developing risk calculation which subsequently prioritised groups for vaccination. The RSS had been working behind the scenes with the JBC for months before this was made public, and many members of it working group had individual advisory roles. Meanwhile, the Royal Society also convened groups DELVE and later Set-C to respond to science questions, it took until the summer for SAGE to be commissioning these well.

One gap here was coordination, which was managed to a degree by engaging with media, especially supported by the Science Media Centre. There was just so much research being produced, spewing out as preprints, that it was difficult to distil into coherent reports. But the central assimilation was managed to some degree by review processes of new publications, 'triage' into SAGE by RAMP. Actually what was the biggest shortcoming was the lack of a research strategy for Disease X - when the strategy first emerged, there were only clear plans for trialling therapeutics, developing vaccines and novel diagnostics. Other priorities in respect of risk stratification and modelling were supported by academic and professional volunteers, with too few disease specialists available - so UKRI put out urgent calls for new research applications in response to the pandemic.

Disease X

Planning for pandemics, the UK government considers 'flu and something like SARS1 in its risk matrix, which considers impact and likelihood. While 'flu is considered a greater threat there, SARS2 actually has features of each: It is a greater transmission risk than SARS1 because people without any or only symptoms spread it and there are lots of those, and its much lower lethality for most than SARS1 giving a different public perception. And it is more dangerous than 'flu killing at least 10 times as often and leaving plenty of other impacts from those with long spells in hospital for ARDS and other organ damage too. In terms of vaccine plans, it also looks like it is different to 'flu with more effective vaccines but waning immunity, and a larger at risk population. Indeed the risk profile of 'flu affects children, while SARS2 barely touches younger children, it seems a lot more dangerous for an unpredictable range of adults. Thankfully, the WHO had recognised that pandemics might be caused by a disease new to science, Disease X, however it is not clear there was any plan for that.

More pointedly, there has been great difficulty shifting from initial conceptual models of SARS2, even as they were based on limited data in unfamiliar settings. Some examples: initial symptoms were cough and fever, with shortness of breath indicating worsening course, but anosmia is surprisingly specific and yet it took months to acknowledge. Initial advice was about hand hygiene, and social distancing, with focus on direct contact and aerosols, and even now there is difficulty talking about ventilation. Asymptomatic proportions of cases were fixed below 20% early on, and despite the best survey in the world consistently showing values above 50%, that consensus figure has not broached 30%. The initial focus on hospital care was understandable, but limited follow up meant it has taken most of a year to understanding readmission data. Indeed the focus on ARDS blinded many to impacts beyond organs like the kidneys which accentuated comorbidities, so sufferers have compiled their own evidence more effectively than scientists. Modellers also struggled, including with public attention to parameters such as R, but community response has been self-managed in RAMP and the INI IDP programme, and also abstruse enough to be fairly openly shared.

Scientific Frontiers

The conclusion is a broad based acknowledgement of the unknown was missing. There was a need to formulate questions, and also to identify research priorities. None of the structures were set up to do this, and so the effective responses have been huge consortia, with dissident individuals engaging through (social) media, often for the worse. Despite two reports now published on the functioning of science advice in this crisis, neither has picked up that there was a fundamental lack of strategic direction. Indeed it was not widely acknowledged that the challenge was Disease X, and most successful projects were familiar in RCTs for therapeutics and development of vaccines (although that has also seen considerable innovation in process). Some examples are still too uncomfortable, such as the collection of virus genome data not having a strategic analytical engagement with the epidemiology. And of course the widely recognised lack of understanding of the community, whether local public health directors, community leaders in the disadvantaged groups worst hit, and the recovering patients themselves has been badly neglected and remains poorly appreciated.

It is not all bad, indeed early modelling started well, and it took time to make use of additional capacity. But much of the evolution has been half-hearted, or subject to further criticism, indeed dumping material on a website served to drown public scrutiny more than offering accountability. Funding mechanisms for grants were criticised for rejecting applications on structural inequalities despite glaring evidence around ethnicity. And gender patterns around the differential mortality affecting men were also not deemed worthy of funding. More generally, responses have come through academic consortia, once again drowning the problems, rather than being investigator led. Some infrastructure like the National Core Studies may have certain critical scale, and the ONS Infection Survey has proven adaptable to innovations for the study of long term symptoms and viral variants. But this ought to have been a science-led emergency response, and it has instead been clinical. At many times politicians were criticised for saying they were following the science, but actually they looked frightened, and eventually settled on trusting the doctors, rather than embracing scientific inquiry.

Exeunt Pursued by the data

Despite the scorn poured on government and SAGE from many, the RCTs of therapeutics have delivered as they should, and the vaccine programme is delivering both astonishing pace and equity. Even the rather Heath Robinson efforts on lateral flow testing devices are starting to break through serious criticisms of their diagnostic suitability in practice. The only group who were really able to take a strategic view, seeing the big picture (do we call it a global pandemic because we don't really believe the scale?) and the lifetime perspective, has been the Science and Technology select committee of the House of Lords. Unravelling extensive unknowns, without continually trying to blame people, many of the conclusions of their hearings are documented here. The utility of models and reliability of testing both came under scrutiny, as did more general science communication with the public. But there has been one blindspot, standing beyond their expertise, and that is data, perhaps leading to a rather naive recommendation of the AMS winter plan.

Government was derided for number theatre by none other than David Spiegelhalter, and the relevant select committee in the Commons has been moved to inquire. But it is clear that no one offering criticism really understands the nature of data, and more particularly models, in such research and this crisis. One of the triumphs of government has been the exemplary dashboard, but serious consideration requires much more data and critical evaluation. Meanwhile, despite its world leading character, the mathematical work done in public has been largely ignored, notwithstanding occasional efforts to understand one particular model. Even the Chief Medical Officer has said that he only considers the hard data and validated 6 week forecasts, not the mathematical modelling which must be the basis of strategic plans. The scientists made an abortive attempt to run independent data briefings, but they never had a plan of what they actually wanted to communicate if it was not about policy. And the press briefings themselves have improved largely because the science has become more interesting and the journalists, and public, are asking better questions of it.

Informed but not persuaded

David Spiegelhalter has expounded the need to inform not persuade, but when we ask for transparency, what do we actually want, and what does that look like? Intelligent transparency is the subject of my next post but there are fundamental issues to review, around the role of science in society. There can be no democracy without acounting to the public: models are hard work, so there is the need to distil the complexity, communicate with analogy and visual aids, but also to raise public appreciation of their details. Putting large amounts of material into the public domain is a necessary step but its technical nature also serves to obfuscate important developments. On top of that, anything relevant to policy is held back until decisions are made, except when it is dramatically leaked. This has led to formal accusations of cherrypicking data to make political arguments, to which no official rebuttal has even been offered. The role of science in maintaining our value based society can only be sustained if the public are equipped to make sense of decisions taken in their name. When politicians claim to follow the science, they actually follow the data; they flatter to persuade.