Digging deeper into SwissCovid data
Recently, I have gained access to data underlying one of the scientific papers claiming effectiveness of the SwissCovid contact tracing app. As we will see, this data actually tells a very different story from the narrative backed by Swiss federal officials, both in the health administration and the scientific community.
My initial criticism
In a March 2021 blogpost, I dug into a Dec 2020 preprint called Digital proximity tracing app notifications lead to faster quarantine in non-household contacts: results from the Zurich SARS-CoV-2 Cohort Study. This paper, with multiple authors at the University of Zurich, claimed that “contact persons with a risk of infection outside their own household went into quarantine about a day earlier if they received a warning via the app — compared to people without a warning”. This was seen by Swiss public authorities as evidence of effectiveness for the app, and sufficient for maintaining operation of the app (by the so-called “SwissCovid law” the app should have been abandoned without proof of effectiveness; as a separate matter medical devices need scientific evaluation in order to be rolled out).
I thought my blogpost highlighted crucial shortcomings in the analysis, overlooking behavioural effects and attributing positive outcomes to the app rather than other prosocial behaviours of individuals who volunteered to use the app. The key question is whether people who had been notified by the app were already in quarantine then, and if so why they were in quarantine already. I thought this was significant for two reasons:
- it might inform public health communications, which could for instance focus on explaining better how to conduct DIY contact tracing and collaborate with authorities on those matters (for instance by spotting superpsreading events faster in a decentralized way);
- it might give us back a chance to get out of the technosolutionist mindset surrounding evaluation of those apps (not just their conception). Indeed, what is currently evaluated in scientific papers is their operation according to a theoretical model of how these apps are used rather than actual patterns of use.
I thought the second point was particularly important in light of the dithyrambic coverage of a UK study on the NHS app, suffering from much of the same biases (but coated in more advanced statistics…).
However my March 2021 blogpost was criticized by one of the authors of the UK study as “tweaking the data” and “not fair”:
It’s all very interesting: it goes at the heart of anyone’s theoretical mental model of how the app will/should work, and where exactly is the burden to attribute a causal link, in the face of what seem to be genuine, statistically significant correlations between app usage and positive outcomes (in both the Swiss and UK studies), whose existence I do not contest.
Getting access to data
However, in the case of the Swiss study we can do better… one didn’t actually need to “tweak the data” and could simply look at the underlying survey data to try to figure out what was the actual causal link. I had said as much to some of the study authors since Jan 25th 2021, pointing out how their actual evidence had been overstretched in their preprint and then by their colleagues on the Swiss Scientific Task Force and in the media (my emails were ignored). Once the preprint was published in Aug 2021 however, I directly asked — as publicly promised on the preprint server— for access to the underlying data. Access was eventually granted on Dec 4th 2021, after signing a “Data Transfer Agreement” including the following description of my intentions:
I will thus limit my comments here to this NDA frame, although there would be more to say from this data (for instance, one could use this data to conduct additional studies using a different methodology).
The data came in a very simple format: as a list of questions, and a spreadsheet of responses. I was actually shocked to find out the survey included an actual question pertaining directly to the causal link (translation):
Wie wurden Sie als erstes aufgefordert, sich in Quarantäne zu begeben?
- Ich wurde durch den kantonsärztlichen Dienst kontaktiert und dazu aufgefordert
- Ich wurde von der SwissCovid App gewarnt und dazu aufgefordert
- Ich habe die Anweisungen meines Arztes oder einer Gesundheitsfachperson befolgt
- Ich habe die Anweisungen meines Arbeitgebers oder eines arbeitsmedizinischen Dienstes befolgt
- Ich habe die Empfehlungen von nahestehenden Personen (Freunde, Familie) befolgt
- Ich habe mich selbst in Quarantäne begeben, da ich von meinem Kontakt mit dem Indexfall wusste und gemäss der Richtlinien des Bundesamts für Gesundheit handelte
- Ich bleibe ohnehin grundsätzlich zu Hause, um mich und/oder andere Personen zu schützen.
- Anderer Grund
As we see, the second choice is explicitly that the SwissCovid notification was the cause of the quarantine. No need to guess, it’s right there: the question had been asked, but somehow the study authors didn’t relay that in their paper!? In the end, we have the following breakdown:
- out of 109 household contacts, none were warned by SwissCovid before being warned by other means. This is logical: in case of infection one tells their partner directly, the system is not needed and is bound to notify later than the direct notification.
- out of 213 non-household contacts, a grand total of one person (!!!) said they were in quarantine because of a SwissCovid app notification. However that same person also said they were warned by Manual Contact Tracing before receiving the app notification. This is what I had described as an outlandish scenario and very undesirable in my blog post, since it means someone attributed more trust to the digital contact tracing system than the manual contact tracing one, despite the manual contact tracing system being more rapid.
We thus see that the conclusion that “digital contact tracing lead to faster quarantine in non-household contact” is simply a N=0 claim. This might be why the title of the paper eventually published was changed, to the more modest Adherence and Association of Digital Proximity Tracing App Notifications With Earlier Time to Quarantine: Results From the Zurich SARS-CoV-2 Cohort Study.
However the authors of this study still stand by the claim that “contact persons with a risk of infection outside their own household went into quarantine about a day earlier if they received a warning via the app — compared to people without a warning.” It is technically true, but only when you understand the bold “if” as selecting those that were notified, without attributing a causal link to the notification! Most people would interpret that “if” differently, so this is a very misleading statement, and indeed there is a long list of people and institutions who have misunderstood it.
What next?
This is not the whole story though. The Swiss paper does find a statistically significant effect, encapsulated in this picture from the main text.
The red line above mixes two subpopulations: app users who were not notified and non app users who were not notified. Fortunately the published paper includes an additional graph in the supplementary materials (not available in the preprint):
This new graph compares app users to app users (as it should, to avoid selection biases and prosocial behaviour suspicions). It splits app users into two groups, depending on whether they got a notification or not. We see visually that the statistical effect is even stronger. Without additional information a reader of the published paper might think this is even stronger evidence of effect of the app, but in fact once we know none of those individuals attribute their quarantine to the notification, it is even more puzzling: selection effects on contact cases and prosocial effects have nothing to do with this and yet there is a statistically significant effect… so what could the potential causes be? One possibility would be that digital notifications are received preferentially in easier-to-trace cases (because contact have spent more time closer), and that there is thus simply a correlation between receiving a notification and being found quickly (by peer-to-peer or official manual contact tracing, doesn’t matter). I think this is a serious enough possibility (the only one?) that it invalidates the whole methodology of the paper.
I invite the authors to either withdraw their paper, or write a complementary one incorporating this criticism. I also invite the authors to signal these various errors to the Federal Office of Public Health. Indeed, the latter as manufacturer of the app has reporting obligations to SwissMedic (the Swiss agency for Therapeutic Products) concerning evaluations of effectiveness (and errors in these evaluations).
