Experts demolish studies suggesting COVID-19 is no worse than flu

Frustrated statisticians and epidemiologists took to social media this week to call out substantial flaws in two widely publicized studies trying to estimate the true spread of COVID-19 in two California counties, Santa Clara and Los Angeles.

The studies suggested that far more people in each of the counties have been infected with the new coronavirus than thought—that is, they estimated that true case counts in the two counties are up to 85 times and 55 times the number of the currently confirmed cases in the counties, respectively. Accordingly, this suggests that COVID-19 is far less deadly than thought. The large case counts in relation to unchanged number of deaths put COVID-19’s fatality rate in the same range as seasonal flu.

How dangerous is this?

We dig into the details of the studies below, but it's important to note that neither of them have been published in a scientific journal, nor have they gone through standard peer-review for scientific vetting. Instead, they have been posted online in draft form (a commonplace occurrence amid a rapidly evolving pandemic that inclines researchers to have fast access to data, however uncertain).

The findings seemed to support minority arguments that COVID-19 may be no worse than seasonal flu (a leading cause of death in the US) and that the restrictive mitigation efforts currently strangling the economy may be unnecessary. In fact, three researchers who co-authored the new studies have publicly made those exact arguments.

In a controversial opinion piece in the biomedical news outlet STAT, population health researcher John Ioannidis, at Stanford, argued back in mid-March that the mortality rate of COVID-19 may be much lower than expected, potentially making current lockdowns “totally irrational.” Health policy researchers Eran Bendavid and Jay Bhattacharya, also both at Stanford, made a similar argument in the Wall Street Journal at the end of March. They called current COVID-19 fatality estimates—in the range of 2 percent to 4 percent—“deeply flawed.”

Ioannidis is a co-author of the study done in Santa Clara county, and Bendavid and Bhattacharya were leading researchers on both of the studies, which appeared online this month.

The new studies seem to back up the researchers’ earlier arguments. But a chorus of their peers are far from convinced. In fact, criticism of the two studies has woven a damning tapestry of Twitter threads and blog posts pointing out flaws of the studies—everything from basic math errors to alleged statistical sloppiness and sample bias.

In a blog review of the Santa Clara county study, statistician Andrew Gelman of Columbia University detailed several troubling aspects of the statistical analysis. He concluded:

I think the authors of the above-linked paper owe us all an apology. We wasted time and effort discussing this paper whose main selling point was some numbers that were essentially the product of a statistical error.

I’m serious about the apology. Everyone makes mistakes. I don’t think they authors need to apologize just because they screwed up. I think they need to apologize because these were avoidable screw-ups.

A Twitter account from the lab of Erik van Nimwegen, a computational systems biologist at the University of Basel, responded to the study by tweeting the quip “Loud sobbing reported from under reverend Bayes' grave stone.” The tweet refers to Thomas Bayes, an 18^th century English reverend and statistician who put forth a foundational theorem on probability.

Pleuni Pennings, an evolutionary biologist at San Francisco State University, noted in a blog regarding the Santa Clara study that “In research, we like to say that ‘extraordinary claims require extraordinary evidence.’ Here the claim is extraordinary but the evidence isn’t. Also, we learn that even if a study comes from a great university—this is no guarantee that the study is good.”

Harvard epidemiologist Marc Lipsitch stated on Twitter that he concurred with similar statistical criticisms made online. He added a “kudos” to the authors for conducting the study and “providing one interpretation of it (which supports their ‘it's overblown’ view).”

So what has all of these researchers up in arms?

The aim of the studies

Both studies primarily aimed to estimate how many people in each of two counties had been infected at some point with SARS-CoV-2. This is an extremely important endeavor because it can tell us the true extent of infection, help guide efforts trying to stop transmission, and better assess the full spectrum of the COVID-19 disease severity and the fatality rate.

Because diagnostic testing has been so limited in the US and many COVID-19 cases appear to present with mild or even no symptoms, researchers expect the true number of people who have been infected is much higher than we know based on confirmed cases. There is no debate about that. But just how much higher is the subject of considerable debate.

The researchers went about their studies by recruiting small groups of residents and testing their blood for antibodies against SARS-CoV-2. Antibodies are Y-shaped proteins that the immune system makes to target specific molecular foes, such as viruses. If a person has antibodies that recognize SARS-CoV-2 or its components, that suggests the person was previously infected.

Santa Clara

In the Santa Clara county study, researchers recruited volunteers using Facebook and had them come to one of three drive-through test sites. They ended up testing the blood of 3,330 adults and children for antibodies. They found 50 blood samples, or 1.5 percent, were positive for SARS-CoV-2 antibodies.

They then adjusted their figures to try to estimate what positive tests they would have gotten back if their pool of volunteers better matched the demographics of the county. The volunteer pool skewed toward certain zip codes in the county and were enriched for women and white people relative to the county’s real makeup. The researchers’ adjustment ended up nearly doubling the prevalence of positives, bringing them from 1.5 percent to an estimated 2.8 percent.

They then adjusted the data again to account for inaccuracies of the antibody test. There are two metrics for accuracy here: sensitivity and specificity. Sensitivity relates to how good the test is at correctly identifying all true positives. Specificity relates to how good the test is at correctly identifying all the true negatives—in other words, avoiding false positives.

According to the authors of the Santa Clara study, the sensitivity and specificity data on their antibody test led them to estimate that the true prevalence of SARS-CoV-2 infections ranged from 2.49 percent and 4.16 percent.

Based on the population of the county, that would suggest somewhere between 48,000 and 81,000 people in the county had been infected. The confirmed case count at the time of publication was only 956. That puts their infection estimate 50 to 85 times higher than the confirmed cases.

The researchers then estimated an infection fatality rate (IFR) with that large number of estimated infections and an estimate of only 100 cumulative deaths (including from infections at the time. Deaths lag behind initial infections, potentially for weeks). They calculated an IFR of 0.12 percent to 0.2 percent. This falls in the ballpark of seasonal flu, which has an estimated fatality rate of about 0.1 percent.

Los Angeles

There is less data available from the Los Angeles study. In an unusual move—even by today’s pandemic standards—the findings were initially announced in a press release from the county’s public health department, which provided little in the way of statistical and methodological details. A short draft of the study (PDF found here) has also circulated online, but it still has less information on the methods than the Santa Clara study. Also, the draft has even higher prevalence estimates than the press release. It’s unclear why the estimates differ, but we’ll mainly focus on the conclusions formally released from the health department.

Generally, for the study, researchers used data from a market research firm to randomly select residents and invite them to get tested at one of six testing sites. The researchers set up quotas for participants by age, gender, race, and ethnicity to match the population characteristics of the county. Their goal was to recruit 1,000 participants.

They tested 863 adults using the same antibody test used in the Santa Clara study, which was made by Premier Biotech, of Minneapolis, MN. Of the tests given, 35 (or 4.1 percent) were positive. According to the press release, the adjusted data suggested that 2.8 percent to 5.6 percent of the county’s population had been infected with the new coronavirus.

Given the county’s population, that suggests that 221,000 to 442,000 adults in the county had been infected. That estimate is 28- to 55- times higher than the 7,994 confirmed case count at the time. As in the Santa Clara study, that puts the IFR in the range of 0.3 percent to 0.13 percent, closer to the IFR of seasonal flu.

Comments are closed.