Estimating Effectiveness of Influenza and COVID-19 Vaccines: The “Test-Negative” Design

According to recent news reports, HHS decided to initially delay publication of, and then not publish, a CDC-led study that estimates the effectiveness of the COVID-19 vaccine over the past winter season. The study had already passed internal reviews by CDC scientific and editorial teams, and its methodology, as indicated by a reportedly leaked copy of the manuscript, is one that is widely used by researchers around the world for studying respiratory virus vaccine effectiveness. Only a few weeks prior, CDC had published a study on influenza vaccine effectiveness using the same methodological approach, and the approach has also been used by CDC in similar studies of COVID-19 vaccine effectiveness in prior years. An HHS spokesperson said the decision not to publish the study was due to concerns from National Institutes of Health Director and acting Director of the Centers for Disease Control and Prevention (CDC) Jay Bhattacharya regarding the method used to calculate vaccine effectiveness.  To provide context for public discussion of the unpublished study, this policy brief provides an overview of the methodological approach in question – the test-negative case control design – and discusses some of its strengths and limitations in assessing vaccine effectiveness for respiratory diseases like influenza and COVID-19.

What is vaccine effectiveness and how do scientists estimate it?

CDC defines vaccine effectiveness (VE) as “how well vaccination works under real-world conditions to protect people against health outcomes such as symptomatic illness, hospitalization, and death.” This contrasts with “vaccine efficacy,” which refers to how well a vaccine works in ideal, controlled conditions such as a clinical trial. At a basic level, a VE study compares health outcomes (like illness or hospitalization) and vaccination status in different groups of people, in order to generate an estimate of the level of protection that a vaccine provides against those health outcomes. Researchers typically calculate the magnitude of the difference – the level of protection – between groups as a percentage (e.g., a vaccine can have a VE of 50% against illness if that illness was 50% less common in the vaccinated group compared to the unvaccinated group). There are several epidemiological approaches for conducting these types of studies, including “observational” approaches such as cohort studies that follow groups of vaccinated and unvaccinated people over time and measure how frequently health outcomes occur in each group, and case-control studies that identify groups of people with a particular health outcome (cases) and those without that outcome (controls) and determine the frequency of vaccination in each group. A common observational approach used to study VE for respiratory illnesses is a type of case-control study called the “test-negative” design. An “experimental” design such as a randomized controlled trial (where participants are randomized into vaccinated and unvaccinated groups and the vaccine is compared to a placebo) of an existing, FDA approved vaccine is generally considered unethical, meaning observational approaches such as the test-negative design are preferable when assessing VE for vaccines already widely used and shown to be safe and effective.   

What is the test-negative study design?

In recent years the “test-negative” design has become the most prevalent approach for studying and monitoring VE of COVID-19 and influenza vaccines in “real world” situations. The test-negative design is a version of the case-control approach where researchers define a clinical outcome of interest (such as influenza-like-illness or respiratory disease symptoms), identify people seeking health care who present with these same symptoms, then test individuals for the pathogen of interest (such as influenza virus or COVID-19). Those that test positive for the pathogen are cases and those that test negative are controls, and researchers can then compare how many in each group were vaccinated. With this information at hand, researchers can determine VE, which is calculated as 1 minus the odds ratio (the odds of vaccination in cases over the odds of vaccination in controls).  For example, in a hypothetical study with 200 participants, 100 were test positive for infection (cases) and 100 were test-negative (controls). When vaccination status is checked, it is distributed across groups as shown in Table 1.

In this case, the odds of vaccination among cases is 30/70 = 0.43, and in controls it is 60/40 = 1.5.  The odds ratio would be 0.43/1.5 = 0.29. Therefore VE in this hypothetical study would be (1 – 0.29) x 100% = 71%, indicating that the vaccine was about 71% effective at preventing the outcome among the vaccinated compared to the unvaccinated.   

Researchers will also typically use statistical techniques to control for differences between groups in other characteristics, such as age, comorbidities, and prior infection history.  The as-yet unpublished CDC study of COVID-19 VE and the recently published study of influenza VE use the test-negative approach. It is also the approach often used to estimate VE in respiratory disease research networks for influenza and COVID-19 in the United Kingdom, Australia, Canada, and across Europe.

What are the strengths and weaknesses of this approach?

The test-negative approach, which relies on people seeking care and health facilities for identifying cases and controls, has become the dominant study design for several reasons. It is often easier to conduct a test-negative design compared to other approaches because both cases and controls are identified in the same location presenting with similar symptoms, whereas other approaches might require seeking controls from the community to match with cases. It’s oftenefficient because it uses existing surveillance and diagnostic testing infrastructure at health care facilities. The test-negative approach also reduces the likelihood of a common bias in case-control designs, which is the potential for differences in health care-seeking behavior between cases and controls. That is, vaccinated people may systematically differ from unvaccinated people in ways that affect healthcare-seeking behavior, which creates the potential for biased VE estimates when community controls are used.

However, there are also methodological issues and potential biases that can occur with test-negative designs. Care must be taken in defining and correctly identifying persons with the illness or health outcome of interest, so that consistency is maintained over time and potentially across multiple locations. It is important to identify study participants systematically according to pre-defined criteria. The quality and consistency of the tests used is very important, as some types of tests (such as rapid diagnostic tests) are less sensitive/specific compared to others (such as RT-PCR or viral culture); lower quality tests can lead to misclassification of cases vs. controls. In addition, in situations where widespread or mandatory testing is in place (for example, when all incoming patients in a facility are tested for COVID-19, as was common for a period during the pandemic), then asymptomatic individuals could be classified as cases even though they may differ in important ways from other cases that are symptomatic or have more severe disease. There are also potential concerns about how to handle prior infections and prior vaccinations because if the groups differ in a systematic way in these areas it can introduce a bias to the study. For COVID-19 studies, in particular, VE estimates unadjusted for infection history can underestimate VE. There are other potential biases that exist in test negative designs as well. Table 2 presents a summary of key strengths and weaknesses of the test-negative design for studying COVID-19 and influenza.

As with all epidemiological studies, poor study design, inconsistent implementation, and failure to take into consideration important biases and confounders can lead to misleading results. However, when well-designed and implemented with consistency and attention to detail, test-negative study designs can produce accurate estimates of vaccine effectiveness. In fact, studies have shown that effective test-negative designs can produce results highly consistent with randomized trials (the “gold-standard” of epidemiological study designs) when compared directly. The reportedly leaked manuscript of the disputed COVID-19 vaccine study indicates methods that are in line with previously published VE studies of influenza and COVID-19 vaccines, suggesting that the current controversy could be a result of increased scrutiny of COVID-19 vaccine effectiveness studies at this particular moment.