This week, Johnson & Johnson began delivering millions of doses of coronavirus vaccine across the United States after obtaining an emergency permit from the Food and Drug Administration. The most important point in getting the green light was a test that Johnson & Johnson performed to measure the effectiveness of the vaccine.
Efficacy is an important concept in vaccine trials, but it is also difficult. For example, if a vaccine has an efficacy of 95%, it does not mean that 5% of the people receiving the vaccine get COVID-19. And just because one vaccine gets a higher efficacy rating than the other in trials, does not necessarily mean it is better. Here’s why.
For statisticians, effectiveness is a measure of how much a vaccine reduces the risk of an outcome. Johnson & Johnson, for example, noted how many people who still received a vaccine received COVID-19. Then they compared it to how many people got COVID-19 after getting a placebo.
Sign up for The Morning New York Times newsletter
The difference in risk can be calculated in percentage. Zero percent means that vaccinated people run just as many risks as people who received the placebo. One hundred percent means that the risk has been completely eliminated by the vaccine. At the US test site, Johnson & Johnson determined that the efficiency is 72%.
Efficiency depends on the details of a trial, such as where it took place. Johnson & Johnson conducted trials at three locations: in the United States, Latin America, and South Africa. The overall efficiency was lower than in the United States alone. One reason for this seems to be that the trial took place in South Africa after a new variant hit the country. The variant called B.1.351 has mutations that make it possible to evade some antibodies produced by vaccination. However, the variant did not render the vaccine useless. Far from it: in South Africa, Johnson & Johnson’s efficiency was 64%.
Efficiency can also change when scientists look at different outcomes. Johnson & Johnson’s vaccine, for example, had an 85% efficacy rate against severe cases of COVID-19. This is important to know because it means that the vaccine will prevent many hospitalizations and deaths.
When scientists say that a vaccine has an effectiveness of, for example, 72%, it is a point estimate. This is not an exact prediction for the general public, as trials can only look at a limited number of people – in the case of Johnson & Johnson’s trial, about 45,000 volunteers.
The uncertainty surrounding an estimate of the point can be small or large. Scientists suggest this uncertainty by calculating a series of possibilities, which they call a confidence interval. One way of thinking about a confidence interval is that we can trust 95% of it that efficiency falls into it. If scientists used this method to find confidence intervals for 100 different samples, the efficiency would fall within 95 of the confidence intervals.
Confidence intervals are strict for trials in which many people become ill and there is a sharp difference between the outcomes in the vaccinated and placebo group. If few people get sick and the differences are small, the confidence intervals can explode.
Last year, the FDA set a goal for coronavirus vaccine trials. Each manufacturer must demonstrate that a vaccine has an efficacy of at least 50%. The confidence interval should not fall below 30%. A vaccine that meets the standard offers the kind of protection found in flu vaccines – and will therefore save many lives.
So far, three vaccines – made by Pfizer and BioNTech, Moderna and Johnson & Johnson – have been approved in the United States after their trials showed that they exceeded the FDA’s threshold. AstraZeneca and Novavax, which are constantly undergoing US trials, have published efficacy results from studies in other countries. Meanwhile, the manufacturers of the Sputnik V vaccine have published results based on their trial in Russia.
For a number of reasons, it is not possible to make an exact comparison between these vaccines. One vaccine may have a higher estimate than the other, but their confidence intervals may overlap. This effectively makes their results indistinguishable.
To complicate matters, the vaccines have been tested on different groups of people at different stages of the pandemic. In addition, their effectiveness was measured in different ways. For example, Johnson & Johnson’s effectiveness was measured 28 days after a single dose, whereas Moderna was measured 14 days after a second dose.
What is clear is that all three of the vaccines authorized in the United States – made by Johnson & Johnson, Moderna and Pfizer and BioNTech – significantly reduce the risk of getting COVID-19.
What’s more, all the vaccines seem to have a high effectiveness against more serious outcomes such as hospitalization and death. No one who received Johnson & Johnson’s vaccine, for example, had to go to the hospital for a COVID-19 infection after receiving an injection. Sixteen people who received the placebo did so. This equates to 100% efficiency, with a confidence interval of 74.3% to 100%.
A clinical trial is just the beginning of the investigation into any vaccine. Once widely used, researchers track its performance. Instead of efficacy, these scientists are now measuring effectiveness: how much the vaccine reduces the risk of a real-world disease, in millions of people rather than in thousands. Early studies on the effectiveness of coronavirus vaccines confirm that it offers strong protection.
In the coming months, researchers will be monitoring this data to see if it becomes less effective – either because the immunity to the vaccine is declining, or because a new variant is emerging. In both cases, new vaccines will be created, and manufacturers will provide new standards for their effectiveness.
This article originally appeared in The New York Times.
© 2021 The New York Times Company