Some people asked me how the mathematics of vaccine testing works. So, because vaccine trials are actually a bit different than a traditional drug trial, and the popular press isn’t really clear on how they work (to put it mildly), I thought I would explain, in quite general terms how they work. Unfortunately, the mathematics to fully explain how any clinical trial works is non-trivial and I haven’t figured out how to explain it without using more mathematics than I want to assume in this blog (but I’m working on it!).
First off, the way a vaccine trial works is you randomly divide the group of trial participants into two. You then give the vaccine candidate to half and a placebo to the other half. Then you tell them to go about their lives. Over time you hope to see a difference in the number of people who got infected in the two groups. You also will also look at the severity of the disease in people in the vaccinated group versus the unvaccinated group. (The flu vaccine protects some people and makes the disease less severe in others for example.) Depending on the efficacy of the vaccine, you may need less than 100 cases in the unvaccinated group before you can begin to check for efficacy.
What seems to be often confusing the media (and certainly the stock market) is that a vaccine trial has to do two things. We obviously need to determine if the vaccine is effective i.e. are people who get the vaccine significantly less likely to get the disease or if you do get the disease, is it much less severe. Both will be tracked in the trials. As I said above, the flu vaccine works both ways for example.
Anyway, determining efficacy is a relatively easy problem in biostatistics, especially if you have a vaccine that greatly reduces the chance of getting the disease or of death. For example, suppose you get 100 cases in the vaccinated group and 400 cases in the unvaccinated group. You certainly know the vaccine is effective. If only half the people die in the vaccinated group compared to those dying in the unvaccinated group, you would be very very happy.
More precisely, you compare the number of people getting the disease or getting severely ill in the vaccinated and placebo group, looking for a statistically significant difference. Then you make a determination of a range for the vaccine’s efficacy in either reducing the number of cases or their severity. A crappy vaccine may show a statistically significant reduction of cases in a big enough Phase 3 trial, but you probably wouldn’t want to use it if, say it only was 20% effective in reducing the number of cases – well unless it was very effective in reducing severity of cases. (The FDA says they will require a minimum of 50% reduction in the number of cases as the mark of efficacy for a Covid 19 vaccine.)
But what people are sometimes forgetting is that we also need to determine if it is safe. A vaccine that prevents infection but has bad side effects, can’t be approved. Alas, the media (and the stock market) seem to be focusing on the efficacy signal without realizing that safety takes far longer to determine. Yes, we may know if a vaccine candidate is effective as little as 30 days after the trial starts. (And looking at the prevalence of Covid 19 in the testing areas, I think it is quite likely.) That’s why people are talking about “having a vaccine” by November.
But you certainly won’t know only a month into a trial with 15,000 people vaccinated, if the vaccine candidate causes any low frequency bad %$^&.
Heck, you won’t be sure about very low frequency events even with a six month trial. This is why vaccines are tracked in their first few years of use – that’s usually called a Phase 4 trial by the way.
But I need to end by saying that determining safety is not only a statistical question, it is a judgment call. Us math types can pretty easily tell you if bad results in the vaccinated group are statistically significant by comparing the number of bad results in the vaccinated group with the number of bad results in the placebo group and applying certain statistical tests. We can also tell you how likely it is you will see bad results in a trial of 15,000 vaccinated people – again assuming estimates on how likely that bad result will happen. But whether those trade-offs are worth it and the vaccine is worth giving, that is not something that we have any special insights into.
But here’s the thing, speaking only as a layperson here, I think that when you are going to give something to billions of healthy people, you want to be as sure as you can of its safety before you release it into the wild for it’s “Phase 4” trials. Yes it a balance in the case of a deadly disease like Covid 19, but we could not only hurt a lot of people by an early approval, we will poison the well for any vaccine for Covid 19 by releasing one without adequate safety testing. So every time you see something like this: https://www.washingtonpost.com/washington-post-live-coronavirus-vaccines-and-treatments-pfizer-ceo-Albert-Bourla/ please, please remember the difference between efficacy and safety and be very very wary.
On a personal note, none of my medical friends who think about this stuff think you can even hope to have enough safety data in less than 6 months. And, independent of what my friends and former students are telling me, six months is also considered, by the majority of the scientific community as the absolute minimum amount of time to run a Phase 3 vaccine trial. So I would suggest we go along with the majority and not the outliers.
So I want to stress that having a vaccine this year is a pipe dream precisely because efficacy and safety are two very different questions and take different amounts of time to check.