Category: The Pandemic

Noah Feldman, a very smart law professor at Harvard wrote a terribly misguided column for Bloomberg on an emergency use authoritrization for a Covid 19 vaccine (https://www.bloomberg.com/opinion/articles/2020-09-04/we-should-cheer-a-covid-19-vaccine-even-from-trump-s-fda). I wrote this response and submitted it to Bloomberg but they decided not to publish it.

My mom used to tell me that just because I was able to do something, it didn’t mean I should. She would say: “think about what could go wrong first”. Well, just because the FDA has the power to issue an emergency use authorization (EUA) for a COVID-19 vaccine, doesn’t mean they should. Why? Because effectivity and safety take very different amounts of time to determine.

To see why there will be a time difference, you have to understand a bit about how vaccines are tested. The first two phases are small, often less than 500 people combined. In these phases they test for obvious safety issues (phase 1) and try to find the right dose(s) to try (phase 2). Then, if phase 1 and phase 2 trials are successful, it’s on to a much larger phase 3 trial – 30,000 people seems to be what they are using for Covid 19 trials.

In a phase 3 vaccine trial, you randomly divide the participants into two groups. Half get the vaccine candidate and the other half a placebo. Then you tell them to go about their lives. Over time, you hope to see a difference in the number of people who got infected between the two groups. You also look at the severity of the disease in people in the groups. For example, the flu vaccine protects some people and makes the disease less severe in others.

Depending on the efficacy of the vaccine, you need surprisingly few cases before you get an “efficacy signal”. For example, suppose there were 300 cases in the unvaccinated group and 100 cases in the vaccinated group. Then we will know the vaccine is 66.67% effective (300-100/300). This is way above the 50% effective threshold that the FDA said they will use for efficacy for a COVID-19 vaccine. You don’t even need a 300/100 split, even a 100/35 split would give a statistically significant signal.

So how long will it take before you get enough infections in the placebo group before you know if you have met the FDA’s 50% threshold for vaccine efficiency? For a virulent disease like Covid 19, if you are testing your vaccine in places where the disease has a high prevalence as indicated by the test positivity rate, hospitalization rates, etc., not very long. I wouldn’t be surprised if it took as little as 30 days after you have given enough people their two doses – you wouldn’t even have to have finished enrolling all 30,000 participants! That seems to be why some reckless individuals talk about “having a vaccine by November”. Still, suppose, as seems likely to me, we have an efficacy signal sometime in November, should the FDA issue an emergency use authorization? Absolutely not! There is no way we will know if the vaccine is causing low frequency (very bad) side effects by then.

Here are some examples: perhaps the scariest one occurred in 1935 when John Kolmer tested an attenuated polio vaccine. He tested in it 10,000 children (no placebo group – those were the dark ages of clinical trials). Five of these children (1 in 2000) died of polio and 10 more (1 in 1000) were paralyzed. So, these horrific side effects occurred in “only” .15% (15/10,000) of the cases. Statistical speaking this is relatively rare and so you wouldn’t necessarily find this kind of side effect that quickly.

A phase 3 trial for a proposed vaccine for Staphylococcus aureus caused more organ failure and death in patients getting the vaccine than those who didn’t get the vaccine. (It also wasn’t particularly effective.)

There is a known, though rare complication of certain vaccines called “antibody-dependent enhancement” (ADE) where a vaccine makes some people end up far worse if they catch the disease than people who never got the vaccine. The best known example of ADE has been for Dengue fever but ADE has been shown to occur with coronaviruses like the one that causes SARS and MERS in animal studies!

Let me reiterate: you simply can’t know if a Covid-19 vaccine causes low frequency bad $#%^ as quickly as you know about its efficacy.

Heck, you can’t be sure about very low frequency events even with a six month trial of 30,000 people. This is why vaccines are tracked in their first few years of use – that’s usually called a phase 4 trial. During the Ford administration, also eager for a vaccine approval before the election, the swine flu vaccine caused Guillain-Barré syndrome, a very very serious side effect, in about 1 in 5,000,000 vaccinated people.

But there is a much larger issue – even if one makes the argument that having an effective vaccine is worth risking rare side effects that might only be discovered by waiting a few more months. That is the effect on how the public sees vaccines if those rare side effects happen. Unfortunately, we live in a time where vaccines are questioned by a significant portion of the public. An EUA for any Covid 19 vaccines, based solely on a quick efficacy signal, could completely poison the well for all vaccines, we can’t risk that most of the major drug companies developing a vaccine claim they understand this. They have said they will not seek an EUA unless and until they are happy with the safety profile. I sincerely hope this is the case!

I always love back of envelope calculations so I thought I would do some for the number of freezers needed for the Pfizer vaccine which, as I pointed out in my last post, needs a -80C medical grade freezer. First off, let’s assume we want to have lots of vaccination stations and we want to get everyone vaccinated within a few months. That means I will assume each vaccine station gets a 3 cubic foot -80C freezer- they cost about 13k each retail, say 10k wholesale?

https://www.thermofisher.com/order/catalog/product/ULT390-10-A#/ULT390-10-A

Each box holds 6,300 2ml vials which I will assume are roughly enough for one person’s dose, round down to 6,000 as is customary in a back of envelope calculation. (And yes I know it’s possible they will use larger vials that hold more individual doses but for a back of envelope calculation, I don’t think that is relevant.)

Since the vaccine requires two doses a month apart, for the sake of this back of envelope calculation, I’m just going to assume we want to have storage for 125 million doses per treatment center at a time because we want to distribute 500 million doses over a month. We need to distribute the vaccines via a truck that would also store the vaccine at -80C so let’s assume we will have the capacity for 75 million doses stored on trucks as they move to the treatment centers from the manufacturing centers. So the trucks themselves need a fair number of freezers because ordinary refrigerated trucks don’t come close to -80C.

O.K. we need to store 200 million doses by assumption because we need to duplicate some capacity to deal with both transport and treatment stations. So we get about 34,000 of these freezers needed at a cost of about 340 million. Not bad $$ wise, but that’s a lot of freezers that we will need to manufacture relatively quickly – and these are not your garden variety chest freezers, since they have to be ultra-reliable at -80C. Also, we need to equip each of the delivery trucks with a 50amp diesel generator to power the freezers – they probably run about 3k each and let’s assume each truck gets two large 20 cubic feet -80C freezers that each hold roughly 40,000 doses, so each truck can deliver 80,000 doses. (Turns out the cost of a 20 cubic foot freezer isn’t much more than a 3 cubic ft freezer interesting enough: https://www.thermofisher.com/order/catalog/product/ULT390-10-A#/ULT390-10-A) 

Anyway, we then need roughly 1,000 diesel generator equipped trucks then to transport the vaccine doses around,, so using a figure of say $3,000/per generator that is a basically irrelevant $3 million for the generators.

Now, of course, you can assume that the vaccine will take, say a year to distribute, and the numbers come down considerably – by a factor of maybe 25. I think I’m not going to hope for that and maybe a six-month rollout is what is going to happen. So my back of envelope calculation leads to about three or four thousand freezers needed, which seems far more doable.

Well, no this isn’t actually the right simplification and my mistake  shows the trouble with back of envelope calculations. leave out something and they go all cattywampus. In this case, I think we also have to have enough vaccine stations to mitigate the number of people who can reasonably wait in line or in their cars. If we don’t, the lines for vaccines will make the lines we saw for testing look like a walk in the park. Oh, and yes we also need to have at least one vaccine station in each town of say more than 10,000 people and less than 25,000 people. (There are 1,572 towns with a population of between 10,000 and 25,000 in the United States.) Anyway, looking at this table: https://www.statista.com/statistics/241695/number-of-us-cities-towns-villages-by-population-size/ and assuming we want a minimum of one vaccine station for each town and say at least one per 25,000 people, I get that even for a slower roll-out, we may need at least 10,000 -80C freezers – and maybe more. 

(Of course, all this may not be necessary. The Moderna mRNA vaccine looks like it can be shipped and stored in ordinary freezer trucks. Other vaccine candidates seem to have even less severe storage requirements.)

I was hoping to finish a blog about the latest press release by Moderna on their Phase 1 trial of their vaccine candidate for seniors. (Sheesh I wish they would release preprints instead of press releases…)

But I just saw this and wanted to bring it to the attention of my readers. It’s going to complicate the distribution of mRNA viruses that is for sure:

“Executives from Moderna and Pfizer on Wednesday separately told the Centers for Disease Control and Prevention’s Advisory Committee on Immunization Practice on Wednesday that mRNA-1273, which is Moderna’s coronavirus vaccine candidate, requires a storage temperature of negative 4 degrees Fahrenheit. BioNTech and Pfizer’s candidates, BN1162b2 and BNT162b2, need to be stored in negative 94 degrees Fahrenheit.” (https://www.marketwatch.com/story/moderna-and-pfizers-covid-19-vaccine-candidates-require-ultra-low-temperatures-raising-questions-about-storage-distribution-2020-08-27)

That means dry ice level cold storage for Pfizer’s and an ordinary freezer for Moderna’s. A quick Google search shows that even a small -80c (-112f) freezer holding 6,000 doses costs about 13k by the way, so the costs of the number of freezers needed won’t be trivial. And, more importantly, I am willing to bet we don’t have enough freezers to store enough doses of the vaccine. (My next blog will do a back of envelope calculation of costs and number of freezers needed.) So, I sure hope someone is thinking about getting them manufactured in the right quantities, now.

Oh, and yes, if it turns out that the Pfizer candidate is the only vaccine that proves really effective in the first batch, poor countries, with minimal infrastructure are so screwed, and even delivering the vaccine to rural parts of the United States requires an infrastructure, I’m also willing to bet, we don’t yet have.

Understanding the difference between these two concepts is vital to understanding the results of clinical trials. Heck, it’s vital to understanding just about every intervention in medicine. Get your patient to stop smoking and (obviously) both their absolute risk and relative risk of getting lung cancer drops dramatically. But the numbers are quite different, and as you will see absolute risk reduction is always a smaller number then relative risk reduction, sometimes a dramatically smaller number. Confusing them is a horrible mistake for a medical professional. 

I’d venture to say that every doctor, epidemiologist, MPh is supposed to understand the difference between the two concepts just like they should understand the difference between other basic concepts like systolic vs diastolic blood pressure. So, I find it crazy that FDA Commissioner Stephen Hahn M.D. forgot or confused them. In any case, he was forced to issue an incredible mea culpa after the crazy press conference on convalescent plasma a few days ago.

“I have been criticized for remarks I made Sunday night about the benefits of convalescent plasma. The criticism is entirely justified,” the commissioner said in a string of tweets. “What I should have said better is that the data show a relative risk reduction not an absolute risk reduction.”

Anyway, understanding the difference between these two concepts is not only important to understanding fatuous press releases from drug companies or you deciding if you want to undergo a specific treatment, it’s also important in dealing with risk assessment in all areas, so I thought I would write a blog explaining the difference.

Absolute risk reduction is the easiest to understand. Usually, it is easiest to think of this as the percentage you lessen the bad crap from happening after you do something good like regularly taking your medicine or stopping smoking – or getting a vaccine. As an example, suppose a new drug drops the percentage of patients dying from 15% of the patients to 12%. Then the absolute risk reduction is 3%.

The trouble with absolute risk reduction is that, especially for drug interventions, it is usually a small number and so hard to completely understand what it means. And what’s worse is that the behavioral economists have found that we seem wired to focus on the small (3% reduction) as the measure of the success of the intervention, missing the forest for the trees. It seems a lot of people would just say “3% doesn’t seem like such a big deal.”

And that is really really bad because any drug with this kind of absolute reduction in deaths is actually a pretty good drug. Enter relative risk reduction which does allow you to focus on the forest and not the trees. Yes, relative risk reduction is always going to result in more spectacular numbers and so, using it requires caution. But heck, if it works to get a patient to take their medicine it can be a good thing- sometimes.

Going back to our hypothetical drug, the relative risk reduction in deaths is a percentage but it is actually a little tricky to conceptualize. So seeing the calculation first is easier than understanding the definition! All you have to do is put the absolute risk reduction (3%) in the top of the fraction and the baseline risk (15%) in the bottom

Relative Risk Reduction = (3%/15%) or 20%!

Now the definition may make some sense: relative risk reduction is: “the percentage of baseline risk that is removed as a result of a treatment”.

The problem with only using relative risk reduction in an explanation is that it depends completely on the baseline. Without knowing the baseline and its size, relative risk reduction numbers alone are often useless. 

Here are some examples. Suppose the risk of getting sick is 90% and a vaccine brings this down to 50%. The absolute risk reduction is 40%. So the relative risk reduction is:

40%/90% or about 45%

That’s an OK (not great) reduction but it is going to be statistically significant for a vaccine trial which tend to be large!

But now suppose the risk of dying from a disease is 2% and a drug drops this to 1% in a trial. Then the relative risk reduction is even better:

1%/2% or 50%

but this may not be statistically significant depending on the size of a trial even though this is a larger relative risk reduction than our vaccine.

So let me end by reiterating that public health officials need to be especially careful when citing risk statistics to clearly say which they are telling you about. While confusing absolute reduction and relative reduction rates is a mortal sin, citing a relative rate reduction without a baseline is a lesser but still bad, sin. Fail to report complete information or showcase headline numbers without context because you want to have the numbers look “great”, and you will end up, like FDA commission Hahn did, looking like an idiot at a press conference.

Some people asked me for the official definition of vaccine efficacy (or VE as it is usually abbreviated). As I suspect you might be thinking, it’s a fraction expressed as a percent. For example, suppose you had 300 cases in the unvaccinated group and 100 cases in the vaccinated group. The VE is then gotten by putting the number of cases in the unvaccinated group in the bottom of the fraction (the denominator)  and the difference between the unvaccinated cases and the vaccinated cases in the top of the fraction (numerator). So in our example, this is:

VE = (300-100)/300  = 2/3 = 66.67%

Some people define it as the percentage reduction of cases you got in the vaccinated group compared to the unvaccinated group but personally I think this is one of those examples where looking at the fraction is easier to understand than parsing the words in the definition!

(unvaccinated-vaccinated)/unvaccinated

Elementary algebra then allows us to compute how many cases in the vaccinated group we can have per hundred cases in the unvaccinated group to achieve the 50% vaccine efficiency that the FDA says they will need before a vaccine meets the efficiency threshold. If X is the number of cases in the vaccinated group, we only need to solve the equation:

(100-X)/100 = .5

which gives X = 50 as you might expect. So no more than 50 cases in the vaccinated group for each 100 cases in the unvaccinated group gives us our efficacy signal.

Fauci is a great virologist, but he doesn’t seem to (automatically) do a back of envelope calculation before throwing out a statement. And so one of his most recent statements (on voting), doesn’t pass the smell test.

So what happened? Fauci said this week there is “no reason” Americans can’t vote in person for the 2020 presidential election, so long as voters follow proper social distancing guidelines amid the coronavirus pandemic.

“I think if carefully done, according to the guidelines, there’s no reason that I can see why that not be the case,” Fauci told ABC News this week. “If you go and wear a mask, if you observe the physical distancing, and don’t have a crowded situation, there’s no reason why we  shouldn’t be able to do that.”

Huh? Every 880 people standing on line¹ requires a one mile line. And, since we are voting in November, quite commonly in the cold and rain. Calculating how many people will be trying to vote at any given time is hard, one can imagine doing the voting over many days. Still, it does seem that, given the reduced number of polling stations, isn’t it likely it will be in the many hundreds pretty much continuously in many places? Anyway, one thing I do know is that you take your estimate of the number of people and divide it by 880 to convert it into miles of people standing on line!

Some people asked me how the mathematics of vaccine testing works. So, because vaccine trials are actually a bit different than a traditional drug trial, and the popular press isn’t really clear on how they work (to put it mildly), I thought I would explain, in quite general terms how they work. Unfortunately, the mathematics to fully explain how any clinical trial works is non-trivial and I haven’t figured out how to explain it without using more mathematics than I want to assume in this blog (but I’m working on it!).

First off, the way a vaccine trial works is you randomly divide the group of trial participants into two. You then give the vaccine candidate to half and a placebo to the other half. Then you tell them to go about their lives. Over time you hope to see a difference in the number of people who got infected in the two groups. You also will also look at the severity of the disease in people in the vaccinated group versus the unvaccinated group. (The flu vaccine protects some people and makes the disease less severe in others for example.) Depending on the efficacy of the vaccine, you may need less than 100 cases in the unvaccinated group before you can begin to check for efficacy.

What seems to be often confusing the media (and certainly the stock market) is that a vaccine trial has to do two things. We obviously need to determine if the vaccine is effective i.e. are people who get the vaccine significantly less likely to get the disease or if you do get the disease, is it much less severe. Both will be tracked in the trials. As I said above, the flu vaccine works both ways for example.

Anyway, determining efficacy is a relatively easy problem in biostatistics, especially if you have a vaccine that greatly reduces the chance of getting the disease or of death. For example, suppose you get 100 cases in the vaccinated group and 400 cases in the unvaccinated group. You certainly know the vaccine is effective. If only half the people die in the vaccinated group compared to those dying in the unvaccinated group, you would be very very happy.

More precisely, you compare the number of people getting the disease or getting severely ill in the vaccinated and placebo group, looking for a statistically significant difference. Then you make a determination of a range for the vaccine’s efficacy in either reducing the number of cases or their severity. A crappy vaccine may show a statistically significant reduction of cases in a big enough Phase 3 trial, but you probably wouldn’t want to use it if, say it only was 20% effective in reducing the number of cases – well unless it was very effective in reducing severity of cases. (The FDA says they will require a minimum of 50% reduction in the number of cases as the mark of efficacy for a Covid 19 vaccine.)

But what people are sometimes forgetting is that we also need to determine if it is safe. A vaccine that prevents infection but has bad side effects, can’t be approved. Alas, the media (and the stock market) seem to be focusing on the efficacy signal without realizing that safety takes far longer to determine. Yes, we may know if a vaccine candidate is effective as little as 30 days after the trial starts. (And looking at the prevalence of Covid 19 in the testing areas, I think it is quite likely.) That’s why people are talking about “having a vaccine” by November.

But you certainly won’t know only a month into a trial with 15,000 people vaccinated, if the vaccine candidate causes any low frequency bad %$^&.

Heck, you won’t be sure about very low frequency events even with a six month trial. This is why vaccines are tracked in their first few years of use – that’s usually called a Phase 4 trial by the way.

But I need to end by saying that determining safety is not only a statistical question, it is a judgment call. Us math types can pretty easily tell you if bad results in the vaccinated group are statistically significant by comparing the number of bad results in the vaccinated group with the number of bad results in the placebo group and applying certain statistical tests. We can also tell you how likely it is you will see bad results in a trial of 15,000 vaccinated people – again assuming estimates on how likely that bad result will happen. But whether those trade-offs are worth it and the vaccine is worth giving, that is not something that we have any special insights into.

But here’s the thing, speaking only as a layperson here, I think that when you are going to give something to billions of healthy people, you want to be as sure as you can of its safety before you release it into the wild for it’s “Phase 4” trials. Yes it a balance in the case of a deadly disease like Covid 19, but we could not only hurt a lot of people by an early approval, we will poison the well for any vaccine for Covid 19 by releasing one without adequate safety testing. So every time you see something like this: https://www.washingtonpost.com/washington-post-live-coronavirus-vaccines-and-treatments-pfizer-ceo-Albert-Bourla/ please, please remember the difference between efficacy and safety and be very very wary.

On a personal note, none of my medical friends who think about this stuff think you can even hope to have enough safety data in less than 6 months. And, independent of what my friends and former students are telling me, six months is also considered, by the majority of the scientific community as the absolute minimum amount of time to run a Phase 3 vaccine trial. So I would suggest we go along with the majority and not the outliers.

So I want to stress that having a vaccine this year is a pipe dream precisely because efficacy and safety are two very different questions and take different amounts of time to check.

It’s hard to present data in an insightful way. I always recommend that people check out the works of the statistician (and artist!) Edward Tufte (https://www.edwardtufte.com/tufte/) to learn more about how to do it. Graphs can be especially problematic because of how easily they can be used to mislead – even with no intention of misleading.

So it should come as no surprise that I am rarely happy with the graphs I see in the media. But I was surprised and so happy to watch this segment on Rachel Maddow. It was just about perfect in explaining why using a single graph that has daily death rates in the entire United States is so misleading, I really suggest you watch it.

https://www.msnbc.com/rachel-maddow/watch/dire-u-s-covid-19-death-rate-seen-in-graph-excluding-ny-nj-ct-88162885745

And, after that, you can watch this segment, which should depress the &*^% out of you:

https://www.msnbc.com/rachel-maddow/watch/why-trump-s-failure-to-lead-national-response-prolongs-coronavirus-crisis-88166981530

Remember syllogisms? You  know:

1. All men are mortal
2. Socrates is a man
3. Therefore Socrates is mortal

While this kind of reasoning dominated logic for 2,000 years, it is too limiting and modern (mathematical) logic goes far beyond these methods of reasoning. But hey they still work and are worth trotting out from time to time – like when deciding whether to open schools during the worst pandemic in 100 years. So here’s one I was thinking about as I listened to some of the insane discussion about opening schools on the news and from the politicians:

1. People at a high risk of dying from Covid 19 should do everything they can to avoid exposure to Covid 19.
2. Teachers are people.
3. Therefore, teachers at a high risk of  dying from Covid 19 should do everything they can to avoid exposure to Covid 19.

Now we throw in a external fact or two derived from the best science we currently have (and yes I know it does take us beyond the limiting world of the classic syllogism):

1. Roughly 18% of school teachers are in a high risk group because of their age and I don’t how many are in a high risk group because of medical issues.
2. Teens and tweens transmit the virus at least as efficiently as adults (https://wwwnc.cdc.gov/eid/article/26/10/20-1315_article) but even younger children can transmit it to adults.

Conclusion the first: At least 18% of teachers are playing Russian roulette if they step foot into a classroom in the fall and asking them to do so borders on the unspeakable.

O.K. time for another (slightly non-standard form) of a syllogism:

1. Children (especially those from lower income or immigrant communities) often live in multi-generational households.
2. Multi-generational households are very likely to contain people at a high risk of dying from  Covid 19.
3. Therefore children are often very likely to encounter people in their households who are at a high risk of dying from Covid 19.

Conclusion the second (using B above): If schools reopen, children who live in multi-generational households are very likely to transmit what is a  deadly disease to their older relatives.

I could keep on coming up with lots more syllogisms about how crazy reopening schools in your community are if your community has not “crushed the curve”, but I’m not going to. Why? Because I can’t help thinking about people like my Mom, who would certainly have been in a high risk group for most of her teaching career when I was growing up, or how older adults in poor or immigrant communities are the ones most likely to die from a rush to open schools. 

This particular defiance of elementary logic applied to the question of opening schools soon,  just makes this mathematician sick to his stomach.

Derek Lowe has his take on them and I was pleased to see that I basically got it right.

https://blogs.sciencemag.org/pipeline/archives/2020/07/20/new-data-on-the-cansino-vaccine

https://blogs.sciencemag.org/pipeline/archives/2020/07/20/new-data-on-the-oxford-az-vaccine

Next, Wired has a very good article on the need to inject some realism into the discussion by the mainstream media about the vaccines. As they point out, yes we should be encouraged, but we best tone down the hype or we will give the anti-vaxers more ammunition when a sucessful vaccine has some unpleasant side effects. (Which, I hasten to say, is likely a consequence of them being effective in raising an immune response.)

https://www.wired.com/story/covid-19-vaccines-with-minor-side-effects-could-still-be-pretty-bad/