Looking at stuff through the eyes of a mathematician – Gary Cornell's blog

I always love back of envelope calculations so I thought I would do some for the number of freezers needed for the Pfizer vaccine which, as I pointed out in my last post, needs a -80C medical grade freezer. First off, let’s assume we want to have lots of vaccination stations and we want to get everyone vaccinated within a few months. That means I will assume each vaccine station gets a 3 cubic foot -80C freezer- they cost about 13k each retail, say 10k wholesale?

https://www.thermofisher.com/order/catalog/product/ULT390-10-A#/ULT390-10-A

Each box holds 6,300 2ml vials which I will assume are roughly enough for one person’s dose, round down to 6,000 as is customary in a back of envelope calculation. (And yes I know it’s possible they will use larger vials that hold more individual doses but for a back of envelope calculation, I don’t think that is relevant.)

Since the vaccine requires two doses a month apart, for the sake of this back of envelope calculation, I’m just going to assume we want to have storage for 125 million doses per treatment center at a time because we want to distribute 500 million doses over a month. We need to distribute the vaccines via a truck that would also store the vaccine at -80C so let’s assume we will have the capacity for 75 million doses stored on trucks as they move to the treatment centers from the manufacturing centers. So the trucks themselves need a fair number of freezers because ordinary refrigerated trucks don’t come close to -80C.

O.K. we need to store 200 million doses by assumption because we need to duplicate some capacity to deal with both transport and treatment stations. So we get about 34,000 of these freezers needed at a cost of about 340 million. Not bad $$ wise, but that’s a lot of freezers that we will need to manufacture relatively quickly – and these are not your garden variety chest freezers, since they have to be ultra-reliable at -80C. Also, we need to equip each of the delivery trucks with a 50amp diesel generator to power the freezers – they probably run about 3k each and let’s assume each truck gets two large 20 cubic feet -80C freezers that each hold roughly 40,000 doses, so each truck can deliver 80,000 doses. (Turns out the cost of a 20 cubic foot freezer isn’t much more than a 3 cubic ft freezer interesting enough: https://www.thermofisher.com/order/catalog/product/ULT390-10-A#/ULT390-10-A) 

Anyway, we then need roughly 1,000 diesel generator equipped trucks then to transport the vaccine doses around,, so using a figure of say $3,000/per generator that is a basically irrelevant $3 million for the generators.

Now, of course, you can assume that the vaccine will take, say a year to distribute, and the numbers come down considerably – by a factor of maybe 25. I think I’m not going to hope for that and maybe a six-month rollout is what is going to happen. So my back of envelope calculation leads to about three or four thousand freezers needed, which seems far more doable.

Well, no this isn’t actually the right simplification and my mistake  shows the trouble with back of envelope calculations. leave out something and they go all cattywampus. In this case, I think we also have to have enough vaccine stations to mitigate the number of people who can reasonably wait in line or in their cars. If we don’t, the lines for vaccines will make the lines we saw for testing look like a walk in the park. Oh, and yes we also need to have at least one vaccine station in each town of say more than 10,000 people and less than 25,000 people. (There are 1,572 towns with a population of between 10,000 and 25,000 in the United States.) Anyway, looking at this table: https://www.statista.com/statistics/241695/number-of-us-cities-towns-villages-by-population-size/ and assuming we want a minimum of one vaccine station for each town and say at least one per 25,000 people, I get that even for a slower roll-out, we may need at least 10,000 -80C freezers – and maybe more. 

(Of course, all this may not be necessary. The Moderna mRNA vaccine looks like it can be shipped and stored in ordinary freezer trucks. Other vaccine candidates seem to have even less severe storage requirements.)

I was hoping to finish a blog about the latest press release by Moderna on their Phase 1 trial of their vaccine candidate for seniors. (Sheesh I wish they would release preprints instead of press releases…)

But I just saw this and wanted to bring it to the attention of my readers. It’s going to complicate the distribution of mRNA viruses that is for sure:

“Executives from Moderna and Pfizer on Wednesday separately told the Centers for Disease Control and Prevention’s Advisory Committee on Immunization Practice on Wednesday that mRNA-1273, which is Moderna’s coronavirus vaccine candidate, requires a storage temperature of negative 4 degrees Fahrenheit. BioNTech and Pfizer’s candidates, BN1162b2 and BNT162b2, need to be stored in negative 94 degrees Fahrenheit.” (https://www.marketwatch.com/story/moderna-and-pfizers-covid-19-vaccine-candidates-require-ultra-low-temperatures-raising-questions-about-storage-distribution-2020-08-27)

That means dry ice level cold storage for Pfizer’s and an ordinary freezer for Moderna’s. A quick Google search shows that even a small -80c (-112f) freezer holding 6,000 doses costs about 13k by the way, so the costs of the number of freezers needed won’t be trivial. And, more importantly, I am willing to bet we don’t have enough freezers to store enough doses of the vaccine. (My next blog will do a back of envelope calculation of costs and number of freezers needed.) So, I sure hope someone is thinking about getting them manufactured in the right quantities, now.

Oh, and yes, if it turns out that the Pfizer candidate is the only vaccine that proves really effective in the first batch, poor countries, with minimal infrastructure are so screwed, and even delivering the vaccine to rural parts of the United States requires an infrastructure, I’m also willing to bet, we don’t yet have.

Understanding the difference between these two concepts is vital to understanding the results of clinical trials. Heck, it’s vital to understanding just about every intervention in medicine. Get your patient to stop smoking and (obviously) both their absolute risk and relative risk of getting lung cancer drops dramatically. But the numbers are quite different, and as you will see absolute risk reduction is always a smaller number then relative risk reduction, sometimes a dramatically smaller number. Confusing them is a horrible mistake for a medical professional. 

I’d venture to say that every doctor, epidemiologist, MPh is supposed to understand the difference between the two concepts just like they should understand the difference between other basic concepts like systolic vs diastolic blood pressure. So, I find it crazy that FDA Commissioner Stephen Hahn M.D. forgot or confused them. In any case, he was forced to issue an incredible mea culpa after the crazy press conference on convalescent plasma a few days ago.

“I have been criticized for remarks I made Sunday night about the benefits of convalescent plasma. The criticism is entirely justified,” the commissioner said in a string of tweets. “What I should have said better is that the data show a relative risk reduction not an absolute risk reduction.”

Anyway, understanding the difference between these two concepts is not only important to understanding fatuous press releases from drug companies or you deciding if you want to undergo a specific treatment, it’s also important in dealing with risk assessment in all areas, so I thought I would write a blog explaining the difference.

Absolute risk reduction is the easiest to understand. Usually, it is easiest to think of this as the percentage you lessen the bad crap from happening after you do something good like regularly taking your medicine or stopping smoking – or getting a vaccine. As an example, suppose a new drug drops the percentage of patients dying from 15% of the patients to 12%. Then the absolute risk reduction is 3%.

The trouble with absolute risk reduction is that, especially for drug interventions, it is usually a small number and so hard to completely understand what it means. And what’s worse is that the behavioral economists have found that we seem wired to focus on the small (3% reduction) as the measure of the success of the intervention, missing the forest for the trees. It seems a lot of people would just say “3% doesn’t seem like such a big deal.”

And that is really really bad because any drug with this kind of absolute reduction in deaths is actually a pretty good drug. Enter relative risk reduction which does allow you to focus on the forest and not the trees. Yes, relative risk reduction is always going to result in more spectacular numbers and so, using it requires caution. But heck, if it works to get a patient to take their medicine it can be a good thing- sometimes.

Going back to our hypothetical drug, the relative risk reduction in deaths is a percentage but it is actually a little tricky to conceptualize. So seeing the calculation first is easier than understanding the definition! All you have to do is put the absolute risk reduction (3%) in the top of the fraction and the baseline risk (15%) in the bottom

Relative Risk Reduction = (3%/15%) or 20%!

Now the definition may make some sense: relative risk reduction is: “the percentage of baseline risk that is removed as a result of a treatment”.

The problem with only using relative risk reduction in an explanation is that it depends completely on the baseline. Without knowing the baseline and its size, relative risk reduction numbers alone are often useless. 

Here are some examples. Suppose the risk of getting sick is 90% and a vaccine brings this down to 50%. The absolute risk reduction is 40%. So the relative risk reduction is:

40%/90% or about 45%

That’s an OK (not great) reduction but it is going to be statistically significant for a vaccine trial which tend to be large!

But now suppose the risk of dying from a disease is 2% and a drug drops this to 1% in a trial. Then the relative risk reduction is even better:

1%/2% or 50%

but this may not be statistically significant depending on the size of a trial even though this is a larger relative risk reduction than our vaccine.

So let me end by reiterating that public health officials need to be especially careful when citing risk statistics to clearly say which they are telling you about. While confusing absolute reduction and relative reduction rates is a mortal sin, citing a relative rate reduction without a baseline is a lesser but still bad, sin. Fail to report complete information or showcase headline numbers without context because you want to have the numbers look “great”, and you will end up, like FDA commission Hahn did, looking like an idiot at a press conference.

Some people asked me for the official definition of vaccine efficacy (or VE as it is usually abbreviated). As I suspect you might be thinking, it’s a fraction expressed as a percent. For example, suppose you had 300 cases in the unvaccinated group and 100 cases in the vaccinated group. The VE is then gotten by putting the number of cases in the unvaccinated group in the bottom of the fraction (the denominator)  and the difference between the unvaccinated cases and the vaccinated cases in the top of the fraction (numerator). So in our example, this is:

VE = (300-100)/300  = 2/3 = 66.67%

Some people define it as the percentage reduction of cases you got in the vaccinated group compared to the unvaccinated group but personally I think this is one of those examples where looking at the fraction is easier to understand than parsing the words in the definition!

(unvaccinated-vaccinated)/unvaccinated

Elementary algebra then allows us to compute how many cases in the vaccinated group we can have per hundred cases in the unvaccinated group to achieve the 50% vaccine efficiency that the FDA says they will need before a vaccine meets the efficiency threshold. If X is the number of cases in the vaccinated group, we only need to solve the equation:

(100-X)/100 = .5

which gives X = 50 as you might expect. So no more than 50 cases in the vaccinated group for each 100 cases in the unvaccinated group gives us our efficacy signal.

Fauci is a great virologist, but he doesn’t seem to (automatically) do a back of envelope calculation before throwing out a statement. And so one of his most recent statements (on voting), doesn’t pass the smell test.

So what happened? Fauci said this week there is “no reason” Americans can’t vote in person for the 2020 presidential election, so long as voters follow proper social distancing guidelines amid the coronavirus pandemic.

“I think if carefully done, according to the guidelines, there’s no reason that I can see why that not be the case,” Fauci told ABC News this week. “If you go and wear a mask, if you observe the physical distancing, and don’t have a crowded situation, there’s no reason why we  shouldn’t be able to do that.”

Huh? Every 880 people standing on line¹ requires a one mile line. And, since we are voting in November, quite commonly in the cold and rain. Calculating how many people will be trying to vote at any given time is hard, one can imagine doing the voting over many days. Still, it does seem that, given the reduced number of polling stations, isn’t it likely it will be in the many hundreds pretty much continuously in many places? Anyway, one thing I do know is that you take your estimate of the number of people and divide it by 880 to convert it into miles of people standing on line!

Some people asked me how the mathematics of vaccine testing works. So, because vaccine trials are actually a bit different than a traditional drug trial, and the popular press isn’t really clear on how they work (to put it mildly), I thought I would explain, in quite general terms how they work. Unfortunately, the mathematics to fully explain how any clinical trial works is non-trivial and I haven’t figured out how to explain it without using more mathematics than I want to assume in this blog (but I’m working on it!).

First off, the way a vaccine trial works is you randomly divide the group of trial participants into two. You then give the vaccine candidate to half and a placebo to the other half. Then you tell them to go about their lives. Over time you hope to see a difference in the number of people who got infected in the two groups. You also will also look at the severity of the disease in people in the vaccinated group versus the unvaccinated group. (The flu vaccine protects some people and makes the disease less severe in others for example.) Depending on the efficacy of the vaccine, you may need less than 100 cases in the unvaccinated group before you can begin to check for efficacy.

What seems to be often confusing the media (and certainly the stock market) is that a vaccine trial has to do two things. We obviously need to determine if the vaccine is effective i.e. are people who get the vaccine significantly less likely to get the disease or if you do get the disease, is it much less severe. Both will be tracked in the trials. As I said above, the flu vaccine works both ways for example.

Anyway, determining efficacy is a relatively easy problem in biostatistics, especially if you have a vaccine that greatly reduces the chance of getting the disease or of death. For example, suppose you get 100 cases in the vaccinated group and 400 cases in the unvaccinated group. You certainly know the vaccine is effective. If only half the people die in the vaccinated group compared to those dying in the unvaccinated group, you would be very very happy.

More precisely, you compare the number of people getting the disease or getting severely ill in the vaccinated and placebo group, looking for a statistically significant difference. Then you make a determination of a range for the vaccine’s efficacy in either reducing the number of cases or their severity. A crappy vaccine may show a statistically significant reduction of cases in a big enough Phase 3 trial, but you probably wouldn’t want to use it if, say it only was 20% effective in reducing the number of cases – well unless it was very effective in reducing severity of cases. (The FDA says they will require a minimum of 50% reduction in the number of cases as the mark of efficacy for a Covid 19 vaccine.)

But what people are sometimes forgetting is that we also need to determine if it is safe. A vaccine that prevents infection but has bad side effects, can’t be approved. Alas, the media (and the stock market) seem to be focusing on the efficacy signal without realizing that safety takes far longer to determine. Yes, we may know if a vaccine candidate is effective as little as 30 days after the trial starts. (And looking at the prevalence of Covid 19 in the testing areas, I think it is quite likely.) That’s why people are talking about “having a vaccine” by November.

But you certainly won’t know only a month into a trial with 15,000 people vaccinated, if the vaccine candidate causes any low frequency bad %$^&.

Heck, you won’t be sure about very low frequency events even with a six month trial. This is why vaccines are tracked in their first few years of use – that’s usually called a Phase 4 trial by the way.

But I need to end by saying that determining safety is not only a statistical question, it is a judgment call. Us math types can pretty easily tell you if bad results in the vaccinated group are statistically significant by comparing the number of bad results in the vaccinated group with the number of bad results in the placebo group and applying certain statistical tests. We can also tell you how likely it is you will see bad results in a trial of 15,000 vaccinated people – again assuming estimates on how likely that bad result will happen. But whether those trade-offs are worth it and the vaccine is worth giving, that is not something that we have any special insights into.

But here’s the thing, speaking only as a layperson here, I think that when you are going to give something to billions of healthy people, you want to be as sure as you can of its safety before you release it into the wild for it’s “Phase 4” trials. Yes it a balance in the case of a deadly disease like Covid 19, but we could not only hurt a lot of people by an early approval, we will poison the well for any vaccine for Covid 19 by releasing one without adequate safety testing. So every time you see something like this: https://www.washingtonpost.com/washington-post-live-coronavirus-vaccines-and-treatments-pfizer-ceo-Albert-Bourla/ please, please remember the difference between efficacy and safety and be very very wary.

On a personal note, none of my medical friends who think about this stuff think you can even hope to have enough safety data in less than 6 months. And, independent of what my friends and former students are telling me, six months is also considered, by the majority of the scientific community as the absolute minimum amount of time to run a Phase 3 vaccine trial. So I would suggest we go along with the majority and not the outliers.

So I want to stress that having a vaccine this year is a pipe dream precisely because efficacy and safety are two very different questions and take different amounts of time to check.

It’s hard to present data in an insightful way. I always recommend that people check out the works of the statistician (and artist!) Edward Tufte (https://www.edwardtufte.com/tufte/) to learn more about how to do it. Graphs can be especially problematic because of how easily they can be used to mislead – even with no intention of misleading.

So it should come as no surprise that I am rarely happy with the graphs I see in the media. But I was surprised and so happy to watch this segment on Rachel Maddow. It was just about perfect in explaining why using a single graph that has daily death rates in the entire United States is so misleading, I really suggest you watch it.

https://www.msnbc.com/rachel-maddow/watch/dire-u-s-covid-19-death-rate-seen-in-graph-excluding-ny-nj-ct-88162885745

And, after that, you can watch this segment, which should depress the &*^% out of you:

https://www.msnbc.com/rachel-maddow/watch/why-trump-s-failure-to-lead-national-response-prolongs-coronavirus-crisis-88166981530