![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
(no subject)
Apr. 30th, 2020 08:55 amTW: Covid
An observation borne of spending too much time watching my facebook feed, and also of that time two years ago where I got really obsessed and read a ton of books about the 1918 pandemic:
People have been sharing various graphs of the spread of the 1918 flu on social media, generally to try to impart some sort of useful lesson about how to approach COVID this time. One I've seen several times contrasts Philadelphia's response to St. Louis. St. Louis was more aggressive in shutting down schools quickly, says the note under the graph, and therefore had less deaths. One thing Crosby makes very clear in America's Forgotten Pandemic is that this is a very tricky game to play.
First of all, you know how frustrating it is to try to look at COVID graphs and figure out how much is because of undertesting and how much is real signal? THERE WAS NO TEST FOR THE 1918 FLU UNTIL THE 1930S. All people could do is track deaths from pneumonia and hope the data was real. And moreso even than that, death and disease record keeping as we know it today largely was developed as a response to the realization after the fact that the bureaucracy had not been able to keep up with the 1918 pandemic. So the numbers you will see in these graphs are extremely inaccurate. There may be significant numbers of deaths missed entirely, for both innocent and malicious reasons.
In one of the most fascinating bits of the book, Crosby points out that the 1918 pandemic seems to have missed San Francisco's Chinatown entirely while raging through the rest of the city. Well, either that or it hit Chinatown the same as it hit the rest of the city, but racist bureaucrats in City Hall were not able to track the deaths in Chinatown. One of those two seems more likely.
Second, not only because of bad recordkeeping but also because cities are complicated eco-systems where many factors drive the spread of disease, it is extremely hard to tell when a particular intervention worked or not.
Crosby highlights the contrast between two smallish California cities, Stockton and Fresno, I want to say, but I don't recall for sure. One imposed a mandatory mask in public ordinance, the other didn't. Their graphs are not noticeably different. Here again, we could conclude that the recordkeeping simply didn't hold up, but Crosby points out that once you go past the graph you get into the qualitative details of how an ordinance is implemented. It's not clear that useage of masks differed greatly between the two cities. Some people in the city that required masks didn't wear them, some in the city that didn't require did wear them. And there wasn't as strong a stay in place requirement, so people would go outside, don a mask, visit a friend, take off their mask, then go back outside with the mask again. Protection against spread from contact with random strangers, but not protection from interacting with an asymptomatic carrier you know.
In other places, the data seems to show significant differences between cities that did and didn't have mask ordinances, but even there, Crosby cautions against drawing too much from that. After all, they didn't have N95 standards back then. The virus was small enough to pass clean through any masks of the time. And all of these cities were performing various other epidemiological interventions that may or may not have influenced the health of their citizens. And how a disease chooses to spread can seem random even without intervention. So maybe the masks helped, or maybe it was something else entirely.
It's extremely hard to assess what parts of a complex intervention made a difference and which parts didn't. This is why the emphasis we've been hearing so much from real medical experts about the value of double blind control experiments. Even those can have flaws in the methodology, but you learn a lot more firmly about what you can trust from real experiments than from trying to parse out the lessons from complicated 'natural experiments'.
But I don't make this post to caution about trying to learn lessons from the past. I write to caution about trying to learn lessons from the future. In spite of our significantly advanced science and significantly advanced bureaucracy, America and America's cities in particular remain incredibly complicated eco-systems that react unpredictably to complicated medical interventions. As we get more data about what is happening right now in the pandemic, it will be extremely tempting to read simple lessons into the data... Mayor So-and-So waited to do X and caused more deaths, Governor So-and-So did Y quickly and saved lives. Be careful of those narratives, it's extremely likely that there will be flaws in the data, and flaws in the analysis.
An observation borne of spending too much time watching my facebook feed, and also of that time two years ago where I got really obsessed and read a ton of books about the 1918 pandemic:
People have been sharing various graphs of the spread of the 1918 flu on social media, generally to try to impart some sort of useful lesson about how to approach COVID this time. One I've seen several times contrasts Philadelphia's response to St. Louis. St. Louis was more aggressive in shutting down schools quickly, says the note under the graph, and therefore had less deaths. One thing Crosby makes very clear in America's Forgotten Pandemic is that this is a very tricky game to play.
First of all, you know how frustrating it is to try to look at COVID graphs and figure out how much is because of undertesting and how much is real signal? THERE WAS NO TEST FOR THE 1918 FLU UNTIL THE 1930S. All people could do is track deaths from pneumonia and hope the data was real. And moreso even than that, death and disease record keeping as we know it today largely was developed as a response to the realization after the fact that the bureaucracy had not been able to keep up with the 1918 pandemic. So the numbers you will see in these graphs are extremely inaccurate. There may be significant numbers of deaths missed entirely, for both innocent and malicious reasons.
In one of the most fascinating bits of the book, Crosby points out that the 1918 pandemic seems to have missed San Francisco's Chinatown entirely while raging through the rest of the city. Well, either that or it hit Chinatown the same as it hit the rest of the city, but racist bureaucrats in City Hall were not able to track the deaths in Chinatown. One of those two seems more likely.
Second, not only because of bad recordkeeping but also because cities are complicated eco-systems where many factors drive the spread of disease, it is extremely hard to tell when a particular intervention worked or not.
Crosby highlights the contrast between two smallish California cities, Stockton and Fresno, I want to say, but I don't recall for sure. One imposed a mandatory mask in public ordinance, the other didn't. Their graphs are not noticeably different. Here again, we could conclude that the recordkeeping simply didn't hold up, but Crosby points out that once you go past the graph you get into the qualitative details of how an ordinance is implemented. It's not clear that useage of masks differed greatly between the two cities. Some people in the city that required masks didn't wear them, some in the city that didn't require did wear them. And there wasn't as strong a stay in place requirement, so people would go outside, don a mask, visit a friend, take off their mask, then go back outside with the mask again. Protection against spread from contact with random strangers, but not protection from interacting with an asymptomatic carrier you know.
In other places, the data seems to show significant differences between cities that did and didn't have mask ordinances, but even there, Crosby cautions against drawing too much from that. After all, they didn't have N95 standards back then. The virus was small enough to pass clean through any masks of the time. And all of these cities were performing various other epidemiological interventions that may or may not have influenced the health of their citizens. And how a disease chooses to spread can seem random even without intervention. So maybe the masks helped, or maybe it was something else entirely.
It's extremely hard to assess what parts of a complex intervention made a difference and which parts didn't. This is why the emphasis we've been hearing so much from real medical experts about the value of double blind control experiments. Even those can have flaws in the methodology, but you learn a lot more firmly about what you can trust from real experiments than from trying to parse out the lessons from complicated 'natural experiments'.
But I don't make this post to caution about trying to learn lessons from the past. I write to caution about trying to learn lessons from the future. In spite of our significantly advanced science and significantly advanced bureaucracy, America and America's cities in particular remain incredibly complicated eco-systems that react unpredictably to complicated medical interventions. As we get more data about what is happening right now in the pandemic, it will be extremely tempting to read simple lessons into the data... Mayor So-and-So waited to do X and caused more deaths, Governor So-and-So did Y quickly and saved lives. Be careful of those narratives, it's extremely likely that there will be flaws in the data, and flaws in the analysis.