Today’s post is brought to you by my sponsor, Mechanize. They’re hiring junior software engineers at $300K/year base salary.
What statistics belong in the Annals of Fake Statistics? Healthcare administration costs ballooning tens of times over in the past few years? Nearly 70,000 deaths a year due to lacking insurance coverage? You’re more dangerous to schizophrenics than they are to you? Men kill their spouses far more often than women do? (Although see.)
There are many candidates, but I would definitely tender the following fake facts:
Medical error is responsible for 251,000 American deaths a year, making it the country’s third leading cause of death;
Misdiagnosis kills 371,000 Americans per year;
Misdiagnosis leaves 424,000 Americans permanently disabled each year.
I recently saw these claims being made in a clip from an episode of the Joe Rogan Experience podcast. These claims are false, but they’ve permeated the discourse and found a way to survive despite their obvious absurdity; in fact, without proper prompting, several frontier LLMs will faithfully regurgitate these erroneous figures.
A Quarter-Million Medical Error Deaths?
This infamous estimate comes from a 2016 British Medical Journal publication by former FDA Commissioner Marty Makary and researcher Michael Daniel. The article achieves an estimate of 251,454 deaths from medical error each year by averaging four independent estimates of different quantities. I say ‘different quantities’, because none of the numbers being averaged was an estimate of the rate of death from medical error.
The first estimate comes from a 2004 estimate by HealthGrades. For Makary and Daniel’s purposes, this study looked at Medicare enrollees admitted to hospitals between 2000 and 2002 and provided an estimate of the number of deaths that could be flagged for investigation for “failure to rescue” and because the death took place in a member of a group that ordinarily has a low-mortality rate. Makary and Daniel then extrapolated the death count from this study forward based on the increase in the number of total hospitalizations by 2013, arriving at a figure of 251,454
The second estimate is taken from Classen et al.’s 2011 study on treatment-related adverse events at three tertiary care hospitals in 2004. This study involved counting up potentially treatment-related adverse events in hospitals, without evaluating whether those events were preventable or if a different pattern of care would have made them less severe. The study looked at 795 hospital admissions, found 9 fatal events, and from this, Makary and Daniel concluded that 1.13% of hospital admissions resulted in a preventable fatal event, which they chalked up to medical error and then extrapolated to total U.S. hospital admissions in 2013. But in addition to being based on an unwarranted assumption of preventability, this number is explicitly wrong and the original article’s count was corrected from 9 to 8 in a 2011 errata that Makary and Daniel should’ve seen. Using this number would drop the count from Makary and Daniel’s 400,201 to 357,502 with rounding and 356,388 without it, dropping estimated medical error deaths to 240,829 (i.e., roughly 10,000 fewer deaths).
The third estimate comes from a 2010 HHS Office of Inspector General report on a small number of Medicare beneficiaries. The report reviewed adverse events and recorded 12 fatalities among 838 events, with Makary and Daniel choosing to use the study’s 44% preventability number for adverse and temporary harm events to adjust this to an estimate of 219,579 medical error-attributable deaths. The study actually said 5 of 12 deaths were clearly preventable and 1 was likely preventable, so they should’ve chosen a rate that was either slightly higher (6/12) or slightly lower (5/12), for estimated death counts of 253,575 or 211,313. But this is a curious calculation, since it assumes the fatal error rate is the same in Medicare and non-Medicare populations, implying that, against an all-cause mortality benchmark of 3.56% for Medicare and 1.01% for non-Medicare, the fraction of deaths attributable to error is 17% versus 62% for non-Medicare, and if we bring that forward, the error-attributable fraction should have fallen for compositional reasons, contrary to their calculations.
The final estimate comes from Landrigan et al.’s 2010 estimation of temporal trends in patient harm in 10 North Carolina hospitals. The authors of this study identified 14 fatality-contributing events across 2,341 hospital admissions, judging 9 of these to be preventable. From this, they got 134,581 medical error-attributable deaths, even though that doesn’t follow from the study’s fatality-related numbers. This is because, instead of using the rate based on the 9/14 preventable events, they used the study’s overall event preventability, which included non-fatal events. The real estimate—assuming we can generalize from this sample, which we cannot—would be 136,157.
Can you grok what’s going on here? Makary and Daniels took four different rates, from four populations that weren’t nationally-representative, generally with very imprecise estimates, understood that their derived rates had to be caveated but despite that, only caveated two of four estimates, then extrapolated each rate forward in time, and called the mean of those numbers the medical error-attributable death rate when not even one of those was the medical error death rate in its study! They then stated that this was likely an underestimate because “the studies cited rely on errors extractable in documented health records and include only inpatient deaths.”
I’m going to note a few concrete issues with this. I’ll start with the claimed sources of underestimation that Makary and Daniels suggested. The first one, about errors, is almost-certainly wrong and they got the bias backwards. There is substantial empirical work on this. For example, Hayward et al. reported that, absent a large number of reviewers, rates of preventable death in U.S. hospitals tend to be overestimated. The low reliability does not result from classical error, it results in cases of non-preventable death being wrongly classified as preventable based on unreliable judgment. For two, there’s much more harm from care not delivered than from harm delivered, as out-of-hospital cardiovascular event, violent death, (etc. etc. etc.) rates make very obvious, unless we assume that inpatient deaths are actually far rarer than Makary and Daniels suggested, and medical error-attributable deaths outside of hospitals, very common. (This is inherently absurd given how little medical care occurs outside of hospitals and especially so given how little can be linked in any way to subsequent fatalities, as well as how low the post-discharge fatality rate is.)
Makary and Daniels’ underreporting objection is also silly on its face because the studies they cited explicitly dealt with it. They are correct that underreporting to incident systems does occur, but the studies they cited dealt with research reviews. The OIG explicitly found that 14% of its harm events were identified by hospital incident systems, but you can’t then extrapolate upwards, because the OIG’s estimate is based on the full set of identified events, not just the 14%.
Makary and Daniels also should not have extrapolated high medical error-attributable fatality rates forward like they did. The reason why this is wrong is because rates have almost-certainly fallen over time. Using just studies Makary and Daniels should’ve been familiar with, we know, for example, that AHRQ recorded a 17% decline in hospital-acquired conditions between 2010 and 2013, Wang et al. recorded a 4.7% annual decline in adverse events per 1,000 hospitalizations between 2005 and 2011, and their cited Landrigan et al. study discussed possible preventable harm rate declines, although it didn’t find any in its own nationally-non-representative sample. If you were to extrapolate from Makary and Daniels’ studies with improvements in care in mind and with consistent deflation of estimates to account for preventability, then medical error deaths would drop somewhere between COPD and suicides to the fourth-leading cause of death. And Makary and Daniels should’ve known this!
The most fundamental issue with Makary and Daniels’ estimate is just that if we actually try to estimate the medical error-attributable fatality rate, it’s clearly much lower than what they estimated.1
The best estimate comes from Hayward et al. This estimate was based on a review of medical records produced between 1995 and 1996 across 7 different VA medical centers. The records reviewed were oversampled for conditions with high rates of preventability. This study doesn’t have a particularly good sample, but it does have particularly good handling of its data: the authors generated data with 14 separate reviewers, standardized reviewing instruments, and then they modeled the effects of unreliability and skewness. The estimate the authors achieved was that 6% of patients would’ve left the hospital alive had they been provided the best possible care, but with 3-month prognoses and review skew taken into account, “only 0.5% of patients who died would have lived 3 months or more in good cognitive health if care had been optimal”. If we call it 2-3 preventable deaths per 10,000 admissions, the annual death count probably clocks in under 10,000, and that’s with the 1990s’ high error rate.
Other studies have provided estimates that are more relevant to medically attributable deaths than Makary and Daniels’. Rhee et al., for example, reported that, across 6 U.S. hospitals in 2014-15, just 3.7% of sepsis-associated deaths were definitely or moderately likely to be preventable. Tatara et al. found that 8.6% of deaths from non-ventilator hospital-acquired pneumonia across 4 hospitals 2016-21 were likely or very likely preventable. Bates et al. reviewed 2,809 admissions across 11 hospitals in Massachusetts and concluded that of 7 fatal adverse events, just 1 was preventable.
The most defensible recent estimate that I can come up with comes from applying Rodwin et al.’s 2020 systematic review estimate—3.1% preventability—with the AHRQ’s estimate of 769,460 inpatient deaths in 2023. This combination leads to an estimated 23,853 medical error-attributable deaths (~17-32k). If you were to apply Bates et al.’s numbers to the AHRQ’s, you’d end up at 12,004 instead. This discrepancy should give some indication of how weak estimates are, given the combination of rare events and unreliable classifications.
In short, Makary and Daniels did not produce estimates that hold up to scrutiny. They took unrepresentative and extreme samples, irrelevant estimates, figures that were later revised, numbers they—conveniently—knew had to be adjusted downwards instead of being kept as they were, and they produced an estimate that was too high. What’s worse is that the concepts they dealt with were not even coherent: to get even remotely close to their estimates, we must assume that a failure to provide optimal care is a “medical error” that realistically—not just ‘theoretically’—could’ve been prevented in an active care setting. Given the reality of hospitals, where there are not unlimited human or capital resources to get everything that could be right done correctly, the theoretical maximum estimates from ‘optimal’ care projections are still too high.
A Third of a Million Misdiagnosis Deaths?
The 371,000 deaths from misdiagnosis figure comes from Newman-Toker et al. Thsi study was a review of disease incidence data from 2012-14 that was paired with literature-derived error error and harm rates to project that each year, diagnostic errors led to 795,000 deaths or permanent disabilities in the U.S. Neither the error counts, nor the implications of the projection forward, or the split between disability and death are empirically supportable or interpretable as the authors did.
As already noted: conditions regarding error have gotten better with time and counts tend to be inflated by unreliability that’s typically unaddressed in studies. Now, more specifically, Newman-Toker et al. cited two studies—Singh et al. and Zwaan et al.—to support their conclusions about death and disability.
Singh et al. rated the potential severity of 190 diagnostic errors, and put 27 cases in the potentially fatal category—it did not report on 27 observed deaths. Most of Singh et al.’s diagnostic errors were identified through subsequent hospitalizations that were considered unexpected, meaning that—as Singh et al. noted—, the errors were likely more severe than errors generally. The estimate from Singh et al. cannot be paired with national disease counts to obtain a representative estimate of harms because it does not supply representative harm probabilities. Nor can Singh et al. be used to determine the split between error-associated deaths and disabilities, because it skews towards greater-than-expected harms to an unknown degree. Additionally, deaths following diagnosis are not necessarily deaths that a proper diagnosis would have prevented or slowed down at all.
Zwaan et al. reviewed 7,926 records sourced from 21 different Dutch hospitals, concerning admissions in 2004. They identified 80 diagnosis-related adverse events, with reviewer adjudication about the presence and preventability of adverse events, but not about their consequences. 29.1% of events were labeled as contributing to deaths, but it wasn’t clear if they were causally involved. In any case, this is an old estimate from a different country, with limited review reliability, being combined with a known-unrepresentative estimate to get a split between disabilities and deaths to somehow come together and provide us with an estimate of America’s misdiagnosis death rate.
Fundamentally, this estimate is not based on numbers that actually establish anything like the misdiagnosis-related death rate. That’s not what these sources can provide.
Half a Million Misdiagnosis Disabilities?
The figure of 424,000 permanent disabilities from misdiagnosis is the flip side to the 371,000 death figure. It is based on the same studies, and thus based on an exaggerated baseline count and an inaccurate split that leads us the Newman-Toker death and disability numbers from said count. Worse is that Singh et al. merely classified 36 of 190 diagnostic errors as potentially causing permanent damage; they did not find that 36 patients wound up with permanent disabilities due to misdiagnoses. Zwaan et al. also had a classification problem because their numbers had nothing to do with permanence: they were about disability at discharge!
The outcome that should have been measured here is new or worsened persistent disability attributable to a diagnostic error. That was not measured.2
Each of these mistaken estimates is an example of the core problem with what I’ve dubbed ‘attribution studies’. The way attribution studies work is by attempting to project the numbers of uncounted things—deaths, disability, displacements, bankruptcies, whatever—based on proxy inputs gathered from elsewhere. The problem is that if you control the inputs, you control the outputs, and if you want a given output, it’s not hard to get it. With each of these examples, the inputs were dubious, so the outputs were too, and our public discourse has been tainted as a result.
This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.
A message from my sponsor, Mechanize:
We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.
To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.
Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.
And again, they should’ve known this. The works they cited noted reservations and definitions that, had they been careful, they would’ve realized invalidated their conclusions.)
Other relevant estimates help to get us closer to what the figure should be—far lower—, but they still don’t provide exact fits for misdiagnosis-driven deaths or permanent disability rates. Generally, the more severe the sample, the higher the estimate; the more representative the sample, the lower the estimate; the better the misdiagnosis definition, the lower the estimate; etc. In short, sources of exaggeration are prevalent.
I stopped watching Rogan when he abandoned reason for madness.
On to the topic at hand, I remember feeling superior knowing these numbers growing up and knowing there was very low hanging fruit that wasn’t being picked up. But what I really needed was an anecdotal education in healthcare for old people.
An endoscopy (or literally 100 other low risk procedures) in a very frail patient could kill them, but if it did, they’d probably have died of pneumonia or some such within weeks anyway, bedridden. That these are low risk and have potentially large benefits for diagnosis and management, they’re done *all the time*. Lots of old, frail patients means lots of dying to what are not particularly scary procedures simply due to volume. But the numbers add up fast, and soon it looks like hospitals are in fact killing factories (which is silly). If you review the cases individually you would see barely any length of life lost due to “medical accidents”, and that doesn’t factor in the benefits at all.
Your numbers look right and feel right. I really like this post.
The 1999 book, To Err Is Human: Building a Safer Health System, claimed large numbers of inpatient deaths were occurring each year. Its conclusions were often discussed during my medical training from '08-'16.
So much has changed since that report/book was published, and I've long wondered what a better system of reporting harms, errors, injuries, and deaths world look like. With the incredible spread of electronic health records since about 2008, it is sad that more reliable vital statistics are still so frustrating to get and riddled with the types of complications you described (and more!)