The idea that vaccines cause autism rose to prominence after a fraudulent 1998 paper in The Lancet argued that the combination measles, mumps, and rubella (MMR) vaccine caused a fake condition dubbed “autistic enterocolitis”. The paper was eventually retracted, but the damage was done; public discourse had been seeded with the notion that vaccines might cause autism. Various poor arguments making the case followed and, nowadays, around 10% of the U.S. population seems to believe that vaccination leads to autism.

There’s ample evidence that makes it nearly certain that vaccines do not cause autism, but the literature on this subject is actually surprisingly limited in terms of how much of it comes from causally informative studies. Most of the extant evidence has to do with mechanistic reasoning, cohort studies, and international comparisons.1 Though these are convincing, they’re not enough for certainty. To that end, I’ve compiled a wealth of causal evidence that you can access at this website: vaccineeffects.com.

The way my method works is simple. Different jurisdictions implemented vaccine mandates and rollouts at different times. The first affected cohorts should show jumps in autism rates that remain elevated above the preexisting trend afterwards. Combined with data on actual uptake, these jumps allow us to estimate the extent to which autism rates rose as a result of vaccines being given out, because earlier cohorts won’t have been exposed to nearly the same degree. Or, restated graphically:

So, what happens when we leverage this design on the universe of available data? This:

For those unfamiliar with risk ratios, a ratio of 1 means no difference between groups. Numbers above 1 denote positive effects—increases—in the treated group and numbers below 1 indicate negative effects—reductions. When the confidence interval goes over 1, that means the effect is not statistically significant (i.e., the p-value is greater than 0.05 in the case of 95% confidence intervals). As you can see, the effect of vaccination is not statistically significant or even anywhere near being significant: the central estimate is 0.97—a reduction in autism, if anything—and the 95% confidence interval ranges from 0.84 to 1.13, ruling out even modest effects in either direction. Or, in other words, strong, causal evidence now shows: vaccines do not cause autism.2

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

Spend enough time online and you’ll probably hear claims like:

• 89% of Americans have lived paycheck-to-paycheck in the last year;

• The proportion of Americans living paycheck to paycheck has changed from 50% in 2022 to 69% in 2025 before falling to 48% in 2026;

• Most Americans live paycheck to paycheck;

• 78% of Americans are living paycheck to paycheck, a six percent increase in just one year;

• 76% of Americans are one paycheck away from poverty;

• 67% of Americans live paycheck to paycheck;

• 66% of Americans live paycheck to paycheck;

• 63% of Americans consider themselves to be living paycheck to paycheck;

• 63% of Americans have been living paycheck to paycheck since the pandemic began;

• 48% of Americans are currently living paycheck to paycheck;

• A majority of Americans earning $100,000 are living paycheck to paycheck;

• 36% of those earning $250,000 or more live paycheck to paycheck;

• Americans living paycheck to paycheck own 60% of credit card debt;

• 37% of Americans lack enough money to cover a $400 emergency expense;

• A majority of Americans cannot afford a $1,000 emergency expense;

• 69% of Americans have less than $1,000 in savings.

Read enough of these headlines and you’ll get the impression that the American precariat is large—perhaps unsustainably so. But, notice too, that these headlines disagree with one another. How, for example, could it be true that 48% and 78% of Americans are living paycheck-to-paycheck? Which is right? The links contain numerous internal inconsistencies, too. For example, PYMNTS’ estimates show a higher annual average percentage (70%) than the monthly readings it summarizes.

Something is amiss!

The truth is that the concept of living paycheck-to-paycheck is poorly measured. It is both inconsistently defined to mean many different things and the samples used to measure it are often highly unrepresentative. Plus, surveys are reported on carelessly.

Consider the claim that 37% of Americans lack the money to cover a $400 emergency expense. That seems extreme, but that’s because it’s not an accurate description of the quantity. The actual thing being measured is ‘Would you cover a $400 emergency expense with cash or its equivalents?’, and the follow-up question of ‘Could you pay it by any means?’ was answered in the affirmative by 88% of people.1

If you ask people directly whether they’re living paycheck-to-paycheck, most—55% here—say ‘yes’. However, of that sample, just 21% of them define that condition as being unable to pay monthly bills without their next paycheck. This very directly gives the game away: most people don’t have a good definition of what it means to live paycheck-to-paycheck, and that’s reflected across different surveys, with their wildly varying numbers and implications.

If we want to obtain a realistic estimate of the portion of Americans living paycheck-to-paycheck, we have to think about what we’re trying to measure. For example, if we’re talking about cushion beyond the next paycheck, the Survey of Consumer Finances (SCF) data for 2025 suggests the best estimate is 23%, with a range of 17-34% depending on pay frequency and cushion threshold. If we pare that down to just people for whom there’s no cushion because of low income—defined as income below two-times the poverty line plus bad year shocks—, the number becomes 9-13%. If the number is households earning $100,000 per year or more with no cushion, it’s 11%, but this drops to 6% if they’re also not saving.

If we use a more expansive definition of paycheck-to-paycheck, then in the SCF and the Fed’s Survey of Household Economics and Decisionmaking (SHED) we get to about 47% of households fitting the criteria. But most of these households also hold some savings each month, so they’re clearly not ‘paycheck-to-paycheck’ because they’re actively able to build cushion. But what about just going based on self-reported struggle? In the SHED, that’s 27-30% reporting they’re “Just getting by”.

If we want to get a best estimate, none of the above will do. We need a standard, because these all disagree while possibly being correct and also possibly being biased in their own ways. So, my procedure to get there using the SCF involves, first, figuring out who has no cushion. A household counts as having no cushion if the cash it can reach would run out before the next paycheck, meaning that they’re living hand-to-mouth. This covers checking, savings, money market and prepaid accounts, plus directly held stocks, bonds, funds, and savings bonds, minus credit card balances and excluding retirement accounts and home equity, with ‘no cushion’ meaning that the amount of those things is at or below half of one two-week paycheck. That’s 23.3% of households.

We then need to do some subtraction. We need to see who can’t afford a cushion by comparing each household’s usual yearly income with the federal poverty line for their size. Household size is the head, the spouse or partner, and their children, and if they’re below twice the poverty line, we can—reasonably, but debatably—count the household as being unable to afford a cushion due to non-lifestyle reasons. About 22% of households fall under that line and 42% have no cushion, representing 9.1% of all households.

Next, we need to figure out who are precarious due to having an unusually bad year. About half of above-line households report that their income last year was more than 10% below its usual level, or their expenses were unusually high. With a bad year, 23.3% have no cushion, and without one, 12.8% do. We count only the gap, 10.5 points, applied to households that had a bad year, adding up to 4.2% of all households.

Finally, we do some addition. 9.1% plus 4.2% equals 13.3% of households, or about 18 million. The remaining 10% have no cushion, adequate income, and no bad year to blame for their finances. Then, we weight each household by its size, giving 13.7% of people, or about 42-47 million people.2

So, how many Americans are living paycheck-to-paycheck? It’s a judgment call ranging from ~7-18% depending on how you treat the threshold, non-nuclear families, and so on. But what it is not is an estimate like ‘65% of Americans live paycheck-to-paycheck.’

But Americans Are Starving in the Streets!

If we use the Fed’s SHED, we see that some 7.8% of adults report sometimes or often (2.3% often) feeling like they’re unable to get enough food. These people are considered ‘food-insecure’, and even though they may not be living paycheck-to-paycheck per se, if we take their claims seriously, then they are definitely economically struggling. But there are three big issues here. Before getting to them, let’s look at some numbers:

The first issue to note is that food insecurity is paradoxically associated with obesity. This might seem odd, but recall: in the modern world, it’s the poor who tend to be fatter. This is precisely because they are not starving, or even anywhere near being at-risk of it except in the most rare and extreme of cases.3

The second issue to note is that food insecurity is associated with reporting differences. Namely, the food-insecure report eating about the same amounts of calories as the food-secure while also reporting that they ate much less than usual. It may be the case that, to the food-insecure, a day without what would be a binge to normal people is anomalously scant for them.

The final issue is that there’s a missing right tail of events driven by food-insecurity. Namely, wasting away and starving: virtually no one actually starves due to lack of food in the U.S. To be fair, there is a way to claim otherwise, but it’s based on a misreading of death certificates. To get to that misreading, look at this chart:

Notice how America has seen a massive rise in malnutrition-related deaths over the past decade or so? This has been used to make the case that America is falling behind and apart. But it’s an illusory phenomenon driven by diagnostic changes, not a substantial increase in malnutrition. Specifically, in October 2007, Medicare began paying for severe malnutrition, so while claims for malnutrition death were flat in following years, the share classified as severe began to rise. Next, the Academy of Nutrition and Dietetics and the Parenteral and Enteral Nutrition Society published new criteria making it easier to diagnose malnutrition in the sick and elderly in 2012. Finally, in 2014, Medicare stopped letting hospices use “adult failure to thrive” and “debility” as main diagnoses, so malnutrition became the natural substitute. As a result, we see a rise where we’d expect: old people dying at home or in nursing homes.4

That missing right tail also shows up with the very small number of deaths listed as being due to lack of food alone in the National Center for Health Statistics’ (NCHS’) data on underlying causes of death for 2024. In that data, we can see that there were 22 strict starvation from lack of food deaths in the whole United States. Even this number overstates how many Americans are actually starving to death. The reason this number is an exaggeration is because the deaths are not really about starvation.

Two of America’s 22 lack of food deaths in 2024 were young children who died at home and the incidents were ruled accidents—that’s caregiver failure. Three involved dementia patients, one of whom was designated with a failure to thrive. Another four involved alcohol dependence, cirrhosis, liver failure, and chronic pancreatitis. Two involved swallowing problems among very old—78, 81—patients. One case was a 67-year-old woman dying at home with wasting due to self-neglect. Eight cases involved serious illness getting in the way of eating, including heart and kidney failure, COPD, diabetes, and a broken hip, with most of the cases being among the elderly. There were just two cases with nothing else listed: a 40-year-old man who died at home with dehydration and an 83-year-old man in the hospital, with no explanations for either case provided.

America has virtually abolished true poverty of the sort that leads people to lose their lives. It is a society so wealthy, filled with such abundance, and so obviously so, that when someone claims Americans are dying on the streets from empty stomachs or that they’re genuinely living hand-to-mouth and paycheck-to-paycheck with no cushion against adversity, we should look at them with at least a hint of revulsion, because they’re clearly trying to lie to us.

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

3

Here’s how that looks distributionally. This view suggests that there’s higher variance among the food-insecure. That could be driven by people genuinely struggling to get food, but given the sex difference, that seems dubious.

In 2015, El Salvador was the murder capital of the world. Now it has the lowest murder rate in the western hemisphere—below even Canada.

El Salvador’s high homicide rate was actually an American product. Hundreds of thousands of Salvadorans fled during the country’s civil war circa 1980-92, and many of those refugees ended up in Los Angeles. Some number of those people got involved in gang violence. Naturally, they gravitated towards establishing their own ethnic gangs, giving birth to MS-13 and, in some cases, joining the existing 18th Street Gang. Until 1996, deporting these people was difficult, but when Congress passed the Illegal Immigration Reform and Immigrant Responsibility Act (IIRIRA), that changed.

The first wave of IIRIRA deportees included some 500 Salvadoran gang members who were sent back to the country not long after it had demobilized its army and only just started building a police force. The flow of criminals was staggering, and it couldn’t have come at a worse time for El Salvador.

Criminal Salvadorans brought the Los Angeles gang model with them. They began carving neighborhoods into territories, setting up lookouts along major roads and interchanges, and doing whatever they could to keep police, informants, and members of rival gangs out while keeping ‘their people’ in. They began to monopolize the urban cityscape of San Salvador and then spread this habit to other cities. Eventually, they were able to capture whole swathes of the population in gangland.

One recent paper found that residents right inside gang territories were 50 percentage points more likely to work inside said territory and 12 percentage points less likely to work for a large firm compared to people living just 50 meters away from them. Living in gangland also evidently lowered their incomes and made it harder to obtain an education, because these differences didn’t exist prior to the arrival of the gangs.

What made these gangs so much more violence-prone than the gangs and cartels in other Latin American countries is that they lacked access to the lucrative drug trade. Instead, they lived off extortion: of households, of shopkeepers, even of bus lines, strippers, fishermen, tourists, bureaucrats, and banks. The gangs were a mixture of poor and predatory to all, which made for a decidedly unpopular combination.

Mano Dura

El Salvador spent two decades alternating between different failed strategies for dealing with the gangs: truces and Mano Dura—the Iron Fist. The Iron Fist amounted to a generally mild crackdown, and because it failed to go far enough, it backfired in some ways. Take the 2003-04 instantiation. This resulted in filled-up prisons, but the prisons ended up becoming centralized headquarters for the gangs.

In 2012, the government asked for a truce: gangsters outside of prison would do less violence in exchange for moving gang leaders out of maximum security, to more favorable prison conditions. This halved the homicide rate overnight, but the truce didn’t hold and the terms of peace eroded, resulting in open urban warfare between gangs and police forces in 2015.

What ultimately broke this multi-decadal tension between gangs and government was Bukele declaring enough to be enough. After a particularly violent weekend in which the gangs killed 87 people in 72 hours, Nayib Bukele and the public had enough. Bukele was suddenly politically empowered to declare the régimen de excepción—the state of exception—and to begin El Salvador’s first real, extensive crackdown.

The police began snatching up gang members identified by their tattoos. This is generally where the non-Salvadoran understanding of what the government did ends, but it goes substantially further than this. In fact, various gangs had already told their members to stop getting facial tattoos after the first Mano Dura. By 2022, many gangsters were unblemished by ink.

The crackdowns were really a community effort and many Salvadorans were proud to snitch on their local gangster, often putting their lives at risk to do so. Police were also happy to finally go after everyone they had been monitoring over the years. The effort ultimately broke the gangs and drove the homicide rate to where it lies today by putting them behind bars at a truly stunning scale.

Are We Sure?

There are many people who would like to dismiss El Salvador’s success for whatever reason. The most commonly given explanation given for how El Salvador achieved its low homicide rate is regression to the mean: they were at the highest rate in the world, so the only place to go from there was down. There are many problems with this explanation. For one, El Salvador’s homicide rate decline was the most stunning ever:

This is highly unusual. Some part of it could be regression to the mean, but we know it’s not because we can take the analysis to a much more granular level.1 In fact, we can see that Bukele’s initial, pre-crackdown territorial control take-back plan cut homicides right away; the state of exception cut them even further at a more rapid clip:

This movement from a homicide rate that was normal for the region to a level that’s exceptional for the entire hemisphere was clearly directly policy-driven and unprecedented. Moreover, it’s not just homicides that were affected. Under the earlier Control Plan, the government had evidently reached an agreement with the gangs, resulting in fewer homicides, but no less extortion. When Bukele actually went after the gangs and finally decided to put them all in prison, reported extortion fell too:

Is Bukele Suspicious?

Nayib Bukele is right-wing with both a libertarian and an authoritarian bent, and his methods were decidedly conservative-coded to the rest of the world. It is unsurprising, then, that he’s generated a lot of distrust from the largely left-leaning establishment involved with NGOs. Reports have been made about human rights abuses, wrongful detentions, secret deals with gangs that are the real explanation for the murder rate going down… Frankly, there’s a lot people have said, but it’s all highly exaggerated.

Consider three points.

First, surveys run by outsiders agree with the public sentiment assessed within the country and with the election results.2 Trust in the National Police is up, criminal victimization is down, and feeling safe to walk alone at night is way up. Plus, people even believe in democracy more (all of this, among other things).

Second, falsification tests don’t look suspect. Gangs didn’t cause traffic deaths and they weren’t noted for sexual violence. Those things were unaffected by the crackdown. If Bukele’s government was just lying to make everything look good, presumably these would improve too.

Third, other potential sources of homicides that have gone uncounted have also been declining. This refers to police killings—usually of gang members—, disappearances, and bodies found in mass graves dug up from the era when the country was embroiled in a civil war. All of this is down, leaving little in the way of room to argue that there are hidden deaths not entering the count. Even adding every disappearance report to the 2024 murder rate, it’d be down 67% from 2021 and 89% from 2015.

But What Did It Cost?

El Salvador did suspend due process to get the crackdown done. This resulted in close to 2% of the population being incarcerated, largely without trials.

The crackdown has also resulted in about 8,000 people being wrongly detained because they were mixed up with gang members. More tragically, somewhere between 265 and 504 people died in state custody between April 2022 and December 2025. But these numbers must be kept in context. For the wrongfully imprisoned, they didn’t lose their lives, just a few months or years. For those who died in prison for whatever reason, their numbers pale in comparison to the number of deaths averted by the crackdown. Given how many lives were saved, it’s hard to argue that the crackdown wasn’t worth it.

Salvadorans agree: they strongly support Bukele, his government, the crackdown, and its continuance. They remember the hell of life under the gangs and they don’t want to return to it. The benefits far outweigh the costs for the vast majority of the population.3

Does Fixing the Murder Rate Help the Growth Rate?

I visited El Salvador back in January of this year. I noted how it definitely feels safe. Random people I spoke to praised Bukele on the streets, talked about how different things are, left their doors unlocked, and I was even fine to walk around what felt like a shantytown without any real sense of danger. The country is safe; far safer, in fact, than the norm for the other Latin American countries I’ve been to (most of them).

One thing I speculated about was that even though El Salvador is now clearly safe, this might not improve the growth rate. It also might not even improve the crime rate in the longer term, because the void left by the gangs might eventually get filled—not by murder, so much as by pettier forms of crime. But I hope I’m wrong. It would be incredibly nice if safety reforms were enough to make an economy boom.

Growth in El Salvador does not stand out through 2025.

Comparing El Salvador to a synthetic version of it assembled from its neighbors as a benchmark reveals that it’s not doing that much better than its peers. In fact, without Bolivia in the sample, its boost over peers is not yet significant.

There are signs of change; it’s not all dismal. For example, domestic investment has boomed. Unfortunately, foreign investment hasn’t followed suit. Construction activity is up 48% over 2019, tourism earnings—clearly downstream of improved safety—have doubled in the same period, and new business registrations—also likely downstream of gang cleanups—have doubled just as well.

One method that’s commonly used to evaluate countries’ growth is to look at nightlights. When economic activity increases, consumption tends to rise, and that leaves a signature we can see from space because people want to have their lights on for longer, at greater intensities and utilizations, and so on. This has been used to show that tariffs hurt economic growth in targeted countries, that COVID lockdowns hurt economic activity, that authoritarian countries fudge their GDP numbers, and more. El Salvador does not look exceptional by this metric:

Household surveys also don’t show improvements to economic activity. Household incomes, earnings, employment, and women’s labor force participation are all not different comparing areas with high pre-crackdown violence to those with lower levels.

The most encouraging recent number is that school attendance is up for late teens—the ones most likely to have been wrapped up with gangs. But this is not a very big result, and it’s unclear if this will translate to benefits for the country down the line.

The best evidence that the crackdown paid off in economic terms comes from a recent study that showed $20-$25 per month greater incomes in municipalities with above-median gang presence.

The paper’s measure of gang presence is based on a policy registry of gang members, and it’s not public. Drats! An earlier paper from these authors defined municipal gang presence as the 18 municipalities in which deported gang leaders were born. Fine! I used this measure and found that, in percentage terms, there’s basically identical growth between places where gang leaders were and were not born. This comes with the impression of higher gains to those places with gang leaders because they had higher initial incomes.4 This suggests the signal in the paper is misleading about total economic benefits to the country.

Is ‘El Salvador’ Repeatable?

Can other countries just do what Bukele did, lock up their criminals and call it a day? Theoretically, yes, but it’s difficult. The conditions in El Salvador were perfect for fixing their crime problem because of the unique nature of their poor gangs. Other countries have nominally attempted to do what Bukele did, and they’ve failed.

Ecuador and Honduras both made attempts, but their issues were not ripe for a fix, and their efforts were too partial, not going nearly far enough to actually make a dent. Ecuador and Honduras didn’t have the leverage El Salvador did. El Salvador fought poor gangs that relied on extortion, not cocaine cash that can finance resistance and pull in successor groups; El Salvador targeted everyone simultaneously instead of focusing on single groups so as to avoid power vacuums getting filled; El Salvador retook the prisons from the gangs; El Salvador went all-in and arrested almost 2% of the population, something no one else attempted to do.

In a sense, El Salvador’s experience is definitely repeatable. A good candidate country would be Brazil, where a large share of the homicides are due to gangs and other forms of organized crime. But try doing this in the U.S. and you’d have a harder time because such a comparatively small share of the homicides can be attributed to gang violence. Incarcerating more people to deal with much more random sorts of violence is possible and would work at some scale, but it’d end up being more expensive and less durable.

Let’s Summarize

El Salvador crushed its gangs, rendering its violent crime and homicide rates among the lowest in the western hemisphere. This feat is impressive and undoubtedly real. This has led some commentators to speculate that El Salvador will now experience rapid economic growth, that it’ll take off because it’s a safe place with a government that can reliably guarantee the streets are clear of crime and the people are free from the abuses of their fellow men. Tourists? Investors? Safe, so it should grow.

But so far, this just hasn’t happened. Order has arrived, but it’s not ‘institutions’, and it’s not yet growth either. It’s an open question where El Salvador goes now. Over the coming years, we will learn if the security miracle will lead to an economic one.5

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

3

Worldwide Governance indicators, of course, suggest El Salvador has largely gotten worse though. Voice and accountability are down, political stability is up, and overall, El Salvador is described as a worse place to live. This is clearly wrong given the material conditions on the ground and how the citizenry feel about them.

5

All the information needed to reproduce these results and more and to get to the data in these figures is in this little reproduction package I asked Claude to assemble into a PDF. Let me know if it or I made any errors!

Replication

4.07MB ∙ PDF file

Download

Download

I post quite a bit about one of my favorite AI companies, Pangram. I think Pangram makes an amazing product: they’ve got the only working AI text detector, and they’re also quick to update it in response to malicious text humanizers that crop up. I trust the Pangram team and I know their product is real and works as it says on the tin. Why, then, do so many writers seem to disagree?

If you read about Pangram, you’ll quickly see a bunch of people claiming that it doesn’t work. Worse, they’ll sometimes claim that it’s biased. Pangram allegedly discriminates against autistic authors, against anti-vaccine authors, against people I’d politely describe as ‘conspiracy theorists’ on both sides of the aisle, and, most prominently, against African writers. How? Well, it supposedly flags their work as AI!

There are a lot of articles about Pangram and other AI detectors ‘mistakenly’ flagging Black and African authors as using AI to do or assist with their writing. The most prominent piece that comes to mind for me was published in the Pan African Review. The piece was an unusual breakout for the Review: the journal generally enjoys a pretty limited circulation. I suspect this is because of the import of the topic area and the utility of providing an example to point to when people want to explain why they were ‘wrongly’ flagged as using AI for writing. So, I’ll consider the perspective of this piece to be emblematic of a community’s wider concerns given how much it resonated.

The piece is authored by one Timi Olagunju. Olagunju is “a lawyer and public policy expert working at the intersection of tech, governance, and development.” His claim is that writing in formal English will be flagged as AI-generated by AI text detector:

Consider an international student in the United States. She has spent years being told that her English must be grammatically correct. She avoids slang, checks every verb and structures each paragraph carefully. Then a detector labels her work artificial because it is controlled and predictable. First, the institution tells her to master standard English through IELTS and TOEFL. Then it treats mastery as suspicious.

This allegedly explains why “African (or black) writers” are liable to have their work flagged by AI detectors:

For many African writers like Falade, English is both a language and an inheritance shaped by colonial history. In Nigeria, Kenya, Ghana and other former British colonies, generations of students were taught forms of English influenced by British education. Teachers emphasized correctness, formal composition and structure.

Is this the case? We can test this right away by inputting Olagunju’s own article into Pangram and… it receives a 100% AI score with high confidence. Surely he wouldn’t use AI to write, right? Let’s consider his thesis in good faith and perish the thought—for now. Surely the other writers in the Pan African Review wouldn’t use AI to write either, right?

To test the extent of AI adoption in the Pan African Review, I looked at how much Pangram flagged articles there over time. Unfortunately, the site is currently down, so I had to extract whatever articles I could from the Internet Archive.1 What we see right away is a total rejection of Olagunju’s thesis: articles were never flagged as AI-generated before the advent of LLMs, and the share flagged as AI-written has rapidly crept up afterwards. This is not what we expect if African writing generally is to be considered liable to be flagged as AI.

Another test of Olagunju’s thesis is to simply look at his writing. He wouldn’t lie about this stuff, would he? All of his past work before LLMs came out should return a 100% AI score with at least moderate confidence. Alas, quite literally none of it that I could find does. All of it shows up as human-written. It’s only after LLMs came out that his work started being flagged as AI-generated.

But maybe we’re at fault here. Maybe Pangram trained on the Pan African Review and Olagunju’s work and they flag pre-LLM work as AI-generated because it’s in the training set. To get at this question, I asked Pangram CEO if that was the case: was “Pangram trained on the articles in the Pan African Review?” “It was not.”

Pangram doesn’t discriminate—not against Africans, not against the disabled, not against people who think mRNA vaccines contain nanobots. Pangram detects AI.2

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

2

Let those who disagree bring forward evidence like I did. To check all the Pangram results I ran, open up this PDF generated by our friend DeepSeek V4.1 Flash:

Pangrammed

244KB ∙ PDF file

Download

Download

It’s widely understood that anti-vaccine activism is predicated on illiteracy, conspiracist ideation, and a resistance to updating in response to empirical findings, but it’s less well-known that anti-vaccine activists (anti-vaxxers, colloquially) are also generally innumerate and—despite their invocation of drawn-out arguments, biological mechanisms, and chains of causation—they don’t think through the numbers they like to cite. The evidence of this is everywhere in their arguments, but nowhere is it more distinct than when they try to invoke statistics in support of anecdotes. Consider the McDowell triplets.

The story goes:

All three of Brenda McDowell's triplets, she attests, 'shut down' within hours of each other after receiving their routine pneumococcal vaccines — later to be diagnosed with Autism Spectrum Disorder.

Alternatively:

Healthy triplets. One pneumococcal vaccine. One single day. Within hours, all three babies changed forever.

Claire screamed uncontrollably… then went completely silent.

Her brothers Richie and Robbie lost their smiles, their sounds, their milestones.

All three diagnosed with severe autism. Only one has ever partially recovered.

And on the odds, anti-vaccine activists have provided three estimates. For example, “A geneticist explained that the chance to this happening to 2 of our triplets would be 1 in 4 million. It happened to all 3.” Alternatively, anti-vaccine activist Steve Kirsch has proffered two probabilities: “The chance of that happening if autism is happens randomly to kids is less than 1.66e-13” and, from the same person:

To see 3 (or more) events in a specific 24-hour window when you expected to see 3e-5 events is:

>>> poisson.sf(2,3e-5)
4.499898751214992e-15

We’d want to multiple this by 1,000 since it could happen on any of the days for the first child. This is still orders of magnitude from ever happening.

All of this might sound convincing if you ignore the discrepancies1 and also don’t think about it for even a few seconds. But if you put any amount of thought into this, it falls apart. It obviously does not demonstrate any risk from vaccines. Think about it: if this is true, and vaccination caused all of these children to suddenly become autistic, what would the implied effect of vaccination be on the odds of autism? We can calculate that!2

If 90% of people get vaccinated and the prevalence of autism is 1%, the implied odds ratio is 4.893. Moreover, the rate of autism among the unvaccinated should be 0.22% versus 1.09% for the vaccinated, meaning that 77.6% of all autism is directly caused by vaccines, and in particular, by the vaccine these children received, the pneumococcal conjugate vaccine (PCV). Is this prediction even remotely plausible? No.

We know that PCV is unrelated to autism in individual-level data—a clear prediction failure—and that autism doesn’t spike in the years after the vaccine is introduced—a causally-informative prediction failure!4 Take a look:

Where does this leave the anecdote? Well, it’s obviously wrong and the extreme numbers cited by proponents of it must be off. We know this is the case, because it must be the case given the observed numbers. For it to be any other way, the autism rate would have to be enormously understated or the vaccination rate overstated or… something else that’s obviously not happening.

What’s really happening in this anecdote is that autism runs in families—it’s highly heritable—to the tune of younger siblings of autistic children having a 10-20% chance of having autism themselves (this study estimated 18.7%) and the timing of vaccination coincides with roughly the time when many parents notice their kids are autistic. Across millions of vaccinated families, large numbers of parents will begin to notice their kids’ autism around the time they get shots, and if they’re all getting their appointments together like these triplets were, then a coincidence of diagnostic timing is to be expected.

The story of the McDowell family relies on vividness to make it seem like something is amiss and vaccines are to blame, but it’s statistically quite probable. In fact, we can just work through Kirsch’s spectacular numbers to get at why we do expect to see examples like this. Here’s a diagram:

From left to right, we have estimates from different scenarios. On the left, you have Kirsch’s implausibly small probability estimate that he—as we’ve shown, improperly—believes establishes that vaccines cause autism. This estimate comes from his tweet and it’s based on taking an autism risk of 1 per 100 by age 4 in 2007 with onset somewhere between ages 1 and 4 (i.e., 1,095 days), noting three kids means 3e-5 expected cases per day, and then computing the Poisson chance of three or more events on one day as poisson.sf(2, 3e-5) and multiplying by 1,000 for “any of the days”, giving roughly 4.5e-12, but using the per-child rate, giving 1.66e-13. This estimate assumes children are independent, that autism begins on a specific, datable day, that every day in the window is equally likely and dated separately per child, and that nobody went looking for the family—all incorrect.

The next estimate to the right simply takes his inputs and computes the number based on his assumptions correctly. The assumptions remain wrong; the point, however, is to show that an implausibly small number still follows from them even when you do the calculation correctly. But what happens when we start correcting the erroneous assumptions? The next estimate removes independence: given the first child is autistic, the next child has an 18.7% chance of also being autistic. From the same study, we also see that, given two siblings are affected, the risk for the third one is 32%. Incorporating this alone raises the probability by about 600 times, and this understates how much the probability should improve, because triplets also share a womb and birth timing.5

The number to use isn’t 1/1,095²—”three independent onset days”—, but q, the chance that parents tie three events to one visit. This means, given three autistic, same-age siblings vaccinated together, we expect this outcome to happen in approximately 1-in-6,700 triplet sets. We know this for a few reasons: autism doesn’t begin on a single day (it’s developmental); triplets share developmental timing and appointments; and parents date the onset of autism after diagnosis and anchor to memorable events like well visits. In other words, diagnoses are not independent draws!6

Finally, we have the probability of at least one family like the McDowells. This is based on an extremely conservative calculation taking 30,000 U.S. triplet sets born between 2000 and 20207 and multiplying by the previous probability of 1.5e-4. P(at least one) = 1 - (1 - p)ᴺ ≈ 1 - e⁽ᴺᵖ⁾ ≈ 0.99 (0.83 to >0.999). Even this is conservative, as it assumes families are independent, it leaves out quadruplets, other countries, and it omits both earlier and later years.

We can confidently state that this anecdote is uninformative because a case like this is expected by chance. At most it could move the odds very slightly. Once we also acknowledge that its implied effects aren’t realized in the general population data, we’re left with the realization that this whole thing was pointless to bring up.8

This is All They Have

This sort of poor-quality evidence stated without any caveats or rigorous interrogation is all that anti-vaccine activists have to go off. They cite lurid anecdotes and don’t check their implications; they get confused by confounded and selected associations; they fail to recognize the role of institutions in driving reporting on disease; and they very generally do not understand the conditions they’re talking about. To end on an example, consider the fact that the apparent ‘rise’ in autism is entirely due to diagnostic drift. This is well-established, and yet anti-vaccine activists try to explain it: they blame it on vaccines, even though there’s nothing to explain!

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

implied_or <- function(prev = 0.01, cov = 0.90, n_kids = 3, thr = 0.5) {
  rr <- 1 / (1 - thr^(1 / n_kids))        # RR at which P(all n cases vaccine-caused) = thr
  p0 <- prev / (cov * rr + 1 - cov)       # unvaccinated risk, holding overall prevalence fixed
  p1 <- rr * p0                           # vaccinated risk
  data.frame(coverage   = cov,
             RR         = rr,
             OR         = (p1 / (1 - p1)) / (p0 / (1 - p0)),
             risk_unvax = p0,
             risk_vax   = p1,
             PAF        = 1 - p0 / prev,          # share of all autism caused by vaccines
             excess_pp  = 100 * (prev - p0),      # percentage points of the overall rate
             max_BF     = (1 / cov)^n_kids)       # most the anecdote can favour any effect over none
}
rbind(implied_or(cov = 0.90), implied_or(cov = 0.92))

On June 29, 2023, the Supreme Court declared by a 6-2 vote that American universities that accept federal funds were no longer allowed to racially discriminate when making admissions decisions. The case—SFFA v. Harvard—concerned the fact that Harvard was making decisions using students’ race as a variable in admissions.

Advocates on the side of Harvard argued that racial discrimination was necessary practice to preserve diversity. Opponents argued that racial discrimination was wrong and illegal. The opponents are best described as ‘correct’; but the advocates do also have a point, albeit one that should only be responded to with ‘So what?’: racially-fair admissions should undermine diversity, because certain groups—namely, Asians and Whites relative to Blacks and Hispanics in the U.S.—tend to have higher levels of academic achievement.

So, three years on, has the verdict in SFFA v. Harvard resulted in universities becoming less ‘diverse’ and more reflective of what a fairly-admitted student body would look like? The answer is ‘partly’, not entirely. A recent article in Econ Journal Watch looked into this and reached just that conclusion. But why? From the changes or lack thereof alone it’s hard to tell. But the most likely possibility is that universities are still discriminating even though the Supreme Court has declared doing that to be illegal.

The Department of Justice has been looking into this issue using the methods I outlined when I showed that Columbia and NYU were still discriminating. They’ve recently opened cases against UCLA’s medical school, Yale’s medical school, UC Davis’ medical school, UC San Diego’s medical school, Duke Law School, George Washington’s medical school, UC Berkeley Law School, UC San Francisco’s medical school, and UCLA Law School. They’ve also sued Harvard for withholding race-related admissions documents, run broader compliance reviews at Stanford, Berkeley, UCLA, and UC Irvine, and began a broader investigation into fifteen medical schools.

Every university the DOJ has opened an investigation into so far is almost-certainly discriminating. We know this because each of the universities shows a considerably higher odds of admissions for students from favored racial groups—Blacks and Hispanics—as opposed to the ones that universities have historically wanted to discriminate against—Whites and Asians:

• Berkeley gives Blacks ~6-times higher odds of admissions at the same LSAT and undergraduate GPA (UGPA) levels.

• UCLA Law gives Blacks 7-times higher odds of admission and Hispanics 2-times higher odds of admission than Whites.

• Duke gives Blacks 3.5-times higher odds of admission than Asians at the same LSAT and UGPA levels.

• Yale invites Blacks as much as 29-times more often for interviews than it invites similarly qualified Asians.

• UCSF is 12.6- and 4.6-time as likely to admit Blacks and Hispanics, respectively, as it is to admit Whites with comparable MCAT scores, UGPAs, and socioeconomic characteristics.

• UC San Diego was nearly 3-times as likely to admit Blacks as Whites.

• UCLA Medicine admits much less qualified Blacks and Hispanics than Whites and Asians.

• George Washington Medicine admitted less qualified Black and Hispanic applicants than applying Whites and Asians.

There are only two ways to obtain these patterns of results:

1. Discrimination; or,

2. There are compensatory factors that make less academically qualified Blacks and Hispanics as worthy of admissions as their more academically qualified White and Asian peers.

Proponents of racial discrimination frequently argue in favor of the latter idea. But it’s absurd. For one, they’re forced to argue for an incredible coincidence: that admissions departments are making the same sorts of decisions without explicitly using race that they made when we know they explicitly used race. People are free to argue that, but it’s obviously unlikely given that there are no such factors that reliably proxy race in the way they’d need. They also don’t have any room to stand on the idea that there are compensatory factors generally. The reason for this is simple: admissions departments don’t collect information about factors that are plausibly compensatory.

A valid non-discrimination explanation for universities continuing to admit student populations that look demographically like their pre-SFFA ones requires a few things:

1. It needs to use variables that are available to admissions departments: whatever admissions decisions are based on must be collected and available before the admissions decision;

2. Variables need to be acceptably measured: they should be assessed consistently across races, without functioning as a racial preference or identity substitute;

3. They need to show the necessary racial disparity: they should favor the group receiving higher admissions odds, conditional on tests, grades, and other academic achievement-related controls;

4. They need to be strong enough: the conditional advantage the variables provide should, when combined with their actual admissions weight, explain the gap.

And though this isn’t required, to be considered valuable:

5. They need incremental predictive value: they should predict relevant outcomes like student GPAs, course completion, in-class behavior, etc. beyond the purely academic criteria they’re being weighted against.

Every college the DOJ is investigating is an open-and-shut case because none of them has an admissions department that gathered data on variables fitting these criteria. We know what they collect because it’s public; we know how the differences in what they collect tend to vary across groups; and we know how much predictive value they have in the general population. There’s no explanation besides discrimination.

Take Berkeley Law as an example. Their admissions department collects 2-4 letters of recommendation, a personal statement, a résumé, and they don’t have a general interview to speak of. Each of these things tends to favor White and Asian students, which means that if they were used, they would actually make it harder for Berkeley to make its case unless they were an outlier school in terms of how these things correlate with grades, test scores, and demographics. To make things worse, most of what Berkeley Law collects data on is unreliable. In Kuncel et al.’s meta-analysis of letters of recommendation, for example, they found that to achieve a reliability of 0.80—roughly the bare-minimum for how reliable something should be—students would need to submit six letters, but Berkeley Law collects at most 4 per student

Unlike test scores, the variables Berkeley Law collects are also usually racially biased, and in any case, they can be easily used to infer student race even if admissions departments never explicitly ask for students to report their race. For good measure, the Supreme Court ruled that sort of proxying illegal, but—and this is part of the evidence—Berkeley Law has decided to engage in this practice anyway.

In the case of Berkeley Law, the DOJ noted that the school’s dean Erwin Chemerinsky explicitly and publicly discussed replacing explicit racial preferences with proxies and presenting such a policy with race-neutral justifications in order to keep discriminating. Communications within the admissions department were uncovered and they explicitly called for maintaining diversity despite litigation risk. Application questions created by these people and their coworkers also expressly invite students to disclose their race. Finally, to make things clearer with some contextual evidence, Chemerinsky explicitly discussed concealing racial considerations in faculty hiring, suggesting that he wants racial discrimination in multiple areas, not just admissions.

Evidence like this was present in every case the DOJ has opened so far, from Duke Law to George Washington’s medical school. The DOJ is not resting on the strong statistical evidence that universities are discriminating; they’re also showing that we know what they’re doing. And the truth is, they know that we know what they’re doing, too. They discuss it openly! The fact that we know that they know that we know what they’re doing should make these cases easy to prosecute1, but only time will tell.

It’s clear now that the universities cannot be trusted. The documentation from the universities themselves indicates beyond any reasonable doubt that they have continued engaging in racial discrimination. In several cases, they’ve been caught planning to do so, admitting to doing so, hinting at it, and so on. In each case, it’s clear: making racial discrimination illegal was insufficient. We must go further.

Universities must be compelled to affirmatively justify their admissions decisions. They need to publicly provide the data and the model used to select students, and they need to tell us how it works—explicitly—and then show us—verifiably—that they used only the information provided on students’ applications, without proxying race. If they find that to be a too great a burden, then we only have one course of action: defund them. Ban their researchers from grants, ban their students from public loans, take away their tax benefits and accreditations and confiscate their endowments until they become compliant or voluntarily cut themselves off from the public teat. The universities will only learn if it hurts.

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

Growing up, it was a common trope that cultures that we now associate with poverty, violence, and oppression were once world-leading in promoting prosperity and moral progress as well as the quintessentially human act of inventing. This popular belief was ultimately not driven by an earnest desire to promote awareness of empirically true but neglected facts. Instead, it was supported by anecdotes and quotations of dubious reliability and relevance, by scholarship that has turned out to be exaggerated or fraudulent, and by the promotion of these things in popular media.

My favorite example of this has to do with the invention of many of the inventions credited to the so-called Islamic Golden Age by a rogue Wikipedia editor.

Have you ever heard the name “Jagged_85”? If you’re a history buff, you’ve probably at least heard of some of his claims. For example:

• Muslims invented funnels and sieves

• Al-Razi invented dissolving substances and sublimation

• Jabir ibn Hayyan invented cheese glue

• Ibn Sahl first described the parabolic mirror

• An Islamic water clock from the 12th century had the earliest copper pipes

• Avicenna formulated momentum as weight times velocity

• The Brethren of Purity proposed a heliocentric universe in the 10th century

• Darwin drew on Ibn Miskawayh’s work to develop the theory of evolution

• Al-Ghazali directly influenced Descartes’ later philosophical work

There are many other claims I could add to this list: Taqi al-Din invented the telescope sometime in the 1570s; Avicenna discovered microorganisms; Muhammad ibn Musa discovered gravity between planets; Al-Biruni discovered elliptical planetary orbits; Al-Jazari invented flow regulators and feedback control; Raqqa had a major glass factory where 300 different glass recipes were used, or—most amazingly—Al-Battani discovered relativity a millennium before Einstein.

Jagged_85 vandalized thousands of articles while making over 87,000 Wikipedia edits in the time between starting in 2005 and being caught in 2012. Those edits have led to hundreds of false claims misattributed to history books and papers, and now, those claims are alive on the internet, being repeated carelessly and promoted as if they’re true, when they are anything but. Several of these claims have even seeped into the scholarly literature, and now they’ve gained some credence as true facts by virtue of being repeated by scholars misled by what they read in Jagged_85’s Wikipedia articles!

How Do False Claims Make it to Wikipedia?

The way a Jagged_85 claim works is straightforward. He rarely ever fully fabricated claims and they always had some sort of source document attached to them, from which you could usually sort-of see what he was getting at. As an example, take the claim that Muslims invented funnels and sieves between 721 and 925. Jagged_85 did cite a source, it just didn’t say what it was cited for. What it actually provided readers was a list of laboratory equipment used by Muslim scholars, among which one could find funnels and sieves. The crucial issue, however, is that it never said they invented them, and in fact, it showed that they had been in use since Antiquity by Roman and Greek scholars. That last bit went unmentioned in places where the work was cited.

What about the claim that Al-Razi invented dissolving substances and discovered sublimation? There was a source cited for this claim too, but what it said was that Al-Razi described their operations, not that he discovered them. That achievement was given to the Greeks. Jabir ibn Hayyan’s invention of cheese glue? Sourced to a document that said he translated a medieval recipe; other sources show us that adhesives of this variety were in use in ancient Egypt. Ibn Sahl inventing the parabolic mirror? This came from a source that mentioned plenty of earlier investigators. And funny enough, when Jagged_85 made that citation, he didn’t understand what he was talking about, as he called the mirror “refracting”, as opposed to ‘reflecting’.

The edits provided by Jagged_85 had every appearance of being scholarly, but the citations were nevertheless stretched and exaggerated, and sometimes contradicted the things they were cited to say. This was noticed in 2010 in a prominent case where Jagged_85 invented the concept of Avicennian logic and then conflated Avicennism with that concept:

The article on 'Avicennism' says that "Avicennian logic had an influence on early medieval European logicians such as Albertus Magnus". Yet the idea that most influenced these scholars, including Albert, was Avicenna's distinction between essence and existence, which was a genuine innovation, and a departure from Aristotle. This idea had a profound impact, in different ways on Aquinas, Giles of Rome, Henry of Ghent, Scotus and eventually Ockham. But it is a metaphysical idea, not a logical one. As far as I am aware, no medieval writer discusses Avicenna's logic (as opposed to his commentary on the Metaphysics, which they frequently cite). Kneale's great work The Development of Logic mentions Avicenna twice, neither in connection with logic. Bochenski does not mention him at all. None of the articles in the main secondary sources on medieval philosophy mention Avicenna's logic (although they discuss his other achievements at length).

[This] proves a clear weakness in the Wikipedia 'verifiability principle'. [… Jagged_85 frequently used] scholarly works not available on the internet. Most of Wikipedia's editors are amateurs who have no access to a university library. Thus they cannot check a source from a journal, or an old or obscure book that would only be found in a library.

Despite this, if you look up Avicennian logic or Avicennism today, you can find distorted notions of what Avicennism is, and plenty of people talking about Avicennian logic, as if it’s a real thing, with attributions to the sources used by Jagged_85. But these attributions are false; the sources don’t support the claims scholars have made about them, and because they were miscited in the ways Jagged_85 cited them, we know they got their backing from his Wikipedia edits.

How Do False Claims Make it to Other Scholars?

Scholars are, in a word, lazy. But the reason for this isn’t some especial problem with scholars, more so, it’s a reflection of the fact that most humans are lazy. If you can read a Wikipedia summary to get your information, why not do that? It’s far easier and less complex than going to the secondary sources Wikipedia cites, and it’s easier yet than going to the primary sources those secondary sources cite. In some cases, it’s easier to use Wikipedia and to rip off its bibliography section than it is to access the relevant works. If you need a citation, that’s a commonly used route, unfortunately.

It’s unsurprising that so many scholars have parroted the confabulations Jagged_85 lied about.1

The scholarly influence of these false claims hasn’t gone unnoticed by academics. There are papers and articles that have specifically noted the influence of Jagged_85 on literatures as wide-ranging as psychology and medicine. In the most bizarre cases, scholars get misrepresented by Jagged_85 and then update their own claims to support the misrepresentations.

Consider University of Manchester professor emeritus Salim Al-Hassani, the maintainer of the 1001 Inventions project and MuslimHeritage.com website. In an article dated 2004—well before the Jagged_85 editing spree began—he described the inner workings of the elephant clock, flow regulation, closed-loop operation, etc., but nowhere did he say these were the first appearances of the concepts. In 2007, Jagged_85 edited Wikipedia to claim that Muslims invented these things with the elephant clock, citing al-Hassani. Not long after, in 2008, al-Hassani published another article that made the same sorts of claims, with similar wording! These errors were then reprinted widely, including by businesses that still have the claims up today.

How Do False Claims Make it to You?

False claims laundered from Wikipedia make it into academic circles and further into the zeitgeist, because they inform people about things they can say and should believe. They go further than that because they are believed, and because, in these cases, they are plausible—by virtue of academic acceptance, Wikipedia presence, scholarly bona fides, etc.—and people want to believe them; it is fashionable to talk about an exaggerated Muslim Golden Age and era of invention. Muslims are newly being introduced into Western societies, so just as Christopher Columbus began to be celebrated to welcome Italians into America, we now find it worthwhile to talk about how Western scholars and historical geniuses were actually inspired by Muslims in ages long past, forgotten, and perhaps even intentionally obscured.

The most widely circulated false ideas achieve their prominence through becoming educational: they are now the sorts of facts people should know, not just academic obscurities. For example, Salim al-Hassani set up an exhibit at London’s Science Museum, and London journalist Olivia Sterns saw the exhibit and decided to write about it for CNN. The article included the false claim ultimately sourced from Jagged_85, that the crank was invented by Al-Jazari in the 12th century.

In a similarly poor and viral showing, Sean Carroll evidently used Wikipedia to try to figure out who had coined momentum, and in doing so, he quoted an error that originated with edits made by Jagged_85. He attributed the thing to Avicenna, using wording derived from Jagged_85. Avicenna, as it turns out, was miscited, and this wasn’t the first time. People might also have been exposed to Jagged_85 edits about Avicenna by way of Google Books descriptions, listings on sites like Booktopia and Goodreads, and so on, which have all featured improper descriptions of translations of Avicenna’s Canon of Medicine, claiming that the author—Avicenna—invented randomized controlled trials and risk-factor analysis.

What the Jagged_85 saga shows us is that with a little gumption, anyone can flood our shared information environment with garbage. They can have lies and exaggerations cited by thousands of scholars, educational programs and pamphlets, museum exhibits, journalists, and search engines alike, because no one checks, and people get lazy.

Who knows how many Jagged_85 edits have survived in the public consciousness, making people feel wrongly ‘informed’ about Islam and the Muslim world? Knowing just a few dozen of them, I can personally attest that I see them multiple times a year. I’m also occasionally privy to other famous Wikipedia-derived false claims like those about the electric toaster inventor Alan MacMasters—who never existed despite making it onto multiple BBC programs—, the ones about the nonexistent gas chambers in Warsaw (the real gas chambers were elsewhere), or the various articles on Chinese deportations and nonexistent silver mines in Russian history invented by the editor Zhemao. Who knows how much literature, how many TV programs, how many ‘fun facts’ are nothing more than the invention of some unsavory Wikipedian?

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

What statistics belong in the Annals of Fake Statistics? Healthcare administration costs ballooning tens of times over in the past few years? Nearly 70,000 deaths a year due to lacking insurance coverage? You’re more dangerous to schizophrenics than they are to you? Men kill their spouses far more often than women do? (Although see.)

There are many candidates, but I would definitely tender the following fake facts:

1. Medical error is responsible for 251,000 American deaths a year, making it the country’s third leading cause of death;

2. Misdiagnosis kills 371,000 Americans per year;

3. Misdiagnosis leaves 424,000 Americans permanently disabled each year.

I recently saw these claims being made in a clip from an episode of the Joe Rogan Experience podcast. These claims are false, but they’ve permeated the discourse and found a way to survive despite their obvious absurdity; in fact, without proper prompting, several frontier LLMs will faithfully regurgitate these erroneous figures.

A Quarter-Million Medical Error Deaths?

This infamous estimate comes from a 2016 British Medical Journal publication by former FDA Commissioner Marty Makary and researcher Michael Daniel. The article achieves an estimate of 251,454 deaths from medical error each year by averaging four independent estimates of different quantities. I say ‘different quantities’, because none of the numbers being averaged was an estimate of the rate of death from medical error.

The first estimate comes from a 2004 estimate by HealthGrades. For Makary and Daniel’s purposes, this study looked at Medicare enrollees admitted to hospitals between 2000 and 2002 and provided an estimate of the number of deaths that could be flagged for investigation for “failure to rescue” and because the death took place in a member of a group that ordinarily has a low-mortality rate. Makary and Daniel then extrapolated the death count from this study forward based on the increase in the number of total hospitalizations by 2013, arriving at a figure of 251,454

The second estimate is taken from Classen et al.’s 2011 study on treatment-related adverse events at three tertiary care hospitals in 2004. This study involved counting up potentially treatment-related adverse events in hospitals, without evaluating whether those events were preventable or if a different pattern of care would have made them less severe. The study looked at 795 hospital admissions, found 9 fatal events, and from this, Makary and Daniel concluded that 1.13% of hospital admissions resulted in a preventable fatal event, which they chalked up to medical error and then extrapolated to total U.S. hospital admissions in 2013. But in addition to being based on an unwarranted assumption of preventability, this number is explicitly wrong and the original article’s count was corrected from 9 to 8 in a 2011 errata that Makary and Daniel should’ve seen. Using this number would drop the count from Makary and Daniel’s 400,201 to 357,502 with rounding and 356,388 without it, dropping estimated medical error deaths to 240,829 (i.e., roughly 10,000 fewer deaths).

The third estimate comes from a 2010 HHS Office of Inspector General report on a small number of Medicare beneficiaries. The report reviewed adverse events and recorded 12 fatalities among 838 events, with Makary and Daniel choosing to use the study’s 44% preventability number for adverse and temporary harm events to adjust this to an estimate of 219,579 medical error-attributable deaths. The study actually said 5 of 12 deaths were clearly preventable and 1 was likely preventable, so they should’ve chosen a rate that was either slightly higher (6/12) or slightly lower (5/12), for estimated death counts of 253,575 or 211,313. But this is a curious calculation, since it assumes the fatal error rate is the same in Medicare and non-Medicare populations, implying that, against an all-cause mortality benchmark of 3.56% for Medicare and 1.01% for non-Medicare, the fraction of deaths attributable to error is 17% versus 62% for non-Medicare, and if we bring that forward, the error-attributable fraction should have fallen for compositional reasons, contrary to their calculations.

The final estimate comes from Landrigan et al.’s 2010 estimation of temporal trends in patient harm in 10 North Carolina hospitals. The authors of this study identified 14 fatality-contributing events across 2,341 hospital admissions, judging 9 of these to be preventable. From this, they got 134,581 medical error-attributable deaths, even though that doesn’t follow from the study’s fatality-related numbers. This is because, instead of using the rate based on the 9/14 preventable events, they used the study’s overall event preventability, which included non-fatal events. The real estimate—assuming we can generalize from this sample, which we cannot—would be 136,157.

Can you grok what’s going on here? Makary and Daniels took four different rates, from four populations that weren’t nationally-representative, generally with very imprecise estimates, understood that their derived rates had to be caveated but despite that, only caveated two of four estimates, then extrapolated each rate forward in time, and called the mean of those numbers the medical error-attributable death rate when not even one of those was the medical error death rate in its study! They then stated that this was likely an underestimate because “the studies cited rely on errors extractable in documented health records and include only inpatient deaths.”

I’m going to note a few concrete issues with this. I’ll start with the claimed sources of underestimation that Makary and Daniels suggested. The first one, about errors, is almost-certainly wrong and they got the bias backwards. There is substantial empirical work on this. For example, Hayward et al. reported that, absent a large number of reviewers, rates of preventable death in U.S. hospitals tend to be overestimated. The low reliability does not result from classical error, it results in cases of non-preventable death being wrongly classified as preventable based on unreliable judgment. For two, there’s much more harm from care not delivered than from harm delivered, as out-of-hospital cardiovascular event, violent death, (etc. etc. etc.) rates make very obvious, unless we assume that inpatient deaths are actually far rarer than Makary and Daniels suggested, and medical error-attributable deaths outside of hospitals, very common. (This is inherently absurd given how little medical care occurs outside of hospitals and especially so given how little can be linked in any way to subsequent fatalities, as well as how low the post-discharge fatality rate is.)

Makary and Daniels’ underreporting objection is also silly on its face because the studies they cited explicitly dealt with it. They are correct that underreporting to incident systems does occur, but the studies they cited dealt with research reviews. The OIG explicitly found that 14% of its harm events were identified by hospital incident systems, but you can’t then extrapolate upwards, because the OIG’s estimate is based on the full set of identified events, not just the 14%.

Makary and Daniels also should not have extrapolated high medical error-attributable fatality rates forward like they did. The reason why this is wrong is because rates have almost-certainly fallen over time. Using just studies Makary and Daniels should’ve been familiar with, we know, for example, that AHRQ recorded a 17% decline in hospital-acquired conditions between 2010 and 2013, Wang et al. recorded a 4.7% annual decline in adverse events per 1,000 hospitalizations between 2005 and 2011, and their cited Landrigan et al. study discussed possible preventable harm rate declines, although it didn’t find any in its own nationally-non-representative sample. If you were to extrapolate from Makary and Daniels’ studies with improvements in care in mind and with consistent deflation of estimates to account for preventability, then medical error deaths would drop somewhere between COPD and suicides to the fourth-leading cause of death. And Makary and Daniels should’ve known this!

The most fundamental issue with Makary and Daniels’ estimate is just that if we actually try to estimate the medical error-attributable fatality rate, it’s clearly much lower than what they estimated.1

The best estimate comes from Hayward et al. This estimate was based on a review of medical records produced between 1995 and 1996 across 7 different VA medical centers. The records reviewed were oversampled for conditions with high rates of preventability. This study doesn’t have a particularly good sample, but it does have particularly good handling of its data: the authors generated data with 14 separate reviewers, standardized reviewing instruments, and then they modeled the effects of unreliability and skewness. The estimate the authors achieved was that 6% of patients would’ve left the hospital alive had they been provided the best possible care, but with 3-month prognoses and review skew taken into account, “only 0.5% of patients who died would have lived 3 months or more in good cognitive health if care had been optimal”. If we call it 2-3 preventable deaths per 10,000 admissions, the annual death count probably clocks in under 10,000, and that’s with the 1990s’ high error rate.

Other studies have provided estimates that are more relevant to medically attributable deaths than Makary and Daniels’. Rhee et al., for example, reported that, across 6 U.S. hospitals in 2014-15, just 3.7% of sepsis-associated deaths were definitely or moderately likely to be preventable. Tatara et al. found that 8.6% of deaths from non-ventilator hospital-acquired pneumonia across 4 hospitals 2016-21 were likely or very likely preventable. Bates et al. reviewed 2,809 admissions across 11 hospitals in Massachusetts and concluded that of 7 fatal adverse events, just 1 was preventable.

The most defensible recent estimate that I can come up with comes from applying Rodwin et al.’s 2020 systematic review estimate—3.1% preventability—with the AHRQ’s estimate of 769,460 inpatient deaths in 2023. This combination leads to an estimated 23,853 medical error-attributable deaths (~17-32k). If you were to apply Bates et al.’s numbers to the AHRQ’s, you’d end up at 12,004 instead. This discrepancy should give some indication of how weak estimates are, given the combination of rare events and unreliable classifications.

In short, Makary and Daniels did not produce estimates that hold up to scrutiny. They took unrepresentative and extreme samples, irrelevant estimates, figures that were later revised, numbers they—conveniently—knew had to be adjusted downwards instead of being kept as they were, and they produced an estimate that was too high. What’s worse is that the concepts they dealt with were not even coherent: to get even remotely close to their estimates, we must assume that a failure to provide optimal care is a “medical error” that realistically—not just ‘theoretically’—could’ve been prevented in an active care setting. Given the reality of hospitals, where there are not unlimited human or capital resources to get everything that could be right done correctly, the theoretical maximum estimates from ‘optimal’ care projections are still too high.

A Third of a Million Misdiagnosis Deaths?

The 371,000 deaths from misdiagnosis figure comes from Newman-Toker et al. Thsi study was a review of disease incidence data from 2012-14 that was paired with literature-derived error error and harm rates to project that each year, diagnostic errors led to 795,000 deaths or permanent disabilities in the U.S. Neither the error counts, nor the implications of the projection forward, or the split between disability and death are empirically supportable or interpretable as the authors did.

As already noted: conditions regarding error have gotten better with time and counts tend to be inflated by unreliability that’s typically unaddressed in studies. Now, more specifically, Newman-Toker et al. cited two studies—Singh et al. and Zwaan et al.—to support their conclusions about death and disability.

Singh et al. rated the potential severity of 190 diagnostic errors, and put 27 cases in the potentially fatal category—it did not report on 27 observed deaths. Most of Singh et al.’s diagnostic errors were identified through subsequent hospitalizations that were considered unexpected, meaning that—as Singh et al. noted—, the errors were likely more severe than errors generally. The estimate from Singh et al. cannot be paired with national disease counts to obtain a representative estimate of harms because it does not supply representative harm probabilities. Nor can Singh et al. be used to determine the split between error-associated deaths and disabilities, because it skews towards greater-than-expected harms to an unknown degree. Additionally, deaths following diagnosis are not necessarily deaths that a proper diagnosis would have prevented or slowed down at all.

Zwaan et al. reviewed 7,926 records sourced from 21 different Dutch hospitals, concerning admissions in 2004. They identified 80 diagnosis-related adverse events, with reviewer adjudication about the presence and preventability of adverse events, but not about their consequences. 29.1% of events were labeled as contributing to deaths, but it wasn’t clear if they were causally involved. In any case, this is an old estimate from a different country, with limited review reliability, being combined with a known-unrepresentative estimate to get a split between disabilities and deaths to somehow come together and provide us with an estimate of America’s misdiagnosis death rate.

Fundamentally, this estimate is not based on numbers that actually establish anything like the misdiagnosis-related death rate. That’s not what these sources can provide.

Half a Million Misdiagnosis Disabilities?

The figure of 424,000 permanent disabilities from misdiagnosis is the flip side to the 371,000 death figure. It is based on the same studies, and thus based on an exaggerated baseline count and an inaccurate split that leads us the Newman-Toker death and disability numbers from said count. Worse is that Singh et al. merely classified 36 of 190 diagnostic errors as potentially causing permanent damage; they did not find that 36 patients wound up with permanent disabilities due to misdiagnoses. Zwaan et al. also had a classification problem because their numbers had nothing to do with permanence: they were about disability at discharge!

The outcome that should have been measured here is new or worsened persistent disability attributable to a diagnostic error. That was not measured.2

Each of these mistaken estimates is an example of the core problem with what I’ve dubbed ‘attribution studies’. The way attribution studies work is by attempting to project the numbers of uncounted things—deaths, disability, displacements, bankruptcies, whatever—based on proxy inputs gathered from elsewhere. The problem is that if you control the inputs, you control the outputs, and if you want a given output, it’s not hard to get it. With each of these examples, the inputs were dubious, so the outputs were too, and our public discourse has been tainted as a result.

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

Thomas Jefferson is the well-known Founding Father who wrote the Declaration of Independence. Sally Hemings was his three-quarters White (‘quadroon’) slave and his wife’s half-sister. She was tasked with being the attendant to his daughter Martha.

In 1787, Martha took Sally on a trip to Paris, where Hemings’ slave status was not recognized, making her legally free. During that time, she stayed on as Jefferson’s servant, and over the course of their 26-month French sojourn while she was a free woman, Jefferson and Hemings began a romantic relationship. At the time, she was roughly 15 years old and Jefferson was a mid-40s widower.

When it came time to return to the States, Jefferson negotiated with Hemings: come home, return to slavery, and you’ll be afforded considerably leniency in work, and your kids will all be manumitted when they come of age. After returning to America, Hemings had four children and Jefferson and his estate eventually freed all of them.

This bit of history has caused considerable political consternation. There are those who want to tar Jefferson for this relationship for all manner of reasons. At the time, the main focus was on the relationship being miscegenation, and thus illicit for that reason. Nowadays, the main focus is on the relationship being necessarily coercive due to Hemings’ status as a slave. Regardless, there was and remains controversy.

Part of the controversy about the relationship between Jefferson and Hemings was about whether it even resulted in children. It’s certainly the case that Jefferson and his estate freed Hemings’ children, and it’s also the case that slave-owners disproportionately freed slaves who were their children. However, this on its own does not establish that Jefferson was the father. Jefferson may have simply kept his word and freed Hemings’ children regardless of parentage.

This perspective runs into several issues. For example, people at the time frequently attested to and mocked Jefferson for his involvement with Hemings, and they even claimed that Hemings’ kids had a striking resemblance to Jefferson. Moreover, we know from genetic tests that there are certainly many people alive today who are related to Jefferson.

In 1999, a genealogical project established that Jefferson’s Y-chromosome was passed down to the modern-day, establishing for many that he had fathered Hemings’ children. But this finding is not unambiguous. Two years later, the Thomas Jefferson Heritage Society published a report noting that the Y-chromosomal evidence was consistent with Hemings also being impregnated by a different Jefferson man. This perspective deserves some credit precisely because there were candidate Jefferson men in contact with Hemings beyond Thomas: his brother and his nephew.

That back-and-forth leaves us where we were until a few days ago: unsure if Jefferson was actually the father of Hemings’ children, or merely a bystander to the impregnation of one of his favored slaves by a male relative. But now, the Smithsonian has sequenced Jefferson’s whole genome—mystery no more!

We now have percentages of autosomal DNA sharing between Thomas Jefferson and a handful of Hemings’ descendants. This is the DNA that’s not mitochondrial or from the sex chromosomes. You share 50% of it with either of your parents and your full siblings share an average of 50% with you as well (vs. 25% for half-siblings and avuncular and materteral relatives, and 12.5% for first-cousins), with a range of relatedness due to the randomness involved in inheritance.

The new paper on the Jefferson descendants lists Hemings descendants (with identification via labels like ‘M5’ and ‘E9’) who share 5.01%, 4.27%, 3.66%, 3.64%, 3.50%, 2.09%, 1.01%, and 0.55% of their DNA with Thomas Jefferson. The expectation for fifth-generation direct descendants is an average of 3.125%. For a brother, it’s 1.563%, and for a nephew, it would ordinarily be half as much, but Thomas Jefferson's nephew was more related to him than an ordinary nephew because Randolph married his and Thomas’ first-cousin. This made Thomas’ nephew 31.25% related to him rather than the usual 25%, making the expectation about 0.977% rather than 0.781%. For comparison, we also have DNA from two descendants of Jefferson and his wife—not Hemings—, and they share 2.57% and 2.69% of their DNA with Jefferson.

Because the expected levels of relatedness differ between descent from Thomas or his brother, we can categorize each of these observations based on their consistency with either option to obtain a likelihood that the kids came from either potential father. Arranging each observation by the order they’re presented in the paper, they’re positioned like this relative to the expected parental distributions:

If we go from the least-Thomas Jeffersonian person to the most-Thomas Jeffersonian person, we can see the likelihood of a match to Thomas rising to a very high level. The probability that Thomas was the father rather than his brother reaches 851:1, and the probability of Thomas instead of his nephew reaches an even higher 6.8m:1.

These results directionally match the more complicated simulation result presented in the paper, which makes sense given that they’re both driven by the same underlying logic. The authors’ preferred likelihood of Thomas compared to his brother is 2,213:1.1 Clearly it’s highly likely Thomas Jefferson had kids with Sally Hemings.

To alter the judgment that Jefferson fathered at least one of Sally Hemings’ children, one or both of two things would’ve needed to happen: gross error in DNA sample identification and/or handling; or major problems in sampling. I seriously doubt either possibility. It seems one of the country’s historical mysteries is now solved.2

You can find my replication package for this project by clicking the link here. Enjoy!

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

Why are men more violent than women? ‘It’s the testosterone.’ Why do some men end up in jail? ‘They’re high-test.’ Why do teens commit more crime than 40-somethings? ‘Testosterone decline.’ Why are bodybuilders always flying into a rage? ‘Roid rage!’

These are all popular beliefs about testosterone, and there are many more like them. Many—and I’d argue a majority—of the public believe that testosterone is some sort of rage or aggression chemical, that it radically increases risk-taking, aggravation, the propensity to commit violent acts, and so on. But is there much evidence for this?

The first thing to establish is the baseline for testosterone. Male testosterone levels are generally not very high, and they’re quite stable even into the beginning of old age. The big decline comes in the elderly years, and its cross-sectional shape is likely impacted by selection.1 take a look:

Given how little testosterone levels appear to decline with age, it would be surprising if testosterone explained the age-crime curve—the well-known relationship whereby the propensity to crime jumps up after puberty and into early adulthood and declines into middle age and beyond. In fact, they do not line up:2

An even simpler method to understand the relationship between testosterone and crime is to simply compare members of the general population to criminals. If we take a look across the whole literature, we get two pictures: violent criminals have about 18% higher testosterone levels than members of the general population (i.e., ~71st-percentile testosterone levels) and sex offenders have completely normal testosterone levels.

One possible issue with this comparison is that incarceration—where testing generally takes place—could lead to reduced testosterone levels due to the diet, exercise, and social conditions of prison. This is not a major worry. There’s one study of the effects of incarceration on testosterone levels, and even though it used salivary testosterone measurements—which are more context- and obesity-, aging-, and insulin-sensitive than serum estimates—it found that changes after admission largely reverted after three months. Most prisoners tested in the literature are not new, so whatever underestimation happens due to incarceration lowering testosterone is not a very large problem.

But is this even interesting? Is testosterone correlated with criminal behavior in the general population? What about among prisoners? Looking at the whole literature including the various meta-analyses that have been done, the correlation between testosterone and criminal violence is just not strong. The correlations are dragged up by old, imprecise estimates from select samples; even then, they’re small, around 0.10.

Notice: I alluded to the fact that the meta-analytic estimates are dragged upwards by publication bias. If we correct for this, the correlations are very small indeed. Summarizing it, the correlation between aggressive behavior and testosterone is about 0.05 to 0.10 among the general population, and it appears to be a bit higher among offenders, at about 0.15 to 0.20—still quite a small correlation, especially given the stereotypes associated with testosterone.

Even this is overstating things. The extent to which criminals have higher testosterone levels is not that great, and it’s easily explicable by confounders. For example, they could be in status competitions or competing for sex more, they could have different diets, their violent surroundings could lead to higher testosterone, etc. The E-value—a measure of how large a confounder would need to be to overturn a result—for the prisoner-general population difference is small, at about 1.65, where anything below 2 is considered fragile. This could be easily be overturned by controlling for differences in substance use, the use of age-mismatched controls, smoking, and so on.

Another way to get at criminality’s relationship with testosterone is to look at recidivism. For the purposes of predicting sexual reoffending at least, testosterone seems to fail. It doesn’t make models any better than chance. This helps shore up our inferences about criminals, because this is about post-conviction behavior, and even this is unduly optimistic since the AUC is from the testing data.

These correlations are useful, but there is more, better evidence, too: there are lots of testosterone administration experiments! The ones that look at aggression turn up results that are remarkably similar to the general population correlations. That is, there’s not that much evidence of effects on aggression. Moreover, we also have some experiments on testosterone administration where the object wasn’t to measure aggression, but instead to assess impacts on health. In those studies—notably, TRAVERSE—, aggression incidents were recorded as side effects. They didn’t happen.

Let’s say you were to apply these numbers to explaining why men are more violent than women. Could it be done? Linear extrapolation suggests ‘no’—the gap in violent behavior is far too large to be explained by testosterone levels.

What about race? Could differences in testosterone levels explain why Black men are more violent than White men, for example? The answer to that is also no, for three reasons: total testosterone levels are highly similar with only a slight Black edge; free testosterone levels are highly similar with a slight White edge—the behaviorally-affective type, appearing contrary to predictions; and, the Black-White violent behavior gap is far larger than this could explain, much like the gap between the sexes.

We also have cohort studies available to see how testosterone levels affect men over time. In the CDC’s Vietnam Experience Study—one of their archival datasets put together to assess the impacts of wartime exposures to combat experiences and chemicals like Agent Orange—we have serum testosterone levels followed by years of subsequent measurements of behavior and self-reports of rage and short temper. There’s no correlation with self-reported anger, and the associations with dishonorable discharge, drug abuse, drunkenness, drunk driving, and remarriage are pitifully small at most.

One way to get at the effects of testosterone that I’ve seen suggested is to look at the effects of castration on sex offenders. It’s well-known that sex offenders who are castrated are far less likely to re-offend. But castration in studied settings has been voluntary, opening the results up to self-selection problems.

If we compared hormonal castration to other treatment methods, the possibility of a self-selection issue becomes apparent: hormonal treatments also lower testosterone, but they don’t result in nearly as large an effect. In fact, without this issue, the apparent reduction in reoffending is just not that large at all, like a Cohen’s d of 0.19.

The studies on castration—chemical or surgical—are ambiguous because the reason for the reduced rates of reoffending could be due less to reduced aggression and more due to reduced desire—a separate issue. No one contests that testosterone makes people much more aroused or that it helps with motivation. But those are distinct from aggression and could be independently related to certain types of criminal offending. And, to make this less ambiguous, recall that sex offenders’ testosterone levels are not unusually high or low at baseline.3

There is a group that has unusually high testosterone levels, and it’s bodybuilders! Anabolic steroid users, most of whom bodybuild, do have elevated rates of criminal offending. But this is because they’re selected: the people who take anabolic steroids are people who engage in other risky behaviors, have preexisting anger issues, etc. And in fact, despite having testosterone levels far above most criminals, they tend not to be criminals themselves.

Studies that control for polysubstance use—meaning drugs used beyond anabolic steroids—find that the risk associated with steroid usage mostly evaporates. I doubt that it’s just the other drugs driving this, and I see the use of other drugs mostly as an indication of other behaviors, but nevertheless, surely residual confounding remains, making the observation of steroid users, at most, consistent with small effects.

We can also make this less ambiguous by looking at trial results. In those, steroid users don’t appear to hulk out and fly into a rage that observers detect. The self-reported bump is also small, and it could just be expectancy, meaning that people who take steroids expect to be angrier, so they report being a bit angrier. Or, they could even be confused about their emotions, mistaking motivation for aggression. Regardless, there’s just very little evidence of strong ‘roid rage’ effects.

What about games? In the setting of games, we can see if testosterone administration or baseline levels predict aggressive behavior, risk-taking, teamwork, competitiveness, and so on. With acute administrations, basically nothing shows up in economic games. This matches another meta-analysis I covered recently, and a true (near?) null aligns with the observation that the various small studies on this topic have effects in all directions for outcomes like cooperation, reciprocity, generosity, and so on.

What about so-called ‘supermales’, men with two Y chromosomes? They are disposed to higher levels of criminality, but curiously, this is not due to testosterone. Their levels are actually not meaningfully elevated, and their heightened propensity for crime is exactly what’s predicted from their much lower than average IQs!

I think I’ve basically covered everything now, but let’s look at yet another route through which androgens might affect the propensity to be aggressive, do crime, etc. That is, prenatal exposures and the so-called ‘dual-hormone hypothesis’, whereby testosterone only affects aggression when cortisol is low. We should have a low prior on both things mattering for a few reasons. For one, prenatal testosterone doesn’t track adult testosterone very well for either sex. For two, cortisol is highly unreliable, so for this to be true and recorded as such in a meaningfully-sized sample would be incredible. For three, the dual-hormone hypothesis concerns an interaction for a small effect. We’re hoping for a suppressor effect, and those are rare in general. These fail:

Even natural experiments like (Complete) Androgen Insensitivity Syndrome, Twin-Testosterone Transfer, and Congenital Adrenal Hyperplasia all also fail to support the idea that testosterone is strongly involved in aggressive behavior. We can also go beyond those natural experiments and look at a much more permanent, lifelong testosterone effect via genes known to be associated with testosterone. That is, we can use Mendelian randomization. When we do this in the U.K. Biobank, what comes up is that genetically-predicted testosterone levels aren’t related to risk-taking, nervousness, mood swings, neuroticism, or irritability. Bupkes! Long-term bupkes!4

Testosterone is the ‘male hormone’. It’s stereotypically associated with everything to do with men: risk-taking and derring-do, aggression and violence, willpower and strength, balding and boners, and so much more. But much of this is not because testosterone does or causes those things, but rather, because it is the male hormone.

Men produce lots of testosterone; it masculinizes the body, it invigorates those who take it, and it can even cure depression. It can provide motivation, it can get you off the couch, and it can clear your mind. But taking it does not make one do all the stereotypical things men do. Men make testosterone; testosterone doesn’t make men.

As to the question of why men are more violent than women, I don’t have a great answer. But I do have an idea: what if it has to do with capability? What if the reason men are so much more aggressive is because it’s instilled in people that if they can be, they ought to be? What if it’s a social interaction, where people recognize that someone is capable of violence, so they fill the role?

I referenced the following studies in this piece: Geniole 2020; Book 2001; Archer 2005; Rada 1983; Cadena-Molina 2025; Cantos 2024; Giotakos 2004; Seim & Dwyer 1988; Wong & Gravel 2018; Dabbs & Morris 1990; Mazur & Booth 2014; Weinberger 2005; Lösel & Schmucker 2005; Schmucker & Lösel 2017; Kristensen 2021; Lundholm 2015; Klötz 2006; Chegeni 2021; Pope 2000; Tricker 1996; TRAVERSE / Lincoff 2023; Huit 2026; Dekkers 2019; Nave 2020; Pratt, Turanovic & Cullen 2016; Kingston 2012; Studer 2005; Witkin 1976; Götz 1999; Schiavi 1984; Stochholm 2012; Raznahan 2023; Zhao 2022; Cannarella 2025; Leggett 2010; Pasterski 2007; Robinson 2013; Reinisch 1981; Harrison 2021; Hanson & Morton-Bourgon 2009; Thompson, Dabbs & Frady 1990; Dreber, Nave et al. 2025; Sánchez Rodríguez 2026; Eisenegger 2010.

September 21, 2026 Update: Homosexuals

One line of evidence that I failed to note above has to do with sexuality. Lesbians (slightly higher) and homosexual males have similar testosterone levels to their same-sex heterosexual counterparts. Despite this, they vary considerably in their rates of violent and nonviolent criminal behavior, with lesbians reportedly committing considerably more offenses than heterosexual women and homosexual males committing considerably fewer offenses than heterosexual males. When it comes to actual criminal records, the ordering is heterosexual men > homosexual men > lesbian women > heterosexual women.

This variety of evidence is like the CAIS or SHBG-race evidence cited above: it doesn’t align with the strong, testosterone-centric hypothesis about misbehavior, but it is also ambiguous about other ideas. Nevertheless, it clearly fits alongside the other evidence.

September 21, 2026 Update - II: Trans Men

Trans men are women who take testosterone in order to transition into being men. They are a very self-selected group, so I didn’t mention them above, but because enough people asked about them, I thought I’d take the time to mention that there are some studies on them. These studies are uniformly uncontrolled and the whole literature is small, but they are still useful for helping to probe if there are nonlinear effects of testosterone in the in-between space between typical male and typical female levels, which show almost no overlap.

The first study I’ll cite covers several others, as it’s a systematic review of the relationship between testosterone therapy and aggression proxies. This review found that the evidence on aggression is weak: the self-reported tendency to react aggressively increased in 3/4 studies (3-months follow-up), angry emotions increased in 1/5 studies (7-months follow-up), and 1/3 studies reported a decrease in hostility. These results do not seem to support the view that there’s a strong, nonlinear increase in aggression in the range between the top of the female testosterone range and the bottom of the male one.

The second study I’ll cite came out this year. This one was a longitudinal study following 178 trans people who started taking testosterone over the course of a year. The study was no increases in reported anger or aggression, and in fact, participants were more likely to show a decrease in measures of irritability, anger, aggression, and externalizing problems.

The literature on trans men supports my general point expressed throughout this article and makes it more difficult to allege there are nonlinear effects of testosterone. It is, however, hard to interpret, because the observed mood improvements could be a psychological effect that follows from gender affirmation, reduced dysphoria, and so on, negating what would otherwise have been increase in aggressive behavior with testosterone. Because it would be unethical to randomly dose non-dysphoric women with cross-sex levels of testosterone like trans men take, it’s basically impossible to overcome this inferential problem. This is why I didn’t mention these studies earlier.

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

4

When we do the Mendelian randomization across samples using iPSYCH+PGC outcomes with the same instruments from the U.K. Biobank, we can avoid any issues with the IV becoming OLS since the samples don’t overlap, and in doing so we arrive at the result that testosterone predicts autism, but not ADHD. ADHD is a better violence predictor than autism, and there is at least one theory (the Extreme Male Brain Hypothesis) that predicts the autism result. Curiously, a female testosterone GWAS does not result in predicting autism. The architectures by sex are quite distinct, so I’m not sure what to think of this result.

’s most famous graph depicts the strange phenomenon whereby every racial group in America likes themselves, except for White liberals, who like other groups more than they like Whites.

I’ve seen this graph enough times that I got curious about how well it replicated, and what else we could learn from this sort of data. So, I grabbed the data and set about answering some questions.

#1: Does it replicate? Yes. The plot itself holds up in adjacent waves of the same survey.

#2: Does a self-derogating split across ideological lines extend to other races? No. Other races consistently like themselves, regardless of their politics. It’s just White liberals.

#3: Has it always been this way? No. White liberal self-derogation is a recent phenomenon, timed after the so-called ‘Great Awokening’ that came with Obama’s re-election and fomenting of race relations.

#4: Do White conservatives show an opposing trajectory shift? No. White conservatives are about as distinct from one another as they’ve always been. They even seem to have started going in the direction of their liberal counterparts, with weak evidence that such a shift has been larger for the less conservative among them.

#5: Do White liberals show a dose-response? Yes. More liberal Whites show greater racial outgroup preference.

#6: Do the results for ideology transfer to party? Yes and no. The more liberal party does show the same trajectory as liberals more generally.

The ‘No’ has to do with the dose-response: the split among Democrats is about ideology, not party attachment, and party attachment and ideology are only weakly correlated (r = ). As a result, we get this quite different-looking graph:

#7: Has this happened before? Not in recorded data. If we use the older ANES data going back to 1964, we don’t see this happen any other time. We also see that Blacks have reverted from the trajectory other groups were on, where they were heading towards having less ingroup bias over time. They are back to 1960s levels.1

#8: Is this because White liberals began liking Whites less or liking non-Whites more? Both.

#9: Does this replicate in other U.S. datasets? Yes. This seems to replicate whenever we have representatively-sampled data from the U.S., in which participants are asked a substantially similar question. For example, here’s alternative ANES results, the VOTER survey, and mention of two additional sources and one dead-end.2

#10: Is this responsive to events? Yes. We generally do not have granular enough data to see whether particular events drive this reporting, but we do for one event: George Floyd’s death. We have weekly polling from the time, and right after it happened, White liberals’ anti-ingroup bias doubled. White moderates were affected in the same way to a smaller extent, whereas conservatives were unaffected.

#11: Did the Floyd effect on warmth affect members of other races? Yes. From the same dataset:

#12: Does White liberal outgroup preference replicate in international datasets? No. Across Europe, Australia, Canada, etc., attitudes have trended in the same direction, but they have not become as severe as in America: on net, Whites abroad still tend to prefer their own national and racial ingroups, even if said Whites lean left. We see this replicate in the European Social Survey, Australian Election Study, the British Election Study, and so on. I’ll show two charts to make some points.

First, compare the European Anglosphere to the Continent in terms of views on other-race versus same-race immigrants. The preference for own-race immigrants is smaller in the Anglosphere:

Second, in Britain, the post-Brexit trajectories on immigrant attitudes are curious:

And I’ll toss in a third chart to make a contrary point: where America, Europe, and the rest of the Anglosphere differ: the various in-group attitude measures are most differentiated by politics in the non-U.S. Anglosphere, despite how different the patterns look.

Bonus: The Great Awokening had a lot of focus on Black-White relations. How do different groups feel about Whites relative to Blacks? Outside of non-liberal Whites, everyone now prefers Blacks, at least in polling.3

September 9, 2026 Update: Sex, Education, and Income

Over on X, I’ve posted four additional analyses. These show that, even though liberals are more likely to be women, that doesn’t modify the ideological relationship in the ANES; that education and income trends do not carry over across ideology and now liberals are homogeneous in belief despite differences in income and ideology; and the reaction to George Floyd was greater for women.

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

The Wordsum is a 10-item vocabulary test where you’re presented with a word and you have to then select the closest word to that one from a list. For example, you could be given the word “concern” and your options would be:

• See clearly

• Engage

• Furnish

• Disturb

• Have to do with

This brief test has been instrumental for understanding the relationship between intelligence—which, like all tests, it taps—and all sorts of social variables, from political views to incomes and education. But, despite its repeated administration and the considerable amount of use it’s seen, it’s been decades since it’s been properly validated. Accordingly, we don’t know how g-loaded Wordsum is.

The easiest way to figure out how g-loaded Wordsum is would be to administer a battery of tests with the Wordsum as one of the tests. From there, you would do a factor analysis and the loading would pop right out—simple. But in lieu of such a test, we can still validate the Wordsum through proxy methods. The way this works is simple:

• The National Longitudinal Survey of Youth (NLSY) 1979 and the 1997 version both administered a Word Knowledge test

  • This test is substantially the same as the Wordsum, it’s just longer

  • We have the actual test items for the 1979 cohort

• Both NLSYs (‘79 and ‘97) also administered a more extensive cognitive battery

• Both NLSYs and the GSS have extensive sets of demographic variables

• We can match the samples demographically

  • If we do this in time and place, it’s plausible that conclusions from the NLSY and the GSS can be transferred between one another

• We can reconstruct a ‘quasi-Wordsum’ in the NLSY ‘79 by matching items to the difficulty of the Wordsum items in the GSS

  • We can validate the similarity of this quasi-Wordsum to the real one by examining how it correlates with variables found in both the GSS and the NLSYs

• We can assess the g-loading of the quasi-Wordsum

• Thus, we can pretty much get a measurement of the g-loading of the real Wordsum

I noticed this was possible while working on a post about the relationship between intelligence and political alignment and how it’s changed over time. I was particularly inspired when I noticed that the IQ-by-politics gradient was four-times steeper in the NLSY ‘79 than in the GSS in the time around Obama’s election. Look:

I figured this was due to the fact that the NLSY ‘79’s testing was much more extensive, so I thought of how I could test that, and the fact that we had item data immediately came to mind because I had worked with it before. Accordingly, I took this snapshot from this era and began the demographic matching to the GSS before attempting to construct a Wordsum analogue. Remaking the plot with the matching plus the quasi-Wordsum ready to go, the difference is readily apparent, as is the validity of the quasi-Wordsum as a proxy for the genuine article:

After more extensive vetting, including looking at how well income and education were predicted, I concluded that the quasi-Wordsum does function like Wordsum, and the linkage with between Word Knowledge and other tests more generally is similar enough across the NLSY ‘79 and ‘97 that we can use the latter—which lacks item-level data for proper quasi-Wordsum construction—to validate the GSS’ Wordsum, too. All of this will be viewable in the replication package with the politics post I’m about to make.

For now, let’s look at a comparison of the AFQT and the quasi-Wordsum to figure out why it predicts politics worse. The answer is about half reliability differences (quasi-Wordsum is a short test with range restriction, like the real Wordsum is in the GSS), most of the residual is about low g-loading, and a tiny share is due to non-g ability variance. This looks like so:

As to the question behind this post, about how g-loaded the Wordsum is, the answer in the NLSY ‘79 is that the g-loading is 0.68 if we include the rest of the Work Knowledge items as indicators for the g factor, and it’s 0.62 without that, indicating a role for non-g ability variance. Disattenuating leads to g loadings of 0.83 and 0.76. In the NLSY ‘79 and ‘97, the g-loadings for the Work Knowledge subtest are 0.84 and 0.82, respectively (not significantly different), so we can reasonably assume psychometric comparability, and thus that the quasi-Wordsum g-loading in the NLSY ‘97 would be the implied Word Knowledge loading if the test had the reliability of a version with ten items: 0.66 (0.81).1

The Wordsum test is highly g-loaded and is a suitable proxy for intelligence, even if it would be nice to have a broader, more reliable, and less range-restricted test score.2

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

America has few large cities and they usually have a bad rap. They’re known for being crime-ridden, inhospitable places dotted by ‘bad neighborhoods’ and poor architecture, with hostile design elements, poorly-maintained streets, and unenticing public art. The prototypical example of this is Detroit, Michigan, which was once a booming industrial powerhouse containing nearly two million people, museums and gorgeous art deco homes, towers, and squares, and America’s highest household incomes. Now, it would be all too polite to call it a ‘hellhole’, and it’s not alone.

America’s cities are, by and large, bad.1 The easiest way to see this is to understand the primary reason why they’re so bad: crime. The big benefit of cities for countries is that they’re economic engines of progress due to agglomeration effects: benefits and cost-savings that appear because companies, services, and workers are geographically collocated in them. But agglomeration effects are negatively related to crime. Look:

But is this causal? Does crime reduce the extent of agglomeration economies? To a major extent, yes, and we know it thanks to a suite of different disasters—bombings, fires, earthquakes, hurricanes, etc.—, to random events that displaced populations—the Mariel Boatlift, Japanese internment, close elections that appointed discriminatory mayors, etc.—, to the historical period known as the Great Migration—the mass movement of African Americans from the southeastern U.S. to the rest of the country’s urban areas. My replication package deals with all of these, and they produce convergent results, but for this article, I’ll just talk about the clearest example.

The Great Migration started just before World War I and ended just after the passage of the Civil Rights Act. The era was marked by African Americans leaving the South for two main reasons: to escape Jim Crow laws and to seek economic opportunity in the factories of the North.

The Great Migration accelerated thanks to World War II, when draft- and casualty-related labor shortages allowed Blacks to move from low-skilled to semiskilled jobs en masse2, and it accelerated once again with the Supreme Court’s 1948 decision in Shelley v. Kraemer, which forbade neighborhoods from having state-enforced racial covenants—agreements to evict Blacks from certain neighborhoods and to prevent them from buying homes in them. What remained post-Shelley were merely private, much more loosely-enforced agreements not to sell property to members of certain races.

Labor shortages and racial covenants provide us with causal leverage: the former are driven by random deaths and pre-war demographics that are displaced by draft boards; the latter are driven in all sorts of unclear ways and broken due to preferences that are plausibly unrelated to the existence of the racial covenants in the first place. Interesting, no? Accordingly, where African Americans wound up and the occupations they were able to enter there had a large degree of randomness to them.

These facts, coupled with African Americans’ persistent association with violent crime, allow us to leverage their inflows to assess effects on agglomeration economies via crime. We can say ‘via crime’, since a mere racism angle doesn’t fit the data given that other racially distinct but lower-crime demographics that were discriminated against do not seem to lead people—in particular White people, in the phenomenon known as ‘White flight’—to leave from the places African Americans show up3, even controlling for socioeconomic status, and because anti-agglomeration effects are particularly distinct for violent crime, which is the type of crime African Americans are associated with, as opposed to property crime.

Cities that received more African Americans during the Great Migration saw larger increases in homicide.4

Cities attempted to counter this, and those that received the largest African American inflows increased hiring and funding of police in response.

Nevertheless, the homicide rates increased, violent crime overtook America’s urban centers, and White flight went into full swing, with Whites leaving metropolitan areas in favor of suburbs and new homes further afield. As a result, city total populations and population densities declined and labor productivity became worse off for it.5

Agglomeration is about two margins: density and returns to density, and crime is bad for both. We can observe and quantify the costs of crime-reduced density and density returns. The cost of the Great Migration density effects alone is about $888 per year per American, and the effect of the 1960s to early 1990s crime wave is another $392 per capita, but even those are lower bounds, as they’re just the costs we see.6

Cities are also arranged worse, leading to tangible and intangible losses. On the tangible side, cities are arranged in strange ways and their architecture is hostile so people can avoid crime; on the intangible side, patents are foregone, nightlife is more limited, police presence has to be greater and more concentrated, segregation is stronger, and NIMBYism—which is suboptimal for policy, as it also harms agglomeration, labor mobility, etc.—is reasonable as a means of exposure control.

We, as humans, not just Americans, miss out in untold ways because of the foregone agglomeration effects that result from our failure to have the gumption, the gusto to reclaim our cities from crime and dysfunction, let alone poor governance.

You can explore my replication package and all of its analyses by clicking here.

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

3

This is not reverse-causal. You can see this from the events I analyzed or from charts like this lagged one, which is expanded on in the replication materials:

Also, notice, the 1950s prior-decade result is nonsignificant. This is due to those inflows being very small.

5

Thus, estimated agglomeration benefits fell:

6

Chart-wise, the observable costs look like so:

It is also worth noting that this estimate excludes the direct cost of crime. It is just loss via the agglomeration channel. It also does not include any estimate of the effect of the Great Migration helping to agitate for the modern welfare state in the Civil Rights era whose legislation it inspired in order to quell the riots the resulted in Northern cities, and it does not quantify the unseen losses in innovation, industries that could more easily be preyed on by China as agglomeration economies fell, or having to enact more extreme policing and redistribution policies at the local level to accommodate Great Migration migrants, etc.

It is also worth noting that the Great Migration could have been a net positive financially despite the fact that it hurt agglomeration so much. This could be the case due to Black workers’ income gains as a result of relocating to Northern metropolitan areas where their labor would be more productive, even if, say, as Derenoncourt (2022) found, the move lowered long-run upward mobility (regression to the mean could explain this and thus flatters my estimate below). The counterfactual world where it happened without harming agglomeration—perhaps because cities worked to quell resulting crime issues more aggressively?—would’ve just been even richer.

So, I went ahead and estimated this. This estimate is only about private income gains. It omits gains to the South (i.e., wages of those who stayed rose as labor left) and the general equilibrium output from moving labor to more productive locales, and I’ll note that the density loss reflects a revealed preference for lower density in response to crime rather than a pure deadweight loss per se. The central estimate is that Black gains were 28% as large as Black migration-induced losses and the optimistic estimate is that they were 45% as large, and as I noted, this barely counts all the unseen and political downsides. Even if you count out Black-carried losses due to the agglomeration losses, the estimates of Black gains relative to aggregate losses remain lopsided. Here’s a helpful chart:

September 8, 2026 Update: Someone asked me about migrants’ role in North-South income closures. The direct effect would be making them closer together through the induced harms, but if there’s a federal policy effect such that the migrants inspired policy that resulted in the South catching up, then the gains from the migrants could be underestimated. But unless you fully attribute catch-up to that or a similar channel, it’d be hard for the gains to outweigh the harms.

In 1974, physicist Richard Feynman gave a commencement speech at CalTech entitled “Cargo Cult Science”. The title of the speech came from the cargo cults of the south Pacific, the belief system that arose among Melanesian islanders that they could make plane-shaped entities carrying supplies come down and enrich them by building things that looked like planes and runways and by marching in straight lines with sticks that looked like guns. After all, they had seen American airmen do just that and the airmen were rewarded.

Feynman took this notion and observed that something much like it happens in academia today: researchers regularly find themselves in their own cargo cults, doing rituals, hoping for boons, and achieving little for much the same reasons that the real cargo cults achieved little. I won’t recapitulate the whole speech, but I will note what I view as its most important bit: a relatively simple rat experiment that expanded into a testing battery so expansive no one had done anything like it at the time and no one has done anything like it since.

Feynman writes:

[T]here have been many experiments running rats through all kinds of mazes, and so on—with little clear result. But in 1937 a man named Young did a very interesting one. He had a long corridor with doors all along one side where the rats came in, and doors along the other side where the food was. He wanted to see if he could train the rats to go in at the third door down from wherever he started them off. No. The rats went immediately to the door where the food had been the time before.

The question was, how did the rats know, because the corridor was so beautifully built and so uniform, that this was the same door as before?

Can you see what professor Young is doing? The notion is simple: he’s trying to figure out why he got the result he did. Getting the result alone is insufficient to know why it happened. So, professor Young must do more experimentation to eliminate possibilities. He starts doing that by painting the doors to mess with the texture, but the rats can still tell where the food is. Then, he plays around with the smells, but the rats can still tell. Next, he realizes there might be something up with the lights in the lab, so he addresses that possibility, and still, the rats can tell.

Professor Young goes through possibility after possibility and eventually figures out that the way the rats are finding the food is through vibrations. So, he designs an experimental setup in which the maze is on top of some sand and thus the noises are blunted and the only way for the rats to learn the right door is to actually learn it. But, without going through each and every possibility, professor Young could’ve made a much more naïve conclusion; to really start learning about how rats learn, he had to run experiment after experiment to even make it possible. But then:

I looked into the subsequent history of this research. The subsequent experiment, and the one after that, never referred to Mr. Young. They never used any of his criteria of putting the corridor on sand, or being very careful. They just went right on running rats in the same old way, and paid no attention to the great discoveries of Mr. Young, and his papers are not referred to, because he didn’t discover anything about the rats. In fact, he discovered all the things you have to do to discover something about rats.

The subsequent research world was, unfortunately, unaware of the conditions that need to be met to be legitimate. It had developed into a cargo cult, where people knew the trappings of what research in the domain looked like, but without addressing the realities like professor Young did, real research would only be possible by chance.

Today, I read a paper on semaglutide (the active ingredient in the GLP-1RA drugs Wegovy and Ozempic) extending the lifespans of mice. The drug allegedly increased lifespans by 100 days—a long time for a mouse! This paper was cargo cult science.

Two years ago, a brilliant paper touching on caloric restriction benefits to lifespan (‘the Pabis paper’) came out and it showed that the lifespan extending effects were concentrated among naturally short-lived animal model strains. What this means is that chronic caloric restriction, where animals eat fewer calories than they would like over a long period, only seem to make animals live longer if they were already predicted to live short lives. But for animals that were longer-lived, the benefits were just not there; the observed ‘life extension’ was not so because it depended on there being problems addressed by caloric restriction that aren’t found in normally healthy animals.

By analogy to professor Young’s maze-running rats, the paper had found a condition that had to be met to show animal lives were being meaningfully extended: researchers must show that the animals would’ve already lived a healthy, normal-length life, and that the lifespan extension resulted in something beyond normally expected lifespans. Comparing treated animals to controls that are themselves unhealthy, bred poorly, or raised poorly just doesn’t lead to a conclusion that even plausibly generalizes to normal, healthy animals.

The new semaglutide paper does not meet this standard: the researchers behind it failed to set up an experiment that could inform us about semaglutide’s effects on extension of healthy, normal lifespan.

The Pabis paper discussed B6 mice, the types used in the semaglutide paper. It also advised avoiding the use of unhealthy, metabolically abnormal non-wild-type control animals and the semaglutide paper described their mice as healthy, naturally-aged, wild-type mice. But this is where they diverged: the Pabis paper recommended using controls with a median lifespan near the strain’s healthy optimum lifespan, which for these mice is around 900, plus or minus 50 days; the control in the semaglutide paper reached a median lifespan of just 742 day, and the treated group only reached 834.1

The semaglutide study attempted to anticipate this objection, but did so in a self-defeating way. They referenced six studies on B6 mouse lifespans to note that the control group had a lifespan that was close to typical. But the numbers in the citations don’t support their case. For example, the Ogiso et al. paper found mean lifespans of 798 ± 161 days for B6J females and 763 ± 163 days for B6N females, which includes early deaths—meaning it’s almost-certainly mechanically brought down relative to the numbers in the semaglutide study.

This simply cannot do! This study did not use animals with long enough lifespans at baseline to inform us about life extension! What is the point of such a study?!

Unfortunately, the semaglutide study has already been widely shared and applauded. People are using it to claim that semaglutide helps extend lifespans; they’re calling GLP-1s longevity drugs; they’re saying weight loss, increased movement, and so on will make people live much longer lives. All of this is unlikely to move maximal lifespan even if it helps to bring up the left tail of lifespan for the same reason that caloric restriction helps particularly messed up animals and not normally healthy ones.

This study reads to me much the same as observational studies saying things like ‘Owning a dog helps people live longer!’ (confounded by selection into who owns dogs) and ‘Vitamin D supplements will help your X, Y, and Z!’ (confounded by selection into who has high levels of vitamin D). People see the airmen flying their planes and dropping off cargo; they see the soldiers marching and eating their rations; they adopt dogs, take useless supplements, and run pointless studies. And we’re all worse off for it.

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

I recently read a new article that claimed that America’s housing affordability crisis was being driven not by a lack of supply, but by runaway inequality. I like this article. I don’t like it because it makes a convincing case—it doesn’t—, but rather because it sets us up for a teachable moment: it’s so bad that we can learn a lot about how not to do science or economic reasoning if we understand it.

The Literature Review

The article is mostly qualitative, involving the authors sketching a variety of weak, misleading, or completely incorrect arguments, without making a single important or useful point that isn’t simply a restatement of something widely known and understood. It starts off with a literature review that’s both selective and misrepresents the papers it cites. Their review is used to support this argument:

1. Housing regulations are local while housing markets are regional;

2. Upzoning in one locality can redistribute construction within a region rather than increasing total regional construction;

3. Many reforms raise land values while having mixed or tiny short-run effects on construction;

4. Therefore, deregulation is not a reliable way to make regions affordable.1

To put it bluntly, this is an annoying, lazy, and unhelpful style of argumentation. Noting a bunch of caveats for other people to deal with, without actually quantifying or qualifying their importance is unserious scholarship. It’s a caution fetish. If you’re going to attempt critique, try to be helpful and thorough when you do; you’ll be doing a service to yourself and to readers, and if you aren’t thorough, you might mislead people into thinking that the things you brought up are more important than they are.

We can easily add some quantitative color to the authors’ arguments, but before doing that, I’d recommend understanding something: there’s been a national permitting downturn. By that I mean, the number of permits to build new homes dramatically declined in a short period of time, right before the Great Financial Crisis (GFC), and the recovery has not yet been complete.2

Some metros saw slower and others saw faster recoveries. The extent of recovery can be leveraged to estimate the effect on housing prices.3 This isn’t causally informative without also incorporating the effects of policy reforms across different metros, but we have plenty of examples of that, so if you check my replication package, you’ll notice several plausibly causally informative specifications and results matching the general thrust of what I’m going to outline here and what other authors tend to find in the broader literature. That is, that more permitting means lower housing prices.

I first tested whether aggregate permits and housing stock growth were correlated across metropolitan statistical areas (MSAs). Pure redistribution among municipalities doesn’t have the ability to explain changes in total (population-adjusted) construction. Pure redistribution wasn’t even close to being supported. Instead, there was evidence that a weaker post-GFC permitting recovery predicted less housing stock growth, less household growth, less net domestic migration, and more rent growth. Other measures agreed, and using a holdout of pre-COVID era data showed that weaker permitting recovery before COVID was associated with more rent growth during COVID.

The result that areas that did more permitting see lower rents down the line survived state fixed-effects, multiple definitions including three based on permitting and one based on actual housing stock growth, and I was able to investigate the lag structure and building type specificity of permitting effects, too. I found that the benefits to housing prices appear to show up years down the line—which makes sense, given prices take time to adjust—and they’re particularly (2x) pronounced for multifamily unit construction.

The authors’ conclusions don’t seem very well-supported by publicly available data. The authors could’ve known this and reformulated their conclusions or at least qualified them better by saying that they only plausibly amount to quibbles. They didn’t. Instead, they went on to argue

1. Overall vacancy is now higher than it was in 1970;

2. Between 2000 and 2020, household formation didn’t exceed housing supply growth in major urban areas;

3. Average household sizes fell from 3.2 to 2.6 people in that period;

4. Accordingly, if housing were genuinely scarce, vacancies would be falling and people should be forced to live in larger households.

These facts do not align like the authors suggest though. There are actually several critical and well-known problems with how they did their reasoning.

Firstly, a “vacant” unit is everything from people’s seasonal and second homes to units that are between tenants to units that are located where there’s little demand, and very critically, units that are condemned or otherwise uninhabitable. National vacancy statistics are incapable of identifying whether rental housing is scarce in a particularly in-demand metropolitan area unless one gets extremely lucky. Secondly, household size is endogenous to age, fertility, marriage, income, migration, housing quality, and other aspects of demographic composition. Declining household size can itself increase the demand for housing—and it does! Finally, comparing total household growth with total household stock growth completely neglects prices! It doesn’t mean a lot for affordability if supply ‘keeps up’ by prices rising enough to stall household formation, exclude certain households, inducing out-migration, and preventing in-migration.

The authors could have tested this idea too. The idea is to use permitting numbers to predict stock growth and subsequent effects on household and resident numbers as well as vacancies and rent growth. Weak permitting recovery does indeed seem to come with lower housing stock growth, subsequently fewer households and residents being locally accommodated, tighter vacancies, and higher rent growth.

The authors then reviewed some literature on supply-to-price elasticities, emphasizing that builders like to build where demand and prices are rising so the correlation between construction and prices is affected by endogeneity. They note some modest, <1 elasticities and suggest that stock shortfalls are therefore unimportant. This point does not follow in any case, as we’re not really interested in the rent change this year divided by the permits this year, we’re interested in something more like the cumulative rent differences several years prior over the cumulative completed difference in housing stock over time. The lag tests act as relevant evidence here, and the authors should have done them themselves.

As the last bit of their review, the authors looked at a small subset of the evidence on filtering, the process by which housing becomes more affordable as it gets older. To get at this, they looked at Rosenthal’s “repeat income” model, which, in their review, implied that there’s roughly 2.5% annual downward filtering for rental units, as measured by occupant income, and 0.5% annual downward filtering for owner-occupied housing, with just about 0.31% annual price depreciation for multifamily rental housing. They note, correctly, that filtering is heterogeneous across markets and it can reverse in some cases.4 Nothing to test here given their narrow focus.

The Calculator Fallacy

With their background out of the way, the meaty thing the authors did was a simulation. For their simulation, they chose six particularly expensive commuting zones—San Jose, San Francisco, New York, Los Angeles, Washington, and Boston—and they paired them with Census data on median recent-mover one-bedroom rents, median annual wages for full-time workers without college degrees, and an “affordable” monthly rent equal to 30% of those workers’ wages. They then specified an annual housing stock growth rate of 1.5% and calculated R = C \times E + F, where C is annual construction, E is the price elasticity to supply, and F is filtering or depreciation. They then compound that until the initial rent reaches their affordability threshold: T = \frac{log(W/H)}{log(1-R/100)}.

The two simulation scenarios were “Optimistic” and “Pessimistic”, and they were defined by construction rates of 1.5% in both, respective elasticities of 1.0 and 0.2, filtering of 2.5% versus depreciation of 0.3%, and respective assumed annual declines of 4% and 0.6%. With these scenarios in hand, the amount of time required to reach affordability ranged between 9 and 22 years optimistically and 60 to 150 if we’re going by the authors’ definition of pessimism.

But this is basically a pointless exercise. There’s no causality indicated by this simulation. Their simulation is, in fact, just a calculator: you input parameters and get mechanically determined figures out, and they may or may not correspond with reality. The result they get is the result they put in, and due to the simulation’s lacking parameters, it’s doomed to be unrealistic and to have inputs that are highly sensitive to the specification. For example, with defensible specifications based on numbers and ranges we can find in the paper and its sources, San Francisco could take between 18.3 and 124.1 years to become affordable.

What if we added migration? Well, it could take much more or much less time for every city to reach affordability then. You can actually do this with any number of plausibly contributing parameters, but regardless of what you add or subtract from the equation, you never get any closer to being able to make a conclusion for a simple reason: being able to calculate a number doesn’t mean the number matters.

Just building homes may take a long time to make median market rents affordable for low-wage workers, sure. This is a conclusion that’s supported by the authors’ exercise because in some universe, the simulation could be sufficient to describe reality. But the exercise cannot be used to make claims like ‘Supply has limited effects on rents.’ That can’t follow because the equation can suggest supply materially lowers rents while also leaving them unaffordable for particular households, no contradiction.

So What?

Using Census data from 1970 to 2021, the authors showed that among employed workers aged 16-64, there’s been much faster real wage growth for college and graduate workers than for those with lower levels of education. This is a well-known phenomenon, but the explanation actually seems quite lost on the authors. Most of it simply reflects that higher levels of education have become normalized and having some education is normal now, too. Sorting into higher educational categories and out of the lower ones has become a major force shaping the composition of different educational categories: a college degree is now normal, not impressive; only graduating high school is now below-average, not above-average; etc.

Presenting these education facts doesn’t really tell us much of anything, but the authors want to use them to convey that inequality has increased, thus providing them a mechanism for their main claim, that increasing inequality has caused the housing crisis. They basically argue by insinuation, by vibes.

To do this properly, unlike what the authors did, you’d use wages instead of education, and you’d regress real rent changes on changes in wage dispersion over time, using multiple margins and different timings. I tested the effect of the 90/10 ratio, 90/50, 50/10, 75/25, 90/25, 75/10, the Gini coefficient, top-5% and top-1% shares, etc. Coefficients ranged from piddling to paltry and were unstable and virtually always null with one-year lags.

Instead of running those sorts of informative tests, the authors then went on to show a scatterplot of commuting zone college-educated shares versus their subsequent college share growth rates and a chart showing a comparison of commuting zones by their 1970 college shares for the top decile for most initially educated cities versus those around the median. The comparison involved indexing two differences: absolute wage differences between city groups and differences in their college/noncollege-educated wage ratios.

This is supposed to show evidence of growing spatial inequality and “the shift from an American economy rooted in building physical objects to one producing knowledge", but it doesn’t show divergence across all metropolitan areas, a relationship between income and rent divergence, inequality being an operative variable, housing supply playing no role, or a conclusion that’s robust to different cutoffs for the margin for inequality. The authors just kind of showed some descriptive statistics and left it at that, when they could’ve done some testing.

The way I did the testing was in stages. In the first stage, I answered the question: do initially richer metropolitan areas subsequently pull further ahead? Between 2000 and 2009, initial pay levels predict convergence, 2009-19, there’s nothing, 2019-24, there’s a slightly negative relationship, and over the whole period 2000-19, the estimate is negative and marginally significant (p = 0.02). This is not what the authors expected.

In the second stage, I answered the question: does local pay growth predict rent growth? Between 2000 and 2009, rent growth correlated slightly with pay growth, the same held for 2009-19, and for 2019-24, the relationship fizzled out. So then, I answered the final big question the authors left unaddressed: do pay and rent dispersion move together geographically? The answer here is definitively ‘no’. From 2000 to 2019, employment-weighted pay dispersion rises roughly 5%, whereas rent dispersion rises some 38%, and the metropolitan-area 90/10 pay gap was flat—with a decline by 2024—whereas the rent gap on the same margin continued widening.

The authors were kind of gesticulating at a spatial sorting mechanism that evidently doesn’t matter in the ways they wanted it to. They then went on to do even worse. Look at their Figure 3:

Based on this figure, the authors tried to argue that, rents rising with mean income while noncollege-educated wages lag shows a wedge that indicates the fundamental inequality mechanism at work driving up rents to unaffordability. But if you’re familiar with this website, you probably know that you shouldn’t make this sort of conclusion based on comparisons of trending levels of things. Trending variables are frequently highly correlated even when their short-run movements are uncorrelated and there’s no causal relationship. This is how we get ‘organic food causes autism’ or ‘the president’s age is causing the U.S. to increase in age, r = 1’.

More problematically, this is a select sample, not an illustrative set, and it deviates from the other figures the authors presented in that there’s no label indicating how inflation was dealt with. So, I ran three tests:

1. I checked the employment-weighted national data for 2000-24 and found a real income/rent level correlation of r = 0.872 (which is substantially outlier-driven), but an annual change correlation of just r = 0.220;

2. I added metro and year fixed-effects. The real level elasticity was 0.628 and the model R² was a staggering 0.922, but this extreme variance explained was due to the large number of metro and year fixed-effects in the model. Income’s partial R² was just 0.0065;

3. I used annual differences.

   1. With year effects, the contemporaneous real pay elasticity was 0.045 (p = 0.392);

   2. With pay growth lagged by a year, the elasticity was 0.152 (p = 0.001).

Basically, real levels track but the trend moves slowly, income really explains little, and income is a lagged demand input, while the “almost perfect” tracking between mean income and rents that the authors illustrated overstated how much income explained. They basically did a lot of handwaving to justify nonsense eyeballing-based conclusions, and none of it stands up to scrutiny.

What Are We Even Doing Here?

The inequality-based mechanism cited by the authors is:

1. Mean income rises, driven by affluent households;

2. Mean income determines housing demand and rents;

3. Median and lower-tail wages fail to keep up with this top-led rise;

4. Rents become unaffordable.

Thus, the authors should predict that mean income growth should outperform median income growth in predicting changes in rent. So—and I admit this is a very crude test—I fit simultaneous regressions including mean and median income and weak permitting recovery as a sort of horse race. Because mean and median income are highly correlated, partitioning the variance between the two can be hard, but the authors still predict that the mean should be better than the median. Alas, it is not: alongside the median—which predicted much more rent growth—the mean generally predicted slightly negative rent growth. Weak permitting recovery (meaning under-permitting relative to the pre-GFC regime) consistently predicted higher rent growth.

This could reflect endogenous migration and selection whereby higher rents expel poorer households and raise the observed median, but that doesn’t rescue the authors’ inequality claims! If we want to really test those, we’ll compare the effects of a weak permitting recovery and wage dispersion growth—at any margin or Gini—directly, and we’ll find…

As I noted at the beginning of this article, I ran a lot more tests and used a lot more measures and looked into a lot more scenarios than I’ve elaborated on here, and more importantly, than the authors of the paper I’m focused on likely even considered. There is, to be frank, basically no case for what they argued, and it’s very odd they argued what they did given that it was fully possible to go and test many of their claims and predictions, and every test came up wanting.5

I like the paper this post is about, not because it’s good, but rather, because it’s so bad. If you can understand why it’s bad, you can understand why so many simulation studies are worthless; why selective and partial literature reviews can mislead you; the importance of thinking through the implications of your claims and ensuring that they hold up before you put them out there for the world to judge; etc. This is a terrible piece of scholarship, but it would make for a great article for undergraduate economics or statistics students to spend an hour or two disassembling.

The study is basically a work of advocacy and an expression of ideology, not a contribution to the field of economics or to our collective knowledge. What it says that is true is widely known and reported by authors who’ve surrounded those facts with less baggage; what it says that’s novel is false or exaggerated to the point of being misinformation.

To the extent the paper might be intended to inform on policy—and on that note, all it seems to want to do is denigrate supply-side reforms—it should be laughed at. The paper is all about ‘can’: inequality can push up rents. But does it ‘tend’ to? No, and policy papers need to be about how policies ‘tend’ to work. Removing restrictions on building housing does tend to reduce rents and home prices. Would said reforms do so as a rule in each and every place they’re implemented? Certainly not. But that’s not what’s relevant for policymakers. They need to know what to expect, not to hear a bunch of quibbles that make it sound like the general tendency of a policy is to do nothing or to have the reverse of its realized effects.

We should qualify our policy expectations rather than thinking solely about general tendency, but we should not do so like this paper did, in a manner that makes doing anything seem futile. If these sorts of leftists eking out an existence in the academy had their way, we would simply do nothing because of the possibility that doing something leads to everything going wrong when we know things should go great.

You can see the replication and extension code for this post by clicking here!

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

2

The timing of the recovery window doesn’t really matter. I tested the effect of making the window start in every year from 2010 to 2014 and the estimated effects of slower permitting rate recovery were statistically and practically indistinguishable across windows. See also:

5

One of my favorite tests was when I looked at the recent reversal in wage dispersion that started under the Biden administration. Though the authors would predict that it would make housing more generally affordable, it didn’t lead to any deviation from trend. I also thought messing around with minimum wage data resulted in some cool findings:

Data centers are, in a word, unpopular. They’ve quickly gone from the sorts of infrastructure projects that communities are neutral towards or welcome to being the sorts of projects that enjoy nearly universal revulsion. This is exaggeration, but not by much: the latest polling suggests about 75% of the population dislikes data centers.

Why? Data centers are clean, they pay high taxes, during their construction they pay a lot of blue collar workers and after construction they continue to stimulate the local economy, and they have very little impact on things like local utility costs or housing prices. But this is not what people believe. When you poll them, they seem to consistently believe that data centers are bad: they pollute and they increase costs.

But data centers do not greatly stress local resources or increase costs, so this is an odd set of views. If you’re very online, you might expect that the top reason for opposition to data centers would be something AI-related; that’s at least something real and worthy of concern, whether it’s an AI extinction risk fear or dismay about having one’s job replaced by machines. Alas, that is simply not it, and to suggest otherwise is to suggest that survey respondents are lying, and much worse, it’s a view that can’t even be falsified: if we believe survey respondents are actually worried about AI and they’re just fibbing or answering in confusion, then we can’t justify that view.

Because so many surveys have now been run, we can triangulate what portion of the resistance to data centers is driven by specifically AI-related sentiment. The triangulation evidence is as follows, from strong evidence to evidence that’s merely contextual:

• Explicit AI Wording. Generic wording produces considerable opposition, and adding explicit AI wording adds only a relatively small penalty. For example, Change Research in April 2026 found generic wording produced 51% opposition versus AI-specific wording, which produced 55% opposition.

• Matched AI-Purpose Comparisons Within Surveys. Across different survey instruments, AI-specific purposes consistently make data centers less popular, but not by a lot compared to the baseline levels of resistance. Echelon recorded AI purpose wording made support “less likely” by 17 points in February; Groundwork found it reduced favorability by 12 points in March; Rainey found that it added 6 points of opposition in May; Echelon found that it added 9 points of opposition in June. To contextualize those values, recall how much people already disliked data centers.

• Combined Study-Level Estimates Across the Polling Corpus. 41 direct-siting studies have been done between March 2025 and August 2026, and in the narrowest date-adjusted comparisons of exact local questions, predicted decided opposition is 65.1% generic versus 71.7% AI-specific. Across all distances, that increment averaged 7.7 points, so it’s fair to say AI adds 6-8 points over an already low level of support and high level of resistance.

• Temporal Divergence. In Annenberg’s February-July 2026 polls, local opposition to data centers increased by 12 points, but negative expectations about AI dropped by 3 points in the same period. Opposition rose across every subgroup they analyzed while negativity about AI moved in both directions, but generally down.

• Repeated Generic and AI-Specific Polling Over Time. All 13 comparable, repeated series became more negative on data centers by between 3 and 39 points, with the largest changes happening under generic wording, not AI-specific wording. Moreover, the opposition followed the same pattern we would expect if NIMBYism were to blame for opposition: it increased from 14 to 51% for a national generic question, from 26 to 65% for a local one, and from 40 to 73.5% for a near-home one. Were AI to blame, locality shouldn’t matter.

• Randomized Facility Comparators. Searchlight’s March polling found that opposition at five miles was 39% for warehouses, 54% for data centers, and 69% for workerless automated factories. Puck News’ June polling found more that data centers had 4% less opposition that nuclear power plants, but AI-specific data centers were just 5% more disliked than those plants. The gaps weren’t large and they’re not consistently tilted against AI-specific worries.

• Spontaneous Supplied Reasons. In Gallup’s Spring poll and Fox’s July poll, when opponents of data centers were provided with the opportunity to supply their reasons, AI concerns ranked much lower than environmental ones. In the Gallup poll, 14% cited negative AI views and 13% cited specific concerns; in Fox’s polling, 11% cited negative AI views. 50% of each poll’s respondents cited environmental concerns.

• Respondent AI Worldviews. Milltown’s polling suggested that negative AI views were common among opponents and not among data center supporters, and harm to the economy was associated with huge swings against data centers. People who thought they would be harmful were 28.1 points more likely to be opponents, and those who had career pessimism were 15.2 points more likely to oppose. Notice the contrast there: general fears versus personal ones, and swings so large that if these were common motivations, AI-specific data centers should have even less support and greater opposition.

• AI Opinions versus Local Opposition. In Fox’s July polling, 54% of participants had unfavorable views of AI versus the 70% who opposed data centers.

• Prompted Concern Batteries. Across different pollsters, environmental concerns lead, and AI concerns range from 5% as the single biggest to an outlier 52% in a multiple-response battery, with an average far below many other issues.

• Material Condition Moderation. Presenting participants with data centers that run off of clean or have their own dedicated power improves support, local electricity and water concern alleviation outperforms talking about winning the race with China, and cost and resource protections move support among 65-85% of people (note: this is not saying that it moves 65-85% to support, but that most people move in that direction).

• Age and AI Labor Patterns. AI job losses are expected to lead among ages 18-24, but opposition and environmental worry are nearly flat across ages. In some polls, like Fox’s recent one, the under-30 and 65+ differ substantially on sentiment towards AI, but they’re almost identical in how much they locally oppose building.

• The ‘Literally Who?’ Reality. When you poll people on which corporate leaders, companies, and products they recognize and how they feel about them, the results don’t comport with the idea that AI woes drive concern.
  
  99% or more of people have heard of AI, 99% or more have heard of Google, about 98% know of Microsoft, 95% know of Meta, 97% know of Amazon and Apple, etc. But move down from household names and awareness falls off: 82% for OpenAI, 64% for Nvidia, 62% for Anthropic, 51% for DeepSeek, 45% for Palantir, and 30% for DeepMind. The products don’t match the corporation awareness: ChatGPT sits at 92%, Gemini at 84%, Copilot at 81%, Grok at 64%, Claude at 63%, Perplexity at 53%, and specialized products are usually at 40% or less. Musk and Zuckerberg enjoyed high recognizability (98% and 95%, respectively), but beyond that, major AI company CEOs are not widely known: 57% for Sam Altman, 52% for Peter Thiel, 49% for Larry Ellison, 35% for Jensen Huang, and 26% for both Alex Karp and Dario Amodei. Then, the ratings don’t even match! Karp is viewed positively among those who know him, and Dario is, too!1
  
  But then, problematically for the narrative that AI is driving concern, it’s polling better over time, more people are using it over time, and favorability is higher among users than nonusers, with signs of some increase over time, unrelated to history of prior use. Moreover, people simply don’t even seem to believe that massive AI job loss or extinction concerns are realistic possibilities!

There are several lines of evidence that point to the most realistic conclusion: data center concerns are about the things people say they’re about. People are actually worried about things that are not realistic concerns, like electricity prices and water availability. Opposition is not due to worries about AI extinction risk, AI job loss, AI threatening people’s happiness and family time, or anything else like that. AI only adds a small bit of extra ‘oomph’ to the opposition, and the reason is ambiguous. For example, when asked about data centers versus AI-specific data centers, people could oppose it more because they think AI data centers have worse environmental impacts!

There are a lot of things we need to consider in order to get the public to support building more data centers. To do that, I suggest we stop drawing up and relying on narratives that explain why people in our circles might be against data centers, and start, instead, listening to the people who are actually on the ground and in the polls telling us—explicitly—why they oppose them. All signs suggest they’re not lying; they point instead to the much simpler fact that the opposition is mostly just NIMBYs.

Here is the list of surveys I reviewed and/or referenced, listed in field dated order. 2023: Washington Post/George Mason Schar School, 2023; CNBC/Generation Lab youth-and-money study, October 2023.

2024: Axios Harris Poll 100, 2024; CNBC/Generation Lab youth poll, January 2024; Morning Consult AI Super Users, March 2024–March 2025; CNBC/Generation Lab youth poll, April–May 2024; Data for Progress/Accountable Tech, July 2024.

2025: Anne Arundel Community Survey, 2025; HostingAdvice, 2025; Wason Center Virginia survey, January 2025; Public Policy Polling/Tech Oversight Project, January 2025; Change Research, March 2025; Airedale/Modine neighborhood survey, March 2025; Illinois data-center tax-incentive survey, March 2025; Overton Insights, June 2025; North Carolina LCV Foundation, July 2025; Data for Progress/WE ACT, July 2025; Ohio data-center tax-exemption survey, July 2025; South Carolina Policy Council, July 2025; Embold Research/Heatmap, August 2025; Arizona data-center survey, August–September 2025; Minnesota data-center accountability survey, August–September 2025; AARP national 50+ utility-cost survey, September 2025; AP-NORC/EPIC Climate Change survey, September 2025; Mississippi Omnibus Survey, September 2025; Georgia data-center tax-and-bills survey, September 2025; Fairleigh Dickinson University New Jersey survey, October 2025; Marquette Law Wisconsin Poll, October 2025; Data for Progress Alaska, October 2025; Morning Consult, October 2025; Texas data centers and electric-grid survey, October 2025; Harvard Salata Institute/YouGov, November 2025; Hoosier data-center and energy survey, November 2025; Ipsos/Redfin, November 2025; Elon/North Carolina data-center electricity-cost survey, November 2025; Emerson Pennsylvania data-centers-and-AI survey, November 2025; DHM Washington energy survey, November 2025; Morning Consult, November 2025; Caltech/YouGov, December 2025; Navigator Research/Global Strategy Group, December 2025; CCAN Action Fund/Global Strategy Group, December 2025; Change Research, December 2025; Harris Poll data-center survey, December 2025; Blue Rose Research, December 2025; Emerson national poll, December 2025; YouGov daily survey, December 2025; Louisville Metro Office of Planning survey, 2025.

2026: Value of Water Index, 2026; ACSI AI Platforms Study, 2026; Cherokee Nation citizen survey, 2026; Reach3 Insights, 2026; YouGov AI Index/BrandIndex, 2026; Wason Center Virginia survey, January 2026; Georgia pre-legislative survey, January 2026; Morning Consult/Stateline, January 2026; Public First/POLITICO, January 2026; Pew Research Center, January 2026; South Carolina Policy Council, January–February 2026; Climate Power/Blue Rose Research, January–February 2026; Amber Integrated/Oklahoma Watch, February 2026; Embold Research/Heatmap, February 2026; Glengariff Michigan survey, February 2026; SLU/YouGov Missouri technology survey, February 2026; Ohio voter energy survey, February 2026; Marquette Law School Poll, February 2026; Axios Harris Poll 100, February–March 2026; Nexus Pulse, February 2026; Annenberg Public Policy Center/SSRS, February–March 2026; Echelon Insights, February 2026; Quinnipiac University, February 2026; Rainey Center, February 2026; Clean Virginia, March 2026; Gallup, March 2026; South Dakota issues poll, March 2026; Elon University North Carolina poll, March 2026; Marquette Law Wisconsin Poll, March 2026; Illinois data-center regulation survey, March 2026; Searchlight near-home facility comparison, March 2026; Pew energy-bill survey, March 2026; Quinnipiac University, March 2026; Fairleigh Dickinson University, March 2026; Washington Post/George Mason Schar School, March 2026; AARP Oklahoma 50+ utility survey, April 2026; Montana Conservation Voters, April 2026; University of Houston SPACE City Panel, April 2026; Center for American Progress utility-price message testing, April 2026; Zencity/TransformGov, April 2026; Change Research, April 2026; South Dakota News Watch/Chiesman Center, April 2026; Climate Power/Global Strategy Group, April 2026; Searchlight energy-price survey, April–May 2026; Glengariff/Detroit Regional Chamber, April–May 2026; Alabama Republican-primary survey, May 2026; Conservation Colorado, May 2026; Hahn/Escalent, May 2026; Ipsos/Redfin, May 2026; Morning Consult consumer-AI reputation study, May 2026; Reuters/Ipsos, May 2026; Ipsos Consumer Tracker, May 2026; Embold/Compass on-site energy experiment, May 2026; Milltown Partners, May 2026; Change Research Florida, May 2026; UMass Lowell Maine survey, May 2026; Embold Research/Heatmap, May 2026; Rainey Center experiments, May 2026; Utah Stratos survey, May 2026; FM3/Net Zero California, May 2026; UNH New Hampshire AI survey, May 2026; Justice Research Group/Working Families Power, May 2026; The Argument/Embold company-favorability survey, May–June 2026; Environmental Polling Consortium and partners, June 2026; University of Virginia Data Center Opinion Project, June 2026; Utah SD-21 Republican-primary survey, June 2026; YouGov Millennial crossbreaks, Q2 2026; YouGov fame and popularity trackers, through Q2 2026; CBS News/YouGov, June 2026; Reuters/Ipsos AI survey, June 2026; University of Texas/Texas Politics Project, June 2026; AI Policy Institute, June 2026; Echelon Insights, June 2026; Data for Progress moratorium survey, June 2026; Fox News/Beacon Research/Shaw & Company, June 2026; Annenberg Public Policy Center/SSRS, June–July 2026; VCU Commonwealth Poll, June–July 2026; PPIC/Ipsos California survey, June–July 2026; ARI/Morning Consult, July 2026; Hancock County open survey, July 2026; Public First/POLITICO, July 2026; Harvard Harris Poll, July 2026; Empire Center New York energy poll, July 2026; Quinnipiac University, July 2026; Marquette Law Wisconsin Poll, July 2026; Fox News/Beacon Research/Shaw & Company, July 2026; Emerson national poll, July 2026; DHM Oregon Pulse, July 2026; Elon University North Carolina poll, July 2026; Wyoming public-lands poll, July 2026; Data for Progress Alaska, July–August 2026; Napolitan News, July 2026; CNBC/Generation Lab AI-leader trust poll, August 2026; Embold Research/Heatmap, August 2026.

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

Depending on the poll, the two biggest issues the public has with data centers are:

1. They take all the water;

2. They increase electricity prices.

The former complaint is scientifically illiterate, dead and done, and gets its limited intellectual backing from analyses that were shown to have included mathematical errors that inflated water use estimates and others that have spuriously attributed water use elsewhere to data centers. The latter complaint has been found to be not just false but backwards for the period before today’s hyperscale buildout, so proponents of the idea have shifted their defense to stating ‘That might’ve been true, but now the buildouts are radically increasing electricity prices!’

I’ve decided to take this possibility seriously. To that end, I’ve made a continuously-updated website that assembles that latest available data on data center energizations and electricity prices. You can explore the website—wattsmybill.org—by clicking here.

What did I find? Firstly, there is an effect of new data centers on electricity prices. However, the effect is small, and it appears to plateau. Secondly, the effect of new data centers on electricity prices is driven most sharply by large, recent buildouts. Finally, projections based on the existing data suggest there’s plenty of room to build out data center capacity without major effects on electricity bills.1

Here’s how average price effects from a “hyperscale”—250MW—data center being built locally look for a typical household as of August, 2026; in short, they’re small:

If you want to play around with the data, check the expected effects of data centers on your bill, or just explore all the analyses I ran, go check out the site by clicking here! If you have requests for additions to the website, then go and leave a comment below.2

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

Data centers are very unpopular right now, for all sorts of reasons. The latest polling is abysmal: people want a data center in their community less than they want a new nuclear power plant, and they really don’t want to live near a new nuclear power plant.

The reasons people supplied for not wanting a data center will be familiar to you if you’ve seen other polling. The most slanted beliefs were that data centers strain the electrical grid and that they consume extreme amounts of water, pushing up the utility bills of local residents. People even tend to believe things that are extraordinarily obviously untrue, like that data centers tend to lead to increased taxes and lower property values. Apparently they don’t see the contradiction, but I digress.

The unique thing in this poll was that the pollsters sought to see if they could change people’s views. They wanted to test if they could unravel people’s misinformed views about data center harms by telling them the truth. Unfortunately, people don’t tend to believe the truth, even when you present it to them directly: they don’t think data centers generate tax revenues, that they make AI more widely available, that they create high-paying jobs, power modern business applications, or—critically—that they actually don’t use very much water.

Given how much disdain people hold at baseline, how strongly they hold their most errant beliefs, and how unconvinced they are by statements of fact, you might expect supplying facts to have little effect. You’d be right! People’s opinions moved in the direction of supporting data centers, but by a barely significant amount.1

How Can We Usher in the Future?

There can be little room for doubt that AI is the future of the global economy. To develop and power that AI, we need to increase the available amount of computational power, which means more data centers. But the public opposition to data centers is acting as a very effective Luddite revolt against realizing the benefits of this future by bringing the promise of AI to fruition. How exactly do we stop it?

My first thought was to look at the results of other persuasion experiments. When researchers have assessed the impacts of political campaign messaging on persuasion, they’ve generally come up empty-handed. Experiments on persuading people to be less hateful through interpersonal conversation sometimes turn up results, but in the best analyses, the effects are meager. The effects of online political advertising on the propensity to vote for a given party have also been estimated to be small and they fade fast, much like TV ads. Common shortcuts like deploying people with shared demographic characteristics to do persuading also don’t seem very valuable.

Persuasion experiments in the published literature turned out to be mostly irrelevant, so I was disappointed by them. But, I did find some NIMBY messaging experiments.

Monkkonen and Manville asked 1,300 LA County adults about hypothetical apartment buildings proposed on their block. They presented the participants with arguments against buildings the apartments, and all such arguments helped to convince people to be against building, often by a very considerable degree. For example, telling people that the developer serves to make large profits increased opposition by 20 whole percentage points. To get people to support building required basically bribing them: most participants (63%) increased their support for projects if developers offered community benefits.2

Walker, Wiersma and Bailey ran a postal survey in England, where they asked people about an offshore wind farm near them. Some participants were asked about it without any framing, others were told about community benefits, and the rest were told about the wind farm’s community benefits and common counterarguments. Without the counterarguments, people became more supportive of the wind farm when they were presented as a boon to the area. The whole support boost evaporates the moment wind farm project benefits are labeled ‘bribery’.

Liebe, Bartczak and Meyerhoff asked Germans and Poles about wind farms near them and found that, firstly, Poles are more accepting or new turbines than are Germans, and secondly, people more readily accept wind farms when they’re owned by local citizens, when the electricity is consumed regionally instead of being exported, and when citizens perceive that their community is involved in the planning process. A lucky finding was that those who had already been exposed to wind turbines were more accepting of additional wind turbines, but it’s not exactly clear why this is. It could be that exposure indexes prior support rather than an exposure effect per se.

Vuichard, Stauch and Dällenbach asked Swiss people about what would increase support for local wind farms, focusing on financial inducements. They found that the only thing that worked to raise acceptance was a local resource tax that benefitted everyone in the community. This was politically split. Conservatives just weren’t convinced at all, and leftists had generally high support for the wind farm and only had their support boosted with the wind resource tax idea. However, centrists actually saw notably increased support with every inducement—they just saw the most with the resource tax idea.

Knauf ran a conjoint experiment in Germany where they found something similar: financial benefits for wind energy projects increased support, but only among those who already supported them or who didn’t have any strong views. Among those who were opposed, benefits failed to move them.

García et al. found that, in Norway, the amount of required compensation to increase local support for wind farms was contingent on how the benefits were delivered. Private payments needed to be much larger than compensation delivered in the form of local services or infrastructure improvement, in order to generate the same support increases.

Muñoz and Tormos ran a survey in Catalonia where they found that framing a wind or solar project as being done “somewhere in Catalonia” led to more support compared to when it was framed as being in one’s own municipality. Opposition to projects also increased with scale, but it decreased when benefits to the environment or municipal tax reductions were offered.

Doberstein, Hickey and Li found that Canadians in a fast-growing, car-dependent mid-sized city were strongly opposed to new developments on their street, but were less opposed to developments elsewhere in town. They softened their stance and became more supportive when the message noted the public benefits of residential densification, but they were unconvinced by private benefits or by the presentation of expert views on benefits. They were also able to have their views shifted by learning what their neighbors felt about the development.

To summarize the literature on NIMBY/YIMBY messaging experiments:

• People care about the community benefits of new building.

  • They want public, widely-shared benefits.

  • They don’t want private benefits per se.

• People care about feeling involved in the process of development.

• Some people cannot be convinced in the short term by any reasonable message.

• When it seems like proponents of building are personally benefitting, opposition increases dramatically.

Opposition to building is driven by a mixture of disdain for unsightly builds, demands to preserve community and environmental character, pure resent, and so on. These are generally not rationally motivated concerns, which helps to explain why fact-based messaging doesn’t work that well: you can fully convince someone that they were wrong about some particular empirical claim about building’s harms or lack of benefits, but they still won’t budge. I would avoid persuasion techniques generally.

It is likely that enough of the fact-based pushing can win out in the longer term, as always happens with technologies. People resisted the internet, the automobile, the telephone, and even electrification writ large, but familiarity, aging, and progress eventually wear down resistance and absurd myths tend to die—who besides crazies now believes that electrical wires make the communities they’re in sick? But this is a poor assurance to people who want to bring that progress about sooner.

For near-term progress, the strategy is clear: give people what they think want, first of all, even if you don’t give them what they’d get through a real negotiation process. Run roughshod over them with any permitting workarounds possible while giving the appearance that locals have been listened to, in messaging. Ensure they know building is costless for them: pay for electricity, pay for water. Then talk about the implicit and actual bribery that comes in the form of construction jobs, permanent white-collar jobs, and property tax abatements, more funding for schools and parks, whatever.

At every stage in history, progress has been resisted, and in some places and times it’s been halted entirely, resulting eventually in a humiliating, unnecessary national defeats. We now face the same challenge and rising to it is our choice to make.

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

I recently read an article that claimed climate change research has peaked, as measured by the number of papers being written on the subject since 2020 rapidly declining. The paper’s recorded decline is dramatic: the number of peer-reviewed publications labeled “climate change” went from a peak of about 900,000 in 2020 to a near-term nadir below 200,000 in 2026.

Science Is Not The Answer

We Passed Peak “Climate Change” (um) Research

Here is a graph of the yearly number of total papers that have the keyword “climate change” in them…

Read more

2 months ago · 86 likes · 32 comments · William M Briggs

The article explains this dramatic change in terms of “the teats [being] squeezed dry”, the field of climate change research being “on repeat”, totally hollowed out of new territory to look into, data to model, and hypotheses to propose. I don’t believe this explanation, but I could believe some explanation like ‘People have realized that we’re going to beat climate change, so the energy is going elsewhere’ and ‘The decline could have accelerated recently due to papers reclassifying themselves away from being about climate change in order to avoid the ire of the Trump administration.’

However, before we try to explain phenomena, we must determine if there’s anything to explain. As I’ve noted before, abrupt changes are usually fake and we have a duty to determine if what we’re critiquing or supporting is valid to the greatest extent we can.

So, how did I check if this article was correct? Three ways. First and weakest, I looked at the article’s own source: Google Scholar. I checked if there was a generalized service-wide decline that mirrored what the blogpost noted for climate change-related research. As it turns out, there was! Fields with more hits in 2020 had considerably fewer hits in 2025. This is odd, but it’s not explicable by something specific to climate change research changing.1

Second, I looked into another repository, OpenAlex. I tried to replicate the peak the blogpost observed, but, well, take a look:

Third, I replicated this again. After all, we don’t want a game of warring repositories; we don’t want to just play Google Scholar off against OpenAlex, so I also looked at Semantic Scholar, Crossref, OpenAIRE, PubMed, Europe PMC, and DOAJ.2

The question of whether climate change research paper numbers have declined in the COVID era is an empirical one that’s worthy of looking into, and the answer is ‘no’. Accordingly, we have no reason to speculate about why it’s declined, as the truth is that it hasn’t. But we do have reason to repeat once again that we must verify things need to be explained before we explain them; we must avoid that harmful impulse.

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

# Root fields (OpenAlex level-0 concepts)
- Computer science
- Physics
- Medicine
- Biology
- Engineering
- Materials science
- Political science
- Chemistry
- Philosophy
- Geography
- Mathematics
- Psychology
- Art
- Business
- Sociology
- Geology
- Environmental science
- History
- Economics

# Top subfields (level-1 concepts, ranks 1-10 by works count)
- Humanities
- Artificial intelligence
- Optics
- Internal medicine
- Law
- Biochemistry
- Quantum mechanics
- Remote sensing
- Organic chemistry
- Genetics

# Sampled subfields (level-1, every 5th rank from 15)
- Statistics
- Surgery
- Geometry
- Social psychology
- Information retrieval
- Metallurgy
- Endocrinology
- Neuroscience
- Theology
- Computer vision

# Low end (level-4 concepts by rank)
- Nucleofection
- Gestational period
- Diafiltration
- Emperipolesis
- Sclerodactyly
- Ocrelizumab
- Divertor
- Neutral beam injection
- Magnetic energy
- Langmuir probe
- Magnetohydrodynamic drive
- Cyclotron radiation
- Neutron emission
- Ballooning
- Transcriptome
- Superconducting Super Collider
- Cyclotron resonance
- Genomics
- Single-nucleotide polymorphism
- Gene expression profiling
- Repressor
- DNA methylation
- Acceleration voltage
- Magnetic particle inspection
- Clade

# Low end (level-5 concepts by rank)
- TSG101
- Articular cartilage damage
- Proteogenomics
- Protein isoform
- Durvalumab
- Pandemic
- Magnetic amplifier
- Edge-localized mode
- Central core disease
- Genetic association
- Coronavirus
- Genome-wide association study
- SNP
- DNA Transposable Elements
- KEGG
- Good agricultural practice
- Social distance
- Prompt neutron
- RNA-Seq
- Conservation-dependent species
- Neutron probe

# Original battery
- climate change
- the
- cancer
- protein folding
- machine learning
- quantum computing
- gene expression
- randomized controlled trial
- solar cell
- coral reef
- stochastic volatility
- nonlinear Schrodinger

The obvious answer to the headline question is that you would ask it a question in the same way you would ask a person, right? For some queries, I think that’s acceptable—(‘How big is the sun?’, ‘What’s 10+10?’, ‘Find a well-known mathematical conjecture and solve it. Make no mistakes.’)—, but often enough, it’s just not. In my experience, the key issues are obeisance, confidence, ‘nuance’, and tacit knowledge.

I’m going to consider three recent cases that I believe to be illustrative of these issues.

America’s High-Cost Healthcare

In 2014, (henceforth, “RCA”) produced an analysis that decisively debunked a number of popular talking points about American spending, healthcare, and so on, and he popularized Actual Individual Consumption and its importance to the blogosphere and the Twitterati. This blogpost has produced an inordinate amount of ire because left-leaning (= center-left and further) and far-right people interested in politics often hold strong, non-empirical views and corresponding positions on the topic of American healthcare.1

The main thing that RCA documented is that America is not a massive outlier in global healthcare spending once its vast wealth is accounted for. In other blogposts, he expanded on several potentially important points, debunking common ideas about the roles of inequality, prices (see also), administration2, and much more.

What do LLMs like Claude and Grok say when you ask them about RCA’s work and conclusions? If you ask naïvely, you will find that they deliver completely wrong answers. Not only that, but often enough, they’ll explicitly reference RCA and claim to contradict him, while citing evidence that he already dealt with. As an example, using a question from , Claude (on Max effort) appears to do just that.

When asked to evaluate arguments, Claude often begins by conceding that one side makes some points, then going and arguing against their case along some other lines, like faux-steelmanning. That’s typical of modern LLMs and that’s how this started off.

In its concession to RCA, Claude got numbers wrong, understating RCA’s case. For example, it said “AIC per capita runs something like 30-50% above the big Western European economies.” Is this the case? No. AIC was >50% higher than the EU28 economies, >50% higher than the 11 core EU economies, and 55-60% above the “big” economies (assuming these refer to Germany, France, the U.K., Italy, and Spain).

Claude then started to talk about how RCA’s argument was “fragile”, alleging that it was ‘circular’, meaning that high health spending is included in AIC, so its denominator is inflated. Is this true? Yes, but RCA dealt with this obvious objection and showed it was not explanatory. Therefore, Claude should not have brought it up; it was ‘prebunked’. The next point of fragility wasn’t even right: Claude said being “on trend” was sensitive to functional form—not true among valid forms—and the choice of small countries to include—part of the whole point of using AIC, to fix the issues with said small countries’ GDP numbers being off due to tax haven status. The next point of fragility is also not even a point, it’s a restatement of what RCA argued: that the trend line is evidence of common causes of health spending across countries. Claude, however, took issue with RCA’s arguments and said “Sitting on the trend line isn’t evidence of efficiency—it’s compatible with Baumol effects plus waste that scales with income everywhere.” How is restating the thesis a counterargument?

In the next paragraph, Claude made exaggerated claims about the role of administration and billing overhead—things RCA dealt with—, price dispersion across countries, branded drug costs—an indication that Claude knew that drug cost concerns were substantially overstated due to generic drug availability—, and then it cited an incorrect study that RCA already explicitly dealt with. The study—dealt with here—argued that utilization differences don’t explain America’s high health spending, but it did so by using select utilization measures that don’t relate much to health spending or have high U.S. residuals, which are things RCA, once again, explicitly explained. Yet, Claude cited this work when it was already shown to be in error, and then it went on to talk about another thing RCA already explained, before switching topics to talking about lifespans instead of health spending, seemingly just to say RCA was correct on that topic in the middle of a paragraph on spending—something I can only read as ‘attempting to sound like the model is being fair’.

Claude then ends its discussion by making a data-free inference that the U.S. is not normal for its wealth and bringing up three different things America’s alleged “dysfunction” is concentrated in, one of which wasn’t relevant, one of which made no sense, and another of which has some truth to it, but was already dealt with by RCA! Had Claude been seeking a proper answer, it would’ve gathered multiple years of the relevant, publicly available data and tested its proposition, or at the very least, said it was unsure.

Other LLMs reason similarly incoherently and also dodge the facts, the things RCA actually said, and they repeat what’s been published in papers, even when it’s irrelevant or wrong. If you read how the LLMs reason in response to MattY’s question, you’ll notice that they say a lot of things that seem to be relevant, but which really aren’t. As such, they appear to provide a lot of information, but they really just use a lot of words to say very little at all. They do not inform, they misinform.

But notice how the responses are written. We see LLMs that write in a way that, from a human, would be considered confident, even though you can press it on any number of these points, and it will concede. We see LLMs that stick to scholarly consensus, preferring debunked points published in JAMA to the debunkings published on a blog. We see LLMs that repeatedly says RCA makes some points, but that he’s overall wrong when everything is considered.3

Woke Madness

A few days ago, I saw posting his blogpost Does Therapy Culture Explain the Ideological Mental Health Gap? over on Twitter. I noted that his preferred theory was not viable since the measurement of this gap comes with evidence for measurement invariance when comparing liberals and conservatives.

He then asked me why the finding of measurement invariance rules it out, I explained, and he took to asking ChatGPT what it thought. The issue with using ChatGPT as a spot-check here is that it doesn’t have an understanding of the topic. In fact, in its response, it affirmed I was correct, but didn’t realize it!

Think about what the LLM should’ve known to say. There are many options consistent with how measurement invariance is defined and what it means—which, I’ll note, is specialist knowledge—, but my preferred response from an LLM would’ve involved interrogating the premise: how could therapy culture explain a liberal-conservative gap and be consistent with identical measurement parameters for both liberals and conservatives?

The implication of said identical parameters—that is, the state of measurement invariance—is that the causes of differences in the thing being measured must act the same between groups. But imagine that: how could they?

For one, presumably those who go to therapy would conceive of the questions used to measure mental illness in different ways, ways atypical of society-at-large but normal in the world of therapy. For two, is it plausible that this works for a trait like depression given what it’s measured by? Consider the most commonly used depression scale, the PHQ-9. Here’s a model of it:

Now, imagine where the variable “Therapy culture” would slot into that diagram. To retain measurement invariance, it would have to affect the PHQ9 latent variable or at least operate on all of the measured variables identically to how the PHQ9 latent variable appears to. Since depression is almost-certainly not a ‘real’ latent variable, but rather something more like a network, we can safely constrain our theories to be about the latter type of variable: a factor mimic.

Candidate explanations for filling this role are basically unimaginable: how would therapy culture cause a connection between anhedonia and sleep troubles, and also to every other symptom as it needs to? How could it act across a trait continuum where the symptoms can go either way, such as with too much or too little sleeping, or too much or too little eating? Therapy culture sounds quite magical! Add in that it must explain more conditions than depression—and those sometimes come with pathognomonic signs—and “therapy culture” as explanation really stretches the limits of what’s believable. It may have some role somewhere, but it’s no general explanation.

I’ll leave the theoretical speculation about how therapy culture could be consistent with measurement invariance to the people who believe in its role, since I don’t, and it’s their case to make. I’ll instead simply point out what the LLM does wrong here. For one, it doesn’t understand my objection. This is visible when the LLM mistakenly affirms what I’ve said while claiming to contradict it. It’s also visible when the LLM outlines at a high level what conditions have to hold for measurement invariance to be attainable, without elaborating on the complex and, I think, practically impossible theoretical constraints that would have to hold for the theory to make any sense.

If you prompt the LLM to think more deeply about how this theory could work and you keep poking holes, it eventually concedes the point and says it’s unlikely that therapy culture plays the all-important role it’s been ascribed. But it lacks the tacit knowledge, or the instinct to automatically ply tacit knowledge, to make these issues apparent from the outset at the asking of a seemingly simpler question. But this question is not really simple, and it takes asking a lot of questions to grasp that if you’re not already familiar with the ins-and-outs of psychometric methods. This is true of many questions, so if you ask any of them, for a correct answer, you have to be in a position where you likely already know the answer, making the LLM superfluous.

David Sinclair’s Sudden Blindness Epidemic

David Sinclair recently went on the Joe Rogan Experience podcast and lied egregiously about the side effects of GLP-1 receptor agonist drugs like Ozempic and tirzepatide. The lie that I called out was that “NAION… is now twice as more [sic] prevalent because of the GLP-1 drugs.”

Because David mentioned “sudden blindness” writ large in this section of his interview, I looked at all types of sudden/acute vision loss and found that their incidence hadn’t changed between 2016 and 2023 in national-level electronic health record data. We’re not well-powered for NAION specifically because it’s so rare, but the prevalence of that hadn’t even moved in the right direction. Nevertheless, I got a lot of comments like this one: an LLM was asked who was right, and the LLM suggested Sinclair might’ve been right.

The basis of the LLM’s confusion has to do with a simple mix-up. This is their logic:

1. There are cohort studies that suggest GLP-1RA users have a higher rate of NAION;

2. More people are using GLP-1RAs than used to;

3. Therefore, rates of NAION have increased.

Can you tell where the issue lies? Fundamentally, nothing in this chain of logic demonstrates that the rate of NAION has increased. To demonstrate that NAION has increased, this method just won’t do; instead, you’ll have to count the number of NAION cases and show that they increased.

But, you may ask, doesn’t that logic follow? And the answer is ‘no’, because it only follows under the restrictive condition that the identified effect size is causal, correct, and applied to an appropriate population that we can generalize from. If any of these is off, then you could find arbitrarily large effect sizes without any effect even existing.

I showed that the studies Sinclair was referring to were wrong in three ways:

1. The effect sizes have declined in newer, larger, better-controlled studies. The magnitudes he’s citing are thus unlikely to be correct;

2. The populations analyzed in his preferred studies were not representative enough. In fact, better representativeness has come with smaller and smaller effect sizes;

3. I looked at the measured population rate of NAION and a variety of other conditions and saw no increase. In fact, for NAION—which has such a low rate that changes are hard to determine—the direction of the effect wasn’t even right.

If you just ask the LLMs about Sinclair’s statements, they don’t think to bring up any of this. They may say that the base rate of NAION is low and thus the benefits of taking a GLP-1RA outweigh the risks for almost everyone, but they will not generally walk down the chain of logic and go ‘Hold up, something isn’t right. Where’s the doubling evidence?’ Instead, they make the same error as Sinclair and unfortunately many other scholars of similar ilk.4

Can you see the issues here? LLM answers sound right—the machine provides confident prose—, but to get the correct answer, you would need to know to press it in ways that are not obvious to most people. If you do not, then LLMs will state mainstream views and interpretations, default to what’s commonly understood on sites like Reddit (where much of the training data is derived) and in mainstream news sources like the Wall Street Journal or New York Times. They will incorrectly claim that open-and-shut questions require considering multiple angles to understand, and yet they won’t know when to probe deeper on questions because they lack the tacit knowledge to default to theoretical thinking about the things they’re asked about.

To make matters worse, LLMs will provide inconsistent answers depending on who they believe to be asking a question. I’ve seen this first-hand when, in my own sessions, I get something passable, and a friend asking a question similarly gets rubbish (and vice-versa!). In fact, this behavior has been documented systematically:

To answer the question ‘How do you ask an LLM a question?’, well, I’m afraid I don’t quite have an answer. I don’t think anyone does, as it’s too open-ended. For the question ‘Can you find a mathematical conjecture and disprove it?’ there’s likely enough detail there for it to go off on its own and return something substantive. For the question ‘What is the meaning of life?’ you’ll doubtless be disappointed no matter how many follow-up questions you manage to plow into the text box. Good luck.

This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.