An “attribution study” is the name I’ve given to studies that purport to provide estimates of important real-world quantities through extrapolations from estimates in other, uninformative studies. There are a lot of these papers and they often come with conclusions like ‘Gas stoves responsible for 13% of American asthma’, ‘68,000 die each year due to health insurer denials’, ‘DOGE killed a million billion Africans with its program cuts’.

This style of study is not inherently bad. There are ways to do it right, but they require more effort than multiplying what are oftentimes pretty random numbers together. I’ll walk through the example from that tweet, the claim that “68,000 excess deaths per year… can be attributed to people who delay or avoid care because they lack health insurance.”

The source for the 68,000 death claim is Galvani et al. (2020), a paper in The Lancet that does, indeed, claim that “ensuring health-care access for all Americans would save more than 68 000 lives and 1.73 million life-years every year compared with the status quo.” Read that carefully and you’ll notice it’s not quite the same claim that was made on Twitter; to get anywhere, we have to be generous in treating this as commensurable to the tweeted claim about people dying from delayed or avoided care. But, if we grant that the claims are the same, we can move on to the big reason why they’re invalid: the methods. Here’s what the study did:

Galvani et al. took age-specific rates of being uninsured, multiplied them by the size of the population at those ages, and reasoned that, if that group had a 40% lower age-specific mortality rate, it would have had 68,531 fewer deaths. Thus, they conclude, expanding insurance coverage would have saved those roughly 68,000 lives.

The first issue with this is that the gap between the uninsured and insured is not necessarily due to the lack of insurance. It could be due to factors associated with not having insurance, like living on the streets, being schizophrenic, being poor, and so on. The second issue with this is that the gap is not reliably estimated. One of the hints of this comes from the author list of the source study: two of the authors are behind the infamous fake healthcare administration costs graph that you might’ve seen making the rounds on X, Instagram, Facebook, Reddit, etc.

The source study used the NHANES III, which is, coincidentally, a dataset that I’ve used for several of my most recent articles. The authors excluded Medicare, Medicaid, and VA/CHAMPUS recipients1 and adjusted for age2, sex, race, income, education, self- and physician-rated health status, body mass index, leisure exercise, smoking, and regular alcohol use and found that after those controls, the uninsured were 40% more likely to die. But what if we do a more extensive control strategy, like what I did in my articles on seed oils, ultraprocessed foods, and ice cream? The mortality differential collapses.

Moreover, if we use more waves of NHANES data or years in the mortality file, it also collapses and sometimes reverses, even without more extensive controls. These adjusted results are not causally informative per se. But, what they do tell us is that the estimated 40% uninsured age-specific mortality delta is not so certain, and not so easy to attribute to insurance. In fact, we can show this in a more serious way by looking at other ways that the insured and uninsured differ. For example, the uninsured are fatter, shorter, less likely to be White, and their diets differ in a number of pretty significant ways. As another negative control, they’re just as likely to die from accidents—which the medical system is generally not responsible for—as they are in general. We can’t attribute these findings to not having insurance coverage. Even by refusing to dichotomize education, letting drinkers report any number of drinks rather than at most 6, and extending the follow-up, the mortality gap shrinks a lot.3

Other studies have looked at the differences in mortality between the insured and uninsured, and they tend to agree with what I’ve found, that there is a lot of confounding to the relationship between insurance status and everything else. But we don’t even need to keep referring to cohort studies. We do have causally informative work that we can cite to get at this question.

A number of these were discussed during the Hanson debate on healthcare utility over on ’s blog Astral Codex Ten. The gist of the studies where people were freely provided with insurance is that mortality was not meaningfully affected, and what little evidence there was for any effect was marginally significant. That means no deaths to attribute to insurance, but, we have to also note that those studies were underpowered to detect small effects, so those might still be on the table. The study of letters sent out by the IRS reminding people to get insured had more power and found a small, but significant effect, where it would take 1,587 people getting health insurance to save one life in the two years following getting insured, with no effect among children and young adults.

Astral Codex Ten

Highlights From The Comments On Hanson And Health Care

Read more

2 years ago · 76 likes · 169 comments · Scott Alexander

What happens if we attempt to re-estimate the original 68,000 number given everything we now know about it? Since the hazard ratio to use is so close to zero, the number of deaths that we can attribute to this excessively broad definition of ‘deaths from delay or denial of care’ is also nearly zero.

We likely won’t ever have a good estimate of the ‘true’ number of people who die from denial or delay of medical care, except that it is small. And it must be this way given the structure of mortality in the population: the overwhelming majority of deaths happen in old age, when there’s government-provided insurance; via EMTALA is is illegal to deny emergency care; and most other deaths among the young are due to things that are outside of the control of the medical system! Our health system is obviously not set up to deny people to death; there is no serious argument that it is.

But this is neither here nor there. The issue for this article is this sort of study, where dubious numbers are selected to promote some sort of narrative that they can hardly support. A previous instance of this that I dealt with some three years ago had to do with gas stoves. A study came out that purported to show that almost 13% of the asthma in America was due to gas stove exposure, but the estimate was based off of a deeply flawed meta-analysis. I redid the meta-analysis and the effect shrank. I then found that a study that had a larger sample than the entire literature up to that point found that there was no significant effect whatsoever, so the estimate ought to be zero.

Despite the gas stove study’s flaws, it was acclaimed. And, as with the Galvani et al. paper, the authors were clearly on a mission: they work for a company with the goal of getting rid of gas stoves! And they’re not alone. Consider the following studies:

• Galea et al. attributed 245,000 deaths in 2000 to low education, 176,000 to racial segregation, 162,000 to low social support, 133,000 to individual-level poverty, and 119,000 to income inequality, with a further 39,000 to area-level poverty—an unbelievable 37% of all the deaths, which was obviously not true given subsequent changes in these variables.

• Krueger et al. attributed 145,243 deaths in 2010 to people having less than a high school degree (vs. a high school degree), 110,068 deaths to having some college experience rather than graduating, and 554,525 deaths to having anything less than a baccalaureate degree. Mind you, this obviously errant conclusion doesn’t even hold if you use more extensive controls in their dataset (the same one I used in this article).

• Brady, Kohler and Zheng attributed 183,003 deaths in 2019 to current poverty and 295,431 deaths to cumulative poverty. They did this in the Panel Study of Income Dynamics so, as with the previous study, we can be sure it doesn’t hold up to using more extensive controls.

• Avanceña et al. argued that a universal basic income of $12,000 a year would save between 42,000 and 104,000 deaths annually, with a negative income tax saving 19,000 to 67,000.

• Lee et al. argued that physical inactivity caused 9% of premature mortality worldwide, amounting to 5.3 million deaths in 2008. Similarly, Rezende et al. argued that sitting was responsible for 3.8% of all deaths in 2015 and Singh et al. claimed that sugary beverages were responsible for 184,000 deaths worldwide in 2010.

• Makary and Daniel claimed that medical error was the third leading cause of death in America.

• Bor et al. claimed that police killings of unarmed Black American caused the Black population to suffer 55 million poor mental health days.

What these studies do is fundamentally problematic from a scientific perspective. They take estimates of exposure effects that are only in a scant few cases causally identified—implying that we’re sure they represent an actual effect of something on something else rather than a mere association—, and they make these potentially non-causal estimates appear as though they’re causal by attributing some number of deaths, disease cases, or whatever else to them. They contain the usually unstated argument that differences between identifiable groups must be due to the things those groups are defined by, like minimum wage recipients having higher mortality than salaried workers because of their lower wages. These results are mere reflections of their assumptions and they cannot stand.4

End the blight by joining me in not publishing, supporting, or listening to the results of un-rigorous, non-causal attribution studies.5 We deserve something a little better.

Many medical treatments, dietary choices, and physical activities seem to induce “nocebo effects”—harm through expectation, even when expectations aren’t met.

The nocebo is a variant of the placebo effect, but it’s far more real than placebo effects tend to be. Nocebo effects are insidious and pervasive, and they drive a lot of the online discussion surrounding topics like mold, fluoride, MSG, sugar, seed oils, and more. For example, people who self-identify as having non-celiac gluten sensitivity have side effects from the belief that they’ve consumed gluten more often than they have effects that can be ascribed to the actual consumption of gluten.

The impact of the nocebo effect on public health knowledge and practice is shockingly large. To make this clear, I’ll talk about the case of statins: very cheap, highly-effective medications that greatly reduce the risk of cardiovascular disease, extending lifespans by years for people who start them early enough.1

Despite their incredible efficacy, statins get a lot of hate, largely because of misperceptions about their side effects. Patients who run into issues with statins are deemed ‘statin intolerant’ and ideally shifted to alternative treatment options like intensive lifestyle interventions, ezetimibe, PCSK9 or ACL inhibitors, fibrates, or bile acid sequestrants. The portion of patients who actually become intolerant ranges from about 5% in trials to 17% in observational studies. When those patients are asked why they’re quitting statins, the answer for about 60% of them is ‘muscle symptoms’—principally myalgia. As noted in a recent review: “muscle-related side effects, the fear of side effects, and perception of side effects are among the common reasons patients often report for declining to start and for discontinuing statin therapy.”

This might seem like a lot of patients are stopping statins for a legitimate reason. But they’re mostly quitting because of the nocebo effect: they believe statins have side effects that they largely do not. Several experiments speak to this fact.

ASCOT-LLA: This study had two phases, one blinded and randomized, and another that was unblinded and unrandomized. In the RCT phase, participants took either atorvastatin or a placebo and reported on side effects. The critical, muscle-related side effects occurred at a rate of 2.03% per year in the statin group and 2.00% per year in the placebo group—virtually identical rates. But when blinding was removed, participants who were on statins reported an annual rate of muscle-related problems of 1.26% versus 1.00% for those who weren’t on statins. It took knowing about the statins for them to generate muscular problems!2

SAMSON: This study took people who had previously stopped statins because of side effects and cycled them through atorvastatin, placebo, and no-tablet months to see how their side effect reports progressed. When people went on statins or placebo pills, they started reporting more symptoms, and they did so to similar degrees. When they came off either pill, they recovered to similar degrees, at the same rate. It was taking any pill that mattered to patients, not taking statins specifically. Placebo was enough.

StatinWISE: This study had patients who had previously stopped or were considering stopping statins because of muscle symptoms alternate between two-month periods of taking either statins or placebo. Participants rated their muscular symptoms at the end of each treatment period and there was no difference between periods taking statins or placebo. The difference was a minuscule -0.11 points on the 0-10-point visual analogue scale, nonsignificantly favoring statins. An alternative measure is withdrawals from the study, and by that measure, there were no significant differences: 9% of participants dropped during statin periods and 7% dropped during placebo period—a nonsignificant result favoring placebo.

Two-thirds of participants who completed the study intended to stick with treatment.

The nocebo-eliciting trials are clear, as are the results from a review of rechallenge trials in which doctors try to get people back on statins after they come off. That review showed no significant differences in global and myalgia symptom ratings, and 36% of the rechallenged participants assigned to statins developed an intolerance versus 26% on placebo. It is also commonly-stated consensus that over 90% of patients who stopped treatment due to side effects are able to successfully continue on a statin when they’re rechallenged, though much of this is based on observational studies, where that percentage visibly do that, albeit without randomization.

All of this is no surprise when placebo-controlled trials show support for very few of the side effects listed on statin labels, and what they do show is universally meager. A recent review of nineteen trials with 123,940 participants in it included this diagram:

If you can’t tell, there is not a single clinically-meaningful effect size pictured in that ring. Moreover, almost every result is null, and given that they almost all cover a very limited range, they are almost all very precise nulls.3 The largest positive effect—meaning one in which statins lead to side effects—is equivalent to about r = 0.10 or d = 0.19. It’s practically nothing, and, as the authors have noted elsewhere: “any theoretical adverse effects of statins on cardiovascular risk that might arise from [their side effects] (or, indeed, from any other mechanism) are already accounted for in the overall reduction in cardiovascular risk that is seen with statin therapy”.4

I’m not going to keep harping on statins. They are a great example where there’s a real issue, but its scale has been so overstated and overhyped that it’s helped to generate nocebo effects in real-world patient populations. Celiac is like this in the sense that it’s a real and serious issue, and it seems to have given birth, by a feat of logic, to non-celiac gluten sensitivity, which is also more strongly nocebo- than substance-driven.

Examples like these speak to the power of belief, suggestion, or whatever you’d like to call it. But nocebo effects aren’t the only way belief matters in health.

People can also be prompted to think they’ve been exposed to things because that have become salient. For example, after a series of high-profile court cases about talc and ovarian cancer hit the news, people started remembering that they used talc. What an odd thing to suddenly remember, and such convenient timing, too!5

Ultimately, what I’m getting at is that the old adage that patients always lie has a kernel of truth to it. But the lies aren’t always to doctors per se, and they’re not always malicious or conscious. In fact, what I’m really saying is: I think the person that patients are the most likely to lie to is none other than themselves.6

Grocery stores are filled with gluten-free bread, sweets, and pastas that look odd and taste worse. Not only that, there are also plenty of gluten-free replacement suggestions like swapping spaghetti with zucchini noodles, replacing tortilla wraps with lettuce, using cauliflower rice instead of cous cous, or putting a portobello mushroom on your beef patties instead of opting for a traditional hamburger bun.

A lot of this has to do with the rise of non-celiac gluten sensitivity (NCGS): supposed harms from the consumption of gluten among people who do not have the celiac autoimmune disorder that causes the body to bloat, become anemic and tired, and induce diarrhea when gluten is consumed. NCGS sufferers claim to have many of these same issues without having the genetic causes or the inflammation that marks traditional celiac disease.

Despite its impact on grocery stores, restaurants, and the memeplex, seeming to verify that it’s real and the market is responding to something substantial, NCGS has all the hallmarks of a disease that’s not real, at least in its own terms. I’ll clarify what I mean by example:

Do you know the difference between inflammatory bowel disease (IBD) and irritable bowel syndrome (IBS)? They have similar acronyms, and they both appear to be about the gut, but that’s about where the similarities end.

IBD is an umbrella term for autoimmune inflammatory conditions like Crohn’s disease and ulcerative colitis, whereas IBS is a ‘Disorder of Gut-Brain Interaction’, or DGBI. DGBIs don’t appear to come with any functional or structural gut problems, but instead arise from people with irritable minds disturbing their guts: they get stressed, they eat poorly, they get annoyed by minor issues that more resilient people might be able to brush off, and they imagine symptoms that probably aren’t there. It’s usually the case that IBS looks more like a mental disorder than a bona fide bowel condition. It is meaningfully psychosomatic.

IBS and NCGS are overlapping conditions, whereas IBS and celiac disease have much less overlap. NCGS sufferers are also about 70-80% female, whereas celiac disease is much more evenly split by sex (~55-60% female). Celiac disease is largely gastric and presents with a variety of physical symptoms, whereas NCGS commonly presents with symptoms like headache, brain fog, anxiety, confusion, depression, fatigue, and diffuse pain—all of which are marked symptoms of other largely-psychosomatic illnesses like long COVID and fibromyalgia. Moreover, celiac diagnosis tends to be symptom-driven following a clinical work-up due to specific symptoms or an incidental finding during other evaluations, while NCGS diagnosis is usually a self-diagnosis or something suggested by an alternative medicine doctor or dietitian rather than a real doctor. And, while untreated celiac disease is associated with a wide array of nutritional deficiencies, the association between NCGS and nutritional deficiencies is both different and less extreme.

NCGS looks like a DGBI, not a genuine autoimmune disorder, and it is certainly not what it appears to be. NCGS is definitely not about reactivity to or an intolerance of gluten per se. The best way to grok this is to look at the results of the landmark challenge studies highlighted in last year’s Lancet review.

The seminal publication on NCGS is Cooper et al.’s challenge study from all the way back in 1980. In it, Cooper et al. subjected eight adult women without celiac to eating a diet with gluten and after they did, they saw some meager signatures of gastrointestinal response. Later reviews have flagged this as low-quality evidence for NCGS, but it’s worth mentioning because it’s the first real clinical report we have.

After the popularity of the condition had grown, we got another study some thirty years later. This was a double-blind, placebo-controlled (DBPC) trial including 34 non-celiac patients, where 68% of the patients given gluten-rich food reported inadequate NCGS symptom control, compared to 40% of those who were not given gluten. All of the effects, however, were subjective, and there was nothing to show for anti-gliadin antibodies, fecal lactoferrin, celiac antibodies, hsCRP, or measures of intestinal permeability.

Two years later, the same research group did another DBPC trial, this time taking aim at a possible connection between NCGS and IBS. In this trial, they gave their 37 non-celiac participants diets that were low in FODMAPs—the types of carbohydrates that make you feel gassy and bloated—and then placed them on high-gluten, low-gluten, or control diets. With low FODMAPs, participants were a lot better off, and when they were exposed to the higher FODMAP diets, they got worse, but not especially so with gluten.

This is key: people with DGBIs like IBS tend to be sensitive—hypersensitive, in fact. When they eat things that would make any of us bloated, they seem to feel much worse than we do. With the same bloating, the same gut distension, the same gassiness, they report feeling relatively excruciating pain, where normal people would feel practically nothing. A subset of them—one that’s more female and rates higher on several additional mental disorder measures—also frequently report feeling more pain broadly, and these individuals have less complete recovery when they’re put on a low FODMAP diet, because their issue is a broader pain perception one (or at least, that’s how they report it),1 and relieving painful bloating is only a partial fix.

A study the following year showed practically no reaction in a 22-person DBCP trial of non-celiac NCGS sufferers, with the only reaction being psychological (p = 0.01 for one of two possible dietary subgroup comparisons on a depression subscale score for the STPI) rather than physical (cortisol, gastrointestinal symptoms).

The next year, a 35-person DBCP trial assessed NCGS sufferers’ ability to distinguish gluten-containing from gluten-free flour. They found that surreptitiously exposing participants to gluten only elicited symptoms in one-third of NCGS patients, with just 34% even correctly identifying gluten flour and 49% incorrectly identifying a gluten-free flour as having gluten in it. Among those who incorrectly identified the gluten-free flour as containing gluten, they had stronger reactions. In the same year, a 61-person DBCP trial of non-celiac NCGS sufferers came out. In it, participants either took gluten or rice starch capsules, and the result was p-hacked: the authors registered that they would report on 15 intestinal symptoms and 13 extraintestinal symptoms, but they ended up only reporting the ones that showed significant differences, and each such difference had a p-value that was dubious, ranging from 0.019 to 0.047. And, only 3 of 59 subjects who completed the study showed any evidence of gluten sensitivity—consistent with chance!

In 2018, another DBCP trial found FODMAP (specifically the ‘O’, fructans) sensitivity rather than gluten sensitivity. That year, we also got a DBPC in children, with 1,114 gastrointestinally-dysfunctional children enrolled and the 3.3% who showed a correlation between gluten ingestion and symptoms taken off to do a crossover gluten diet challenge. Among those kids who did the challenge, 39% showed worse symptoms with gluten, but the majority showed no evidence of any issue, and the number of those who showed an issue is absolutely small. More pressingly, the p-values for group differences are misreported and a few challenge results that are reported as significant are instead nonsignificant and misreported. For example, in Table 3, results reported as p < 0.005 or <0.001 were instead 0.030, 0.082, 0.498, and 0.027.

Finally, we got a larger, stronger study of 84 NCGS sufferers. This one had a unique factorial design in which participants were sorted based on the expectation of eating or not eating gluten, and then they were also given or not given gluten within different expectancy groups. Expectancy—not actual gluten content—dominated. The participants who expected gluten and got it or did not get it had comparable responses, and those who didn’t expect it weren’t affected. Or, in other words, there was a nocebo effect: the anticipation of gluten was the source of the problem. Gluten was innocent! In other trials (and, indeed, in reviews), nocebo effects crushingly outnumber anything that could look like actual gluten sensitivity (but which might in truth be FODMAPs or some other stimulus).

The picture we get from challenge studies is one in which NCGS looks like little more than IBS, complete with the same demographics, similar treatments, and anticipation effects that show it’s substantially psychological in nature. Meanwhile, there’s no meaningful or consistent evidence for gluten actually being involved. Despite extensive searching, there are no NCGS biomarkers, nor any identifiable severe acute reactions, and there’s no evidence of physical effects on the gut, immune reactions, or anything you might expect if people are, as they think, sensitive to gluten.

There is no reason to believe that a non-celiac sensitivity to gluten actually exists.2

How Do We Treat Mental Illness Disguised as Food Issues?

But there are still NCGS sufferers, people who wrongly think that they’re affected by gluten when they’re suffering due to something else. These are people who are more likely to have IBS or some other functional gastrointestinal/mental disorder characterized by hypersensitivity and typically alleviated through dietary restrictions intended to cut down on or distract them from specific, generally tolerable conditions like bloating and gas. They usually have mental comorbidities and they frequently claim to have a bundle of other conditions that are not what they seem.

The easy answer on what we should do for these people is to restrict their diets in a way that alleviates their symptoms by cutting out FODMAPs, fructans, or whatever else contributes something towards states they’re hypersensitive to. To the extent—which is major—that their symptoms are ultimately due to some sort of mental illness, we’ll need to keep looking at that angle. It would be unethical to only treat what might be a delusion built from a mental illness when it could be possible to treat the underlying mental illness that creates false food sensitivities.3

$100 million to find a box2 within one year, with 52 updates about your distance to the box along the way. If you don’t get the box, you die. If you’re offered these terms, take the bet because you’re basically being promised $100 million!

This type of problem has a name: multilateration. This is a well-studied geometry concept and it’s fundamental to modern Global Positioning Systems (GPSs). With it, you can pinpoint the location of practically anyone or anything, to the degree of precision you’re allowed by going off of just a few separate distance readings.

Practically, multilateration means that you are going to make a lot of money on this bet, and you’ll be able to do it quickly. As an applied example, consider one of my hobbies: geocaching. In geocaching, you are tasked with finding an object like a container given only its GPS coordinates, down to three decimal places in Degrees Decimal Minutes (DD° MM.mmm′) format or five decimal places using Decimal Degrees (DD.ddddd°). If you’re unfamiliar, the latter is standard latitude and longitude, and when you get down to that degree of precision, that’s about where GPSs cut out, and it allows you to be within about six feet of the container of interest.

Six feet is close enough that you can’t miss the container. But during geocaching, people are right at the object when they write down the coordinates. How long would it take you to find an arbitrary object at that level of precision given only a set of distances? The answer is surprisingly little time! If you’re given exact distances, to an arbitrary level of precision, then it should take you at most two iterations after the initial distance to get to five decimals of precision in Decimal Degrees. Then, it’s over.

I’m calling this level of precision the “Direct find zone”, because you can’t miss it:

If you were given geocache-level precision on the distance estimates, that would mean giving you three decimal places, so a distance like 100.123 miles or 7.651 miles. When your distances are that precise, it will take you up to about five weeks to get very clear on the location of the cache if the distances are provided in miles, and three weeks if they’re provided in kilometers.3

But let’s be even more fair. Let’s assume that you’re not given arbitrarily precise distances, but are instead given precision to a single decimal place, so you might be told your $100 million check is in a box 100.1 miles away, or that it’s 39.3 miles from your current position. In that scenario, you’re still in a good position: it should take you at most about 18 weeks with miles or 16 with kilometers, then you’ll find your box!

Now, let’s see an applied example of this. I’m going to show you four randomly selected scenarios in what should be the hardest country to get around in distance-wise. In Russia, if you can get around and you get geocache-tier distance estimates, you’ll find the box to four decimal places quickly. Nowhere is a safe hiding spot if you’re smart about where you go to receive the next distance estimate.4

So, what’s my advice? If you’re getting reasonably precise distances, take the bet!

Now imagine if we didn’t live with the convenience of modern air travel. Imagine instead that we lived in the early 20th century and you took this bet in a big country like America, Canada, China, or Russia. Well, then, you might be boned.

I don’t like the term ‘ultraprocessed food’. It’s vague, and the vagueness might even be intentional. The inventors of the most popular food processing classification scheme—NOVA—have failed to provide a detailed breakdown of which foods count as processed, unprocessed, or ultraprocessed. Despite that, it’s currently all the rage to talk about how bad ultraprocessed food is for health.

Take a peanut in a shell. That’s an unprocessed food (Group 1), but when you crack it open, that’s a minimally-processed food (Group 2). If you want to get dangerous, crush it up into a paste, add some salt, and now you’ve got peanut butter, a processed food (Group 3). If you add some sugar on top and put in a dab of vegetable oil to stop the oil from separating, you now have ultraprocessed peanut butter! (Group 4)

By today’s received wisdom, you went from something benign and safe to something whose consumption is dangerous and liable to promote all manner of health ills, from heart attacks to diabetes to cancer. But in that example, there’s little nutritional change going from Group 1 to Group 4, and that example isn’t particularly unusual. Per NUPENS—the creators of NOVA—if you slice a potato and bake it, it’s unprocessed, just like a peanut in a shell. But, if you take that potato and fry it, it jumps right into Group 4 and becomes an ultraprocessed food. Shake it off and dry it out and there’s no nutritional difference for your body to detect, so who cares that it jumped groups?

If you add a teaspoon of honey to plain yogurt at a factory, it goes straight from Group 1 to Group 4. If a factory grinds roasted peanuts into peanut butter, they go from Group 1 to either Group 3 or 4—it’s not completely clear. Bread baked at home slots into Group 3, but an identical recipe baked in a factory with a single preservative goes in Group 4.

When you see examples like fortified breakfast cereal (Group 4) being classified worse than white rice (Group 1) even though it’s substantially healthier, it can make the system look useless. But that’s how it is:

• Fruit is minimally processed and fruit in sugar syrup is processed;

• Applesauce with added sweetener can be minimally processed or just processed, but if you sweeten it with high-fructose corn syrup, it’s ultraprocessed;

• Beer, cider and wine are merely processed whereas distilled spirits are ultraprocessed;

• Margarine and spreads are ultraprocessed while butter is a culinary ingredient;

• Powdered milk is objectively processed but it’s listed as minimally processed, while almond milk just requires soaking, grinding, filtering, adding water, homogenizing, and pasteurizing—steps that have no nutritional impact whatsoever—and as a result, it’s listed as either processed or ultraprocessed.1 Compare this to normal cow’s milk, which retains minimally processed status after homogenization and pasteurization.

After you’ve familiarized yourself with NOVA, it becomes apparent that the classification scheme is capturing something about ‘naturalness’ and industrial context rather than nutrition. All food processing classification methods are attempts to provide scientific-sounding language that makes industrial-scale food production sound dubious. They fill a niche for health and wellness influencers who want to moralize about food and nutrition but know that if they just said ‘added sugar is bad’, their complaints would be obvious and their disdain would be morally unsatisfying.2

But moralizing catches on. There’s demand for ways to praise and celebrate the aesthetics of basic cooking and to attack complex food preparation, just as there’s demand for ways to attack modern lifestyles in general. But demand doesn’t substitute for nutrition guidance: good guidance still needs nutritional facts, not expressions of anxiety about modernity.

Empirically-Speaking

I used the NHANES dataset to look into the question of what ultraprocessed food intake was even correlated with. I checked basic correlations, moved up to extensive matching exercises, and took the analysis plausibly causal by checking if ultraprocessed food consumption mattered within households. I found that the picture is unclear. For starters, read this:

With basic ordinary least squares (OLS) regressions, ultraprocessed food intake is correlated with higher white blood cell count, lower HDL, lower blood pressure—wait what? These correlations aren’t even consistently bad, they’re all over the place. The next outcome is lower HbA1c, which is the opposite of what it should do if it’s harmful. Then, if we switch to the plausibly-causal household-level analysis, only three outcomes remain significant, but they’re at least directionally consistent: lower HDL (bad), lower GGT (good, marginally significant), and lower uric acid (good).

This is a bizarre set of relationships that only gets worse when you look at dose responses. For some reason, it looks like the effects are nonlinear. For HDL, for example, going from the bottom quintile of ultraprocessed food consumption through the next two quintiles seems to not matter to any meaningful degree. It’s only when you get into the top two quintiles that things get bad.

If we change up our dose-response analysis and look at all-cause mortality—the hardest of hard outcomes—then the results become even less clear. It actually looks like a little ultraprocessed food consumption is related to more death, and a lot of consumption is related to no elevation in death risk whatsoever!

What I suspect is that food processing is a noisy concept, so it’s bound to produce noisy results. But this concern should be moderated because ultraprocessed food consumption is consistently related to a few things—things it can’t possibly affect.

There are stronger and more monotonic relationships between ultraprocessed food consumption and obvious confounders like the probability of being White or male, and even the age of the study participant. If we benchmark our statistical findings based on these impossible ones, then there’s nothing left for ultraprocessed food harms: nothing bad is very strongly related to it.

What we end up finding is that ultraprocessed food consumption is not a good variable to correlate with anything else because it is an almost mirror-image of whole food intake (r = -0.95).3 Getting your calories from whole foods has been standard dietary advice since at least 1980 when the Dietary Guidelines for Americans began recommending it. When government and professional organization recommend something, the people who follow those recommendations tend to be better-off and healthier, regardless of diet, making it so that in the future, the associations between what those guidelines recommend and all manner of good outcomes are confounded.

In other words, because ‘we know’ the inverse of ultraprocessed food—whole food—is good, we’ll tend to find that ultraprocessed food is bad or neutral, even if it is, in fact, good. And, because it’s such a poorly defined concept, we’ll also find that it produces plenty of noise. Accordingly, most epidemiological research on ultraprocessed foods is liable to be worthless—even more so than nutrition research generally.

The truth of all of this also makes for an interesting conclusion: it’s incredible that ultraprocessed food can look so neutral given how bad it ought to look. It might not be so bad after all!

Replication Files, a Preprint, All the Evidence, and Q&A

Read more

There’s a widespread belief online that, around the creation of modern welfare states, wealthier people went from being fatter than poor people to being skinnier. There has been inversion, but it doesn’t seem timed with the advent of welfare states. More likely, it’s tied to national development and when countries become sufficiently rich, the poor become fatter even without welfare states. There’s a clear case for this belief.

First, as countries get richer, they tend to get fatter.1

Second, we can see that, in the countries with good data, the poor become fatter than the rich before welfare states in some cases, and before welfare states become significant in some of the rest.

Third, in the United States, we have excellent data on this based on nationally-representative cohorts from 1961 onwards. In that time period, the low-income versus high-income obesity delta has been fairly constant, contrary to the prediction one might make if continuous exposure to welfare states drove poor people to become fat.2

Before the era of representative cohort studies, there are various data points like estimates at the time of the draft for WWII, or estimates from Union Army veterans decades after the Civil War, and if we want to go even less population-representative, we can look at a comparison of legislators from New York and prisoners, and in that whole period, we see that the high-income are fatter than the low-income.3

In modern America, what does this relationship even look like? It’s actually not quite ‘poor people are fat.’ In fact, this shows up overall and it has since the Midtown Manhattan study in 1954, but it only does so consistently in women. It did in men for a few decades, but nowadays, poor and rich men have similar obesity rates!

One question is whether this is just due to a race confound, where Blacks and Hispanics tend towards both higher obesity rates and lower being poorer. If you read the above chart carefully, you’ll realize this can’t be the case in general, and could only be the case within a given sex, and it is! Take a look:

Among men, middle-class Whites are fattest, and both poor and rich are similarly fat. Among Blacks, the gradient is strongly reversed from expectations, and among Hispanics, it’s more moderately reversed. Among women, however, being skinny is status-associated, and this relationship is strong within each racial group, so the overall result where poor people seem to be fat is basically just driven by women.

I’ve gotten this question a lot: How can this be? How can countries get fatter with growing national wealth, when wealthier people tend to be skinnier?

There’s a simple answer: Individual and ecological correlations have different causes!

At the level of countries, national wealth enables more people to get fat if they want to. Your wealth is more determined by national wealth than by anything you can do on your own. At the individual level, people who have good personal habits tend to go further and earn more in life, and overlapping habits help people to avoid undesirable obesity, which is also something that makes it harder to get rich in its own right.4

The studies I referenced or consulted were Bodenhorn (2010), Carson (2009) and (2013), Costa (1998)5, Linares and Su (2005), Moore, Stunkard and Srole (1962), Goldblatt, Moore and Stunkard (1965), the NHES I (1960-62), the NHANES I (1971-75), II (1976-80), III (1988-94), and Continuous (1999-2023), CenSoc WWII Army Enlistment v1.1, Society of Actuaries (1959) and (1980) for the Build and Blood Pressure Study (1959) and Build Study (1979), Actuarial Society of America Medico-Actuarial Mortality Investigation (1912), Staub et al. (2010), Schoch, Staub and Pfister (2012), Staub et al. (2016), Quanjer and Thompson (2021), Kark and Rasmussen (2005), Lissner et al. (2019), Marmot et al. (1991), Sobal and Stunkard (1989), McLaren (2007), Jaacks et al. (2019), Singh et al. (2011), Lehmann et al. (2016), de Saint Pol (2009), Sánchez-García et al. (2021), Bann et al. (2018), Perkins et al. (2018), Templin et al. (2019), Kuehl and Vuegelers (2008), Staub et al. (2011), Komlos and Brabec (2010), and ICPSR 2877.

While you’re here, go check out my article on whether GLP-1s can end the obesity epidemic!

1

Relative to European countries and the United States, the slope of this relationship tends to be a little shallower for Asian countries and a little steeper for Hispanic and African countries. You can sort-of see that here:

These charts used GDP, which overstates the development of tax havens. It would be preferable to use Actual Individual Consumption (AIC) to get around this issue, but we don’t have good continuous measurements of AIC, so we’ll have to settle for the Gross National Income (GNI). Using GNI, we get this replication of the first chart:

And the GNI version of the second chart looks like this:

As a final note, the median figure in the first plot is not an accurate depiction of international tendencies because it’s differentially reweighted along the GDP continuum. A similar issue applies to the regional aggregates in the second chart, but it’s less of an issue there because those countries tend to be more similar substantively and in terms of their ranges on both variables.

April 4, 2026 Update: Several people have asked me if this relationship holds within the United States between different racial groups. It likely does. With GNI, the plot looks like this:

With GDP, the shape for the U.S. data points is basically unchanged, except the graph is more compressed due to high GDP/lower GNI outliers, and the shape of the Median line differs, as above. So, for visibility, just take the GNI plot. Now, if you look at that plot, you’ll notice that the slopes do not appear to differ, but the intercepts do. This is even clearer if we go back as far as nationally-representative sampling allows, with the NHES I dataset:

Unfortunately, the large-scale, nationally-representative datasets started in 1961, meaning that they happened after substantial development. As I recorded on Twitter/X, the slope is sharpest in early development and it appears to considerably flatten out at higher levels.

The data from 1961 is largely after the expected point where the slopes should differ the most. And think about this: in the distant past, when resources were much more limited, the world was much poorer, life was more physically arduous, etc., and it was difficult for most people to be fat, different races’ BMIs should have been more similar—and indeed, the obesity epidemic has involved increasing variance.

To get at whether the slopes really differ within the U.S., we need older data. We do have some in sources like this, where the Black-White BMI gap in the 19th century held stable for a long time and was very small, but that data mostly concerns males, and the modern Black-White BMI difference mostly concerns females. Luckily, we do have female data!

The problem with the female data from the 19th century is that it’s a bit selective: it concerns prisoners. With male data, we have soldiers, laborers, photographs, and prisoners, and the estimates all converge on the conclusion that there was little Black-White BMI gap in the past, and with female data, we have photographs, prisoners, and not much else, but, reassuringly, these too converge on the idea that BMIs were similar in the past. More importantly, in the modern day, female prisons tend to recapitulate the Black-White BMI gap seen in the general population. In Nowotny, Rogers and Boardman’s analysis of the 2004 Survey of Inmates in State and Federal Correctional Facilities compared to the NHANES data we’ve been using, they found the following Black and White female obesity rates:

• Non-incarcerated: 26.5% for White, 45% for Black (+18.5pp; +69.8%)

• Incarcerated: 22.9% for White, 40.9% for Black (+18pp; +78.6%)

• Difference between contrasts: -0.5 (-6.6 to 5.6, p = 0.87; 0.97, 0.70 to 1.37 in relative terms)

So, with some assumptions about the relative constancy of representativeness in prisons, we can get to the conclusion that the slopes did vary predictable in the U.S., in a way that explains the vast majority of the modern intercept difference.

Unfortunately, we can’t repeat this with other groups because the data just doesn’t exist. But it remains a priori likely that there were major slopes differences within the U.S. for each group, which we can see in said groups having different intercepts today, and the past being a place where the range of BMIs was relatively constrained within and between races.

Maybe you’ve seen the headlines: “Harvard can’t figure out why ice cream eaters are healthier”; “The best nutritionists can’t explain it: eating ice cream helps your heart!”; “Studies show a mysterious health benefit to ice cream. Scientists don’t want to talk about it.” If you haven’t, consider yourself blessed, because you were saved from the experience of reading something very obviously wrong.

The idea that ice cream—a food item we can all agree is sugary, fatty, and delicious—is healthy for your heart is principally derived from an offhanded comment in a dissertation. Quote:

In our study [of the HPFS, NHS, and NHSII datasets], ice-cream intake was inversely associated with CVD risk among diabetes patients. This result was unexpected given its considerable saturated fat content (6.7 g per 100g) and sugar (21.2g of per 100g) of which, the majority is added sugar (>60%) and the rest is comprised of naturally occurring lactose. Nonetheless, in a previous study conducted among HPFS, the NHS, and the NHSII participants free of disease at baseline, ice-cream intake was also associated with a lower T2D risk. This result was in agreement with the those from two additional cohort studies, and in a meta-analysis it was estimated that consuming 10g of ice-cream per day vs. 0 g was associated with a 19% lower T2D risk. On the other hand, Ice-cream [sic] intake was not associated with risk of total stroke or stroke subtypes.

What are the results? Ice cream was marginally associated with cardiovascular disease (OR = 0.82, p = 0.047), whereas it was nonsignificantly related to coronary heart disease (p = 0.067) and to stroke (p = 0.37) risk.

These are not very impressive results; they’re also not even consistent within the paper. For example, at baseline the study found no relationship between ice cream intake and cardiovascular disease and the relationship only became marginally significant after adding a bundle of controls, including for total dairy product intake. It also wasn’t related at all to cardiovascular disease, coronary heart disease, or stroke risk among diabetics; it wasn’t related to cardiovascular disease among people who had self-reported HbA1c levels; it was related to 33% lower risk among those who reported insulin or other diabetes medication use; the main effect on cardiovascular disease risk became nonsignificant when an 8-year lag was added; and, among ice cream’s correlates including dairy (r = 0.17), low-fat dairy (-0.06), cheese (0.11), cream (0.10), fermented dairy (0.07), it was only modestly correlated with high-fat dairy intake (0.46), not correlated with sherbet consumption (-0.02), or consumption of skim and whole milk (0.11 and 0.001), with a modest correlation with yogurt consumption (0.32).

The pattern of results in this paper is seemingly not robust despite The Atlantic’s portrayal, and, frankly, it doesn’t seem that meaningful given that the authors’ analysis didn’t go very deep: they didn’t use many controls or a sophisticated control strategy; they didn’t attempt to deal with measurement error; and they didn’t use any designs to figure out if the association was driven by any sort of confounding. It was standard nutritional epidemiology, with all of its defects.

As summarized by John Ioannidis about a decade ago: nutritional epidemiology’s “effects” typically represent the impacts of the cumulative biases present in the research rather than the impacts of eating particular foods. What really matters is the crud factor.

Empirical Reanalysis

To look into the ice cream → health hypothesis, I opened up the complete NHANES data and went through progressively more sophisticated models, from basic regressions through doubly-robust estimation, all the way to using family fixed-effects. I wanted to see if the ice cream associations held up, and if they also came with absurd consequences. To make a long paper short, (1) they don’t, and (2) it did.

There are a lot of analyses in the paper, but here I’ll focus on just a few.

Firstly, in Panel A, you’ll notice one set of analyses. The OLS results are the standard cross-sectional correlations, where ice cream seems to relate to things like lower inflammation, higher HDL, lower blood pressure, and so on. But after adding just a few covariates to account for basic confounding, these effects are generally reduced and in the cases of blood pressure, HbA1c, and eGFR, the signs of the effects flip.

When we use very extensive matching on more than 100 different variables, the only thing that survives is that ice cream is associated with being slightly fatter. When we switch over to a plausibly causal analysis—comparing people in the same home—we get a similar picture, where the person who eats more ice cream is a bit fatter, but they’re also a bit less likely to be depressed. But in any case, all of these effects are tiny: certainly too small to matter, but significant because of my large sample size.

The ‘Ice Cream Good’ Narrative

The idea that ice cream is a health food and nutrition researchers can’t explain that fact is contrary to evidence. Ice cream doesn’t look particularly good for you if you examine the association with even a little rigor.

This fact runs counter to theorists like Ray Peat and to the public’s demand for counterintuitive findings. And, if the narrative that’s sprung up around this idea is true—that researchers at elite institutions couldn’t explain this finding—then it indicts a lot of academia, because ‘Ice Cream Good’ is obviously false and it was never on strong ground to begin with.

Replication Files, a Preprint, All the Evidence, and Q&A

Read more

Given all of that, why are people worried about seed oils? Not for any good reasons. Most of the anti-seed oil hype has to do with feelings that they’re unnatural, that they’re prepared in bad or disgusting ways, and so on—not on the basis of any meaningful, clinically acceptable evidence. The best arguments are based on mechanistic reasoning, but mechanistic reasoning sets up a game of ‘he said, she said’ because proponents of replacing saturated fat with mono- and polyunsaturated fats, seed oils, whatever, also have their own technical, mechanistic arguments and the bulk of the causal evidence.

The usual outcome when the trials and genetic evidence are pointed to is that opponents of seed oils will make some sort of excuse. They seem fully capable of always finding a way to say that seed oil intake was mismeasured or some obscure control variable was needed and not present. These excuses reek; I have never seen one that’s principled or which seed oil opponents will reliably stick to. In one moment, a study will be deficient because it used the wrong measure, and in the next, when that measure seems to correlate with something bad, they’ll accept it once more.

Let’s establish a baseline. Everyone says ‘correlation isn’t causation’, so let’s just check what the correlations between seed oil intake and various health variables are.

For this analysis, we’ll be using the CDC’s nationally-representative National Health and Nutrition Examination Survey (NHANES) dataset. For all analyses, we’ll be controlling for age with natural cubic splines with four degrees of freedom, survey weights,1 normalized weights for logistic regression,2 and robust standard errors for the weighted least squares regressions.3

To measure seed oil intake, I will be using dietary and plasma linoleic acid and, to supplement, I’ll also be using dietary and plasma n-6:n-3 ratios. All of these has been used and apparently been suitable for seed oil bashing by opponents of seed oils. Thus, they must be valid generally, and the only objection will have to do with the measurements in the NHANES. But, the measurements in the NHANES have also been used to note increasing seed oil intake and to attack seed oils, so again, opponents will have to denounce their evidence if they want to attack these results.

Seed oils aren’t even correlated with bad outcomes. Nor is canola oil intake or the n-6:n-3 ratio. But saturated fat is, in case you were wondering. Here’s a revealing big-picture graph: people who got the largest share of their diets from linoleic acid have the lowest risk of dying over just over 30 years of follow-up.

Cross-sectionally, linoleic acid was associated with lower inflammation, cholesterol, and triglycerides, with more worrying positive associations with HbA1c, which are the same or worse for saturated fat.

The bad associations for linoleic acid are driven by dietary confounding—more linoleic acid just means eating more. This is easily corrected, just switch to plasma measures of linoleic acid. With those, the bad associations fall away, the good associations remain, and saturated fat looks worse. Adding in propensity score-matching, most associations diminish; going further with doubly-robust estimation, they decline even more, but some signals remain. Here’s the summary:

After all’s said and done, in the matching analyses, linoleic acid is associated with lower inflammation, higher HDL, lower HbA1c, and not a lot else. And if you’re not satisfied by that, well, there are several supplementary analyses I ran, which include just looking at reported seed oil intake effects directly. For those, there’s not a lot there either!

I also went further and tested the mechanistic claims made by seed oil critics. I found that linoleic acid was not associated with arachidonic acid; it didn’t relate to higher atherogenic particle risk; it didn’t seem related to thyroid suppression or autoimmunity; there was no evidence of vitamin E depletion or oxidative damage; even cooking oils specifically didn’t seem to matter; and there was bupkes for clotting.

Consistent with the causal evidence and the best observational evidence from humans—including, for example, a massive analysis of the Nurses’ Health Study/II/Health Professionals Follow-up Study, the latest meta-analysis of prospective cohort studies, the National Institutes of Health AARP Diet and Health Study, and an analysis of the U.K. Biobank—seed oils not only don’t seem to be harmful, they appear to be healthful. This analysis is not causally informative, but there’s not even any confounded evidence for harm here. The cliché is that correlation isn’t causation, but seed oils don’t even have correlation-level evidence that they’re bad.

I end with a quote:

Seed oils people do what people often do when attempting to build up a losing case. They rely on a broad panoply of weak, circumstantial, points—mechanistic research in rodents, bs studies—along with plenty of winks and nudges, to give off a strong impression that their argument is correct beyond a doubt. It’s then left to others to show that each one of their points is specious, one by one, and that what they’ve built up is, in reality, a false impression based on a lot of garbage/irrelevant studies and unjustified assumptions. Even if some of the peculiarities they manage to find can’t fully be ruled out, the case overall is far less impressive than [they’ve] made it out to seem.

Replication Files, a Preprint, Sensitivity Analyses, and Q&A

Read more

I just saw this post from Jonathan Haidt claiming that a ‘powerful’ study showed that adolescents who increased social media use had lower cognitive performance compared to adolescents who didn’t increase their social media use.

Naturally, I opened up the paper, read it, and noticed that the model was not a causally informative one. This struck me as odd because the authors definitely had access to the ability to do within-person analyses—to use each person as their own control—to obtain a causally informative result, and they could’ve also used siblings as controls for one another. They could’ve even done both for thoroughness!

Thankfully, I have colleagues with access to this data. So, I drafted some code and pinged a model specification over to them to retest this result immediately. Now, I’ve produced those results for you. Before I show you that, let’s compare what the authors did to what I was able to do within a few minutes.

Nagata et al.’s Model: They used group-based trajectory modeling, where they classified 7,500 participants in the Adolescent Brain Cognitive Development (ABCD) cohort into three latent social media use trajectories across baseline, Year 1, and Year 2 (no/very low, low increasing, high increasing). They then regressed Year 2 cognitive scores on trajectory group dummies, controlling for baseline cognition, age, sex, race/ethnicity, household income, parent education, ADHD and depressive symptoms, non-social media screen time, and site.

Their estimand is the average adjusted difference in Year 2 cognition between trajectory groups—a between-person quantity that is conditional on observables. They can only identify a causal effect with their analysis if the covariate set captures all relevant confounders. Any unmeasured variable that correlates with both social media use and cognition—genetics, parenting quality, household cognitive stimulation, neighborhood and school quality, etc.— will bias the estimate.

My Models:

1. Family fixed-effects: These absorb all between-family variation by demeaning within families. The estimand is the within-family association: among siblings in the same household, does the one who uses more social media perform worse? This controls nonparametrically for everything shared within families—parental genetics, socioeconomic status, parenting style, neighborhood, school district, etc.—whether measured or unmeasured. Identification requires only that siblings’ social media use differs for reasons unrelated to their sibling-specific cognitive endowments.1

2. Person fixed-effects: These absorb all stable individual characteristics by demeaning within person across waves. The estimand is the within-person association: does an individual’s own increase in social media use predict their own cognitive change? This controls for everything stable about the person—their genetics, their personality, their baseline ability, their family background, etc. Identification requires that the timing of social media uptake is unrelated to time-varying individual shocks to cognitive endowments.

3. Panel family fixed-effects: These involve stacking both waves and demeaning within families across all person-wave observations. This is the most efficient within-family estimator because each sibling contributes two observations, increasing the variation available for identification and reducing standard errors by 9-20% relative to the cross-sectional family fixed-effects result.

4. I also ran negative control analyses—showing that their specification produces significant effects on outcomes that cannot possibly be affected—, I computed the minimum detectable effects across methods, I checked for nonlinearities, and I tested covariate sensitivity.

First thing’s first, I replicated their results. I was able to do that. They are not frauds.

Next, I extended their results with a within-family analysis and a within-person analysis which knocked out their results and showed that they were not robust.

I also showed that the difference between my results and their results was not due to power, because I had more than enough power to detect smaller effects than theirs.

I then looked at whether their methods would suggest that social media use causes impossible outcomes: racial admixture, polygenic scores, and cognitive ability measures from earlier in time; and implausible outcomes: brain volumes. They sometimes could, but not within families!2

And to top all this off, I checked for nonlinearities in the forms of quadratic terms and categorical bins for 0, >0-0.5 hours, 0.5-1 hours, and >1 hours of social media use. Nothing was significant save for one outcome that showed a non-monotonic pattern that was inconsistent with a dose-response relationship.

And how long did it take me to find that their results didn’t hold up? Minutes.

It was trivially easy to specify and run these models and to output the results. It took practically no additional effort to go from the uninformative models used by Nagata et al. to using an actually informative model. Metallic laws, anyone?

Social Science is Always Like This

There are a lot of things that are deeply wrong with social science but, I’ll argue, none is so bad as the abject laziness with which researchers approach every subject. It took me mere minutes to correct a result that looked, to me, to be patently absurd. And let’s be frank: it is absurd. Social media does not make people less intelligent and anyone telling you it does is lying.

When you measure well, concussions don’t even lower intelligence and years of education don’t raise it. With those facts in mind, it should seem patently ridiculous to think that selecting into another hour a day of Twitter is going to make kids dull.

Nagata et al. didn’t get the right result because they’re at least one of the following: ignorant, lazy, biased, or publication farmers. They might not have known that they could test their results—ignorance. They might not have wanted to fit proper, informative models even though they knew they could—sloth. They might have wanted to make social media look bad and not cared that people could see they didn’t run appropriate models—bias. Or they might be looking to overturn their own results in a future publication, with this one intended to pad their publication and citation numbers—farming.3

Whatever their reason, Nagata et al. aren’t atypical (and I don’t want to insult them; I think calling them ignorant or lazy is descriptive and correct). Consider the issue of measurement invariance. This is a critical thing in fields where objective measurements like height and weight aren’t being used and the measurements are instead of hypothetical quantities like group perceptions, neuroticism, intelligence, or what-have-you. To compare groups or individuals across times, you have to establish that those measurements are comparable over time, and despite this possibility being routinely available to researchers, almost no one tests it, and when they do test it, they tend to test it incorrectly. Yet, if you can’t compare measurements over time or between populations, any differences might be due to that incomparability rather than due to variation in the target trait. So, why don’t researchers assess measurement invariance?

Well, they’re lazy. They’re so lazy that they rarely ever do the required robustness and sensitivity analyses to assure that their results are as they are, or to let them triangulate in on something causally informative.

The field of economics has made a lot of strides in this regard. The typical economics paper in a modern, high-tier journal is littered with tables full of robustness tests4, and these allow economists say that their results hold up. In fact, they’ve gotten so good at this that I like to cite something called Henderson’s Law: “When you read an econometrics study done after 2005, the probability that the researcher has failed to take account of an objection that a non-economist will think of is close to zero.”5

But we’re not yet at the point where the rest of the social sciences—including genetics and medicine—can confidently say the same. In fact, it’s more typical for there to be glaring errors that even laymen could spot.

I don’t know how to fix this. I imagine any fix has to be internal to a field. In economics, this was self-led based on an outcry to “Take the Con out of Econometrics”, but it was also possible because economists tend to be smart. The people in other fields tend to be considerably less intelligent and, especially, less capable at or interested in doing mathematics-heavy work, even if they’ve got plenty in terms of raw smarts. If I had to place a wager, I’d say the average sociologist, social scientist, or even medical doctor cannot write even simple proofs.6 What hope do they have to notice something like that a method such as Non-Linear Mendelian Randomization is just not workable?

Happy Note

In 2019, I gave a conference presentation that documented that methodological innovations precede theoretical improvements rather than vice-versa in the domains of differential psychology and analytical chemistry. I think those conclusions held up well, and they present the possibility that social science doesn’t have to be this way.

The fix I have in mind is to have methodologists steward their fields. The people who have a deep understanding of and desire to improve on and implement different methods are the real heroes of science, and to the extent they’re empowered, they can force compliance. Consider the issue of ‘small sample sizes’. Once people started pointing out that many samples used in the social sciences were too small to answer their questions after Ioannidis’ Why Most Published Research Findings Are False and its’ knock-on work, it became a rallying notion. Suddenly, a study could be dismissed out of hand based on a sample being too small, and lots of bad work became unpublishable!

This was by no means perfect; the complaint is often taken too far because the people brandishing the excuse to dismiss a given paper usually do so without the knowledge of what a small sample actually means. Nevertheless, this was progress. Suddenly, tons of papers that would very likely be bad died in the cradle and the number of false-positives was probably comparatively meager. And this shows us the way!

The way is this: increase the salience of methods and reduce the friction to using them. Make software packages that easily implement new methods. This will result in a proliferation of careless results—as has happened with the literature on Mendelian Randomization—but it will nevertheless improve the average quality above simpler methods and lacking checks. We also need to increase the salience of the need for assurances. Authors should be compelled by editors, reviewers, and their coauthors to check twice if they’re going to cut once; it costs effectively nothing to host a large supplement containing robustness tests, so journals should heavily encourage them.

Am I being too optimistic about this? Almost certainly. I live to be disappointed.

P.S. Did you know that the New York Times wrote a slanderous and obviously error-riddled article kvetching about me having access to this data? If not, you can go check that out here!

P.P.S. Here’s my replication code. Unfortunately, I can’t provide the data; you’ll have to apply for access to that yourself.

Have you ever heard about “HAES”? It stands for “Healthy at Every Size” and it’s the idea that people can be healthy regardless of how fat they might be.1 This idea might sound surprising given what we know about how obesity affects health.2 And yet, it has garnered a degree of intellectual cachet. In fact, it’s inspired some meaningful empirical research into the condition of being both fat and healthy, a condition known as metabolically healthy obesity, or MHO.

MHO is a poorly-defined idea.

A review of the MHO concept by the nutritionists Smith, Mittendorfer and Klein (SMK) noted that there “is no universally accepted standard for defining MHO, and more than 30 different definitions have been used in different studies”. Most studies define MHO based on being obese while having some small number of the symptoms of metabolic syndrome.3 That sounds fair enough: MHO is basically defined as having some signals of looking healthy and being obese. But is this even necessary? How prevalent is MHO versus its antithesis, metabolically unhealthy obesity, MUO? In the CDC’s nationally-representative NHANES dataset, the answer is ‘fairly uncommon.’

8.8 to 13.9% of each obesity category qualified as MHO in the latest sampling year based on a fairly strict definition I found had been used in this dataset by Wang et al. Because more of the obese are Class I (i.e., BMIs of 30-34.9) than are in the higher BMI classes, the majority (about two-thirds) of those who are MHO are borderline obese, and with more extreme obesity, MHO is rarer relative to MUO. This is consistent with the findings of the SMK review:

In general, MHO is more common in women than in men, in younger than in older adults, in people with BMIs less than 35 kg/m² than in people with BMIs of 35 kg/m² or higher, and in people of European ancestry than in those from Africa, South America, and South Asia (Indian ancestry).

Notice the line “MHO is more common… in younger than in older adults”. This is crucial. MHO is unlikely to be a stable condition. I tried to find every study that examined how people converted between MHO and MUO and found a lot of them, and they generally agreed: the more time that passes, the fewer people maintain MHO.

Though I’m not confident in the appropriateness of this analysis, it is possible to take the results from these different studies, with all their differences in definition noted, and to make a projection about how stable MHO can be. If we do that, it looks like 8-9/10 cases of MHO become MUO by 20 years in. That’s a theoretically dubious estimate, but I imagine something like it is not that far off the mark. Majority conversion in that time is highly believable.

MHO is defined as obesity with a reduced but not zero metabolic downsides typically associated with obesity, and… it’s not very protective.4 A few select quotes from SMK should suffice:

The risk of developing T2D is much lower in those with MHO than MUO, but is still about 4-fold greater than in those who are MHL [i.e., metabolically healthy and lean], and is directly related to the number of metabolic abnormalities at baseline….

The risk of CVD events (new-onset angina, fatal and nonfatal myocardial infarction, sudden cardiac death, fatal and nonfatal heart failure, and peripheral vascular disease) is lower in people with MHO than in those with MUO, but is still higher in people with MHO than in those who are MHL….

The combined data from five large cohort studies that followed participants for an average of 13 years found that people with MHO and no metabolic syndrome components (excluding waist circumference) did not have an increased risk of all-cause mortality compared with the MHL group; however, the risk of all-cause mortality was greater in participants with MHO versus MHL when participants with one abnormal metabolic risk factor (excluding waist circumference) were included in the MHO group.

The people who are MHO are still at a high risk, and because so many of them convert to MUO, the risk is higher than it might appear at a given ‘healthy’ baseline because so few can stay ‘healthy’. And this has a lot to do with fat itself being bad and, like cholesterol, causing damage that accumulates over time.5 We can see this in a variety of ways.

For one, we know that being fat is bad for glycemic parameters—that is, blood sugar. Bad glycemic parameters lead to cumulative damage because they’re constant stressors on numerous areas of the body, and perhaps also because they lead to obesity. In long-term follow-ups, there’s evidence of metabolic memory such that earlier periods of better or worse glycemic control affect complication risks years down the line. Patients that engage in early, intensive glucose control efforts also show reduced rates of microvascular complications, with numerous systemic benefits including, notably, reduced risk of all-cause mortality.

Mendelian Randomization supports the idea that BMI and abdominal adiposity are causally linked to both insulin resistance and type-2 diabetes, but not clearly to fasting glucose itself. This fits with obesity mostly wrecking insulin sensitivity first, with fasting glucose remaining compensated for a while. An individual could appear ‘MHO’ while being on a path to MUO as their insulin sensitivity degrades.

Moreover, one of the things that predicts going from MHO to MUO is visceral fat accumulation—which tends to happen with time spent being MHO and MUO alike—, and visceral fat seems to be itself bad for glycemic parameters. As a sort of natural experiment, people with lipodystrophy also tend to develop severe insulin resistance, hypertriglyceridemia, fatty liver, and diabetes. However, it’s not clear that just removing this fat fixes the issue. Liposuction cases don’t seem to show improvements in insulin action and glycemic control.

For two, obesity is harmful through mechanical load. Having more tissue also predicts, and quite likely causes, higher rates of cancer, in addition to all the other carcinogenic effects of obesity. It directly leads to sleep apnea and hypoventilation; fatty liver and liver fibrosis; osteoarthritis and musculoskeletal damage; venous thrombosis and impaired fibrinolysis; kidney damage; adverse adipokine signaling; chronic low-grade inflammation; and by being fat, you’ll probably be respected less, leading to whatever health harms follow from being socially rejected and disliked more than might be the case if you were skinnier.

If you’re a guy, you should be worried by obesity because it lowers total and free testosterone and raises the estrogen to testosterone balance because adipose tissue expresses aromatase, which converts androgens to estrogens. As a result, obesity can lead to gynecomastia, lower libido, erection problems, worsened fertility, and reduced muscle mass—obesity-related sarcopenia! For women, the worry runs the opposite direction, with obesity being relatively masculinizing, contributing to the worsening of PCOS and hyperandrogenic abnormalities that can impair fertility.

I could go on.

My point is this: Healthy at every size is quasi-mythical. There are people who appear to have relatively normal metabolic parameters while being obese, but they are generally not very obese, just somewhat fat. Moreover, few of the seemingly healthy obese people can sustain whatever metabolic healthfulness they have for very long, and to the extent they can, it’s probably because whatever damage they’re accumulating isn’t being measured or isn’t being used to define that person as metabolically unhealthy even though they’ll soon become metabolically unhealthy by any measure.

If we want people to be as healthy as possible, we’ll not humor the idea that people can be healthy at every size, because, frankly, it’s just not true.

The studies referenced in the charts are ATTICA, ELSA-Brasil, the Nurses’ Health Study, Hwang et al. (2015), Kang et al. (2017), MESA, KERCADRS, the Kadoorie Biobank, Ahvaz, Gilardini et al. (2018), ELSA-England, Kim et al. (2019), TLGS, NWAHS, CPRD, REGICOR, Whitehall II, Achilike et al. (2015), SATSA, Elías-López et al. (2021), Higashioka et al. (2025), CHARLS, and REACTION.

The NCPP is a huge dataset involving 60,000 pregnancies that happened across twelve U.S. medical centers between 1959 and 1974, with kids followed up for seven years, and measured across thousands of variables, from birth defects to growth trajectories to blood groups to IQs and personalities. Out there somewhere, subsets of the participants have been followed up for decades, but more on that later.

The big problem is that, for some reason, this meticulously gathered dataset that cost taxpayers millions to generate has produced barely any publications. Why? Simple: it’s a complete mess. If you want to access this wealth of data, you have to put in a lot of effort scanning microfiche. Microfiche is so old that most of you don’t even know what it is. Basically, it’s these tiny little slots of 10x15 cm transparent film containing highly reduced, unreadable images of documents, like newspapers, reports, and court and medical records. If you want to access that data, you have to scan that in, and that’s a lot of work!

So, with the help of our handy-dandy artificially intelligent assistant, Mr. Claude, I developed a scan and OCR pipeline that—with unfortunately large amounts of involved work on my part—eventually got all of the available NCPP data digitized. Then, I organized it, about 60% by hand, and 40% by Claude (if he’d errored less, he could’ve done more. Alas…), and made a website where you can search through all the variables, download all the data, and tell me if there are any problems that I might’ve missed.

As an added bonus, I’ve even expanded the dataset! (And I have more extensions planned.) Before I get onto showing you some of the really cool analyses you can run with this data, I want to tell you about those extensions. Want to skip that? Click here.

Dataset Extensions

I precomputed a lot of variables to make life easier for analysts who want to use this data. I provided g factor scores, growth measures, developmental and behavioral scores, and attrition and Census survey weights, but the big value-add is that I provided kinship links.

Kinship links are means of identifying siblings in the dataset. When you have siblings, this opens up lots of new analyses. For example, we can look compare siblings who were breastfed to ones who were not and see who’s more likely to develop allergies, a high BMI, a high IQ, or whatever else we might be interested in. With siblings who are different degrees of relatedness, like half-siblings (Relatedness = 0.25) versus full-siblings (R = 0.50), you can also fit behavior genetic models. Combine that with variables like sex and race and you can compute between-group heritabilities.

I constructed the kinship links in a few ways. For twins, I used determinations based on the ABO, Rh, MNSs, Kell, Duffy, Kidd, P, Lewis, and Lutheran blood groups, extracted from cord blood around birth. This allowed me to take the hundreds of twins—siblings born at the same time—and classify them as either dizygotic (fraternal) or monozygotic (identical) twins. But I didn’t have this data for every twin, so I used the twins I had and used other criteria and even built a Bayesian classifier to disambiguate even more of them.

For the non-twin siblings, I classified full- and half-siblings by way of a scoring system that integrates different types of evidence. For example, if a child’s father is listed as someone different from the father for their sibling, that’s a big bump towards ‘half-sibling’. One of the ways I thought I’d be able to distinguish a few half- from full-siblings was to use ABO blood groups for the mother and the children separately, to identify pairs where it was impossible for a given father’s genotype to produce both kids in a set. Unfortunately, this didn’t lead to anything substantive, but in fact, to identifying impossible combinations where the results for the kids couldn’t have even come from the mother. This meant identifying a very low rate of serological error, which was to be expected from the lab equipment of the time.

My sample ended up substantial, making this a great resource for future studies involving sibling and even cousin controls:1

Want to know what else I brought to this dataset? Go check out the paper!

What Can We Learn With This?

A lot! I have a short paper with some interesting tests over here. I’ll provide a few examples from it. Hopefully this piques your interest and you check out the paper.

General Intelligence is Real

How do we know if general intelligence—g—is real? There are several ways of getting at this question, but one of the most direct is a simple statistical test. This test involves comparing two models: a Common Pathway and an Independent Pathway model. Here’s an illustration of the difference:

If you’re unfamiliar with behavior genetic models, this might look like strange esoterica. Basically, what this shows is estimating the heritability and shared environmental influence on some set of correlated indicator variables, X_1 through X_p. If the variables are correlated because of a latent trait like g, then the additive genetic (A), shared environmental (C), and nonshared environmental (E) influences on the variables run through the latent trait given as P in the diagram. This is a test of whether our understanding of a given construct actually carves nature at the joints, and if the test supports a given latent construct, then it’s much more likely to be real.

Several papers have tested this for a huge number of psychological variables, from personality to intelligence. In the domain of intelligence, the first explicit test affirmed a common pathway model—supporting g—in a Japanese sample. The second explicit test also affirmed a common pathway model, but in a larger study of American veterans. The third explicit test was by me—I found two common pathways and one independent pathway on a gifted sample, which might be explained by Spearman’s Law of Diminishing Returns—, but the third one that was published was done on a Serbian sample, and it supposedly found evidence for an independent pathway model. However, that model was misfit because one of the tests was a very unreliable measure (something I was able to show empirically on a much larger American sample!).2 Finally, there’s a recent common vs. independent pathway modeling study, but it doesn’t actually test these models, it just uses the same words and runs some polygenic score analyses.

But enough about earlier results. I ran these model tests again for the NCPP. The independent pathway model fit best, but what this means is not totally clear because the independent pathway model was so similar to the common pathway model and the phenotypic g factor model.

The congruence coefficients between the A, C, and E loadings from the independent pathway model and the common pathway model come out to 0.996—essential equality—0.859, and 0.840, respectively—high, but not so high as to be identical. Thus, the genetic factor approximates the general factor. Moreover, a combined, vectorized congruence coefficient comes out as 0.885, and variance-weighted, it’s 0.957. The subspace overlap is 0.995 and the implied common variance—for the common pathway model, that means g—congruence coefficient is 0.996 and the rank-1 SVD share is 90.4%, versus a baseline of 68-83%. The common factor share of total variance attributed in the common pathway model is 39.7% versus 50.2% in the independent pathway.

The points of disagreement between the models had to do with the shared environment. By leveraging the independent pathway model, we can see that the shared environment is important for verbal and academic tests (Vocabulary, Comprehension) beyond what you’d expect from the variance directly attributable to g. In other words, the family’s environment affects those tests beyond g, but together, so they end up making the model with just g fit poorly. Perhaps this is why, when people find support for a common pathway, they tend to have dropped the shared environment, or used a sample where the shared environment is no longer expected to be involved due to the Wilson effect, where heritability rises with age, displacing it.

What these results end up telling us is that there’s considerable construct support for g: it appears to be very strongly linked to genetic influence, and not as much to environmental influence. But the model fits acceptably well, it’s just that there’s some subtest-level nuance that precludes it being the best option.

Note: The tests I ran were simple; I plan to come back and test additional factor configurations in the future, so there may still be a common pathway model that’s preferred over the independent pathway model. What I’ve shown is preliminary, but still highly informative because, given the dimensionality of the battery, all well-fitting results will converge, and every conclusion you reach with one model will be much like what you would with any other valid model. To that end…

Gene-Environment Interactions Don’t Matter for g

Even if the independent pathway model weren’t the ‘correct’ model, it can still be highly useful for several reasons. One such reason is that if it fits well enough, it sets us up to do powerful decompositions of the common and total variance across our tests, with implications for the common pathway model given how close to equivalent they are.

One of the things you can do is test whether the A, C, and E factors are actually interactive, rather than representations of additive effects. What this means is that you can determine the extent to which, for example, gene-environment interactions matter in aggregate as an explanation of particular factors, variables, etc. The result of checking this is… not a lot! There’s a small amount of evidence for interactivity that drops to next-to-zero when you model assortative mating (not shown), so the interpretation of aggregate genetic and environmental effects as acting additively is strongly supported and non-additivity is not a worry for us.

Race Differences in IQ Are Heritable

Another cool thing the independent pathway model can do is it can be used to compute the genetic and environmental contributions to group differences. The common pathway model normally can’t be identified to allow this, and doing so involves more complexity than you get in a traditional ACE model.

The ability to do these decompositions was noticed back in 1989, but few people every ran with this. Since the advent of this method, only a handful of papers have used it, with a focus on the Black-White IQ gap in one, a focus on sex differences in alcohol use in another, and a focus on sex differences in conservatism in another. There may be a few more out there that I’m not aware of.

In any case, I’ve applied the same methods here, to the Black-White IQ gap. This table will be hard to read, so I’ll interpret it for you below.

Basically, when we decompose the Black-White IQ gap, we see that the gap can be attributed to about 65% genetic and 35% environmental influence among these seven-year-olds. This is the gap in the common variance. When we include the variance specific to indicators, and not shared among them, the share of the gap that’s genetic rises to 69%.

That too is not the be-all-end-all, either, as there’s also the possibility that assortative mating—which has gone unmodeled—has inflated the shared environmental share relative to the genetic one, since it makes siblings more genetically similar than chance. Modeling values for assortative mating between 0 and 0.61, the common variance genetic share rises from 64.8% to 80.1% at a parental correlation of r = 0.20, and it goes to 97% at a parental correlation of r = 0.55. The total share rises from 69% to 91% at the same time.

The true estimate of the genetic contribution to the Black-White gap at this age (note: the shared environment fades into adulthood, so the genetic contribution will presumably rise) will likely be in-between the baseline estimate and the one with a parental correlation of 0.55. This gap, which we know to be highly heritable, is about 1 d (1.05 SDs) in size at the level of latent g, and a bit smaller at the observed level, which contains non-g variance and error variance.

People Can Personally Gauge Intelligence

When the WISC IQ test is administered, it’s done so by an examiner who listens to the verbal remarks of the child. In this dataset, the examiner also provides their subjective impressions of the each child’s cognitive function, personality, etc. The latent factor from these impressions is strongly correlated with latent g (r ≈ 0.9) and, as it turns out, these impressions are also entirely unbiased by race and they’re 90% as large as the tested Black-White IQ gap.

This might seem a bit boring, after all, the examiner generating the impressions just gave the child an IQ test. However, these impressions are not just tautological. They also correlate highly with a different, shorter set of tests that have objective scoring criteria independent of the examiner’s impression, and they correlate highly with test scores from a few years prior, when the kids were examined by someone totally different as four-year-olds:

What the examiner sees is intelligence, as it’s measured. They can pick up on the positive manifold to a strong enough degree that their impressions from testing someone still manage to correlate about as well with different IQ tests as those IQ tests do with one another. That might seem simple and boring, but it’s incredible!

Brain Size Causes Higher Intelligence, Mediates Race Differences in IQ

In 1994, Arthur Jensen and his colleague Fred Johnson used this dataset to reach some interesting conclusions about brain size, intelligence, and race differences therein. To validate my findings and theirs, I’ve replicated their result and expanded on it.

People with larger brains tend to be smarter. This relationship is known to be causal. In the NCPP, the correlation between head circumference and IQ holds up for Whites and Blacks, and after adjusting for height and weight:

This association holds up within families, although it’s significantly attenuated. Between families, where family-level variables might inflate the association, the relationship is r = 0.272 with WISC FSIQ; within families, comparing siblings of all types, the association is still significant, at r = 0.098 and 0.107 for just the full siblings. (This association grows with age and goes higher with better-quality tests.)

One of the really interesting things that Jensen showed was a practical demonstration of the phenomenon of' ‘supervenience’, or many-to-oneness. The way he did this was by showing that, if you control for brain size, the Black-White IQ gap only slightly shrinks. However, if you control for IQ, the Black-White brain size gap goes away. Raw brain size is one of many factors driving the IQ gap; IQ fully explains the brain size gap, as well as the gap in many other variables. Jensen only tested this with one form of matching, but I’ve extended this to several, all confirming the same result:

Another extension that I undertook was to look into whether any of the tests given to this sample predicted head circumference after taking out g. The answer between families, within families, and using multiple methods including just residualizing for factor scores or testing within a structural equation model was conclusive: at most, there are incredibly meager brain size-performance associations beyond g:

[Head circumference’s] entire association with the cognitive battery operates through g; there is no residual spatial or visual-processing specificity.

Since I’d recently seen people arguing that because males have larger brains than females on average and thus they must also have higher IQs, I decided to test that proposition, too. I found that when girls and boys were matched on IQ, girls still tended to have smaller heads. Given that, you might expect this next result: when they were matched on head size, girls tended to have higher IQs! And for thoroughness, I also found that when matched on weight and height, the slopes relating IQ to brain size were the same in both sexes.

Breastfeeding Doesn’t Affect IQ Scores

Even if breastfeeding didn’t help kids’ health at all, it would still be positively associated with all sorts of wonderful things and negatively associated with all the bad things in the world. The reason for this is that people today ‘know’ that breastfeeding is good, and the people who disproportionately adopt good habits are those who are well-off. Accordingly, their kids will tend to be well-off as well, regardless of the causal impacts of breastfeeding.

To get over this, we have to leverage natural experiments, randomized controlled trials, and within-family tests, where breastfed kids are compared to their bottle-fed siblings. The NCPP offers perhaps the most informative test of this to date, because when the NCPP began, breastfeeding was considered the unhealthy, low-class option. That’s just what people thought at the time, for whatever reason, so as a result, you can see things like 94.5% of mothers with <9 years of education breastfeeding versus 74.5% of college-educated mothers. You can also see that breastfed children averaged -0.65 SD lower WISC FSIQs—that’s almost 10 IQ points!

But, does that mean breastfeeding is actually bad? Well, let’s see. If we control for various maternal socioeconomic status variables, we can’t eliminate this negative association, only reduce it. Adding controls for birth weight, gestational age, smoking, and parity as well as a suite of neonatal health controls also doesn’t eliminate the association. So, naïve social scientists would likely say ‘yes, breastfeeding appears to be harmful.’ However, when we compare siblings who were breastfed to ones who were not, the association evaporates completely:

The bivariate cross-sectional association is large and negative: breastfed children score −0.65 SD lower in WISC FSIQ. Adjusting for demographics reduces this to −0.23 SD; full adjustment yields −0.15 SD. Under mother fixed effects, the coefficient flips sign: b = +0.08 (SE = 0.05, p > 0.1). All five IQ outcomes tell the same story—fully adjusted OLS associations are modest (WISC FSIQ: b = −0.15; VIQ: −0.02; PIQ: +0.05; SB IQ: −0.20; PCA g: +0.01) and all five FE estimates are small and non-significant (b = −0.02 to +0.13, all p > 0.05). The entire cross-sectional association is confounding from the reversed SES-breastfeeding gradient of the era.

This is one of the few times this has been tested. There are not that many within-family tests of the association between breastfeeding and IQ, so this is a big addition, not just because it’s another test, but because it shows that even in an era when the confounding was reversed, we can still back out the correct answer: a null effect!3

Socioeconomic Status and IQ

Within families, there’s exploitable variation in socioeconomic status that we can leverage to see if things like family income actually affect rather than just reflect ability. By using this, we can see that these variables have much small effects on IQ than is suggested cross-sectionally. That is to say, being in a wealthy family doesn’t matter that much for IQ:4

Similarly, I was able to run a variety of tests to see if the family size-IQ association was causal. Normally, this association is negative and strong, but it turns out it’s actually driven by confounding and family size does not predict lower IQs once that’s accounted for. I ran a lot of tests for this, but I’ll show you just one here: I found that among older mothers, a larger family was not related to kids having lower IQs. It was only among the youngest mothers—who are selected for being Black, poor, less educated, and having husbands who are similar—where a large family was associated with a lower IQ. That suggests a selection-driven finding!

A Peaty Non-Finding

I also tested something riffing on Ray Peat’s work. Ray Peat suggested papers born from mothers who had gestational diabetes would have larger brains. I checked this and, between mothers, the association was positive and nonsignificant (b = 0.03, p = 0.57), and it was still nonsignificant within families (b = 0.045, p = 0.88). There was nothing to do with IQ or effects at other ages or when stratifying by race either.

Have My Data, Have My Code

Everything I’ve done here is publicly verifiable. You have my data and you have my code; I’ve put it all online for you to see, to run, to critique, and to iterate on. It is yours, just cite my paper on the dataset if you plan to do anything with it. Doing so is a condition of using the data and/or the scripts I’ve provided, or for building from said data or scripts in any way. That’s not a big ask, so please, just do it, alright? Like so:

Lasker, J. (2026). The Collaborative Perinatal Project: A Modern Data Release.
OSF Preprints. https://osf.io/4tna9

I hope that more people will follow my example; I hope they’ll make their data and their code available to more people, and we can all come together and improve the scientific enterprise in the process. With that said, I hope you all appreciate what I’ve done here by modernizing the wonderful National Collaborative Perinatal Project!

Want to Help?

So, as I mentioned earlier, there are follow-ups for some of the participants in this data. These follow-ups go for various lengths of time, and unfortunately, none of them are publicly available, but all of them should be. The follow-ups I’m aware of are:

• Pathways to Adulthood (Johns Hopkins): Janet Hardy and Sam Shapiro followed 2,694 members of the Baltimore cohort through age 27–33 (1992–94), plus their mothers (G1) and children (G3), creating a three-generation study. 82% of participants were located; 65% interviewed. Archived at ICPSR Study #2420.

• New England Family Study (Boston + Providence): Stephen Buka (Brown), Jill Goldstein, and Larry Seidman (Harvard) have followed 17,741 individuals born at the Boston and Providence sites from the 1980s through the present day, with over 4,000 still active. The study has produced landmark findings on schizophrenia risk, ADHD, and adult cardiovascular health.

• Philadelphia-Providence Intergenerational Study: Klebanoff et al. re-contacted 1,782 female CPP offspring from Philadelphia and Providence in 1987–91 to study intergenerational transmission of preterm birth and low birth weight.

• Minnesota CPP 2.0: Logan Spector and Julia Steinberger (University of Minnesota) are conducting an ongoing follow-up of the Minnesota site cohort (now in their late 50s–60s), linking childhood exposures to adult cancer, heart disease, and diabetes via medical records and cancer registries.

• NICHD Mortality Linkage: Edwina Yeung (NICHD) linked 44,174 NCPP mothers to the National Death Index through 2016, enabling research on associations between pregnancy complications and cause-specific mortality across the lifespan.

These follow-up datasets are not included in this release because they are held separately by the individual institutions and were never incorporated into the central NICHD public-use files. The Pathways to Adulthood data is available through ICPSR; for other follow-ups, you have to contact the individual sites directly. I’ve tried, to no avail. If anyone would like to do that on my behalf, you can contact me on here or on GitHub with the results and I will gladly add them to the data release in later versions.

2

On the subject of that Serbian sample, we can actually use a bit of basic math to flip the interpretation of the result. Whereas the authors assumed—wrongly, given the unreliable WCST measure—that they had affirmed an independent pathway model, their common factor loading matrix (the loadings from A_C and E_C) suggest a common pathway is substantially correct. The way I flip the interpretation is through a singular value decomposition (SVD) of the common factor loading matrix from the study. You can obtain that from the authors’ diagram:

The first thing to note is that the model is obviously misspecified because there are some loadings that are exactly zero. The second thing to note is that an SVD suggests that 87.6% of the variance is rank-1, meaning that this data is about 90% compatible with proportionality, and thus something close to the common pathway model, there’s just a small 12.4% share that is not.

Quantitative qualification ought to be used whenever people are doing independent vs. common pathway modeling, because with a large sample size, less than total unidimensionality, and with measurement error, it is almost-certain that even when you have, say, 95% support for a common pathway, you’ll reject it in favor of the independent pathway model because large-enough samples practically ensure less parsimonious models will win.

However, the quantitative evidence should probably be better than just an SVD—which needs to be calibrated—and should include stuff like loading comparisons between models. Unfortunately, with just the data provided by Nikolašević et al., we’re not able to get that. But, I do go on to show how this can be done for the NCPP data in the supplementary materials.

I was inspired to do this by various GenomicSEM papers that have broken down the extent to which genetic effects on intelligence tests and other measures are consistent with affecting the general dimension (a la a Common Pathway) versus specific indicator variables (a la an Independent Pathway).

Sometimes they’ll give up and the result achieved will live on in the file-drawer, that place where results go to rarely ever be seen again. Sometimes researchers will try hard and they’ll be able to publish a null result or one with an effect in an unexpected direction. Unfortunately, one thing researchers often do—sometimes without even knowing there’s a problem—is they’ll figure out a way to misrepresent their results.

Researchers who get a null they didn’t want or a result they didn’t expect will sometimes look at different outcomes where the result might be different.1 Other times, they’ll decide to test the effect of different control variables, since those can change the significance or sign of the results. And in yet other cases, they’ll play around with excluding certain observations, transforming their variables, and just generally doing anything they can to change their result—sometimes a lot, sometimes a little, but usually ‘favorably’: the result becomes what they wanted, whether that’s the same result but significant or a different one entirely.

One seriously under-discussed strategy to do this that gets used all the time is searching for a subgroup in the data to re-estimate the effect. For example, say we run a drug trial and we find an effect of size d = 0.12, but the result isn’t significant. Unpublishable! But what if the effect is significant among women? You can split the data and… voila! The effect is significant in women (p = 0.047), so you’re good to go.

It’s time to rewrite your paper and conclude that you found something that affects women, but not men. Among men, the effect was nonsignificant (p = 0.663), and by standard procedure, that means nothing happened to them. Right? Perhaps you can see the problem in this chart:

The results for men and women is strikingly similar, so even though the results for men were not significant, it’s still not right to say that the effect varied by sex. The male result is just noisier, and that’s explained by the sample size difference. But the rule in academia is ‘publish or perish’, so you’ll have to focus on the significant female result while ignoring, downplaying, or implying—and perhaps even outright saying—that it contradicts the male one.

This might not seem like a mistake—you did find a significant female result—but it’s a pretty major one. What’s happened here is that, through data dredging, a fragile result that’s likely to be a fluke was dug up. This is an “interaction”: a difference in coefficients between subgroups, and it’s not significant, meaning that the difference in coefficients between groups is not statistically supported, so cutting up these groups and making conclusions about their different effects is not justifiable

Interaction effects take much larger samples to justify than do estimates of regular effects. To put that into perspective, you need an eight-times larger sample to detect an interaction than you do to detect a main effect with an effect size of 0.25 d.2

If you have enough grouping variables in your data, you can easily get significant results by looking for subgroups where the effect is significant, but because interactions take so much power to support reliably, they’ll also tend towards coming from null interactions and failing in replication. This has an unexpected upside: it makes it easy to detect p-hacking!

The reason why this is a shortcut to p-hacking detection is that most researchers—including those who go dredging for significant results—forget to test their interactions. In a lot of cases, these interactions are talked about but are not significant, so you can tell someone was p-hacking when they report them without a preregistration to prove their decision to test the interaction wasn’t just an ad hoc way to achieve significance.

There are so many examples of post hoc interaction tests the authors forgot to actually test to dredge up significant results, and they are often stunningly incriminating. I’ll provide one example from a potentially fraudulent study. The study I have in mind is the single randomized, controlled trial (RCT) for functional medicine, and it’s about coaching for elimination dieting.

For reference, functional medicine is a pseudoscience that its practitioners claim can reveal ‘root causes’ of people’s health problems through the use of unproven diagnostic tests and therapies that lack rigorous scientific backing.3 The premise is that chronic diseases are downstream manifestations of upstream imbalances in things like gut health, toxin burden, nutritional status, hormones, etc. After a functional medicine doctor (if you can call them that) runs enough specialized biomarker panels, they’ll claim they can address these ‘root’ imbalances through dietary change, supplements, ‘detoxification’, and other lifestyle alterations, resulting in a reversal of disease states that traditional medicine supposedly overlooks.

What functional medicine actually does is it bilks desperate people through prescribing extensive, often unnecessary, unvalidated, and unreliable lab tests like advanced lipid panels, organic acid tests, food sensitivity panels—which they often make in-house and rarely subject to empirical scrutiny—and so on, then funneling those patients into long-term supplement regimens and repeat testing to ‘monitor their progress.’ Patients are caught in a test-supplement-retest cycle that costs lots of money and delivers nothing better than traditional medicine—at best. At its worst, it prescribes quack cures that waste patient time and money while causing serious harm.

A near-synonym for this approach is ‘integrative medicine’, denoting medical care that involves elements of both traditional medicine and alternative medicine. With luck, the alternative medicine is useful, but it rarely ever is. Functional medicine is like that in that it’s usually quackery, not ‘integrative’ in any meaningful sense: it takes real biological concepts like the existence of the microbiome, epigenetics, and inflammation, and builds unfalsifiable clinical narratives around them. Functional medicine practitioners can always find something wrong about a patient by running enough tests, so they can always recommend another supplement or intervention. Then, to check how well that’s working, they can always order another set of labs.

Functional medicine has no endpoint where patients are told ‘you’re fine, stop coming.’

The authors of this functional RCT compared “functional medicine health coaching support” to “a typical self-guided elimination diet” with respect to effects on “dietary compliance and patient-reported health and quality of life.” They preregistered their plan on the NIH’s clinical trials website, stating that they planned to test one main outcome and two secondary ones. The registered main outcome was “Change from baseline PROMIS-Global Health score” (PROMIS GH) and the secondary ones were “Change from baseline Medical Symptoms Questionnaire score” (MSQ) and “Adherence/compliance to the elimination diet.”

This RCT was run by functional medicine practitioners and coaches with a vested, financial interest in functional medicine and it was funded by The Institute for Functional Medicine in collaboration with the Functional Medicine Coaching Academy. The three affiliations that the eight authors provided were The Center for Integrative Medicine, The Institute for Functional Medicine, and Functional Medicine Coaching Academy. But, they stated incorrectly: “The authors have no conflicts of interest to disclose.”4

Before even getting to the first word of the abstract, there seems to also be a second incorrect statement: “The datasets generated during and/or analyzed during the current study are publicly available.” But there is no repository link—no OSF, Zenodo, Figshare, or Dryad deposit to be found, and not even a link to the authors’ personal websites or a repository maintained by their funding institution. This makes sense, too, since when they registered their clinical trial, they stated that there was no “Plan to Share Individual Participant Data”.

The authors of this trial submitted a registration on September 16th, 2022 and the first version of their registration that was considered to have passed quality control checks was submitted on September 20th. That first viable version said that they were aiming to enroll 150 people. In their September 30 progress update to the NIH, they said the trial had started on the 26th, and that 150 people were enrolled. They maintained this claim in their final update on December 1st of that year, but when you check the paper, you see 139 participants mentioned. What happened to the other 11? One person allocated to control didn’t sign a consent form and ten people allocated to the intervention arm didn’t receive the intervention, had no time to commit, had health issues, etc. Is this important? We’ll see.

The authors’ provided rationale for their sample size is inconsistent, suggesting that they likely didn’t plan their trial based on the power analysis they claim to have performed. They wrote:

Assuming a 2-tailed α of 0.05, 80% power, and a standard deviation of 3 points in the difference of PROMIS GH means between arms, a sample size of 120 participants was estimated to detect a difference of approximately 1 point in PROMIS GH scores between study arms. This minimally detectable difference fell comfortably beneath the minimally important change in PROMIS GH scores. In order to account for unknown levels of dropout in this intervention in which the consent, intervention, and data collection were all performed online, a conservative estimate of 25% dropout was employed when recruiting potential participants. As such, 139 participants were randomized to either the functional medicine health coaching intervention or self-guided control arms of the study.

Can you see the arithmetic problem here? If you need 120 completers, then you need 160 participants. If you start with 150, cutting 25% gets you to 113 (112.5), so by their own admission, they were underpowered given “a conservative estimate of 25% dropout”. The “As such” before stating that they enrolled a sample size that their own calculations showed was too small with their “conservative” attrition estimate is curious, to say the least.

But, at least for power, those missing participants really didn’t matter because, to make their situation worse, the power analysis is completely wrong. With a two-tailed p-value of 0.05 for 80% power and an SD of 3 points, the required per-arm sample size is given by:

Using their provided values, z₀.₉₇₅ is 1.96, z₀.₈ is 0.84, σ is 3, and δ is 1. Plug in the numbers and you get just over 141. That means the required sample size to achieve 80% power per arm is 141-142, but they only enrolled 139 total. They needed 282-284 and got less than half that! The actual power maths out to <50%. Even being incredibly generous and using a one-sided test5, they would have had 72% power for detecting a 1-point difference with a standard deviation of 3.

And I want to reiterate: I am being incredibly generous. PROMIS scores are T-scores, so they come with a mean of 50 and a standard deviation of 10 in the norming sample. The authors knew this: they explicitly stated that “PROMIS GH t score distributions are standardized such that a 50 represents the mean… with the standard deviation around the mean of 10 points.” That means that they should have done their power calculations with a standard deviation that was more than three-times what they claimed to do their power analysis for. If we calibrate based on that, then with a one-tailed test, they would need more than 1,200 people per arm to achieve 80% power. With a proper two-tailed test, they’d need almost 1,600. With a standard deviation of 10, 120 people total, and an insistence on detecting a 1-point difference, the power is just under 14%6; with a proper two-tailed test, their power comes out to just under 8%—less than one-tenth of what they aimed for.

Looking at their baseline standard deviations for either group confirms that the PROMIS scores they had were substantially more variable (e.g., PROMIS GH — mental health SD = 7.6 for the control group, pre-intervention; SD = 6.8 for the intervention group, pre-intervention) than what they might have planned for.

I say “might have planned for” because it seems the preregistration of this trial was not real. The authors first submitted a failed clinical trial registration attempt on September 16th, but in their paper they wrote “Participants… were recruited via email invitation between September 10-19, 2022.” The International Committee of Medical Journal Editors (ICMJE) definition of a prospective registration for a trial requires registration before the first participant is enrolled, not before the study start date. Sending recruitment invitations is considered enrollment activity, so by their own admission, prospective participants were enrolled on the wrong dates.

If enrollment began after submission on September 20th, then the trial was prospectively registered by the ICMJE definition. But, if it began on the 26th—as they claimed—, then it was also not, as the authors claimed, a 10-week trial. If you check the clinical trial registration again, you see that the study completion date is listed as November 30th, meaning that the trial lasted ~9.3 weeks, not 10 weeks. The only ways they could’ve arrived at 10 weeks for the length of the trial would have been to start it before registering it officially, to have rounded the number of weeks upwards without saying it, to have used an atypical and unstated definition of a ‘week’, or to have made an error.

The study has yet more issues. For example, in Table 3, they claim to be reporting the baseline characteristics of the study sample, but they’re not really doing that. They are, instead, reporting the baseline characteristics of the follow-up sample—that is, the sample after attrition. This is a big deal because that means we don’t see the post-randomization sample’s characteristics, only what remains after randomization has potentially been broken.7 Alarmingly, the authors also said they had 100% follow-up from participants who provided baseline data, which is 125 people. But in their flow diagram, they don’t say anything about excluding participants because of a failure to provide baseline data. The only way to make sense of their result is to suggest that they made an error—I don’t doubt they did!—or something like that they didn’t actually collect baseline data at baseline, meaning that it’s retrospectively gathered. But, as will be revealed, we know this too is wrong.

Throughout the study, the numbers reported in the tables also don’t seem to really matter, because the authors change the baselines randomly. For example, they wrote that 87.8% of the sample was younger than 55 years old, it was 89.9% female, it was 70.5% White, and it was 69.1% Americans. These numbers are impossible to get from a sample size of 125, but they work with 139. These numbers are stated to be for “the study sample”, but immediately following their presentation, we’re greeted to a table saying that the study sample has different, discrepant values for these things, and a sample size of 125.

When it comes to actually estimating effects, the authors claimed that they used an intention-to-treat (ITT) approach, meaning they used the data from everyone who was randomized into each group. However, this is not correct given their statement that they randomized 150—or 139, as they also stated—but only 125 provided the baseline data and were included in the analysis, meaning that, for one, an ITT analysis was impossible, and for two, they admit they didn’t do it. They did a completer analysis, and received results for a potentially nonrandom sample!

The actual substance of those estimates is—you guessed it!—wrong.8 The table they’re presented in shows a pre sample size of 74 for the control group and 65 for the intervention group, with post sample sizes of 64 and 61, yet they say only 125 participants provided baseline data. But I see they provided data right there in Table 4! Did the authors actually collect psychological measurements without more prosaic information like age, race, or the other details that they probably had to collect during the training module the sample was recruited from, a training module that we know they collected at least some demographic criteria from, because they reported the training and licensure breakdown for it?

The study is also full of careless, numerical mismatches and misstatements. For example, in the abstract, they state that there were 64 people in the intervention and 61 in the control group, but later on in the Results section, the numbers become 61 in the intervention group and 64 in the control arm. The article does have an Erratum on it that corrects two other errors in the abstract, but it missed these issues! Their flow diagram describing the sample size through recruitment, randomization, follow-up and analysis also shows 150 people were randomized, but elsewhere they said “139 participants were randomized”, indicating that the authors conflated randomization with receipt of treatment.

One can forgive small errors, but only if there aren’t so many major ones. Onto results!

The p-values for the estimated differences between groups were not consistent with the authors’ reported means and standard deviations, nor with an ITT approach, a post-score comparison, or a change score comparison from the initial sample, or for that matter, from the types of t-tests they claimed to run. The result could be compatible with something to do with completers, but we just don’t know given what’s provided. Under any analytical model that can actually be reconstructed from their paper, the difference between groups for the MSQ score and the PROMIS GH physical score is highly significant. Why didn’t they report this? Instead, they reported that the difference was not significant. As such, they had to resort to other methods to reach a functional health-favoring conclusion.

After getting nonsignificant main results, the authors came to the conclusion that:

[F]unctional medicine health coaching was feasible and associated with superior elimination diet compliance as compared to a typical self-guided elimination diet control situation… [And] between-group improvements favored the functional medicine health coaching arm among participants who were experiencing greater symptomatology at baseline.

They got to this from nonsignificant results by explicitly doing subgroup analyses—searching for a group the result was significant in, without reporting the interaction.

The feasibility claim and the claim of superiority for functional coaching was based on doing a median split of each measurement and then checking if the changes between the intervention and control groups were significant in either of the groups that had higher and lower levels of symptoms.

This part of the paper is extremely poorly reported, even beyond how poorly the rest of the paper is reported. We receive no sample sizes or standard deviations, and we don’t know which pre means were used, so the result is impossible to reproduce. The result is apparently greater improvement with the functional health intervention for the people with more symptoms, to the tune of p = 0.0038 for the PROMIS GH mental health scale—which was not what they preregistered they were going to analyze—and p = 0.047 for the MSQ scale.

They didn’t preregister this analysis, they created the subgroups in a way that dramatically increases the risk of false-positives, and they did a large number of additional tests, all without any multiple comparisons correction. Doing any multiple comparison correction would have rendered all of these ad hoc and underspecified results nonsignificant!

And all of these problems exist before the perhaps more important problem that their measures were just awful! To get to the result that functional medicine coaching leads to superior elimination diet compliance, they just asked participants to provide one self-rating of how well they followed the elimination diet. A 1-10 self-rating! This is highly subjective, vulnerable to social desirability, demand effects, and the mere fact that the people in the intervention were more involved by virtue of receiving five direct coaching sessions versus those in the control arm getting to view some webinars. A single-item self-perceived adherence rating is not a robust measure of actual dietary compliance, and especially not in a sample that was recruited from among a group of people who had already shown an interest in functional medicine.

The only ‘gold standard’ study of how functional methods work seems to be irredeemably bad and there are many grounds on which it not only deserves, but commands a retraction, from the authors’ major undisclosed conflicts of interest, to their faulty registration and analytical deviations, to the numerous reporting errors, discrepancies, and even the erroneous claim about data being provided, along with their inexplicably poor understanding of study design, and their all-too-obvious p-hacking.

Looking for more to read? Why not learn how one study often outperforms a meta-analysis:

Wearables are all the rage. From FitBits to WHOOP bracelets to Oura rings to smart glasses, continuous glucose monitors (CGMs), metabolism-tracking patches, and the Apple Watch, everyone in the fitness space seems to be wearing something that tracks their health, whether they’re monitoring their VO2 max and heart rate variability or just the basics like sleep and heart rate.

A lot of people in that space also think wearables help to improve health. Naturally, then, many have taken to promoting wearables as a tool for health improvement. But do wearables actually help? It’s intuitive that they might. For example:

• Wearables can make invisible health markers visible so people can act on them.

• Wearables can provide immediate feedback on health habits: do more steps, sleep more, eat a slice of pie, and you’ll get numbers back right away so you can know how to adjust your habits.

• Many wearables come with reminders, nudges, and trackers that might help people to stick to their habits and form healthier ones.

• Some wearables use gamification and include social and personal accountability mechanisms like badges, streaks, leaderboards, and workout sharing, possibly providing users with an external locus of control.

• Wearables support N-of-1 learning, which could feel empowering and promote consistency.

• Some wearables provide clinic-level monitoring and early detection for serious health issues like atrial fibrillation, sleep apnea, and so on, and users can treat them like a way to get early alerts that let them act sooner for better results.

Notice my qualifying words: “can”, “might”, “possibly”, “could”. All of this is well-and-good in theory, but theory only matters if it translates to the real world! Does it? No.

Just the Evidence

The biggest question for the most common types of wearables is Do they increase physical activity? We’re looking for biobehavioral feedback, priming, sunk costs—whatever you want to argue from—and we do see evidence for it. Basically every review suggests that there is a small-to-modest effect on physical activity.

Sometimes this effect is tenuous. For example, in Au et al.’s 2024 systematic review and meta-analysis on wearable activity trackers for children and adolescents, trim-and-fill rendered the effect size for daily steps nonsignificant (-0.01; 95% CI: -0.35-0.33). For measured “moderate-to-vigorous physical activity” (MVPA, a common outcome in this literature), the effect size held together after adjustment, but was small (-0.14) and marginally significant (p = 0.01). In Wu et al.’s 2023 systematic review and meta-analysis on wearable activity trackers for older adults, they were underpowered to investigate publication bias, but their trim-and-fill for MVPA nonetheless still halved their effect size from about 0.54 d (0.36-0.72) to about 0.25-0.26 d (~0.05-0.46), though it retained significance (marginally: p ≈ 0.02 after correction).

Effects in this literature start off not being very large, and they get meaningfully smaller if you’re able to account for publication bias. Some reviews have gone so far as to note that, the fewer the number of studies in the meta-analysis, the more positive the results seem to be, which is a clear signal of publication bias. Charted:

A lot of hype, from rarely-reported, relatively novel outcomes—something that would help a researcher stand out when it comes to getting cited!—is probably driven by selective publication, which tends to inflate effect sizes.

The effects in the literature get even smaller and more statistically dubious when you remember that the analysts are often medical doctors, and MDs are not generally all that good at stats.1

For example, here’s something any economist but few doctors would catch. In the Wu et al. review I cited, they reviewed three other outcomes: daily steps, total daily physical activity, and sedentary time. There looked to be evidence of publication bias for all of these outcomes, but there was too little power to tell. The thing the economist would’ve pointed out though, is that each result-level meta-analysis included multiple estimates per study, without clustering.

If a qualified economist had been on the team, they would have noticed the standard errors were unclustered and told the exclusively-medical team that their results were too precise because the results were dependent on one another—they basically ended up estimating the same things, or methodologically-dependent things multiple times, with no accounting for that fact! If their meta-analytic results had been estimated with clustering, the significance and effects would change, usually subtly, but sometimes dramatically.

For total daily physical activity, accounting for clustering means going from a significant 0.21 d to a nonsignificant 0.29 d ; for daily steps, the result goes from a highly significant 0.59 d to a still highly significant 0.57 d, but now there’s a significant trim-and-fill result that brings it down to a still-significant 0.48 d; for sedentary time, the result practically doesn’t change, but the trim-and-filled does make the result nonsignificant (d = -0.09, p = 0.07).2

These and more obvious, sometimes-important, sometimes-not statistical issues are the norm for wearable studies, and if you accounted for them consistently, you would find that the already modest effects wind up being considerably smaller than they appear. Throw in that many of the studies have bad designs—how do you even blind a wearable study?—, highly motivated volunteers, temporary novelty and participation effects, and issues with attrition and compliance, and reasonable people will conclude that the effects on these proxy outcomes need to be downweighted a lot.

And that last sentence is really important: these are proxy outcomes. No one should actually care about whether outcomes like physical activity and daily steps move up unless they elicit health changes—the real outcomes of interest! Fortunately for us, because these proxy outcomes are so widely accepted, researchers tend to treat them as primary, and they publish papers on the basis of significant results for said proxies, with less focus on actual health improvements. That makes inferences with respect to actual health improvements biased upwards to the extent they correlate with the proxy outcomes, but otherwise effectively unbiased. So, what do those show? Not a lot!

Consider this: in Ferguson et al.’s 2022 review, they found that meta-analyses—which are usually uncorrected for the issues noted earlier—generally do not show much in the way of physiological benefits, with smaller effects in healthier and more population-representative samples.

The evidence for psychosocial outcomes is even worse:

This comes up again and again: there seem to be effects on promoting healthy behaviors, but they don’t translate to effects on body composition, physical risk improvements, psychosocial improvements, etc., and where they do, the samples are usually at-risk, so generalizing to the general population is not advisable.

If we move away from activity monitoring towards CGMs, we see that, though diabetics obviously benefit and are protected by having them, normal people don’t get a whole lot of benefit. Think about this for just a moment and it becomes really obvious: What is a normal person even supposed to do with a CGM? Are they going to monitor their diet better? For what purpose? So they avoid glucose spikes that don’t tend to affect healthy people’s health much at all? It’s hard to even take CGMs seriously as something to recommend the general, non-diabetic population.

When we give nondiabetics CGMs, we see weak evidence for small improvements in postprandial and fasting glucose levels and variability, steps and walking time, weight, and a handful of other health proxy outcomes.3 There is virtually no evidence for changes to hard outcomes. And frankly, we shouldn’t expect anything, because there’s just not much in the way of health information for a normal person to glean from a CGM.

At best, I suspect non-diabetic CGM users will get what my friends and I have experienced: you learn what spikes your glucose, and you manage to cut down on alerts with some habit changes of dubious importance. The novelty decays into background noise; the alerts and reminders become less frequent and more obviously errant and ignorable; once you’ve learned that you sleep worse when you drink and less sleep makes you more tired, the wearables just become a source of cognitive load.4

Before wrapping up, I want to mention something else: many wearables are unreliable. Counting steps and heartbeats is easy, generally reliable, and not worth a whole lot. Sleep duration? Sleep staging? SpO₂? Atrial fibrillation? Don’t trust this stuff too much. Calories and energy expenditure? Stress? Cuffless blood pressure? Really don’t trust that.

One review found accurate step counts and heart rates with poorer reliability for energy expenditure estimates. A new study of the Apple Watch found results that were sometimes good, but for the more advanced stuff it was all over the place, with this fairly expensive device having only a moderately accurate step count and sleep tracking. Reliability is massively heterogeneous, both within and between devices and certain measurements need routine recalibration that users might forget to do.5

The best evidence in favor of mass wearable adoption is a chain of “can’s”: Wearables can surface hidden health signals; wearables can nudge behavior; wearables can sustain motivation.

The empirical record says that chain is not very strong; in fact, it’s usually broken.6

Behavioral effects are small and shrink further once you account for statistical and sampling biases; engagement and compliance fade as device novelty decays; and the endpoints that actually matter—weight, cardiovascular risk, mental health, HbA1c, and other hard outcomes—rarely move outside of high-risk, tightly-coached samples (and even their gains tend to attenuate with time). Add that most headline metrics are noisy or model-driven and you end up selling people devices that mostly add cognitive load and spiffy dashboards, only rarely resulting in people getting healthier.

Telling everyone to use wearables will generate a lot of data but not much health.

The algorithm over on Twitter updates too quickly. If you like a post, you’ll get a lot of exactly that style of post on your feed, even if you don’t care for it. One of the ways that’s showing up for me recently is in a surge in posts kvetching about the latest anti-vaccination nonsense. An example from the Joe Rogan Show stuck out to me: in it, the the interviewee makes his case that early hepatitis B vaccination does, in fact, cause autism, and he makes it on the basis of just one study: Gallagher and Goodman (2010).

After the very obvious isolated demands for rigor—around 1:16—the guest says:

There is one study out there regarding Hep B vaccines and autism. It’s from Gallagher and Goodman out of University of Stony Brook, it’s in the peer-reviewed literature, and it showed that kids that got Hep B vaccines versus those that didn’t in the first six months of life had three-times the rate of autism—statistically significant. Gallagher/Goodman, University of Stony Brook, it’s on PubMed. That is the only study of Hep B vaccine and autism that you will find in the peer-reviewed literature. If you do it based on science, on the published literature, that’s the only one out there.

The study he mentions is, indeed, the only published study on the highly-specific topic of hepatitis B vaccination in the first six months of life. It is an atrocious study—as all anti-vaccine studies have proven to be—and we are lucky that its data is public.

Bad Studies With Public Data Fall Apart Readily

One of the virtues of economists is that for papers published in high-profile venues, they demand a replication package—a collection of all the code and, usually, the data required to replicate a set of findings—to ensure researchers ran the tests and got the results they claimed to. Unfortunately, a replication package is not always available; in those cases, fraudulent analysis, fraudulent data, and fraudulent presentation can linger, sometimes for many years and with significant effects on real-world policies and in some instances, they can even have effects on court cases.

Gallagher and Goodman’s (G&G) study uses the National Health Interview Survey (NHIS). Given that the NHIS data is public, we’ll be able to be virtuous just like the economists: we’ll be able to check if G&G’s results hold up!

Reproducing Results

The first thing to do is to mimic their exact model specification. G&G presented results from two models but ran several more. The ones they presented results from were univariate and multivariate logistic regressions using hepatitis B vaccination to predict parent-reported autism diagnoses. Can we reproduce them? For the most part, yes!

To reproduce G&G’s results, we have to subset the NHIS down to (1) just boys, (2) aged 3-17, (3) who have shot records, (4) and were born before 1999, (5) and had files containing autism and neonatal status, (6) and non-missing covariates, for the multivariate model. The outcome of interest is parent-reported autism diagnosis and having that on file was also required to enter the sample. This pares down the sample from 86,996 to 44,841 (boys-only), to 33,710 (ages 3-17), to 7,936 (shot records), to 7,737 (born before 1999), to 7,454 (autism and neonatal status), and finally, to 7,055 (no missing covariates).1

The univariate and multivariate results mostly hold up, even though I did not have their original SAS code available to handle the data exactly how they did. The differences in data parsing likely explain why when I used their specified sample, I found two more autism cases than they did. Who knows why that happened? But at least we got basically the same statistics, confirming that I am ‘good to go’ for deeper analysis. Let’s start said deeper analysis by talking about multiple comparisons.

Multiplicity

As you perform more and more statistical tests, the odds of getting a statistically significant result drift higher and higher. This means that a p-value of 0.05 is not that surprising once you’ve run a test ten times; at that point, it’s more like a p-value of 1-(1-0.05)^10 ≈ 0.40, assuming independent tests.2 To ensure a reported p-value maintains its nominal interpretation (i.e., that a p-value of, say, 0.05 actually means ‘a 5% chance of a result at least as extreme under the null’), researchers have to use multiple comparisons corrections, like the Bonferroni correction,3 to either adjust the thresholds for significance or adjust the p-values themselves to control the error rate.

However, because G&G conducted an unknown number of tests, we cannot know the size of the ‘family’ to adjust for. We know that G&G ran a lot of tests, so the adjustment we have to do must be large. At a minimum, they tested the primary exposure coefficient across the univariate and multivariate models for the boys and they reported those results. Among the ones they state that they checked the results without reporting for are the girls’ versions of the boys’ models they showed, the boys model with no birth year restriction, the boys model with and without the mothers’ high school graduation or higher as a covariate, they tested the relationship between hepatitis B vaccination and a non-autism composite, and they tested for vaccine-autism relationships for the varicella and MMR vaccines. Counting just this stuff and the associated tests in their models and you get to 35 individual parameter tests at minimum, not even counting whether they also ran univariate models for girls without birth year restriction, whether they ran a sex-combined model before deciding to stratify, and any exploratory looks based on the ethnic subgroups whose differences from the main sample they highlighted in their Table 2B.

G&G obviously ran a huge number of tests, but even adjusting for just the four univariate tests eliminates their main results. With just the four univariate tests in the family, the hepatitis B result is not robust to Bonferroni correction (the most conservative; a passing p-value becomes 0.0125), to Holm-Bonferroni (less conservative), or even to Benjamini-Hochberg correction (even less conservative). The only variable that passes is that two-parent households come with significantly less autism, and that’s not their focal hypothesis and also not biologically sensible at all.

More pointedly, using the p-values from our reproduction of G&G’s results, being non-Hispanic White as opposed to anything else and coming from two-parent household both retain their significance, but the hepatitis-autism association is never significant in the first place, let alone after any corrections.

Even going multivariate and using the same extremely favorably limited corrections, hepatitis B does not survive Bonferroni or Holm-Bonferroni correction, and it squeaks by Benjamini-Hochberg (0.031 vs. a threshold of 0.0375) in their specification, but not in our reproduction (in which it’s nonsignificant to start, despite our finding two additional cases). With the closer-to-correct family size of at least 35, nothing they found is significant and everything is far from it.

Conclusion I: G&G’s result is not robust to attempts to reproduce their exact sample and specification or to multiple comparisons correction. Their results are marginal and in the range of 0.01 to 0.05, making them especially likely to have been p-hacked.

Robustness to Their Sampling Choices

Let’s run through the models G&G did, but didn’t report on. This means going beyond reproducing their results to adding the data from 2003—which they had access to but decided to omit for some unknown reason—, removing the birth year restriction so we use all birth years and not just those born before 1999—which they justified via thimerosal exposure—, and then using the results for both sexes.

Their results are not robust and are borderline at best in each year before adding sex. The only reason the authors supply for stratifying by sex in the first place is that boys have an autism rate that’s more than four-times the female rate. That’s not even a justification, really, it’s just a background fact and not a preregistered analytical decision, so it makes no sense to have done it in the first place.

In fact, the authors said the finding that for girls there was a marginally significant protective effect was “paradoxical”. How? There’s no paradox here at all, they just used that to dismiss having to report the results for the girls, almost-certainly because they contradicted their anti-vaccine thesis. But though the girls’ results are treated as anomaly, they are just as real and legitimate as the boys’ results and the authors offered quite literally no reason whatsoever for dropping them, and there is no good reason to think they should be dropped. If there is a biological effect of vaccination on autism risk, we have zero reason to think it’s sex-specific.

Conclusion II: G&G’s result is not robust to their sampling choices. The biggest issue is arbitrarily dropping girls, which provided them with their marginally-significant headline results. Correcting this renders their results nonsignificant.

Robustness to Their Sampling Choices

G&G used three controls in their analyses: non-Hispanic White race vs. the rest, two-parent households vs. the rest, and maternal education being high school or higher vs. the rest. They justified these on the basis of prior literature.

For the race control, they said that “black race has been shown to be associated with increased risk for autism” but also that another recent study had found “hat nonwhite children showed decreased risk for autism diagnosis” and yet another had found “Black race [is] associated with later autism diagnosis.” They found that being non-White was associated with roughly 65% lower odds of being diagnosed with autism.

For the two-parent household control, they said that “Absence of one parent was shown to be a risk factor for delay in vaccination” and “decreased risk for autism diagnosis.” They found that two-parent households had about 70% lower odds of autism diagnoses, so evidently the latter confounder wins! (For clarity, I don’t believe this is explanatory.)

For the education control, they said that “Higher maternal education level has been associated with greater likelihood of autism diagnosis.” They found that the higher-educated mothers were about 2.3-times as likely to register a child having a diagnosis.

G&G had literature justifications, but the set of justifications is incredibly sparse, especially given that several of their citations mentioned other possible controls that they had access to! For example, they’re missing birth weight, gestational age, maternal age, parity, healthcare utilization, insurance status, urbanicity, etc. Moreover, some of their cited risk factors now go the opposite way in more modern data: nowadays, Blacks get higher rates of diagnosis, the low-educated do too, and single mothers get their kids diagnosed more often. This is likely because people now see autism diagnoses as a way to obtain access to services, and those groups use said services more often.

G&G even noted that earlier authors had discussed selection bias that they could have addressed, and they didn’t! Quote: “van Damme et al. (2000) suggested that selection bias might play a role because parents of children diagnosed with a medical condition would be more likely to seek medical advice and preventive care for their child.” When we control for healthcare utilization, however, the result is never robust:

When we add in other theoretically-justified controls, without or in addition to our healthcare utilization controls, the result is once again, not robust. Notably, even just switching to a continuous measure of maternal education instead of an inappropriately discretized one (this lowers statistical power) is sufficient to take the result fully nonsignificant.4

Combine these important—as acknowledged by G&G and their citations!—controls and you get a very clearly null result. Remove the inappropriate sex stratification too, and this is just such an obvious nothingburger of a result:

Conclusion III: G&G’s result is not robust to their covariate choices. Moreover, they excluded several covariates they or their citations mentioned as important, while handicapping one of their covariates they did use by inappropriately discretizing it. With any of these, their result would have gone from slightly below the significance threshold to slightly above it. They quite evidently p-hacked their way to their result.

Now They’re Just Being Unkind

At this point in the paper, G&G seem to insult the reader’s intelligence. Though they’re concerned with neonatal males, they start talking about injections that are typically not given to neonates. They write:

Specificity of the exposure was also tested by separately substituting the varicella and measles–mumps–rubella vaccinations for the neonatal hepatitis B vaccination in the model for autism; again, significant associations were absent.

Varicella? Given at 12-15 months. MMR? 12-15 months as well. Hib? Polio? Two months. DTP/DTaP? 2, 4, and 6 months.

Though they portray this test as evidence that the effect is specific to the hepatitis B vaccine, what they are in fact telling you is that they don’t expect you to check their results, because this obviously cannot be significant. The data shows only 11 neonatal male MMR injections, 6 for varicella, 19 for DTP/DTaP and another 19 for polio, and just 48 for Hib. These are more likely to be date entry errors than neonatal vaccination!

There were zero autism cases among this group, for the simple reason that both vaccination and autism are rare. As a result, we must be underpowered, and we must be getting misled by G&G. Or they’re just that daft. Jury’s out.

Even being generous to G&G and assuming that by “substituting”, they are referring to ever receiving the varicella or MMR vaccines at later ages, this provides a completely different exposure window, so it’s not a valid specificity test! Either they checked an empty test or an invalid comparison, but regardless, they duped reviewers. A real specificity test was unavailable to them, because to do it would require another vaccine that’s also routinely given at birth with substantial uptake, and hepatitis B is essentially the only one in the U.S. schedule, which makes a test impossible.

Conclusion IV: G&G’s claimed vaccine specificity tests were impossible to run with their data. And yet, they claimed to run them. They have to be dishonest, daft, or both.

They Don’t Expect You To Check, But They’re Lying

To show that the disease effect is specific to autism, G&G claimed to have run a bunch of other analyses of different conditions.

First, to test for disease specificity of the outcome affected by the exposure, parental report of any one or more of a group of outcomes with no known relation to autism, autoimmunity, or vaccines— i.e., Down’s syndrome, cystic fibrosis, cerebral palsy, congenital or other heart problems—was substituted for autism diagnosis in the multivariate logistic regression model; null effects were found.

The problem here is one that I never would have expected a non-methodologist to catch on even a careful read. It’s either nefarious or stupid:

G&G created a composite outcome that was based on various rare problems to check against for hepatitis B vaccine effects, but said composite is full of empty outcomes because of age-at-diagnosis issues and rarity leading to tiny case counts, so the result is an artificial, constructed null. It being null (OR = 0.88, p = 0.65) isn’t even good for their case, since the result is not significantly different from their headline result (p = 0.054)!

Unless G&G are just really bad at being scientific, what they did was malicious. And it’s not even the correct way to test for specificity by disease! You do not throw together a bunch of random diseases, because there is no biological sense to that. What pathway is expected to elicit effects from vaccination to… a composite of a bunch of random, disconnected diseases? It’s theoretical nonsense.

So, splitting it up, what do we see? Well, we do not see disease specificity. Take a look:

Instead of seeing a lack of association as G&G wrongly claimed, we see associations with some other diseases, and these associations are actually more robust than the male-only autism association. Even worse, the associations are impossible! Cystic fibrosis is fully genetic, caused by mutations in the CFTR gene. It cannot have anything to do with vaccination! For girls, we see a significant—and still, stronger than the male autism association—effect on cerebral palsy, but cerebral palsy is caused by prenatal or neonatal hypoxia, infections, trauma, prematurity, etc.

The idea of these things being caused by vaccination is preposterous. They are far more likely to reflect a bias—like one that would be obtained through increased healthcare utilization as a confounder—or an error, and they are obviously virtually meaningless, since they’re based on so few cases.

Conclusion V: G&G’s claimed disease specificity was misleading in the extreme, and actually testing their claims, we see that the supposed ‘effect’ of hepatitis B vaccination is, in fact, not specific to autism. This analysis was so incompetently done that it had to be an indictment of G&G’s abilities or an indictment of their character.

Let’s Be Serious About Falsification

We can and should go beyond the sorts of outcomes that G&G mentioned to mislead readers. If we do that, we’ll see that the supposed effects of hepatitis B are much more ambiguously bad than they let on.

Consider, for example, that hepatitis B vaccination is related to significantly and substantially and very robustly lower rates of learning disability and ADHD/ADD. It’s also somehow related to very high rates of cystic fibrosis, and—amazingly—to both higher rates of food and digestive allergy and lower rates of hay fever.

None of these results make much sense. Obviously the hepatitis B vaccine doesn’t raise kids’ IQs or help them sit down in their chairs in class. It also doesn’t cause genetic conditions that it’s as robustly associated with as it is with autism, such as muscular dystrophy or sickle cell anemia, and it doesn’t protect against Down’s syndrome, which is detectable before birth. And the idea of it both causing and reducing rates of allergy suggests a degree of biological specificity that nothing has ever been shown to achieve. And if we add the girls in, we see this:5

Adding in girls—because there was never any reason to exclude them in the first place—we see that the autism finding becomes nonsignificant, while the others that were significant, remain significant, and a few more become significant, but they’re mostly in the direction of vaccination being good (lower rates of asthma and respiratory allergy) or implausible in its effects (cerebral palsy). If G&G were more serious, they’d have written about how hepatitis B vaccination makes kids smarter! Instead, they wrote about one of their less robust findings. Pity.6

Conclusion V: Using G&G’s own methodology paints a murkier picture of hepatitis B vaccination effects. Their claimed link to autism is much more tenuous than the highly robust negative link to learning disability and ADHD. Clearly both things reflect confounding.

Obvious Tests to Run

G&G’s specification isn’t even the most obvious specification to run. They had no theoretical justification for running the tests they did, so why didn’t they run other ones? Presumably, if asked, they could fall back on saying that they were doing some sort of test of the childhood vaccination. However, that would leave them in the awkward position of having done a very poor quality test!

If other plausibly tests support their results, surely their case is helped. If they show something else, then they have explaining to do. So, I ran some dose-response and timing tests, and here’s what I got: bupkes.

Effect Exaggeration

All of these results are based on extremely small samples of kids with autism, like 9 cases here, 13 cases there. The cystic fibrosis result is based on 3 or 5 cases depending on if you just use boys or the whole group. These results are incredibly tenuous, and displacing a single boy from the group with both hepatitis B vaccination and autism to the other group would result in a nonsignificant effect. When your result depends on one kid in a cohort of thousands, consider disbelief, consider recognizing that you have very low statistical power.

One of the funny consequences of low statistical power and p-hacking is that published effects are huge. This is because the effects have to be in order to be significant given how imprecisely they’re estimated. We can directly assess the impact of this effect size exaggeration by projecting how much statistical power we had.

This study obviously has very little statistical power. With 32 autism cases and 9 of them being exposed, this is a very tiny effective sample for a rare outcome. The exposure applies to 18% of the sample, so the comparison is very asymmetric. G&G also searched across subgroups (boys, born <1999 only, neonatal timing only) to land on the one specification that hit their magical p < 0.05 alpha threshold. At 6% power, the true OR consistent with this design is about 1.23, and the observed OR is exaggerated 2.2-times over. At 8%, the true OR is 1.32, or 2-times exaggeration.

If the true OR was 1.05, they had 5% power to find it (exaggeration: 24x). If it was 1.20, they’d have had 7% power (7x). If it was a 50% increase in autism risk, they’d have had 13% power (3x). Even a doubling in autism risk is just 28% power (2x). Obviously this was an underpowered exercise and the p-hacking evinced above means that whatever we wound up with is a severely incorrect and unrealistic effect size.

Is Thimerosal Even Associated With Autism?

The stated motivation for testing the association between the hepatitis B vaccine given in a specific set of years and autism is that the vaccines once contained thimerosal. Of course, they no longer do, but autism has kept on rising with no detectable changes related to the phaseout of this ingredient. This makes sense, because heavy metals do not play a causal role in autism, and the amount of mercury in vaccine preparations was always a vanishingly small amount that the body reliably purges right away. So, is thimerosal vaccination associated with autism anywhere? No.

Wrap-Up

Gallagher and Goodman’s paper is currently under investigation by the venue that published it. That’s with good reason, because the paper was obviously p-hacked, it’s filled with what is at best evidence of incompetence and at worst evidence of fraud, and it’s caused harm by misleading people about vaccines that allow us to keep down the prevalence of a horrible, preventable disease. Let’s contribute to that investigation with some points:

• G&G’s result is fragile. If one kid swaps groups, the result becomes nonsignificant

• G&G’s analysis isn’t causally informative in the first place

• G&G’s specification is cherrypicked

• G&G’s result does not survive multiple comparisons/multiplicity correction

• G&G claimed to run tests they either did not, or which they misreported

• G&G ran tests that made no sense

• G&G’s study was obviously underpowered, leading to an inflated effect size

• G&G’s own methodology shows that hepatitis B vaccination is actually beneficial

• G&G’s theory of their paper has been disproven time and again

All of this should’ve been caught in peer review but wasn’t. What are we doing here?

The whole anti-vaccine movement was borne out of fraud. The Wakefield Lancet Controversy was a scam to demonize one type of vaccine to sell more of another, and even though the fraud was obvious and the results were never plausible, the movement has not died. In fact, the movement is stronger than ever, and it’s presently ruining the health of thousands of American children who it’s foisting measles and other preventable diseases on. Ironically, anti-vaxxers often talk about other people doing fraud in support of vaccines; thankfully, the childhood vaccine schedule has never relied on fraud and there is no way to make the case it has with any evidence.

Replication Files, Analytic Extensions, and Commentary

Read more

I’ve read the entire literature on the Flynn effect. I’ve reanalyzed all there is to reanalyze, conducted novel analyses, reproduced and replicated results, and helped researchers with papers on the subject. My understanding of the Flynn effect is this:

Increases in scores across cohorts (so-called ‘Flynn effects’) and decreases in scores across cohorts (so-called ‘anti-Flynn effects’ and ‘Woodley effects’ alike) are attributable to psychometric bias: measurement non-invariance that, when corrected, tends to moot or at least substantially reduce the scale of temporal trends. The psychometric location of the Flynn effect is primarily on those subtests with lower g-loadings, and there is usually a g-unrelated affinity for certain skills and group factors. The Flynn effect is primarily about test-taking sophistication and norm obsolescence.

This understanding is an accurate summation of the whole literature that I would challenge anyone to disagree with on empirical grounds. On that note, I know exactly how it could be challenged, and I want to explain why such a challenge is invalid.

There are a handful of studies that are allegedly studies of the Flynn effect (and some that are adjacent to it) which purport to show real, measurement invariant test score decreases and increases across various times and places. While most Flynn effect studies are driven by measurement bias, these other studies sometimes actually aren’t. These studies are instead driven by sampling bias.

An exemplary study that showed a positive Flynn effect is Ang, Rodgers and Wänström’s one in which they used the Children of the National Longitudinal Survey of Youth (CNLSY) cohort to showcase the Flynn effect in the PIAT Mathematics test. What they found was that five, six, seven, eight, nine, ten, eleven, twelve, and thirteen-year-olds’ scores trended up in each year of the survey data. The extent of these trends was similar by sex, race, and urbanicity, but children from more educated and higher-earning households showed a stronger Flynn effect—that is, greater score increases in successive years of the data.

So, in short, what they did is they looked at all the people who were a given age in a given year and, in later years, people of said ages tended to have higher scores.

Do you see the problem here?

If you don’t, that’s perfectly alright. None of the peer reviewers or editors at Intelligence or any other journal publishing similar analyses noticed the problem either. Once I say it, it’ll become a glaring problem and you might even notice it in other papers. I’ll give you another hint before coming out and telling you. Take a look at this chart that uses data from the same sample:

Children born later in the CNLSY sample had mothers who were (1) better educated; (2) smarter; (3) more likely to be White as opposed to Black. Ang, Rodgers and Wänström’s results are clearly confounded: smarter kids are born later because smarter people tend to have kids later! Fair to Ang, Rodgers and Wänström, they recognized this possibility and controlled for measures of mothers’ intelligence. But, they did not have measures of fathers’ intelligence and their measures have some degree of unreliability, so the control is likely not enough to remove all the residual confounding.

If we take the raw score trend in the CNLSY for kids’ PIAT Mathematics, Reading Recognition, Reading Comprehension, and PPVT performance and we plot it, we can clearly replicate the results obtained by Ang, Rodgers and Wänström. If we control for maternal characteristics relevant to kids’ IQs, such as a mother’s education level and her own intelligence, we can reduce the extent of the “Flynn effect”. But, doing that leaves behind a considerable trend. It’s only when we compare siblings in the same family that we see the effect entirely evaporate. In fact, it goes slightly negative!1

In other words, what Ang, Rodgers and Wänström had dubbed a “Flynn effect” was actually just a compositional effect. We just couldn’t see that this was the case until we controlled away all the confounding characteristics that led to later births also being smarter births. The Flynn effect in the CNLSY was much like the effect of breastfeeding, where it also dissipates within families but it won’t fully disappear if you just throw a handful of the most obvious maternal characteristic controls at it. If you already grok that example, this style of chart might be familiar to you:

Here we have two Flynn effects, both conflated wildly in the literature, and yet totally distinct in character. The Flynn effect in the CNLSY is generated by following a cohort of mothers sampled at different points in their fertile window and noticing a compositional within-cohort selection on who reproduces when. The actual (anti-)Flynn effect, as opposed to this compositional effect, is when you have representative snapshots of the population at a given point in time, and the later group scores better (worse).

If you compared two test norming samples 25 years apart, you would theoretically be looking at two random draws from the population separated only by time, and any compositional changes would be real compositional changes, rather than selection into childbirth. But in the CNLSY analysis, it’s not about a sequence of representative samples, it concerns the children of a single birth cohort, and the older mothers just happened to be better off in many ways that led to a selection-driven false Flynn effect.

In my big article on the Flynn effect and its peculiar inconsistencies, I mentioned another sample that was quite like this. Oberleiter et al. claimed to find a large (~5-IQ point), measurement invariant anti-Flynn effect—that is, real declining ability—in Germany over a brief ten year period (2012-22).

The issues with this result were two-fold. First, their result was not, in fact, measurement invariant. Maybe that’s enough to disqualify it entirely; we can’t know without the study’s data. Second, there were changes in the underlying population being covered in their comparisons. Their results were based on comparisons of three large, population-representative cohorts of secondary school students, but in this period, secondary school attendance became much less selective and Germany’s young demographics meaningfully changed in that period, too.

The question I asked when I first discussed Oberleiter et al.’s study was whether it was more believable that German kids became one-third of a standard deviation dumber in ten years or that there were sampling issues like we just saw above in the CNLSY. I lean towards thinking that the explanation is more likely the latter.

That study has precedent, too. An earlier study from Germany suggested that a reform that compressed the number of years kids were in school but otherwise maintained the curriculum made German kids a whopping nine IQ points dumber basically overnight. But, this was due to selection. As I showed in my article on it, studies that could account for selection into the sample showed an entirely different result: precise null effects and potentially even small benefits. (My article also covered other negative, sampling-driven results in other countries, like Denmark and the United States.)

Yes, This Matters

There are surprisingly many papers out there that conflate ‘the Flynn effect’, as in the psychometric bias-driven change in scores between cohorts, with ‘the Flynn effect’, as in the sampling bias-driven change in scores between select cohorts. These select cohorts can be like subsequent generations of university students—who have become more population-representative as attendance has increased and have accordingly become less elite—or the kids of a birth cohort—where brighter people delay fertility so the kids born later tend to be brighter.

I really wish this were not the case since it misleads people into thinking that there are oftentimes huge and fast population-level changes that just are not so. Alas, that is the sort of thing the public wants: they desire big, flashy results to think deeply about. I’ll repeat my conclusion from my earlier article on the other type of Flynn effect:

People want to think that population intelligence shifts up and down and that people become smarter and dumber all the time, but the reality is that the population practically hasn’t changed any more than you’d predict from shifts in demographics. People want explanations, and they often think they have them, even though most common explanations [for the Flynn effect]—education, technology exposure, family size, hybrid vigor, blood lead levels, genomic imprinting, pathogen levels, nutrition, IQ variability, social multipliers, etc.—cannot be compatible with [its] boring and academic psychometric reality.

1

I posted a version of this chart on X/Twitter earlier. In that chart, the within-family result is slightly different because I included controls for birth rank and a superfluous model specification that also included a likely-invalid control in the form of income. Both birth rank and income are invalid due to collinearity.

Within families, birth rank and birth year were far too collinear to separate (r = 0.84). Including birth rank as a control in the within-family model absorbs some 70% of the within-family birth year variance, leaving the Flynn effect identified off of just 8.6% of the original variation. This made for incredibly fragile estimates that were certainly not robust. Without birth rank, the effect is -0.07 SDs (p = 0.26 with OLS and 0.35 with HC5 robust standard errors) and with birth rank, it’s +0.21 SDs (p = 0.047 with OLS and 0.088 with HC5 SEs). If we really want to account for birth order effects, it would be wiser to use an external calibration. If we used the numbers from, e.g., Black et al.’s 20050 Norwegian register data analysis, we would still end up with a nonsignificant result though. And if we went further and did additional robustness tests, the effect would also not stand up to scrutiny. (I’ll get to these below.)

Adding family income to the unadjusted analysis as a control flips the sign to -0.24 SDs and it’s significant, which is an obvious overcorrection. The problem here is that income is mechanically confounded with birth year in the CNLSY. Because the mothers occupy a narrow birth cohort (1957-64), a child’s birth year is almost perfectly determined by maternal age at birth. Income follows a life-cycle pattern where it increases with age through the years that are typically a woman’s full child-bearing years, meaning that later-born children are observed when their mothers are older and typically earning more. This is decidedly not a real intelligence confounder, but rather, it’s the mothers’ age-earnings profile masquerading as a birth year effect and invalidating our birth year-based analysis. Unfortunately—and I checked even though I knew it wouldn’t work—inflation adjustment doesn’t help here because the confound is about the shape of an individuals’ lifetime age-earnings curve rather than nominal price level issues.

The income example is clear, but the issue with birth rank might still feel tougher, because with a very large sample, this shouldn’t be an issue. Unfortunately, we just don’t have a sample large enough. In fact, if we use birth rank, our power is too low to be acceptable to detect reasonably-sized effects. Look at this chart:

By dropping the birth rank control, we become powered (80%) to detect a Flynn effect that’s roughly one-quarter the size of the typical 3-points-per-decade (~0.75 points) benchmark people tend to cite approvingly. With the birth rank control, we have enough power to detect an effect roughly twice that size. If we recognize the simple fact that birth order effects are (1) not sufficiently large to make much of a different and they’re (2) especially noisy at the ages being looked at here, then we realize there’s even less reason to worry.

Now, we should also talk about robustness. In my analysis presented above, I dealt with complete cases for all four of the big ‘IQ tests’ in the battery and I noted the HC5 robust results. If we instead average available cases with three or more tests for a nice, standardized, per-person score composed of as much as we can get, then we gain about 1,000 people (n = 9,009) and the raw result goes to +0.64 SDs, reduces to a similar degree with controls, and becomes -0.02 SDs (p = 0.64) within families. If we use multiple imputation (m = 20), we can get the sample size up to 9,070, achieve the baseline result of +0.65 SDs versus +0.29 SDs with maternal education and g controlled and -0.01 SDs (p = 0.86) within families.

With multiple imputation, the number of test instances available per-person is large enough that a panel model can be used and can meaningfully reduce the size of our standard errors. Thus, we get a maximally large, maximally precise result. The way I do this is multi-step:

1. Demean Y and X at the mother level: composite_dm = composite - mother_mean(composite), birth_year_dm = birth_year - mother(mean(birth_year).

2. Regress: composite_dm ~ birth_year_dm - 1 (intercept-free due to demeaning).

3. Cluster SEs at the child level (i.e., vcovCL(…, cluster = child_id)), since multiple test occasions from the same child are not independent.

4. Apply a degrees-of-freedom correction: sqrt((n-1)/(n-n_families-1))

With the average of 1.57 test occasions per child available with complete cases, the clustering at the child level that’s required to make this work leads to somewhat larger SEs than just using complete cases without attempting to use panel data. With the 3.24 obtained with multiple imputation (harmonic mean k = 2.70), we gain a lot of data, largely for the PPVT, which over half of the kids were missing at some point. This shrinks SEs by ~4% and our final result is +0.01 SDs (SE = 0.00153/year, p = 0.77).

Since Ang, Rodgers and Wänström analyzed only the PIAT Mathematics subtest, I thought I would replicate with their result with single observations, no birth rank control, etc., just to match what they did. In the PIAT Mathematics, the raw trend is +0.83 SDs versus a within-family +0.06 SDs (p = 0.43). For Reading Recognition, the raw trend is +0.71 SDs versus +0.02 SDs within families (p = 0.73). For Reading Comprehension, it’s +0.50 SDs versus -0.15 SDs within families (p = 0.03), which is significant, but is not robust to the methods described above (e.g., HC5 SEs p = 0.053, and other methods lower this more). Finally, for the PPVT, the trend is +0.94 SDs at baseline and -0.15 SDs within families (p = 0.07, and 0.112 with HC5 SEs).

In 2018, Bratsberg and Rogeberg observed that the Flynn effect and its reversal in Norway could be entirely captured within families. I attempted to replicate their result with this data, but due to my smaller sample, my options were more limited. Nevertheless, I used a GAM to check nonlinearity in a manner that’s at least theoretically comparable (in the sense of letting us visualize reversals if they’re present). The test meant taking all sibling pairs (i.e., those with the same mother; yes, I know this is not the same as 50% relatedness, but the results are robust to that, so don’t quibble), taking the residuals, and fitting a penalized spline to those as a function of birth year. By just using the sibling subsample without imputation (result is robust to that), this cuts the sample size a bit, but not meaningfully. Here’s the result:

In other words, I could not replicate their result, but should that have been possible in the first place? It’s possible the trends in Norway and the U.S. in each cohort’s timeframes were not psychometrically alike. Alas, without their data, who can really say?

One other thing I checked was whether the PPVT-R to PPVT-III transition around 2004 affected anything, In principle, that could’ve created an artificial discontinuity in scores if the versions weren’t on the same scale. I don’t think this is an issue for five reasons.

First, there’s no version flag, and I assume the BLS—who maintains the dataset—handled the transition credibly internally. Second, there’s no score discontinuity in 20040. Residualized PPVT scores drift smoothly across the transition, going up by +2.1 in 2002, +2.9 in 2004, and +4.3 in 2006. No jump at 2004 as you might expect. Third, raw score changes are consistent across 2002-2006 and the 19% of children who first enter the sample in 2004 track the pre-2004 trend rather than jumping to some other level. Four, there’s not extra non-invariance associated with the year of the change. And finally, I checked!

As noted above, the PPVT all-waves result was -0.15 SDs with HC5 p = 0.11. Pre-2004, it’s -0.24 SDs with HC5 p = 0.004, so it does get stronger, but it’s still quite different from the raw result, the difference from which is what I was concerned about. A 4-test composite with PPVT pre-2004 yields -0.28 SDs with HC5 p = 0.0007, and a 3-test PIAT composite yields -0.01 SDs with HC5 p = 0.88. With multiple imputation and child clustering as above, the all-waves result goes to -0.20 SDs (p = 0.016), the pre-2004 result goes to -0.31 SDs, compared to the 4-test PPVT pre-2004 result becoming -0.33 SDs, and the 3-test PIAT becoming +0.01 SDs, with significance and non-significance maintained.

As you can see with the GAM results, there’s a dip in the within-family results that goes away. I’m not sure why. This also replicates in quartile-binned results:

It doesn’t matter, but I would like to know why this is, and though we can’t credibly test it, I think the answer might be birth order, which we dropped for reasons explained above. When we add it back, the PPVT pre-2004 -0.24 SD result goes to +0.07 SDs (p = 0.65) and it’s similar for the other specifications. This might be due to birth order effects being larger for verbal/vocabulary measures than on achievement tests, as is often reported in the literature. The PPVT is a receptive vocabulary test, after all. But as noted above, for reasons of power, this is hard to look into, and very likely to be beyond our means with the present sample.

As a final note, the cross-sectional CNLSY results are more extreme in a latent variable model, and the differences between kids born in different birth year quartiles or just later and earlier years were mostly measurement invariant. But, factor score-based results were still nil within families. Measurement invariance was achievable, because nothing really changed.

Sex differences in intelligence are of perennial interest to large swathes of the reading public. Being fair to practically everyone, I’ll concede that they are a potentially interesting topic, and if there are differences, they might help us make sense of the world: If men are smarter than women, that can help to explain why there are more men in positions of eminence; if women are smarter than men, that can force us to confront a reality where culture has somehow suppressed the contributions of half of the population. Getting to the bottom of this could be important!

In a recent article on the topic (you should also check out my earlier article on sex differences in knowledge), I critiqued a paper that claimed to show women were smarter, but which used an invalid method to do so. In short, they awarded women extra points for being more risk-averse and concluded that was an underrated sign of intelligence. Clearly absurd. Whatever the case, in that article, I also described what would be required to actually test which sex was smarter. The ideal study features:

• Excellent sampling. In many cohorts, the intellectually disabled, troublemakers, goofs, and drop-outs tend to be excluded, and this tends to artificially boost male performance. We need a cohort that’s not impacted by sampling problems.

• Latent variable modeling. To provide a valid group comparison, you need to achieve measurement invariance and to ensure that you’re measuring the right thing—that is, intelligence, g, general intellectual ability as distinct from specific skills that might be more common in one or the other sex because of mere content exposure.

• A quality test battery. Latent variable modeling can’t overcome a bad or sex-biased set of tests. We need good tests with diverse content, or the tests might favor one or the other sex by construction.

The closest study that I’ve found to actually overcoming this issue is Deary et al.’s 2007 analysis of the National Longitudinal Survey of Youth (1979; NLSY ‘79) sample. This study features a high-quality test battery with ten tests covering diverse content domains, given to a large, nationally-representative sample, and analyzed in an admittedly crude way with a latent variable model, comparing opposite-sex siblings. This comes close to overcoming every issue with typical sex difference studies, and their result was interesting:

Males show a very small (Cohen's d = 0.064) but significant advantage on the g factor extracted from the AFQT. Males score significantly higher on the g factor from the ASVAB, though the effect size is again very small (Cohen's d = 0.068). The strongest finding is for significantly greater variance in male scores. The standard deviations of the g factors from the ASVAB and the AFQT have male:female standard deviation ratios of 1.16 and 1.11, respectively.

This result is strong and important, and it is a shame that it hasn’t been replicated. So, with that said, I will do two things: I’ll reproduce and extend the result in the NLSY ‘79 and I’ll replicate it in the Children of the NLSY (CNLSY) and the NLSY ‘97 cohorts. With that said and done, it’ll be possible to finally make a pronouncement on that always-interesting question: Who’s smarter, men or women?

The Data, The Models

All of our raw data is publicly-available and comes from the Bureau of Labor Statistics. It can be accessed through the NLSInfo website. All of the data linking together participants by household and indicating their levels of relatedness (e.g., full-siblings, half-siblings, cousins, etc.) can be accessed through the publicly-available NLSYLinks R package.

The models we’re using are theoretically-motivated. They’re based on published analyses of the tests batteries used in these studies, and they’re supplemented by noting and observing the theoretical coherence of different factors in the analyses. For the NLSY ‘79 and ‘97 cohorts, we’ll be using this hierarchical model that features a general factor, g, and four group factors that have to do with speed, mathematics, verbal ability, and technical knowledge. By using a higher-order g, we obviate psychometric issues with oversampling particular abilities like spatial or reading.

For the CNLSY, we have fewer subtests, and so we’re forced to fit a single-factor model with a set of tests that arguably favors girls because of its greater focus on one of the abilities that girls tend to excel in net of g: reading. This could confound both mean estimation and variance estimation, so in order to get the most coherent results, we will endeavor to estimate the variance difference where the mean difference is zero.

For our sampling, we’ll match the prior literature and use published schemes, but in the section below entitled Robustness, I will match all the samples on sampling schemes, and I’ll be presenting several results with different schemes anyway, for a glimpse into how strong these results are. By cohort:

• NLSY ‘79:

  • All 10 ASVAB subtest results present

  • Age-residualized on birth year with a quadratic polynomial (age range = 16-23)

  • All male-female sibling pairs within households or relatedness, R = 0.50 (“Gen1Housemates”)

  • Deduplicated (each person once in the CFA)

  • MLR estimator with FIML imputation (effectively listwise)

• NLSY ‘97:

  • All 10 ASVAB subtests present

  • Age-residualized on birth year (quadratic; age range = 13-17)

  • R = 0.50 pairs only, since the NLSY ‘97 lacks a household grouping variable comparable to the ‘79 one

  • Deduplicated

  • MLR with FIML

• CNLSY:

  • At least 3 of 5 tests

  • At least one observation per child

  • Age-residualized on interviewed age (CSAGE; quadratic; age range = 5-18)

  • All male-female pairs within MPUBID (a household variable) or R = 0.50 (“Gen2Siblings”)

  • Deduplicated

  • MLR with FIML (68% complete cases and FIML handles the rest)

Do Our Models Fit?

Yes! Our latent variable models are almost fully invariant, meaning that males and females are comparable and that the factors influencing their performance are shared between sexes.1 The only areas in which there are deviations from full strict factorial invariance have to do with the intercepts, and specifically for the NLSY ‘79’s General Science and Auto & Shop subtests and the NLSY ‘97’s Paragraph Comprehension subtest.

Partially invariant models freeing these parameters leave more than enough constrained parameters to measure latent g credibly across the sexes, and they at least make some theoretical sense. For example, for General Science and Auto & Shop, boys might have been more interested in the subjects, leading to additional learning about them and raised intercepts as a result. That means that, net of measured ability, boys did better (roughly 0.25 d better) in those specific subtests.

Curiously—and this makes less theoretical sense—boys also did better than expected in Paragraph Comprehension and they did so by a similar degree to their boost for General Science and Auto & Shop. This means that even though girls did better in this subtest, they would’ve done even better without bias. In fact, the gap would’ve roughly doubled in size in girls’ favor, but only for the NLSY ‘97 and not the ‘79.

With our modeling efforts being such a success, we can now review our actual results!

So, Who’s Smarter?

The question of who’s smarter has to do with differences in g, not specific abilities. This question is directly testable within our latent variable models. All we have to do to test it is constrain the means to equality and see if the model fits better or worse. To get to the means, we have to start by first testing if the variances are equivalent, and as it turns out, they are not.

In each dataset, male g is much more variable, as it is in most datasets out there. Importantly, this means that the observation of greater male variance is not attributable to sampling.2 In fact, because sampling issues tend to cut off the left tail, they should generally reduce the male-female variance difference. Consistent with that, the generally-observed variance ratio of 1.16 is smaller than these. If you look on the right-hand side of this plot, you’ll see far larger variance ratios: 1.49, 1.21, and 1.37!

If you look on the left-hand side of the plot, you’ll see the other interesting detail: virtually zero difference in g! The mean differences in the NLSY ‘79, the NSLY ‘97, and the CNLSY are between 0.32 IQ points favoring girls and 0.33 IQ points favoring boys. Using the extreme tips of the 95% CIs doesn’t make things much different (-1.35 to 1.40). In other words, the difference is minuscule!

Let’s make a strong statement:

S1: Sex differences in intelligence have to do with the variances, rather than the means.

Some people prefer to conceptualize intelligence differently. They are wrong to want to do so and they are concerned with considerably more minor constructs, but we can still help to satisfy them by looking at the differences in the group factors in the hierarchical models from the NLSY ‘79 and the NLSY ‘97. Take a look at the gaps:

The mean differences in Processing Speed and Technical Knowledge were consistent between cohorts. For Mathematical Reasoning and Verbal Comprehension, they were not, and now the male-favoring difference in Mathematical Reasoning is slightly reversed.3 But how much does this matter? Not a lot. These group factors explain very little performance. You can see this very clearly from the implausibly high variance ratios for these different factors, which follow naturally from the limited variance.

Let’s make another strong statement:

S2: Sex differences in cognitive ability have less to do with intelligence and more to do with specific abilities. These appear to be more malleable than intelligence.4

Let’s also look at this at the subtest level so we can get a better idea two things: (1) of the impact of specific skill differences between the sexes, and (2) of observed mean differences on observed variance ratio differences. We’ll do the latter first.

Simple averaging of the VR₀ figures across batteries delivers a male:female variance ratio of 1.182 when there are no sex differences.

Now, let’s look at differences in the subtests after we take out g and the group factors. What we see is that there’s not a whole lot left over5 but sometimes there’s something, implying that there are average sex differences—not just individual differences—in specific test-relevant skills.

Let’s make yet another strong statement:

S3: There are male-female differences in test performance that go beyond measured intelligence and broad ability factors, having more to do with skills that are specific to particular tests.

Finally, let’s look at the development of male-female differences.

One thing you might’ve noticed is that the variance ratio VR₀ in the CNLSY sample is a lot smaller than the ones concerning developed adults from the NLSY ‘79 and ‘97. The age range for the CNLSY is 5-18, so it concerns a lot of the time when boys and girls are still developing, starting and then completing puberty. This could theoretically impact sex differences in both the means and the variances. Luckily, the CNLSY is a longitudinal cohort, so we can use its data to assess what’s going on.

First, let’s look at the CNLSY cross-sectionally. For the sake of power, I’ve binned the results instead of attempting to get continuous results, because there are sometimes not enough people at later ages to get anything even remotely certain. To get more power than I would have just using full-siblings, I’ll also look at the results for any other kids in the house, which includes things like co-resident cousins and half-siblings. The results are interesting. Take a look:

For the mean, it looks like differences might decline with age as boys enter puberty. We know that by early adulthood—as indexed by the older NLSY ‘79 and ‘97 samples—the gap is gone, so this suggestive result here looks to be confirmed. But to really make this certain, we need to look at the same individuals over time in a longitudinal analysis. We can do that with the CNLSY data, and the result looks highly similar:

These results support an early female advantage that fades into nothingness alongside some consistent, possibly increasing level of greater male variability, but they aren’t very statistically certain yet. Nevertheless, let’s make a final, albeit weak statement:

W1: There are developmental trends in the male-female differences in g which may lead to the dissipation of early female advantages as people mature into adulthood.

Post-publication addition: A reader asked me to look into the development of the differences cross-sectionally in each cohort, so I’ve produced an updated plot doing just that. You can see it at this footnote.6

Sex Differences in Intelligence: Broad Strokes

There are often sex differences in performance on IQ tests. The reasons for those differences do not appear to be due to differences between the sexes in intelligence, but instead, to differences in lower-level abilities—like specific mathematical, verbal, and spatial ability, speed, and so on—and differences in skills specific to certain tests. Differences also have to do with males being more variable at all levels: g, s, residuals.

This much seems to be certain. These findings are confirmed thrice over in nationally-representative data with high-quality tests and large samples, with results spanning multiple generations. They are substantially robust to different methodological choices, sample inclusion criteria, adjustments for age, and so on (see below).

To my mind, the questions that remain about sex differences in IQ are much less generally interesting than the ones that have been answered. Many people are fascinated by big potential differences between races, sexes, countries, ages, and more, but they’re much less likely to have their interest piqued by questions about developmental minutiae. So, hopefully I’ve sated their interest and they can move on. Hopefully, the rest of us can move on to deeper questions than top-level ones about the mere existence of differences in means and variances and their general locations.

Hopefully.

Subscriber Sections:

• Robustness

• Sex and Aging

• Brain Size and the Lack of an Intelligence Gap

• Other Unanswered Questions, Directions For Future Research

• Sibling Interaction

Robustness

Let’s talk about how our specifications affect our results and how well things hold up.

Read more

All the nations of the world have come together to compete to see who has the greatest athletes. It’s time for the Olympics.

I will assume that every reader is familiar with what the Olympics are, and I’ll endeavor to explain something interesting about them: host advantage. As it turns out, the Olympics do not seem entirely fair; hosting countries have notable advantages: they win more medals and take home a higher share of the prestigious golds! Why this is has always been mysterious. Visually, the host advantage looks like:1

The Olympic host advantage is large, but it substantially fades out. There’s some room to think that hosting countries might invest more effort into finding and training athletes, but that’s unlikely to be all there is to this, since they don’t perform as well in subsequent games. We’ll have to dig deeper.

To start, we’ll look at the difference-in-differences effect of hosting on winning medals:

The effect is large, it is statistically significant, and it significantly fades, too. The effect is also robust to alternative specifications2 and it affects both total and gold medals.3 Does it hold when controlling for (World Bank 2015 US$) GDP Per Capita and population size?4 Yes! So those variables on their own are not what explains the host effect either:5

So far, we’ve narrowed down the possibilities somewhat, but we still haven’t reached any interesting answers as to what the host advantage really reflects. This next test does get us closer: the host effect is relatively larger for smaller countries.

Why might small countries gain more from hosting the Olympics? The answer isn’t immediately obvious until you consider that the host gets to bring more athletes. Whoever is hosting will tend to have a big boost to their delegation size because their athletes will automatically qualify for many events that they might not have been able to submit athletes for had they not been the host.

Consistent with this, hosting results in a huge bump to the number of athletes a country sends and then afterwards, they return to a more normal, but slightly elevated number in subsequent games.

With just these variables in mind, we can decompose the host advantage6 and see that delegation size actually explains more than 100% of the host bonus! What this suggests is that hosting countries send a lot more athletes, but said athletes may not be as high-quality as the athletes they’d normally send, so they end up less successful on average, while bringing home more medals because they’re more numerous. Although, this residual is not significant, so let’s not make too much of it.

So, is that it? Possibly, but I think we can go deeper. For one, we need to address other possibilities. For example, what about jet lag and climactic effects? To proxy both, we can just look at the distance to the host city that sending countries are and see how the medal counts and per capita medal counts are. As it turns out, distance isn’t meaningful in any case: overall (shown) or as latitude and longitude (not shown).

There’s also not some sort of distance-related cultural spillover effect, where nearby countries who host draw medals from countries that near them that might practice in similar sets of sports.

The real kicker for the Olympic host effect has to do with judging. The host, as it turns out, exerts considerable influence over two things that impact judging: for one, the judges themselves! For two, the crowds, as hosting countries get more seats for their countrymen at the events, and this might matter because there’s evidence home field advantages have to do with influencing people like referees.

When we look at the Olympic host bonus, after we control for delegation size, there is no bonus. However, there is a significant interaction by medal type: the host may not gain significantly in terms of total medals beyond delegation effects,7 but they still earn more gold medals. That effect is only significant for events with subjective judging criteria:

A final concern could be that the host advantage reflects that hosts can select to add new events, and they’d be likely to select new events that favor them. Fortunately, this is only recent and we know which events have been added, and that they benefitted Japan and France, but not by enough to significantly alter any of our results so far. But while we’re on the subject of Japan, let’s focus on it a little more.

When Japan recently hosted the Olympics, COVID precautions made it so there were virtually no crowds. They had a normal-sized overall host advantage (p = 0.94 for total medals and p = 0.38 with a delegation size control), but no crowds. Given the lack of a crowd, the crowd size influence channel on judging should have been eliminated. We can thus leverage this natural experiment to figure out if it’s the crowds swaying judges or if it’s the judges being biased in home countries for some other reason.

As it turns out, the answer has nothing to do with crowd size. The subjective-only boost Japan received—even net of its choice to add skateboarding and karate as events—was completely normal!8 Japan's delegation-residualized judged-sport bonus at the fanless Tokyo Games was statistically indistinguishable from the typical host's (p = 0.59), and the overall judged-sport residual remains highly significant (p < 0.001) even with Tokyo excluded.9 Whatever mechanism drives the subjective scoring advantage, it operates with or without a home-based crowd.

Why Do Hosts Do Better?

Let’s go through things that we now know do not explain why Olympic hosts do better:

• Being wealthier

• Coming from bigger countries

• Spillovers with neighbors

• Being able to propose events10

• Jet lag

• Climate familiarity

• Higher athlete quality via additional funding, training, etc.

• Favorable crowds11

And now let’s talk about the pathways that really matter:

• Being able to send more delegates is the primary mediator of the host advantage

• Being able to influence judges is the primary mediator of residual host advantage in earning gold medals

The reason Olympic hosts earn more medals at the Olympics is overwhelmingly because they send more athletes. The reason they earn more gold medals above and beyond sending more athletes is because they influence judges.12

2

Alternative specifications listed below:

We can also subject the effect to a permutation test and see that it holds up, but this isn’t saying much since that’s practically just repeating the p-value:

3

Here’s another visualization method:

5

This is also robust across different specifications:

7

You can see this illustrated like so, and also take note: the extra delegation size effect is not just mediated by hosting countries being more likely to take part in sports involving large teams, which tend to have fewer medals per Olympian. There is a host effect across all competition types. And yes, I am aware there are some subjective team sports, I just didn’t want these figures to have captions that were too bulky.

After more than a century of study, we actually don’t have an answer to the question of which sex tends to be smarter. We know with considerable certainty that men are more variable, but we don’t know if they also tend to be more or less capable. This isn’t for want of trying: lots of researchers have looked into this question and come to varying conclusions on the matter! A new Journal of Political Economy (JPE) paper by economists Harrison, Ross, and Swarthout (HRS) provides the following answer:

Contrary to received literature, women are more intelligent than men, compete when they should in risky settings, and are more literate.

Such a definitive answer deserves serious consideration. Unfortunately, on further inspection, it turns out that the authors’ conclusions are not warranted by their data and their methods are so poor that they would never be taken seriously by any qualified psychometrician. Their paper provides an example of peer review failing.

How Did They Make Their Conclusions?

HRS were able to reach the conclusion that women are more intelligent than men by redefining intelligence in a way that favors women. They didn’t show that existing measurements were flawed or that their preferred definition of the trait is more consistent with how intelligence is generally understood than the standard definition is. Instead, HRS declared by fiat that they had a better measure and, by that measure, women are more intelligent.

This may sound like ‘the authors were just playing word games’, and that’s because they were! They measured something different from what people typically mean by ‘intelligence’, called it ‘intelligence’, and declared that women are ‘more intelligent’. By the same token, I could say women are taller if I define height as ‘height over weight’ rather than as, well, height. If I did, I hope most people would recognize that I’m just misusing words. Unfortunately, the peer reviewers and editors at the JPE did not.

Their work is less clever than it is annoying. But let’s be overly generous to HRS.

How Exactly Did They Make Their Conclusions?

HRS started from a reasonable enough observation: conventional intelligence tests record only whether a response is correct or incorrect, discarding potentially useful information about respondent confidence. When someone selects answer C on a multiple-choice question, we don’t know if they were certain it was right or if they thought it was slightly more likely than answer B, or even if they were just guessing.

This does seem like a genuine loss of information. Two people may both answer incorrectly, but one might have narrowed down the answers to two plausible options while the other had no idea what was going on. Traditional scoring, where you’re either right or wrong, conflates these failure modes. Similarly, two people might both answer a question correctly, but one might have been certain while the other was just lucky. Traditional scoring makes these into equivalent successes.

HRS proposed that this missing information—calibrated confidence—is itself a component of intelligent behavior. Knowing what you know, and knowing what you don’t know is certainly cognitively valuable; a person who recognizes the limits of their knowledge can seek help, defer judgments, or hedge better, whereas a person who is confidently wrong may produce catastrophe. From their perspective, intelligence measurement should reward not just rightness or wrongness, but calibration: he who reports uncertainty when he’s uncertain and confidence when he’s confident is more intelligent than he whose subjective confidence remains irrelevant.

To capture calibrated confidence, the authors adapted Raven’s Advanced Progressive Matrices (RAPM) to a new response format.

Traditionally, a test-taker looks at a matrix pattern with a missing element and picks among eight options to complete the pattern. If they’re correct, they earn a 1, and if they’re wrong, they earn a 0, with their total score being the count of correct responses. By contrast, in the adaptation, respondents allocate tokens among different responses according to their subjective beliefs about which is correct. If they’re certain response number five is correct, they can place all their tokens on it, and if they’re split between options five and three, they can split their tokens accordingly.

In the adaptation, respondents’ allocations are rewarded using a Quadratic Scoring Rule (QSR)—a proper scoring rule from decision theory—, which incentivizes truthful probability reports under risk neutrality. If a respondent places all their tokens on the correct answer, they earn $2, and if they hedge uniformly, they’re guaranteed $1.13, regardless of the correct answer. The authors call the resulting measure “efficiency” and define it as the fraction of the maximum possible earnings people achieve, thus rewarding accuracy (modal correctness) and calibration (appropriate confidence).

With traditional scoring, men outperform women, but with efficiency scoring, women outperform men. The authors interpreted this as evidence that women are smarter.

What’s the Problem?

Efficiency scoring changes what’s being measured. This is a fundamental issue; it means that the authors’ conclusions are only valid if what they’re measuring is closer to ‘intelligence’ than what the RAPM measures with standard scoring. This cannot be.

Efficiency scoring introduces variance from constructs that are distinct from intelligence. These constructs can be called metacognitive calibration, confidence, and risk preference, but as long as you understand them, the particular names are irrelevant.

The principal finding of intelligence research is the positive manifold: the observation that all valid measures of cognitive performance are positively correlated in the general population. The most common perspective holds that the positive manifold arises because each test loads on a common factor—g—plus test-specific factors. In matrix form, that comes out to:

where \lambda is the vector of g-loadings—regression betas on the general factor—, S contains the specific factor loadings, and \psi are the specific factors. The positive manifold appears because all elements of \lambda are positive, so higher g means higher performance on every test. Contrarily, consider what happens when you replace accuracy scoring with efficiency scoring: you introduce a new component into the measurement model! For simplicity, we’ll just add \kappa, for metacognitive calibration, and \rho, for the inverse of risk aversion. The efficiency score loads on these three latent factors, like so:

Risk-averse individuals (those high on \rho) hedge toward the floor. On easy items, this hurts them because they tend to leave money on the table by not concentrating their tokens on correct answers. On hard items, this helps them because they secure the floor winnings of $1.13 instead of gambling for scraps. This means that \delta_j is negative for easy items and positive for hard items.

Consider what this means for the covariance structure of the items. For two items, j and k, with different difficulties:

If j is easy and k is hard, then \delta_j<0 and \delta_k>0, so \delta_j\delta_k<0. The risk preference component contributes negative covariance between easy and hard items. This works against the positive manifold! It means the authors have introduced a factor that makes performance on easy items negatively correlated with performance on hard items, conditional on g. The empirical consequence of this is that the factor structure of efficiency scores should be messier than the factor structure of accuracy scores. You might see…

• Lower overall loadings on the general factor

• The emergency of spurious difficulty factors

• Negative residual correlations between item pairs at opposite ends of the difficulty spectrum—a violation of the local independence assumption of latent variable models

If Harrison et al. had computed a correlation matrix of item-level efficiency scores and extracted factors, I would predict they’d find worse fit to a unidimensional model than they would find with accuracy scores. Moreover, they would not find something consistent with the key findings of intelligence research, but instead, with a novel understanding of intelligence as a behavioral combination of multiple constructs, including what is typically considered to be intelligence. This is clearly absurd.

What Should They Have Done Instead?

Testing which sex is smarter is straightforward and simple. All you have to do is administer a large, diverse test battery, score the answers as right or wrong, and then fit an appropriate latent variable model if such a model holds between the sexes (meaning that there’s no bias). Then, check which sex has a higher level of g. That’s all!

In the absence of psychometric bias, the authors cannot claim that there is any important measurement issue with typical testing, aside from, say, a given battery being weighted towards content that favors one or the other sex. All they can do is quibble about intelligence being something different from what the field of intelligence research has understood it as. They’re stuck arguing that it’s not one thing that can be measured, but multiple things that come together from different traits, one of which is what the relevant field considers intelligence to be, and the others of which are clearly measured with bias across demographics like sex and race.

On its own, this should be considered devastating: HRS are just playing word games and they’ve failed to justify their conclusions from playing those games. They provided no reasons to think intelligence needs to be redefined, let alone as they’ve chosen to define it. As I’ll discuss below, they don’t have room to play this game at all. Their preferred definition of intelligence is simply unjustifiable—full-stop.

Table of Contents

• Further Issues With the Harrison, Ross and Swarthout Intelligence Study

• Harrison, Ross, and Swarthout’s Competitiveness and Literacy Results

• Why Extant Studies Don’t Move the Needle on the Question of ‘Who’s Smarter?’

• The Ideal Study

• What is Intelligence?

Read more

Table of Contents

• National Results

  • By Race

  • By Sex

• Representative State Results

  • By Race

• Comparing With Previous Years

  • Overall

  • Michigan

• ACT Results

  • By Race

  • By Sex

  • Representative State Results

• America’s Future Elites

• Methods and Bonus Charts

• Have Suggestions?

National Results

Let’s start by looking at SAT results for the whole country. Do note, taking the SAT is selective; few states test every high schooler. At the national level, the gaps are at practically the same levels they’ve been at for decades for the well-identified groups. Most groups have fairly similar variances as well, with the exception of Asians, whose scores are considerably more variable than are the ones for other groups. Maybe this is because high performers have more diverse performance, and/or maybe it’s because “Asian” is a more diverse category than the other ones.

If we look at test scores split by sex at the national level and among Michiganders—who all take the SAT, making the results representative for that state—then we see that men are a bit more variable and they seem to be a bit higher-performing at the national level, but not in Michigan.

Representative State Results

Read more

For three-quarters of my life, from about ages 6-7, I seriously thought that I was allergic to guaifenesin and dextromethorphan. I’d always been told that I was allergic to these drugs and that they cause me to go into anaphylactic shock.

But that was wrong. I’m not allergic at all. In fact, I don’t have any allergies!1

Millions of people are probably in the same boat I was, albeit with different drugs listed for their allergies.2 The most common example of this—but by no means the only one—is penicillin.

Some six to ten percent of people have been labeled allergic to penicillin, but it’s likely that at least ninety to ninety-five percent of these labels are incorrect and the labeled individuals do not actually have a penicillin allergy. Nevertheless, these false labels result “in unnecessary avoidance of β-lactam antibiotics, leading to worse clinical outcomes, increased rates of Clostridioides difficile infections and multidrug-resistant organisms, greater health care costs, and longer hospital stays” for millions of people each year.

‘How does this happen?’, you might ask. People are usually mislabeled through an abundance of caution. Doctors frequently see something minor, like a skin rash following administration of an antibiotic when they see a patient with an infection, and in order to ensure it doesn’t happen again, they just mark the patient as allergic to that antibiotic. This is usually benign, and most doctors don’t give it a second thought. But, as noted above, this is bad for public health, so we really ought to care.

‘But how can we fix this?’, you might wonder. Well, it’s easy. We need to get systematic about delabeling—the process of testing whether people are actually allergic to something like penicillin, guaifenesin, dextromethorphan, pollen, some random drug or various foodstuffs, and so on, and then removing that label from people’s records if it turns out they don’t react.

The testing is very simple and applies to most drug allergies. I’ll describe my case.

After looking at how uncommon the allergies I thought I had were, I took to looking up the details of my case. In mine, there was nothing that would positively differentiate guaifenesin from dextromethorphan, and both were extremely rare allergies, that have no reason to occur alongside one another. Therefore, it was unlikely that I had both allergies. So, I grabbed a handy-dandy epinephrine pen (just in case) and sat down for a supervised ‘direct oral challenge’: I took a pill containing just guaifenesin, at the highest available over-the-counter dose, recorded the time, and sat, for two hours, monitoring symptoms the whole time.

After two hours, I had no hives, no angioedema, no wheeze, no cough, no hypotension… nothing. By two hours, such a lack of reaction makes it extremely unlikely that I’m truly allergic, and virtually impossible that I have anything like a severe, life-threatening allergy. So next, I repeated this on my lonesome, and again, no reaction. Therefore, I concluded, I do not have an allergy. Whatever resulted in that label being added to my file, it was in error.3

This finding weakly lowered the prior odds that I had the other allergy on biological grounds and it moderately reduced the odds on case-history grounds. So, the next day, I did another supervised direct oral test, this time with dextromethorphan, and two hours later, nothing! Then, I took more on my lonesome and, once again, nothing. I’m not really allergic to either drug!

If people did tests like these, they could get delabeled for most of their medication allergies, and they could benefit from being able to use those medications as a result. The benefits of this wouldn’t just be personal, they’d be social, as less effective care for things like infectious diseases means more spread and longer periods of suffering. If we get people delabeled, we can waste less money on poor-performing medicines, reduce human suffering, and more effectively curb the risk of many common diseases.

Consider this result for the commonly prescribed antibiotic penicillin: about half of the people flagged as having penicillin allergies in their electronic health records were classified as high-risk, but only one person looked high-risk on allergists’ review, and just one actually tested positive for their allergy!

Penicillin allergies (like so many others): commonly labeled, uncommonly verified. If there’s an annoying allergy with a dubious origin in your medical file, consider going to an allergist and taking a supervised test.