Today’s post is brought to you by my sponsor, Mechanize. They’re hiring junior software engineers at $300K/year base salary.
In 1974, physicist Richard Feynman gave a commencement speech at CalTech entitled “Cargo Cult Science”. The title of the speech came from the cargo cults of the south Pacific, the belief system that arose among Melanesian islanders that they could make plane-shaped entities carrying supplies come down and enrich them by building things that looked like planes and runways and by marching in straight lines with sticks that looked like guns. After all, they had seen American airmen do just that and the airmen were rewarded.
Feynman took this notion and observed that something much like it happens in academia today: researchers regularly find themselves in their own cargo cults, doing rituals, hoping for boons, and achieving little for much the same reasons that the real cargo cults achieved little. I won’t recapitulate the whole speech, but I will note what I view as its most important bit: a relatively simple rat experiment that expanded into a testing battery so expansive no one had done anything like it at the time and no one has done anything like it since.
The first principle is that you must not fool yourself—and you are the easiest person to fool.
Feynman writes:
[T]here have been many experiments running rats through all kinds of mazes, and so on—with little clear result. But in 1937 a man named Young did a very interesting one. He had a long corridor with doors all along one side where the rats came in, and doors along the other side where the food was. He wanted to see if he could train the rats to go in at the third door down from wherever he started them off. No. The rats went immediately to the door where the food had been the time before.
The question was, how did the rats know, because the corridor was so beautifully built and so uniform, that this was the same door as before?
Can you see what professor Young is doing? The notion is simple: he’s trying to figure out why he got the result he did. Getting the result alone is insufficient to know why it happened. So, professor Young must do more experimentation to eliminate possibilities. He starts doing that by painting the doors to mess with the texture, but the rats can still tell where the food is. Then, he plays around with the smells, but the rats can still tell. Next, he realizes there might be something up with the lights in the lab, so he addresses that possibility, and still, the rats can tell.
Professor Young goes through possibility after possibility and eventually figures out that the way the rats are finding the food is through vibrations. So, he designs an experimental setup in which the maze is on top of some sand and thus the noises are blunted and the only way for the rats to learn the right door is to actually learn it. But, without going through each and every possibility, professor Young could’ve made a much more naïve conclusion; to really start learning about how rats learn, he had to run experiment after experiment to even make it possible. But then:
I looked into the subsequent history of this research. The subsequent experiment, and the one after that, never referred to Mr. Young. They never used any of his criteria of putting the corridor on sand, or being very careful. They just went right on running rats in the same old way, and paid no attention to the great discoveries of Mr. Young, and his papers are not referred to, because he didn’t discover anything about the rats. In fact, he discovered all the things you have to do to discover something about rats.
The subsequent research world was, unfortunately, unaware of the conditions that need to be met to be legitimate. It had developed into a cargo cult, where people knew the trappings of what research in the domain looked like, but without addressing the realities like professor Young did, real research would only be possible by chance.
Today, I read a paper on semaglutide (the active ingredient in the GLP-1RA drugs Wegovy and Ozempic) extending the lifespans of mice. The drug allegedly increased lifespans by 100 days—a long time for a mouse! This paper was cargo cult science.
Two years ago, a brilliant paper touching on caloric restriction benefits to lifespan (‘the Pabis paper’) came out and it showed that the lifespan extending effects were concentrated among naturally short-lived animal model strains. What this means is that chronic caloric restriction, where animals eat fewer calories than they would like over a long period, only seem to make animals live longer if they were already predicted to live short lives. But for animals that were longer-lived, the benefits were just not there; the observed ‘life extension’ was not so because it depended on there being problems addressed by caloric restriction that aren’t found in normally healthy animals.
By analogy to professor Young’s maze-running rats, the paper had found a condition that had to be met to show animal lives were being meaningfully extended: researchers must show that the animals would’ve already lived a healthy, normal-length life, and that the lifespan extension resulted in something beyond normally expected lifespans. Comparing treated animals to controls that are themselves unhealthy, bred poorly, or raised poorly just doesn’t lead to a conclusion that even plausibly generalizes to normal, healthy animals.
The new semaglutide paper does not meet this standard: the researchers behind it failed to set up an experiment that could inform us about semaglutide’s effects on extension of healthy, normal lifespan.
The Pabis paper discussed B6 mice, the types used in the semaglutide paper. It also advised avoiding the use of unhealthy, metabolically abnormal non-wild-type control animals and the semaglutide paper described their mice as healthy, naturally-aged, wild-type mice. But this is where they diverged: the Pabis paper recommended using controls with a median lifespan near the strain’s healthy optimum lifespan, which for these mice is around 900, plus or minus 50 days; the control in the semaglutide paper reached a median lifespan of just 742 day, and the treated group only reached 834.1
The semaglutide study attempted to anticipate this objection, but did so in a self-defeating way. They referenced six studies on B6 mouse lifespans to note that the control group had a lifespan that was close to typical. But the numbers in the citations don’t support their case. For example, the Ogiso et al. paper found mean lifespans of 798 ± 161 days for B6J females and 763 ± 163 days for B6N females, which includes early deaths—meaning it’s almost-certainly mechanically brought down relative to the numbers in the semaglutide study.
This simply cannot do! This study did not use animals with long enough lifespans at baseline to inform us about life extension! What is the point of such a study?!
Unfortunately, the semaglutide study has already been widely shared and applauded. People are using it to claim that semaglutide helps extend lifespans; they’re calling GLP-1s longevity drugs; they’re saying weight loss, increased movement, and so on will make people live much longer lives. All of this is unlikely to move maximal lifespan even if it helps to bring up the left tail of lifespan for the same reason that caloric restriction helps particularly messed up animals and not normally healthy ones.
This study reads to me much the same as observational studies saying things like ‘Owning a dog helps people live longer!’ (confounded by selection into who owns dogs) and ‘Vitamin D supplements will help your X, Y, and Z!’ (confounded by selection into who has high levels of vitamin D). People see the airmen flying their planes and dropping off cargo; they see the soldiers marching and eating their rations; they adopt dogs, take useless supplements, and run pointless studies. And we’re all worse off for it.
This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.
A message from my sponsor, Mechanize:
We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.
To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.
Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.
The Pabis paper also calls for testing across sexes and the semaglutide paper was female mouse-only, and it calls HET3 mice advantageous, but the semaglutide paper just uses inbred B6. These aren’t huge deals, but they do mean we lack a check on plausibility and they limit generalization further.
Gwern did a follow-up on this. The reason it was ignored was mostly because it was never published, because the group just... never bothered, with ANY of their research.
One point in favor of "publish or perish"!
https://gwern.net/maze
As far as I understand things semaglutide was originally introduced to treat obese patients with type 2 diabetes and other co-morbidities so exactly the type of reduced life expectancy patients you mention. And semaglutide has been very successful with these patients. Expecting normal-life-expectancy patients to react the same is worth a look but the experimental protocols need to very careful or they do lead. as you write, to cargo cult science.