How Do You Ask An LLM a Question?
LLMs are making people's takes more ambitious and confident, even when they're wrong
Today’s post is brought to you by my sponsor, Mechanize. They’re hiring junior software engineers at $300K/year base salary.
The obvious answer to the headline question is that you would ask it a question in the same way you would ask a person, right? For some queries, I think that’s acceptable—(‘How big is the sun?’, ‘What’s 10+10?’, ‘Find a well-known mathematical conjecture and solve it. Make no mistakes.’)—, but often enough, it’s just not. In my experience, the key issues are obeisance, confidence, ‘nuance’, and tacit knowledge.
I’m going to consider three recent cases that I believe to be illustrative of these issues.
America’s High-Cost Healthcare
In 2014, Random Critical Analysis (henceforth, “RCA”) produced an analysis that decisively debunked a number of popular talking points about American spending, healthcare, and so on, and he popularized Actual Individual Consumption and its importance to the blogosphere and the Twitterati. This blogpost has produced an inordinate amount of ire because left-leaning (= center-left and further) and far-right people interested in politics often hold strong, non-empirical views and corresponding positions on the topic of American healthcare.1
The main thing that RCA documented is that America is not a massive outlier in global healthcare spending once its vast wealth is accounted for. In other blogposts, he expanded on several potentially important points, debunking common ideas about the roles of inequality, prices (see also), administration2, and much more.
What do LLMs like Claude and Grok say when you ask them about RCA’s work and conclusions? If you ask naïvely, you will find that they deliver completely wrong answers. Not only that, but often enough, they’ll explicitly reference RCA and claim to contradict him, while citing evidence that he already dealt with. As an example, using a question from Matthew Yglesias, Claude (on Max effort) appears to do just that.
When asked to evaluate arguments, Claude often begins by conceding that one side makes some points, then going and arguing against their case along some other lines, like faux-steelmanning. That’s typical of modern LLMs and that’s how this started off.
In its concession to RCA, Claude got numbers wrong, understating RCA’s case. For example, it said “AIC per capita runs something like 30-50% above the big Western European economies.” Is this the case? No. AIC was >50% higher than the EU28 economies, >50% higher than the 11 core EU economies, and 55-60% above the “big” economies (assuming these refer to Germany, France, the U.K., Italy, and Spain).
Claude then started to talk about how RCA’s argument was “fragile”, alleging that it was ‘circular’, meaning that high health spending is included in AIC, so its denominator is inflated. Is this true? Yes, but RCA dealt with this obvious objection and showed it was not explanatory. Therefore, Claude should not have brought it up; it was ‘prebunked’. The next point of fragility wasn’t even right: Claude said being “on trend” was sensitive to functional form—not true among valid forms—and the choice of small countries to include—part of the whole point of using AIC, to fix the issues with said small countries’ GDP numbers being off due to tax haven status. The next point of fragility is also not even a point, it’s a restatement of what RCA argued: that the trend line is evidence of common causes of health spending across countries. Claude, however, took issue with RCA’s arguments and said “Sitting on the trend line isn’t evidence of efficiency—it’s compatible with Baumol effects plus waste that scales with income everywhere.” How is restating the thesis a counterargument?
In the next paragraph, Claude made exaggerated claims about the role of administration and billing overhead—things RCA dealt with—, price dispersion across countries, branded drug costs—an indication that Claude knew that drug cost concerns were substantially overstated due to generic drug availability—, and then it cited an incorrect study that RCA already explicitly dealt with. The study—dealt with here—argued that utilization differences don’t explain America’s high health spending, but it did so by using select utilization measures that don’t relate much to health spending or have high U.S. residuals, which are things RCA, once again, explicitly explained. Yet, Claude cited this work when it was already shown to be in error, and then it went on to talk about another thing RCA did explain, before switching topics to talking about lifespans instead of health spending, seemingly just to say RCA was correct on that topic in the middle of a paragraph on spending—something I can only read as ‘attempting to sound like the model is being fair’.
Claude then ends its discussion by making a data-free inference that the U.S. is not normal for its wealth and bringing up three different things America’s alleged “dysfunction” is concentrated in, one of which wasn’t relevant, one of which made no sense, and another of which has some truth to it, but was already dealt with by RCA! Had Claude been seeking a proper answer, it would’ve gathered multiple years of the relevant, publicly available data and tested its proposition, or at the very least, said it was unsure.
Other LLMs reason similarly incoherently and also dodge the facts, the things RCA actually said, and they repeat what’s been published in papers, even when it’s irrelevant or wrong. If you read how the LLMs reason in response to MattY’s question, you’ll notice that they say a lot of things that seem to be relevant, but which really aren’t. As such, they appear to provide a lot of information, but they really just use a lot of words to say very little at all. They do not inform, they misinform.
But notice how the responses are written. We see LLMs that write in a way that, from a human, would be considered confident, even though you can press it on any number of these points, and it will concede. We see LLMs that stick to scholarly consensus, preferring debunked points published in JAMA to the debunkings published on a blog. We see LLMs that repeatedly says RCA makes some points, but that he’s overall wrong when everything is considered.3
Woke Madness
A few days ago, I saw Richard Hanania posting his blogpost Does Therapy Culture Explain the Ideological Mental Health Gap? over on Twitter. I noted that his preferred theory was not viable since the measurement of this gap comes with evidence for measurement invariance when comparing liberals and conservatives.
He then asked me why the finding of measurement invariance rules it out, I explained, and he took to asking ChatGPT what it thought. The issue with using ChatGPT as a spot-check here is that it doesn’t have an understanding of the topic. In fact, in its response, it affirmed I was correct, but didn’t realize it!
Think about what the LLM should’ve known to say. There are many options consistent with how measurement invariance is defined and what it means—which, I’ll note, is specialist knowledge—, but my preferred response from an LLM would’ve involved interrogating the premise: how could therapy culture explain a liberal-conservative gap and be consistent with identical measurement parameters for both liberals and conservatives?
The implication of said identical parameters—that is, the state of measurement invariance—is that the causes of differences in the thing being measured must act the same between groups. But imagine that: how could they?
For one, presumably those who go to therapy would conceive of the questions used to measure mental illness in different ways, ways atypical of society-at-large but normal in the world of therapy. For two, is it plausible that this works for a trait like depression given what it’s measured by? Consider the most commonly used depression scale, the PHQ-9. Here’s a model of it:
Now, imagine where the variable “Therapy culture” would slot into that diagram. To retain measurement invariance, it would have to affect the PHQ9 latent variable or at least operate on all of the measured variables identically to how the PHQ9 latent variable appears to. Since depression is almost-certainly not a ‘real’ latent variable, but rather something more like a network, we can safely constrain our theories to be about the latter type of variable: a factor mimic.
Candidate explanations for filling this role are basically unimaginable: how would therapy culture cause a connection between anhedonia and sleep troubles, and also to every other symptom as it needs to? How could it act across a trait continuum where the symptoms can go either way, such as with too much or too little sleeping, or too much or too little eating? Therapy culture sounds quite magical! Add in that it must explain more conditions than depression—and those sometimes come with pathognomonic signs—and “therapy culture” as explanation really stretches the limits of what’s believable. It may have some role somewhere, but it’s no general explanation.
I’ll leave the theoretical speculation about how therapy culture could be consistent with measurement invariance to the people who believe in its role, since I don’t, and it’s their case to make. I’ll instead simply point out what the LLM does wrong here. For one, it doesn’t understand my objection. This is visible when the LLM mistakenly affirms what I’ve said while claiming to contradict it. It’s also visible when the LLM outlines at a high level what conditions have to hold for measurement invariance to be attainable, without elaborating on the complex and, I think, practically impossible theoretical constraints that would have to hold for the theory to make any sense.
If you prompt the LLM to think more deeply about how this theory could work and you keep poking holes, it eventually concedes the point and says it’s unlikely that therapy culture plays the all-important role it’s been ascribed. But it lacks the tacit knowledge, or the instinct to automatically ply tacit knowledge, to make these issues apparent from the outset at the asking of a seemingly simpler question. But this question is not really simple, and it takes asking a lot of questions to grasp that if you’re not already familiar with the ins-and-outs of psychometric methods. This is true of many questions, so if you ask any of them, for a correct answer, you have to be in a position where you likely already know the answer, making the LLM superfluous.
David Sinclair’s Sudden Blindness Epidemic
David Sinclair recently went on the Joe Rogan Experience podcast and lied egregiously about the side effects of GLP-1 receptor agonist drugs like Ozempic and tirzepatide. The lie that I called out was that “NAION… is now twice as more [sic] prevalent because of the GLP-1 drugs.”
Because David mentioned “sudden blindness” writ large in this section of his interview, I looked at all types of sudden/acute vision loss and found that their incidence hadn’t changed between 2016 and 2023 in national-level electronic health record data. We’re not well-powered for NAION specifically because it’s so rare, but the prevalence of that hadn’t even moved in the right direction. Nevertheless, I got a lot of comments like this one: an LLM was asked who was right, and the LLM suggested Sinclair might’ve been right.
The basis of the LLM’s confusion has to do with a simple mix-up. This is their logic:
There are cohort studies that suggest GLP-1RA users have a higher rate of NAION;
More people are using GLP-1RAs than used to;
Therefore, rates of NAION have increased.
Can you tell where the issue lies? Fundamentally, nothing in this chain of logic demonstrates that the rate of NAION has increased. To demonstrate that NAION has increased, this method just won’t do; instead, you’ll have to count the number of NAION cases and show that they increased.
But, you may ask, doesn’t that logic follow? And the answer is ‘no’, because it only follows under the restrictive condition that the identified effect size is causal, correct, and applied to an appropriate population that we can generalize from. If any of these is off, then you could find arbitrarily large effect sizes without any effect even existing.
I showed that the studies Sinclair was referring to were wrong in three ways:
The effect sizes have declined in newer, larger, better-controlled studies. The magnitudes he’s citing are thus unlikely to be correct;
The populations analyzed in his preferred studies were not representative enough. In fact, better representativeness has come with smaller and smaller effect sizes;
I looked at the measured population rate of NAION and a variety of other conditions and saw no increase. In fact, for NAION—which has such a low rate that changes are hard to determine—the direction of the effect wasn’t even right.
If you just ask the LLMs about Sinclair’s statements, they don’t think to bring up any of this. They may say that the base rate of NAION is low and thus the benefits of taking a GLP-1RA outweigh the risks for almost everyone, but they will not generally walk down the chain of logic and go ‘Hold up, something isn’t right. Where’s the doubling evidence?’ Instead, they make the same error as Sinclair and unfortunately many other scholars of similar ilk.4
Can you see the issues here? LLM answers sound right—the machine provides confident prose—, but to get the correct answer, you would need to know to press it in ways that are not obvious to most people. If you do not, then LLMs will state mainstream views and interpretations, default to what’s commonly understood on sites like Reddit (where much of the training data is derived) and in mainstream news sources like the Wall Street Journal or New York Times. They will incorrectly claim that open-and-shut questions require considering multiple angles to understand, and yet they won’t know when to probe deeper on questions because they lack the tacit knowledge to default to theoretical thinking about the things they’re asked about.
To make matters worse, LLMs will provide inconsistent answers depending on who they believe to be asking a question. I’ve seen this first-hand when, in my own sessions, I get something passable, and a friend asking a question similarly gets rubbish (and vice-versa!). In fact, this behavior has been documented systematically:
To answer the question ‘How do you ask an LLM a question?’, well, I’m afraid I don’t quite have an answer. I don’t think anyone does, as it’s too open-ended. For the question ‘Can you find a mathematical conjecture and disprove it?’ there’s likely enough detail there for it to go off on its own and return something substantive. For the question ‘What is the meaning of life?’ you’ll doubtless be disappointed no matter how many follow-up questions you manage to plow into the text box. Good luck.
This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.
A message from my sponsor, Mechanize:
We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.
To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.
Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.
Treat this explanation as shorthand. I’m not getting into it any deeper than this.
A related problem I mentioned in my previous post is that things like advertising and all the other non-healthcare functions of insurance companies are included in the U.S.’ administrative cost estimates. Additionally, several of their functions’ costs get counted, when they’re in the tax administration share for other countries. This explains a portion of the U.S.’ administrative cost residual that we see over the years. But, as RCA notes, overall, administration doesn’t explain much anyway.
This part is particularly ironic, because the LLM was not thorough and RCA was.
This error is incredibly pervasive, tempting, and obvious if you just stop and ask ‘Where’s the evidence?’ And yet, it keeps being made by researchers, journalists, and laymen who may have been miseducated by the aforementioned groups. One example I noted last year had to do with COVID vaccines causing cancer. The study implied an effect so large that it would be immediately noticeable in cancer or mortality statistics, and yet no such effect reared its head. Accordingly, we know the study’s estimates were flawed.