Today’s post is brought to you by my sponsor, Mechanize. They’re hiring junior software engineers at $300K/year base salary.
The Wordsum is a 10-item vocabulary test where you’re presented with a word and you have to then select the closest word to that one from a list. For example, you could be given the word “concern” and your options would be:
See clearly
Engage
Furnish
Disturb
Have to do with
This brief test has been instrumental for understanding the relationship between intelligence—which, like all tests, it taps—and all sorts of social variables, from political views to incomes and education. But, despite its repeated administration and the considerable amount of use it’s seen, it’s been decades since it’s been properly validated. Accordingly, we don’t know how g-loaded Wordsum is.
The easiest way to figure out how g-loaded Wordsum is would be to administer a battery of tests with the Wordsum as one of the tests. From there, you would do a factor analysis and the loading would pop right out—simple. But in lieu of such a test, we can still validate the Wordsum through proxy methods. The way this works is simple:
The National Longitudinal Survey of Youth (NLSY) 1979 and the 1997 version both administered a Word Knowledge test
This test is substantially the same as the Wordsum, it’s just longer
We have the actual test items for the 1979 cohort
Both NLSYs (‘79 and ‘97) also administered a more extensive cognitive battery
Both NLSYs and the GSS have extensive sets of demographic variables
We can match the samples demographically
If we do this in time and place, it’s plausible that conclusions from the NLSY and the GSS can be transferred between one another
We can reconstruct a ‘quasi-Wordsum’ in the NLSY ‘79 by matching items to the difficulty of the Wordsum items in the GSS
We can validate the similarity of this quasi-Wordsum to the real one by examining how it correlates with variables found in both the GSS and the NLSYs
We can assess the g-loading of the quasi-Wordsum
Thus, we can pretty much get a measurement of the g-loading of the real Wordsum
I noticed this was possible while working on a post about the relationship between intelligence and political alignment and how it’s changed over time. I was particularly inspired when I noticed that the IQ-by-politics gradient was four-times steeper in the NLSY ‘79 than in the GSS in the time around Obama’s election. Look:
I figured this was due to the fact that the NLSY ‘79’s testing was much more extensive, so I thought of how I could test that, and the fact that we had item data immediately came to mind because I had worked with it before. Accordingly, I took this snapshot from this era and began the demographic matching to the GSS before attempting to construct a Wordsum analogue. Remaking the plot with the matching plus the quasi-Wordsum ready to go, the difference is readily apparent, as is the validity of the quasi-Wordsum as a proxy for the genuine article:
After more extensive vetting, including looking at how well income and education were predicted, I concluded that the quasi-Wordsum does function like Wordsum, and the linkage with between Word Knowledge and other tests more generally is similar enough across the NLSY ‘79 and ‘97 that we can use the latter—which lacks item-level data for proper quasi-Wordsum construction—to validate the GSS’ Wordsum, too. All of this will be viewable in the replication package with the politics post I’m about to make.
For now, let’s look at a comparison of the AFQT and the quasi-Wordsum to figure out why it predicts politics worse. The answer is about half reliability differences (quasi-Wordsum is a short test with range restriction, like the real Wordsum is in the GSS), most of the residual is about low g-loading, and a tiny share is due to non-g ability variance. This looks like so:
As to the question behind this post, about how g-loaded the Wordsum is, the answer in the NLSY ‘79 is that the g-loading is 0.68 if we include the rest of the Work Knowledge items as indicators for the g factor, and it’s 0.62 without that, indicating a role for non-g ability variance. Disattenuating leads to g loadings of 0.83 and 0.76. In the NLSY ‘79 and ‘97, the g-loadings for the Work Knowledge subtest are 0.84 and 0.82, respectively (not significantly different), so we can reasonably assume psychometric comparability, and thus that the quasi-Wordsum g-loading in the NLSY ‘97 would be the implied Word Knowledge loading if the test had the reliability of a version with ten items: 0.66 (0.81).1
The Wordsum test is highly g-loaded and is a suitable proxy for intelligence, even if it would be nice to have a broader, more reliable, and less range-restricted test score.2
This was a timed post. The way these work is that if it takes me more than an hour to complete the post, an applet that I made deletes everything I’ve written so far and I abandon the post. You can find my previous timed post here.
A message from my sponsor, Mechanize:
We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.
To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.
Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.
The 0.81 is the disattenuated loading.
Yes, I know this analysis leaves some wiggle room and contains some error due to the use of proxying rather than direct measurement. The real results would almost-certainly be close.