Why America’s Racial Poverty Statistics Are a Lesson for Researchers

What if a single government employee could tell you an entire literature was wrong?

Sometimes when you survey people about something simple like their age, they put down the wrong answer.¹ Instead of writing that they were born in 1970, someone might put down that they were born in 1870, or 1979. To make matters worse, sometimes the people doing data entry make errors when they record people’s data; 1970 might become 1870, 19700, 1979, or any other errant number. These are sources of measurement error and, as it turns out, when you ask people about whether they’re receiving welfare, there’s a lot of measurement error.

I was recently fascinated by this paper on measurement error in reported SNAP, Social Security, unemployment insurance (UI), general assistance, TANF, Medicaid, Medicare, and the Indian Health Service, and private pension receipt. The paper looked at self-reported rates of receiving benefits in the popular and widely used American Community Survey (ACS), the Current Population Survey Annual Social and Economic Supplement (CPS ASEC or just CPS), and the Survey of Income and Program Participation (SIPP), and it compared how those rates stacked up to official administrative statistics from the IRS, the Center for Medicare and Medicaid Services, the Indian Health Service, the Social Security Administration, and several states’ SNAP and TANF administrators. This was made possible through the Person Identification Validation System and the Social Security Administration’s Protected Identification Keys making it possible to link participants in the ACS, CPS, and SIPP to their administrative records. Simple enough.

Let’s start by looking at who receives SNAP benefits. Administrative data suggests that the prevalence of food stamp receipt is underestimated in all races, but that the measurement error is greater for Blacks and Hispanics than for Asians and Whites. Or to put things differently, Black and Hispanic welfare usage is more severely underestimated than is the welfare usage of Asians and Whites.

In addition to the prevalence of SNAP receipt being underestimated in surveys, the average dollar amount received is also underestimated. In the first study (second study), Whites averaged a reported receipt of $3,324 ($3,409) versus a true average of $3,660 ($3,607). For Blacks, these figures were $3,648 ($3,825) and $4,080 ($4,172); for Hispanics, they were $3,576 ($3,528) and $4,092 ($3,638); and for Asians, they were $3,708 ($2,803) and $3,912 ($3,286). The reality is that the government gives out SNAP to more people than survey estimates would lead us to believe. Is the same true for TANF and unemployment insurance? Yes:

When it comes UI, the average amount different groups received was also underestimated in the 2010 estimate from the CPS and the 2011 estimate from the SIPP. For Whites, the CPS and SIPP estimates were $8,065 and $7,194 versus real average receipt of $8,808 and $9,301. For Blacks, survey-based receipt was estimated at $6,917 and $5,762 versus real average receipt of $7,990 and $8,573. Hispanics followed the same pattern, with survey-estimated receipt of $7,556 and $5,963 versus real average receipt of $8,803 and $8,562. Likewise, Asian receipt was underestimated at an average of $9,421 and $8,023 versus the reality of $10,990 and $9,664. As with UI, however, average TANF receipt prevalence was underestimated, but unlike with UI, the average amount received was overestimated. For Whites, they were estimated to receive $2,567 on average versus a real value of $2,196. For Blacks the estimate was $3,420 versus a reality of $2,640, and for Hispanics, the estimate of $3,006 was about twice their reality of $1,455. Asian numbers for TANF could not be estimated due small sample sizes requiring suppression of the results.

The numbers for pension receipt are perhaps the most stunningly underestimated. In data from 2013-14, 40.3% of Whites said they were receiving pensions versus 74.6% who actually were. Similarly, for Blacks, Hispanics, and Asians, just 31.6%, 23.6%, and 21% reported receiving pensions versus true proportions of 65.7%, 48.3%, and 45.3%. People also underestimated how often they received general assistance, Indian Health Services, Medicare, and Medicaid.

Across diverse studies and measures, we repeatedly see evidence that some people overreport service usage, but more often and more severely, we see evidence that people tend to underreport, and they do so differentially by race. This matters because it has the added implication that poverty rates have been misestimated due to misreporting in surveys, and that misestimation has contributed to the appearance of poverty among Blacks and Hispanics relative to Asians and Whites.

---

The Bureaucrat’s Dilemma

These results suggest (at least) three important things:

1. American poverty is less prevalent and less racialized than surveys suggest;

2. Papers on poverty statistics, racialized progress against poverty, and benefits over- and under-receipt have been systematically wrong for years;

3. Someone at the IRS or Social Security Administration could have noticed these facts and enlightened the researchers contributing to this literature.

If we change the “someone” in point #3, we can generalize it very broadly:

Somewhere out there, there is social scientific work being done on important subjects with the limited data that’s available, and the conclusions rendered by that work will be overwrought or simply wrong due to the limitations of the data. If additional, already-existing data was more accessible, potentially costly errors could be avoided.

I can access the ACS, the CPS, and the SIPP straight away, there’s no need to wait. But if I want to access administrative data to confirm any conclusions I might draw from those surveys, I’ll probably have to wait a very long time and possibly forever. The data needed to avoid messing up simply isn’t open enough for me to avoid potential mistakes made through no fault of my own.

Now imagine I’m a medical doctor and I want to make sure my patient gets the best medication for their condition. I look at the latest meta-analysis or systematic review on the comparative efficacy of different, say, depression drugs. Unfortunately, it’s highly probable that I’ll see something different from what the FDA knows to be the truth. Take a look:

When the FDA reviews the evidence underlying a drug approval decision, they’re privy to all the results from the trials, not just what trial runners decide to present in the published literature. Since what those trial runners prefer to present is often an overstated, overly positive view of drug efficacy, the reality is that, outside of the FDA, we’re receiving a distorted image of how well drugs work. If I’m a doctor tasked with recommending my patient an antidepressant, the FDA’s failure to force authors to be truthful in the presentation of their results and to present all results including the negative ones might influence me to prescribe the wrong drug. A lack of transparency matters, and that’s more than apparent from these two readily available examples that are certainly not even close to being the most important among umpteen others.

Lawmakers need to open up government-collected and/or funded data, lest the lives and efforts of researchers, patients, and everyone else continue being wasted.

1. One of my favorite examples is about how less educated people tend to provide more erroneous height estimates. Another that I find interesting is that more extreme response styles are related to lower intelligence and that fact can be leveraged to provide measurements of it that are probably crude at the individual level, but seem to be good for comparing aggregates.