Lying for a Climate Crusade

The Consumer Product Safety Commission is on a mission that has nothing to do with safety

As reported two days ago by the Los Angeles Times, the U.S. Consumer Product Safety Commission (CPSC) is considering banning gas stoves, federalizing a move already embraced in some locales. The basis for this was a nine-paragraph paper that came out near the end of last month in the MDPI publication International Journal of Environmental Research and Public Health.

The paper took an estimate of the effect of gas stoves in the home on asthma rates obtained from a meta-analysis and entered the estimate into a formula to estimate the proportion of the US’ (and several state’s) asthma that was theoretically attributable to gas stoves. The estimates were huge! A whopping 12.7% of the US’ asthma was estimated to be due to gas stoves, and this was even more extreme in Illinois, where 21.1% of asthma cases were estimated to be due to the pernicious effects of gas stoves in the home. This result should rightfully take readers aback. This is huge. We have the potential to prevent millions of future asthma cases by simply electrifying homes. With the CDC’s estimated cost of asthma per person of $3,300 in 2011 prices ($4,368 now), spending to replace gas stoves with electric ones is a steal!

But these results were not so clear-cut, and I will explain why the more likely explanation is ignorance and fraud.

---

Let’s go over the meta-analysis that Gruenwald et al. referenced, Lin, Brunekreef & Gehring (2013) , only touching on those sources that have to do with asthma and gas stoves. The first study they used for this was Behrens et al. (2005) and it came with what Lin, Brunekreef & Gehring interpreted as an odds ratio, but what was, in fact, a prevalence ratio. The difference between the odds ratio (OR) and prevalence ratio (PR; i.e., the RR) is subtle. The OR is (A/B)/(C/D) where A is the prevalence of a condition in an exposed group and B is their total population, while C is the prevalence of a condition in a nonexposed group and D is that group’s total population. This condenses to AD/BC. The PR is (A/(A+B))/(C/(C+D)). The OR and PR equations for the opposite directions are BC/AD and (B/(A+B))/(D/(C+D)). These are important to note for two reasons: magnitudes and sign consistency.

The formulae clarify that the PR will always be less than the OR. When a PR is entered into an OR meta-analysis, the likelihood of publication bias can be affected, but more importantly, the effect size of a PR with the correct sign will be understated compared to similar ORs and a PR with the opposite of the prevailing sign for ORs will fail to bring the meta-analytic mean closer to its hypothetical true value because it is smaller than it should be. This means that it can, more often than an OR, fail to change the sign of a meta-analysis’ estimated effect when it should. For causal inference, the PR is also notably inappropriate: “PR can only be used in cross-sectional studies and not in clinical trials or cohorts since the former measures prevalence instead of risk.”

So let’s redo this meta-analysis and avoid using the studies with prevalence ratios when we can’t get their odds ratios. That eliminates the first study on asthma prevalence, Behrens et al. (2005). The second study was Carlsten et al. (2011), a study in a cohort at high risk for asthma - a group who should be more strongly affected by their environments than members of the general population because of the nature of asthma risk. Strangely, the OR used in the meta-analysis was an adjusted one rather than the raw one. The adjusted OR was 1.4 while the unadjusted one was (8/(8+29))/(63/(63+280)) = 1.177 (0.28 - 4.87). Since the meta-analysis authors did not have access to original data and could not make comparable adjustments across studies, they should have gone with the unadjusted estimates for comparable scaling factors, so we will. More importantly though, since they weren’t working with experiments, their adjustments needed to be motivated by causal theories; for all we know, adjustment was biasing or homogenizing. Whatever it did, it almost certainly contributed between-study variance that was not due to the gas stoves in question.

The next study was Casas et al. (2012), from which the meta-analysis authors had the option of using the unadjusted OR or ORs with three levels of adjustment and they chose the highest and most precise available one instead of the unadjusted one. So, instead of 1.33 (0.92 - 1.93), we will use 1.10 (0.80 - 1.50). Dekker et al. (1991) provided an adjusted OR of 1.95 (1.41 - 2.68) and an unadjusted OR of 1.94 (1.25 - 3.03). Diette et al. (2007) had an OR of 0.97 (0.33 - 2.81). Dodge (1982) had an OR of 1.24 (0.18 - 8.70). Garrett et al. (1998) did not provide the requisite sample sizes for exposed and nonexposed children in their sample to allow the calculation of the ORs and they did not provide CIs for gas stove exposure’s unadjusted OR. To use this datapoint, we can take the unadjusted OR of 2.11 and the adjusted CI of 1.06 - 4.72, but this might not be strictly admissible, so the results with or without them can be calculated. I suspect that were the requisite data available, the result from this study would not be significant since it barely is after considerable adjustment. Hessel et al. (2001) gave us numbers for asthmatics and non-asthmatics with gas stoves, but did not give us numbers for asthmatics without gas stoves. However, through arithmetic, we can take the OR they provided to get the number of asthmatics without gas stoves and arrive at either 136 or 137 because 1.6 = (27/60)/(137/487) where C was the missing value. The reason this is important to note is that Hessel et al. provided a CI from 1.0 to 2.8 and we cannot know if they rounded the lower end up to 1, incorrectly presenting a nonsignificant result as significant. Recalculating, their OR was 1.60 (0.66 - 3.88). Hölscher et al. (2000) was another study that might need to be removed because only the adjusted OR was provided alongside insufficient data to calculate the OR. Their AOR was 0.59 (0.26 - 1.33). Lin et al. (2012) did not provide enough data to recalculate their ORs and they provided AORs. It was also not clear which of their AORs should have been used, since they provided three different measures of asthma and three different comparisons. The asthma measures were prevalent (ever diagnosed plus symptomatic in last 12 months), incident (ever diagnosed), and symptoms (last 12 months) while the comparison groups were early versus never gas cooking, ever versus never, and current versus never. These things mattered since the 9 all-age effect sizes ranged from 1.06 to 1.29. Unsurprisingly, the meta-analysis used 1.29 (0.98 - 1.69). For the sake of fairness, let’s take the median effect size. Ordered by size, the AORs were 1.06, 1.08, 1.08, 1.10, 1.13, 1.13, 1.15, 1.19, and 1.29. Because there were no principled way to order the 1.13s there, the CIs were averaged for 1.13 (0.915 - 1.395).

Maier et al. (1997) was another study in which PRs were presented and used in the meta-analysis instead of ORs. Incidentally, this example also included a protective estimate for gas stoves, of 0.9 (0.6 - 1.3). Yet another study that is incorrectly limited in its ability to pull down the meta-analytic mean estimate while slightly militating against publication bias concerns! For sure, publication bias corrections might be larger sometimes with more constrained negative estimates for a literature, but that isn’t likely here. Luckily for us, Maier et al. included enough to calculate the ORs. The correct OR was 0.84 (0.35 - 2.03). McConnell et al. (2002) also gave the PR (as the adjusted RR), which the meta-analysis used, but in this case, they were above 1! But they were only above 1 because McConnell et al. wanted them to be. When it came to the “wheeze” symptom, these was a 4.2% rate of “Yes” and a 4.5% rate of “No”. For “no wheeze”, these were 2.1% and 1.4%, respectively. How could these both lead to positive estimates? They had to be reflected to be above 1 and to get the exact estimates provided, the adjustment had to explain the remainder of the discrepancy. The real combined wheeze plus no wheeze group OR was 1.34 (0.77 - 2.36). I see no reason why these groups were initially separated, since all that did was generate two dependent estimates for the meta-analysis, again militating against a finding of publication bias.

Melia et al. (1977) was another study that provided two ORs for some reason. The OR for this study was 1.53 (0.82 - 2.88). Ponsonby et al. (2000) was, bizarrely, the source of two PRs that were differently adjusted. The first was 1.44 and the second was 1.84, while the unadjusted OR was 1.70 (0.35 - 8.25), but the meta-analysis authors went with the PR of 1.84. Ponsonby et al. (2001) was another study providing an adjusted PR (1.20) that was used in the meta-analysis, but it did not provide enough information to compute the OR so it had to be dropped. Spengler et al. (2004) only provided the adjusted OR and did not provide enough information to compute the OR, so they can be used or dropped with their AOR of 2.28 (1.04 - 5.01). Tavernier et al. (2005) provided an OR of 0.69 (0.24 - 1.95). Volkmer et al. (1995) provided two ORs: one from the total sample living in South Australia and one from a subset living in the greater Adelaide region of South Australia only. The meta-analysis authors decided to use the higher one, 1.24, which was from the subset. I will use the complete sample’s OR because there’s no real reason not to; this was 1.19, but the unadjusted CI was not provided, so the adjusted CI of 1.02 - 1.30 had to be used. The final study was Willers et al. (2006), which provided an OR of 1.43 and an adjusted OR of 1.50, which the meta-analysis used. Instead, I used the unadjusted OR (1.43, 0.87 - 2.37).

Let’s first look at the relationship between our 17 odds ratios and their standard errors. In the original take on the meta-analysis, these were supposedly not very well-related. With the new values (logged):

Now the bigger the effect, the less precise. Though not unlike the estimates in this meta-analysis, this was marginal and only significant when sample size-weighted, which is hardly admissible because of the dependence of the independent variable on N. The meta-analytic estimate also became 1.174 (0.981 - 1.367) - no longer significant, and, what’s more, biased. Trim-and-fill suggested adding four studies, dropping the meta-analytic estimate to 1.1048 (0.8906 - 1.3190). As a final note, subgroup analyses by outcome, as in the original meta-analysis, were inconclusive, with meta-analytic ORs for current versus lifetime asthma of 1.30 versus 1.04.

Simply fixing the issue with PRs and the doubling of certain study’s datapoints reduced the estimated effect of gas stoves to nonsignificance. It would be amusing that the errors in the meta-analysis all seemed to go towards a larger effect if it weren’t so unfortunate that the meta-analysis was ever published at all. Given how horribly flawed it was and how strong its incredibly weak conclusions were claimed to be, good scientists should be aware of its dubiousness and ought to campaign for its retraction.

As a final note for this section, there is good reason to distrust meta-analyses. When the data going into one is bad, the ultimate result is likely to be bad regardless of the corrections applied. If you merely reduce the badness of a meta-analysis, you will rarely reverse the sign or nullify the effect even when the true sign goes in the opposite direction or the effect doesn’t exist for no other reason than that the mass of estimates goes the way the literature stands and you could not reverse it because studies going the other way were never published due to bias or whatever other reasons. Even applying liberal corrections for publication bias will not fix the low quality of typical meta-analyses. That’s what this paper found comparing large preregistered multi-lab replication project’s results with those of meta-analyses: if anything, the most liberal publication bias corrections were still too conservative.

One way around the garbage in-garbage out issue of meta-analysis is to rely on the highest-quality study or studies alone. This is rarely an option because there are usually not stand-out studies capable of reversing an entire field’s conclusions. In the case of gas stoves and asthma, we are lucky that there was such a study from the same year as the original highly-flawed meta-analysis and it even shares a coauthor. The study’s data was obtained between 1997 and 2004 and involved 512,707 children in 108 centers in 47 countries - more countries and children than the rest of the meta-analyzed literature combined. The study had a simple conclusion for the link between gas stoves and asthma: nothing. It delivered a clear null! Here’s the relevant table:

---

Let’s apply our new and improved meta-analytic estimate to the calculation of PAF, the population attributable fraction of cases. Gruenwald et al. cited Knibbs et al. (2018) for the formula for this quantity representing the proportion of cases in the population that are attributable to an exposure. This quantity is a bit dumb, and I would hope any half-decent economist would see the problem with it immediately. Here’s the formula:

The relative risk can be swapped out for the OR, as noted by Knibbs et al. The important detail here is that this statistic is very sensitive to the effect size (RR/OR)), but not so much to the proportion of exposed people in a population (p). If we take an OR of 1.1 and find a p of .4, 3.8% of the thing being affected with our given effect size is attributable to it. Barely anything changes doubling p to .8, as the PAF only becomes 7.4%. However, if even 10% of the population is exposed to something with an OR of 2, the PAF is 9%, and with 40% exposure, it’s 28.57%.

The meta-analysis Gruenwald et al. cited was deeply flawed, with vastly overestimated effect sizes of 1.32 for lifetime asthma and 1.42 for current asthma. These compare to 1.30 and 1.04 in the revised meta-analysis where PRs are not incorrectly conflated with ORs, unadjusted ORs are used when possible to make estimates usable at all, studies are not randomly split into multiples, and the largest estimate from a given study is not always picked to enhance the force of the meta-analysis.

The flaws are deeper still because every estimate is noncausal and confounded with all sorts of exposures that are related to use of gas stoves, from poverty to race to things as seemingly benign as distance from a person’s home to a curb. To obtain a causal estimate, one might use data from a project like Moving to Opportunity, but no such estimates exist. The air pollution literature more generally has many estimates that use causal inferences methods like instrumental variables and event studies. Based on Bagilet & Zabrocki-Hallak’s recent review, it seems there’s only one well-powered instrumental variable study in the whole literature, so it is clearly highly exaggerated and untrustworthy.

What happens if we compute the PAFs for every variable like gas stoves in the home? That is, what if we produce a PAF for whether a parent is a smoker? What about if two parents are smokers? Dampness, humidifier use, ownership of a bird, dog, cat, or any other pet, geographic variables aplenty, socioeconomic status, number of bad weather days, ventilation quality, and so on? What if we use genetic admixture, since African admixture is known to be related to asthma risk? What if we just convert the heritability of asthma to an odds ratio and plug it into our PAF formula? Everyone has genetics and the OR of with a heritability of 50% (and it might be more heritable than that) would be 37.622, leading us to a PAF of 97.34%! If we add in Gruenwald’s estimate of 12.7%, a literature-derived estimate of secondhand smoking’s PAF of 11.5%, a dampness estimate of 10.2%, a pet ownership estimate of 31.3%, and an African admixture estimate of 9.16% (because African Americans are 12% of the population and average 80% African with an OR of 2.05), we can then explain 172% of the PAF of asthma! That’s not even all the variables out there - even some large effect variants may have ORs greater than two! A correlation of r = 0.30 is an OR of 3.129 (and ORs of 1.32 and 1.11 are r’s of .076 and .029, respectively), which means it would have a PAF of 45.82%, or roughly five times its variance explained.

This statistic cannot be used to inform policy unless it is causally derived and accurately estimated. It is too correlated with too many other variables and its effect is not actually known to any reasonable degree. Secondhand smoke is a great example of this because if one parent smokes, the other tends to as well. If we use “father smokes” and “mother smokes” as separate variables, we can explain far more than “either/both parent smokes” because the two variables are imperfectly correlated. We can even add them to more recapitulations like “suspected secondhand smoke exposure” or “parents friend smokes” or “parents consume alcohol” to explain even more. The second author of Gruenwald et al., Brady Seals, evidently didn’t appreciate that PAF couldn’t be used with this level of naïveté since she approvingly linked a study that provided different PAFs accounting for around 120% of the US’ childhood asthma. Are some kids getting double- or triple-asthma?

But let’s take this seriously for a minute. Let’s compute the PAFs like Knibbs et al. suggested, by resampling the PAF statistic with some suggested values of our effect size and exposed population in mind. Gruenwald et al. used data from the American Housing Survey (AHS) to estimate the proportion of Americans with gas stoves. For some reason they used 2019 instead of 2021, but who cares, let’s just do what they did. In 2019, the survey had 123,672 responses about the type of cooking fuel used in the home. 74,452 used electricity and 103 used other. The rest used piped (43,469) or bottled (5,648) gas as their cooking fuel. That means that 39.72% of American households cooked with gas. So with the incorrect OR of 1.34, we get pretty close to Gruenwald et al., since we get 0.1190. However, with the corrected OR of 1.1048, 11.90% becomes 3.996%. They might have been able to justify using the 1.42 from the meta-analysis, looking only at current asthma, delivering an underestimate. A mistake, to be sure, since despite potential conceptual differences, these were often not well-distinguished between studies and the estimates for gas stove effects on current and lifetime asthma didn’t statistically differ. But let’s be very serious and do the resampling procedure they did.

Gruenwald et al. never provided their code or data, but I’ll provide code so people can reproduce this. I did this in R, like they claimed to do, and I did it with the boot package. My seed was 123. First, the function I used to compute PAF was this:

PAF <- function(data, index) {

p <- data[index, 1]

RR <- data[index, 2]

return((p * (RR - 1)) / (p * (RR - 1) + 1))}

Next, I generated 10,000 values of p around a mean of .3972 with an SE of \sqrt{\frac{.3972 * (1 - .3972)}{123672}} = 0.0014. I generated 10,000 values of OR with a mean of 1.1048 and an SE of \frac{1.3190 - 0.8906}{3.92} = 0.1093. Next, I simply did

PlotDF ← data.frame(PAF = boot(PORDF, PAF, R = 10000)$t0)

For the quantiles, I did quantile(PlotDF$PAF, c(.05, .5, .95)) and I used ggplot2 to plot the results. Here they are. I’ve labeled the 5th percentile and 95th percentile of results. The median was 0.0392. I also labeled Gruenwald et al.’s proposed PAF.

Their result simply isn’t happening. What about for other states? Gruenwald et al. looked at Illinois (PAF = 0.211), California (0.201), New York (0.188), Massachusetts (0.154), Pennsylvania (0.135), Texas (0.117), Colorado (0.108), Ohio (0.095), and Florida (0.030). The bootstrapped values with the correct OR were 0.068, 0.067, inestimable for 2019 because data was unavailable so 2017 (most recent year) data indicated 0.068, 0.048, inestimable for 2019 because data was unavailable so 2021 data indicated 0.044, 0.036, 0.028, 0.034, and inestimable for 2019 because data was unavailable so 2021 data indicated 0.009. The real values were fractions of what Gruenwald et al. reported. They didn’t even have a consistent rank-order because of small changes in responding (and perhaps real-life conditions) across survey years, an expected result given that the effects being dealt with are so minute that a small amount of sampling error or change over time could alter everything. Anyway, let’s plot these next to each other, noting for the sake of completeness that none of the corrected estimates are significant.

---

The results are very wrong, but there’s something even worse about this paper: an undeclared conflict of interest. The first two authors, Talor Gruenwald and Brady A. Seals are employees of RMI, Carbon-Free Buildings, which is, according to their website, “a non-partisan, non-profit organization that works to transform global energy systems across the real economy.” Their organization’s aims are to “Raise public awareness of health and climate costs of fossil fuels in buildings.”, “Design and advocate for carbon-free buildings policies in 20 key US states that represent 70% of direct gas use.”, “Retrofit large numbers of existing buildings to be all-electric, grid-interactive, and efficient.”, “Create buildings industry platforms to support dissemination of technology, supply chain development, and business-led interventions.”, “Leverage US successes to influence global supply chains and scale to China, India, Southeast Asia, and Africa.”.

In other words, this is a work of blatant activism by people with an organizational interest in these extremely overwrought results. Even worse, less than a month after the study was published online (21 December, 2022), the CPSC cited the study as a reason to ban gas stoves. The remit of the CPSC is not pollution. However, someone inside the organization clearly wanted to ban gas stoves, and this supplied a reason because this suggests gas stoves are a consumer health hazard. It suggests far more than what the evidence really does. At this point, one might ask why the EPA hasn’t dealt with this given that it concerns pollution, but their remit does not include indoor pollutants. Interests aligned and a garbage study intended to influence policy was shat into a low-tier MDPI journal and immediately received for its purpose as a weapon to be used by the appropriate regulator.

---

This paper does do one interesting thing: it raises the question of why asthma rates are up when virtually every exposure that could increase asthma is on the decline. The air and water emissions people are exposed to are way down! For example:

And despite more miles driven, vehicle emissions are also down:

So much for living at the curb. This is all so odd! Homes are much better insulated than in the past, dampness, mold, smoking, and gas exposure have all fallen basically across the board. Record numbers even own electric vehicles and have gas detectors installed in their homes. The only variables that contribute to asthma that are noticeably increased are obesity, sedentism, and pet ownership. The other option is increased diagnosis. Our immense modern wealth makes the luxury of chronic disease diagnosis available where in the past it simply wasn’t. The threshold for noticing has dropped and the nomothetic web surrounding phenomena like asthma and autism are suspect. The studies in this literature absolutely cannot be taken for granted.

---

This was a clear-cut case of science being brandished for activism, and judging by the paper’s reception in the popular press and on social media, this obviously slapdash and compromised work was happily received. Next steps are clear: the editors and reviewers involved in the publication of this paper need to be brought to light and punished and the paper needs to be retracted with an apology from the journal.

Perhaps after years of being bullied, the public still hasn’t understood the costs of letting regulators wield activists’ “science” to run roughshod against the public interest. Let’s hope this fails.