What you read in an abstract, a title, or even in the body of a paper might be the opposite of what papers actually show, and sometimes result are just made up
In my experience, even well educated researchers have great difficulty understanding how QRPs can undermine the meaning of their results. In part, it is a matter of being "hard to get a man to understand something when his livelyhood depends on not understanding it." But also it is the difficulty in imagining that just a few QRPs can create an irreproducible, meaningless result. Perhaps all researchers should be required to perform sensitivity analyses even in fields where.that isn't the norm. I remember being very excited about a surprising result from a pilot project and getting ready to try to replicate it when I saw that the effect was entirely driven by a single extreme outlier.
Could you expand on your writing technique? These 1-hour posts are impressive.
I assume you do all the research beforehand and gather all the elements before writing? You must have an idea of the overall outline too I suppose? It must have been hard on the first posts, no? Would be interested to read more on this.
Related to your recent X thread on natural sources for drugs, you (or readers, if already familiar to you) may be interested in the history of Chinese hamster ovary cells in medical research and drug production.
The storybook characterizations "colliders" are true enough. These problems are common, though less so in economics today. In top journals, the fallacies tend to be more subtle than your examples.
Then you turn to an obscure study on immigration that appears to make no sense on its face. It compares crime rates in different communities with different shares of immigrants. The comparison is fallacious at the start, because immigrants do not randomly choose where they live. They probably live in poor communities with more crime problems, because it's all they can afford. The supposed collider ("suspect rate") is never defined, but it doesn't matter---there's no compelling study design to begin with.
But why do you choose this study of all studies? Was it impactful? Did it persuade anyone? The way you build context, and then choose this particular study, appears designed is to mislead, to implicitly suggest that the truth is opposite of this study's finding. But the reality is that immigrants commit crimes at a far lower rate than natives do. This fact cuts the data differently, looking at individuals rather than communities. It is indisputable.
A recent study looking at a long history of this fact:
The abstract thing reminds me of a paper I saw brought up multiple times as supposedly showing that "the science" supports transwomen in women's sports. https://link.springer.com/content/pdf/10.1007/s40279-016-0621-y.pdf. I checked out the paper, and the abstract proclaims "Currently, there is no direct or consistent research suggesting transgender female individuals (or
male individuals) have an athletic advantage at any stage of their transition (e.g. cross-sex hormones, gender-confirming surgery) and, therefore, competitive sport policies that place restrictions on transgender people need to be considered and potentially revised." Then you look at their review of 39 sources. Only 8 of these were actual studies, and 7 of those were basically just asking transgender-identified people if they enjoy sports. Exactly one study even tried to address whether there was an advantage, and of course it found that biological males taking cross-sex hormones have an advantage over biological women.
Sometimes I wonder if I'm being lied to when the author assures me that his very thoughtful and well-structured post was written in under an hour.
To be fair, in a lot of these timed posts, the content for certain sections is practically pre-written over on my Twitter. For example, https://x.com/cremieuxrecueil/status/1773936474079002634
How many failed timed posts have happened? And why are okay with having your work deleted if you miss the timer?
In my experience, even well educated researchers have great difficulty understanding how QRPs can undermine the meaning of their results. In part, it is a matter of being "hard to get a man to understand something when his livelyhood depends on not understanding it." But also it is the difficulty in imagining that just a few QRPs can create an irreproducible, meaningless result. Perhaps all researchers should be required to perform sensitivity analyses even in fields where.that isn't the norm. I remember being very excited about a surprising result from a pilot project and getting ready to try to replicate it when I saw that the effect was entirely driven by a single extreme outlier.
What is “suspect rate?”
What’s currently stopping journals from requiring data to be published in papers (or somehow provided *without* needing a request)?
It seems like very low-hanging fruit to at least make outright fraud more likely to be caught, and probably less attractive as a result.
Could you expand on your writing technique? These 1-hour posts are impressive.
I assume you do all the research beforehand and gather all the elements before writing? You must have an idea of the overall outline too I suppose? It must have been hard on the first posts, no? Would be interested to read more on this.
Thank you for the great content.
Related to your recent X thread on natural sources for drugs, you (or readers, if already familiar to you) may be interested in the history of Chinese hamster ovary cells in medical research and drug production.
The storybook characterizations "colliders" are true enough. These problems are common, though less so in economics today. In top journals, the fallacies tend to be more subtle than your examples.
Then you turn to an obscure study on immigration that appears to make no sense on its face. It compares crime rates in different communities with different shares of immigrants. The comparison is fallacious at the start, because immigrants do not randomly choose where they live. They probably live in poor communities with more crime problems, because it's all they can afford. The supposed collider ("suspect rate") is never defined, but it doesn't matter---there's no compelling study design to begin with.
But why do you choose this study of all studies? Was it impactful? Did it persuade anyone? The way you build context, and then choose this particular study, appears designed is to mislead, to implicitly suggest that the truth is opposite of this study's finding. But the reality is that immigrants commit crimes at a far lower rate than natives do. This fact cuts the data differently, looking at individuals rather than communities. It is indisputable.
A recent study looking at a long history of this fact:
https://www.nber.org/papers/w31440
There are many ways to mislead.
The abstract thing reminds me of a paper I saw brought up multiple times as supposedly showing that "the science" supports transwomen in women's sports. https://link.springer.com/content/pdf/10.1007/s40279-016-0621-y.pdf. I checked out the paper, and the abstract proclaims "Currently, there is no direct or consistent research suggesting transgender female individuals (or
male individuals) have an athletic advantage at any stage of their transition (e.g. cross-sex hormones, gender-confirming surgery) and, therefore, competitive sport policies that place restrictions on transgender people need to be considered and potentially revised." Then you look at their review of 39 sources. Only 8 of these were actual studies, and 7 of those were basically just asking transgender-identified people if they enjoy sports. Exactly one study even tried to address whether there was an advantage, and of course it found that biological males taking cross-sex hormones have an advantage over biological women.
"the higher the proportion of foreigners in an district" should be in *a* district.
"rather than district- or region-level data" has an extra hyphen after 'district.'
Yup, I remember being ticked the first time I saw this https://entitledtoanopinion.wordpress.com/2012/05/06/rushton-is-the-spengler-of-race-realism/