I am very surprised at genetics looking so bad. Geneticists are mostly very aware of the problems of multiple hypothesis testing and do a lot about it. I really want to see how the Z statistics were collected.
One additional note is that economics often uses p < 0.1 as a secondary threshold, with top journals like QJE putting stars next to such results, while other fields' journals would not (using a different symbol or none at all).
Thanks. Would you be so kind as to elaborate on how you arrived at the following conclusions: “we can calculate that at 80% power and with all real effects, 12.60% of results should return p-values between 0.05 and 0.01. Under the null hypothesis, only 4% would be in that range”?
"In fact, with this knowledge, we can calculate that at 80% power and with all real effects, 12.60% of results should return p-values between 0.05 and 0.01. Under the null hypothesis, only 4% would be in that range. But there are problems comparing this to the observed distribution of p-values. To name a few"
Is the point (and its corresponding graph) that if we have a set of studies with an 80% statistical power and their effects are genuine, the distribution of their z-values should be something akin to the 'Alternative with 80% Power' graph, having an average z-value of ~3.7, as opposed to the observed clustering around 1.96?
Economics is mostly made up to begin with. It's much easier to create consistency inside a pseudo-scientific simulation of "reality" than within the natural sciences.
>So with incredibly low power, what proportion of results should be between p = 0.05 and 0.01? The answer is far fewer until you drop just below 50% power
Am I misreading? I’m assuming if you shift the real-effect distribution to the left, the area under our target Z-scores (converted to p-values) should increase even before we get to 50% power. The “far fewer” line makes it seem like that’s not the case. Maybe I’m misunderstanding?
I am very surprised at genetics looking so bad. Geneticists are mostly very aware of the problems of multiple hypothesis testing and do a lot about it. I really want to see how the Z statistics were collected.
What math papers are reporting p-values? Stats?
One additional note is that economics often uses p < 0.1 as a secondary threshold, with top journals like QJE putting stars next to such results, while other fields' journals would not (using a different symbol or none at all).
Thanks. Would you be so kind as to elaborate on how you arrived at the following conclusions: “we can calculate that at 80% power and with all real effects, 12.60% of results should return p-values between 0.05 and 0.01. Under the null hypothesis, only 4% would be in that range”?
"In fact, with this knowledge, we can calculate that at 80% power and with all real effects, 12.60% of results should return p-values between 0.05 and 0.01. Under the null hypothesis, only 4% would be in that range. But there are problems comparing this to the observed distribution of p-values. To name a few"
Is the point (and its corresponding graph) that if we have a set of studies with an 80% statistical power and their effects are genuine, the distribution of their z-values should be something akin to the 'Alternative with 80% Power' graph, having an average z-value of ~3.7, as opposed to the observed clustering around 1.96?
Economics is mostly made up to begin with. It's much easier to create consistency inside a pseudo-scientific simulation of "reality" than within the natural sciences.
Ot-- if I wanted to learn statistics of the type you do, theoretically and practically, what would you recommend doing?
It seems wrong for economics to be at the top of a hierarchy of the sciences:
https://pubmed.ncbi.nlm.nih.gov/20383332/
>So with incredibly low power, what proportion of results should be between p = 0.05 and 0.01? The answer is far fewer until you drop just below 50% power
Am I misreading? I’m assuming if you shift the real-effect distribution to the left, the area under our target Z-scores (converted to p-values) should increase even before we get to 50% power. The “far fewer” line makes it seem like that’s not the case. Maybe I’m misunderstanding?
Sus.