Really difficult problem when there's some many possible specifications and you just eke out significance. Quite the waste of resources. Someone could plausibly justify one specific model and a barely significant result, so maybe we should judge based on people's expected replicability rate using the z-curve like Ulrich Schimmack (https://replicationindex.com/2021/01/19/personalized-p-values/). We have much more sophisticated ways of judging than "peer reviewed" "prestigious institution" "high impact journal" but that's what the public knows and many academics still believe means quality. And so, they can exploit this way of judging quality of research against controversial hypotheses which they exclude--sometimes in the case of Nature Human Behavior they explicitly state this!
I continually (without success) try to understand this pretzel logic that outcomes for minorities in areas of healthcare and educational attainment can somehow be tied to getting treated or taught by fellow minorities. If this were true, the empirical evidence would be clear.
For example, wouldn't Inner-city school testing scores would be some of the highest in the nation?
It also assumes that the physician population is representative of the patient population and that patients are randomly assigned. Neither is likely to be the case.
You mention a study design that would allow to claim causation and not only correlation: a mother gives birth to many children with the help of doctors of different races.
But even if a mother has a single child, can’t we have an RCT? I.e. the doctor would be assigned “randomly.” Of course some people wouldn’t want to be included in that study, but for people who are willing, assuming we observe correlation, can’t we automatically claim causation?
I thought about mentioning this in the article but ended up not doing that because an RCT/random doctor assignment could work in theory, but it might be problematic in the American context since so many mothers have a preferred physician they want to be there and hospitals they want to be at for their birth. Because of this, it might not be feasible to run a large RCT, and places with random assignment of doctors might be places many mothers avoid, hurting generalizability.
"All we can do to combat it in the long run is to cultivate a culture of radical data openness. Many people don’t want that, however, as they fear data will be misused."
Or perhaps we go back to appointing Court justices upon merit, rather than "check boxes" or political ideology. I point this comment to all the recent justices, not just Jackson. The statistical concepts you outline are tricky and even had me going back to my student days (after 50 year, one forgets much). However, that's not the big issue. The issue is that Jackson doesn't know what she doesn't know and therefore makes these errors of analysis and will do so in the future as she is guided by her ideology, rather than fact and reason.
Really difficult problem when there's some many possible specifications and you just eke out significance. Quite the waste of resources. Someone could plausibly justify one specific model and a barely significant result, so maybe we should judge based on people's expected replicability rate using the z-curve like Ulrich Schimmack (https://replicationindex.com/2021/01/19/personalized-p-values/). We have much more sophisticated ways of judging than "peer reviewed" "prestigious institution" "high impact journal" but that's what the public knows and many academics still believe means quality. And so, they can exploit this way of judging quality of research against controversial hypotheses which they exclude--sometimes in the case of Nature Human Behavior they explicitly state this!
I had never heard of personalized p values. Thank you for sharing the link!
I continually (without success) try to understand this pretzel logic that outcomes for minorities in areas of healthcare and educational attainment can somehow be tied to getting treated or taught by fellow minorities. If this were true, the empirical evidence would be clear.
For example, wouldn't Inner-city school testing scores would be some of the highest in the nation?
It also assumes that the physician population is representative of the patient population and that patients are randomly assigned. Neither is likely to be the case.
You mention a study design that would allow to claim causation and not only correlation: a mother gives birth to many children with the help of doctors of different races.
But even if a mother has a single child, can’t we have an RCT? I.e. the doctor would be assigned “randomly.” Of course some people wouldn’t want to be included in that study, but for people who are willing, assuming we observe correlation, can’t we automatically claim causation?
I thought about mentioning this in the article but ended up not doing that because an RCT/random doctor assignment could work in theory, but it might be problematic in the American context since so many mothers have a preferred physician they want to be there and hospitals they want to be at for their birth. Because of this, it might not be feasible to run a large RCT, and places with random assignment of doctors might be places many mothers avoid, hurting generalizability.
Vinay Prasad evaluated a similar study claiming that in counties with more black doctors, black patients live longer: https://open.substack.com/pub/sensiblemed/p/does-black-representation-save-lives.
"All we can do to combat it in the long run is to cultivate a culture of radical data openness. Many people don’t want that, however, as they fear data will be misused."
Or perhaps we go back to appointing Court justices upon merit, rather than "check boxes" or political ideology. I point this comment to all the recent justices, not just Jackson. The statistical concepts you outline are tricky and even had me going back to my student days (after 50 year, one forgets much). However, that's not the big issue. The issue is that Jackson doesn't know what she doesn't know and therefore makes these errors of analysis and will do so in the future as she is guided by her ideology, rather than fact and reason.
Wasn’t this an effect of giving to the older, more experienced, more often white doctors those cases of higher risk?
There’s at least one critique of a study, maybe not this study, claiming a selection bias for assigning doctors in high risk cases.
No one has provided evidence to that effect to my knowledge.