This is an absolutely superb article for anyone who is interested in interpreting the scientific literature. I have been aware of most this kind of thing on an intuitive level, and obviously spend a lot of time wrangling with issues like fraud in my review of alcohol https://thingstoread.substack.com/p/is-alcohol-good-for-us and even more in my review of parapsychology https://thingstoread.substack.com/p/the-rising-star-of-parapsychology . My very favorite way to go is to investigate things personally and on my own, but I only have so much time and money for that, and it still leaves everyone else to wonder whether I'm falsifying my data or shopping around for effects like everyone else.
Ultimately reading your essay here, it looks like the thing to do is generally to accept evidence in an ascending hierarchy of credibility: no studies < small studies < large studies < meta-analyses < strong studies. Identifying the strong studies unfortunately may require a bit of savvy, and trying to convince clueless people of what you're doing won't always work. What you really, really want is a massive, meticulously designed, preregistered study - and even then, good luck that the people who carried it out were honest. It's quite a tragedy the way politicization and declining levels of trust in the developed world are leaving people without the time and talent for this sort of thing with fewer and fewer credible sources to rely upon in figuring out what to believe.
This is an absolutely superb article for anyone who is interested in interpreting the scientific literature. I have been aware of most this kind of thing on an intuitive level, and obviously spend a lot of time wrangling with issues like fraud in my review of alcohol https://thingstoread.substack.com/p/is-alcohol-good-for-us and even more in my review of parapsychology https://thingstoread.substack.com/p/the-rising-star-of-parapsychology . My very favorite way to go is to investigate things personally and on my own, but I only have so much time and money for that, and it still leaves everyone else to wonder whether I'm falsifying my data or shopping around for effects like everyone else.
Ultimately reading your essay here, it looks like the thing to do is generally to accept evidence in an ascending hierarchy of credibility: no studies < small studies < large studies < meta-analyses < strong studies. Identifying the strong studies unfortunately may require a bit of savvy, and trying to convince clueless people of what you're doing won't always work. What you really, really want is a massive, meticulously designed, preregistered study - and even then, good luck that the people who carried it out were honest. It's quite a tragedy the way politicization and declining levels of trust in the developed world are leaving people without the time and talent for this sort of thing with fewer and fewer credible sources to rely upon in figuring out what to believe.