Yes! "Training a model without testing it" is exactly what so much social science research amounts to. Plotting the marginal "fitted values," i.e., predictions, of a model as a function of some feature to make some point about what has been learned from the data, but without ever actually, you know, testing the model's predictions out-of-sample.
This article is imo entirely correct on methods, but I also want to talk about how to improve the chosen topics.
Back when physics was in its infancy, it started with quantifying things that people have already noticed. For example, gravity. People had already noticed that things tend to fall down, rather than up, or the movements of celestial bodies. The hard part was generating a specific formula on how it works in-detail. Only *after* considerable hard work we found that intuitions on at least some points were wrong, such as that the force of gravity is actually identical between differently-weighted objects. But many if not most intuitions turned out roughly correct! Likewise for many other things; The basic motivating question for many researchers was extremely often "why is this thing, which we all can perceive and all agree on, the way it is?". This does not preclude eventually overturning some intuitions, but that is generally much further down the line!
In social sciences however, I regularly see even honest students and researchers start from the assumption that all "folk theories" are wrong and they only try to work out how they are wrong. Upturning intuitions is their entire raison d'etre. If you don't get "surprising" results, you're at best boring and uncool or at worst vilified. Best example is imo stereotypes; If I ask friends who literally studied psychology, many will declare with confidence that all stereotypes are wrong and have been thoroughly debunked by psychological research. Yet stereotype accuracy is one of the most robust and replicable findings there is. But psychologists shouldn't stop there! We need to quantify how accurate they are, we need to find how different parameters influence accuracy, and so on. The good news is, this is at least being worked on by a small group of researchers; The bad news is, many others try their best to entirely ignore these findings.
To be frank, you can't do science this way. You should start from the things that regular society can broadly agree on AND which is backed up by replicable, strong findings- stereotype accuracy, the primacy of selection effects in education, the robustness of intelligence measures, and so on. From there, you can work yourself towards more contentious, variable or difficult topics. But you shouldn't start from those.
This avoids another problem we could see here: If you start out with investigating the things that most people have been agreeing on for a long time, an argument like "maybe attitudes changed, dunno" will seem non-credible. And this is very good, since we definitely shouldn't waste money on findings that very plausibly won't last even just a few years.
Nothing is a panacea. But what I love the most about pre-registration is that the researchers are FORCED to publish whatever results they find. They cannot just shove it under the table. For studies such as stereotype threat or police treatment of minorities, most of these liberal academics would just quietly not publish any result that goes against the prevailing narrative that blacks are being discriminated against in a systemic way. That's simply not the case.
You seem to have entirely missed the point that preregistration does not fix this in favor of Waging The Good Fight. You're free to spend your time how you like, but you're commenting below an article that explains how preregistration is an insufficient fix for the issue of malicious or self-fooling researchers futzibg with data until it says the right thing.
It seems to me that science should be modularized, with experiment designers doing preregistrations crowdsourcing from experimentalists. This ensures that preregistrations are sufficiently detailed and removes the incentive where you want your own hypothesis to be true. You can even include prediction markets in this system, selecting which datasets to award and include for analysis based on bets of whether or not they replicate.
Gotta mention the (journal-published) paper by Ioannidis - "Most findings published in academic journals are false" or similar title. Short and informative.
Your article builds sooo nicely on this. Taking it further, into considering how the issues might get fixed, and how research might get focused on acquiring and sharing meaningful useful knowledge.
Think you should get it published in a journal - maybe the same one where Ioannidis published his...
Yes! "Training a model without testing it" is exactly what so much social science research amounts to. Plotting the marginal "fitted values," i.e., predictions, of a model as a function of some feature to make some point about what has been learned from the data, but without ever actually, you know, testing the model's predictions out-of-sample.
This article is imo entirely correct on methods, but I also want to talk about how to improve the chosen topics.
Back when physics was in its infancy, it started with quantifying things that people have already noticed. For example, gravity. People had already noticed that things tend to fall down, rather than up, or the movements of celestial bodies. The hard part was generating a specific formula on how it works in-detail. Only *after* considerable hard work we found that intuitions on at least some points were wrong, such as that the force of gravity is actually identical between differently-weighted objects. But many if not most intuitions turned out roughly correct! Likewise for many other things; The basic motivating question for many researchers was extremely often "why is this thing, which we all can perceive and all agree on, the way it is?". This does not preclude eventually overturning some intuitions, but that is generally much further down the line!
In social sciences however, I regularly see even honest students and researchers start from the assumption that all "folk theories" are wrong and they only try to work out how they are wrong. Upturning intuitions is their entire raison d'etre. If you don't get "surprising" results, you're at best boring and uncool or at worst vilified. Best example is imo stereotypes; If I ask friends who literally studied psychology, many will declare with confidence that all stereotypes are wrong and have been thoroughly debunked by psychological research. Yet stereotype accuracy is one of the most robust and replicable findings there is. But psychologists shouldn't stop there! We need to quantify how accurate they are, we need to find how different parameters influence accuracy, and so on. The good news is, this is at least being worked on by a small group of researchers; The bad news is, many others try their best to entirely ignore these findings.
To be frank, you can't do science this way. You should start from the things that regular society can broadly agree on AND which is backed up by replicable, strong findings- stereotype accuracy, the primacy of selection effects in education, the robustness of intelligence measures, and so on. From there, you can work yourself towards more contentious, variable or difficult topics. But you shouldn't start from those.
This avoids another problem we could see here: If you start out with investigating the things that most people have been agreeing on for a long time, an argument like "maybe attitudes changed, dunno" will seem non-credible. And this is very good, since we definitely shouldn't waste money on findings that very plausibly won't last even just a few years.
Nothing is a panacea. But what I love the most about pre-registration is that the researchers are FORCED to publish whatever results they find. They cannot just shove it under the table. For studies such as stereotype threat or police treatment of minorities, most of these liberal academics would just quietly not publish any result that goes against the prevailing narrative that blacks are being discriminated against in a systemic way. That's simply not the case.
You seem to have entirely missed the point that preregistration does not fix this in favor of Waging The Good Fight. You're free to spend your time how you like, but you're commenting below an article that explains how preregistration is an insufficient fix for the issue of malicious or self-fooling researchers futzibg with data until it says the right thing.
It seems to me that science should be modularized, with experiment designers doing preregistrations crowdsourcing from experimentalists. This ensures that preregistrations are sufficiently detailed and removes the incentive where you want your own hypothesis to be true. You can even include prediction markets in this system, selecting which datasets to award and include for analysis based on bets of whether or not they replicate.
Gotta mention the (journal-published) paper by Ioannidis - "Most findings published in academic journals are false" or similar title. Short and informative.
Your article builds sooo nicely on this. Taking it further, into considering how the issues might get fixed, and how research might get focused on acquiring and sharing meaningful useful knowledge.
Think you should get it published in a journal - maybe the same one where Ioannidis published his...
Very nicely written!
Populism and science don't work together efficiently