I have a degree in stats and I can follow what he did, though I wouldn't have thought to do what he did. It takes solid knowledge of statistics as well as domain knowledge to know what the right analytical moves in a field are, and this is not my field. But if you know what you want to and should do, an AI can give you the code to do the analysis instantly.
The broader problem is that the set of people who know the stats and the field is far far smaller than the set of people doing work in the field.
I actually felt some sympathy for the original authors when I read that comment. I'm not so sure that it's as easy as it is for them as opposed to Cremieux.
How about creating an LLM based tool that reads papers and flags specific or potential issues? It might not be perfect, but it could be a way to do a little stress testing at scale.
C, my guess is that social scientists are on average not so much lazy but comparably dumb and biased midwits. Sorry for my French. They don't earn much money either. Being able to write math proofs will take 98% of college graduates out of the running. After all, if you look at math BSc students - already a strong adverse selection - the most challenging thing they need to do is write proofs.
My own private joke is that if philosophy students needed to graduate as Maths BSc (after all, maths is a subspecialty of philosophy), it would have prevented the appearance of midwit philosophers. Maybe even Judith Butler, lol!
I wonder if you're a little blinded by your own mathematical aptitude. I have a PhD in molecular biology and I struggled to follow your methods here, let alone think about applying them in minutes to any dataset (I'm no longer active in the lab but when I was, I tended to enlist help from statisticians to interrogate my data). I definitely would fall into the ignorant category, and wonder if AI tools could be built to help bridge the gap for folks that don't have ready access to a mathematician to dig in properly...?
There is a problem that a lot of maths depends on intuitive understanding. AI would simply apply probabalistic adaptation of what is there in a segment of www. So if say 20% of users have used this test to do A, B or C, the AI would recommend it. C is operating on the level of an full professor level of biostatistics at a major university. Think of people like Bradley Efron etc...
Disputing nothing, just adding context: when last I checked – about five years ago – fixed effects were not part of the standard toolkit of personality or social psychologists. In PhD training for those disciplines, they were taught glancingly or not at all. So I’m not surprised to find that they don’t appear in articles even when there’s a good case to be made for them.
I have an accusation! "Trivially easy" varies from person to person. Cut it out!
Now, on remedial measures for the rest of us.
Remember SPSS? In its day it made a positive contribution.
If you want social science to improve, you must make it easy for others. Data checking packages are one thing, but hypotheses testing packages would help. I can remember the rise of confirmatory factor analyses, and as an editor of a special volume, saying that of course we had to use the procedure, though it seemed cumbersome and possibly too severe by the standards of the time. Jelte was right.
You outline most of this anyway, so nothing novel in my remarks. Just remember cultural lag. It takes us time to learn new tricks. Remedial education can work if there is plenty of training, and lots of help to get it right.
Finally, academia might be better off just dying a quiet death
I pretty much know exactly what I want to write before my fingers hit the keyboard. I've often been talking about the topic with friends in-person for a few days ahead of time.
Look now Mr. Cremieux, I can totally accept your conclusions about many social scientists either not having the right skills (or being too lazy to apply them. Although I don't think they're all lazy though, many work hard, with a different emphasis), but to say that this is because economists are more intelligent in general... It's not that long ago that evidence was a word most economists hadn't even heard of, and instead DSGE dreamworld modeling was the king of all economics, until that edifice came crashing down in the face of reality. It's good that there was a credibility revolution in economics, but it did take about 50 years for it to permeate the field, despite whatever superior intelligence you attribute to economists. And still one might wonder if everyone got the memo.... (And even now textbook introductions are full of falseties that only exist within fairytales of perfect competition, but that's a somewhat different issue).
So I think it's mostly cultural, in the sense that most will think these methods are good enough, and also the skills needed for more rigorous methods of statistical analysis are not widespread. Considering the importance of statistics in social science, educational programs could probably use a little more of it (and they should just teach R instead of relying on SPSS, but that's also a different story), but there are good reasons why social sciences should also emphasize non-mathematical knowledge and skills. It's good to be critical about this (and I get it, you need to draw their attention somehow, so an edgy title calling all social scientists lazy is fine), but intelligence is not the critical factor here.
Thanks; regarding "...one of the following: ignorant, lazy, biased, or publication farmers." it could be a set up for becoming an expert witness at a fancy trial.
I thank the Heavens I am able to ignore social medial 99% of the time. But many scientists are very lazy as they might do experiments already knowing the results they want. Still, the money is great, I bet.
Your statement that education doesn't impact intelligence has piqued my interest. This is my first read of your substack so please forgive me if you've talked about it previously. I just read that each year of school increases IQ scores by between 1 to 5 points. I'd love to hear why you think this isn't the case as I was skeptical as well but this is not my area of expertise.
Do you think your intelligence would have been affected by not having a single year of school? If you hadn’t learnt English or maths, wouldn’t that somewhat affect your measured IQ? I’m confused on this point.
"It was trivially easy to specify and run these models and to output the results."
I have 3 technical degrees from MIT and don't know enough statistics to follow your explanation.
Give yourself more credit.
I have a degree in stats and I can follow what he did, though I wouldn't have thought to do what he did. It takes solid knowledge of statistics as well as domain knowledge to know what the right analytical moves in a field are, and this is not my field. But if you know what you want to and should do, an AI can give you the code to do the analysis instantly.
The broader problem is that the set of people who know the stats and the field is far far smaller than the set of people doing work in the field.
I actually felt some sympathy for the original authors when I read that comment. I'm not so sure that it's as easy as it is for them as opposed to Cremieux.
The blog post doesn't describe what the model actually looks like, how can you expect to be able to understand what the tables are saying?
Sounds like there is a gap in the market for a specific career. A statistician who’s job it is to review & ok these studies
That's assuming there's demand...
How about creating an LLM based tool that reads papers and flags specific or potential issues? It might not be perfect, but it could be a way to do a little stress testing at scale.
haidt moment
AI will change this hopefully. Working on it.
C, my guess is that social scientists are on average not so much lazy but comparably dumb and biased midwits. Sorry for my French. They don't earn much money either. Being able to write math proofs will take 98% of college graduates out of the running. After all, if you look at math BSc students - already a strong adverse selection - the most challenging thing they need to do is write proofs.
My own private joke is that if philosophy students needed to graduate as Maths BSc (after all, maths is a subspecialty of philosophy), it would have prevented the appearance of midwit philosophers. Maybe even Judith Butler, lol!
I wonder if you're a little blinded by your own mathematical aptitude. I have a PhD in molecular biology and I struggled to follow your methods here, let alone think about applying them in minutes to any dataset (I'm no longer active in the lab but when I was, I tended to enlist help from statisticians to interrogate my data). I definitely would fall into the ignorant category, and wonder if AI tools could be built to help bridge the gap for folks that don't have ready access to a mathematician to dig in properly...?
There is a problem that a lot of maths depends on intuitive understanding. AI would simply apply probabalistic adaptation of what is there in a segment of www. So if say 20% of users have used this test to do A, B or C, the AI would recommend it. C is operating on the level of an full professor level of biostatistics at a major university. Think of people like Bradley Efron etc...
Disputing nothing, just adding context: when last I checked – about five years ago – fixed effects were not part of the standard toolkit of personality or social psychologists. In PhD training for those disciplines, they were taught glancingly or not at all. So I’m not surprised to find that they don’t appear in articles even when there’s a good case to be made for them.
By the way, Verbruggen at City Journal substack just quoted your study on smartphone use in a cautious, though positive way :D
Wait, that paper in the diabetes journal is actually just about calculating the area under a curve? I must be misunderstanding something
You didn't miss anything!
I have an accusation! "Trivially easy" varies from person to person. Cut it out!
Now, on remedial measures for the rest of us.
Remember SPSS? In its day it made a positive contribution.
If you want social science to improve, you must make it easy for others. Data checking packages are one thing, but hypotheses testing packages would help. I can remember the rise of confirmatory factor analyses, and as an editor of a special volume, saying that of course we had to use the procedure, though it seemed cumbersome and possibly too severe by the standards of the time. Jelte was right.
You outline most of this anyway, so nothing novel in my remarks. Just remember cultural lag. It takes us time to learn new tricks. Remedial education can work if there is plenty of training, and lots of help to get it right.
Finally, academia might be better off just dying a quiet death
James
Is the data used publicly available? I'd love to have a look myself.
It's restricted access.
Thank you.
How exactly do you manage to write dense and informative articles like this in under an hour?
Admittely this one isn't as dense as the ones on Nocebos and Gluten intolerance, but still, those were timed too..
I pretty much know exactly what I want to write before my fingers hit the keyboard. I've often been talking about the topic with friends in-person for a few days ahead of time.
Look now Mr. Cremieux, I can totally accept your conclusions about many social scientists either not having the right skills (or being too lazy to apply them. Although I don't think they're all lazy though, many work hard, with a different emphasis), but to say that this is because economists are more intelligent in general... It's not that long ago that evidence was a word most economists hadn't even heard of, and instead DSGE dreamworld modeling was the king of all economics, until that edifice came crashing down in the face of reality. It's good that there was a credibility revolution in economics, but it did take about 50 years for it to permeate the field, despite whatever superior intelligence you attribute to economists. And still one might wonder if everyone got the memo.... (And even now textbook introductions are full of falseties that only exist within fairytales of perfect competition, but that's a somewhat different issue).
So I think it's mostly cultural, in the sense that most will think these methods are good enough, and also the skills needed for more rigorous methods of statistical analysis are not widespread. Considering the importance of statistics in social science, educational programs could probably use a little more of it (and they should just teach R instead of relying on SPSS, but that's also a different story), but there are good reasons why social sciences should also emphasize non-mathematical knowledge and skills. It's good to be critical about this (and I get it, you need to draw their attention somehow, so an edgy title calling all social scientists lazy is fine), but intelligence is not the critical factor here.
Thanks; regarding "...one of the following: ignorant, lazy, biased, or publication farmers." it could be a set up for becoming an expert witness at a fancy trial.
I thank the Heavens I am able to ignore social medial 99% of the time. But many scientists are very lazy as they might do experiments already knowing the results they want. Still, the money is great, I bet.
Your statement that education doesn't impact intelligence has piqued my interest. This is my first read of your substack so please forgive me if you've talked about it previously. I just read that each year of school increases IQ scores by between 1 to 5 points. I'd love to hear why you think this isn't the case as I was skeptical as well but this is not my area of expertise.
The effects of education seem to be specific to certain skills but not to affect general intelligence, g.
Do you think your intelligence would have been affected by not having a single year of school? If you hadn’t learnt English or maths, wouldn’t that somewhat affect your measured IQ? I’m confused on this point.