Goes along with the huge lit on how people are bad at probabilities and normative reasoning generally, make conjunction fallacies etc.
I think that's interesting, but only half the picture: it just means that how we do reason is different from normative accounts in important ways and has different priorities (more pragmatic, social, contextual, anchored to available info, etc).
Task then becomes to work out what human judgements are for, when human judgements can shine best, etc, and how to use human and model best together
Indeed. Well known well before the AI era that algorithms work better, c.f. Checklist Manifesto. The fact that a superior solution already exists but is virtually unused because workers "know better" than an algorithm implies that AGI implementation will have higher barriers than getting the right answer.
Is there any reason to believe this wouldn’t hold for endeavors like venture capital? VCs place a ton of emphasis on unstructured interviews and some rather odd heuristics (Thiel heavily weights email response speed).
Tyler Cowen and Daniel Gross (a VC) have a book on screening and hiring in which they argue in favor of unstructured interviews. They present some empirical evidence on personality testing and IQ, but I never thought they offered a compelling argument that their clinical judgment could escape this curse.
A lesson universities learned the hard way when they got rid of standardized tests in the name of "equity". They seem to have forgotten why they were instituted in the first place.
Fascinating read! I’m Harrison, an ex fine dining industry line cook. My stack "The Secret Ingredient" adapts hit restaurant recipes (mostly NYC and L.A.) for easy home cooking. Dm me if interested in a recommendation swap — we’re growing fast!
Poor writing skills. You do not define the topic until paragraph 15. Why do you assume that your readers share your definition of the term "holistic judgment?"
"Humans do not accurately weight difference pieces of information in their heads to obtain optimal predictions"
In this case, I think holistic has no pre-specifiable definition. The definition totally depends upon the whole that the speaker has in mind. Are we talking about a whole human body, a whole nation, the whole human population? It is incumbent upon the speaker to clarify which hole we are discussing in any context. In this context, which factors are we discussing and including in the whole?
Incidentally, that is why holistic is such a marvelous weasel word, because one can say, "I think we have to approach this holistically," and sound very comprehensive without specifying what the hell is meant.
He is obviously not talking about a “whole human body”, or a “whole nation”. Context makes this clear. The context is admissions where students could be evaluated on many dimensions. Considering all of these dimensions (even those that should be disregarded) is holistic judgment. It’s evidently clear what the author is referring to.
Your last sentence is partly the point of the article, in that administrators have used the term to upweight irrelevant factors or plainly discriminate.
I think that it would be clearer to state that single factor judgments are often more useful than multi-factorial judgments. The point of this article is that only one factor, GMA, was ultimately shown to be predictive rather than a number of factors.
In this case, it turns out that GMA was the factor that should have been favored above all others, but that could not have been known for certain until the end of the experiment when the success of the various hires was evaluated, and the other factors were shown to be non-predictive.
I think we're talking about factors being considered in parallel rather than factors somehow operating as one, which would be a true definition of holistic.
It remains an open question whether the human mind can really operate holistically as opposed to multifactorially or as opposed to single-factorially masquerading as multifactorially.
I think there's a problem with your definition. Holistic comes from a Greek word holos, which means the whole, so its definition cannot be a consideration of distinct factors. From what the author of this post suggests it's the integration of the consideration of distinct factors.
What percent of college educated readers do you believe are familiar with the term? I have read the Washington Post and the New York Times daily for the last five years and I never ran across it. When you are addressing a general office audience it is simply good procedure as a writer to define a term in first two paragraphs.
What percent of college-educated readers here do you believe are familiar with quickly finding the meaning of unfamiliar terms, including their meaning in a particular context, using a dictionary or Google?
In this case, I think holistic has no pre-specifiable definition. The definition totally depends upon the whole that the speaker has in mind. Are we talking about a whole human body, a whole nation, the whole human population? It is incumbent upon the speaker to clarify which hole we are discussing in any context. In this context, which factors are we discussing and including in the whole?
Incidentally, that is why holistic is such a marvelous weasel word, because one can say, "I think we have to approach this holistically," and sound very comprehensive without specifying what the hell is meant.
Some on the right rail against elites and some on the right want to anoint themselves as the new elite, and they are absolutely certain that the rest of us don't count.
Goes along with the huge lit on how people are bad at probabilities and normative reasoning generally, make conjunction fallacies etc.
I think that's interesting, but only half the picture: it just means that how we do reason is different from normative accounts in important ways and has different priorities (more pragmatic, social, contextual, anchored to available info, etc).
Task then becomes to work out what human judgements are for, when human judgements can shine best, etc, and how to use human and model best together
For whatever reason, Government is ferociously hostile to evaluating people accurately -- i.e., with objective data and actuarial analysis.
"California ‘No Robo Bosses Act’ would bar AI from making personnel decisions
New legislation in California would prohibit employers from using automated decision-making systems in personnel management tasks." https://statescoop.com/california-no-robo-bosses-act-ai-personnel-decisions-2025/#:~:text=California%20state%20Sen.,termination%20decisions%20without%20human%20oversight.
>Doctors don't know what p-values are
>Doctors don't answer the breast cancer question well
I'm sure psychotherapists are much better.
Indeed. Well known well before the AI era that algorithms work better, c.f. Checklist Manifesto. The fact that a superior solution already exists but is virtually unused because workers "know better" than an algorithm implies that AGI implementation will have higher barriers than getting the right answer.
Is there any reason to believe this wouldn’t hold for endeavors like venture capital? VCs place a ton of emphasis on unstructured interviews and some rather odd heuristics (Thiel heavily weights email response speed).
Tyler Cowen and Daniel Gross (a VC) have a book on screening and hiring in which they argue in favor of unstructured interviews. They present some empirical evidence on personality testing and IQ, but I never thought they offered a compelling argument that their clinical judgment could escape this curse.
There's just good reason to believe it *would* hold.
This is an awesome article
If you publish the formula for something as high stakes as an admissions judgment, how do you prevent its being Goodharted?
Use criteria that are difficult to Goodhart, such as test scores, class ranks, and weighted GPAs.
A lesson universities learned the hard way when they got rid of standardized tests in the name of "equity". They seem to have forgotten why they were instituted in the first place.
Fascinating read! I’m Harrison, an ex fine dining industry line cook. My stack "The Secret Ingredient" adapts hit restaurant recipes (mostly NYC and L.A.) for easy home cooking. Dm me if interested in a recommendation swap — we’re growing fast!
check us out:
https://thesecretingredient.substack.com
Poor writing skills. You do not define the topic until paragraph 15. Why do you assume that your readers share your definition of the term "holistic judgment?"
"Humans do not accurately weight difference pieces of information in their heads to obtain optimal predictions"
I think it’s clear that “holistic” means to consider each factor when judging. The term is also widely used (especially since SFFA v Harvard).
In this case, I think holistic has no pre-specifiable definition. The definition totally depends upon the whole that the speaker has in mind. Are we talking about a whole human body, a whole nation, the whole human population? It is incumbent upon the speaker to clarify which hole we are discussing in any context. In this context, which factors are we discussing and including in the whole?
Incidentally, that is why holistic is such a marvelous weasel word, because one can say, "I think we have to approach this holistically," and sound very comprehensive without specifying what the hell is meant.
He is obviously not talking about a “whole human body”, or a “whole nation”. Context makes this clear. The context is admissions where students could be evaluated on many dimensions. Considering all of these dimensions (even those that should be disregarded) is holistic judgment. It’s evidently clear what the author is referring to.
Your last sentence is partly the point of the article, in that administrators have used the term to upweight irrelevant factors or plainly discriminate.
I think that it would be clearer to state that single factor judgments are often more useful than multi-factorial judgments. The point of this article is that only one factor, GMA, was ultimately shown to be predictive rather than a number of factors.
In this case, it turns out that GMA was the factor that should have been favored above all others, but that could not have been known for certain until the end of the experiment when the success of the various hires was evaluated, and the other factors were shown to be non-predictive.
I think we're talking about factors being considered in parallel rather than factors somehow operating as one, which would be a true definition of holistic.
It remains an open question whether the human mind can really operate holistically as opposed to multifactorially or as opposed to single-factorially masquerading as multifactorially.
I think there's a problem with your definition. Holistic comes from a Greek word holos, which means the whole, so its definition cannot be a consideration of distinct factors. From what the author of this post suggests it's the integration of the consideration of distinct factors.
What percent of college educated readers do you believe are familiar with the term? I have read the Washington Post and the New York Times daily for the last five years and I never ran across it. When you are addressing a general office audience it is simply good procedure as a writer to define a term in first two paragraphs.
What percent of college-educated readers here do you believe are familiar with quickly finding the meaning of unfamiliar terms, including their meaning in a particular context, using a dictionary or Google?
In this case, I think holistic has no pre-specifiable definition. The definition totally depends upon the whole that the speaker has in mind. Are we talking about a whole human body, a whole nation, the whole human population? It is incumbent upon the speaker to clarify which hole we are discussing in any context. In this context, which factors are we discussing and including in the whole?
Incidentally, that is why holistic is such a marvelous weasel word, because one can say, "I think we have to approach this holistically," and sound very comprehensive without specifying what the hell is meant.
Lol
Do you find Venn diagrams amusing? The question is always what is inside the circle and what isn't? What is included in the whole and what isn't?
Prestigious article 🧲💯
Did you mean prodigious? Who is assigning this prestige?
Elite human capital. I write about them sometimes 🧲💯🚀
It's amazing.
Some on the right rail against elites and some on the right want to anoint themselves as the new elite, and they are absolutely certain that the rest of us don't count.
You sound woke. Are you woke right or woke left?
Calling someone woke is as lazy as calling someone toxic. Do better.
you sound burdened by what has been
I am unbought and unbossed. I belong to no groups.