RE nonhuman intelligence and g. My guess is that human intelligence has something to do with abstract thought enabled by recursion, and that this is unique to our species. Some suggestive evidence: language (even vocab size) is the single test most correlated with g, and abstract thought is one of the first things evidenced in Behavioral Modernity right before Homo Sapiens conquer the world (including the rest of Homo). There could also be many older biological considerations that also contribute to g, like general health or synapse speed. But the kernel of human g is unique, I think. That's also relevant to LLM "IQ," given their answers are produced by very different processes (which, importantly, are not built on top of recursive self-awareness).
Anyway, good treatment of human vs nonhuman intelligence.
Interesting. I am surprised by these results, mostly because of the blue line graph in my post which indicates that Claude-3 was much better at the easy answers than the hard answers (suggesting a strong correlation with how humans perform on the questions.) https://www.maximumtruth.org/p/ais-ranked-by-iq-ai-passes-100-iq
If you have human test data, could you correlate human percentages correct with Claude's percentages correct? Mine looked like: https://postimg.cc/06DSbqdt.
When I look at your plot, I see a Spearman correlation of 0.60 between 1-ordered difficulty and percent correct. In my data, the Spearman correlation between correctness and the IRT difficulty parameter is 0.64 for the no training group, 0.79 for the easy training group, 0.68 for the full training group, and 0.87 for the difficult training group, versus 0.15 for Claude 3, and 0.30 for the correlation between difficulties and discriminations. Human correctness is correlated at -0.16, 0.09, -0.09, and 0.36 with the discriminations, versus 0.02 for Claude 3's correctness, so at least that's similar, but it's indeterminate.
Where humans seemed to fare worse with more rules (https://postimg.cc/SnZvdRRv), Claude seemed to fare randomly. I'm not sure what's up.
Regardless, it's still not clear how to compare LLM intelligence test performance to human intelligence test performance and I have strong doubts that they're comparable even if LLM and human difficulties can be made to ordinally match up.
Enjoyed reading the article. But one criticism I have is it seems to answer a question that nobody is asking. I don't think anyone believes that a 100 IQ LLM is exactly the same as a 100 IQ human. When it came to going into the details of how they are different, I feel the article came up short.
One clarification here. I also liked the part about differentiating between memorization and intelligence in humans. But are we claiming that LLMs are doing memorization? If so, why wouldn't they score much higher on IQ-tests? Note that Claude-3 is the only LLM to have cracked 100 points, and the previous best was GPT-4 at 85 points. And if they aren't doing memorization, can't whatever they are doing be called reasoning? Is there a 3rd process of gaining knowledge besides memorization and reasoning?
No, it was not claimed that what LLMs are doing is memorization. The distinction between memory and intelligence was brought up to show that different things predict different things. If that were the claim, I still don't see why it would follow that LLMs must do very well on IQ tests when we don't have the training data to see what test-relevant information they've been shown. In the case of my data, I'm reasonably certain it wasn't exactly in the training data, but similar content probably was.
And no, ¬memorization != reasoning and there are certainly many ways to obtain knowledge besides memorization and reasoning, even in humans. Our language learning instincts provide an example of that.
I think I have figured out the source of the confusion. It is in the conflation of "reasoning" with "intelligence". While I agree that ¬memorization doesn't imply reasoning, because there can be other techniques like pattern-matching, I would argue it does imply intelligence.
That's why in the AI world we mainly care about training dataset leakage while testing LLMs. Having seen the training data is basically a proxy for memorization, and it's something we don't want. But if the LLM performs well on unseen data, we don't care HOW it gets the answer, the fact that it does is all that matters. (In ML terminology, this is called generalization accuracy).
Pet peeve: I get very annoyed when people say LLMs aren't doing "real reasoning" without simultaneously explaining why the reasoning performed by humans is "real". If this keeps happening I might go full Scott Aaronson and starting calling people meat chauvinists. 8)
✅The closest non-human #resemblance to the #growth in #humans are #plants.
👇🏽Here’s why…
➡️💨🔥The righteous are like ‘Trees’ Proverbs 4:18; Jeremiah 17:8; Psalm 92:12-15; ‘Jesus is Peace’; ‘Righteous Lion’ Proverbs 28:1; and
‘Longing Fulfilled Tree’ Proverbs 13:12
✅Jesus compares having #Faith to a #mustardSEED
💨Matthew 17:20
”And Jesus said unto them, Because of your unbelief: for verily I say unto you, If ye have faith as a grain of mustard seed, ye shall say unto this mountain, Remove hence to yonder place; and it shall remove; and nothing shall be impossible unto you.“
Looks like footnote 2 cuts out mid sentence. Curious what the rest says!
Fixed!
Interesting! For a (related) argument that the kind of intelligence/wisdom we associate with being human is irreducible, see https://tmfow.substack.com/p/artificial-intelligence-and-living
RE nonhuman intelligence and g. My guess is that human intelligence has something to do with abstract thought enabled by recursion, and that this is unique to our species. Some suggestive evidence: language (even vocab size) is the single test most correlated with g, and abstract thought is one of the first things evidenced in Behavioral Modernity right before Homo Sapiens conquer the world (including the rest of Homo). There could also be many older biological considerations that also contribute to g, like general health or synapse speed. But the kernel of human g is unique, I think. That's also relevant to LLM "IQ," given their answers are produced by very different processes (which, importantly, are not built on top of recursive self-awareness).
Anyway, good treatment of human vs nonhuman intelligence.
Interesting. I am surprised by these results, mostly because of the blue line graph in my post which indicates that Claude-3 was much better at the easy answers than the hard answers (suggesting a strong correlation with how humans perform on the questions.) https://www.maximumtruth.org/p/ais-ranked-by-iq-ai-passes-100-iq
Not sure how to reconcile those findings.
If you have human test data, could you correlate human percentages correct with Claude's percentages correct? Mine looked like: https://postimg.cc/06DSbqdt.
When I look at your plot, I see a Spearman correlation of 0.60 between 1-ordered difficulty and percent correct. In my data, the Spearman correlation between correctness and the IRT difficulty parameter is 0.64 for the no training group, 0.79 for the easy training group, 0.68 for the full training group, and 0.87 for the difficult training group, versus 0.15 for Claude 3, and 0.30 for the correlation between difficulties and discriminations. Human correctness is correlated at -0.16, 0.09, -0.09, and 0.36 with the discriminations, versus 0.02 for Claude 3's correctness, so at least that's similar, but it's indeterminate.
Where humans seemed to fare worse with more rules (https://postimg.cc/SnZvdRRv), Claude seemed to fare randomly. I'm not sure what's up.
Regardless, it's still not clear how to compare LLM intelligence test performance to human intelligence test performance and I have strong doubts that they're comparable even if LLM and human difficulties can be made to ordinally match up.
I don't have such human test data, though the people at Mensa Norway probably do; maybe I'll ask them for it.
Enjoyed reading the article. But one criticism I have is it seems to answer a question that nobody is asking. I don't think anyone believes that a 100 IQ LLM is exactly the same as a 100 IQ human. When it came to going into the details of how they are different, I feel the article came up short.
One clarification here. I also liked the part about differentiating between memorization and intelligence in humans. But are we claiming that LLMs are doing memorization? If so, why wouldn't they score much higher on IQ-tests? Note that Claude-3 is the only LLM to have cracked 100 points, and the previous best was GPT-4 at 85 points. And if they aren't doing memorization, can't whatever they are doing be called reasoning? Is there a 3rd process of gaining knowledge besides memorization and reasoning?
No, it was not claimed that what LLMs are doing is memorization. The distinction between memory and intelligence was brought up to show that different things predict different things. If that were the claim, I still don't see why it would follow that LLMs must do very well on IQ tests when we don't have the training data to see what test-relevant information they've been shown. In the case of my data, I'm reasonably certain it wasn't exactly in the training data, but similar content probably was.
And no, ¬memorization != reasoning and there are certainly many ways to obtain knowledge besides memorization and reasoning, even in humans. Our language learning instincts provide an example of that.
Appreciate the response.
I think I have figured out the source of the confusion. It is in the conflation of "reasoning" with "intelligence". While I agree that ¬memorization doesn't imply reasoning, because there can be other techniques like pattern-matching, I would argue it does imply intelligence.
That's why in the AI world we mainly care about training dataset leakage while testing LLMs. Having seen the training data is basically a proxy for memorization, and it's something we don't want. But if the LLM performs well on unseen data, we don't care HOW it gets the answer, the fact that it does is all that matters. (In ML terminology, this is called generalization accuracy).
Pet peeve: I get very annoyed when people say LLMs aren't doing "real reasoning" without simultaneously explaining why the reasoning performed by humans is "real". If this keeps happening I might go full Scott Aaronson and starting calling people meat chauvinists. 8)
A problem: genuine novel utility that while not super powerful on its own yet, can be leveraged to create power that was formerly not possible.
Language is a hell of a drug!
✅The closest non-human #resemblance to the #growth in #humans are #plants.
👇🏽Here’s why…
➡️💨🔥The righteous are like ‘Trees’ Proverbs 4:18; Jeremiah 17:8; Psalm 92:12-15; ‘Jesus is Peace’; ‘Righteous Lion’ Proverbs 28:1; and
‘Longing Fulfilled Tree’ Proverbs 13:12
✅Jesus compares having #Faith to a #mustardSEED
💨Matthew 17:20
”And Jesus said unto them, Because of your unbelief: for verily I say unto you, If ye have faith as a grain of mustard seed, ye shall say unto this mountain, Remove hence to yonder place; and it shall remove; and nothing shall be impossible unto you.“
https://bible.com/bible/1/mat.17.20.KJV
✅Plants #communicate with each other
👇🏽👇🏽👇🏽
https://duckduckgo.com/?q=talking+plants&ia=web
✅Plants have #feelings and can respond to #touch
👇🏽👇🏽👇🏽
https://duckduckgo.com/?q=plants+have+feeling+and+respond+to+touch&ia=web
✅Plants have #personality
#Entomology #Nematology
👇🏽👇🏽👇🏽
https://duckduckgo.com/?q=plants+have+personality&ia=web
✅Plants #listen
👇🏽👇🏽👇🏽
https://duckduckgo.com/?q=plants+listen+for+danger&ia=web
💨People are like soil, and words (i.e., communication, songs, hymns,
disagreements/agreements, affirmations, and confessions) are the seeds planted within that grow into ideas and beliefs.
✅According to Matthew 13, Mark 4:1-9, and Luke 8:4-8 people produce a crop that sustains or dissipates.
✅Moms plant seeds within the womb before the child is even born.
👇🏽👇🏽👇🏽
https://duckduckgo.com/?q=maternity+impacts+baby+mental+health&ia=web
The #environment influences the child's #development from the #womb into #adulthood through #epigenetics.
👇🏽👇🏽👇🏽
https://duckduckgo.com/?q=the+environment+influences+the+child%27s+development+from+the+womb+into+adulthood+through+%23epigenetics.&ia=web
✅#Humanity is impacted
by the transmission of primary #perceptions, #understanding, and #discernment, including but not
limited to secondary #consciousness of #COperceptions, #COunderstanding, and #COdiscernment, which affects future #outcomes of #families.
This article was a great read but we’re not that far from understanding #TRUEintelligence only made possible through the indwelling of #HOLYSPIRIT
✅It’s so much we know and still so much to learn and understand 🤔
I’m fascinated with #learning— it’s still so much I need to learn. Humanity need to want to learn more and more in order to continue thriving, Amen 🙏🏾