To be fair to his thesis, there are well-documented cases of African English linguistic quirks being picked up by LLMs and becoming quintessential ‘LLMisms’, the most famous probably being ‘delve’:
Now obviously this phenomenon surely isn’t so great as to explain a 100% AI result, but there isn’t *nothing* to this. It is plausible enough that African writing sounds like AI at least a little, especially to the human ear.
My understanding is that pangram is a neural network classifier, and it could possibly have learnt a heuristic like “if this text was clearly written before LLMs, output ‘human’”. Which would also explain pre-LLM texts not getting flagged.
It seemed to be AI-written to me, so when I saw some say Pangram told them "100% human" I used the free plan to check myself, and it said 29% AI. Others claimed Pangram told them "100% AI" and various other numbers, akin to rolling a D100. It could be that all those other people were just making it up, or not pasting in the whole article for some reason, but even if a lot of people are honestly holding it wrong, it at least means that you can't trust Pangram numbers that get reported without checking each discussed passage yourself.
This is extremely funny.
To be fair to his thesis, there are well-documented cases of African English linguistic quirks being picked up by LLMs and becoming quintessential ‘LLMisms’, the most famous probably being ‘delve’:
https://www.theguardian.com/technology/2024/apr/16/techscape-ai-gadgest-humane-ai-pin-chatgpt
Now obviously this phenomenon surely isn’t so great as to explain a 100% AI result, but there isn’t *nothing* to this. It is plausible enough that African writing sounds like AI at least a little, especially to the human ear.
That is what the arguments are, but none of this pre-LLM writing actually gets flagged as AI-generated despite them.
My understanding is that pangram is a neural network classifier, and it could possibly have learnt a heuristic like “if this text was clearly written before LLMs, output ‘human’”. Which would also explain pre-LLM texts not getting flagged.
The texts do not require date information to be classified. In fact, it doesn't change any judgments when it's included.
People are desperate to get out of admitting just how pervasive the practice has gotten, and worse still the demographic spread of said phony writers.
The whole thing is shameful. Particularly the demonisation of Pangram.
I lost some confidence in Pangram based on the Hacker News comments on this (otherwise unrelated) story: https://news.ycombinator.com/item?id=49857899
It seemed to be AI-written to me, so when I saw some say Pangram told them "100% human" I used the free plan to check myself, and it said 29% AI. Others claimed Pangram told them "100% AI" and various other numbers, akin to rolling a D100. It could be that all those other people were just making it up, or not pasting in the whole article for some reason, but even if a lot of people are honestly holding it wrong, it at least means that you can't trust Pangram numbers that get reported without checking each discussed passage yourself.
That could be the case. Luckily, I've preempted this concern by linking all of the Pangram reports.