In short: start with a dataset of question and answer pairs, where each question has been answered by two different LLMs. Ask the model you want to evaluate to choose the better answer for each pair. Then measure how consistently it selects winners. Does it reliably favor some models over the questions, or does it behave close to randomly? This consistency is a strong proxy for the model’s intelligence.
It is not subject to dataset leaks, lets you measure intelligence in many fields where you might not have golden answers, and converges pretty fast making it really cheap to measure.
Interesting, but couldn't a model "cheat" in this task by being very good at telling model outputs apart? How far do you get with a classifier simply trained to distinguish models by their output?
It seems to me many models - maybe by design - have a recognizable style which would be much easier to detect than evaluating the factual quality of answers.
In theory, yes! If this metric ever becomes a widely used standard, one would have to start accounting for that...
But, in practice, when asking a model to pick the best answer they see a single question / answers pair and focus on determining what they think is best.
It presumes some models are better than others (and we do find that providing data with a wide mix of model strengths improves convergence) but it does not need to be one model, and it does not even need to be transitive.
I like the high level idea! (how do we test intelligence in a non functional way?)
I'm effect, the different response types are measuring how the models respond to a context-free novel environment. I imagine humans would also respond on a variety of ways to this test, none of which are necessarily incorrect from the perspective of intelligence testing .
Many tests of human behavior (eg, n behavioral economics) create some pretense context to avoid boarding the response that is actually being measured. For example, we may invite a participant to a study of color preference, but actually measure how fast they complete the task when the scientist has/hasn't bathed in a week (or whatever).
Likewise, for llm intelligence testing, you could create pretext tasks and context, and perhaps measure what the model considered along the way, instead of the actual task outcome.
These aren't task free. They're just implicit task, "figure out what you're expected to do". These sort of riddle tasks are 100% dependent on who does the expecting.
This is not a new idea. Traditional IQ tests pivoted to them (they weren't originally like that), and no doubt they have great "discriminative power", because having the ability to figure out what's expected of you and not getting intimidated by cryptic and obtuse tasks put before you, are certainly extremely valuable skills in e.g business and politics.
But I always respected real tasks more. A question on a math test is honest; if it doesn't precisely define what's expected of you, the taskmaster has done a bad job, not you. It still can be extremely demanding.
An implicit task, by comparison, smells more of riddles, gnosticism. Do you know the way? Do you know the genre? (Once you know the genre of implicit tasks typical to IQ tests, you can easily increase your performance by a lot).
For that matter, this idea isn't new to machine learning either. Francois Chollet did it already, and he was IMO just as wrongheaded in thinking implicit tasks are somehow more indicative of "true intelligence" than explicit ones.
Aren't LLMs just super-powerful pattern matchers? And guessing "taps" a pattern recognition task? I am struggling to understand how your experiment relates to intelligence in any way.
Also, commercial LLMs generally have system instructions baked on top of the core models, which intrinsically prompt them to look for purpose even in random user prompts.
LLMs are pattern matchers, but every model is given specific instructions and response designs that influence what to do given unclear prompts. This is hugely valuable to understand since you may ask an LLM an invalid question and it is important to know if it is likely to guess at your intent, reject the prompt or respond randomly.
Understanding how LLMs fail differently is becoming more valuable than knowing that they all got 100% on some reasoning test with perfect context.
There's definitely more than "just" pattern matching in there - for example, current SOTA models 'plan ahead' to simultaneously process both rough outlines of an answer and specific subject details to then combine internally for the final result (https://www.anthropic.com/research/tracing-thoughts-language...).
This is very interesting. Especially the last part where it shows gpt-5.2 and gpt-oss and their very similar and unique outcome of being 90%+ Serious.
I tested this locally and got the same result with gpt-oss 120b. But only on the default 'medium' reasoning effort. When I used 'low' I kept getting more playful responses with emojis and when I used 'high' I kept getting more guessing responses.
I had a lot of fun with this and it provided me with more insight than I would have thought.
Game playing is the next frontier. Model economically valuable tasks as games and have the agents play/compete. Alphabench and Vendingbench show the potential of this approach.
A decade of reinforcement and agentic learning was spent playing games (Google Deepmind AlphaGo, AlphaStar, OpenAI Five), including against each other. So what makes it a new frontier?
Its application to LLMs to push capabilities. We're going to tap out expert feedback, and objective/competitive arenas are going to be the only way to progress at a reasonable speed.
The difference is going to be instead of starting from pre-existing games and hoping that "generalizes" to intelligence, this time people are going to build gamified simulators of economically valuable stuff. This is feasible now because we can use LLMs to help generate these games much faster than we would have been able to previously.
On alternative ways to measure LLM intelligence, we had good success with this: https://arxiv.org/abs/2509.23510
In short: start with a dataset of question and answer pairs, where each question has been answered by two different LLMs. Ask the model you want to evaluate to choose the better answer for each pair. Then measure how consistently it selects winners. Does it reliably favor some models over the questions, or does it behave close to randomly? This consistency is a strong proxy for the model’s intelligence.
It is not subject to dataset leaks, lets you measure intelligence in many fields where you might not have golden answers, and converges pretty fast making it really cheap to measure.
Interesting, but couldn't a model "cheat" in this task by being very good at telling model outputs apart? How far do you get with a classifier simply trained to distinguish models by their output?
It seems to me many models - maybe by design - have a recognizable style which would be much easier to detect than evaluating the factual quality of answers.
In theory, yes! If this metric ever becomes a widely used standard, one would have to start accounting for that...
But, in practice, when asking a model to pick the best answer they see a single question / answers pair and focus on determining what they think is best.
Doesn't that presume that one model dominates the other?
It presumes some models are better than others (and we do find that providing data with a wide mix of model strengths improves convergence) but it does not need to be one model, and it does not even need to be transitive.
I like the high level idea! (how do we test intelligence in a non functional way?)
I'm effect, the different response types are measuring how the models respond to a context-free novel environment. I imagine humans would also respond on a variety of ways to this test, none of which are necessarily incorrect from the perspective of intelligence testing .
Many tests of human behavior (eg, n behavioral economics) create some pretense context to avoid boarding the response that is actually being measured. For example, we may invite a participant to a study of color preference, but actually measure how fast they complete the task when the scientist has/hasn't bathed in a week (or whatever).
Likewise, for llm intelligence testing, you could create pretext tasks and context, and perhaps measure what the model considered along the way, instead of the actual task outcome.
These aren't task free. They're just implicit task, "figure out what you're expected to do". These sort of riddle tasks are 100% dependent on who does the expecting.
This is not a new idea. Traditional IQ tests pivoted to them (they weren't originally like that), and no doubt they have great "discriminative power", because having the ability to figure out what's expected of you and not getting intimidated by cryptic and obtuse tasks put before you, are certainly extremely valuable skills in e.g business and politics.
But I always respected real tasks more. A question on a math test is honest; if it doesn't precisely define what's expected of you, the taskmaster has done a bad job, not you. It still can be extremely demanding.
An implicit task, by comparison, smells more of riddles, gnosticism. Do you know the way? Do you know the genre? (Once you know the genre of implicit tasks typical to IQ tests, you can easily increase your performance by a lot).
For that matter, this idea isn't new to machine learning either. Francois Chollet did it already, and he was IMO just as wrongheaded in thinking implicit tasks are somehow more indicative of "true intelligence" than explicit ones.
Aren't LLMs just super-powerful pattern matchers? And guessing "taps" a pattern recognition task? I am struggling to understand how your experiment relates to intelligence in any way.
Also, commercial LLMs generally have system instructions baked on top of the core models, which intrinsically prompt them to look for purpose even in random user prompts.
LLMs are pattern matchers, but every model is given specific instructions and response designs that influence what to do given unclear prompts. This is hugely valuable to understand since you may ask an LLM an invalid question and it is important to know if it is likely to guess at your intent, reject the prompt or respond randomly.
Understanding how LLMs fail differently is becoming more valuable than knowing that they all got 100% on some reasoning test with perfect context.
There's definitely more than "just" pattern matching in there - for example, current SOTA models 'plan ahead' to simultaneously process both rough outlines of an answer and specific subject details to then combine internally for the final result (https://www.anthropic.com/research/tracing-thoughts-language...).
Eh that is still encompassed by the term “pattern matching” in this context. Sure it’s complicated, but it’s still just a glorified spell checker.
And we're just glorified oxidation. At some point the concept of "emergent systems" comes into play.
I'm an LLM naysayer, and even I have no trouble seeing, or accepting, that they're much more than glorified spell checkers.
This is very interesting. Especially the last part where it shows gpt-5.2 and gpt-oss and their very similar and unique outcome of being 90%+ Serious.
I tested this locally and got the same result with gpt-oss 120b. But only on the default 'medium' reasoning effort. When I used 'low' I kept getting more playful responses with emojis and when I used 'high' I kept getting more guessing responses.
I had a lot of fun with this and it provided me with more insight than I would have thought.
Game playing is the next frontier. Model economically valuable tasks as games and have the agents play/compete. Alphabench and Vendingbench show the potential of this approach.
A decade of reinforcement and agentic learning was spent playing games (Google Deepmind AlphaGo, AlphaStar, OpenAI Five), including against each other. So what makes it a new frontier?
Its application to LLMs to push capabilities. We're going to tap out expert feedback, and objective/competitive arenas are going to be the only way to progress at a reasonable speed.
The difference is going to be instead of starting from pre-existing games and hoping that "generalizes" to intelligence, this time people are going to build gamified simulators of economically valuable stuff. This is feasible now because we can use LLMs to help generate these games much faster than we would have been able to previously.
whats the assistant prompt being used for these? i dont think ive ever gotten these joking responses back to anything
Typo:
"The behvior summary"
[flagged]
tap tap tap tap tap
tap tap tap tap tap tap tap
tap tap tap tap tap tap tap tap tap tap tap