The turkey is fed by the farmer every morning at 9 AM.
Day 1: Fed. (Inductive confidence rises)
Day 100: Fed. (Inductive confidence is near 100%)
Day 250: The farmer comes at 9 AM... and cuts its throat. Happy thanksgiving.
The Turkey was an LLM. It predicted the future based entirely on the distribution of the past. It had no "understanding" of the purpose of the farmer.
This is why Meyer's "American/Inductive" view is dangerous for critical software. An LLM coding agent is the Inductive Turkey example. It writes perfect code for 1000 days because the tasks match the training data. On Day 1001, you ask for something slightly out of distribution, and it confidently deletes your production database because it added a piece of code that cleans your tables.
Humans are inductive machines, for the most part, too. The difference is that, fortunately, fine-tuning them is extremely easy.
That’s where Bayesian reasoning comes into play, where there are prior assumptions (e.g., that engineered reality is strongly biased towards simple patterns) which make one of these hypotheses much more likely than the other.
> The Turkey was an LLM. It predicted the future based entirely on the distribution of the past. It had no "understanding" of the purpose of the farmer.
But we already know that LLMs can do much better than that. See the famous “grokking” paper[1], which demonstrates that with sufficient training, a transformer can learn a deep generalization of its training data that isn’t just a probabilistic interpolation or extrapolation from previous inputs.
Many of the supposed “fundamental limitations” of LLMs have already been disproven in research. And this is a standard transformer architecture; it doesn’t even require any theoretical innovation.
I'm a believer that LLMs will keep getting better. But even today (which might or might not be "sufficient" training) they can easily run `rm -rf ~`.
Not that humans can't make these mistakes (in fact, I have nuked my home directory myself before), but I don't think it's a specific problem some guardrails can solve currently. I'm looking for innovations (either model-wise or engineering-wise) that'd do better than letting an agent run code until a goal is seemingly achieved.
LLM's have surpassed being Turing machines? Turing machines now think?
LLM's are known properties in that they are an algorithm! Humans are not. PLEASE at the very least grant that the jury is STILL out on what humans actually are in terms of their intelligence, that is after all what neuroscience is still figuring out.
You clearly underestimate the quality of people I have seen and worked with.
And yes guard rails can be added easily.
Security is my only concern and for that we have a team doing only this but that's also just a question of time.
Whatever LLMs ca do today doesn't matter. It matters how fast it progresses and we will see if we still use LLMs in 5 years or agi or some kind of world models.
> You clearly underestimate the quality of people I have seen and worked with.
I'm not sure what you're referring to. I didn't say anything about capabilities of people. If anything, I defend people :-)
> And yes guard rails can be added easily.
Do you mean models can be prevented to do dumb things? I'm not too sure about that, unless a strict software architecture is engineered by humans where LLMs simply write code and implement features. Not everything is web development where we can simply lock filesystems and prod database changes. Software is very complex across the industry.
> You clearly underestimate the quality of people I have seen and worked with
"Humans aren't perfect"
This argument always comes up. The existence of stupid / careless / illiterate people in the workplace doesn't excuse spending trillions on computer systems which use more energy than entire countries and are yet unreliable
I agree with Dijkstra on this one: “The question of whether a computer can think is no more interesting than the question of whether a submarine can swim.”
I really wish all these LessWrong, what is the meaning of intelligence types cared enough to study Wittgenstein a bit rather than hear themselves talk; it would save us all a lot of time.
Intelligence as described is not the entire "requirement" for intelligence. There are probably more layers here, but I see "intelligence" as the 2nd layer, and beneath that later is comprehension which is the ability to discriminate between similar things, even things trying to decieve you. And at layer zero is the giant mechanism pushing this layered form of intelligence found in living things is the predator / prey dynamic that dictates being alive or food for something remaining alive.
"Intelligence in AI" lacks any existential dynamic, our LLMs are literally linguistic mirrors of human literature and activity tracks. They are not intelligent, but for the most part we can imagine they are, while maintaining sharp critical analysis because they are idiot savants in the truest sense.
Two concepts of intelligence and neither have remotely anything to do with real intelligence, academics sure like to play with words. I suppose this is how they justify their own existence; in the absence of being intelligent enough to contribute anything of value, they must instead engage in wordplay that obfuscates the meaning of words to the point nobody understands what the hell they're talking about, and confuses the lack of understanding of what they're talking about for the academics being more intelligent than the reader.
Intelligence, in the real world, is the ability to reason about logic. If 1 + 1 is 2, and 1 + 2 is 3, then 1 + 3 must be 4. This is deterministic, and it is why LLMs are not intelligent and can never be intelligent no matter how much better they get at superficially copying the form of output of intelligence. Probabilistic prediction is inherently incompatible with deterministic deduction. We're years into being told AGI is here (for whatever squirmy value of AGI the hype huckster wants to shill), and yet LLMs, as expected, still cannot do basic arithmetic that a child could do without being special-cased to invoke a tool call. How is it that we can go about ignoring reality for so long?
> With recent advances in AI, it becomes ever harder for proponents of intelligence-as-understanding to continue asserting that those tools have no clue and “just” perform statistical next-token prediction.
??????? No, that is still exactly what they do. The article then lists a bunch of examples in which this in trivially exactly what is happening.
> “The cat chased the . . .” (multiple connections are plausible, so how is that not understanding probability?)
It doesn't need to "understand" probability. "The cat chased the mouse" shows up in the distribution 10 times. "The cat chased the bird" shows up in the distribution 5 times. Absent any other context, with the simplest possible model, it now has a probability of 2/3 for the mouse and 1/3 for the bird. You can make the probability calculations as complex as you want, but how could you possibly trout this out as an example that an LLM completing this sentence isn't a matter of trivial statistical prediction? Academia needs an asteroid, holy hell.
[I originally edited this into my post, but two people had replied by then, so I've split it off into its own comment.]
One question is how do you know that you (or humans in general) aren't also just applying statistical language rules, but are convincing yourself of some underlying narrative involving logical rules? I don't know the answer to this.
I keep coming back to this. The most recent version of chatgpt I tried was able to tell me how many letter 'r's were in a very long string of characters only by writing and executing a python script to do this. Some people say this is impressive, but any 5 year old could count the letters without knowing any python.
The calculations are internal but they happen due to the orchestration of specific parts of the brain. That is to ask, why can't we consider our brains to be using their own internal tools?
I certainly don't think about multiplying two-digit numbers in my head in the same manner as when playing a Dm to a G7 chord that begs to resolve to a C!
Intelligence is not just about reasoning with logic. Computers are already made to do that.
The key thing is modeling. You must model a situation in a useful way in order to apply logic to it. And then there is intention, which guides the process.
Our computer programs execute logic, but cannot reason about it. Reasoning is the ability to dynamically consider constraints we've never seen before and then determine how those constraints would lead to a final conclusion. The rules of mathematics we follow are not programmed into our DNA; we learn them and follow them while our human-programming is actively running. But we can just as easily, at any point, make up new constraints and follow them to new conclusions. What if 1 + 2 is 2 and 1 + 3 is 3? Then we can reason that under these constraints we just made up, 1 + 4 is 4, without ever having been programmed to consider these rules.
It's the same kind of schizm that has lead to a lot of hate and mass murder over the last century or so:
Abstraction/dimensionality reduction vs. concrete logic and statistics
Concrete statistician: I can learn the problem in its full complexity, unlike the dumdum below me.
Abstract thinker: I understand it, because I can reduce its dimensity to a small number of parameters.
CS: I can predict this because I have statistics about its past behavior.
AT: I told you so.
CS: You couldn't possibly know this, because it has never happened before. You suffer from the hindsight bias.
AT: But I told you.
CS: It has never happened, you couldn't possibly have statistics of when such things occur. You were just lucky.
CS: I'm smart, I can be taught anything
AT: You are stupid because you need to be taught everything.
War (or another sort of mass death or other kind of suffering) emerges.
Memory foam doesn't really "remember" the shape of my rear end but we all understand the language games at play when we use that term.
The problem with the AI discourse is that the language games are all mixed up and confused. We're not just talking about capability, we're talking about significance too.
The author defines American style intelligence as "the ability to adapt to new situations, and learn from experience".
Then argues that the current type of machine-learning driven AI is American style-intelligent because it is inductive, which is not what was supposedly (?) being argued for.
Of course current AI/ML models cannot adapt to new situations and learn from experience, outside the scope of its context window, without a retraining or fine-tuning step.
intelligence/understanding is when one can postulate/predict/calculate/presume something correctly, from concepts about it, without that thing (or similar) ever been in the training/past (or even ever-known).
Yeah, not all humans do it. It's too energy expensive, biological efficiency wins.
As of ML.. Maybe next time, when someone figures out how to combine deductive with inductive, in zillion small steps, with falsifying built-in.. (instead of confronting them 100% one against 100% the other)
I'd argue you can have much more precise definition than that. My definition of intelligence would be a system that has an internal of a particular domain, and it uses this simulation to guide its actions within that domain. Being able to explain your actions is derived directly from having a model of the environment.
For example, we all have an internal physics model in our heads that's build up through our continuous interaction with our environment. That acts as our shared context. That's why if I tell you to bring me a cup of tea, I have a reasonable expectation that you understand what I requested and can execute this action intelligently. You have a conception of a table, of a cup, of tea, and critically our conception is similar enough that we can both be reasonably sure we understand each other.
Incidentally, when humans end up talking about abstract topics, they often run into exact same problem as LLMs, where the context is missing and we can be talking past each other.
The key problem with LLMs is that they currently lack this reinforcement loop. The system merely strings tokens together in a statistically likely fashion, but it doesn't really have a model of the domain it's working in to anchor them to.
In my opinion, stuff like agentic coding or embodiment with robotics moves us towards genuine intelligence. Here we have AI systems that have to interact with the world, and they get feedback on when they do things wrong, so they can adjust their behavior based on that.
Marx is fair play, but one of the most prominent cases of understanding everything in advance is undoubtedly Chomsky's theory of innate/universal grammar, which became completely dominant on guess which side of the pond.
The turkey is fed by the farmer every morning at 9 AM.
Day 1: Fed. (Inductive confidence rises)
Day 100: Fed. (Inductive confidence is near 100%)
Day 250: The farmer comes at 9 AM... and cuts its throat. Happy thanksgiving.
The Turkey was an LLM. It predicted the future based entirely on the distribution of the past. It had no "understanding" of the purpose of the farmer.
This is why Meyer's "American/Inductive" view is dangerous for critical software. An LLM coding agent is the Inductive Turkey example. It writes perfect code for 1000 days because the tasks match the training data. On Day 1001, you ask for something slightly out of distribution, and it confidently deletes your production database because it added a piece of code that cleans your tables.
Humans are inductive machines, for the most part, too. The difference is that, fortunately, fine-tuning them is extremely easy.
"fine-tuning them is extremely easy." Criminal courts, jails, mental asylums beg to disagree.
This issue happens at the edge of every induction. These two rules support their data equally well:
data: T T T T T T F
rule1: for all i: T
rule2: for i < 7: T else F
That’s where Bayesian reasoning comes into play, where there are prior assumptions (e.g., that engineered reality is strongly biased towards simple patterns) which make one of these hypotheses much more likely than the other.
yes, if you decide one of them is much more likely without reference to the data, then it will be much more likely :)
LLM’s seem to know about farmers and turkeys though.
> The Turkey was an LLM. It predicted the future based entirely on the distribution of the past. It had no "understanding" of the purpose of the farmer.
But we already know that LLMs can do much better than that. See the famous “grokking” paper[1], which demonstrates that with sufficient training, a transformer can learn a deep generalization of its training data that isn’t just a probabilistic interpolation or extrapolation from previous inputs.
Many of the supposed “fundamental limitations” of LLMs have already been disproven in research. And this is a standard transformer architecture; it doesn’t even require any theoretical innovation.
[1] https://arxiv.org/abs/2301.02679
I'm a believer that LLMs will keep getting better. But even today (which might or might not be "sufficient" training) they can easily run `rm -rf ~`.
Not that humans can't make these mistakes (in fact, I have nuked my home directory myself before), but I don't think it's a specific problem some guardrails can solve currently. I'm looking for innovations (either model-wise or engineering-wise) that'd do better than letting an agent run code until a goal is seemingly achieved.
LLM's have surpassed being Turing machines? Turing machines now think?
LLM's are known properties in that they are an algorithm! Humans are not. PLEASE at the very least grant that the jury is STILL out on what humans actually are in terms of their intelligence, that is after all what neuroscience is still figuring out.
You clearly underestimate the quality of people I have seen and worked with. And yes guard rails can be added easily.
Security is my only concern and for that we have a team doing only this but that's also just a question of time.
Whatever LLMs ca do today doesn't matter. It matters how fast it progresses and we will see if we still use LLMs in 5 years or agi or some kind of world models.
> You clearly underestimate the quality of people I have seen and worked with.
I'm not sure what you're referring to. I didn't say anything about capabilities of people. If anything, I defend people :-)
> And yes guard rails can be added easily.
Do you mean models can be prevented to do dumb things? I'm not too sure about that, unless a strict software architecture is engineered by humans where LLMs simply write code and implement features. Not everything is web development where we can simply lock filesystems and prod database changes. Software is very complex across the industry.
> You clearly underestimate the quality of people I have seen and worked with
"Humans aren't perfect"
This argument always comes up. The existence of stupid / careless / illiterate people in the workplace doesn't excuse spending trillions on computer systems which use more energy than entire countries and are yet unreliable
I agree with Dijkstra on this one: “The question of whether a computer can think is no more interesting than the question of whether a submarine can swim.”
I really wish all these LessWrong, what is the meaning of intelligence types cared enough to study Wittgenstein a bit rather than hear themselves talk; it would save us all a lot of time.
I fully agree with your sentiments. People really need to study a little!
Intelligence as described is not the entire "requirement" for intelligence. There are probably more layers here, but I see "intelligence" as the 2nd layer, and beneath that later is comprehension which is the ability to discriminate between similar things, even things trying to decieve you. And at layer zero is the giant mechanism pushing this layered form of intelligence found in living things is the predator / prey dynamic that dictates being alive or food for something remaining alive.
"Intelligence in AI" lacks any existential dynamic, our LLMs are literally linguistic mirrors of human literature and activity tracks. They are not intelligent, but for the most part we can imagine they are, while maintaining sharp critical analysis because they are idiot savants in the truest sense.
Two concepts of intelligence and neither have remotely anything to do with real intelligence, academics sure like to play with words. I suppose this is how they justify their own existence; in the absence of being intelligent enough to contribute anything of value, they must instead engage in wordplay that obfuscates the meaning of words to the point nobody understands what the hell they're talking about, and confuses the lack of understanding of what they're talking about for the academics being more intelligent than the reader.
Intelligence, in the real world, is the ability to reason about logic. If 1 + 1 is 2, and 1 + 2 is 3, then 1 + 3 must be 4. This is deterministic, and it is why LLMs are not intelligent and can never be intelligent no matter how much better they get at superficially copying the form of output of intelligence. Probabilistic prediction is inherently incompatible with deterministic deduction. We're years into being told AGI is here (for whatever squirmy value of AGI the hype huckster wants to shill), and yet LLMs, as expected, still cannot do basic arithmetic that a child could do without being special-cased to invoke a tool call. How is it that we can go about ignoring reality for so long?
Addendum:
> With recent advances in AI, it becomes ever harder for proponents of intelligence-as-understanding to continue asserting that those tools have no clue and “just” perform statistical next-token prediction.
??????? No, that is still exactly what they do. The article then lists a bunch of examples in which this in trivially exactly what is happening.
> “The cat chased the . . .” (multiple connections are plausible, so how is that not understanding probability?)
It doesn't need to "understand" probability. "The cat chased the mouse" shows up in the distribution 10 times. "The cat chased the bird" shows up in the distribution 5 times. Absent any other context, with the simplest possible model, it now has a probability of 2/3 for the mouse and 1/3 for the bird. You can make the probability calculations as complex as you want, but how could you possibly trout this out as an example that an LLM completing this sentence isn't a matter of trivial statistical prediction? Academia needs an asteroid, holy hell.
[I originally edited this into my post, but two people had replied by then, so I've split it off into its own comment.]
One question is how do you know that you (or humans in general) aren't also just applying statistical language rules, but are convincing yourself of some underlying narrative involving logical rules? I don't know the answer to this.
> Probabilistic prediction is inherently incompatible with deterministic deduction
Prove that humans do it.
I keep coming back to this. The most recent version of chatgpt I tried was able to tell me how many letter 'r's were in a very long string of characters only by writing and executing a python script to do this. Some people say this is impressive, but any 5 year old could count the letters without knowing any python.
How is counting not a technology?
The calculations are internal but they happen due to the orchestration of specific parts of the brain. That is to ask, why can't we consider our brains to be using their own internal tools?
I certainly don't think about multiplying two-digit numbers in my head in the same manner as when playing a Dm to a G7 chord that begs to resolve to a C!
Many people would require an intelligent entity to successfully complete tasks with non-deterministic outputs.
Intelligence is not just about reasoning with logic. Computers are already made to do that.
The key thing is modeling. You must model a situation in a useful way in order to apply logic to it. And then there is intention, which guides the process.
Our computer programs execute logic, but cannot reason about it. Reasoning is the ability to dynamically consider constraints we've never seen before and then determine how those constraints would lead to a final conclusion. The rules of mathematics we follow are not programmed into our DNA; we learn them and follow them while our human-programming is actively running. But we can just as easily, at any point, make up new constraints and follow them to new conclusions. What if 1 + 2 is 2 and 1 + 3 is 3? Then we can reason that under these constraints we just made up, 1 + 4 is 4, without ever having been programmed to consider these rules.
It's the same kind of schizm that has lead to a lot of hate and mass murder over the last century or so: Abstraction/dimensionality reduction vs. concrete logic and statistics
Concrete statistician: I can learn the problem in its full complexity, unlike the dumdum below me.
Abstract thinker: I understand it, because I can reduce its dimensity to a small number of parameters.
CS: I can predict this because I have statistics about its past behavior.
AT: I told you so.
CS: You couldn't possibly know this, because it has never happened before. You suffer from the hindsight bias.
AT: But I told you.
CS: It has never happened, you couldn't possibly have statistics of when such things occur. You were just lucky.
CS: I'm smart, I can be taught anything
AT: You are stupid because you need to be taught everything.
War (or another sort of mass death or other kind of suffering) emerges.
What are some concrete examples of wars that you believe emerged due to this schism?
Memory foam doesn't really "remember" the shape of my rear end but we all understand the language games at play when we use that term.
The problem with the AI discourse is that the language games are all mixed up and confused. We're not just talking about capability, we're talking about significance too.
This is kind of bait-and-switch, no?
The author defines American style intelligence as "the ability to adapt to new situations, and learn from experience".
Then argues that the current type of machine-learning driven AI is American style-intelligent because it is inductive, which is not what was supposedly (?) being argued for.
Of course current AI/ML models cannot adapt to new situations and learn from experience, outside the scope of its context window, without a retraining or fine-tuning step.
intelligence/understanding is when one can postulate/predict/calculate/presume something correctly, from concepts about it, without that thing (or similar) ever been in the training/past (or even ever-known).
Yeah, not all humans do it. It's too energy expensive, biological efficiency wins.
As of ML.. Maybe next time, when someone figures out how to combine deductive with inductive, in zillion small steps, with falsifying built-in.. (instead of confronting them 100% one against 100% the other)
I'd argue you can have much more precise definition than that. My definition of intelligence would be a system that has an internal of a particular domain, and it uses this simulation to guide its actions within that domain. Being able to explain your actions is derived directly from having a model of the environment.
For example, we all have an internal physics model in our heads that's build up through our continuous interaction with our environment. That acts as our shared context. That's why if I tell you to bring me a cup of tea, I have a reasonable expectation that you understand what I requested and can execute this action intelligently. You have a conception of a table, of a cup, of tea, and critically our conception is similar enough that we can both be reasonably sure we understand each other.
Incidentally, when humans end up talking about abstract topics, they often run into exact same problem as LLMs, where the context is missing and we can be talking past each other.
The key problem with LLMs is that they currently lack this reinforcement loop. The system merely strings tokens together in a statistically likely fashion, but it doesn't really have a model of the domain it's working in to anchor them to.
In my opinion, stuff like agentic coding or embodiment with robotics moves us towards genuine intelligence. Here we have AI systems that have to interact with the world, and they get feedback on when they do things wrong, so they can adjust their behavior based on that.
Marx is fair play, but one of the most prominent cases of understanding everything in advance is undoubtedly Chomsky's theory of innate/universal grammar, which became completely dominant on guess which side of the pond.