Compute, bytes of ram used, bytes in model, bytes accessed per iteration, bytes of data used for training.
You can trade the balance if you can find another way to do things, extreme quantisation is but one direction to try. KANs were aiming for more compute and fewer parameters. The recent optimisation project have been pushing at these various properties. Sometimes gains in one comes at the cost of another, but that needn't always be the case.
We will not see memory demand decrease because this will simply allow AI companies to run more instances. They still want an infinite amount of memory at the moment, no matter how AI improves.
The hyperscalers do not want us running models at the edge and they will spend infinite amounts of circular fake money to ensure hardware remains prohibitively expensive forever.
Oh it gets worse than that, the money which caused all of this by OpenAI was taken from Japanese banks at cheap interest rates (by softbank for the stargate project), and the Japanese Banks are able to do it because of Japanese people/Japanese companies and also the collateral are stocks which are inflated by the value of people who invest their hard earned money into the markets
So in a way they are using real hard earned money to fund all of this, they are using your money to basically attack you behind your backs.
But what if it becomes "good enough", that for most intents and purposes, small models can be "good enough"
There are some people here/on r/localllama who I have seen run some small models and sometimes even run multiple of them to solve/iterate quickly and have a larger model plug into it and fix anything remaining.
This would still mean that larger/SOTA models might have some demand but I don't think that the demand would be nearly enough that people think, I mean, we all still kind of feel like there are different models which are good for different tasks and a good recommendation is to benchmark different models for your own use cases as sometimes there are some small models who can be good within your particular domain worth having within your toolset.
The mainframe analogy is close but I think the key difference is that mainframe->PC was driven by hardware getting cheap, while LLM efficiency needs algorithmic breakthroughs which are way less predictable. My bet is we get a split: anything latency-sensitive (code completion, local assistants) goes to edge as soon as models fit on consumer hardware, because you can't cheat physics on network round trips. But training and heavy reasoning stays centralized -- the data gravity just gets worse as models improve. Also I keep going back and forth on whether stuff like MoE and speculative decoding is "better math" or just "better engineering." Feels like an important distinction since they have very different cost curves.
I don't think we are there yet. Models running in data centers will still be noticeably better as efficiency will allow them to build and run better models.
Not many people would like today models comparable to what was SOTA 2 years ago.
To run models locally and have results as good as the models running in data centers we need both efficiency and to hit a wall in AI improvement.
None of those two conditions seem to become true for the near future.
Citation needed. I've heard this quite often, but so far, I haven't seen proof of the stated causality.
PS: This doesn't mean that better public transportation could deliver more bang for the buck than the n-th additional car lane. But never ever have I heard from anybody that they chose to buy a car or use an existing car more often because an additional lane has been built.
Despite the shortage, RAM is still cheaper than mathematicians.
The same could be said about other IT domain... When you see single webpages that weight by tens of MB you wonder how we came to this.
This is one of the basic avenues for advancement.
Compute, bytes of ram used, bytes in model, bytes accessed per iteration, bytes of data used for training.
You can trade the balance if you can find another way to do things, extreme quantisation is but one direction to try. KANs were aiming for more compute and fewer parameters. The recent optimisation project have been pushing at these various properties. Sometimes gains in one comes at the cost of another, but that needn't always be the case.
Sigh. Don't make me tap the sign [1]
[1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Can we say something about the compression factor for pure knowledge of these models?
We will not see memory demand decrease because this will simply allow AI companies to run more instances. They still want an infinite amount of memory at the moment, no matter how AI improves.
If models become more efficient we will move more of the work to local devices instead of using SaaS models. We’re still in the mainframe era of LLM.
The hyperscalers do not want us running models at the edge and they will spend infinite amounts of circular fake money to ensure hardware remains prohibitively expensive forever.
> of circular fake money
Oh it gets worse than that, the money which caused all of this by OpenAI was taken from Japanese banks at cheap interest rates (by softbank for the stargate project), and the Japanese Banks are able to do it because of Japanese people/Japanese companies and also the collateral are stocks which are inflated by the value of people who invest their hard earned money into the markets
So in a way they are using real hard earned money to fund all of this, they are using your money to basically attack you behind your backs.
I once wrote an really long comment about the shaky finances of stargate, I feel like suggesting it here: https://news.ycombinator.com/item?id=47297428
> If models become more efficient
Then we can make them even bigger.
> Then we can make them even bigger.
But what if it becomes "good enough", that for most intents and purposes, small models can be "good enough"
There are some people here/on r/localllama who I have seen run some small models and sometimes even run multiple of them to solve/iterate quickly and have a larger model plug into it and fix anything remaining.
This would still mean that larger/SOTA models might have some demand but I don't think that the demand would be nearly enough that people think, I mean, we all still kind of feel like there are different models which are good for different tasks and a good recommendation is to benchmark different models for your own use cases as sometimes there are some small models who can be good within your particular domain worth having within your toolset.
The mainframe analogy is close but I think the key difference is that mainframe->PC was driven by hardware getting cheap, while LLM efficiency needs algorithmic breakthroughs which are way less predictable. My bet is we get a split: anything latency-sensitive (code completion, local assistants) goes to edge as soon as models fit on consumer hardware, because you can't cheat physics on network round trips. But training and heavy reasoning stays centralized -- the data gravity just gets worse as models improve. Also I keep going back and forth on whether stuff like MoE and speculative decoding is "better math" or just "better engineering." Feels like an important distinction since they have very different cost curves.
I don't think we are there yet. Models running in data centers will still be noticeably better as efficiency will allow them to build and run better models.
Not many people would like today models comparable to what was SOTA 2 years ago.
To run models locally and have results as good as the models running in data centers we need both efficiency and to hit a wall in AI improvement.
None of those two conditions seem to become true for the near future.
I disagree. I think a sharp drop in memory requirements of at least an order of magnitude will cause demand to adjust accordingly.
Department of Transportation always thinks adding more lanes will reduce traffic.
It doesn't, it induces demand. Why? Because there's always too many people with cars who will fill those lanes.
Citation needed. I've heard this quite often, but so far, I haven't seen proof of the stated causality.
PS: This doesn't mean that better public transportation could deliver more bang for the buck than the n-th additional car lane. But never ever have I heard from anybody that they chose to buy a car or use an existing car more often because an additional lane has been built.
Jevons paradox https://en.wikipedia.org/wiki/Jevons_paradox