Everything I read seems to suggest that RAM capacity is going to grow at 20-25% a year, which just doesn't seem good enough. Even in consumer use cases, phones and laptops would benefit greatly by double the amount of RAM. And then obviously, the AI need is gigantic.
I don't see it going away. I mean, it may not grow as fast as now, but I don't see it growing away either. I get why the memory makers do not want to bankrupt themselves, but it feels like there's got to be some way to push that risk off onto model providers and other people in the ecosystem to allow us to grow ram capacity more like 50% per year.
I mean the biggest risk is Chinese CXML benefit and capturing markets that others are leaving hanging and then being able to compete and push out the others when costs start to normalize.
As for 20-25% growth not being enough, I think it's not that far off, if we assume data center build out plans hit a wall and slow down significantly, and the AI heat starts to cool off.
I don't think 20-25% may be enough in the short term but if the AI build out stops within this year, we have a massive oversupply instead of a under supply.
Makes prior assumptions that getting tens of gigs of ram is cheap thrown out the window. Would likely lead to super fast SSDs such as optain being way more valuable
It is one of the thing with consumer when they remember they brought it at the absolutely lowest price point when DRAM maker were bleeding money.
Those are not normal pricing. Before the pricing collapse in early 2020, 96GB DDR5 would have cost about $450 to $500. And I will need to restate again the cost of DRAM hasn't really changed much in the past 20 years. Its price just goes up and down in cycles.
So in reality it is more like going from $500 to $1300. But consumer felt it was more like going from $200 to $1300.
Crucial are already selling DRAM made by CXMT. And China are already throwing money at it. I doubt the memory bubble will burst in next 12-24 months. As in going back to money losing DRAM pricing. As they will all pivot to HBM or other money making products. But the bulk of lower end consumer DDR5 or LPDDR5 will goes to Chinese Foundry. Assuming they have figure out how to do them well. Which they have improved but are still so far away from industry leaders.
Normally memory maker will push the next DDR standard to market just to push out Chinese competitors, I am not sure it will work the same this time around. DDR5 have plenty of other usage / demands.
I really don’t want to give anyone ideas, but doesn’t this make the Nvidia 5090 an unbelievably good deal right now?
The VRAM in the 5090 is only made by one country in the world.
The 50xx series is special, because its ram is so dependent on a single commodity. It’s not like a 4090 or a 3090; their VRAM chips have been around for years.
If there’s a shortage or interruption in DDR7 VRAM, it seems like every GPU that requires it would explode in value.
I hope I don’t regret posting this because I’d really like to buy one myself…
Which surely is the highest it'll ever be! You're suggesting that the price will go down in the future? Would love to hear more about your thought process!
Also had to fork over almost $1k for a 64G DDR5 kit a few weeks back. At least AMD chips large L3 cache allows folks to get away with lower grade udimms.
Also had to do an Intel build, and there was no way we were going cudimm at current prices.
I'm not moving past my DDR4 build (and the 32 GB of DDR4 2133 MHz backup chips I still have around from way back, before I got the current 3200 MHz ones) until the prices go back to being at least partially sane. This also means that CPU manufacturers are not getting my money (since the 5800X is fine for now) and I have no reason to get a new GPU either (though admittedly the B580 isn't perfect).
This is 100% going to kill the home built pc market. When I started building gaming pcs, the top top card was 750$ (NZD). Now they’re 10,000 just for the gpu and another 1-2000 for ram.
People used to get into gaming pcs as an affordable hobby, now it’s making general aviation look like plan B.
Bought a second hand Dell server a week ago. The entire rig with a 12-core CPU and 32GB DDR4 ecc RAM cost as much as I'd pay to buy 64 GB of DDR RAM alone. I hope there's an end to this absurdity soon enough otherwise the pain will affect other markets too. I read the other day that PC case sales have collapsed by more than 40%.
I feel like by the time the AI bubble bursts the PC market will be irreparably damaged. Manufactures who have been making "enterprise" parts aren't going to go back to making consumer parts because there will be no market for it. And with a glut of datacenters not making any money on slop, they are going to be repurposed for saas, stuff like OnShape but for every application.
Most users don't seem to care about storing everything they generate in cloud services and this could easily be sold as an alternative to owning "expensive" desktop or laptop hardware.
It's still unclear to me: the shortage is semiconductor boules / wafers? or the shortage is semiconductor fab process step availability?
As long as the discussion seems focused on memory, I'd suspect the latter, but if its really the semiconductor boules/wafers, then I'd expect the boule growers to profit, not the memory makers, who just pass on the cost.
It’s fab capacity. Fwiw dram is different enough that fabs are not transferable between dram memory and other usages. It’s nice to think ‘wow if they made the current 10nm dram on the latest 2nm processes it’d be much faster’ but it doesn’t work that way. The specific size is needed for the capacitance. Sram can be made on fabs that make other circuitry since it’s transistor not capacitor based but is less dense.
I wonder why the hyperscalers aren't vertically integrating more and building their own fabs. Sure, a fab costs a billion dollars, but they're currently spending hundreds of billions of dollars purchasing chips from NVidia and others.
I'm not sure if they should vertically integrate, it would probably be a better idea to directly fund the expansion of capacity, much like Apple does when they scale up a new technology for iPhones.
However, that the hyperscalers and AI companies aren't doing this says a lot about their true beliefs about how much future demand AI will have.
AI companies claim they will need a ton of massive expansion, but are unwilling to take on the risk of the capital needed for that expansion.
I'm hearing a lot of sad whining from AI folks about how these chip makers are holding them back, but who actually has the money to finance the expansion easily? Chip makers have been through this game far longer, when Sam Altman went around claiming it was time for $7T of fabs the AI companies made it clear that they were willing to make ridiculous claims, eliminating credibility.
What's needed now is for them to funnel a tiny amount of their massive piles of cash into financing fabs directly.
Do we though? DLSS 5 changes that somewhat from a “we need powah” to “we need models”. I think the future consumer GPU market will be tuned for image and world inference while workstation cards will be tuned for image and video inference. The old way of thinking about this will come to an end when we stop looking at the render loop as the be-all-end-all…
You're probably thinking about jevons paradox. But you slightly mis-stated. It is the phenomenon that increasing the efficiency of resource consumption can end up increasing total consumption.
As you stated it, it would merely be a property of (nearly) all demand curves. Jevons paradox only happens sometimes. It isn't a law.
An example of where it stopped happening is with gasoline in developed countries. Cars having better fuel efficiency doesn’t make me drive further to the grocery store or work.
Generally when someone replaces their vehicle the new one is more fuel efficient than the old one even if I bought the same car.
Here’s the thing, what if memory manufacturers take this opportunity to collude and basically never reduce the price of memory below the current levels since it’s too hard for a new competitor to just rise up and undercut them? Everything I hear about is how hard and risky it is to spin up a new fab.
And by doing this, they ensure local LLMs never become feasible for the vast majority of people and AI companies solidify subscriptions forever.
Keeping prices at this level is precisely how one or more competitor will rise up. Making memory isn’t super hard. That’s why it is a commodity. The problem with the memory market is that up and down cycles have bankrupted the vast majority of players in the past. Now we only have 3 players left except for a few smaller ones in China.
The reason memory prices can stay high for years in this mega cycle is because the 3 players will be very cautious on overbuilding. They’d rather under build, make great profit (not maximum) and reduce the risk of going bust if this suddenly ends.
Same for TSMC in chips.
Great opportunity for Chinese companies though. This shortage is exactly what Chinese companies need to scale.
When Samsung had to sell memory at a loss after COVID, no one came to save them. They buffered their memory division using profits from their other businesses. That’s how Samsung survives memory downturns.
According to some stories, this is how Samsung convinced TSMC to not enter the memory business - that you need a nation or other lines of business to prevent bankruptcies.
If the collude to say make the price $1000 for a component that costs them $100(including opportunity costs), then either a new company or a greedy company in the collusion can make their price secretly $900 and get massively more profit.
Right now their opportunity cost is too high.
> risky it is to spin up a new fab
You don't need a new fab. You can build memory in 20 years old fab.
They will respond when people are loud enough. If memory stays at $1200 for 128GB for years and investigative journalists say it could be colluding, enough people will make enough noise.
Everything I read seems to suggest that RAM capacity is going to grow at 20-25% a year, which just doesn't seem good enough. Even in consumer use cases, phones and laptops would benefit greatly by double the amount of RAM. And then obviously, the AI need is gigantic.
I don't see it going away. I mean, it may not grow as fast as now, but I don't see it growing away either. I get why the memory makers do not want to bankrupt themselves, but it feels like there's got to be some way to push that risk off onto model providers and other people in the ecosystem to allow us to grow ram capacity more like 50% per year.
I mean the biggest risk is Chinese CXML benefit and capturing markets that others are leaving hanging and then being able to compete and push out the others when costs start to normalize.
As for 20-25% growth not being enough, I think it's not that far off, if we assume data center build out plans hit a wall and slow down significantly, and the AI heat starts to cool off.
I don't think 20-25% may be enough in the short term but if the AI build out stops within this year, we have a massive oversupply instead of a under supply.
I bought 96GB of RAM a couple of years ago for ~$250. That same RAM now costs $1200!
I paid $279 for crucial 96gb DDR5 5600 MHz SO-DIMM ram October 22 of last year. Amazon has the same kit going for $1,048.90 right now.
Makes prior assumptions that getting tens of gigs of ram is cheap thrown out the window. Would likely lead to super fast SSDs such as optain being way more valuable
It is one of the thing with consumer when they remember they brought it at the absolutely lowest price point when DRAM maker were bleeding money.
Those are not normal pricing. Before the pricing collapse in early 2020, 96GB DDR5 would have cost about $450 to $500. And I will need to restate again the cost of DRAM hasn't really changed much in the past 20 years. Its price just goes up and down in cycles.
So in reality it is more like going from $500 to $1300. But consumer felt it was more like going from $200 to $1300.
Crucial are already selling DRAM made by CXMT. And China are already throwing money at it. I doubt the memory bubble will burst in next 12-24 months. As in going back to money losing DRAM pricing. As they will all pivot to HBM or other money making products. But the bulk of lower end consumer DDR5 or LPDDR5 will goes to Chinese Foundry. Assuming they have figure out how to do them well. Which they have improved but are still so far away from industry leaders.
Normally memory maker will push the next DDR standard to market just to push out Chinese competitors, I am not sure it will work the same this time around. DDR5 have plenty of other usage / demands.
> Crucial are already selling DRAM made by CXMT.
Crucial was disestablished this year.
He probably meant Corsair which is the DRAM brand selling CXMT produced chips.
I really don’t want to give anyone ideas, but doesn’t this make the Nvidia 5090 an unbelievably good deal right now?
The VRAM in the 5090 is only made by one country in the world.
The 50xx series is special, because its ram is so dependent on a single commodity. It’s not like a 4090 or a 3090; their VRAM chips have been around for years.
If there’s a shortage or interruption in DDR7 VRAM, it seems like every GPU that requires it would explode in value.
I hope I don’t regret posting this because I’d really like to buy one myself…
if you can buy one!
The RTX 5090 is faster than an H200. It just has less ram (32 vs 141), doesn't have NVLink, and technically isn't allowed to be used in a datacenter.
The datacenter GPUs sell at an 80% margin. They're incredibly overpriced. But the laws of supply and demand are undefeated and so here we all are.
It's gone up like 300% in cost in the last year.
Which surely is the highest it'll ever be! You're suggesting that the price will go down in the future? Would love to hear more about your thought process!
I recently built a system at insane ddr4 prices ($2000 for 256gb). But that’s only after seeing how ddr5 prices were 3-4x that!
Also had to fork over almost $1k for a 64G DDR5 kit a few weeks back. At least AMD chips large L3 cache allows folks to get away with lower grade udimms.
Also had to do an Intel build, and there was no way we were going cudimm at current prices.
Yeah I upgraded all of my systems to DDR5 last year, so now I have to buy for ddr5 memory upgrades.
I'm not moving past my DDR4 build (and the 32 GB of DDR4 2133 MHz backup chips I still have around from way back, before I got the current 3200 MHz ones) until the prices go back to being at least partially sane. This also means that CPU manufacturers are not getting my money (since the 5800X is fine for now) and I have no reason to get a new GPU either (though admittedly the B580 isn't perfect).
Awful time for gamers and PC hobbyists not fully into AI.
This is 100% going to kill the home built pc market. When I started building gaming pcs, the top top card was 750$ (NZD). Now they’re 10,000 just for the gpu and another 1-2000 for ram.
People used to get into gaming pcs as an affordable hobby, now it’s making general aviation look like plan B.
also for ones fully into AI
Bought a second hand Dell server a week ago. The entire rig with a 12-core CPU and 32GB DDR4 ecc RAM cost as much as I'd pay to buy 64 GB of DDR RAM alone. I hope there's an end to this absurdity soon enough otherwise the pain will affect other markets too. I read the other day that PC case sales have collapsed by more than 40%.
I feel like by the time the AI bubble bursts the PC market will be irreparably damaged. Manufactures who have been making "enterprise" parts aren't going to go back to making consumer parts because there will be no market for it. And with a glut of datacenters not making any money on slop, they are going to be repurposed for saas, stuff like OnShape but for every application.
Most users don't seem to care about storing everything they generate in cloud services and this could easily be sold as an alternative to owning "expensive" desktop or laptop hardware.
It's still unclear to me: the shortage is semiconductor boules / wafers? or the shortage is semiconductor fab process step availability?
As long as the discussion seems focused on memory, I'd suspect the latter, but if its really the semiconductor boules/wafers, then I'd expect the boule growers to profit, not the memory makers, who just pass on the cost.
So which is it?
It’s fab capacity. Fwiw dram is different enough that fabs are not transferable between dram memory and other usages. It’s nice to think ‘wow if they made the current 10nm dram on the latest 2nm processes it’d be much faster’ but it doesn’t work that way. The specific size is needed for the capacitance. Sram can be made on fabs that make other circuitry since it’s transistor not capacitor based but is less dense.
Dram is just extremely specialised.
Time to let ASML sell to the Chinese memory producers … or not.
I wonder why the hyperscalers aren't vertically integrating more and building their own fabs. Sure, a fab costs a billion dollars, but they're currently spending hundreds of billions of dollars purchasing chips from NVidia and others.
I'm not sure if they should vertically integrate, it would probably be a better idea to directly fund the expansion of capacity, much like Apple does when they scale up a new technology for iPhones.
However, that the hyperscalers and AI companies aren't doing this says a lot about their true beliefs about how much future demand AI will have.
AI companies claim they will need a ton of massive expansion, but are unwilling to take on the risk of the capital needed for that expansion.
I'm hearing a lot of sad whining from AI folks about how these chip makers are holding them back, but who actually has the money to finance the expansion easily? Chip makers have been through this game far longer, when Sam Altman went around claiming it was time for $7T of fabs the AI companies made it clear that they were willing to make ridiculous claims, eliminating credibility.
What's needed now is for them to funnel a tiny amount of their massive piles of cash into financing fabs directly.
I wish I had figured that out a year ago. MU up ~10x, SNDK up ~37x. My crystal ball is woefully under performing.
I assume that memory manufacturers don’t really care where the money is coming from, as long as the "numbers go up" game is working.
NVIDIA in their recent quarterly report stopped categorizing "Geforce" as a single category, and merged it into "Edge-Computing".
If you are a PC Gamer or PC Enthusiast as I am, then we have some dark times ahead.
Do we though? DLSS 5 changes that somewhat from a “we need powah” to “we need models”. I think the future consumer GPU market will be tuned for image and world inference while workstation cards will be tuned for image and video inference. The old way of thinking about this will come to an end when we stop looking at the render loop as the be-all-end-all…
Or, we could be fucked.
A commodity rapidly increasing in price. What could go wrong?
Why did this happen so suddenly?
Why were tech savy investors unable to figure this out when the datacenter craze had already started?
How to explain this lag between quickly rising demand for all datacenter components besides memory?
The seminal paper about AI came out in 2017
Nine years later, we’re feeling the impact.
If the investors had known in 2017 that Google’s paper was going to light the fuse on AI, they would have built more manufacturing capacity.
Because AI caught everyone off guard, there is over a trillion dollars pouring into the sector in the last four years or so.
There’s a HUGE incentive for the RAM companies to slow roll sales.
If they DON’T keep prices high, the supply of RAM might simply run out.
There are plenty of companies who can’t even BUY the ram that they want for a 5X markup. There are SKUs going to 10x and 20x due to scarcity.
This is supply and demand at a crazy scale.
The algorithm advances are going to crash this so hard.
Or will more efficient algorithms just mean we run even more AI models, increasing the demand for AI chips even more?
I mean, god willing, but it'll be just as likely that we'll blissfully consume 100 million token contexts in that case.
isn't there a law for that? as things become cheaper you consume more?
Jevons paradox: https://en.wikipedia.org/wiki/Jevons_paradox
You're probably thinking about jevons paradox. But you slightly mis-stated. It is the phenomenon that increasing the efficiency of resource consumption can end up increasing total consumption.
As you stated it, it would merely be a property of (nearly) all demand curves. Jevons paradox only happens sometimes. It isn't a law.
An example of where it stopped happening is with gasoline in developed countries. Cars having better fuel efficiency doesn’t make me drive further to the grocery store or work.
Generally when someone replaces their vehicle the new one is more fuel efficient than the old one even if I bought the same car.
Jevons paradox.
https://en.wikipedia.org/wiki/Jevons_paradox
jevons paradox
Anyone invested in Micron stock?
Here’s the thing, what if memory manufacturers take this opportunity to collude and basically never reduce the price of memory below the current levels since it’s too hard for a new competitor to just rise up and undercut them? Everything I hear about is how hard and risky it is to spin up a new fab.
And by doing this, they ensure local LLMs never become feasible for the vast majority of people and AI companies solidify subscriptions forever.
Keeping prices at this level is precisely how one or more competitor will rise up. Making memory isn’t super hard. That’s why it is a commodity. The problem with the memory market is that up and down cycles have bankrupted the vast majority of players in the past. Now we only have 3 players left except for a few smaller ones in China.
The reason memory prices can stay high for years in this mega cycle is because the 3 players will be very cautious on overbuilding. They’d rather under build, make great profit (not maximum) and reduce the risk of going bust if this suddenly ends.
Same for TSMC in chips.
Great opportunity for Chinese companies though. This shortage is exactly what Chinese companies need to scale.
> Making memory isn’t super hard.
Then why do only 3 companies make it?
Bankruptcy risks.
When Samsung had to sell memory at a loss after COVID, no one came to save them. They buffered their memory division using profits from their other businesses. That’s how Samsung survives memory downturns.
According to some stories, this is how Samsung convinced TSMC to not enter the memory business - that you need a nation or other lines of business to prevent bankruptcies.
The market has stabilized to 3 players.
...And why do they go bankrupt?
Because it's an incredibly capital intensive process, involving billions of dollars of investment into manufacturing infrastructure.
That is to say, making memory is quite hard.
The technical process of making memory is relatively easy. Hence, it is a commodity.
I didn’t say owning a memory business is easy.
Making the memory can be much easier than predicting future demand.
Placing the bet isn't as hard as making an accurate prediction.
> up and down cycles have bankrupted the vast majority of players in the past
Exactly, so what’s the incentive for anyone to sink half a billy into building out more capacity.
The existing players get to rest on their laurels and succeed whether or not the AI bubble busts.
The incentive is that your 2 competitors will build more than you and gain market share on you if you are too conservative.
Samsung, SK Hynix, and Micron all have to balance between capex spending, making as much profit as possible, and risk of bankruptcy.
So are the new competitors currently in progress of starting up? Because it takes at least several years.
Only Chinese companies have a chance. Problem is that China can’t buy EUV machines and the most advanced memory chips now use EUV.
Heck, the US is now pressuring ASML to not sell even DUV machines to China, period.
When costs are high enough, you can recoup that, if you have an appetite for risking the downturn.
Then China will come and eat their lunch. I for one will only buy Chinese RAM from now on, no matter the prices.
If the collude to say make the price $1000 for a component that costs them $100(including opportunity costs), then either a new company or a greedy company in the collusion can make their price secretly $900 and get massively more profit.
Right now their opportunity cost is too high.
> risky it is to spin up a new fab
You don't need a new fab. You can build memory in 20 years old fab.
Then that's a cartel and hopefully regulators will act.
They won’t.
They will. DOJ prosecuted memory makers in the late 90s and 2000s for collusion.
This boom is magnitudes higher than before. The attention will be endless.
That was a very different DOJ.
Current DOJ is corrupt as fuck, it will not happen. Get back to reality.
They will respond when people are loud enough. If memory stays at $1200 for 128GB for years and investigative journalists say it could be colluding, enough people will make enough noise.