It makes sense to me that distributing across more parameters results in models that can be quant more heavily (information theory - more bits available)
I wonder if anyone has figured out how the information is compressed and calculated the amount of information an LLM can hold depending on its size
It makes sense to me that distributing across more parameters results in models that can be quant more heavily (information theory - more bits available)
I wonder if anyone has figured out how the information is compressed and calculated the amount of information an LLM can hold depending on its size
> I wonder if anyone has figured out how the information is compressed and calculated the amount of information an LLM can hold depending on its size
You might want to look at Physics of Language Models[1]. IIRC, the authors estimate it to be ~2 bits of factual knowledge per parameter.
[1]: https://physics.allen-zhu.com/
Anyone with a billion dollars want to try this and report back?
From the paper it appears that it's probably more useful on small-ish models.
What does it cost to train a model like 1-bit Bonsai? Anyone know?