COUNT on RAM: A Different Way to Scale Language Models
TL;DR for operators Improving a language model does not necessarily require proportionally more GPU-resident parameters. COUNT stores corpus-derived continuation statistics in host RAM and converts them into transformer input representations using existing model weights. In controlled experiments, COUNT delivers validation-loss improvements equivalent to 46–89% more standard-transformer training compute, according to fitted scaling laws. An optimized implementation retains more than 93% of baseline training throughput, even with a count table approaching 798 GB. ...