
Weight Watchers for LLMs: Dynamic Dieting Beats Static Selection
Most large language models (LLMs) are trained as if every piece of data is equally nutritious. But just as elite athletes optimize not just what they eat but when and how they eat it, a new paper proposes that LLMs can perform better if we learn to dynamically adjust their data “diet” during training. The Static Selection Problem Traditional data selection for LLMs is front-loaded and fixed: you decide what data to keep before training, often using reference datasets (e.g., Wikipedia) or reference models (e.g., GPT-3.5) to prune the lowest-quality examples. While effective in reducing cost, this approach ignores a key insight: an LLM’s preference for certain types of data evolves over time. ...