When More Problems Stop Helping: RL Data Scaling Becomes an Allocation Problem

TL;DR for operators

A post-training team with a fixed compute budget has several ways to spend it: add more verified problems, generate variants of existing problems, increase difficulty, or expose the model to structurally different tasks. The usual accounting metric—number of training problems—does not tell the team which choice produces the most useful learning experience.

Sancaktar, Zhang, Synnaeve, and Cohen test these alternatives in A Deep Dive into Scaling RL for Code Generation with Synthetic Data and Curricula.¹ Their controlled code-RL experiments show an early plateau when real training data increases from 25K to 81K problems. Adding synthetic data can improve convergence and coding performance, but the result is not “generate as much as possible.” Difficulty distribution matters, medium-difficulty problems often provide the best optimization trade-off, and distributing a fixed 20K synthetic-task budget across four executable environments improves generalization relative to putting all 20K tasks into one environment.

For operators, this makes RL data design an allocation problem. Pass rates and executable rewards can be used not only to score training but to decide which experiences should be generated next. The unresolved question is whether the same allocation rules survive beyond these models, coding environments, binary rewards, and benchmarks.

More real problems hit an early plateau

The paper begins with a scaling result that changes the rest of the analysis. Increasing the real code-RL dataset from 25K to 81K problems does not yield proportional improvement. Performance plateaus early.

That comparison does not prove that additional real data has little value in general. It establishes a narrower constraint inside the tested setting: once the model has already consumed a substantial verified problem set, raw count can cease to be the main bottleneck.

The authors then test synthetic augmentation under controlled training budgets. Adding 20K synthetic problems produces faster and more stable in-domain convergence across the studied Llama3.1-8B Instruct, Qwen3-8B Base, and Qwen2.5-32B Base models. Synthetic augmentation outperforms the 81K real-data baseline on most reported LCB metrics under the fixed-budget comparison. Math transfer also improves for Llama3.1-8B Instruct and Qwen2.5-32B, although Qwen3-8B is comparable or slightly worse in some math settings.

The evidence therefore supports synthetic data as another scaling axis, but not an unrestricted one.

Multi-turn generation uses student performance to shape the next task

The paper’s main methodological contribution is an iterative generation process rather than independent task sampling.

A GPT-OSS 120B teacher generates a task. The same model, operating in a lower-reasoning mode as a proxy student, attempts that task 32 times. Because the environments have executable binary rewards, those attempts produce an empirical pass rate. The teacher receives that rate together with representative successful and failed solutions, then adjusts the task over six turns per seed.

This is difficulty control through observed solvability rather than a verbal request such as “make this problem harder.”

The clearest result from this procedure concerns production efficiency. After validity, solvability, formatting, redundancy, and difficulty filtering, multi-turn generation produces roughly four times as many viable problems as equivalent independent single-turn sampling. That is main evidence for the generation method.

The resulting easy-medium-hard chains are more ambiguous. Explicit stepping stones improve solve rates on medium and hard generated tasks, but downstream benchmark gains are modest and depend on curriculum scheduling. The experiment therefore supports intermediate variants as a useful capability of the pipeline, not as the proven reason the pipeline works.

A separate seed-source ablation also weakens the assumption that synthetic tasks need to begin from curated solved problems. Random 25–50-line snippets from starcoderdata are competitive generation seeds; after matching difficulty distributions, the random-code condition provides greater diversity and higher reported in-domain pass@1 and pass@10.

Medium difficulty produces the most usable training signal

Difficulty is not a monotonic “harder is better” variable in these experiments.

Easy problems are solved frequently and converge quickly, but they can become too concentrated in the training distribution and overfit easier benchmark splits. Very hard problems have the opposite problem: successful trajectories become rare, so training receives relatively few positive signals and convergence slows.

The paper calls this difference reward density: how often the model receives a useful positive training signal rather than near-certain success or near-certain failure. Medium-difficulty problems provide the strongest observed balance between sufficient positive rewards, non-trivial learning, convergence, and performance on harder code splits.

This result also connects to exploration. As RL progresses, the model’s behavior becomes less varied—what the paper describes through declining policy entropy. Spending too much early training on already-solvable tasks can consume that exploration capacity without teaching much that is new. Starting from tasks the model almost never solves creates too little signal to guide improvement.

The curriculum experiments test whether ordering can manage this tension. Reverse schedules can reduce early overfitting to easy tasks. But beginning directly from the hardest split causes slower convergence and substantially greater variance across random seeds. A medium-start reverse curriculum behaves better, while the paper explicitly leaves its advantage over medium-only training unresolved.

So there is no demonstrated universal curriculum. There is evidence for a solvability floor: nominal difficulty is not useful when rewards become too sparse.

Environment diversity helps without increasing the data budget

The paper then changes a different variable while holding problem count fixed.

One condition allocates 20K synthetic problems to induction tasks. Another divides the same 20K budget across induction, abduction, deduction, and fuzzing—5K problems per environment. All four environments use executable binary rewards, but they require different kinds of behavior: synthesizing a function from examples, finding an input for a target output, predicting execution results, or discovering inputs that expose a bug.

The multi-environment condition improves out-of-domain generalization and in-domain pass@10 relative to induction-only training.

This matters because difficulty diversity and environment diversity solve different allocation problems. Difficulty changes where examples sit along a solvability range. Environment diversity changes what kind of reasoning interaction generates the reward. The fixed-budget experiment provides evidence that this second axis contributes independently enough to merit explicit attention.

What post-training teams can change

Cognaptus interprets these experiments as a case for instrumenting the training-data pipeline around learning-signal distribution rather than dataset size alone.

The affected user here is the team operating an RL post-training pipeline. The decision is what experience to generate and train on next. The relevant condition is a constrained compute or problem budget with executable verification. The paper does not establish the same rule for domains where correctness cannot be cheaply executed or where rewards are subjective, delayed, or continuous.

The generation loop still uses the wrong student

One limitation matters directly for deployment of the method. The teacher does not adapt task generation against the actual target model as that model changes during RL. GPT-OSS 120B serves as both teacher and proxy student during offline dataset creation.

That separation simplifies generation, but it means the measured difficulty distribution can drift away from the difficulty experienced by the evolving target policy. An operational implementation might therefore want to measure target-policy pass rates during training and compare them with the proxy-derived labels before allocating another generation batch. That is a Cognaptus inference, not an experiment reported by the paper.

The other uncertainty is more fundamental. Stepping-stone gains are modest outside particular curricula, curriculum differences are sometimes weak, and synthetic datasets can become easy-heavy when their distributions are not controlled. Even synthetic-only training matching real-data LCB performance for Qwen3-8B should not be read as evidence that real problems are generally dispensable.

Measure the distribution, not only the inventory

The paper makes a credible case that verified RL data has more structure than its row count.

Its most operationally defensible results are the higher viable-task yield from pass-rate-guided multi-turn generation, the favorable behavior of medium-difficulty tasks, and the fixed-budget gain from distributing training across multiple executable environments. Together, these results give post-training teams measurable variables for deciding where another unit of generation and RL compute should go.

They do not yet provide a universal synthetic-data recipe. The more durable operating rule is to observe which tasks are trivial, which are nearly unsolvable, which environments are underrepresented, and whether another batch of data is still changing the model. Scaling then becomes a repeated allocation decision informed by training behavior rather than a commitment to accumulating the largest possible verified dataset.

Cognaptus: Automate the Present, Incubate the Future.