On-Policy-Distillation

TL;DR for operators A multi-model agent platform already records much of what a post-training system needs: the request, the model-selection decision, tool interactions, execution failures, verification signals, and eventual outcome. NeoHorse-1 turns those records into training supervision rather than treating them as operational exhaust.1 The strongest practical evidence is not the recursive-self-improvement claim. It is the controlled comparison of training data. Using the same routing-guided training configuration, Qwen3.5-4B trained on routing-harness trajectories reaches a 70.57 average across five benchmarks, versus 64.32 when trained on Toucan public synthetic agent data. The difference is 6.26 points. ...

TL;DR for operators A reasoning model can have useful confidence at one point in its workflow and misleading confidence at another. In a controlled comparison using the same Qwen2.5-7B-Instruct backbone and reasoning-data mixture, on-policy distillation produced the strongest average signal for estimating difficulty before reasoning, supervised fine-tuning supplied particularly useful confidence for stopping weak traces during generation, and reinforcement learning gained the most from confidence-based filtering after traces were complete.1 ...

TL;DR for operators The paper’s main message is simple: if a reasoning model has already walked into a dead end, per-token distillation often keeps supervising it from inside the dead end. A clever loss cap is not a map. A top-k filter is not a tow truck. Trajectory-Refined Distillation, or TRD, repairs the student’s own rollout before using it for distillation. The pipeline is: sample the student’s attempt, ask a teacher or privileged self-teacher to rewrite the trajectory into a better one, then train on the refined trajectory rather than on the original failed rollout. The technical contribution is not “better prompting”, although prompts are used. It is the shift from token-level correction to trajectory-level correction. ...