Keep the Rollout Honest: Miles Treats Throughput and Fidelity as One System

TL;DR for operators

If rollout GPUs keep generating while training GPUs update the policy, higher utilization creates an accounting problem: some trajectories come from older weights, and serving and training stacks can assign different probabilities even when they nominally use the same model. The operating decision is therefore not simply how to remove idle time, but how much staleness and numerical mismatch the loop can tolerate.

Miles v0.1¹ treats rollout, training, and weight synchronization as one system, with explicit controls for trajectory age, token identity, MoE routing, precision, probabilities, memory placement, and weight transport. Its 64-GPU GLM-5.2 run demonstrates frontier-scale asynchronous agentic RL, while component tests show that infrastructure choices can materially change transfer and offload costs. The evidence is strongest for systems feasibility and configuration-specific mechanisms, not for generalized learning gains. For operators, verification depth matters as much as feature coverage.

Asynchrony turns utilization into a data-freshness problem

A synchronous RL loop is easy to reason about but can leave costly hardware waiting. Miles instead supports separate rollout and training GPU pools with a bounded buffer between them.

That buffer records queue size and how old each trajectory group is relative to the trainer’s current policy. A group can be acceptable when it arrives and too stale by the time training consumes it. The scheduling problem therefore becomes a staleness budget: more overlap can improve utilization, but only if the system measures and bounds how far training data has drifted behind current weights.

The main reference run uses 64 NVIDIA GB300 GPUs split evenly between rollout and training, with up to 128 trajectories in flight. Across the first 30 measured steps, median training-step time is 263 seconds. This is main evidence of end-to-end feasibility for that configuration, not a general throughput estimate for Miles.

Exact tokens and probabilities become part of the training record

A second mismatch appears when the trainer reconstructs what the rollout model did from message text. Re-tokenization or message transformation can change the sequence being optimized.

Miles uses token-in-token-out session handling to preserve sampled token IDs and rollout log-probabilities across multi-turn interactions. For mixture-of-experts models, it can also record and replay inference-time expert assignments, because identical tokens can still reach different experts under different numerical paths. That routing-replay mechanism is a targeted fidelity control rather than a universal default: the paper notes substantial memory and communication overhead for long MoE trajectories.

Miles then compares trainer and rollout probabilities for the same sampled token:

When $r=1$, the two stacks agree on that token. Differences can arise from weight staleness, routing, precision, kernels, or other numerical effects. Truncated importance sampling or clip-or-pop can limit how strongly large mismatches affect the policy update.

In the 100-step GLM-5.2 run, average rollout-training log-probability divergence is 0.0369 and ends near its starting level. That supports the paper’s claim that the system can observe and manage mismatch in this setup; it does not establish a universal divergence level for asynchronous RL.

Cluster topology is part of the training design

Miles offers three weight-synchronization paths: NCCL broadcast when ranks share a fabric, peer-to-peer RDMA when direct rank-to-rank reachability exists, and disk-delta updates when shared storage is the practical bridge. The interface is common; the best transport depends on topology.

The P2P measurements are component-level comparisons rather than a general networking benchmark. On tested H100 deployments, P2P reduces update time from 58.30 to 8.48 seconds for GLM-5 at 16 nodes per side and from 53.28 to 7.23 seconds for Kimi K2 at 32 nodes per side. At two nodes per side for Qwen3-30B-A3B, the reduction is smaller: 2.67 to 2.16 seconds. The paper also reports that P2P can be substantially slower on a single node.

Memory placement shows the same dependency. With actor offloading, streaming optimizer state to node-local disk reduces reported offload time for Qwen3-30B-A3B from 24 to 5.2 seconds and reload time from 8.9 to 1.3 seconds. These gains matter when optimizer state is the binding resource, but they remain tied to the measured configuration.

For infrastructure teams, the relevant planning unit is the combination of model size, backend, precision, node topology, interconnect, storage path, and update recipe—not the framework name alone.

Support matrices need verification depth

Miles spans Megatron-LM and FSDP, multiple low-precision formats, LoRA RL, on-policy distillation, supervised training, exact sampled-token alignment for selected configurations, and a diffusion extension. The paper does not present these as uniformly mature.

BF16 is the baseline; FP8 blockwise has broader tested coverage; MXFP8 and NVFP4 are labeled Beta and restricted to narrower model and hardware combinations. LoRA support depends on agreement across the trainer, Miles weight mapping, and the serving engine. Disk-delta transport has backend and deployment constraints. True-on-policy alignment covers only supported dense Qwen3 configurations and guarantees equality for sampled-token log-probabilities, not the full output distribution.

One on-policy-distillation experiment shows why this distinction matters. Held-out response length falls from 14,070 to 6,132 tokens, about 56%, while accuracy moves from 84.0% to 85.2%. With a reported standard error of about 1.6 percentage points, the supported conclusion is shorter responses without a reliable accuracy change.

For deployment governance, “implemented,” “Beta,” “tested,” and “full-scale validated” should be treated as different evidence states.

The frontier-scale run proves operation, not reproducible learning gain

The 100-step GLM-5.2 744B-A40B run on Terminal-bench-2 records a 96% prefix-cache hit rate with roughly 90 to 100 rollout requests active concurrently. Its nine-step moving average of raw task reward rises from 0.438 to 0.556.

The authors explicitly do not call that reward movement a demonstrated model-quality improvement. There is one run, one task distribution, and no separate estimate of run-to-run variation. The run therefore supports a narrower conclusion: Miles can coordinate asynchronous rollout, optimization, synchronization, caching, and numerical checks at frontier-model scale.

That boundary sharpens the operational lesson. Once post-training overlaps work across subsystems, the organization needs explicit contracts for what a trajectory was, which policy produced it, how it was routed, what probability the rollout stack assigned, and how updated weights reached serving. Higher utilization is valuable only when those contracts remain measurable and verified for the configuration actually being operated.

Cognaptus: Automate the Present, Incubate the Future.