Clean Less, Route Better: DataOrchestra Reframes Pretraining Data Curation

TL;DR for operators

A pretraining-data pipeline receives millions of uneven records. Some are unusable, some contain removable noise, some need structural repair or added explanation, and some are already valuable enough that further processing may damage them. The operational problem is therefore not how to apply more cleaning, but how to decide which intervention—if any—each example needs.

DataOrchestra adds a small routing layer that can delete a chunk, preserve it, or send it through selected processing stages. In the paper’s fixed-pipeline comparison, this adaptive approach achieved an average score of 39.99 at an estimated 782 EFLOPs, while the complete Drop+NP+SR+PA pipeline reached 39.59 at 1,236 EFLOPs. The more comprehensive pipeline used substantially more estimated compute and still performed worse.

For data-platform teams, the design choice is between a uniform cleaning factory and an inspectable control layer that allocates filtering and rewriting capacity per example. DataOrchestra’s results support the latter: protect useful source material, reserve expensive transformations for records that need them, and judge routing plans by the actual outputs of downstream tools rather than by abstract recommendations alone.

The most comprehensive pipeline was not the best pipeline

Large data operations often begin with a reasonable assumption: if one cleaning stage helps, applying several stages should produce an even better corpus. The problem is that records do not arrive with the same defects. A transformation that repairs one chunk may remove useful structure from another, while an expansion step may add tokens to material that was already concise and complete.

A fixed pipeline behaves like an assembly line: every retained example passes through the same stations regardless of condition. In the paper’s controlled 0.5B-model comparison, the full sequence of dropping, noise pruning, surface rectification, and pedagogical augmentation produced a 39.59 average score at 1,236 estimated EFLOPs. DataOrchestra selectively invoked those operations and reached 39.99 at 782 EFLOPs.

The 0.40-point performance difference is modest in isolation. Its interpretation changes when paired with roughly 36.7% lower estimated curation inference compute. The result suggests that the additional processing in the fixed pipeline was not merely redundant. Some of it was likely counterproductive.

Adding selected raw text to the adaptive output raised the average further to 40.13 without increasing the reported curation EFLOPs. This is consistent with another finding: rewriting-only approaches improved some knowledge-oriented tasks but often lost performance on language-understanding benchmarks. More synthetic transformation was not uniformly associated with more useful pretraining data.

Preservation is an active curation decision

DataOrchestra splits documents into ordered chunks of no more than 1,024 Qwen3-tokenizer tokens. Each chunk receives one of three top-level decisions: remove it, preserve it without modification, or clean it with an ordered subset of three operations.

The preservation route—called untouch in the paper—is central to the design. It recognizes that already-useful text can be damaged through paraphrasing, compression, expansion, or stylistic normalization. Leaving a chunk unchanged is therefore not the absence of a quality policy. It is the output of one.

Chunks selected for cleaning can receive:

• Noise pruning, which removes irrelevant or defective lines.
• Surface rectification, which repairs presentation, organization, or readability.
• Pedagogical augmentation, which expands material with additional explanation.

These interventions have visibly different effects. Average token counts fell from 576 to 387 under pruning and from 637 to 540 under rectification. Augmentation moved in the opposite direction, increasing the average from 670 to 1,594 tokens. Treating these operations as interchangeable forms of “rewriting” would obscure the different defects they address and the different costs they create.

The routing distribution reinforces this heterogeneity. The orchestrator dropped 25.1% of chunks and preserved 10.8% unchanged. Cleaning assignments overlapped: 55.0% received pruning, 57.8% rectification, and 27.0% augmentation. A chunk could receive several stages, but it did not have to receive all of them.

The decision layer is trained against actual tool behavior

A routing model trained only on abstract labels could learn what a teacher model believes should happen without learning what the downstream tools actually do. DataOrchestra addresses this through execution-grounded plan evolution.

A large teacher first proposes a coarse plan for each chunk. The system then runs the selected tools and evaluates their outputs. Harmful or ineffective stages can be removed. Failed rewrites can be retried up to five times. Rewrites that barely change the source may be discarded. For rectification and augmentation, the system also generates a chunk-specific instruction rather than relying only on a common stage prompt.

This process produces roughly 300,000 chunk-plan pairs used to fine-tune a 1.7B-parameter orchestrator. The result is better understood as a control layer than as a simple quality classifier. It predicts not only whether text is good or bad, but which downstream action is appropriate given the behavior of the available tools.

The paper’s ablations test this mechanism rather than introduce a separate thesis. Removing fine-grained plan evolution reduced downstream performance. Suppressing chunk-specific instructions also lowered the reported average. These tests support the claim that execution feedback and example-level objectives contribute to the final result; they do not isolate a universally optimal plan-generation method.

The benchmark advantage grows with model scale

On RedPajama-V2, DataOrchestra reported the highest average at all three tested model sizes:

The increasing margin is noteworthy because data interventions that register at small scale do not always remain useful as model capacity grows. Here, the comparative advantage expanded from 0.90 to 1.92 average points.

Additional experiments broaden the evidence without making it universal. At 0.5B, the method also produced the highest reported averages on DCLM-RefinedWeb, C4, and FineWeb. In continued pretraining of a 3B model, it led the reported science-reasoning averages on both OpenWebMath and MegaMath.

Rewrite-fidelity analysis offers supporting evidence for the preservation mechanism. The full system achieved an 81.6% no-content-loss rate and a 96.7% no-factual-error rate in a teacher-based evaluation, outperforming the rewriting baselines on those measures. These numbers remain dependent on an LLM evaluator. Manual checks reportedly exceeded 86% agreement for the rewrite-quality assessment, but the reviewed sample’s size and composition were not reported.

What a data-platform team would build differently

What the paper directly shows: Within matched pretraining configurations, conditional routing produced better benchmark averages than uniform individual methods and the full fixed pipeline. It also required fewer estimated inference FLOPs than forcing all retained chunks through every available stage.

What Cognaptus infers: A production curation platform could separate orchestration from execution. The decision layer would emit an inspectable chunk-level plan, while pruning, repair, and augmentation tools remain replaceable modules. Operators could record invocation rates, rejection reasons, token expansion, verifier outcomes, and stage-level cost before admitting transformed text into a training corpus.

This design changes several operational decisions. GPU capacity can be allocated by intervention need rather than corpus size alone. High-quality source text can be protected through an explicit preservation policy. Teams can replace one rewriting component without rebuilding the entire pipeline. Auditors can inspect why a chunk was deleted, retained, or expanded rather than receiving only the final transformed document.

The system also creates a measurable build-versus-route comparison. Planner overhead is justified only when avoided tool calls and improved data utility exceed the cost of orchestration, verification, retries, and logging. DataOrchestra’s reported results make that trade-off plausible, not settled for every deployment.

Local validation must include cost and preservation

The experiments focus on web text and mathematical continued pretraining. They do not establish the same advantage for multilingual corpora, code, legal documents, scientific literature outside the tested benchmarks, or multimodal data.

The implementation is also concentrated within the Qwen3 family: teacher, orchestrator, verifier, pruning tool, and rewriting models all come from related model infrastructure. Whether the training procedure transfers cleanly to heterogeneous commercial tool stacks remains untested.

The benchmark values are averages over the three most recent checkpoints, not independent repeated training runs. The paper provides no confidence intervals or significance tests, limiting how precisely the reported margins can be separated from training variability.

Finally, EFLOPs are analytical estimates of Transformer inference work. They include orchestrator and tool-model passes but not every production cost. Wall-clock latency, memory contention, data movement, scheduling overhead, retry amplification, engineering maintenance, and cloud pricing could alter the realized economics.

A serious pilot should therefore measure two outcomes together: downstream model utility and preservation failures. Lower preprocessing spend is not a sufficient success criterion if the routing layer deletes valuable material or allows rewriting tools to change meaning. Conversely, perfect preservation is not enough if the pipeline retains low-value text and fails to improve the trained model.

Curation becomes a control problem

DataOrchestra’s contribution is not another argument for stronger filtering or more capable rewriting. It is evidence that the allocation of interventions can be as consequential as the interventions themselves.

For operators, the immediate design principle is concrete: do not send every record through the most expensive available transformation. Build a decision layer that can remove, preserve, or selectively repair each example; test its plans against real tool outputs; and measure both model outcomes and processing costs.

The paper provides a strong comparative case for that architecture within its experimental scope. The next step for a production team is not to assume the same savings, but to test whether its own corpora contain enough heterogeneity—and enough valuable text at risk of unnecessary rewriting—to justify orchestration.

Cognaptus: Automate the Present, Incubate the Future.