Scaling Smarter, Not Larger: Why Your AI Dataset Is Probably Wasting Money

The expensive habit of feeding the machine

Data teams have a familiar ritual. The model disappoints. Someone asks for more data. Another person asks for cleaner data. A third person, usually with a spreadsheet and a suspiciously calm face, asks whether the extra labeling budget is approved.

Then the pipeline expands.

More driving clips. More corner cases. More annotated scenes. More storage. More training runs. More dashboards explaining why the latest model is still not quite where it should be.

This habit feels rational because, at small scale, more data often does help. The trouble begins when “more” becomes the strategy. In large physical-AI systems, the question is no longer whether another sample is useful in some abstract sense. The question is whether that next sample improves the metric that still matters, at the point where the system currently sits.

That is the useful discomfort in Scaling-Aware Data Selection for End-to-End Autonomous Driving Systems, which introduces MOSAIC, short for Mixture Optimization via Scaling-Aware Iterative Collection.¹ The paper is framed around autonomous driving, but the deeper argument is broader: an AI dataset is not a pile of assets. It is a portfolio of domains with different expected returns, different saturation speeds, and different effects on the metrics that determine whether the product can actually be used.

Some samples are early-stage growth stocks. Some are mature utilities. Some are decorative collectibles purchased by a data team with too much confidence and not enough marginal analysis.

MOSAIC matters because it gives structure to a question many AI organizations avoid asking directly: which part of the dataset is still earning its keep?

The real problem is not diversity; it is changing marginal value

A common reaction to data inefficiency is to seek diversity. That instinct is not wrong. A self-driving system trained only on calm suburban roads will not become robust by being shown more calm suburban roads. Diversity matters.

But diversity alone is an incomplete rule. So is uncertainty. So is representation coverage. These signals can identify samples that look different, confusing, or geometrically underrepresented, but they do not necessarily answer the operational question: what will this sample improve after the model has already learned from thousands of other samples?

MOSAIC’s central move is to replace static usefulness with changing marginal utility.

The paper studies end-to-end autonomous-driving planners, where the model is not judged by a single neat score such as classification accuracy. It is evaluated through EPDMS, the Extended Predictive Driver Model Score, which aggregates nine rule-compliance metrics, including no collision, drivable-area compliance, driving-direction compliance, traffic-light compliance, ego progress, time-to-collision, lane keeping, history comfort, and ego comfort.

That aggregation matters. A clip from dense urban traffic may help collision-related behavior but do little for lane keeping. A curvy-road clip may improve lane keeping while stressing comfort. A rainy or overcast scene may be useful only after the model has already consumed enough routine conditions. The value of a domain is not fixed. It moves with the training budget.

MOSAIC is therefore not asking the generic active-learning question:

Which samples are informative?

It asks a more commercially annoying question:

Which domain still has the highest next-unit return against the metric bundle we actually care about?

This is the mechanism-first lens through which the paper should be read. The headline result is data efficiency. The more transferable idea is that data collection becomes a sequential budget-allocation problem.

MOSAIC turns the dataset into domains, rankings, and scaling curves

The MOSAIC pipeline has three pieces. None is exotic in isolation. The combination is the point.

First, the data pool is partitioned into domains. In the main autonomous-driving experiments, these domains are based on geolocation: Boston, Pittsburgh, Singapore, and Las Vegas. In a robustness-style variant, the authors also cluster clips using generated captions, producing semantic groups such as calm streets, pedestrian crossings, highways, parking areas, city streets, and rainy or overcast conditions.

Second, samples inside each domain are ranked. The authors use an importance signal derived from model evaluation on the sample. The practical idea is simple: if a domain contains both highly influential and mediocre samples, the selection policy should not treat them equally. It should pull the stronger samples first.

Third, MOSAIC fits a scaling curve for each domain. The paper uses a saturating exponential form:

Here, $\Delta \hat{U}_i(n_i)$ is the estimated improvement in the aggregate utility from adding $n_i$ samples from domain $i$. The parameter $a_i$ captures the domain’s asymptotic contribution, while $\tau_i$ governs how quickly the domain saturates. A domain with a high early gain may be attractive at 200 clips and uninteresting at 2,000. Another domain may look mediocre at first but become valuable after fast-saturating domains have been exhausted.

That is the quiet trick. MOSAIC does not merely decide that Boston or Singapore or Pittsburgh is “important.” It estimates when each one stops being the best next choice.

The selection algorithm then proceeds iteratively. At each step, it computes the first-difference gain from adding one more sample from each domain and chooses the domain with the highest estimated marginal improvement. Then it takes the next ranked sample from that domain. Repeat until the budget is exhausted.

A simplified operational view looks like this:

The result is a selection policy that can shift over time. Early allocation may favor high-return domains. Later allocation may move toward slower, steadier domains once the early domains saturate. This is not philosophical data-centric AI. It is a data budget with a yield curve.

The main evidence: same budgets, better EPDMS

The paper evaluates MOSAIC on OpenScene and Navtrain configurations using Hydra-MDP, a challenge-winning autonomous-driving planner. The authors compare MOSAIC with Random selection, Uncertainty selection, Coreset selection, and Chameleon, a data-mixture baseline.

The main table reports validation EPDMS and BRMR, the Budget Ratio to Match Random. BRMR asks how much data a method needs to match the EPDMS reached by Random at a given budget. Lower is better. A BRMR of 0.20 means the method needs about one-fifth of Random’s data to reach the same score. That is the sort of ratio that makes brute force look less heroic and more like poor procurement.

The headline numbers are consistent across the two datasets:

Two readings are worth separating.

The direct paper result is that MOSAIC outperforms the baselines in EPDMS at the reported budgets. On OpenScene, it reaches a BRMR around 0.15–0.18 in the highlighted budgets, meaning more than 80% fewer samples than Random to match comparable performance. On Navtrain, the sample-efficiency gain is smaller but still large, with BRMR values around 0.30–0.38 in the highlighted budgets.

The business interpretation is not “MOSAIC will reduce your data bill by 80%.” That would be a lovely sentence for a sales deck and therefore immediately suspicious. The better interpretation is narrower and more useful: when the metric bundle is known, and the data pool can be partitioned into meaningful domains, scaling-aware allocation can identify high-yield data much earlier than generic selection rules.

That is enough. A data strategy does not need magic. It needs fewer expensive superstitions.

The metric breakdown shows why aggregate scores can mislead

The paper’s Table 2 is more important than a casual reader might think. It breaks EPDMS into its nine component metrics for OpenScene at 4,000 clips and Navtrain at 1,600 clips.

This table is not just extra reporting. It explains why MOSAIC is different from methods that optimize generic uncertainty or diversity.

In the OpenScene setting, the base model is especially weak on DAC and EC relative to several other already-high metrics. MOSAIC improves DAC from 83.9 in the base model to 93.59 at 4,000 selected clips. Random reaches 90.53, Coreset reaches 92.93, and Chameleon reaches 92.32. MOSAIC is not uniformly best on every submetric, but it is strong where the aggregate score has meaningful headroom.

That distinction matters. A naive method can improve a metric that is easy to improve or already near saturation. The dashboard looks busy. The KPI barely moves. Everyone has a meeting.

MOSAIC’s advantage is that it models the relationship between domains and the aggregate utility. In the authors’ interpretation, it shifts effort away from saturated or less impactful metrics and toward areas that still improve EPDMS. For a business audience, that is the more general lesson: metric design and data selection cannot be separated. If the utility function is wrong, the selection policy will optimize the wrong bargain very efficiently.

A practical organization should therefore ask four questions before copying the MOSAIC idea:

This is why the paper’s mechanism is more interesting than its benchmark table. The benchmark says MOSAIC wins. The mechanism says why the winning allocation changes over the budget.

The ablations explain when the machine should change its mind

The most instructive part of the paper is the ablation section, because it shows the internal dynamics of selection rather than only the final score.

In the OpenScene experiment, the authors fit scaling curves for city-based clusters. Boston and Singapore offer the strongest early gains below roughly 500 clips. Pittsburgh improves more steadily and becomes increasingly attractive at higher budgets. Las Vegas offers the smallest gains and saturates early, but it is not ignored forever. Once Boston, Singapore, and Pittsburgh approach saturation, the marginal value of initial Las Vegas samples becomes competitive enough for MOSAIC to mine from it.

That is exactly what a scaling-aware strategy should do. It should not permanently worship the best early domain. It should change its allocation when the return curve bends.

The paper’s visualized selection process shows this budget-dependent shift: early stages focus on Boston and Singapore; Pittsburgh dominates much of the later selection; after around 3,700 selection rounds, Pittsburgh is exhausted and the policy shifts again. This is not a fixed domain-weighting method. It is a sequential allocation policy reacting to estimated marginal returns.

For business readers, the analogy is straightforward. A customer-support automation model may initially improve most from refund-dispute examples. Later, once refund handling is competent, escalation cases may become more valuable. A warehouse robot may first need common aisle navigation, then rare obstruction patterns, then human-interaction edge cases. A financial AI system may first need clean mainstream disclosures, then messy footnotes, then litigation-related text.

The category with the highest value today is not necessarily the category that deserves the next dollar tomorrow. This is obvious in finance. Somehow, data strategy still needs to be told.

The robustness tests are useful, but they are not a second thesis

The paper includes several tests that should be interpreted carefully.

The caption-based clustering experiment is best read as a robustness test. Instead of using city/geolocation clusters, the authors generate captions for Navtrain clips with Qwen-2.5-VL-32B-Instruct, cluster TF-IDF features into six semantic groups, and rerun the selection comparison. MOSAIC again outperforms the baselines. At the 1,600- and 2,400-clip budgets, it requires 61% and 52% fewer clips than Random to match performance. It also reaches the full-training performance using 2,400 clips, or 42% fewer samples than the full 4,141-clip pool.

The purpose is not to prove that caption clustering is universally superior. In fact, the appendix notes that a first attempt using a sentence-transformer embedding model produced clusters that lacked coherent driving characteristics, while TF-IDF over generated captions was more interpretable. That is a small detail with a large warning label attached: better embeddings do not automatically mean better operational domains.

The clustering/ranking ablation is more diagnostic. The authors test variants without clustering and without ranking. In the low-data regime, up to about 800 clips, much of the improvement comes from ranking: simply taking high-impact samples is already powerful. At larger budgets, however, ranking alone begins to lag behind full MOSAIC because it does not manage domain-level saturation. Clustering without ranking also loses information because it treats samples inside a domain too uniformly.

A concise map of the evidence looks like this:

The compute analysis is especially relevant for managers. MOSAIC requires upfront pilot runs to estimate cluster-specific scaling curves. The authors reduce this overhead by continual training from a base model checkpoint rather than full training from scratch. In OpenScene, they report that MOSAIC is not the strongest method at small compute budgets, but its pilot overhead amortizes over time. At the highest compute budget, MOSAIC reaches the top-baseline performance with 16% less compute, saving about 490 A100 GPU hours; compared with Random, it requires 57% less compute, saving about 1,700 GPU hours to attain the same EPDMS.

So the compute lesson is not “always run MOSAIC.” It is more disciplined: if your project is too small for pilot-run overhead to amortize, simpler ranking or coreset methods may be more practical. If your data program is large enough that repeated training and annotation waste matter, scaling-aware selection becomes much more attractive.

The business value is cheaper diagnosis, not just cheaper training

The obvious business interpretation is cost reduction. Fewer clips. Less annotation. Fewer GPU hours. Cleaner procurement conversations. All pleasant.

But the more durable value is diagnostic.

MOSAIC forces a data team to expose three things that are often hidden inside vague “data quality” language:

1. which operational domains exist;
2. which metrics each domain moves;
3. where each domain’s marginal return begins to decay.

That diagnostic structure is useful even if an organization does not implement MOSAIC exactly. A logistics company training warehouse robots can segment by aisle type, lighting condition, object density, and human proximity. A healthcare imaging team can segment by scanner type, anatomical region, institution, and acquisition protocol. A customer-service automation team can segment by intent class, escalation path, language, and policy ambiguity.

In each case, the business question is not “Do we need more data?” It is:

Which segment still improves the KPI that blocks deployment?

This changes how AI leaders should think about data budgets.

This also changes vendor conversations. If a labeling vendor, simulation provider, or data broker cannot explain which failure mode their data is expected to improve, the buyer is not purchasing training fuel. The buyer is purchasing hope in a zip file.

Where the result applies, and where it should not be stretched

The boundaries of the paper are clear enough, and they matter.

First, the evidence comes from end-to-end autonomous-driving planner evaluation using Hydra-MDP on OpenScene and Navtrain-style configurations. That is a strong testbed for physical AI, but it is still one family of tasks. The result should be treated as evidence that scaling-aware data selection can work in a complex multi-metric physical-AI setting, not proof that the exact recipe transfers unchanged to every robotics, healthcare, or enterprise AI problem.

Second, MOSAIC depends on meaningful clustering. The method assumes that cluster-wise first-order scaling captures the dominant variation in performance. The appendix makes this more precise: interaction terms are not assumed to be zero, but they are treated as residual error. In the Navtrain geolocation analysis, estimated EPDMS overstates actual EPDMS by up to about 1 point at higher budgets. That is modest in the tested setting, but it also tells us the approximation is real, not decorative.

If clusters are random, semantically incoherent, or operationally irrelevant, the method can allocate confidently in the wrong direction. Confidence, unfortunately, is not accuracy wearing a suit.

Third, the ranking signal is not free. The paper’s main ranking method uses EPDMS-based importance. The authors test cheaper alternatives—trajectory imitation loss, gradient norm, and output sensitivity to gradient perturbations—but find that none correlates strongly with EPDMS-based ranking. For commercial adoption, this is not a footnote. It is a cost model. If obtaining the ranking signal requires dense annotations or expensive evaluation, the savings from selection must be weighed against the measurement cost.

Fourth, MOSAIC has upfront compute overhead. Pilot runs are needed to fit the scaling curves. The paper’s compute analysis suggests this overhead can be recovered in the large-budget regime, but not necessarily at small budgets. In other words, MOSAIC is not a magic coupon. It is an investment in measurement. Like most investments in measurement, it is most valuable when the downstream spending is large enough to make being wrong expensive.

What Cognaptus would take from this paper

The immediate lesson is not that every company should implement MOSAIC tomorrow morning. Please do not let a blog article become a Jira ticket by lunchtime.

The better lesson is that serious AI data strategy needs a marginal-utility layer.

For many organizations, the first practical version would be simpler than the paper:

1. define the deployment-relevant metric bundle;
2. segment the data pool into operationally meaningful domains;
3. run small pilot fine-tunes or evaluations for each domain;
4. estimate whether returns are fast-saturating, slow-saturating, or flat;
5. allocate the next labeling or collection budget to the highest expected marginal gain;
6. repeat after the model changes.

This would already be an improvement over the usual ritual of asking for more data with the solemnity of a rain dance.

The strategic point is that dataset size is becoming a weak signal of AI capability. What matters is not just how much data a company owns, but whether it knows the yield curve of its data domains. In physical AI, that yield curve may decide whether a system improves the metric blocking deployment or merely becomes more expensive to train.

MOSAIC’s contribution is to make that curve explicit. It clusters the operating world, ranks samples inside each cluster, estimates how each cluster scales, and then allocates data by marginal gain. The benchmark gains are impressive, but the framework is the part worth carrying into other domains.

The next competitive edge in AI will not come from treating data as an ever-growing warehouse. Warehouses are useful. They are also where companies store things they forgot how to value.

The better question is sharper: what should the next unit of data buy?

Until a data pipeline can answer that, it is probably wasting money.

Cognaptus: Automate the Present, Incubate the Future.