Less Prompt, More Blueprint: MOSAIC and the Data-Science Agent That Keeps Receipts

TL;DR for operators

MOSAIC is best read as a system-design paper, not as another entry in the increasingly crowded genre of “we attached an LLM to Python and hoped for the best.” The paper introduces a structured agentic framework for automated data science where the agent builds an explicit workflow blueprint before generating code, then verifies, executes, and refines candidates using diagnostic feedback and failure-aware offline reinforcement learning.¹

The operational point is simple: data-science automation fails less embarrassingly when it has memory, contracts, and rollback discipline. Charming, yes. Also what production systems usually require.

MOSAIC’s pipeline starts by turning a task and dataset into a semantic profile, including schema, statistics, visual time-series patterns, and engineered features. It then retrieves prior task-solution cases and source-code modules, extracts reusable components, constructs a blueprint specifying module composition and shape constraints, generates executable candidates, validates them by running them, and refines the best candidates over multiple steps.

The paper tests this system on financial time-series forecasting and generation across cryptocurrency, limit-order-book, and stock datasets. Reported results show MOSAIC improving forecasting errors, distributional generation metrics, execution success, and refinement efficiency against AutoML and agentic baselines. On forecasting, the authors report average RMSE reductions over TS-Agent of 8% on LOB, 5% on Crypto, and 3% on Stock, plus a 33% average RMSE reduction over AutoGluon on LOB. On generation, MOSAIC reduces average LOB Marginal distance by more than 32% and Correlation distance by 27% relative to TS-Agent, while maintaining full success coverage across tested dataset-backbone pairs.

The business interpretation is narrower, and therefore more useful. MOSAIC suggests a route toward data-science agents that behave less like stochastic interns with shell access and more like governed modeling systems: reusable prior cases, inspectable module choices, explicit compatibility constraints, execution traces, and failure memory. That matters most in domains where modeling workflows are repeated, expensive to debug, and risky to deploy casually.

The boundary is not decorative. MOSAIC depends on the quality of its case bank, code bank, and refinement knowledge bank. It assumes useful module extraction from heterogeneous repositories. It also spends real compute on execution-based verification. The results are strongest as evidence for structured financial time-series automation, not as proof that a general-purpose data-science agent can now replace the modeling function. The robot has learned to keep a lab notebook. It has not become the lab.

The expensive part of data science is not typing the code

A surprisingly large share of AI-agent discourse treats code generation as the center of data science. This is understandable if one’s model of data science was formed by watching terminal demos. The cursor moves, the code appears, the plot renders, everyone pretends the hard part was syntax.

MOSAIC starts from a more adult premise. Automated data science is not merely the production of runnable code. It is the selection and coordination of data transformations, feature representations, model architectures, training procedures, evaluation protocols, and refinement strategies. These choices are coupled. A model architecture implies assumptions about the data. A training procedure implies assumptions about stability and sample size. An evaluation protocol implies what the model is actually being optimized to do. The paper’s framing is that data science is a structured model-selection and workflow-construction problem.

That framing is the first useful thing about the work. It shifts the unit of automation from “generate a script” to “construct a modeling workflow.” A script is an artifact. A workflow is an argument about why a particular modeling path should be admissible for a particular task.

This is where MOSAIC differs from ordinary AutoML and from many LLM agents. AutoML typically searches over predefined model and hyperparameter spaces. LLM agents can be more flexible, but their flexibility often appears as unconstrained code synthesis followed by debugging. MOSAIC tries to sit between those two failure modes: less rigid than fixed AutoML search, less feral than free-form agentic coding.

The system’s core idea is the blueprint. Before asking the LLM to generate code, MOSAIC constructs an intermediate representation of the workflow: selected modules, composition order, input-output interfaces, dimensional constraints, and execution requirements. The LLM is still present, but it is no longer the author of the entire plan from vibes and token probabilities. It becomes the executor of a structured modeling specification. A modest demotion. A productive one.

MOSAIC turns model-building into a staged control system

The mechanism is easiest to understand as a sequence of narrowing decisions. MOSAIC does not leap from task description to final model. It progressively constructs context.

This is the paper’s main contribution: not that each ingredient is unprecedented, but that the ingredients are assembled into a control loop for data-science workflow construction.

The task profile matters because retrieval quality depends on how the task is represented. MOSAIC’s profile includes schema information, statistical features, visual time-series patterns, and engineered features. The appendix details a semantic-aware EDA process that maps numerical properties into text-based anchors and uses visual reports from standardized plots to help bridge raw time-series data with textual case retrieval. In plainer terms: the system tries to make data characteristics legible to the agent before the agent starts choosing models.

The case bank then provides task-level memory. It contains curated task-solution pairs from financial benchmarks, competition entries, and studies. The code bank supplies implementation-level memory: forecasting models, generative models, and evaluation metrics. The refinement bank adds domain heuristics around preprocessing, training optimization, tuning, and evaluation. This is the unglamorous part of the system, which usually means it is where much of the practical value lives.

A useful enterprise analogy is not “AI scientist.” It is “modeling operations library with an agentic front end.” Less cinematic. More likely to survive contact with a compliance review.

The blueprint is the paper’s quiet governance layer

The blueprint deserves special attention because it is doing several jobs at once.

First, it narrows the search space. Instead of letting an LLM hallucinate an architecture, MOSAIC retrieves top candidate models, analyzes their architecture families, and extracts modules from the corresponding source code. If the candidates share a family, modules can be recombined more directly. If they span families, the top model becomes the backbone and other candidates contribute higher-level design ideas.

Second, it provides compatibility structure. Extracted modules are enriched with deterministic shape analysis and semantic annotations. The blueprint records dimensional constraints and composition order, which means code generation is no longer a freehand sketch. It is constrained by interface contracts.

Third, it creates traceability. If a generated model succeeds or fails, the system has a record of the retrieved cases, selected modules, blueprint constraints, generated code, execution diagnostics, and refinement actions. That is not the same as full formal verification. But it is a much better audit surface than “the agent generated model_v7_final_FINAL.py.”

This also explains why the likely misconception matters. MOSAIC is not merely an LLM code generator. The paper’s claim depends on a chain of structured steps: task profiling, retrieval, module extraction, blueprint construction, execution verification, and refinement. Remove the structure and the system becomes another code agent. Perhaps an impressive one. Also one that may eventually discover a new and expensive way to transpose a tensor.

The ablations support this reading. When the authors remove pieces of the model-generation module, performance drops across metrics. Removing the blueprint, module understanding, or verification weakens the system; the naive LLM variant performs worst in the model-generation ablation. The paper reports that full ModelGen reaches up to 70% win rates across comparisons, while some ablated variants fall near zero.

That is not just a performance detail. It is the causal story the paper wants the reader to see: the structure is not paperwork around the model. The structure is part of the model-building capability.

Failure is not thrown away; it becomes policy training data

The refinement module is the second major mechanism. Here MOSAIC treats model improvement as a sequential decision problem rather than a greedy edit-and-test loop.

That distinction matters. In modeling work, an edit can be locally neutral or temporarily harmful while enabling later gains. Adding normalization may only pay off after a learning-rate change. Changing model capacity may only help after regularization is adjusted. Conversely, a plausible-looking edit can create a shape mismatch, unstable loss, NaNs, timeouts, or an executable model that performs worse.

MOSAIC separates the high-level refinement decision from code realization. The reinforcement-learning policy chooses structured actions such as adding normalization, changing dropout, tuning batch size, modifying regularization, adjusting model capacity, or replacing a component. A frozen LLM executor translates the chosen action into concrete code edits. The edited program is then executed and evaluated.

The reward is based on loss improvement over the refinement horizon. In simplified form, the objective is to choose a policy $\pi$ that reduces cumulative loss changes:

The important engineering move is how MOSAIC treats failure. A hard failure occurs when the edit cannot run: syntax error, shape mismatch, missing dependency, NaN loss, timeout. A soft failure occurs when the code runs but validation loss degrades. In both cases, the system rolls back to a valid checkpoint. But it does not simply erase the bad path. It stores the failed branch as negative supervision and records the failed action in a state-specific invalid-action mask.

This creates a tree of refinement trajectories rather than a single linear history. The system learns not only what helped, but which actions should not be repeated from a similar diagnostic state. The paper uses Implicit Q-Learning for this offline refinement policy, trained on thousands of collected refinement transitions across generated tasks. It also uses invalid-action masking and soft rollback at deployment.

For operators, this is one of the most transferable ideas in the paper. Production automation improves when failed actions are not merely logged as shameful artifacts but converted into constraints, policies, and future avoidance behavior. Failure memory is underrated because it is less photogenic than success. Unfortunately for demos, systems tend to become useful after they remember what not to do.

What the evidence is actually testing

The paper’s evidence stack has several layers. Reading them as one big scoreboard would miss the structure.

The main results are directionally strong. Forecasting improves across Crypto, LOB, and Stock. The authors report that MOSAIC reduces RMSE relative to TS-Agent by 8% on LOB, 5% on Crypto, and 3% on Stock on average, with individual gains up to 19% on LOB using Claude and 8% on Crypto using GPT-5.4. Compared with AutoGluon, the LOB RMSE reduction reaches 33%.

The financial metrics are also meaningful, though they should be read carefully. On Crypto forecasting, the paper reports that average $\Delta$Sharpe is reduced by more than 21% relative to TS-Agent, with GPT-4o showing a 46% reduction. Lower $\Delta$Sharpe means the model’s downstream Sharpe behavior is closer to the reference. This matters because financial forecasting systems are often judged not only by point error but by how errors propagate into portfolio or risk behavior.

For generation, MOSAIC dominates LOB and Stock on several distributional metrics. The paper reports average LOB Marginal distance reductions above 32% and Correlation reductions of 27% compared with TS-Agent, plus a Stock Marginal reduction above 23%. On Crypto generation, MOSAIC beats DS-Agent, ResearchAgent, and Optuna on $\Delta$VaR and $\Delta$ES, but TS-Agent achieves lower tail-risk distances on that dataset. This is precisely the kind of inconvenient detail that makes the result more credible. The paper is not clean-room supremacy; it is structured improvement with edge cases.

Execution reliability is a second major result. MOSAIC maintains 100% success rate across tested dataset-backbone pairs, while DS-Agent and ResearchAgent have reported success rates ranging from 20% to 100% depending on configuration. In business language, this is not a footnote. A slightly better model that fails unpredictably is often worse than a modestly weaker system that executes reliably. The paper’s strongest operational claim is therefore not only “better metrics.” It is “better metrics under a workflow that runs, records, and repairs.”

The ablations say the plumbing is the product

The ablation studies are unusually important because they test whether MOSAIC’s staged architecture is decorative or causal. The answer, in the paper’s evidence, is mostly causal.

The EDA and multimodal ablation removes semantic EDA, feature engineering, and multimodal support. The authors report consistent degradation, especially for generation and financial metrics. In the detailed appendix, removing these components increases LOB Marginal distance sharply for some backbones, and Crypto $\Delta$Sharpe deteriorates substantially in one GPT-4o setting. The likely interpretation is that financial time-series modeling depends on statistical and visual priors: volatility, heavy tails, dependence, nonstationarity, and regime shifts are not incidental texture. They are the problem.

The model-generation ablation is even more central. Removing module understanding, blueprint construction, or verification weakens results; naive LLM generation performs worst. This supports the paper’s mechanism-first thesis. Repository-grounded generation is not merely “LLM plus examples.” It needs extracted modules, shape contracts, composition plans, and execution checks. The blueprint is doing work.

The RL ablation isolates invalid-action masking, soft revert, and trajectory branching. Removing soft revert causes the largest performance drop and increases refinement steps by roughly 20% in the summarized results. Removing trajectory branching and invalid-action masking also hurts. That pattern is coherent: if a system keeps moving forward from degraded executable states, it accumulates damage; if it discards failed edits entirely, it loses supervision; if it forgets which actions failed from which state, it repeats known mistakes. One almost hears the enterprise change-management office applauding quietly.

The modular comparisons add another layer. ModelGen achieves the highest average win rate in both forecasting and generation, reportedly 65% versus 48% and 63% versus 42% against compared alternatives. IQL-based refinement wins 83% of metric comparisons across datasets and LLM backbones and reaches the best incumbent in fewer steps than LLM-only refinement, 8.7 versus 10.0 on average. These are comparison-with-prior-work results, not ablations, and should be treated as evidence that MOSAIC’s implementation choices are competitive against other candidate strategies in this experimental setting.

The case studies show what “composition” means in practice

The case studies are not the main evidence. They are explanatory evidence: they show what the abstract mechanism looks like when the system builds actual models.

In the Crypto forecasting example, GPT-4o generates a hybrid architecture from PatchTST, DLinear, and TimeMixer. Because these models share a decomposition-and-projection style, MOSAIC uses component-level recombination. The generated model combines series decomposition, multi-scale convolution branches, auto-correlation attention, and a gated residual head. During verification, the system detects a projection-head shape mismatch and corrects it by adding separate temporal and channel linear layers. The resulting hybrid reports RMSE 0.205 compared with 0.217 for the best single-model baseline, PatchTST, and reduces Sharpe Ratio Difference from 12.4 to 2.07.

In the LOB generation example, Claude Opus 4 produces a diffusion-based model using Diffusion-TS as backbone, FIDE’s frequency-inflated score block as a direct replacement, and COSCI-GAN’s per-channel coordination idea as an adapted conditioning principle rather than a literal adversarial graft. This distinction matters. The system does not simply paste modules together like a desperate hackathon participant at 2 a.m. It uses family analysis to decide whether to transfer components directly or only borrow ideas. The final model reports Marginal distance 0.513 versus 0.767 for Diffusion-TS alone, and Autocorrelation distance 0.191 versus 0.215.

These examples support the paper’s claim that blueprint-constrained composition can create novel architectures that remain executable. They do not prove that MOSAIC will invent useful architectures in every domain. But they make the mechanism observable, which is valuable in a field that too often asks readers to accept “agent did things” as a method section.

The business value is reusable modeling memory, not autonomous magic

MOSAIC’s business relevance is clearest for organizations that repeatedly build related models over changing datasets: finance, demand forecasting, risk modeling, operations analytics, fraud detection, industrial monitoring, and any setting where the next modeling problem resembles prior problems but is not identical to them.

The direct paper result is financial time-series automation. The broader business inference is that agentic data science should be built around reusable modeling memory and explicit workflow objects.

The most practical near-term use is not letting an agent “own” data science end to end. A more plausible deployment is an internal modeling workbench that can assemble candidates from approved repositories, produce a blueprint, run tests, record diagnostics, and suggest refinements under human review. This is less glamorous than an autonomous quant in a box. It is also less likely to bankrupt anyone because a generated model confused a validation split with an act of destiny.

For regulated or high-stakes analytics, the audit trail may matter as much as the metric uplift. A MOSAIC-like system can record which prior cases influenced the decision, which modules were selected, what compatibility constraints were imposed, what failed during verification, and which refinements were applied or masked. That makes review easier. It also makes failure less mysterious.

The ROI pathway therefore runs through three operational levers:

1. Lower search waste. Reusing prior cases and modules reduces repeated modeling effort across similar tasks.
2. Higher execution reliability. Verification and repair reduce the burden of debugging generated candidates.
3. Better institutional learning. Failed refinements become structured negative supervision rather than forgotten notebook history.

The third lever may be the most durable. Organizations often accumulate code repositories but not modeling memory. MOSAIC treats that memory as a first-class input to automation.

Where MOSAIC still depends on fragile assumptions

The paper is explicit about several limitations, and they are not minor.

The first is knowledge-bank quality. MOSAIC’s performance ceiling depends on the case bank, code bank, and refinement bank. Bad cases, stale code, unrepresentative prior tasks, or weak evaluation templates would degrade the system. The agent is not escaping institutional memory; it is amplifying it. This is wonderful when the memory is curated. It is less wonderful when the memory is a graveyard of abandoned notebooks named after dates no one remembers.

The second is module extraction. MOSAIC assumes source-code modules can be identified, annotated, and recombined reliably. That is plausible in controlled repositories with consistent architecture patterns. It is harder in messy enterprise codebases where modeling logic, preprocessing, training loops, and business exceptions are braided together like historical sediment.

The third is computational cost. Execution-based verification, multiple generated candidates, warm-up refinement, optimization steps, debug retries, and long timeouts are not free. The appendix reports experiments on a server with an NVIDIA H200 GPU and substantial CPU/RAM resources. A production team would need to evaluate whether the quality gain justifies the runtime and infrastructure budget.

The fourth is domain scope. The experiments focus on financial time-series forecasting and generation. That is a demanding and useful domain, but it is not the whole of data science. Structured workflows may transfer well to other repeated modeling settings, but the paper does not demonstrate broad cross-domain generality.

The fifth is guarantee strength. Shape contracts and forward-pass checks help avoid many implementation errors, but they do not prove that a synthesized module is statistically appropriate, causally valid, robust under distribution shift, or aligned with business policy. MOSAIC improves the audit surface. It does not eliminate the need for model governance.

These limitations do not weaken the paper’s practical value. They define where the system should be deployed first: domains with repeated task families, high-quality internal repositories, standardized evaluation, and strong incentives to preserve modeling traceability.

The operator’s checklist for a MOSAIC-like system

A business team should not ask, “Can we buy MOSAIC?” The better question is whether its design principles can be implemented inside existing model operations.

A MOSAIC-like system needs:

The table also reveals why many current agent deployments feel brittle. They start with the LLM and add guardrails afterward. MOSAIC starts by structuring the modeling task and then lets the LLM operate inside that structure. The difference is not philosophical. It is architectural.

The real lesson: agents need intermediate objects

MOSAIC’s deepest lesson is that agentic systems become more useful when they create intermediate objects that can be inspected, executed, revised, and reused.

For data science, that object is the blueprint. For software engineering, it may be a design spec and test plan. For legal operations, it may be an issue map and evidence bundle. For finance, it may be a transaction rationale and risk-control trace. In each case, the agent should not jump from instruction to final artifact. It should build the thing that explains how the final artifact will be produced.

That is how automation becomes governable. Not by making the model more charismatic. Not by adding a longer system prompt. By forcing the system to leave behind structured decisions.

MOSAIC is not the final form of automated data science. It is too domain-specific, too dependent on curated banks, and too compute-heavy to be treated as an off-the-shelf answer. But it is an important design signal. The next useful generation of data-science agents will not be the ones that merely write code faster. They will be the ones that know what they are reusing, why they are composing it, whether it runs, what failed, and which failures should never be repeated.

A model-building agent with memory, contracts, execution checks, and rollback is still an agent. It is just one that has discovered paperwork. Naturally, that is when it starts becoming useful.

Cognaptus: Automate the Present, Incubate the Future.