A Near-Perfect Score on the Wrong Evidence

TL;DR for operators

An autonomous data science agent can generate executable code, report an excellent predictive score, and still produce an invalid modelling pipeline. In a benchmark covering 1,761 pipelines across 220 tasks, methodological failures were widespread, and data leakage was associated with better evaluation rankings.

More sophisticated agents eliminated some basic omissions without consistently preventing leakage. Controlled tests revealed weaknesses in recognizing causal and feature-admissibility constraints.

For enterprise analytics teams, predictive performance and methodological validity require separate acceptance checks. Feature availability, data partitioning, preprocessing, and validation design deserve independent verification. Automated audits can help, but the study does not establish that they can reliably certify a pipeline for production.

The score looked excellent because the model knew too much

An analytics team receives a prediction pipeline from an autonomous agent. The code executes successfully, the reported performance looks exceptional, and the next decision is whether to advance the model toward deployment.

Before accepting it, the team needs to establish whether the model could have produced its predictions using only information available when those predictions would actually be made.

In a Kaggle competition involving problematic internet use, two autonomous data science agents, ReAct and AIDE, exploited features that revealed the prediction target in the local test data. These features were unavailable in the competition’s final test set. The agents reported misleading local scores near 1.0, despite relying on information they could not legitimately use for final evaluation.

This is data leakage: information that should be unavailable during model development or prediction influences the pipeline and makes its measured performance unreliable.

The example comes from The Silent Failures in Agentic Data Science, by Xue Li and colleagues.¹ Their investigation asks whether autonomous data science agents follow sound statistical practices, rather than merely generating working code and competitive scores.

For organizations evaluating autonomous modelling systems, the distinction changes what constitutes sufficient evidence for acceptance.

Across 1,761 pipelines, methodological failures were widespread

The researchers assembled 220 tabular prediction tasks from STRABLE, Kaggle, OpenML, UCI, and MLE-bench. They evaluated three agent frameworks—ReAct, AIDE, and MLEvolve—combined with three language models, producing nine agent–model configurations.

The main benchmark yielded 1,761 pipelines. A two-stage LLM-based evaluator inspected their operations against 15 predefined failure modes, including incorrect task formulation, preprocessing before data splitting, inappropriate validation, and flawed model evaluation.

The experimental design made an 80:20 train/test partition available to agents while expecting them to preserve the separation. This deliberately created an opportunity to observe whether agents would improperly use evaluation information.

The results were substantial: most configurations exhibited at least one detected failure in approximately 95%–100% of pipelines. The main exception, MLEvolve with gemini-3.7-flash, still had failures in approximately 68%.

These figures describe failures detected under the study’s evaluation procedures, not verified rates of production defects.

Importantly, the failure counts cannot be interpreted without considering what an agent actually attempted.

A pipeline that never ensembles models cannot commit a defective ensembling procedure. An agent that skips feature engineering may avoid mistakes associated with feature transformations, but that does not establish a better modelling solution.

Consequently, fewer detected failures can reflect a less complete pipeline rather than superior methodological competence.

This matters when comparing agent frameworks. An organization selecting the configuration with the smallest raw failure count could inadvertently reward the system that performs fewer substantive modelling operations.

Leakage can improve the ranking that selects the agent

The researchers next examined whether specific methodological failures were associated with predictive performance.

Because prediction metrics varied between tasks, they compared pipelines using within-task performance rankings. Their regression analysis adjusted for agent-framework and language-model identity (Section 4.2; Appendix A.6).

The uncomfortable result was that leakage-related failures were associated with more favorable predictive rankings.

The mechanism is understandable. If information from evaluation data enters preprocessing, feature selection, model fitting, or optimization, a pipeline may achieve an artificially favorable score. A performance-focused evaluation can then reward the very procedure that compromises its validity.

The result requires careful interpretation. These are adjusted statistical associations, not causal estimates. The regressions do not establish that introducing leakage causes a particular improvement in rank, nor that leaked pipelines generalize better to genuinely unseen observations.

Nevertheless, the finding identifies a concrete evaluation weakness: a ranking based solely on predictive performance may favor pipelines whose evaluation procedures are invalid.

That weakness becomes especially consequential when autonomous agents repeatedly optimize against accessible evaluation information.

More sophisticated agents fix omissions but can introduce new errors

Greater framework sophistication produced meaningful improvements in some methodological practices.

With gemini-3.7-flash, the frequency of missing validation splits fell from 40% under ReAct to 0% under both AIDE and MLEvolve. MLEvolve also reduced missing feature engineering from 38% under ReAct to 0%, and naive ensembling from 90% to 4% among applicable pipelines (Table 3).

These improvements indicate that additional agent capabilities can make pipelines more complete.

Yet the supplementary failure analysis uncovered a different problem. Approximately 50% of pipelines generated by MLEvolve with gemini-3.7-flash exhibited validation-to-training leakage, despite the framework having a dedicated leakage-checking component (Appendix A.7).

The predefined 15-mode taxonomy also missed additional methodological problems in approximately 37% of all evaluated pipelines. These included previously unclassified forms of validation-to-training and test-to-training contamination.

This supplementary analysis serves a different purpose from the main benchmark: it probes failures that the original evaluation checklist was not designed to capture.

Together, the findings suggest that additional pipeline operations create additional requirements for methodological control. More capable agents may perform steps that simpler agents omit, but completing those steps does not guarantee their correct execution.

For platform teams comparing agent frameworks, the relevant questions therefore include both which operations a system performs and how reliably it preserves constraints throughout those operations.

Controlled tests expose failures that better prompts do not always resolve

The researchers constructed a separate diagnostic suite of 17 tasks based on three datasets, targeting six methodological problems. These included class imbalance, inconsistent formatting, spurious correlations, domain-specific feature engineering, prohibited features, and unsuitable evaluation metrics.

Each agent–task–hint configuration was repeated five times.

In the baseline tests, all six reported agent–model configurations failed every run of the correlation-versus-causation test and the prohibited-feature selection test (Section 4.3).

These failures concern decisions that are not resolved merely by executing code correctly. Agents must understand whether a feature is substantively appropriate, whether it will exist at legitimate prediction time, and whether a statistical relationship supports the proposed modelling choice.

Explicit methodological hints substantially reduced failures for ReAct and AIDE, often approaching zero under the strongest guidance. MLEvolve did not sustain comparable improvements.

The controlled suite is a diagnostic test of specific capabilities, not an estimate of how frequently these errors occur in ordinary enterprise projects. It nevertheless suggests that adding instructions is not a uniformly reliable safeguard across agent architectures.

A separate comparison against first-place human solutions reinforces the distinction between automated pipeline construction and task-specific modelling competence.

Across five Kaggle competitions, the winning human entries outperformed the evaluated agent solutions on both private and public leaderboards (Section 4.4; Appendix A.8). The specialized challenges exposed larger gaps than simpler playground competitions.

In the Problematic Internet Use challenge, the winning solution employed a different problem formulation: predicting a continuous intermediate target, optimizing thresholds, using a Tweedie objective, and combining multi-seed cross-validation with model diversity.

That comparison is informative about the capabilities missing from the evaluated agents, although the human competitors and agents were not matched for development time, computational resources, or expertise.

Pipeline acceptance needs independent methodological checks

For enterprise analytics teams, Cognaptus draws a practical inference from these results: an agent-generated pipeline should not advance solely because its code runs and its reported score exceeds a threshold.

A more defensible acceptance process would examine four questions:

These checks should be applied to the operations actually performed, with task-specific scrutiny where necessary.

Independent review is particularly relevant when an agent’s optimization process has repeated access to evaluation results or when development data contain fields unavailable in deployment.

This is a proposed governance response to the study’s findings, not an experimentally demonstrated production intervention. The paper does not quantify how much such checks would reduce operational incidents, development costs, or deployment failures.

Automated audits remain an incomplete safeguard

The researchers validated their LLM-based failure detector against independent human annotations of 19 pipelines, covering 285 binary assessments.

The detector achieved 89.5% agreement, Cohen’s kappa of 0.62, precision of 0.71, and recall of 0.65 (Section 4.1; Appendix A.4).

The recall figure is especially relevant for governance: the detector missed a meaningful share of failures identified in the annotated sample. Its limited validation set also restricts confidence in estimates for individual or rare failure categories.

Additional boundary conditions matter. The benchmark intentionally exposed evaluation data to agents, its controlled tests contained constructed pitfalls, and the expert comparison covered only five selected competitions. Supplementary memorization checks found no statistically significant effects across six reported comparisons, but did not establish that contamination was absent.

The strongest conclusion is therefore about the existence and diversity of methodological weaknesses in the tested systems, rather than their universal prevalence or the effectiveness of a particular production audit.

Reliability begins before the score is accepted

Autonomous data science agents can substantially automate the construction of prediction pipelines. The study shows why successful construction should not be confused with successful methodological reasoning.

A strong model can exploit unavailable information. A sophisticated framework can eliminate basic omissions while introducing leakage elsewhere. An automated evaluator can detect many errors while missing others.

For analytics organizations, the appropriate response is to preserve predictive performance as one measure of success while requiring separate evidence that the modelling procedure is valid for its intended use.

The decisive question is not only how accurately the agent predicts, but whether the information and procedures producing that accuracy would remain legitimate outside the evaluation environment.

Cognaptus: Automate the Present, Incubate the Future.