Dare to Benchmark: Why Data Science Agents Still Trip Over Their Own Pipelines

Spreadsheet work has a special kind of comedy.

A person asks an AI agent to load a dataset, clean a few columns, train a model, generate predictions, and save a prediction.csv file. The agent writes plausible Python. The model architecture is reasonable. The explanation sounds confident. Then the whole thing fails because the agent forgot to pass the filename into the execution tool.

This is not a philosophical failure of intelligence. It is worse: an operational failure. The machine understood the assignment, admired the assignment, narrated the assignment, and still tripped over the pipe wrench lying directly in front of it.

That is why the arXiv paper DARE-bench: Evaluating Modeling and Instruction Fidelity of LLMs in Data Science is useful.¹ It is not merely asking whether large language models can “do data science.” That question is too vague to be dangerous. The better question is whether data-science agents can follow an executable workflow, respect constraints, produce reproducible outputs, and still build models that perform well.

The answer is: sometimes, but not often enough to justify the casual enthusiasm around “AI data scientists.” A charming demo is not an analytics function. A notebook that works once is not a production pipeline. A generated model that scores well while ignoring instructions is not a reliable employee. It is a very expensive intern with root access.

DARE-bench matters because it turns these uncomfortable distinctions into measurable evidence.

The benchmark’s core move is to make data-science work verifiable

Many AI benchmarks are still built around final answers. Did the model solve the coding problem? Did it answer the math question? Did it pick the right option? That is useful, but data science is not a short-answer exam. A data-science workflow includes file reading, metadata interpretation, preprocessing, feature construction, model training, prediction formatting, and runtime management. One small error can poison the entire result.

DARE-bench is built around a simple but powerful mechanism: if the environment is controlled tightly enough, data-science tasks can be evaluated automatically.

The paper constructs 6,300 Kaggle-derived tasks. These are not only leaderboard-style Kaggle competitions. The authors use the broader Kaggle dataset pool, then apply automated filtering, LLM-assisted task design, post-processing, and sandbox validation to produce standardized tasks across classification, regression, and time-series forecasting. The benchmark includes both training and test tasks, with roughly a 95/5 split, and it uses the most recently updated tasks as the test set.

The important design choice is that DARE-bench does not rely on a human judge saying, “Yes, this looks reasonable.” Data science already gives us harder evidence. Either the output matches the reference workflow, or it does not. Either the prediction file scores well against masked ground truth, or it does not. The agent can write a moving paragraph about its “robust preprocessing strategy” afterward if it likes. The metric is not impressed.

DARE-bench has two major task logics:

That distinction is the paper’s spine. The authors are not treating “good data science” as one monolithic capability. They split it into two skills that businesses often confuse: following the process and optimizing the outcome.

A junior analyst can follow a senior analyst’s modeling protocol but not improve the model. A clever modeler can find a strong predictor while violating the required procedure. In a regulated analytics environment, the second failure is not a success. In a competition setting, maybe it is. Context matters. DARE-bench makes that context explicit.

Obedience and competence are different skills

The most common misconception around AI data agents is that better final accuracy will solve most problems. It will not.

A financial forecasting agent can produce a decent curve while mishandling the time index. A customer-churn model can score well while leaking target information. A marketing analytics pipeline can generate a polished report after silently dropping rows that were supposed to be imputed. The final answer may look acceptable because the final answer is often a very poor witness.

DARE-bench addresses this by separating instruction-following tasks from open-ended modeling tasks.

For classification and regression, the IF variants require agents to reproduce reference workflows. The reference solution generates deterministic predictions under controlled settings, and the agent receives a binary score: match or fail. There is no “mostly followed the instructions” consolation prize. This is harsh, but so is production.

The MM variants, by contrast, care about predictive performance. Classification-MM uses macro-F1, which matters when classes are imbalanced. Regression-MM uses clipped $R^2$, where negative $R^2$ values are clipped to zero so that weak models do not distort comparisons:

Time-series tasks add another useful distinction. In the XF variant, the agent has timestamp, entity identifiers, and exogenous features in the test set. In the CF variant, the test set is constrained to timestamp and entity columns, closer to classical forecasting. This is not just a dataset-format detail. It tests whether the agent can reason temporally rather than simply exploit conveniently available covariates.

The paper’s framing is therefore more operational than theatrical. It does not ask whether a model can “reason like a data scientist.” It asks whether it can survive a controlled data-science assignment with files, metadata, tools, constraints, and output requirements. Less romantic. More useful.

DARE-bench is a pipeline, not just a pile of tasks

The benchmark is also interesting because of how it is built.

The authors begin with Kaggle datasets and filter them by practical criteria such as format, licensing, size, and metadata. They then augment dataset metadata by crawling descriptions and using LLMs to infer task suitability, identify targets, classify column roles, and detect time-series structure. After that, the pipeline creates train/test splits, injects controlled noise for instruction-following tasks, handles chronological splits for time series, and validates instruction-following tasks in a sandbox.

This gives DARE-bench a useful middle position. It is more realistic than toy code-generation datasets, because it inherits messy properties from real user-contributed datasets: missing values, noisy columns, imbalanced labels, multilingual metadata, irregular timestamps, and domain variety. Yet it is more controlled than a random pile of real-world analytics tasks, because each task is packaged with standardized files, instructions, and automated evaluation.

That matters for training. A benchmark that only evaluates is a scoreboard. A benchmark that also produces verified trajectories becomes training infrastructure.

The paper leans heavily into this second role. DARE-bench is not merely a way to embarrass models in public, although it does a respectable job of that. It is designed to create data that can improve models through supervised fine-tuning and reinforcement learning with verifiable rewards.

The mechanism looks like this:

Raw Kaggle datasets
        ↓
Automated task formulation and metadata processing
        ↓
Sandbox validation and standardized packaging
        ↓
Executable evaluation against reference outputs or ground truth
        ↓
Training traces for SFT or rewards for RL

This is the part businesses should pay attention to. The strategic asset is not the benchmark table. The strategic asset is the loop.

The main results are not a leaderboard victory lap

The paper evaluates several proprietary and open-source models under a standardized setting: 5 interaction turns and a 200-second sandbox execution limit per turn. The authors also test other turn/time configurations and find that performance improves meaningfully when moving from 3 to 5 turns, while further turns can help but create a less practical latency/cost tradeoff. That sensitivity test is not the paper’s main thesis. It is a practical calibration step: how much room should an agent receive before evaluation becomes unrealistic?

Under the selected configuration, the results are not flattering.

A few points deserve interpretation.

First, the strongest models are uneven rather than universally dominant. GPT-5 performs strongly on instruction-following columns. Claude-Sonnet-3.7 leads in several modeling and time-series columns. GPT-o4-mini is competitive across many categories. This is already a warning against one-model procurement logic. “Best model” depends on whether the task rewards procedural obedience, predictive modeling, forecasting, or efficient tool use.

Second, canonical time-series forecasting is brutal. Many models are near zero on Time-CF. Even the stronger models remain weak. This matters because forecasting is where business users most eagerly hallucinate value: demand planning, inventory, energy load, cash flow, churn, revenue, risk. Apparently, adding “agentic” to a model does not automatically teach it calendar discipline. Shocking, I know.

Third, Claude-Sonnet-4 underperforms sharply in this setup. The paper attributes this partly to excessive decomposition: the model tends to break tasks into many small operations and hits step limits. This is a useful reminder that more careful-looking behavior is not always better agent behavior. A model can be “thorough” in the same way a committee can be thorough: by ensuring nothing finishes.

Fourth, open-source baselines struggle badly under these constraints. Qwen3-32B and Qwen3-4B perform far below the stronger proprietary models, especially in instruction following and canonical forecasting. This does not mean open-source models are useless. It means that, without task-aligned training, complex data-science workflows remain a harsh environment.

The leaderboard, then, is not the interesting part. The interesting part is diagnosis.

The failure modes show why final-answer accuracy is too narrow

The paper’s qualitative failure analysis identifies several bottlenecks: incorrect tool argument passing, instruction-following failures, brittle reasoning in open-ended tasks, and time-series failures.

These are not all the same kind of error.

This table is where the paper becomes practically useful. Many companies currently evaluate AI analytics agents by asking whether the final output “looks right.” That is a weak test. The output can look right while the process is wrong. Conversely, a technically valid process can fail because the model botched a tool call.

In a production environment, these categories imply different remedies. Tool argument errors suggest interface training, schema enforcement, or stricter function-calling wrappers. Instruction deviations suggest workflow verification and trace checking. Brittle preprocessing suggests better metadata grounding and test cases. Time-series failures suggest domain-specific training and specialized forecasting modules. Excessive decomposition suggests agent budget management, not more inspirational slogans about autonomy.

A single accuracy number hides all of this. DARE-bench tries not to.

The training results are the paper’s strategic contribution

The most important part of DARE-bench is not that it reveals failure. Plenty of benchmarks reveal failure. Some seem to exist primarily for that purpose, like academic blood sport with LaTeX.

The stronger claim is that some failures are trainable.

The authors use DARE-bench training tasks in two ways. For Qwen3-32B, they run supervised fine-tuning using rejection-sampled traces. For Qwen3-4B, they run reinforcement learning using verifiable rewards.

The results are large:

A detail here is worth slowing down for. The paper reports not only overall task scores but also “Model-Perf,” which focuses on prediction quality among successful attempts for modeling tasks. That helps separate two questions:

1. Did the agent produce a valid output at all?
2. When it produced a valid output, was the model any good?

This matters because a training method could, in theory, merely make agents better at formatting outputs without improving actual modeling ability. The reported Model-Perf gains suggest the improvement is not only syntactic discipline. The agents become better at the work after they stop stepping on the rake.

The rejection-sampling strategies also reveal useful tradeoffs:

This is not just methodological decoration. It hints at how enterprise teams might build internal training data. You may want the fastest valid procedure for customer-facing latency. You may want diverse valid traces when workflows vary across departments. You may want best-valid traces when predictive performance matters more than speed. The right supervision depends on the operating model.

The ablation study sharpens the point. Training only on instruction-following data improves IF scores but hurts modeling scores. Training only on modeling data improves MM scores but weakens instruction following. Combined training gives the better balance.

That is the paper’s most useful lesson in miniature: procedure and performance must both be trained. One does not magically emerge from the other.

The appendix tests are not side quests

The paper includes several additional analyses that should be read correctly. Not every table is a second thesis. Some are calibration checks, some are robustness tests, some are implementation details, and some are external validation.

This distinction matters because business readers often misuse benchmark papers in two opposite ways. One group overgeneralizes every number into procurement advice. Another dismisses every number because “our data is different.” Both reactions are lazy, but at least they are symmetrical.

The better reading is more specific. DARE-bench does not prove that a fine-tuned Qwen model is now ready to run your credit-risk modeling pipeline. It does show that verifiable data-science tasks can expose distinct failure modes and that task-aligned training can reduce some of them.

That is enough to change how a serious team should evaluate AI analytics systems.

The business lesson is cheaper diagnosis, not just better agents

For firms building or buying AI data analysts, the obvious takeaway is “benchmark before deployment.” True, but incomplete. The deeper takeaway is to build a verification loop around recurring analytical work.

A useful enterprise version of DARE-bench would not begin with a grand ambition to benchmark all data science. It would begin with a narrow library of high-value workflows:

• monthly sales forecasting;
• churn-risk scoring;
• lead-quality modeling;
• inventory demand planning;
• financial variance analysis;
• customer segmentation;
• campaign performance attribution.

For each workflow, the company would define files, metadata, constraints, allowed tools, expected outputs, and evaluation rules. Some tasks would measure instruction fidelity: did the agent follow the approved process? Others would measure modeling quality: did it improve predictive performance without violating constraints? Failed runs would be classified by failure mode, not thrown into a vague bucket labeled “AI not good enough.”

The operational loop looks like this:

Recurring business workflow
        ↓
Standardized task package
        ↓
Sandboxed agent execution
        ↓
Programmatic scoring
        ↓
Failure classification
        ↓
Targeted prompt, tool, wrapper, or fine-tuning update
        ↓
Regression test before redeployment

This is not glamorous. Good infrastructure rarely is. It is mostly a disciplined way to prevent a chatbot from quietly becoming the weakest link in the analytics stack.

The ROI logic is also more sober than “AI replaces analysts.” DARE-bench points toward three more defensible sources of value:

The phrase “define what they actually value” is doing more work than it appears. Many business AI projects fail evaluation because nobody decides whether the agent is supposed to be obedient, creative, fast, accurate, auditable, cheap, or all of the above. “All of the above” is not a specification. It is a wish list wearing a blazer.

DARE-bench’s contribution is that it forces tradeoffs into the open.

Forecasting remains the rude part of the story

The time-series results deserve separate attention because many business applications are forecasting-heavy. DARE-bench’s Time-CF scores are weak across models, with several baselines at or near zero. The paper attributes failures to invalid output formats, naive heuristics such as last-value or mean forecasts, and limited exposure to complex time-series reasoning.

This is a useful correction to current agent enthusiasm. A model that can write Pandas code and explain ARIMA does not necessarily understand forecasting as an operational discipline. Time-series work requires respecting chronology, avoiding leakage, handling multiple entities, selecting horizons, managing missing intervals, and formatting predictions correctly. It is extremely easy to look competent while being wrong.

The enterprise implication is direct: if your use case depends on forecasting, do not assume a general LLM agent is production-ready because it performed well on ordinary tabular tasks. Test forecasting separately. Include canonical setups where test-time exogenous features are unavailable. Include messy entity identifiers. Include irregular timestamps. Include output-format checks. In other words, evaluate the workflow you actually need, not the workflow that makes the demo behave.

This does not mean LLM agents have no role in forecasting. They may still help with metadata interpretation, feature generation, scenario explanation, anomaly triage, and pipeline orchestration. But the model should be forced to prove forecasting competence under controlled conditions before being trusted with inventory, cash flow, or capacity planning.

A forecast that cannot survive a benchmark is not a forecast. It is a spreadsheet with stage lighting.

What the paper directly shows, and what Cognaptus infers

To avoid turning the paper into a prophecy, it helps to separate evidence from interpretation.

That last row is important. DARE-bench is realistic relative to many benchmarks, but it is not the entire enterprise world in miniature. It is primarily tabular and Kaggle-derived. It does not fully cover multimodal analytics, messy document-image-code workflows, live BI systems, internal access permissions, human approval chains, or long-running organizational feedback loops.

The paper’s own limitations are also practical: generating many executable traces can be expensive, and rejection sampling may bias supervision toward shorter trajectories. That does not invalidate the result. It simply means the benchmark should be read as a strong mechanism demonstration, not as a universal substitute for internal evaluation.

The boundary: DARE-bench is necessary discipline, not complete governance

A company could copy the DARE-bench pattern and still deploy unsafe agents if it tests the wrong tasks. Verification is only as good as the task design. If the benchmark omits privacy constraints, the agent may learn to be accurate and invasive. If it omits cost constraints, the agent may learn to be correct and economically absurd. If it omits approval steps, the agent may learn to automate decisions that should remain supervised.

So the business lesson is not simply “make benchmark, become safe.” The lesson is more demanding:

1. Identify recurring workflows.
2. Define what “correct” means for each workflow.
3. Separate procedural fidelity from outcome quality.
4. Run agents in controlled environments.
5. Score outputs programmatically where possible.
6. Classify failures by mechanism.
7. Use successful and failed traces to improve the system.
8. Re-test after every model, prompt, tool, or data change.

That is not a blog-friendly miracle cure. It is an operating discipline. Unfortunately, operating disciplines are usually what separates an AI pilot from an AI liability.

Conclusion: the future AI data scientist needs a test harness

DARE-bench is valuable because it shifts the conversation from model personality to workflow evidence.

The industry does not need more confident agents that can describe a good data-science process. It needs agents that can execute the process, respect constraints, recover from messy metadata, produce correctly formatted outputs, and improve under verifiable feedback.

The paper’s most important message is not that today’s agents fail. We already knew that, although the tables make the failure taste a little less abstract. The more important message is that failure can be decomposed. Some failures are about tool interfaces. Some are about instruction fidelity. Some are about modeling. Some are about forecasting. Some are about efficiency. Once decomposed, they can be measured. Once measured, some can be trained.

That is where DARE-bench becomes more than a benchmark. It becomes a prototype for agentic data-science infrastructure.

For businesses, the conclusion is simple enough to fit on a dashboard: do not buy an AI data scientist until you know how you will test one. Final accuracy is not enough. A polished notebook is not enough. A confident explanation is definitely not enough; we have had enough confident explanations, thank you.

The next generation of useful AI data agents will not be built by vibes. It will be built by verifiable tasks, executable traces, failure taxonomies, and disciplined feedback loops.

Not glamorous. Very necessary.

Cognaptus: Automate the Present, Incubate the Future.