The Table Is the Task: What DataSpace Reveals About Data-Agent Reliability

TL;DR for operators

A data agent can find the right evidence, perform much of the required analysis, and still fail the task by returning the wrong table. In an audit of 136 failed runs from the strongest tested backbone, 71 failures—52.2%—were attributed primarily to turning the agent’s internal result into the requested output. Sixty of those involved submitting extra or missing columns. Only three failures were attributed to selecting the wrong evidence source.

That changes where deployment teams should look for reliability gains. DataSpace shows that the surrounding agent harness can also move completion accuracy substantially even when the underlying model is fixed: with MiMo-V2.5 held constant, accuracy ranged from 30.98% to 46.34% across five harnesses. For enterprise analytics, model selection, orchestration design, and output-contract validation should be evaluated as separate engineering decisions rather than collapsed into one headline model score.

The analysis can be right while the deliverable is wrong

Consider a familiar analytics workflow. An agent identifies the relevant files, extracts the needed records, performs the calculation, and produces a CSV. The numbers it needed may already exist somewhere in its trajectory. Yet the submitted table contains an extra column, omits a required field, changes numeric precision, or orders rows incorrectly.

For the consumer of that table, the distinction between “almost solved” and “solved” is not academic. A downstream process cannot safely ingest an output whose schema or contents violate the requested contract.

This is the reliability problem examined by DataSpace, a 410-task benchmark for data agents operating over heterogeneous workspaces.¹ Tasks draw on 7,439 artifacts totaling 15.01 GB across CSV, JSON, SQLite, Markdown, PDF, and video, with both Chinese-English cross-language tasks and single-language tasks.

The benchmark does not ask whether the agent can retrieve a relevant passage or generate a plausible analytical explanation. Every task requires the complete requested table.

That output requirement is what makes the failure audit particularly revealing.

Complete-task scoring exposes failures that partial progress can hide

DataSpace scores each task as a complete analytical result. Its evaluator allows harmless representational variation: predicted columns can be aligned one-to-one with reference columns, and values are normalized according to task-specific type and precision rules. Unordered results can be compared as multisets, while tasks with meaningful row ordering retain that requirement.

But semantic tolerance does not mean partial credit. Missing or extra rows and columns still fail. Invalid outputs, runtime failures, incomplete tables, and missing submissions also count as incorrect.

This distinction matters because an evaluation that rewards intermediate progress could make an agent look substantially more capable than the system experienced by its downstream user.

Under the fixed DataSpace-Agent harness, the strongest tested backbone, Grok 4.5, solved 272 of 410 tasks, or 66.34%. GPT-5.6 Sol reached 64.63%, followed by Kimi K3 at 53.41%. The benchmark is therefore far from saturated even for the strongest tested configurations.

The more diagnostic result comes from examining where unsuccessful trajectories diverged.

Among all 136 failed Grok 4.5 tasks, 71 were assigned to the answer-materialization stage. Sixty involved a correct internal result being submitted with extra or missing columns. Another nine altered type, precision, date precision, or value formatting during output.

By comparison, only three failures were attributed to choosing the wrong artifact or evidence source.

The audit does not show that evidence discovery is universally easy. It shows something narrower and more actionable: within these audited failures, evidence-source selection was not the dominant bottleneck.

A further 17 failures involved misunderstanding the target output, requested entities, or row granularity. Combined with the 60 faulty column-projection cases, these two subtypes account for 77 of 136 failures, or 56.6%.

The recurring problem is not merely obtaining information. It is maintaining an accurate representation of what must ultimately be delivered.

The harness can move accuracy by more than fifteen points

The paper also separates model capability from the software environment surrounding the model.

An agent harness is the control layer that determines how a model uses tools, manages context, iterates through actions, executes commands, and decides when and how to submit a result. It is easy to treat this layer as plumbing around the “real” intelligence. DataSpace’s controlled comparison makes that assumption difficult to maintain.

With MiMo-V2.5 held fixed across all 410 tasks, five harnesses produced markedly different completion rates:

The spread is 15.36 percentage points with the backbone unchanged.

This does not identify which individual harness mechanism causes the difference. Each system retains its own native control loop and context-management behavior. But it does establish the engineering relevance of the harness itself: a model benchmark cannot substitute for testing the model inside the execution framework that will actually run the workload.

The backbone comparison reinforces another systems-level point. GPT-5.6 Sol finished only 1.71 accuracy points behind Grok 4.5 while using 74.2% fewer tokens, 50.3% fewer tool actions, and 39.2% less wall-clock time per task. Its reported API cost per task, however, was higher: $0.200 versus $0.169 under provider pricing at evaluation time.

Accuracy, compute consumption, tool activity, latency, and monetary cost can therefore rank systems differently.

Output validation deserves its own reliability layer

The paper directly demonstrates the failure distribution. The following deployment implications are Cognaptus inferences from that evidence rather than interventions experimentally tested by the authors.

First, an analytics agent should face a pre-submission contract check. Before a table becomes consumable, the system can verify expected column count, names or semantic mappings, row cardinality where known, types, numeric and date precision, required ordering, and serialization integrity. The concentration of materialization failures makes these checks plausible candidates for high-leverage engineering work.

Second, teams should benchmark the production harness independently of the backbone. Replacing a model while leaving orchestration unexamined can miss a sizeable source of variance. Conversely, improving control logic may raise completion without changing model weights.

Third, regression testing should score complete outputs, not only intermediate evidence retrieval or reasoning traces. A pipeline that routinely reaches the correct intermediate dataframe but corrupts it during projection or submission is operationally unreliable even if its reasoning appears strong.

Finally, workflows that depend on relational integration deserve explicit stress tests. Across all six backbones, tasks requiring joins were associated with accuracy that was 9.7 to 19.8 percentage points lower than tasks without joins. Tasks requiring evidence from multiple modalities were also 1.8 to 14.0 points lower than single-modal tasks across the six backbones.

Those contrasts identify useful testing dimensions, not causal effects.

Where the evidence stops

DataSpace’s strongest claims concern performance under its own controlled benchmark protocol. The task-characteristic analyses are explicitly descriptive, so the join and multimodal gaps should not be interpreted as estimates of what adding a join or modality would causally do to an otherwise identical task.

Coverage also matters. The benchmark is constructed from BULL and EHRSQL, concentrating tasks in fund, stock, macroeconomic, and healthcare analytics. Cross-language variation covers Chinese and English rather than multilingual analytics generally. Several specialized data-agent systems could not be included because their interfaces, output contracts, modality support, or released implementations were incompatible with faithful evaluation.

The reported comparisons also represent particular model and harness versions evaluated in 2026. They should be treated as system snapshots, not durable rankings.

These boundaries limit how far the numerical scores travel. They do not remove the underlying deployment question that DataSpace makes measurable: can the system convert heterogeneous evidence into the exact analytical object the user requested?

For enterprise data agents, that question belongs at the end of every benchmark—and at the end of every production run.

Cognaptus: Automate the Present, Incubate the Future.