Agent Harnesses

TL;DR for operators If an AI coding assistant can install software, do not assume it will inspect the evidence already available about where that software came from. In Pengyin Shan’s pre-registered demand-side audit,1 only 9 of 1,920 registered trials retrieved a relevant trust signal before the install-or-decline decision. Across 2,114 completed registered and supplementary trials, not one assistant executed a signature or attestation verification command. ...

TL;DR for operators A data agent can find the right evidence, perform much of the required analysis, and still fail the task by returning the wrong table. In an audit of 136 failed runs from the strongest tested backbone, 71 failures—52.2%—were attributed primarily to turning the agent’s internal result into the requested output. Sixty of those involved submitting extra or missing columns. Only three failures were attributed to selecting the wrong evidence source. ...

TL;DR for operators Most agent failures are blamed on the model because blaming “the model” is emotionally convenient and operationally vague. HarnessX makes a more useful claim: the runtime harness around the model — prompts, tools, memory, control flow, tracing, evaluators, safety checks, and training interfaces — is not scaffolding in the disposable sense. It is part of the system’s intelligence surface.1 ...

Cost is usually where beautiful agent demos go to become spreadsheets. A prototype calls an LLM at every step. It reasons, reflects, revises, asks itself whether it should revise the revision, and then, very responsibly, consumes another few thousand tokens to explain why this was necessary. The demo looks intelligent. The invoice looks even more intelligent. ...

Glue is not glamorous. In most AI product discussions, the model gets the spotlight. The harness—the scripts, prompts, validators, retry rules, state files, tool adapters, and stopping criteria around the model—gets treated as plumbing. Necessary, slightly annoying, and best ignored until it leaks. That habit is becoming expensive. The paper Natural-Language Agent Harnesses argues that the surrounding execution system is no longer a secondary implementation detail. It is often the actual unit of agent performance, reliability, and portability.1 The paper’s useful claim is not that “natural language replaces code.” That would be a lovely fantasy for people who have not debugged parsers, sandboxes, or file permissions lately. The sharper claim is that part of the harness can become an editable natural-language policy object, while exact execution remains in code. ...