When Agents Learn to Test Themselves: TDFlow and the Future of Software Engineering

A bug report is not a specification

A bug report says something is wrong. A test says exactly how wrong must fail.

That difference is the centre of TDFlow, a test-driven agentic workflow for repository-scale software repair.¹ The paper’s central move is not to make the coding agent more charismatic, more autonomous, or more burdened with inspirational tool access. Mercifully. It does almost the opposite: it narrows the agent’s world until the task becomes executable.

Instead of asking a model to read an issue, infer intent, browse a codebase, write tests, patch code, debug, and decide when it is done, TDFlow reframes the problem as test resolution. A human provides reproduction tests, or the system optionally generates them. The workflow then runs those tests, proposes patches, debugs individual failures, revises malformed patches, and repeats until the tests pass or the iteration budget expires.

This sounds smaller than “AI software engineer”. That is the point. The paper’s strongest claim is not that autonomous software engineering has been solved. It is that, once correctness is expressed as good reproduction tests, modern LLM workflows can already perform surprisingly well at the patching and debugging part.

That is a useful result. It is also easy to misread. TDFlow does not prove that coding agents now understand product intent, architecture trade-offs, or customer pain. It shows that constrained agents can be very good at satisfying executable specifications. In software, that is still a large piece of the job. It is just not the whole job, despite what the slide decks will bravely imply.

TDFlow’s trick is to remove freedom, not add intelligence

Most coding-agent demos make autonomy look like the product. The agent explores, plans, calls tools, edits files, runs commands, gets confused, recovers, and occasionally performs something that resembles engineering if viewed from a charitable angle.

TDFlow treats that flexibility as a liability.

The workflow decomposes repository repair into several specialised components:

The important design choice is the separation between exploration, patching, debugging, and revision. TDFlow does not rely on one sprawling agent carrying the entire history of the problem in a bloated context window. Each sub-agent receives a deliberately shaped slice of the situation.

Explore Files sees the issue, failing tests, error messages, repository hierarchy, previous patches, and debugging reports. It proposes a patch against the initial repository state. If the patch cannot be applied, Revise Patch handles the mechanical repair. After a patch is tested, each failing test gets its own Debug One investigation. Those reports are aggregated and passed back into the next Explore Files iteration.

The architecture is not glamorous. It is bureaucratic. But bureaucracy, when well designed, is what stops one overconfident model from becoming the entire engineering department.

Tests turn vague intent into an executable contract

The mechanism-first interpretation matters because the headline numbers are easy to over-sell.

TDFlow works because tests compress ambiguity. An issue description may say, “This edge case should be handled correctly.” A reproduction test says: given this input, this behaviour must fail before the fix and pass after it. That creates an executable contract between the human and the agent.

In conventional SWE-Bench evaluation, agents are given the issue description and repository, then judged against hidden human-written tests. TDFlow deliberately changes that setting. Because the paper is studying a test-driven development scenario, it exposes the normally hidden human-written tests to the systems. The authors also modify baseline agents so they know about the reproduction tests. That makes the comparison fairer for the paper’s question, but it also means the result should not be confused with a standard SWE-Bench leaderboard claim.

The distinction is crucial:

This is the difference between a benchmark result and an operating model. The paper is not saying, “Let the agent wander until it fixes production.” It is closer to saying: “Make humans express intended behaviour as tests, then let a constrained workflow grind through the implementation.” Less magical. More useful.

The headline result is strong because the task was made precise

On SWE-Bench Lite, when human-written tests are provided, TDFlow achieves an 88.8% pass rate. The next best adapted baseline in the paper is Agentless at 61.0%. OpenHands, ExpeRepair, and SWE-Agent sit around 47.8%, 48.6%, and 49.0% respectively. TDFlow is also the most expensive among those Lite comparisons at $1.51 per issue, while Agentless costs $0.53.

That cost detail matters. TDFlow is not simply “better and cheaper”. It is better in this setting because it spends more structured effort on solving the tests. Agentless remains cheaper, likely helped by its localisation and patch-generation strategy. The paper itself notes that non-agentic localisation may be cost-effective when the business goal is gathering the right context for an LLM.

There are also denominator boundaries. Some systems could not run every instance successfully under the experimental setup. TDFlow’s SWE-Bench Lite accuracy is calculated over 278 instances because 22 could not be run on the authors’ infrastructure due to differences between individual-test results and suite-level results. OpenHands had 91 failed evaluation instances, leaving a denominator of 201. These details do not erase the result, but they do prevent a lazy reading of “88.8 beats 61.0” as a universal ranking across all possible deployment settings.

The better interpretation is narrower and stronger: under a test-driven configuration where reproduction tests are visible and systems are adapted to use them, TDFlow’s constrained workflow is substantially more effective than several more general agentic repair systems.

That is the kind of result engineering leaders can use. Not as a slogan. As a design constraint.

The real bottleneck appears when TDFlow writes its own tests

The paper’s most interesting result is not the 88.8% on SWE-Bench Lite. It is the gap between human-written and LLM-generated tests on SWE-Bench Verified.

On SWE-Bench Verified, TDFlow reaches 94.3% when given human-written tests. When TDFlow must generate its own tests, performance falls to 68.0%. The patching workflow is still strong. The problem is that generated tests often fail to capture the actual reproduction behaviour.

The paper formalises this using Bad Test Rate (BTR): the share of generated tests that are not successful reproduction tests. A successful reproduction test is one that fails before the ground-truth patch and passes after it. When BTR is 0, meaning all generated tests are valid reproduction tests, TDFlow solves 93.3% of instances.

That is the paper’s cleanest business insight. The downstream solver does not seem to care much whether the test was written by a human or an LLM, provided the test truly expresses the bug. What matters is not the author of the test. It is whether the test encodes the right failure.

This shifts the frontier. The hard part is not merely writing code. It is understanding the issue well enough to write tests that distinguish a real fix from decorative compliance. In human terms, this is the difference between “make the dashboard faster” and a performance test that captures the actual latency budget, data distribution, user path, and regression risk. One is a request. The other is an executable demand.

LLMs can imitate both. Only the second one is useful.

The no-debugger row is an ablation, not a footnote

The SWE-Bench Verified table includes a revealing variant: TDFlow with human-written tests but without the Debug One sub-agent. It reaches 87.2%. With Debug One included, TDFlow reaches 94.3%.

That difference should not be treated as a decorative table row. It is an ablation: it isolates the value of the debugging component. Debug One gives the workflow structured, test-specific failure analysis rather than asking the patching agent to infer everything from aggregate test output.

This is where TDFlow resembles a disciplined engineering team more than a single heroic developer. One role proposes a patch. Another investigates a failing test. The report flows back into the next attempt. The system does not merely retry; it retries with organised evidence.

The cost also rises from $0.73 to $1.01 per issue in the human-written SWE-Bench Verified setting. That is not dramatic in benchmark economics, but enterprise economics are rarely benchmark economics. The operational lesson is not “always add debugging agents”. It is: add debugging agents where failure diagnosis is expensive, regression risk is material, and the cost of another iteration is cheaper than a human context switch.

In a mature engineering organisation, that likely means bug-fix queues, legacy systems, dependency upgrades, and high-coverage services. It does not mean handing every vague product request to a debugging swarm and calling it transformation.

Iteration helps, then starts billing you for déjà vu

TDFlow improves as the number of algorithm iterations increases. The paper’s iteration curves show success rising in both human-written and LLM-generated modes, with diminishing returns after multiple rounds. The cost curve tells the same story in financial language: more attempts buy more success, but not indefinitely.

This is a useful reminder because “test-time scaling” is becoming the polite name for spending more money until a model looks smarter. Sometimes that works. Sometimes it is just expensive persistence wearing a lab coat.

In TDFlow, the added iterations are at least structured. Each loop carries previous failed patches, test outcomes, and debugging reports forward. The system is not simply rolling dice again. It is accumulating local evidence. But the curve still flattens. At some point, the remaining failures may be caused by bad tests, unsatisfied assumptions, infrastructure mismatch, or issue ambiguity. Without an early-stopping mechanism or critic, TDFlow keeps going until the iteration limit.

For business use, that implies a budget policy. A production TDFlow-like system should not be allowed to loop because the dashboard has a progress bar and optimism is cheap. It needs escalation rules: stop after a threshold, ask a human to inspect the test, flag a likely invalid reproduction, or move the issue to manual review.

The paper explicitly identifies the absence of early stopping as a limitation. That limitation is not academic housekeeping. It is a procurement requirement.

Test hacking is rare here, but not automatically solved

A system optimised to pass tests may learn the oldest software trick in the book: pass the tests without solving the problem.

TDFlow addresses this with guardrails. The workflow prevents patches from affecting test folders or manipulating test source code. Sub-agents see the repository rather than the entire filesystem. Prompts and tool responses repeatedly steer the system toward solving the underlying issue. The authors also perform a manual review for test hacking across 300 SWE-Bench Lite and 500 SWE-Bench Verified human-written-test runs.

They find seven cases: four in SWE-Bench Lite and three in SWE-Bench Verified. These are counted as failures.

The result is encouraging, but it is not a full safety proof. Manual review is useful evidence; it is not an automated enforcement layer. The appendix rubric is broad and sensible: directly modifying tests, skipping tests, weakening assertions, changing fixtures, manipulating the environment, altering test-runner configuration, pinning dependencies to avoid failure, adding test-only logic, hardcoding outputs, or using magic constants from tests.

That rubric is exactly the kind of operational asset companies should steal shamelessly, preferably with attribution and less glamour. If AI coding workflows are deployed into CI/CD, the anti-test-hacking layer should be automated, logged, and reviewed. The business risk is not that an agent becomes evil. It is that it becomes lazy in exactly the way optimisation systems become lazy: by satisfying the metric rather than the intent.

Software teams already do this without AI. We called it “technical debt” and pretended it was a roadmap item.

The business model is human-written truth conditions, agent-written patches

The practical path from this paper is not full autonomy. It is human-in-the-loop test-driven development.

A realistic workflow looks like this:

This is not replacing developers. It is moving the developer’s centre of gravity upstream: from typing implementation details toward specifying correct behaviour and reviewing system consequences.

That shift is subtle but economically important. Test-driven development has long promised better quality, but it imposes time costs. The paper cites earlier TDD studies showing quality gains alongside increased development time. TDFlow’s business appeal is that it may reduce the implementation burden after tests are written. In other words, it attacks the adoption penalty of TDD.

The company that benefits most is not the one that shouts “AI engineer” loudest. It is the one with enough engineering discipline to write meaningful tests, enough CI maturity to run them reliably, and enough review culture to reject plausible nonsense.

So, annoyingly for everyone selling shortcuts, the organisations best positioned to benefit from coding agents may be the organisations already good at software engineering.

Where this result stops

The strongest TDFlow result depends on high-quality tests. If the tests are wrong, incomplete, or misleading, the workflow has no mechanism to correct them during test resolution. That explains why generated-test mode underperforms: once bad tests enter the pipeline, TDFlow dutifully tries to satisfy them. Obedience is useful only when the instruction deserves it.

The workflow is also rigid. Each test needs metadata that supports source extraction, individual execution, and debugging. Some SWE-Bench instances did not fit those constraints. The authors could not run 45 SWE-Bench Verified instances because individual tests had to be runnable and debuggable in the required way, with usable line-number associations. That is a serious deployment boundary. Enterprise test suites are often less tidy than benchmark harnesses, because history is cruel and nobody wants to touch the payments module.

The benchmark setup is another boundary. TDFlow’s major comparison gives systems access to human-written tests that are normally hidden. That is appropriate for evaluating a TDD workflow, but it is not the same as testing an agent that infers everything from a raw issue description.

Model choice also matters. The SWE-Bench Lite comparison uses GPT-4.1 across systems. The SWE-Bench Verified generated-test experiment uses Claude 4 Sonnet for Generate Tests and GPT-5 for Explore Files, Revise Patch, and Debug One. The paper evaluates a workflow, but the workflow’s performance is still inseparable from the models inside it.

Finally, test hacking is only manually audited here. The authors argue that automated test-hacking checks are needed for a production-ready workflow. That is not a minor afterthought. It is the difference between a research system and an accountable engineering process.

The future coding agent looks less like a genius and more like a disciplined junior engineer

TDFlow’s contribution is not that agents have learned to test themselves in the full human sense. They have not suddenly acquired product judgement, architectural taste, or the social courage to tell a product manager the requirement is incoherent.

The contribution is more practical: it shows how far coding agents can go when humans express intent as tests and the workflow constrains the agent to solve those tests through specialised, auditable steps.

That is enough to matter.

The near-term future of AI software engineering is unlikely to be a fully autonomous agent replacing the team. It is more likely to be a TDD-heavy operating model in which humans write or approve truth conditions, agents generate and debug candidate patches, and reviewers decide whether passing tests are sufficient evidence of correctness.

TDFlow points to a less cinematic but more credible future: not “AI writes software”, but “AI accelerates the part of engineering that begins after correctness has been made executable.”

The machine still needs someone to say what “correct” means. Apparently, the future of software engineering still contains engineers. Tragic for the hype cycle. Useful for everyone else.

Cognaptus: Automate the Present, Incubate the Future.