Passing Tests Is Not an Audit Trail

TL;DR for operators

When a coding agent changes a production-bound function after a failed test, a reviewer should be able to determine which failure caused each changed region to exist. Final code, passing tests, logs, and commits may show that the program works now, but they can lose the construction history needed to explain individual repairs.

TraceCoder¹ preserves that missing record. It links each repaired code region to the benchmark failure, repair round, raw failure output, and model-supplied explanation associated with the change.

This history could help organizations:

• approve generated changes with evidence tied to each repair;
• investigate regressions and previously handled edge cases;
• retain review records for governance and compliance;
• evaluate coding agents on traceability as well as final pass rates.

The prototype shows that this form of repair history is technically feasible for moderate single-file Python tasks. It does not establish production readiness, validate generated benchmarks, or prove that frequently repaired code is more likely to contain defects.

A passing function can still be unexplained

Consider a coding agent that generates a function, runs tests, repairs the implementation several times, and eventually produces a passing result. A reviewer can inspect the final code and the final test report. Those artifacts establish what exists now and whether the available tests accept it.

They may not establish why a particular guard clause, parser branch, or boundary check was introduced.

Round-level logs help only partially. They can show that a repair occurred after a failure, but subsequent edits may move or replace the affected lines. Git commits can preserve chronological changes, yet the relationship between a specific snippet and the exact failing benchmark may remain implicit. Once several repair rounds accumulate, the reviewer is left reconstructing intent from diffs, prompts, and execution output.

TraceCoder treats that missing relationship as part of the generated artifact. Each code change becomes a persistent event connected to the test that triggered it, the round in which it occurred, the failure output, and the explanation returned during repair. This is observable repair provenance, not access to a model’s hidden reasoning. The explanation is itself a model-produced record and should be judged against the associated failure and code change.

Stable snippet identity comes before better logging

A persistent history requires more than storing additional messages. It requires a way to refer to the same code region as the program changes.

Line numbers are unsuitable for that purpose. Inserting ten lines near the top of a file shifts every later reference. Moving a function changes its location even when much of its content remains intact. Replacing or deleting a block removes it from the current program although it may remain important to an investigation.

TraceCoder assigns snippets lexicographically ordered fractional position keys. Its key-generation algorithm can create a new key between two existing keys while leaving unchanged keys intact. The paper states the ordering guarantee as:

This mechanism allows snippets to be inserted without globally renumbering the surrounding code. Diffing identifies which regions were retained, inserted, replaced, moved, or removed. Replaced and deleted snippets are preserved as inactive records rather than erased, while lineage pointers connect later versions to earlier ones.

The consequence is structural rather than cosmetic. A repair history can remain attached to a code region through subsequent rounds instead of becoming a collection of timestamps referring to unstable line positions.

The database makes repair history reconstructable

TraceCoder stores the resulting history in normalized SQLite tables covering code snippets, benchmarks, benchmark executions, snippet-to-failure links, and repair explanations. Records include creation and deactivation rounds, failure output, and lineage relationships.

This design supports two different views of the same run.

The current-program view reconstructs only active snippets in position-key order. The historical view retains earlier versions and associates each changed region with the event that produced it. A browser interface exposes these records without requiring reviewers to reconstruct them manually through SQL.

The paper’s three case studies serve as illustrative mechanism evidence rather than controlled performance tests. They show histories associated with negative-input handling in a Fibonacci task, parsing and tokenization behavior in an expression evaluator, and boundary logic in Roman-numeral conversion. Their value lies in demonstrating what a reviewer could inspect: the failed edge case, the relevant output, and the code region altered in response.

They do not establish that the explanation is semantically complete, that the repair was necessary, or that the resulting code is correct beyond the available benchmarks.

The prototype records useful history at modest absolute scale

The evaluation covers 30 algorithmic programming specifications under two configurations. The reported aggregates are descriptive because the configurations also differed in model roles, benchmark batching, and implementation revision.

In the second configuration, 18 tasks passed the initial benchmark batch, two required one repair iteration, and ten exhausted the six-round budget. One database reached 5,672 KB, while the remaining databases were much smaller.

These results support prototype feasibility: detailed histories did not normally require large absolute storage. They do not show that the second provider arrangement is superior. Its richer history may reflect independent benchmark generation, a larger initial batch, provider capabilities, task interactions, or code revisions. The study does not isolate those causes.

The changed-snippet percentage also requires careful interpretation. A high value may indicate broader testing and richer recorded repair activity. It may instead indicate weaker initial code, aggressive rewriting, or formatting changes. Conversely, a critical line generated correctly in the first attempt receives no repair tag. The metric measures recorded change coverage, not explanation quality or defect probability.

Where businesses could use the history

For an engineering reviewer, the relevant decision is whether to approve agent-generated code for integration. When a change affects a sensitive function, the reviewer could inspect which benchmark caused the change and whether the resulting edit plausibly addresses that failure. This use depends on trustworthy tests and remains unvalidated for large repositories.

For a maintenance team, the decision is where to begin regression triage. When a later incident resembles an earlier edge case, snippet history could identify regions previously repaired for related failures. The value is faster investigation, not proof that the historical repair caused the new problem.

For compliance or model-governance teams, the decision is whether the organization has retained sufficient evidence about automated changes. A queryable link among tests, failures, edits, and approvals could support review records. Higher-risk environments would still require human-authored or locked specifications, independent validation of generated benchmarks, and explicit approval controls.

For a coding-agent platform team, the decision is which provider configuration to deploy. TraceCoder suggests expanding evaluation beyond pass rate and latency to include benchmark correctness, provenance completeness, parser failures, storage behavior, and recovery from malformed outputs. The paper demonstrates that these dimensions can be recorded; it does not provide a validated provider scorecard.

Generated tests remain part of the risk surface

The most consequential implementation problem reported in the study was not provenance storage. It was benchmark correctness.

The benchmark generator sometimes produced incorrect expected outputs. A repair agent following a faulty test can modify valid behavior until the program satisfies an invalid specification. Traceability would make that sequence visible, but visibility does not prevent the error.

For autonomous repair in production-bound systems, generated tests should therefore be treated as proposed specifications rather than unquestioned ground truth. Depending on risk, organizations may need independent test generation, deterministic checks, human review, benchmark locking, or separation between the system proposing a test and the system authorizing a code change.

TraceCoder can preserve evidence that a change followed a benchmark. It cannot establish that the benchmark deserved authority.

A feasible audit architecture, not a production control

The paper demonstrates an implemented method for retaining fine-grained repair history across iterative code edits. Stable position keys address the identity problem; tombstones and lineage preserve superseded code; relational records make failures and explanations queryable; the viewer makes the history inspectable.

The current evidence does not cover multi-file repositories, compiled languages, complex build systems, production incident rates, or reviewer performance. No developer, auditor, or compliance user study tests whether the interface improves decisions. Provider API failures, malformed JSON, syntax errors introduced during repair, and incorrect generated benchmarks also affected the experiments.

The appropriate next use is controlled evaluation. Teams considering coding agents can instrument a limited workflow, preserve locked reference tests, compare provenance records with ordinary logs and commits, and measure whether reviewers identify risky or unjustified changes more accurately. Until that evidence exists, TraceCoder should be treated as a credible architectural proposal for accountable code generation—not as proof that agent-generated code has become auditable merely because every repair has a record.

Cognaptus: Automate the Present, Incubate the Future.