A Repair Log Is Not an Explanation

TL;DR for operators

When a coding agent changes repository code, the resulting patch and test outcome reveal what was changed and whether the repair succeeded under the evaluation procedure. They do not necessarily explain why the agent selected that code or how its decision relates to the reported requirement.

THEMIS attempts to preserve this missing information through stage-specific repair records. Yet its audit finds complete selected records in only 214 of 300 cases (71.3%), while many available diagnostic summaries have low confidence or limited specificity. A separate paired comparison reports 19 successful repairs versus 9 for a direct workflow, but cannot isolate the graph’s contribution.

For teams operating AI-assisted software maintenance, the implication is to evaluate record completeness, cross-stage consistency, and actual repair success separately. Readable logs can support investigation; they are not independently verified explanations.

A successful patch does not explain how the repair was chosen

A coding agent receives a bug report, identifies a candidate function, modifies its implementation, and submits a patch. A reviewer can examine the changed lines and test results. What may remain unclear is whether the selected function corresponds to the reported behavior, whether the agent preserved relevant constraints, and whether its recorded rationale describes the modification it actually made.

These questions matter when an organization needs to investigate an unexpected edit or determine whether an automated repair can be trusted.

Zewen Tao and Shin-nosuke Ishikawa address this problem in Externalizing Requirement-to-Repair Artifacts as Observable Traces for LLM-Based Program Repair, introducing a system called THEMIS.¹

The authors’ central contribution is an architecture that separates several decisions and observations that might otherwise remain implicit in an automated repair workflow. Rather than evaluating the system exclusively through completed patches, they examine whether intermediate records are available and whether those records visibly agree.

That distinction produces an instructive finding: even a workflow designed around traceability does not consistently retain the information required to inspect its own repair process.

THEMIS separates the repair decision from its subsequent audit

THEMIS combines four modules with different responsibilities.

The Advanced Code Analyzer interprets the issue and selected repository code, generating semantic hypotheses about the suspected fault, expected behavior, and possible repair.

The Enhanced Graph Manager constructs a requirement-code graph. This representation connects statements about required behavior to candidate functions, classes, and dependencies. It distinguishes blocking discrepancies, represented by VIOLATES edges, from non-blocking observations, represented by ADVISORY edges.

Before editing, the graph’s selected findings are distilled into textual guidance, including candidate targets, violation priorities, and proposed modification strategies.

The Developer receives that guidance alongside the raw requirements and semantic analysis. It performs one bounded repair attempt, producing a patch and selected rationale fields such as hypothesis_root_cause, expected_invariant, and patch_strategy.

Finally, the Judge rebuilds the graph after the edit and records conflicts and a repair brief describing the target, related symbols, expected behavior, and other diagnostic information.

The order matters. In the reported configuration, the Judge does not send its assessment back for another Developer revision. Its output is retained for retrospective inspection, not used to correct the patch in an iterative loop.

The paper formalizes the selected observable trace as:

Here, $r$ is a serialized requirement summary, $t$ a target summary, $d$ the Developer rationale, $p$ the patch, and $q$ the post-edit diagnostic.

This representation makes the stages distinguishable. It does not retain the complete runtime graph or establish that the relationships recorded between artifacts are semantically correct.

Complete records were available in only 71.3% of repairs

The authors retrospectively audited nine existing THEMIS log directories covering 300 SWE-bench Lite repository-repair instances.

Their principal continuity measurement examines whether three selected artifact groups are present: conflict history, complete Developer rationale, and core repair-brief fields.

Source: Section 5.1, Table 2.

Completeness is defined using explicit field-presence criteria. For example, a complete Developer rationale requires non-empty root-cause, expected-invariant, and patch-strategy fields. The repair-brief core requires a requirement identifier, target symbol, and expected behavior.

The resulting 71.3% does not mean that 71.3% of repairs were correctly explained. It means that 214 cases contained the selected fields needed for joint inspection.

The remaining 86 cases lacked at least one required artifact group. These are observable failures of record continuity, independent of whether the code change itself succeeded.

Even available records can contain weak diagnostic information. Among 222 repair-brief cores, 131 (59.0%) carried confidence values at or below 0.2, while 138 (62.2%) contained one of the study’s predefined generic issue summaries.

For a maintenance team, these are potentially measurable workflow-quality indicators. Their presence in the study does not establish that acting on them improves repair outcomes.

Matching function names is weaker evidence than matching requirements

The retained artifacts also allow the authors to compare the code target named in the repair brief with the Developer rationale and patch.

This is a lexical correspondence test: after normalizing symbol names, the procedure checks whether the qualified or shorter symbol name appears as a substring in another artifact.

The results show partial consistency:

• The target appears in 134 of 214 eligible rationales (62.6%).
• The target appears in 130 of 207 eligible non-empty patches (62.8%).
• Including related symbols increases patch correspondence to 157 of 207 (75.8%).

Within the 205-case cohort where both surfaces can be compared, 110 cases contain target matches in both rationale and patch, 18 in the rationale alone, 19 in the patch alone, and 58 in neither.

These disagreements identify records worth examining. They do not establish that the mismatching repairs were necessarily incorrect.

One unresolved example makes the distinction concrete: in django__django-16139, the post-edit graph identifies SetPasswordForm while the Developer edits UserChangeForm. The retained records expose this difference, without independently determining which target was semantically appropriate.

There is also a temporal measurement problem. The repair-brief target used for these comparisons is generated after the Developer has edited the code. It is not an independently validated localization target established before execution.

Consequently, agreement between the Judge’s target and the patch might partly reflect information available after editing. Even exact name matches cannot demonstrate that the Developer originally selected the correct function for the right reason.

The study therefore establishes measurable surface correspondence, rather than faithful causal provenance from requirements to edits.

The integrated workflow resolved more cases, but the graph’s effect remains unidentified

The main 300-instance run produced 279 non-empty patches (93.0%) but resolved only 58 benchmark instances (19.3%) under the official SWE-bench harness.

Patch production and successful repair are substantially different outcomes.

The authors also compare THEMIS with a direct same-input workflow on 100 matched instances. They hold constant the instance IDs, model (gpt-5.1-codex-mini), seed, revision count, requirement-string construction, file-selection strategy, and selected files.

Source: Section 5.6, Table 5.

Nine instances were resolved by both workflows, ten by THEMIS alone, and none by the direct workflow alone. The paired resolution difference is 10 percentage points, with an approximate 95% confidence interval of 4.12 to 15.88 percentage points. The exact two-sided McNemar test reports $p=0.002$.

This is evidence of an outcome difference between the two evaluated workflows, not an isolated estimate of what requirement-code graphs contribute.

The configurations differ simultaneously in semantic analysis, graph-derived guidance, Judge records, interfaces, and token budgets. The comparison therefore cannot identify which components account for the improvement. Moreover, post-edit Judge diagnostics cannot explain the observed advantage through additional Developer corrections, because no such correction pass occurs.

The result warrants further controlled evaluation of the integrated architecture. It does not establish that adding a graph alone will improve an existing coding agent.

For software maintenance, auditability requires several independent checks

The practical opportunity suggested by THEMIS is better inspection of automated repair processes.

A maintenance lead responsible for approving AI-generated code changes could require records identifying the reported requirement, candidate target, edit rationale, actual patch, and subsequent diagnostic assessment.

Cognaptus infers that these records would be most informative when monitored through separate measures: whether required fields exist, whether names and claims agree across stages, whether the recorded relationships withstand semantic review, and whether the submitted patch actually resolves the issue.

These checks answer different operational questions. Missing fields indicate an instrumentation problem. Contradictory targets suggest a possible localization or reporting problem. Failed tests indicate an unsuccessful benchmark repair. None automatically identifies the cause of another.

The distinction also changes architecture evaluation. A team considering graph-based guidance should compare it against alternatives under controlled model, context, interface, and budget conditions before attributing any observed performance difference to the graph itself.

Observable records are not yet validated accountability

The study’s conclusions are bounded by its evaluation setting.

The experiments use Python repositories from SWE-bench Lite, a single model configuration, and one Developer revision. The evaluated requirements incorporate benchmark-derived FAIL_TO_PASS test identifiers, optional hints, and sometimes manually supplied semantic contracts, so the results cannot be treated as performance estimates for ordinary issue-only repair.

The graph construction relies partly on heuristic requirement interpretation and mapping. The full runtime graph and complete reasoning records are not preserved in the selected audit projection. Neither the correctness of the recorded links nor their causal relationship to edits is independently established.

Crucially, the study does not test whether human reviewers can use these artifacts to detect failures faster, make better approval decisions, or reduce maintenance costs.

Those outcomes require separate validation.

Repair observability is a foundation, not a verdict

THEMIS demonstrates that an automated coding workflow can externalize selected intermediate decisions and make record continuity, symbol correspondence, and inconsistencies measurable.

Its audit also shows how easily a readable record can be mistaken for a validated explanation. A complete field set establishes availability. Matching symbols establish lexical recurrence. Passing tests establish a particular form of repair success. These properties are related, but none substitutes for the others.

For organizations adopting coding agents, the appropriate standard is therefore more demanding than retaining logs or reporting pass rates alone: preserve the relevant records, verify what they mean, and assess whether reviewers can actually use them to make better decisions.

Cognaptus: Automate the Present, Incubate the Future.