The Summary Kept the Files. The Agent Lost the Fix.

TL;DR for operators

A coding agent can retain the right files and repair evidence after context compaction yet fail to complete work that it could finish with its full history. Compact-Bench¹ measures this failure through matched execution continuations, not summary inspection alone.

Across 14 summarizer backbones, stable compaction-induced damage ranges from 5% to 64% of 22 validated full-history repair successes. The benchmark also finds that configurations preserving more exact technical details do not necessarily complete more repairs. A mechanical extractor retains 79.1% of recurring structured details, versus at most 16.0% for the tested LLM configurations, but its ability to preserve executable progress is not evaluated.

For engineering teams, summarizer selection and compaction-prompt changes should be assessed using actual continuation outcomes, with structured-detail retention as a separate diagnostic. The benchmark establishes the value of distinguishing these measurements; it does not demonstrate that a particular checkpoint design eliminates compaction failures.

The agent found the files but never made the repair

Consider a coding agent that has spent several steps investigating a software defect. It has inspected files, examined tests, and narrowed down what needs changing. Its accumulated history becomes too long, so the system replaces that history with a shorter summary. The repository remains available, and the agent resumes.

In one evaluated Sphinx repair, identified as sphinx-9230, the full-history continuation applied the relevant code edit. The summary-only continuation accessed relevant files but continued examining tests and reproduction code without completing the edit. The summary contained the target file and four listed repair-relevant strings. (Section 4.4, Table 5; Appendix I, Table 14.)

The failure was not simply an inability to locate the code. The two continuations used available evidence differently, and only one reached the repair within its execution budget.

This illustrates a distinction that matters when selecting memory systems for coding agents. A summary can preserve technical evidence without preserving the operational progress that connects previous investigation to the next necessary action. The trace does not identify which particular omission or representation change caused the failure, but it shows why checking summary contents alone is insufficient.

Measuring whether work survives history replacement

Compact-Bench addresses this problem through three complementary evaluations, led by an executable test called Task Preservation.

Rather than asking whether a summary looks complete, the benchmark compares two continuations originating from the same repository state:

• PRE: The agent resumes with its complete preceding execution history.
• POST: The agent resumes with only a compacted replacement summary.

Both start from the restored working-tree snapshot. The continuation agent is Kimi-K3, with matched tools, environment, and an eight-step execution budget. Success is determined by repository FAIL_TO_PASS tests, rather than an LLM judge’s assessment of the summary.

The initial execution history contains at most ten agent steps. The benchmark selects 35 localized repairs across seven Python repositories, but the principal damage analysis concerns 22 repairs that demonstrably succeed with full history.

A repair counts as damaged when PRE passes and POST fails. This conditional comparison matters: it asks how often compaction destroys a demonstrated success, not how often coding agents fail generally.

The other two evaluation tracks examine information preservation. Recurring-Detail Retention measures exact structured values that occur both before an artificial compaction boundary and in subsequent recorded requests. Production-Chain Retention examines summaries actually produced at successive compaction boundaries in real coding-agent sessions.

Together, these tracks distinguish the outcome of continued work from the information available to support that work.

Summarizer choice changes which repairs survive

With the Claude Code compaction prompt fixed, Compact-Bench evaluates 14 summarizer backbones. The results vary substantially.

Source: Section 4.2, Table 2. Selected backbone results; common Claude Code prompt and Kimi-K3 continuation agent.

These differences concern executable outcomes, not just summary style or semantic similarity. Claude Opus 4.8 preserves 21 of the 22 demonstrated successes, while DeepSeek-Flash preserves eight.

The benchmark also tests repeatability. Initially damaged comparisons receive two additional independent continuation samples, with stable damage requiring POST failure in at least two of three runs. Among 126 first-round damaged instance–backbone pairs, 120 meet that criterion. This is a robustness check on observed damage, not a full estimate of each configuration’s outcome distribution.

Task-level disagreement is equally revealing. Across all 14 backbones, only four of the 22 repairs receive unanimous outcomes. For pytest-10081, Claude Opus 4.8 preserves the repair while the other 13 backbones damage it.

That variation supports evaluating summarizers on the actual tasks an agent must resume. It does not establish a universal model ranking: the comparison uses a small, selected sample, a single continuation agent, and a fixed execution budget.

More remembered details do not necessarily mean more completed repairs

The benchmark’s prompt comparison makes the distinction between retention and continuation especially clear.

With Kimi-K3 fixed as the summarizer, five production compaction prompts produce the following outcomes:

Source: Sections 4.2–4.3, Table 3. Repair outcomes use single runs; retention scores come from a separate 146-session dataset.

Kimi Code achieves one more successful continuation than Claude Code, but its measured detail retention is substantially lower. The one-repair difference cannot establish a robust prompt superiority ranking, particularly because the prompt comparison uses only one run per repair.

The retention measure itself is deliberately narrow. Researchers extract structured values such as file paths, commit identifiers, issue references, error types, URLs, and key-value pairs. A value counts as retained if it appears as a case-sensitive substring in the summary. Session-level retention fractions are averaged without weighting by the number of details.

Across 146 sessions containing 9,330 recurring structured details, the highest LLM configuration reaches 16.0%. A bounded mechanical extraction method, anchor_index, reaches 79.1%.

The 63.1-percentage-point gap shows how many literal identifiers generative summaries can omit compared with deterministic extraction. It does not show that mechanical extraction would produce better task continuations; that baseline is not evaluated for executable repair success.

A second comparison reinforces the measurement distinction. Under the same Claude Code prompt, DeepSeek-Flash retains 16.0% of recurring details versus Kimi-K3’s 15.0%, yet damages 14 of 22 repairs rather than four.

The two evaluations use different datasets, so this is not a measured causal relationship between any particular retained detail and repair success. It is evidence that literal recall and executable task preservation cannot safely be treated as interchangeable scores.

Repair traces expose what summary scores miss

The benchmark examines selected damaged cases to understand what happens after compaction.

In four stably damaged Kimi-K3 cases, the target file remains represented in the summary. For pytest-7571, all five audited gold-patch tokens survive, yet the POST continuation fails. For sphinx-9230, the agent accesses relevant files but does not convert its investigation into the successful edit made by PRE.

These case analyses are diagnostic evidence, rather than an ablation establishing the causal importance of a specific missing instruction.

They suggest that compaction evaluation should inspect the state of the work as well as its recorded artifacts: which hypotheses were investigated, which actions were completed, what remains unresolved, and how the resumed agent proceeds. Simply preserving more paths or code fragments may not address failures in using those details.

Production data offer a related, but more limited, view. Among 106 real sessions containing 224 compaction boundaries, 140 boundaries have evaluable future details. At the first boundary, summaries retain an average of 15.7% of structured details appearing in subsequent requests.

Unlike the artificial recurring-detail evaluation, this production measure does not require every future detail to have appeared before compaction. Some apparent omissions therefore may not represent information that the summarizer could have preserved. Later boundary samples also shrink sharply, preventing a reliable conclusion about progressive deterioration across repeated compactions.

How to test compaction changes before deployment

For teams running long-lived coding agents, Cognaptus draws three design and evaluation implications from these results.

First, make task continuation the primary regression test. When comparing a new summarizer, prompt, or context policy, replay representative workflows from matched execution states and check executable outcomes. The benchmark demonstrates that changing the summarizer alone can change which repairs succeed.

Second, measure structured retention separately. Paths, identifiers, completed actions, unresolved issues, and tested hypotheses are plausible candidates for inspectable checkpoint state. Deterministic extraction could complement generated summaries, but Compact-Bench does not test whether this hybrid design improves task success.

Third, inspect divergent trajectories when a continuation regresses. Check whether the resumed agent loses relevant evidence, repeats investigation, fails to perform an identified edit, or interacts with a mismatched execution environment. The benchmark’s interpreter, working-directory, and snapshot controls show why these distinctions matter: an earlier separate-directory evaluation setup directed POST commands outside the judged tree in 22 of 34 cases.

These procedures also have a cost. The full 35-repair study requires approximately 8,000 model calls, while compression accounts for less than 5% of tokens. Most evaluation expense comes from generating continuations and judging executable results. Platform teams budgeting for compaction regression testing should therefore account for replay costs, not merely the price of summary generation.

The evidence supports a testing discipline, not a universal configuration

Compact-Bench’s strongest evidence comes from matched repository snapshots and executable repair outcomes. The repeatability checks strengthen confidence that many observed failures are not isolated sampling accidents.

Its external validity is narrower. The repair set is deliberately localized; the headline comparison depends on only 22 full-history successes; Wilson confidence intervals are wide; and the continuation agent and eight-step budget are fixed. The production retention track is descriptive, while private session payloads and cached repair states are not publicly released for exact independent reproduction.

The study therefore cannot determine which summarizer is best for every deployment or prove that retaining explicit operational state prevents failed continuations.

What it establishes is a more useful evaluation target. When a coding agent crosses a context boundary, successful compression is not demonstrated by a summary that mentions the right evidence. It must also be judged by whether the resumed agent can use its remaining context to complete the work.

For long-running agent systems, continuation success and information retention deserve separate measurements. Compact-Bench provides a controlled way to observe their divergence, and a concrete basis for testing compaction choices against the outcomes engineering teams actually need.

Cognaptus: Automate the Present, Incubate the Future.