The Fine Print Is the Task: Why Long-Context AI Fails After Finding the Answer

TL;DR for operators

When an AI system reads a manual, policy, API specification, case file, or operating procedure, finding the relevant facts is only half the job. It must also discover the local rules that define what a valid answer looks like: required fields, exact labels, ordering constraints, exception handling, validation steps, prohibited actions, and completeness conditions.

A study of 1,899 long-context tasks finds that models pass 76.01% of individual requirements while completing only 22.55% of entire tasks. In other words, the typical failure is not complete confusion. It is an answer that is mostly right and operationally unusable.

The paper’s proposed method, Private Specification-Contract Induction, asks the model to extract a task-specific contract before answering. That contract then guides generation, checking, and repair. With GPT-5.1, strict task success rises from 22.55% to 28.14%; similar absolute gains appear with Gemini 3 Pro and Qwen3.5-27B.¹

The immediate business lesson is not “add four more model calls.” It is to separate two questions that many enterprise AI systems currently collapse:

1. What information is needed to answer?
2. What obligations must the answer satisfy to be accepted?

Retrieval helps with the first. Production reliability increasingly depends on engineering the second.

The model found the anomaly and still failed the task

Imagine an AI system reviewing a long event log.

The model correctly notices that a reserved event ID has been skipped. It explains the gap, identifies the affected records, and gives a sensible diagnosis. Unfortunately, the local operating manual also states that any reserved-ID gap must be logged using the exact flag Sequence_Gap_Warning.

The model writes “missing sequence detected.”

A human reader understands it. The downstream system does not. The answer contains the right content but violates the local protocol.

This distinction is the foundation of Agentic Context Learning with Self-Discovered Specification, a paper that examines why frontier models remain surprisingly weak when a long context defines not just the facts of a task, but its temporary operating environment.¹

The authors call this setting context learning. The model is given information absent from pre-training—perhaps a fictional rule system, a product procedure, a local schema, a regulatory code, or an experimental dataset—and must learn enough from the supplied context to act correctly.

This is not ordinary question answering with a longer document. The context defines both:

• Content: what the answer should say.
• Specifications: how the answer must behave to be locally valid.

Most long-context systems are designed as though the first category were the entire problem. The paper’s evidence suggests otherwise.

A 76% score can conceal a 22% success rate

The study uses CL-Bench, a benchmark containing 1,899 expert-written tasks across four families:

• domain knowledge reasoning;
• rule-system application;
• procedural task execution;
• empirical discovery and simulation.

Each task is evaluated through multiple binary rubric items. Altogether, the benchmark contains 31,592 items, or an average of 16.64 requirements per task. Contexts average 10,400 tokens and reach 65,000 tokens.¹

Under direct full-context prompting, GPT-5.1 passes 76.01% of individual rubric items.

That sounds respectable until the benchmark asks the only question a production system eventually cares about: did the complete output satisfy every required condition?

The strict task pass rate is 22.55%.

The difference between these numbers is not a statistical curiosity. It describes a familiar enterprise failure mode. A model produces an answer that is fluent, detailed, substantially correct, and missing one mandatory field. Or one exception. Or one exact label. Or one validation check. The output looks competent until it touches the workflow that must consume it.

Strict task success can be represented simply as:

where each $r_i \in {0,1}$ indicates whether requirement $i$ was satisfied.

When success is conjunctive, average competence does not rescue a missing obligation. Fifteen correct fields and one absent field still produce a failed transaction.

The authors examine 281 tasks that failed by exactly one rubric item. Among those final missed requirements, 75.3% were specification items, not content items.¹

This is the paper’s first important contribution. It changes the diagnosis.

The model often did not fail because it lacked the answer. It failed because it did not acquire the complete definition of an acceptable answer.

Specifications occupy more of the task than content does

To test whether this pattern extends beyond near-misses, the researchers classify all 31,592 rubric items into three groups:

The classification uses agreement between an LLM-based judge and a deterministic rule-based classifier, with disagreements assigned to the ambiguous category. A separate human audit of 300 items agrees with the non-ambiguous labels in more than 90% of cases.¹

The conservative design matters. The headline finding is not produced by forcing every debatable formatting requirement into the specification bucket. Even after abstaining on 22% of items, specifications remain the largest component of the benchmark.

Nor are these specifications limited to cosmetic output formatting. The paper divides them into eight subtypes:

• validation and edge-case rules;
• exclusions and forbidden actions;
• ordering and sequence requirements;
• schemas and structured outputs;
• source-grounding requirements;
• coverage and completeness conditions;
• exact-phrase constraints;
• role, tone, and style constraints.

Validation and edge-case rules alone account for 42.9% of specification items. Ordering requirements account for another 12.6%.

A company could dismiss punctuation or tone as benchmark fussiness. It is harder to dismiss an omitted exception rule, an invalid calculation threshold, or a maintenance step performed out of sequence.

The rules are available, but the question does not point to them

One alternative explanation would be that the benchmark contains hidden evaluator preferences: requirements that no reasonable model could infer from the supplied material.

The paper investigates that possibility directly.

It finds that 95.5% of specification items are traceable to the task context. Most appear through explicit declarations or validation patterns, with smaller shares communicated through examples and repeated conventions. Only 0.4% are classified as not derivable from the context; another 4.1% are too briefly expressed to judge.¹

The rules are therefore mostly present.

The problem is where they are present.

Only 23.3% of specification items can be inferred from the user query alone. The other 76.7% require reading the surrounding material.

A user might ask:

Analyse these rowing synchronization records and report the operational impact.

The context may separately define:

• which dataset version overrides conflicting artifacts;
• the precise synchronization formula;
• a 90% threshold below which efficiency loss should be calculated;
• the required output tags;
• which findings must be escalated to another agent;
• which causal conclusions the current agent is prohibited from making.

The apparent question is about rowing performance. The real task also includes learning a temporary analytical constitution.

This creates an asymmetry between query relevance and validity relevance.

Query-relevant information helps answer what the user visibly asked. Validity-relevant information determines whether the answer complies with the local system. Retrieval methods are generally optimized for the former.

That is why a retriever may surface the relevant race data and omit a seemingly minor paragraph defining a threshold. Unfortunately, the “minor” paragraph controls the calculation.

Twelve familiar fixes mostly fail to move the result

The paper compares direct full-context prompting with twelve methods drawn from four broad families:

Across these methods, strict pass rates range from 19.00% to 23.33%. None improves on the 22.55% baseline by more than 0.78 percentage points.¹

This result should not be read as proof that retrieval, reflection, or verification are useless. All three solve real problems.

It shows something narrower and more useful: those methods do not automatically solve this problem.

A retrieval system can improve access to answer-bearing evidence while discarding distributed constraints. A generic critic can complain that an answer needs more detail without knowing the exact local obligations that matter. A post-hoc rubric generator may discover a missing requirement after the answer’s structure has already been fixed.

The predictable industry response to weak long-context performance is usually to add more retrieval, more reasoning tokens, or another self-reflection pass. It is a comforting response because the architecture remains conceptually unchanged. The paper’s evidence suggests the architecture may be optimizing the wrong object.

PSCI turns the local rules into a control object

To test the specification-acquisition diagnosis, the researchers introduce Private Specification-Contract Induction, or PSCI.

It uses four sequential stages:

Task + context
      |
      v
1. Induce a private specification contract
      |
      v
2. Generate an answer under that contract
      |
      v
3. Check each obligation against visible answer evidence
      |
      v
4. Repair failed or uncertain obligations

The contract is a structured list of requirements inferred from the current task and context. It can include:

• required fields;
• output shape;
• exact local labels;
• validation steps;
• sequencing constraints;
• exclusions;
• evidence requirements;
• edge cases;
• completeness conditions;
• likely omission risks.

The contract is “private” because the system never sees the benchmark’s public evaluation rubric for the current task. It must reconstruct the obligations from the source material itself.

The method also uses rotated demonstrations from unrelated tasks to teach the desired style and granularity of a contract. These demonstrations do not contain the current task’s context, answer, or rubric. Zero-shot contract induction still improves performance, indicating that the demonstrations refine the representation rather than supply the underlying task information.¹

The contract then becomes a shared control object.

The answer generator sees it before drafting. The checker evaluates the draft against its individual obligations. The repair stage receives localized failures instead of a vague request to “improve the answer.”

That ordering is more important than it may appear.

A checklist generated after drafting is mostly a grader. A contract generated before drafting can influence the structure of the answer: which sections exist, which cases are enumerated, which calculations are performed, and which actions are excluded.

In enterprise terms, PSCI moves policy interpretation upstream.

The main result is meaningful, but nobody should schedule the victory parade

With GPT-5.1 at medium reasoning effort, PSCI raises strict task success from 22.55% to 28.14%.

That is:

• 5.59 percentage points of absolute improvement;
• 24.8% relative improvement;
• a rise in global rubric pass from 76.01% to 81.13%.¹

The gains replicate across two additional model families:

PSCI with medium-effort GPT-5.1 also exceeds the 23.7% result reported for GPT-5.1 at high reasoning effort in the original benchmark evaluation.¹

This does not mean PSCI makes the models reliable. A 28.14% strict pass rate still means failure on more than seven tasks out of ten.

What the result establishes is a causal direction. Explicitly organizing inference around local specifications yields more improvement than simply spending additional reasoning effort on the undifferentiated task.

The improvement also aligns with the diagnosis. Specification-item pass rises from 76.7% to 82.1%, while content-item pass rises from 73.4% to 77.5%. Among rubric items repaired on tasks that change from baseline failure to PSCI success, 77.7% are specification items.¹

The intervention improves the part of the task it was designed to address. This sounds obvious. In AI evaluation, it is almost luxurious.

The ablations identify the mechanism, not a second collection of results

The paper runs 17 ablations. Their purpose is not to introduce seventeen alternative products. They test whether PSCI’s gain comes from the proposed specification mechanism or from familiar confounds such as extra tokens, extra calls, generic checklists, or fortunate prompt wording.

The most informative tests are these:

Three conclusions follow.

The contract is the primary driver

Specification induction alone reaches 26.09%, compared with 22.55% for the baseline and 28.14% for full PSCI. The checker and repair stages add another 2.05 percentage points.¹

This suggests that the first operational priority is not building an elaborate correction loop. It is constructing a useful representation of the local obligations.

A misaligned contract can actively degrade performance

When the researchers replace the task-specific contract with a contract from another task, strict performance falls below the direct baseline.

This result deserves attention outside the benchmark. Contract extraction is not a harmless summarization layer. If the extracted rules are generic, incomplete, stale, or imported from the wrong policy domain, they can steer generation away from the correct answer with impressive discipline.

A control object controls. That is useful only when it is the right object.

Timing changes what the contract can control

When the model answers first and induces specifications afterward, strict pass falls to 23.67%, and rubric pass drops below the direct baseline to 70.67%.¹

Post-hoc editing can insert a label or correct a field. It is less reliable when the missing specification affects global structure:

• which sections should exist;
• which cases must be covered;
• which evidence must be used;
• which calculation path is valid;
• which step order must be preserved.

This explains why generic “draft, critique, revise” loops often disappoint. The critic arrives after the architecture of the answer has already hardened.

The qualitative examples show how competent answers become invalid outputs

The appendix provides two illustrative cases. They are not additional quantitative evidence; the main support comes from the full benchmark, ablations, and human audit. Their value is diagnostic: they make the failure mode visible.

A manufacturing plan follows the facts but not the runbook

In one task, the model must assemble a unit using a source-defined 12-step process. The context contains conflicting revisions, an engineering change notice, a spacer anomaly, a prohibited tool suggestion, and a mandatory hand-spin test.

The direct baseline produces a detailed 7,264-character response and passes 10 of 19 rubric items. It identifies much of the relevant content correctly.

It nevertheless:

• inserts an extra step, shifting the required sequence;
• accepts limited use of an impact driver when the local procedure requires rejecting it;
• mentions a substitute wrench but omits the linked extra-torque requirement.

PSCI passes all 19 items.¹

The baseline does not look unintelligent. It looks like a capable employee improvising around a procedure that the company expected to be followed exactly. Depending on the industry, that is either a minor workflow defect or the first paragraph of an incident report.

A rowing analysis calculates the wrong business object correctly

In another task, the model analyses rowing synchronization. It receives multiple artifacts, a designated authoritative dataset, a source-defined synchronization formula, a 90% threshold for applying efficiency loss, and a required four-part output structure.

The baseline performs substantive numerical analysis and passes 14 of 21 requirements. It reports a synchronization score of 87%, calculates an efficiency loss, and propagates that loss into estimated time and power impacts.

The problem is that its score is calculated incorrectly. Under the source-defined formula, the score should be approximately 92%. Because that exceeds the 90% threshold, the efficiency-loss model should not be applied at all.

The downstream calculations are internally coherent and locally invalid.

This is a particularly useful enterprise analogy. AI systems can produce accurate arithmetic over the wrong specification. Better calculators do not help when the wrong formula has been selected.

What the paper directly shows—and what businesses should infer

The evidence supports several claims with different degrees of confidence.

The distinction matters because the paper does not test a production deployment. It tests a benchmark designed to isolate context learning.

The correct business response is therefore architectural experimentation, not blind replication of the four-call prompt pipeline.

A practical enterprise pattern: retrieve facts, compile obligations, then execute

For an enterprise agent, the paper suggests a control architecture with at least three distinct objects:

Source context
   |
   +--> Evidence set --------> What is true or relevant?
   |
   +--> Obligation contract -> What must the output do?
                                  |
                                  v
                         Generation or action
                                  |
                                  v
                         Obligation-level check

The evidence set and the obligation contract may come from the same documents, but they serve different functions.

A retrieval layer might identify the customer’s transaction history, the relevant policy clause, and the current product terms.

A specification layer might extract that the response must:

• cite the controlling policy version;
• include a mandatory disclosure;
• avoid making a suitability determination;
• escalate transactions above a defined threshold;
• produce a particular JSON schema;
• omit protected fields;
• record a reason code;
• request human approval before execution.

Treating all of this as “context” is technically convenient and operationally vague. Production systems benefit from typed intermediate representations.

The paper’s contract format is one implementation. A more mature enterprise system might compile obligations into:

• schema validators;
• deterministic business rules;
• permission checks;
• policy-as-code;
• required tool calls;
• state-machine transitions;
• human-approval gates;
• post-generation assertions.

The model can discover candidate obligations. It should not necessarily be the final authority enforcing all of them.

The ROI question is routing, not universal deployment

PSCI uses four sequential model calls. Under the paper’s common model and API setting, the direct baseline generates about 18,900 tokens per task with 44.6 seconds of wall-clock latency. PSCI generates about 91,600 tokens and takes 132.2 seconds.¹

Those numbers should not be treated as universal cost forecasts. They depend on the benchmark, prompts, provider, model, and evaluation environment.

They do establish the direction of the trade-off: PSCI is substantially more expensive than single-pass generation.

The business case therefore depends on error economics.

A four-stage specification process may be rational for:

• regulated communications;
• complex financial operations;
• industrial procedures;
• compliance reporting;
• high-value research deliverables;
• irreversible tool actions;
• outputs consumed automatically by downstream systems.

It will be difficult to justify for:

• low-risk summarization;
• conversational brainstorming;
• disposable internal drafts;
• high-volume tasks where minor omissions have negligible cost.

The sensible deployment pattern is adaptive. Use cheaper generation when the specification surface is small and explicit. Escalate to contract induction and enforcement when the context contains distributed procedures, exceptions, schemas, permissions, or hard completeness requirements.

The paper demonstrates that structured computation beats similarly expensive generic computation. It does not demonstrate that maximum computation should be applied to every request.

The remaining bottleneck is enforcement

The paper also compares induced contracts with the held-out evaluation rubrics.

Across 24,640 non-ambiguous items, the contracts strongly cover 60.8% and partially cover 33.4%, for roughly 94% total coverage. Contradictions occur in 0.49% of items. A human audit finds somewhat higher strong coverage than the automatic measurement, suggesting that the automated estimate is conservative.¹

Better contract coverage is associated with better downstream performance. Rubric pass rises from 48.7% for uncovered items to 88.8% for strongly covered items.

Yet the most revealing result appears in the residual failures.

Among rubric items still failed by PSCI:

• 83% are covered by the contract but not executed correctly in the final answer;
• 14.9% are discovery failures;
• 2.0% involve contradictions.¹

Once the system becomes good at identifying the rules, following them becomes the dominant problem.

This places a boundary around the paper’s achievement. Specification extraction does not produce specification obedience.

For production use, an induced natural-language contract may need to be translated into stronger controls. Some obligations can be enforced through code. Some can be validated deterministically. Some require model-based judgment. Some require human review. The most important design decision may be deciding which category each obligation belongs to.

A model checker asking the same model family whether an answer followed a natural-language rule remains a probabilistic control. It is better than no control. It is not the same as an invariant.

A contract can faithfully enforce a bad rule

The method optimizes local validity relative to the supplied context. It does not determine whether the context itself is true, safe, lawful, current, or ethically acceptable.

A contract induction system may extract and enforce:

• an obsolete operating procedure;
• a discriminatory eligibility policy;
• an unsafe shortcut;
• a legally invalid local instruction;
• a malicious rule inserted through prompt injection;
• a policy that conflicts with a higher-level corporate control.

The answer-generation prompt used in PSCI instructs the model to audit the contract against the original source and task. That helps with extraction errors. It does not resolve conflicts between local instructions and external truth, law, safety, or organizational authority.

Enterprise systems therefore need an obligation hierarchy.

Law and safety constraints
          >
Enterprise-wide policy
          >
Approved domain procedure
          >
Task-local instructions
          >
User preferences

A local specification should control the answer only after passing higher-level authorization and provenance checks.

The paper makes AI better at reading the fine print. Governance must still decide whose fine print is allowed to govern.

The benchmark is broad, but it is still one benchmark

The paper’s evidence is unusually detailed for a single-benchmark study: 1,899 tasks, 31,592 rubric items, twelve baselines, three model families, seventeen ablations, classification audits, contract-overlap audits, and a blind review of 100 baseline-fail/PSCI-pass examples.

In the blind audit, human annotators prefer the PSCI answer in 96 of 100 sampled cases and confirm the automatic failure-to-pass flip in 92 cases.¹

That audit addresses an important concern: perhaps PSCI merely learns to write text that pleases the automated evaluator. The human judgments suggest most measured improvements are genuine.

The audit is nevertheless targeted. It samples cases already identified as PSCI wins. It does not estimate total human-evaluated performance across the benchmark, nor does it establish field performance.

External validity remains open in several areas:

• contexts assembled from messy real documents rather than benchmark packages;
• incomplete or mutually inconsistent specifications;
• specifications that change during execution;
• agents taking actions instead of generating static answers;
• interactions among retrieval, tool use, memory, and policy enforcement;
• multilingual or organization-specific documentation;
• latency-sensitive environments;
• adversarially inserted local rules.

These are not ritual limitations. Each affects whether a specification-contract layer will function as a useful control system or merely produce another confident document for nobody to read.

The real lesson is to stop treating all context as evidence

The long-context industry has spent years improving how models find information in large inputs. That work remains necessary.

But a manual is not merely a bag of facts. A policy is not merely a source for question answering. An API document is not merely content to summarize. These objects also define temporary rules of operation.

The paper’s most useful distinction is therefore not between a direct prompt and PSCI. It is between answer-bearing context and validity-bearing context.

The first tells the model what to conclude.

The second tells the system what counts as completion.

Most enterprise failures occur after those two have been blended into a single prompt and entrusted to probability. The model reads almost everything, remembers most of it, produces an excellent answer, and neglects one sentence in the appendix that the actual workflow considers mandatory.

Very intelligent. Very fluent. Rejected by the validator.

The better architecture is not simply to retrieve more text or ask the model to think harder. It is to surface local obligations before generation, keep them active during execution, and enforce each one with the strongest mechanism available.

The answer may be in the context. So is the contract.

Cognaptus: Automate the Present, Incubate the Future.