The Fourth Hop Changes the Risk Profile: Measuring Reliability in Multi-Step LLM Workflows

TL;DR for operators

When one model output becomes input to the next stage, a final accuracy score tells you too little about where reliability is being lost. A workflow may fail because a required fact was never available, because a later composition step is intrinsically harder, or because an earlier mistake was allowed to propagate. Those failure modes call for different controls.

Omanic¹ makes that distinction measurable with expert-reviewed four-hop questions and explicit intermediate answers. Across selected models, the fourth hop is consistently the weakest. Chain-of-thought improves results when constituent facts are available, but its benefit collapses as those facts go missing. Supplying correct prior answers does not remove the fourth-hop weakness, and carrying model-generated answers forward makes it worse.

For operators, the implication is diagnostic rather than prescriptive: measure reliability by stage, test factual coverage separately from composition, and spend verification budget where dependency depth makes failure more consequential. Omanic supports that approach within controlled English four-hop QA; it does not establish universal thresholds for longer, multilingual, or domain-specialized systems.

Reliability deteriorates before the final answer reveals it

Consider a workflow that retrieves a fact, transforms it, combines it with another result, and then makes a final decision. If only the final output is scored, a correct answer can conceal a weak intermediate step, while an incorrect answer gives little indication of whether retrieval, transformation, or composition caused the failure.

The paper turns that problem into a controlled measurement task. OmanicBench contains 967 expert-reviewed four-hop instances, each decomposed into four single-hop questions with intermediate answers. The construction also includes annotated dependency topologies and at least one mathematically grounded hop. Its purpose is diagnostic: not simply to ask whether a model reaches the right endpoint, but to locate where performance changes as reasoning progresses.

That distinction becomes visible in the per-hop results. Under direct prompting, GPT-5.4 scores 85.63%, 86.97%, and 84.80% on Steps 1–3, then falls to 66.08% on Step 4. With chain-of-thought, the same sequence becomes 93.80%, 95.66%, 92.66%, and 79.63%. Claude-Sonnet-4.6 and Gemini-3.1-Flash-Lite show the same ordering: Step 4 is the lowest-accuracy hop under both prompting conditions.

The relevant signal is therefore not merely that four-hop questions are harder. Reliability changes with position inside the dependency chain.

Longer reasoning cannot supply facts the model does not have

Chain-of-thought is an obvious response to multi-step difficulty because it gives the model more room to decompose a problem. Omanic shows an important condition on that benefit.

The researchers group questions according to how many constituent single-hop questions the model answers incorrectly. When all constituent questions are correct, the average chain-of-thought gain across evaluated models is about 21.9 percentage points. When three constituent steps are wrong, the average gain falls to approximately -0.7 points.

This is the paper’s factual knowledge floor: reasoning scaffolding becomes much less effective when the premises required for the chain are unavailable.

The inverse result matters too. Even when the model answers every constituent single-hop question correctly, direct multi-hop accuracy is only about 60%. Possessing the atomic information is therefore necessary but not sufficient. The model must still select, combine, and propagate that information correctly.

For a production system, Cognaptus infers a sequencing rule for diagnosis. Before increasing reasoning length or paying for a more elaborate reasoning prompt, determine whether the failure is caused by missing information. Retrieval coverage and compositional execution are separate problems; additional inference can address only the latter reliably under the conditions tested here.

The cost difference can be substantial. Qwen3-Max averages 14 output tokens under direct prompting and 4,841 with chain-of-thought; GPT-5.4 averages 13 and 451 respectively. Reasoning quality and inference cost therefore need to be evaluated together rather than treating chain-of-thought as a cost-neutral fallback.

The fourth hop has two different sources of failure

A later step might be weak simply because it inherits bad inputs. Omanic tests that explanation by evaluating steps in two ways.

In the first condition, the model receives the correct previous-hop answers. This isolates how difficult the current step is when upstream information is clean. In the second, the model must carry its own generated intermediate answers forward. That introduces the errors a sequential production workflow would actually propagate.

Step 4 remains disproportionately difficult even when correct previous answers are supplied. The later-hop bottleneck is therefore not reducible to cascading mistakes.

Propagation then adds another penalty. Under direct prompting, Step-4 error reaches 33.0% when model-generated intermediate answers are carried through the chain, 4.7 percentage points above the corresponding independent evaluation. Chain-of-thought lowers absolute error but does not remove the amplification pattern.

These two effects imply different operational controls. An intrinsically difficult later step may justify stronger models, decomposition, or targeted verification at that stage. Propagated error instead argues for checking an upstream result before it becomes an input to several dependent operations. A single aggregate accuracy number cannot tell an operator which intervention is appropriate.

Structured training improves more than the benchmark score

Omanic also includes OmanicSynth, a 10,296-instance synthetic training set built around four-hop dependencies. On OmanicBench, Qwen3-8B rises from 25.65% MCQ accuracy in its vanilla form to 53.62% after supervised fine-tuning and 53.77% after supervised fine-tuning plus GRPO. LLaMA-3.3-70B rises from 40.04% to 57.55% after supervised fine-tuning.

The more consequential test is external transfer. Fine-tuned models improve across six reasoning and mathematics benchmarks—GSM8K, ReClor, MATH, PrOntoQA, LogiQA, and AR-LSAT—with a reported average gain of 7.41 percentage points. That does not establish universal transfer, but it reduces the likelihood that the improvement is only adaptation to OmanicBench’s evaluation format.

The paper then asks whether training simply added factual knowledge or changed how information was used. For Qwen3-8B, the authors examine answer uncertainty as correct prior-hop information accumulates. After OmanicSynth supervised fine-tuning, final-answer entropy falls and the trajectory under gold prior-hop context changes from increasing to decreasing. The advantage also grows at later isolated hops.

This is behavioral evidence consistent with better organization and propagation of accumulated information. It is not direct evidence about an internal neural mechanism. The distinction matters: the experiment supports the interpretation that structured training improves compositional use of information, but it does not prove exactly how that change is implemented inside the model.

Production evaluation should separate three failure classes

The paper’s results suggest a practical evaluation design for sequential LLM systems.

This is a Cognaptus inference from the benchmark design, not a production result measured by the paper. The affected users are teams running workflows in which model-generated outputs feed later model calls or automated decisions. The decision is where to place retrieval, model capacity, and verification budget. The relevant condition is dependency: an intermediate result deserves more scrutiny when several later operations rely on it.

The paper also exposes a measurement issue. Claude-Sonnet-4.6 improves its MCQ score from 55.43% to 73.11% with chain-of-thought, yet its open-ended Exact Match and F1 decline. The authors attribute this to verbose responses obscuring the extractable final answer. Production evaluation should therefore distinguish reasoning correctness from response-format and extraction reliability rather than letting one metric stand in for all three.

The four-hop gradient is evidence, not a universal law

OmanicBench is deliberately controlled. It is English-only, uses a fixed four-hop structure, has moderate scale, and underrepresents specialized areas such as legal and biomedical reasoning. Its construction differs from unconstrained production workflows, where tools, retrieval systems, branching agents, retries, and human intervention can change the dependency structure.

The paper therefore supports a method of diagnosis more strongly than a universal numerical rule. It shows that step position, factual prerequisites, and propagated intermediate errors can be separated and measured. It does not establish that Step 4 will always be the critical point, that a 4.7-point propagation penalty should be expected elsewhere, or that four-hop findings extrapolate to six, eight, or twenty dependent operations.

That boundary strengthens the practical lesson rather than weakening it. A production team does not need Omanic’s exact gradient to ask the same diagnostic questions of its own workflow.

Reliability should be measured where dependencies accumulate

Multi-step systems create failure modes that endpoint evaluation compresses into one number. Omanic’s main contribution is to reopen that number and show what is inside: missing premises, harder late-stage composition, and errors inherited from earlier outputs.

For operators, the resulting change is straightforward. Test whether the required information exists before paying for additional reasoning. Evaluate difficult stages with clean upstream inputs before blaming propagation. Then run the same stages end to end to measure how much error the workflow itself adds.

A final-answer benchmark can tell you whether a system succeeded. Step-wise evaluation is what starts to tell you where reliability should be repaired.

Cognaptus: Automate the Present, Incubate the Future.