When Retrieval Learns to Breathe: Teaching LLMs to Go Wide *and* Deep

Opening — Why this matters now

Large language models are no longer starved for text. They are starved for structure. As RAG systems mature, the bottleneck has shifted from whether we can retrieve information to how we decide where to look first, how far to go, and when to stop. Most retrieval stacks still force an early commitment: either search broadly and stay shallow, or traverse deeply and hope you picked the right starting point.

This paper introduces ARK (Adaptive Retriever of Knowledge), an agentic retrieval framework that refuses to choose upfront. Instead, it lets the model breathe—expanding outward when unsure, diving inward when relations matter. The result is a retrieval system that adapts its own search strategy at inference time, without task-specific training.

Background — The unresolved breadth–depth dilemma

Knowledge-graph retrieval lives between two unsatisfying extremes:

• Similarity-based retrievers (BM25, dense embeddings) offer global coverage but flatten relational structure. They find things that look relevant, not things that are connected for a reason.
• Traversal-based agents perform multi-hop reasoning but depend on fragile seed selection. Start in the wrong neighborhood, and you never recover.

Prior work tends to optimize one side of this tradeoff at the expense of the other. Some systems encode shallow neighborhoods into embeddings. Others train policies to walk the graph—often expensively, and often brittle across domains.

What’s been missing is a retrieval process that can dynamically switch modes based on the query itself.

What the paper does — Retrieval as an interactive policy

ARK reframes KG retrieval as a tool-using agent problem. Instead of committing to a fixed pipeline, the language model is given a minimal but expressive toolset:

Crucially, these tools can be composed arbitrarily. Multi-hop traversal emerges naturally by chaining one-hop expansions—not by hardcoding a depth limit.

The agent maintains an ordered list of retrieved nodes and decides, step by step, whether to:

• search globally again,
• expand locally around a promising node, or
• stop.

No predefined hop count. No fixed seeds. No retriever training.

Results — Adaptive behavior beats static pipelines

Across the STaRK benchmark (AMAZON, MAG, PRIME), ARK delivers the strongest average retrieval performance among training-free methods.

Key quantitative takeaway

The interesting part is why this works.

Tool usage mirrors query structure

ARK wasn’t told which queries are textual vs relational. Yet its behavior aligns closely with dataset characteristics:

• AMAZON (text-heavy): ~88% global search
• MAG (relational): ~65% neighborhood exploration
• PRIME (mixed, biomedical): balanced usage

In other words, ARK learns when to go wide and when to go deep—implicitly, from the query.

Compute, but with knobs

Agentic systems are often criticized for latency. The paper doesn’t dodge this—it exposes it.

By varying:

• number of agents (parallel runs), and
• maximum steps per trajectory,

ARK yields a clean budget–performance frontier. Want cheaper inference? Use one agent and shallow steps. Want higher recall on complex queries? Add depth or parallelism.

This explicit tradeoff is refreshing in a space that often hides cost behind opaque pipelines.

Distillation — Making it practical

Perhaps the most underappreciated contribution is label-free agent distillation.

A large teacher model (GPT-4.1) runs ARK and produces tool-use trajectories. A smaller model (Qwen3-8B) is then fine-tuned only on these interaction traces—no relevance labels required.

The result:

• Up to 98.5% of teacher performance,
• At a fraction of inference cost,
• With training completed in hours, not weeks.

This turns ARK from a research prototype into something deployable.

Implications — Why this matters beyond benchmarks

ARK quietly argues for a shift in how we think about retrieval infrastructure:

• Retrieval should be adaptive, not static.
• Graph structure should be navigated, not embedded away.
• LLMs can act as retrieval controllers, not just rerankers.

For enterprises building RAG systems over complex internal knowledge graphs—legal entities, supply chains, biomedical ontologies—this approach is far more aligned with reality than one-shot retrieval.

Conclusion — Small tools, large leverage

ARK doesn’t add more models, more embeddings, or more heuristics. It adds agency.

By exposing just two well-chosen operations and letting the model decide how to combine them, the system resolves a decade-old retrieval tension in a surprisingly clean way.

Sometimes, progress isn’t about inventing new machinery—it’s about finally letting the system choose how to use the machinery it already has.

Cognaptus: Automate the Present, Incubate the Future.