Econometrics bridge: Nearest-neighbor prediction, similarity weighting, information selection, and measurement quality
Estimated time: 105 min
Lab: Open browser lab
Code: Python - R

From nearest-neighbor estimation to retrieval

An econometrician may already know how to predict from observations similar to a new case. In nearest-neighbor or kernel estimation,

\[ \hat m(x)=\sum_i w_i(x)y_i, \qquad \sum_i w_i(x)=1. \]

A retrieval system likewise scores candidates relative to a query and selects relevant items. But retrieving a document is not averaging an outcome: the selected text becomes evidence supplied to another model, which may use it correctly, ignore it, or distort it. Similarity is not the same as factual reliability or causal relevance.

Module 10 separated pretrained parameters from inputs supplied at inference. This module asks what happens when an external collection determines which inputs the generator receives. Unlike fine-tuning, retrieval can update the information presented without changing the generator’s weights.

Parameters and external evidence are different information sources

A standalone model generates using an input \(x\) and learned parameters \(\theta\):

\[ p_\theta(y\mid x). \]

A retrieval-augmented system additionally has a document collection \(D\), a retrieval rule, and a context-building policy. The corpus may be refreshed independently of \(\theta\). This does not ensure that the latest document is indexed, selected, or used faithfully.

Two mathematically distinct RAG constructions

In a latent-document formulation, retrieval can be represented by a distribution \(p_\eta(z\mid x)\) and document-conditioned generation by \(p_\theta(y\mid x,z)\):

\[ p(y\mid x)=\sum_z p_\eta(z\mid x)\,p_\theta(y\mid x,z). \]

This expresses marginalization over possible documents. A common retrieve-then-concatenate pipeline instead constructs

\[ C_k(x;D,B)=\operatorname{Pack}_B\!\left(\operatorname{TopK}(x,D)\right), \qquad p_\theta(y\mid x,C_k), \]

where \(B\) is a context-token budget and \(\operatorname{Pack}_B\) decides which chunks fit. Conditioning on one packed context is not equivalent to summing over document alternatives. This course lab models the second construction, not the original learned marginalization model.

What retrieval can learn—and what this lab fixes

Lexical retrieval compares observed terms (for example, bag-of-words counts, TF-IDF, or BM25). Dense retrieval maps queries and passages into learned embeddings and uses a similarity score. Hybrid systems can combine the two; exact identifiers may suit lexical matching, while paraphrases may benefit from learned representations.

The browser lab uses a deliberately simple fixed term-frequency cosine retriever:

\[ s(q,d)=\frac{v(q)^\top v(d)}{\|v(q)\|_2\,\|v(d)\|_2}. \]

It lowercases and splits text on non-alphanumeric boundaries. No embedding model, ranking parameters, or generator weights are fitted in the exercise. Relevance labels are provided as a teaching benchmark, not inferred from word overlap. This is the same difference between a fitted predictive score and an independently defined evaluation target seen in Module 06.

Where RAG can fail

A reliable diagnosis starts with the earliest stage missing the necessary evidence.

Stage Question Typical failure
Corpus Is the needed evidence stored at all? The current rate or policy is absent
Retrieval Was needed evidence ranked inside top-k? A multi-document question retrieves only one required passage
Context construction Did the selected evidence actually reach the generator? A token budget excludes a lower-ranked passage
Generation Did the answer use the evidence faithfully? Unsupported claim, source misuse, or unnecessary abstention

The lab packs whole chunks in ranking order, stopping when the next chunk will not fit within the stated word-token budget. Production systems may split, compress, rerank, or pack contexts differently. This simple rule makes the information boundary observable.

A source can also be stale, false, contradictory, or adversarial even when all four computational stages succeed. Retrieved text is untrusted data, not a new instruction authority. Provenance and dates matter independently of similarity scores.

Evaluation: do not collapse distinct claims

With gold relevant passages \(G(q)\) and retrieved top-k set \(R_k(q)\),

\[ \operatorname{Recall@k}=\frac{|R_k\cap G|}{|G|}, \qquad \operatorname{Precision@k}=\frac{|R_k\cap G|}{k}, \qquad |G|>0. \]

The context also needs its own evidence coverage: how many required passages survive packing? If the corpus contains no answer, recall@k is not defined, and abstention may be correct. A compound question may require all members of \(G\), not just one.

Keep three claims separate:

  1. Retrieval relevance: were appropriate passages selected?
  2. Evidence support (grounding): can the selected passages justify the answer’s claims?
  3. Answer correctness: is the answer true and responsive to the actual question, including source freshness?

A faithful answer quoting outdated data can still be wrong. A correct answer supplied from prior parametric knowledge may still lack retrieved support. A larger top-k can improve evidence coverage while increasing irrelevant text, context cost, and conflicting claims.

Grounded generation is a separate component

The lab’s answer stage is an explicitly scripted simulator, not a language model. It knows a small set of preauthored facts and their required document IDs. In evidence-constrained mode it returns an answer only if all required documents reached context; otherwise it abstains. An intentionally unsupported mode demonstrates that retrieval success does not force a trustworthy answer.

Because the expected facts are fixed in the lab, a matching document ID is only a controlled proxy for claim-level support. A real evaluation must check cited passages, entailment, source authority, and real-world correctness separately.

Econometrics-to-AI translation

Apply Module 0’s five questions to this system:

Question RAG answer
What is the familiar statistical object? Nearest-neighbor similarity, information selection, and measurement quality
What became learnable? In some systems, retriever embeddings/rankers and generator parameters; nothing is trained in this lab
What objective is optimized? A learned system may use retrieval and generation losses; this lab computes fixed similarity scores
What generates the evidence? External passages and query-specific selected contexts, not newly independent economic observations
Where does the analogy break? Similarity is not truth; presence is not use; document support is not causal or real-world correctness

Interactive browser lab

Run three experiments with the same small corpus and relevance labels:

  1. Retrieval: select a straightforward or compound query, vary top-k, and examine scores, recall@k, and precision@k. Some compound queries require multiple passages.
  2. Context: limit the token budget and inspect which ranked chunks really arrive in the generator context. Distinguish a missed retrieval from evidence lost to packing.
  3. Grounding: switch between evidence-constrained, unsupported, and abstaining answer simulations. Identify the earliest evidence failure and any additional generation defect. Also try a request for a current number that is missing from the corpus.

All calculations run offline, deterministically, without API calls or downloaded models.

Python and R lab

Reproduce the shared corpus, tokenizer, cosine ranking, whole-chunk context packing, and scripted answer checks. Assert tie ordering, probability-free similarity scores, evidence recall, context coverage, and abstention when an answer is unavailable. Compare a low-k compound query with one that has sufficient retrieval and budget, then deliberately choose an unsupported answer.

Practice:

  1. Explain why the latent-document mixture expression and retrieve-then-concatenate systems are not the same probability model.
  2. For a compound question requiring two passages, calculate recall@1, recall@4, and evidence coverage after packing.
  3. Explain a case in which top-k contains every required passage but context construction still loses evidence.
  4. Explain why a citation to a retrieved document does not automatically establish factual correctness.
  5. Distinguish corpus absence, retrieval failure, context failure, and generation failure for a given trace.
  6. Give an example of prompt injection inside retrieved text and explain why it has no authority to override the user’s instructions.

Learner output

Produce a short RAG failure report with query, gold evidence IDs (if any), ranking, chosen k, packed context IDs, retrieval and context metrics, answer status, and a diagnosis of the earliest missing-evidence stage. Explain separately whether the answer is supported and whether its truth has been independently established.

Sources and Further Reading

Sources

  • Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Foundational latent-document RAG formulation. https://arxiv.org/abs/2005.11401
  • Thakur, N. et al. (2021). BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. Retrieval evaluation across tasks and domains. https://arxiv.org/abs/2104.08663
  • Russell, S. and Norvig, P. Artificial Intelligence: A Modern Approach, 4th ed. Background on probabilistic inference and information retrieval.

Further reading