RAG: From Retrieved Documents to Supported Answers
Deterministic offline demonstration, not a language model. The tiny local corpus, fixed cosine retriever, preauthored relevance labels, and scripted answer simulator let you identify each stage independently.
Select one experiment and predict the failure
The budget counts words from the corpus using the lab tokenizer, not actual LLM tokens. Packing stops when the next whole ranked passage does not fit. Changing k cannot add documents absent from the corpus.
Experiment 1 — Retrieval and ranking
Inspect term-frequency cosine scores and independently specified gold evidence. A high-ranked irrelevant passage can push needed evidence below top-k.
Experiment 2 — What reaches the generator?
Top-k selection and context packing are different operations. The system packs whole passages in ranked order and stops when one no longer fits.
Experiment 3 — Evidence support and failure diagnosis
This scripted teaching simulator returns a fixed answer only when all designer-labeled documents are present, or illustrates an unsupported claim or abstention. It does not infer textual entailment or establish truth in the world.
What a live RAG system would additionally need
Source recency checks, quote-level support assessment, conflicting-evidence handling, permission checks, robust prompt-injection boundaries, and independent answer-correctness evaluation. Document IDs in this toy lab are only a controlled evidence proxy.