Measure the Chunk, Not the Container
TL;DR for operators A retrieval system can improve while its benchmark understates the improvement. A generated answer can remain consistent with its evidence while a lexical-overlap metric gives it a poor groundedness score. That is the central operational lesson from Morales, Hegde, and Syed’s urban-discourse system.1 Dense semantic retrieval beat TF-IDF across every reported held-out retrieval metric, including Precision@10 of 0.660 versus 0.545. But when the dense system’s actual retrieved chunks were rated directly, its Precision@10 was 0.94. The evaluation unit changed, and so did the apparent quality of the same retriever. ...