Five Answers, One Bad Retrieval: When RAG Agreement Misleads
TL;DR for operators A production RAG system returns the same answer five times. The operator still has to decide whether to release it, investigate it, or send it for review. Repeated agreement is useful evidence that generation is stable, but it does not show that the system retrieved the right material. All five answers may have been generated from the same empty, incomplete, or incorrect context. In that case, repeated sampling does not independently test the answer; it repeatedly tests the decoder under one defective retrieval state. A wrong answer that remains effectively unchanged across samples is a silent error. ...