Vector Search

Enterprise search has a very old habit wearing a very modern jacket: it averages. A policy document becomes one vector. A runbook becomes one vector. A postmortem full of operational detail becomes one vector. Then a RAG system asks that one vector whether the document is relevant. This is convenient, fast, and usually defensible — until the relevant answer is a narrow paragraph hiding inside a large document. At that point, the retrieval system is no longer searching for evidence. It is asking a crowd to speak for the witness. ...

Search gets expensive quietly. At small scale, an embedding is just a vector. At product scale, it becomes rent: storage rent, memory rent, GPU rent, latency rent, and the recurring emotional tax of explaining why a semantic search feature needs yet another infrastructure budget. Dense embeddings made this bargain feel natural. More dimensions, more semantic capacity. More semantic capacity, better retrieval. Better retrieval, more invoices. Elegant, if one enjoys expensive inevitability. ...

TL;DR for operators Private examples are not harmless just because they sit inside a prompt rather than inside model weights. In-context learning lets teams adapt a general LLM by adding examples at inference time, which is convenient until those examples are medical notes, legal clauses, customer tickets, invoices, or internal decisions that should not be inferable from the model’s output. ...

TL;DR for operators Most enterprise RAG failures are not generation failures. They are retrieval-routing failures wearing a very convincing blazer. The paper behind this article proposes an open-source agentic hybrid RAG framework for scientific literature review: bibliographic metadata and citation relationships go into a Neo4j knowledge graph; full-text PDF chunks go into a FAISS vector store; an LLM-based agent decides whether a user’s question should be answered through GraphRAG or VectorRAG; a Mistral-based generator produces the final answer; DPO is used to improve grounding; and bootstrap resampling is used to report evaluation uncertainty.1 ...

TL;DR for operators Long-context windows are useful. They are also an expensive way to pretend that memory is just a bigger clipboard. The HEMA paper argues for a more operationally realistic design: keep a compressed summary of the conversation always visible, store detailed past exchanges outside the prompt, and retrieve only the details that matter for the current turn.1 That gives the model two different memory behaviours: continuity from Compact Memory and factual recall from Vector Memory. ...