Beyond Search: RAG’s Awakening to Enterprise Spreadsheets

TL;DR for operators

Most enterprise RAG failures do not begin at the chatbot. They begin earlier, when the retrieval system slices policy manuals into arbitrary chunks, flattens tables into textual porridge, ignores metadata, retrieves semantically similar but operationally wrong passages, and then asks an LLM to look confident. Naturally, the LLM obliges. It has excellent manners.

The paper behind this article, “Advancing Retrieval-Augmented Generation for Structured Enterprise and Internal Data,” proposes a more enterprise-aware RAG pipeline: semantic chunking for text, row-level indexing for tables, hybrid dense-plus-BM25 retrieval, metadata filtering, cross-encoder reranking, LLM-based query reformulation, conversational memory, and feedback-triggered re-retrieval.¹

The reported gains are meaningful. Compared with naive RAG, the advanced pipeline improves Precision@5 from 75% to 90%, Recall@5 from 74% to 87%, and MRR from 0.69 to 0.85. Compared with direct LLM generation, the gap is larger: Precision@5 rises from 62% to 90%, Recall@5 from 58% to 87%, and MRR from 0.60 to 0.85. Human or evaluator ratings also rise sharply: faithfulness from 3.0 to 4.6 versus naive RAG, completeness from 2.5 to 4.2, and relevance from 3.2 to 4.5 on a five-point scale.

For operators, the lesson is not “buy a vector database.” That was never a strategy; it was a line item. The lesson is that retrieval quality depends on preserving business structure: rows, columns, policy sections, entities, dates, departments, confidentiality tags, and query intent. This matters for HR policy search, finance reports, compliance review, procurement records, audit support, and any internal knowledge system where the correct answer may live in one row of one table inside one poorly formatted PDF.

The boundary is also clear. The paper is not proof that a universal enterprise RAG system is production-ready. Its evaluation uses mostly public HR-policy and corporate-report-like materials, with simulated enterprise metadata. It acknowledges static indexing, difficulty with complex or nested tables, dependence on explicit user feedback, heuristic query reformulation, and scaling risks around latency and memory. The paper is useful because it shows the shape of a better pipeline. It does not remove the engineering bill. Annoying, but reality does like to be invoiced.

The spreadsheet problem is not a generation problem

Enterprise information is rarely stored as clean prose. It lives in HR policy PDFs, payroll tables, procurement sheets, internal reports, audit records, benefit matrices, onboarding manuals, vendor contracts, and departmental templates that have been copied forward since someone’s heroic Excel era in 2016.

A normal RAG system is comfortable with paragraphs. It embeds text chunks, retrieves passages similar to the user’s query, and gives those passages to a language model. That works reasonably well when the answer is contained in a narrative paragraph: “Employees are eligible for annual leave after six months.” It works less well when the answer depends on row-column relationships: employee type, tenure band, location, department, policy year, exception code, and whether the relevant footnote applies only to contractors in one jurisdiction.

That is the paper’s starting point. The problem is not that LLMs cannot speak business language. They can. Sometimes too fluently. The problem is that the retrieval layer often hands them damaged evidence.

Flatten a table into plain text and the model may see the words but lose the relationships. Chunk a policy by character count and a relevant exception may land in a different chunk from the eligibility rule it modifies. Use dense semantic retrieval alone and the system may retrieve passages that sound conceptually related while missing exact terms such as a department code, regulation label, employee grade, or product SKU. Skip reranking and the first retrieved item may be merely adjacent to the answer, which is a polite way of saying “wrong, but wearing the right shoes.”

The paper’s contribution is therefore best understood as a sequence of repairs. Each layer addresses a specific enterprise RAG failure.

The architecture is not revolutionary in the Hollywood sense. No robot arm bursts through the wall holding a spreadsheet. Its importance is more practical: it assembles known retrieval improvements into a pipeline that respects the formats enterprises actually use.

First repair: stop murdering tables

The most consequential mechanism is table handling. The paper treats tables not as decorative text blocks but as structured objects. It uses tools such as Camelot, with pdfplumber as fallback, and also discusses Azure Document Intelligence for extracting tables into structured forms. Tables are serialised into JSON with file names, row identifiers, column headers, cell values, and related metadata. The key move is row-level indexing: each row can become a retrievable unit.

That sounds small until one imagines the alternative. Suppose a user asks, “What is the reimbursement limit for Grade B employees in Region 2?” A naive RAG system that flattened the table might retrieve the whole reimbursement policy, or a nearby benefits section, or a semantically similar paragraph about travel claims. The model then has to reconstruct the table’s logic from a mangled sequence of words. This is less retrieval-augmented generation than retrieval-assisted archaeology.

Row-level indexing changes the retrieval target. The system can retrieve the row where Grade B, Region 2, and reimbursement limit co-exist as structured fields. This is not merely “more context.” It is better-shaped context.

The paper also distinguishes between table strategies. Storing entire tables as chunks can work for small tables, especially those with fewer than ten rows. But for larger tables it becomes inefficient and imprecise, because row-specific queries force the retriever to bring back too much irrelevant material. Row-level indexing solves a different problem: granularity. It lets the system retrieve the smallest useful unit without pretending the whole table is one thought.

This is where the paper’s title-level promise becomes operationally meaningful. Enterprise spreadsheets and spreadsheet-like tables are not just containers of text. They encode business rules through position. Meaning is in the row, the column, the header, the neighbouring cells, and sometimes the cursed merged cell. A RAG system that ignores that structure is not “general.” It is just underinformed with excellent branding.

Second repair: use dense retrieval for meaning, BM25 for exactness

The paper combines dense semantic retrieval using all-mpnet-base-v2 embeddings with sparse retrieval using BM25. The fusion weight reported in the paper is 0.6 for dense retrieval and 0.4 for sparse retrieval.

This matters because enterprise queries are often hybrids. They contain semantic intent and exact references at the same time.

A user might ask:

“Show the leave encashment rule for employees under the 2023 HR policy.”

The words “leave encashment rule” require semantic matching. The phrase “2023 HR policy” requires exact or near-exact filtering. Dense retrieval may capture the broad concept while missing the specific policy year. BM25 may catch the exact terms while missing semantically equivalent language. Hybrid retrieval exists because both failure modes are common, and enterprises usually do not forgive either.

The paper frames this as a retrieval balance. Dense retrieval supplies conceptual reach; BM25 supplies lexical discipline. The fusion score is not magic. It is an empirical engineering choice. But the broader point is durable: internal knowledge systems need both semantic flexibility and exact textual anchoring.

That distinction is especially important in regulated or quasi-regulated workflows. In HR, legal, finance, and compliance, an answer that is “basically about the right topic” may still be operationally useless. A procurement policy from the wrong department is not a near miss. It is a wrong answer wearing a lanyard.

Third repair: rerank the candidates before the LLM sees them

First-stage retrieval is usually broad. It is designed to avoid missing relevant material. That means it often returns plausible but noisy candidates. The paper adds a cross-encoder reranking layer using an MS MARCO fine-tuned MiniLM model.

The mechanism is straightforward. A dense retriever maps queries and documents separately into vectors, then compares them. This is efficient, but approximate. A cross-encoder evaluates the query and candidate chunk together, allowing richer interaction between the two. It is slower but more precise, so it is usually applied only to the top candidates from the first retrieval stage.

This is not just a technical flourish. In enterprise RAG, the order of retrieved evidence matters because the generator often gives more attention to top-ranked context. If the first item is tangential, the model may answer from the wrong passage. If the correct row or clause appears lower down, the system may have technically retrieved the answer while still failing the user. That is the sort of success only a benchmark spreadsheet could love.

Reranking makes the retrieval layer less gullible. It asks not only “Is this chunk nearby in embedding space?” but “Does this chunk actually answer this query?” That is a more expensive question. It is also the question the business user thought the system was asking all along.

Fourth repair: metadata is not decoration

The paper enriches chunks with metadata using spaCy named entity recognition for locations, dates, and organisational names. It also describes metadata such as document type, department, and confidentiality level, while noting that some of this metadata is simulated for experimentation and could be replaced with real enterprise metadata.

That caveat matters. Simulated metadata is useful for testing pipeline behaviour, but it is not the same as surviving a real enterprise permission model, inconsistent departmental tagging, outdated SharePoint folders, and the mysterious document labelled “final_final_revised_USE_THIS_v3.pdf.” Still, the direction is right.

Metadata turns retrieval from “find similar text” into “find similar text within the right business context.” A finance policy from 2022, an HR policy from 2024, and a legal clause in a vendor contract may all discuss reimbursement, eligibility, or approval authority. Without metadata, the system must infer too much from text alone. With metadata, retrieval can be constrained by department, year, document type, confidentiality, region, or entity.

This is one of the paper’s most business-relevant ideas. Enterprises rarely need an answer from “all documents.” They need an answer from the authorised, current, applicable subset of documents. That difference separates a useful internal assistant from a liability wearing a chatbot icon.

Fifth repair: vague questions need query reformulation, not moral judgement

Users do not always ask clean questions. They ask, “What’s our policy on remote work?” when there are different policies for full-time employees, contractors, jurisdictions, and approval levels. They ask follow-ups like “What about managers?” and expect the system to remember what “what” referred to. Quite unreasonable of them to behave like humans.

The paper uses LLaMA and Mistral models on ChatGroq for query rewriting and expansion. It also keeps up to ten recent interactions in conversational memory. When users provide negative feedback, the system can automatically reformulate the query and retrieve again.

This is a useful design move because it treats retrieval as an iterative process. The first query may be incomplete. The first answer may expose ambiguity. Feedback can become a signal for another retrieval attempt rather than a dead end.

Still, query reformulation is not free intelligence. The paper itself notes that heuristic LLM-based expansion can misinterpret intent or broaden the query too much. In business settings, that can be dangerous. A broader query may retrieve more documents while becoming less faithful to the user’s actual need. The more sensitive the workflow, the more reformulation should involve clarification rather than silent expansion.

For low-risk internal knowledge search, automatic rewriting can improve convenience. For legal, financial, medical, or compliance-heavy workflows, the system should sometimes ask: “Do you mean the 2024 policy for full-time employees in Manila, or the global remote-work guideline?” Yes, this is less magical. It is also less likely to produce confident nonsense.

What the reported results actually support

The paper reports a comparison across three systems: direct LLM generation, naive RAG, and the advanced RAG pipeline. The advanced system performs best across all listed metrics.

The magnitude is large enough to matter. A Precision@5 gain from 75% to 90% means fewer irrelevant chunks near the top. A Recall@5 gain from 74% to 87% means fewer missing pieces. An MRR increase from 0.69 to 0.85 means the first useful result appears sooner, which matters because downstream generation is highly sensitive to context ordering.

The qualitative gains are also important, especially faithfulness. In enterprise deployments, hallucination is often discussed as a model problem. The paper’s results suggest a more practical diagnosis: bad retrieval creates the conditions for hallucination. If the answer generator receives incomplete, misranked, or structurally broken evidence, faithfulness suffers. Improve the evidence and the model suddenly becomes better behaved. Miraculous? Not really. Less garbage went in.

But these results should be interpreted as main evidence for the pipeline, not as a clean ablation proving the isolated value of every component. The paper describes progressive experimentation and notes that hybrid retrieval, reranking, query refinement, and table-aware processing contribute incremental improvements. It does not provide a full component-by-component ablation table separating the exact marginal gain of each mechanism. That distinction matters for implementation planning.

A CTO should not read the table and conclude that every enterprise RAG system must adopt this exact stack, exact embedding model, exact fusion weight, exact reranker, and exact memory length. The evidence supports the architecture pattern: preserve structure, combine semantic and lexical retrieval, rerank, use metadata, and refine queries. It does not prove that this is the globally optimal configuration.

The paper’s tests are doing different jobs

The experimental components in the paper should not be read as one undifferentiated benchmark blob. They serve different purposes.

This distinction is important because enterprise AI buyers often ask one overloaded question: “Does it work?” The better question is: “Which part works, under which data conditions, and at what operational cost?” Less catchy. More useful.

The business value is retrieval discipline, not chatbot sparkle

The paper’s practical relevance is strongest in workflows where the answer must be grounded in internal evidence and where that evidence is partly structured.

Consider HR policy search. A user may need the rule for a specific employee class, region, contract type, or policy year. A naive system may retrieve the general leave policy and miss the exception table. A row-aware system has a better chance of finding the exact applicable line.

Consider finance and audit support. A user may ask for line items above a threshold, transactions from a specific period, or policy clauses governing approval authority. Dense retrieval alone may retrieve semantically relevant accounting text while missing exact numeric or categorical constraints. Hybrid retrieval and metadata filtering become operationally important.

Consider compliance review. A legal or risk team may ask for clauses tied to a regulation, jurisdiction, vendor category, or effective date. In that setting, metadata is not convenience. It is control.

Cognaptus’ business inference is therefore narrow but useful: enterprise RAG projects should be budgeted and evaluated around retrieval architecture, not just model selection. The LLM is the visible surface. The value often comes from the plumbing nobody wants to put in the demo.

A practical deployment checklist would look like this:

This is not a glamorous checklist. That is how one knows it is probably useful.

Where the paper is strongest

The paper is strongest as a systems blueprint. It does not claim a new foundation model, a new embedding theory, or a dramatic mathematical breakthrough. Instead, it composes existing tools into an enterprise-oriented retrieval pipeline.

That is valuable because many failed internal AI deployments fail by composition. Each individual component may look reasonable: PDF parser, chunker, embeddings, vector database, LLM, chat interface. The system still fails because the interfaces between those components destroy structure, discard metadata, or retrieve evidence at the wrong granularity.

The paper’s mechanism-first contribution is to show where those seams matter:

1. Preprocessing decides what evidence can exist. If table structure is lost during extraction, retrieval cannot recover it later.
2. Indexing decides the unit of answerability. A whole document, a paragraph, a section, and a table row are not interchangeable.
3. Retrieval decides what the model is allowed to know. Dense-only retrieval is too soft for exact enterprise references.
4. Reranking decides what the model sees first. Ordering is not cosmetic; it shapes generation.
5. Feedback decides whether failure is terminal. A system that can re-retrieve after negative feedback has a path to correction.
6. Memory decides whether conversations remain coherent. Follow-up questions are normal; stateless retrieval treats them as suspiciously brief new mysteries.

This is the paper’s real usefulness. It gives operators a vocabulary for diagnosing RAG failures without blaming everything on “the model.”

Where the evidence should not be stretched

The paper’s limitations are not decorative. They define where the business interpretation should stop.

First, the evaluation corpus is not a complete mirror of enterprise reality. The paper uses predominantly public HR policies and corporate-report-like materials, along with supplementary public datasets. That is a sensible experimental base, but real enterprises add messy permissions, stale duplicates, multilingual content, scanned images, inconsistent templates, confidential annexes, and users who ask questions like they are texting a colleague during lunch.

Second, metadata is partly simulated. In production, metadata quality is often the product of governance, document management discipline, and access-control integration. If those systems are weak, the RAG pipeline inherits the weakness. The chatbot will not magically create a records-management culture. Tragic, yes.

Third, the paper acknowledges static indexing. If document updates require full reindexing, the system may lag behind the business. For HR, compliance, pricing, procurement, and policy operations, stale retrieval can be worse than no retrieval because it carries the aesthetic of authority.

Fourth, complex tables remain difficult. Row-level indexing helps, but deeply nested tables, hierarchical headers, merged cells, multi-page tables, and context-dependent footnotes remain hard. Many enterprise spreadsheets are less “table” than “ritual object.” A production system needs robust extraction tests before trusting answers from them.

Fifth, the feedback mechanism depends on explicit user signals such as thumbs up or thumbs down. In real settings, users often do not provide feedback unless something is very wrong. Passive signals may help, but they introduce their own ambiguity. Did the user click the document because it was useful, confusing, or the only thing on screen?

Finally, scaling is unresolved. Hybrid retrieval, metadata filtering, reranking, query reformulation, and multiple indices all add system complexity. At moderate size, this may be manageable. At enterprise scale, latency, memory, cost, access control, and monitoring become central design constraints.

The paper points toward the right architecture. It does not make the operational trade-offs disappear. Anyone claiming otherwise is probably selling implementation services, which is one of nature’s more predictable phenomena.

What enterprise teams should do differently after reading this

The practical move is to stop treating RAG evaluation as a generic chatbot benchmark. Enterprise RAG should be tested against the data structures that actually carry business meaning.

For a pilot, teams should build a task set around real internal questions:

• row-specific table queries;
• policy exception queries;
• date-bounded retrieval;
• department-filtered retrieval;
• exact code or entity lookup;
• multi-turn follow-ups;
• ambiguous prompts requiring clarification;
• queries where the correct answer is “not found”;
• queries involving updated or superseded documents.

Then they should evaluate not only whether the final answer sounds good, but whether the retrieved evidence was correct, current, authorised, and sufficiently granular.

A useful internal benchmark should separate retrieval failure from generation failure. If the correct evidence was not retrieved, the model is not the primary culprit. If the evidence was retrieved but ignored, the prompting or generation layer needs work. If the evidence was retrieved but stale or unauthorised, the governance layer is broken. This kind of diagnosis is less exciting than a leaderboard, but it prevents expensive confusion.

The paper’s architecture suggests a layered maturity path:

Most organisations are somewhere between Basic RAG and “we uploaded PDFs and hoped.” The paper is useful because it shows what the next few levels look like.

The quiet shift: from search over documents to reasoning over business records

The title of this article says “Beyond Search,” but the point is not that search disappears. Search becomes more disciplined. Enterprise AI does not need a chatbot that can poetically summarise irrelevant context. It needs a retrieval system that can locate the right clause, row, entity, and document version before the model starts speaking.

That is the shift this paper captures. RAG is moving from document search toward structured knowledge access. The LLM remains important, but it is no longer the whole story. In fact, the more serious the business workflow, the more the retrieval substrate matters.

For Cognaptus readers, the strategic lesson is simple: enterprise AI advantage will not come from plugging the same model into the same vector store as everyone else. It will come from the boring, defensible work of structuring internal knowledge so machines can retrieve it without flattening the business into mush.

That means table extraction. Metadata governance. Hybrid search. Reranking. Query clarification. Incremental indexing. Evaluation sets built from real work. Permission-aware retrieval. Monitoring. Latency budgets. The unglamorous machinery of reliability.

The paper does not solve every one of these problems. But it correctly identifies the direction: RAG systems must wake up to the fact that enterprise knowledge is not just prose. It is structured, conditional, versioned, permissioned, and often trapped inside spreadsheets pretending to be PDFs.

Once that is understood, the future of enterprise RAG looks less like “chat with your documents” and more like “retrieve the correct business state.” A less catchy phrase, admittedly. Also a better product.

Cognaptus: Automate the Present, Incubate the Future.