A reliable enterprise knowledge system is not primarily a prompt or model-selection problem. Prioritise permission-aware retrieval, well-maintained source content, and an evaluation set based on real questions before tuning generation. Treat citations as evidence to review; a cited answer can still be wrong.
Start with the knowledge boundary
Retrieval-augmented generation, or RAG, retrieves relevant content and supplies it to a language model when answering. It is useful when answers must reflect changing organisational material without retraining the model. It does not make source material accurate, current, complete, or authorised.
Define the system’s knowledge boundary before choosing a retrieval design:
- Which repositories are included?
- Which document types and versions are authoritative?
- Which users may retrieve each item?
- How quickly must changes appear?
- What must the system do when evidence is missing or contradictory?
- Which questions require a human decision rather than an answer?
Enterprise content is heterogeneous: policies, records, intranet pages, spreadsheets, scanned PDFs and databases all have different structures and freshness requirements. Microsoft’s Azure AI Search guidance, last updated 4 August 2026, identifies query understanding, multi-source access, token limits, response time and security as separate RAG challenges. Its documented security approaches include document-level trimming, permission metadata and query-time filters. These describe what the platform can do; whether a particular deployment is correctly secured is separate work. Microsoft’s Azure AI Search RAG overview, last updated 4 August 2026
Make authorisation part of retrieval
A permission check after generation is too late. Unauthorised text may already have entered the prompt, logs, caches or model context. The retrieval layer should enforce the user’s effective permissions before content is passed to the model.
Recommended controls include:
- Preserve source-system identifiers, ownership, classification and access-control metadata during ingestion.
- Apply security trimming at query time, not only when documents are indexed.
- Test direct access, inherited access, revoked access, group membership changes and confidential-content edge cases.
- Include negative tests designed to make the system reveal documents the user should not see.
- Treat permission changes and document deletion as freshness events requiring measurable propagation times.
A search index is a derived copy. It can become more permissive than the source if permissions are flattened incorrectly, stale, or applied only at the application interface, so an organisation still needs its own access-control testing.
For New Zealand organisations handling personal information, the Office of the Privacy Commissioner’s guidance, dated 21 September 2023, says the Privacy Act applies across the stages of using AI, including user input, responses and actions based on those responses. It recommends a Privacy Impact Assessment before use and regular updates as risks change. Office of the Privacy Commissioner guidance on AI and the Information Privacy Principles, 21 September 2023
Improve retrieval before changing the model
Retrieval quality depends heavily on document preparation. Preserve headings, tables, page references, dates, document status and links to the canonical source. Remove duplicate versions where possible, or make version precedence explicit. Chunk documents around meaningful sections rather than using only fixed character lengths.
Use more than one retrieval signal for enterprise content:
- Keyword search handles exact policy names, product codes and legal phrases.
- Vector search handles related concepts and terminology variation.
- Hybrid retrieval combines both.
- Reranking can improve ordering after a broader candidate search.
- Metadata filters can constrain results by business unit, jurisdiction, document status or effective date.
The right design depends on the question set. EKRAG, published in May 2025, evaluated enterprise-style questions over corporate documents and assessed correctness, relevance and faithfulness. Its authors reported that some commonly proposed retrieval strategies did not consistently improve enterprise retrieval accuracy, which is a reason to benchmark alternatives rather than assume that a fashionable technique will help. EKRAG: Benchmark RAG for Enterprise Knowledge Question Answering, May 2025
Recent benchmarks also make structured content harder to ignore. T²-RAGBench, published in March 2026, contains 23,088 text-and-table question-context-answer triples and found hybrid BM25 effective on its tested data, while state-of-the-art systems still found the benchmark challenging. ARQA, also published in March 2026, found strong factual retrieval but continuing weaknesses in grounded arithmetic and causal reasoning over annual reports. These are benchmark results, not production evidence, but they support testing tables, calculations and cross-document reasoning explicitly. T²-RAGBench, March 2026 ARQA, March 2026
Anthropic’s contextual retrieval article, published 19 September 2024, describes a provider experiment in which contextual information was added to chunks before indexing, with reported improvements on its tested datasets. The experiment used the provider’s selected tasks, corpora and models. Treat the gains as a hypothesis to test on your own corpus. Anthropic’s contextual retrieval article, 19 September 2024
Design for uncertainty and source conflict
A grounded answer can still be wrong. The retriever may miss the decisive passage, return an obsolete policy, combine incompatible documents, or retrieve a plausible but irrelevant section. The model may then produce a fluent answer that overstates what the evidence supports.
Set explicit response behaviours:
- Answer only when the retrieved evidence meets a defined sufficiency threshold.
- Say when no adequate source was found.
- Show source title, date and location, not just a generic citation.
- Distinguish requirements, recommendations, examples and historical material.
- Surface conflicts instead of silently selecting one document.
- Route high-impact questions to a named owner or review process.
Grounding features differ by provider. Google’s current documentation, last updated 1 October 2026, describes grounding as connecting model output to verifiable sources and supports sources including search, enterprise data and third-party systems. Microsoft’s Azure AI Search RAG overview, last updated 4 August 2026, documents retrieval pipelines that return citations and query metadata. These pages describe product behaviour and supported configurations; independent validation of answer quality, security and fit is separate work. Google Cloud’s grounding overview, last updated 1 October 2026
Evaluate retrieval and answers separately
A useful evaluation set should contain real or carefully redacted questions, including ambiguous wording, terminology differences, multi-document questions, structured tables, outdated-content traps, permission boundaries and questions with no answer in the corpus.
Measure at least four layers:
- Retrieval recall: Did the system retrieve the evidence needed to answer?
- Retrieval precision: Were the returned passages relevant and sufficiently focused?
- Answer faithfulness: Are claims supported by the retrieved evidence?
- Answer usefulness: Is the answer complete, understandable and appropriate for the user’s task?
Also measure abstention quality, citation correctness, permission leakage, latency and freshness. Do not collapse these into one score; a system can improve answer relevance while worsening security or freshness.
Measure adoption through task evidence rather than usage volume alone: completed tasks, repeat questions, correction and escalation rates, and whether citations are opened. High query volume may mean the system is useful, or that users cannot find a reliable answer.
ARES, published in June 2024, evaluated context relevance, answer faithfulness and answer relevance using automated judges calibrated with human annotations. NIST’s LARA publication, dated 13 July 2025, similarly describes using language models to prioritise human relevance assessments rather than replacing them entirely. These approaches can reduce annotation effort, but automated judges should be sampled against human review, especially for specialised terminology and high-consequence content. ARES evaluation framework, June 2024 NIST’s LLM-Assisted Relevance Assessments, 13 July 2025
Manage freshness as an operating capability
Freshness is more than re-indexing on a schedule. Define service expectations for:
- Time from source change to searchable index update.
- Time from permission change to retrieval enforcement.
- Handling of deleted, superseded and withdrawn documents.
- Detection of failed ingestion jobs.
- Identification of stale answers and citations.
Store document effective dates and version status as retrievable metadata, and prefer canonical links and source-system timestamps over copied publication dates. Fast-changing material may need a shorter indexing interval or retrieval directly from the source.
Apply a staged decision test
A sensible pilot should pass these gates before broader release:
- Content readiness: authoritative sources, owners, versions and retention rules are known.
- Permission readiness: access-control tests cover both allowed and denied cases.
- Retrieval readiness: the system retrieves evidence for representative question types, including structured and multi-document questions.
- Answer readiness: responses distinguish evidence from inference and abstain when necessary.
- Operational readiness: freshness, logging, incident response and evaluation ownership are defined.
- Privacy readiness: personal-information handling has been assessed for the proposed sources and workflow.
If the corpus is poorly governed, RAG will surface the confusion faster than it resolves it. If questions require reasoning over structured records, calculations or undocumented business context, a document-only design may be insufficient. If the cost of a wrong answer is high, retrieval should support human review rather than replace it.
What we are watching
Research is moving towards more realistic enterprise benchmarks, claim-level faithfulness checks, multimodal documents, structured reasoning and comparisons between retrieval and long-context approaches. The 2025 Deep Search benchmark extends enterprise evaluation across documents, meeting transcripts, Slack messages, GitHub and URLs, but its corpus is synthetically generated, so its results do not establish production performance. Benchmarking Deep Search over Heterogeneous Enterprise Data, November 2025
These results remain sensitive to corpus design, query type, model, retriever and evaluation method. We are watching whether independent evaluations establish durable relationships between benchmark scores and production outcomes. We are also watching permission-aware retrieval under changing group membership, document deletion and cross-repository inheritance. Finally, organisations should watch for evidence that adoption measures reflect better decisions and completed work, rather than novelty, confidence or answer volume.