All posts

How to improve RAG accuracy: diagnose retrieval, chunking, filters, and evaluation

October 25, 2026 · 11 min read

Separate missing evidence from generation errors before tuning a RAG system. Use an internal help desk to work through recall@k, chunk boundaries, hybrid retrieval, authorization filters, pgvector result shortfalls, and regression evaluation.

RAG accuracy is not improving? Separate retrieval from generation first

Retrieval-Augmented Generation searches for material related to a question and uses it as evidence for an answer. When an internal help desk gives incorrect instructions, prompt tuning alone cannot locate the fault. The source might be missing from the index, retrieval might miss it, context assembly might drop it, or generation might misread it.

Missing evidence means an answer grounded in that source cannot be guaranteed; it does not rule out a model answering correctly by chance. Inspect the chunks actually supplied to generation as well as retrieval results. Full-text logs can contain confidential data, so define a limited, authorized investigation scope.

Build an evaluation set pairing questions, evidence, and permissions

An illustrative set might contain 100 answerable questions, 20 with no answer in the corpus, and 20 asking about unauthorized material. Choose real proportions from workload frequency and failure impact. Record relevant chunk IDs, answer criteria, user and tenant context, and source version or effective date for each question.

Separate examples used repeatedly during development from final evaluation cases to reduce overfitting. Include more than paraphrases from one document: old versions, tables, cross-section explanations, and short questions matter too. Version labels when sources or permissions change.

Calculate recall@k consistently when several chunks are relevant

Use evidence chunks as the evaluation unit. For relevant set G and top-k results R, recall@k = |R ∩ G| / |G|. A macro average gives each question equal weight. Exclude empty-G questions from this average and evaluate them separately, such as by correct abstention rate.

Macro recall@5 is (1/2 + 1 + 1/3) / 3 = 11/18, approximately 61.1%. The share of questions finding at least one relevant item is instead 3/3, or 100%. A micro average pooling relevant-item counts is 3/6, or 50%. State the denominator and aggregation unit before comparing recall scores.

cases = [
    ({"A", "B"}, ["A", "X", "Y", "Z", "W"]),
    ({"C"}, ["C", "X", "Y", "Z", "W"]),
    ({"D", "E", "F"}, ["D", "X", "Y", "Z", "W"]),
]
k = 5
recalls = [len(gold & set(found[:k])) / len(gold)
           for gold, found in cases]
hits = [bool(gold & set(found[:k])) for gold, found in cases]
print(f"macro recall@5={sum(recalls) / len(recalls):.1%}")
print(f"hit@5={sum(hits) / len(hits):.1%}")

Improve chunking by finding missing context before changing length

A chunk saying only “submit within 30 days” can lose the application type and exceptions. Preserve useful headings, product names, and policy versions. Check that table extraction retains column labels and row meaning. If parsing dropped the answer entirely, another retrieval model cannot recover it from the index.

Useful token counts and overlaps vary by corpus. Rebuild and version the index when changing chunking instead of mixing old and new forms. Deduplication can help when near-identical passages fill the top results, but check that it does not remove important exceptions hidden in similar wording.

Missing exact identifiers? Compare hybrid retrieval and reranking

Embeddings may recognize paraphrases while failing to rank exact identifiers well. Compare adding keyword retrieval such as BM25 and fusing candidates. Reciprocal Rank Fusion combines ranks without directly adding incompatible raw scores. Evaluate its benefit on your own questions.

Retrieve a broader candidate set and consider reranking what reaches generation. Reordering cannot restore evidence absent from the candidates. Measure candidate-stage recall separately from recall in the final model context to identify losses in reranking or context limits.

Tenant filters leave too few results? Check where approximate search filters

If five requested results become one despite relevant data existing, the engine may collect nearby candidates before applying a tenant or authorization filter. Assume each of 40 candidates has a 1% chance of belonging to the tenant: expected survivors are 40 × 0.01 = 0.4. That is an assumption about candidate membership, not a guarantee from the tenant’s share of the full corpus.

The pgvector v0.8.0 documentation describes filtering after approximate index scanning and optional iterative scans that continue until enough matches or a scan limit. Compare search breadth, iterative scans, partitioning, and exhaustive search over the authorized set. Use exhaustive results under identical filters as an ANN baseline. Returning the requested count still does not prove business-relevant evidence was retrieved.

Correct evidence is present? Investigate generation and answer checks

The model can still mix current and obsolete sources, overlook conditions, or cite a passage that does not support its conclusion. Grade required facts separately and check whether each citation actually supports the claim. Define when insufficient evidence requires abstention or clarification.

An answer can faithfully reproduce a source that is wrong or outdated. JSON validation also does not measure content correctness. Check IDs and numbers programmatically where possible and use explicit rubrics for textual judgments. LLM judges can be biased by order and verbosity; compare their judgments with human labels.

Validate improvements with fixed questions, sources, and permissions

1. Fix a baseline and record source, index, embedding, search, prompt, model, and evaluator versions.
2. Group failures by stage and category, then select a frequent or high-impact cause.
3. Compare retrieval coverage and answer quality before and after using the same inputs and permissions.
4. Inspect categories such as exact codes, multiple sources, old versions, and unauthorized requests alongside the overall average.
5. Repeat generation and grading to understand variability and whether improvement is stable.
6. Measure retrieval, reranking, and generation latency and cost, then select a configuration meeting release criteria.

Increasing top-k, adding reranking, or using a larger model changes latency and cost too. Define release gates that can reject a higher overall score if disclosure or important procedural errors worsen. Add newly observed production failures to the evaluation set after removing personal data.

FAQ: do bigger chunks and more top-k results always improve accuracy?

• Are larger chunks better? They preserve context but may add irrelevant material. Compare based on missing conditions and the available input budget.
• Should top-k simply increase? Candidate recall can improve while truncation or noise increases before generation. Measure final context too.
• Should a larger model come first? It is an experiment when the right evidence arrives but is misread. Missing retrieval evidence also needs retrieval investigation.

Primary sources on retrieval, evaluation, and authorization

• Azure Architecture Center — RAG information-retrieval phase
• Azure Architecture Center — RAG end-to-end evaluation phase
• Anthropic — Contextual Retrieval: chunks, BM25, embeddings, and reranking
• pgvector v0.8.0 — Filtering and iterative index scans
• Google Cloud — Set up data source access control
• Zheng et al. — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena