How to Reduce Hallucinations in RAG Systems

Most RAG hallucinations are retrieval failures, not model failures

To reduce hallucinations in a RAG system, fix retrieval first: most confident-but-wrong answers happen because the right passage never reached the model, not because the model invented things from nothing. When the retriever returns three plausible-but-irrelevant chunks, the language model dutifully summarises them, and the result reads convincingly false. Improve what the model is given and the majority of hallucinations disappear before you ever touch the prompt — which is why retrieval quality, not model choice, is where this work begins for nearly every team.

The instinct when a RAG system gives a wrong answer is to blame the model or reach for a bigger one. That instinct is usually misplaced and expensive. Here are the techniques that actually move the needle, presented roughly in order of impact so you can spend your effort where it pays off most.

Step 1: Improve retrieval quality

If retrieval is weak, nothing downstream can save you, so this is the highest-leverage place to invest your time. A model can only reason over what it is handed, and handing it the wrong passages guarantees a wrong answer no matter how good the model is.

• Use hybrid search — combine vector similarity with keyword search so exact tokens like names and error codes are not missed by embeddings alone.
• Add a reranker that re-scores the top candidates and reorders them by true relevance; this is often the single biggest quality jump in a struggling system.
• Fix chunking — structure-aware chunks with the document title prepended retrieve far more reliably than blind character splits that shred context.

Step 2: Ground the model and demand citations

Instruct the model explicitly to answer only from the supplied context, and require it to cite which passage each claim came from. Citations do double duty: they keep the model honest by anchoring it to the source, and they let a reader verify the answer instantly instead of trusting it blindly.

An answer with a source a reader can click is a research assistant. An answer without one is a black box you are asked to trust on faith, which is exactly how trust erodes.

Step 3: Give the model permission to refuse

A model that must always answer will always find something to say, even when the context does not support it — that is the behaviour to design out. Detect when retrieval is weak and let the system decline gracefully rather than fabricating a confident response.

• Set a relevance threshold on the top retrieval score and short-circuit to "I could not find this in the documentation" below it.
• Tell the model in the prompt that "I don't know" is a correct and preferred answer when the context is insufficient.
• A calibrated refusal builds more trust than a confident guess that turns out to be wrong, because users forgive "I don't know" far more readily than a fabricated fact.

Step 4: Verify answers against the source

For high-stakes use, add a checking step after generation. Once the model answers, verify that its claims are actually supported by the retrieved passages — a second pass that flags or filters unsupported statements before they ever reach the user.

• Cross-check each claim against the cited context and drop the ones that are not grounded in it.
• Prefer extractive answers — quoting the source directly — over free paraphrase where accuracy is critical.
• Keep context focused; flooding the prompt with marginally relevant chunks invites the model to wander off the source.

Step 5: Measure hallucinations so you can reduce them

You cannot fix what you do not measure, and "it seems better" is not measurement. Build an evaluation set from real questions paired with correct, sourced answers, and score every change against it — including how often the system hallucinates and how often it correctly refuses. Mine your production logs for the failures, because the questions your system gets wrong today are precisely the test cases that will tell you whether tomorrow's change actually helped or merely felt like it did. This discipline is unglamorous, and it is exactly what separates a RAG system that degrades silently from one that improves on purpose.

Prefer it built and managed for you?

BSH Technologies builds production RAG that stays grounded — hybrid retrieval, reranking, citations, calibrated refusal, and evaluation harnesses that track hallucination rates as a first-class metric. If your RAG system answers confidently but not always correctly, talk to BSH Technologies or explore our AI & automation services.