Reducing LLM Hallucinations in Document Q&A
Retrieval alone does not stop a model from inventing answers. Four controls, answer from context, allow not found, quote the source and verify the quote, catch most of it, and a few failure types slip through.

Retrieval does not make a model honest. A preregistered 2024 study of AI legal research tools from LexisNexis and Thomson Reuters, products marketed as grounded in retrieval over case law, found that each hallucinated between 17% and 33% of the time. Those are products with large teams behind them. An internal document assistant built over a weekend has no reason to do better unless it adds controls of its own. The good news is that the controls that reduce LLM hallucinations in document Q&A are mostly plain engineering: a strict prompt, a structured output, and a few lines of code that check the model's work instead of trusting it.
This article covers four of them, in the order they are usually added, and then the failures none of them catch. It assumes a retrieval pipeline already exists; the architecture itself is laid out in private RAG over internal documents.
Three ways a document assistant gets it wrong
"Hallucination" covers several different faults, and each control targets a different one.
- The answer was never retrieved. The right passage is not among the ones handed to the model, so the model fills the gap from its training data or from a near-miss passage. In internal systems this is usually the first case to rule out.
- The answer was retrieved but misread. The passage is there, but the model swaps two figures, merges two clauses, or attributes a sentence to the wrong document.
- The answer is assembled, not stated. The model combines three passages into a conclusion none of them makes, such as adding up amounts from different contracts or inferring a policy from an example.
Keep these apart when reviewing bad answers. A fix for the first does little for the third.
Control 1: answer only from the retrieved text
The first control is an instruction: use the supplied passages and nothing else. Anthropic's prompting guide lists this as "external knowledge restriction", telling the model to use only information from provided documents and not its general knowledge. The idea applies to any model, local or hosted.
A workable system prompt for a local model is short and blunt:
You answer questions about company documents.
Use only the passages below. Each passage has an ID.
Do not use outside knowledge, even if you are confident.
If the passages do not contain the answer, reply exactly: NOT_FOUND
On its own this helps, but less than people hope. Models, smaller local ones especially, can follow the instruction loosely when the question is about something they "know", such as a general tax rule or a common contract term. Treat the instruction as the baseline that makes the next three controls possible, not as the fix.
Control 2: make "not found" a valid answer
A model asked a question tends to produce an answer. A 2025 paper by researchers at OpenAI and Georgia Tech argues that this habit is trained in, because most evaluations score a guess higher than an admission of uncertainty, so models learn to guess. A document assistant has to reverse that incentive explicitly.
Two things make "not found" work in practice.
- Give it a fixed form. A literal token such as
NOT_FOUND, or a field like"found": falsein JSON output, can be detected by code. A polite paragraph saying the documents are unclear cannot. - Show it in the user interface as a result, not an error. "No passage in the HR policies answers this. Searched: leave policy 2025, handbook section 4." tells staff where to look next and teaches them the system's limits.
Measure the rate. If almost no questions ever come back as not found, the model is probably still guessing on some of them. If a large share do, say a third, retrieval is likely missing passages, and the fix belongs in search or in how documents are chunked, not in the prompt.
Control 3: require quoted citations
The third control changes the output format. Instead of free text with a passage number at the end, ask for every claim with the exact words that support it:
{
"found": true,
"answer": "Unused vacation days carry over up to a cap of 5 days.",
"citations": [
{
"passage_id": "hr-handbook-2025#4.2",
"quote": "Employees may carry over a maximum of five (5) unused vacation days"
}
]
}
A quote is better than a bare passage ID for two reasons. For long documents, the same Anthropic guide recommends having the model extract word-for-word quotes first and then work only from them, which forces it to locate the evidence before writing the answer. And a quote can be checked by code, which leads to the control that does the most work.
Most local runtimes can constrain output to a JSON schema or grammar, which removes malformed responses. Ask for short quotes, one sentence or less. Long quotes invite the model to paraphrase while still claiming to quote.
Control 4: check that every quote exists in the source
This is the step that turns a citation from decoration into evidence. After the model responds, the application looks up each cited passage and confirms that the quoted text actually appears in it. No model is involved, so the check cannot itself hallucinate.
A minimal version in Python:
import re
def normalize(s):
s = s.lower()
s = re.sub(r"[\"'`]", "", s)
return re.sub(r"\s+", " ", s).strip()
def quote_is_supported(quote, passage_text):
return normalize(quote) in normalize(passage_text)
def verify(response, passages):
for c in response["citations"]:
text = passages.get(c["passage_id"])
if text is None or not quote_is_supported(c["quote"], text):
return False
return True
Three outcomes follow from the check:
| Result | Likely cause | What to do |
|---|---|---|
| Passage ID was not among the retrieved ones | Model invented a source | Reject the answer |
| ID is valid, quote not found in it | Paraphrase or fabrication | Retry once, then reject |
| All quotes found | Evidence exists | Show the answer with highlighted quotes |
Normalizing case, whitespace and quote marks avoids false rejections from PDF extraction quirks. Resist the urge to accept fuzzy matches above some similarity score. "Payment is due within 30 days" and "Payment is due within 60 days" are 97% similar character by character, and that one digit is exactly the kind of error the check exists to catch.
The cost is small. Take an assistant handling 400 questions a day where 1 in 10 answers fails the check and gets one retry. That is 400 x 0.1 = 40 extra generations a day, a 10% increase in model load. The string check itself takes microseconds.
When an answer is rejected after the retry, show the user the retrieved passages with a note that no verified answer was produced. A list of relevant paragraphs is still useful; a confident wrong sentence is not.
What these controls cannot fix
The four controls together catch fabricated sources, invented quotes and most guessing. They leave several failure types untouched, and staff should know which.
The quote is real but does not support the claim
The model quotes "Contractors are not eligible for the annual bonus" and answers that part-time employees are not eligible. The quote passes the check; the reasoning is wrong. Catching this needs a second step, either a human reading the highlighted quote, or a second model call that asks only whether the quote entails the answer. The second call reduces the problem and adds its own error rate.
The source itself is wrong or out of date
If the 2021 policy and the 2025 policy both sit in the index, the model can faithfully quote the old one. Verification proves the text exists, not that it is current. The fix is upstream: remove superseded documents at ingest, or store an effective date and filter on it.
The right passage was never retrieved
A model that correctly says NOT_FOUND because search missed the passage is behaving well and still failing the user. No answer-side control can help. Stuffing 30 passages into the prompt to avoid misses is not a reliable fix either, since models tend to use the middle of a long context window less well than its start and end. The cure is better search, measured on a fixed set of real questions.
Arithmetic and synthesis across documents
"What did all three vendor contracts cost last year" needs figures from three passages and a sum. Each figure can be quoted and verified, but the total is computed by the model and has no quote to check. Pull the figures out as structured data and do the arithmetic in code.
Prompt injection in the documents
A document that contains instructions to the model can steer the answer while every quote stays genuine. That is a security problem, covered in prompt injection in document-reading AI.
A practical rule for rollout: add controls 1 to 3 on day one, add the quote check before anyone outside the project team uses the system, and route any question involving money, deadlines or legal obligations to a person who reads the highlighted quotes before acting.
Frequently asked questions
Does a bigger local model hallucinate less?
Larger models tend to follow the "only use these passages" instruction more reliably and misread passages less often. They still invent answers when retrieval misses, so the quote check is needed at every size. Testing two sizes on the same set of real staff questions shows whether the larger one is worth its memory cost.
Should the temperature be set to zero?
A low temperature makes output repeatable, which helps when debugging a bad answer, and it removes some random word choices that turn a quote into a paraphrase. It does not stop a model from confidently stating something absent from the passages. Set it low and keep the other controls.
How many retries are reasonable when a quote fails verification?
One retry is a sensible default. If an answer fails twice, the cause is usually missing or ambiguous source text, and further retries mostly produce different wrong answers at extra cost. Log the failures, because a cluster of them on one topic points to a retrieval or chunking gap.
Can staff be told the system does not hallucinate once these controls are in place?
No. The controls make fabricated sources and invented quotes rare and visible, but a real quote can still be used to support a wrong conclusion, and an outdated document can still be quoted correctly. Tell staff that every answer shows its evidence and that they are expected to read it for anything that matters.
Ready to Modernize Your Business?
Get your AI automation roadmap in minutes, not months.
Analyze Your Workflows →