“Hallucination” gets framed as a model defect. It is more accurately a system-design defect. Language models fabricate when they are asked questions they have no grounded answer for and are not given a way to admit uncertainty. The patterns that fix this are mechanical, not magical, and they are the difference between an agent you can put in front of customers and a demo you keep apologizing for.
Why agents make things up
A language model is a machine for producing plausible next words. Ask it your customer’s order status when it has no access to your order system, and it will still produce an order status — a fluent, confident, entirely invented one — because producing answers is what it does. Nothing in the model knows the difference between “I retrieved this” and “I composed this.”
That framing matters because it tells you where the fix lives. You will not prompt your way out of hallucination with “be accurate” instructions, and you will not wait your way out with the next model release. You engineer your way out with the system around the model. In the AI systems we build for clients, that system rests on five patterns.
Give the agent tools instead of forcing it to guess
If the agent needs to know an order status, give it a tool that queries the order system. If it needs a customer’s plan tier, give it a tool. A well-tooled agent answers from the database; a poorly-tooled agent makes a plausible-sounding answer up.
Tool design is where this succeeds or fails in practice. Tools should be narrow and well-described, so the model reliably picks the right one. They should return explicit “not found” and error states, because a tool that returns an empty string invites the model to fill the silence. And every tool call should be logged, so when an answer is wrong you can see whether the model misused a tool or never called one — two different bugs with two different fixes.
Make “I don’t know” a first-class output
Prompts that say “always provide an answer” produce confident garbage when the model is unsure. Prompts that explicitly permit and reward “I do not know, here is what I would need to find out” produce calibrated answers — which is what users actually want from a customer-facing system.
Go one step further and give uncertainty somewhere useful to go. An agent that can hand off to a human, open a ticket, or ask a clarifying question does not need to bluff. Most of the hallucinations we get called in to fix trace back to an agent that had exactly one allowed behavior: answer, no matter what.
Ground generation in retrieved context
RAG works because the model is no longer trying to recall a fact — it is summarizing or extracting from a passage you literally just handed it. Combine that with a citation requirement (“include the source chunk ID for every claim”) and you can audit answers and reject ungrounded ones programmatically, before a user ever sees them.
The honest caveat: retrieval quality is usually the bottleneck, not generation. If the retriever surfaces the wrong policy document, the model will faithfully ground its answer in the wrong policy. Budget as much effort for chunking, indexing, and retrieval evaluation as for the prompt. We covered when retrieval is the right approach at all — versus fine-tuning or plain prompt engineering — in a separate breakdown.
Constrain output structure
Free-form text invites drift. Structured outputs — JSON schemas with required fields, enums instead of open strings, function-call schemas — cut hallucination rates dramatically because the model has fewer degrees of freedom to invent.
The practical version: wherever the agent’s answer feeds another system, define a schema and validate against it. A response that fails validation gets rejected and retried automatically instead of flowing downstream. An agent that must pick a refund reason from seven enum values cannot invent an eighth. You are not trusting the model to behave; you are making misbehavior unrepresentable.
Run evals on the cases that already broke
Every production hallucination becomes a new eval case. Over time the suite becomes a tight contract for what the agent is and is not allowed to say. Models change; the contract endures. When a new model version ships, you rerun the suite and know in an hour whether the upgrade is safe — instead of finding out from a customer.
This is why our engagements turn success criteria into an eval suite during the proof-of-concept phase rather than after launch. It is the same rhythm we described in how a real AI engagement runs: the evals you write to prove the system works become the regression harness that keeps it working.
The system is the product
Notice what none of these patterns require: a frontier model, a secret prompt, or luck. The reliable production agent is not the smartest model with the cleverest prompt. It is the well-tooled, well-grounded, well-evaluated system around a strong-but-conventional model. The model is mostly interchangeable. The system around it is the product — and it is ordinary, disciplined software engineering, which is exactly why a senior team gets it right and a rushed pilot does not.
If you have an agent in pilot that works most of the time but occasionally, confidently invents facts, the fix is almost always at the system level, not the model level. Bring us the transcript of its worst answer — the first 30 minutes are free, and that transcript usually tells us which of the five patterns is missing.