Published:
Get personalized guidance to refine your product strategy and achieve market success.
Large language models generate the next plausible sequence of words based on patterns in their training data. They don't have a built-in mechanism that distinguishes "I know this for a fact" from "this sounds like the kind of thing that would be true." Left unconstrained, a model asked about a specific company policy will produce something that reads exactly as confident whether it's grounded in a real source or not, which is precisely what made the Air Canada case so damaging. The customer had no way to know one part of the website was reliable and another wasn't.
This is the gap that separates a demo from a production system. A lot of AI agent development services can get a chatbot answering questions convincingly in a sales demo. Far fewer have actually built in the guardrails that keep it from confidently inventing an answer once it's live and talking to real customers about real policy.
The fix isn't "use a smarter model." Hallucination risk gets managed architecturally, not by hoping the underlying model has improved enough to stop making things up. A few things we treat as non-negotiable when we build agents for regulated or compliance-sensitive conversations:
Retrieval-grounded answers, not open generation. The agent should be pulling from your actual, current source-of-truth documents, policy pages, product data, account records, rather than generating an answer from general training knowledge. If the answer isn't in the retrieved source material, the agent shouldn't be answering from memory.
A real "I don't know" path. This sounds obvious and is routinely skipped. An agent that's allowed to say "I'm not certain, let me connect you with someone who can confirm that" is dramatically safer than one that's optimized purely to sound helpful in every turn. Refusal behavior has to be designed in deliberately; it's not a model's default setting.
Escalation to a human before commitment, not after. In Moffatt's case, the failure that actually cost Air Canada wasn't the hallucination itself, it was that nothing caught it before the customer acted on it. A well-built agent routes anything touching a binding commitment, refund eligibility, a medical instruction, a financial term, to human confirmation before the conversation closes, not as a support ticket filed afterward.
Logging and audit trails from day one. Every regulated industry eventually needs to answer "what did the agent actually say, and when." An agent without a full conversation log and decision trace is a liability problem waiting to surface, independent of whether it hallucinates or not.
Testing against adversarial and edge-case prompts before launch, not after a customer finds the gap. Air Canada didn't get caught by a hypothetical, they got caught by an actual customer asking an actual question their system wasn't ready for. Red-teaming an agent against exactly those kinds of edge cases before it goes live is cheaper than finding out about the gap from a tribunal ruling.
According to Salesforce Research, 35% of customers say they'd rather work with an AI agent than a human if it meant they didn't have to repeat themselves.
If you're evaluating vendors, the useful question isn't "can your agent handle complex conversations." Every agency will say yes to that. The sharper questions are: What happens when the agent doesn't know something, does it guess or does it say so? Is every answer traceable back to a specific source document? Who reviews edge cases before launch, and how? A custom AI agent development company that's actually built for regulated use cases should have concrete, specific answers to all three, not a general assurance that "the model is very capable."
If you're weighing whether to build this in-house, through a no-code platform, or with a dedicated AI agent development agency, hallucination risk in exactly the conversations that matter most to your business, refunds, eligibility, compliance-adjacent questions, is one of the clearest signals of whether you're looking at a real production system or a demo that hasn't been tested against the case that eventually finds you.
We build AI agents with retrieval grounding, escalation logic, and audit trails as the starting point, not an add-on requested after something goes wrong. If you're scoping an agent for a regulated or compliance-sensitive workflow, our AI agent development team can walk through what that actually looks like for your specific conversations.
Next in this series: why the handoff between an AI agent and your CRM or phone system is usually where a working pilot quietly breaks in production.
.png)