When AI Introduces Errors — and How Teams Catch Them
AI reduces human error — and introduces new error types of its own: hallucinations (confident, fabricated output) and automation bias (people over-trusting AI and waving its mistakes through). Independent benchmarks put hallucination rates anywhere from ~1.5% of sentences in clinical note generation to 17–34% of legal research queries, and teams catch these errors not with a single tool but with layered oversight: engineered workflows, human-anchored evaluation, and — before anything else — knowing which AI systems they run.
We’ve written about how AI reduces human error in daily operations, and that case still stands. This article is the other side of the ledger: what the research says about the errors AI itself introduces, the well-documented incidents where those errors cost real money, and what actually works to catch them. No fear-mongering — just the numbers, with their dates and their caveats.
See HowThe Errors AI Introduces: Hallucinations, by the Numbers
The best-documented AI-introduced error is the hallucination: output that is fluent, confident, and wrong — a fabricated citation, an invented product policy, a symptom the patient never mentioned. The first thing the research makes clear is that there is no single “hallucination rate.” Measured rates vary by an order of magnitude depending on the task, the configuration, and the benchmark — so any number you read (including the ones below) comes with a date and a model generation attached.
| Task / domain | Measured hallucination rate | Study & model era |
|---|---|---|
| Clinical note generation | 1.47% of generated sentences — but 44% of those rated “Major” (could affect diagnosis or treatment) | npj Digital Medicine, 2025 — GPT-4, a 2023-era model, on simulated consultations |
| Grounded RAG responses (source material provided) | 6.6–16.4% of responses across frontier models | FaithJudge, EMNLP 2025 — 2024–25 model generation |
| Legal research (commercial AI tools) | >17% to >34% of queries | Stanford RegLab/HAI — tool versions of May 2024; peer-reviewed 2025 |
| Short factual Q&A | 27–81% of non-refused answers, depending on model | HalluLens (Meta AI), ACL 2025 — 2024-generation models |
Three of these deserve unpacking, because the details are where the operational lessons live.
Legal research: domain tools hallucinate despite “hallucination-free” marketing. A Stanford RegLab/HAI benchmark (Magesh et al.) ran 200+ pre-registered legal queries, graded by legal experts, against the leading commercial legal AI tools as they stood in May 2024. LexisNexis’s Lexis+ AI and Thomson Reuters’s Ask Practical Law AI produced incorrect information in more than 17% of queries; Westlaw AI-Assisted Research in more than a third — even though vendor marketing at the time promised tools that “avoid hallucinations” and deliver “hallucination-free” linked citations. Fairness requires noting the dispute: Thomson Reuters and LexisNexis contested the methodology — arguing, among other things, that Ask Practical Law is a how-to product rather than a research tool — and cited lower internal figures. The study was later peer-reviewed in the Journal of Empirical Legal Studies (2025), and it remains the only pre-registered, expert-graded public benchmark of these tools.
Clinical notes: rare, but disproportionately dangerous when they happen. A 2025 study in npj Digital Medicine had 50 clinicians annotate 450 AI-generated clinical notes (GPT-4, a 2023-era model, on simulated consultations). Hallucinations appeared in only 1.47% of generated sentences — but 44% of them were classified as “Major,” capable of affecting diagnosis or treatment, versus 16.7% of omissions. When a model invents content, it is more dangerous per error than when it leaves something out. One disclosure matters here: the authors were employees of Tortus AI, a clinical-AI vendor, including its CEO — and, to their credit, the paper reports the errors of their own product category in detail.
RAG: it helps, it does not fix. Retrieval-augmented generation — giving the model your documents to ground its answers — is often sold as the cure for hallucinations. The FaithJudge benchmark (a study by Vectara researchers with the University of Waterloo, EMNLP 2025 Industry Track) measured how often frontier models hallucinate even when handed the relevant source material: from 6.65% for the best model tested (Gemini-2.5-Pro) up to ~16% for several others — roughly one grounded answer in fifteen, at best. Note the affiliation: Vectara sells RAG and hallucination-detection products, which makes the finding notable — the Stanford legal study independently corroborates it, with RAG-based legal tools hallucinating at far higher rates.
And the rates are task-shaped, not model-shaped: Meta AI’s HalluLens benchmark (ACL 2025) found that on short factual Q&A, 2024-generation models hallucinated in 27–81% of the answers they didn’t refuse — and that rankings reshuffle between task types. GPT-4o, among the stronger models on other HalluLens tasks, fabricated information about non-existent entities in 42.3% of cases where Llama-3.1-405B did so in only 6.9%. The model that leads on one hallucination task can rank near the bottom on another.
Why doesn’t this just get patched? Researchers at OpenAI and Georgia Tech (three of the four authors employed by OpenAI; the paper is a preprint, and its framing is debated) argue that hallucination is structural: pretrained models face statistical pressure to produce plausible falsehoods whenever falsehoods are indistinguishable from facts in training data, and the errors persist after fine-tuning because most benchmarks reward confident guessing over admitting uncertainty. Whatever the final academic verdict, the operational takeaway is uncontroversial: plan for hallucinations as a property of the technology, not a bug awaiting a patch.
The Human Side: Automation Bias
Hallucinations are only half the story. The other half is what happens on the human side of the screen — because an AI error only becomes an organizational error when a person accepts it.
Human-factors research has studied this for decades under the names automation complacency and automation bias. The foundational review (Parasuraman & Manzey, 2010, in Human Factors) established findings that map uncomfortably well onto today’s AI assistants: complacency is an attention-allocation problem, not laziness — it emerges when monitoring the automation competes with other tasks for attention; it produces both omission errors (missing what the system failed to flag) and commission errors (following its wrong recommendations); it affects experts as well as novices; and — critically — it cannot be trained away by instructions alone. Training and experience reduce it; they do not eliminate it.
The LLM-era evidence points the same way:
- Confidence in AI is associated with less critical thinking. A CHI 2025 survey study by Microsoft Research and Carnegie Mellon (319 knowledge workers, 936 real work tasks) found that the more confident workers were that the AI could handle a task, the less critical thinking they reported applying (β = −0.69, p < 0.001) — while confidence in their own ability to do and to evaluate the work was associated with more. The data is correlational and self-reported, but the direction is exactly what complacency theory predicts.
- Training alone does not protect. In a single-blind randomized clinical trial published in NEJM AI (44 physicians, single site, fielded June–August 2025), participants completed 20 hours of AI training — including how to critically evaluate AI outputs — and then worked diagnostic vignettes with ChatGPT-4o, which had been seeded with deliberately erroneous recommendations in half of them. Exposure to the erroneous advice cut diagnostic accuracy by an adjusted 14.0 percentage points (84.9% vs 73.3%, p < 0.0001). Trained physicians remained vulnerable — which is the precise, honest framing: not that training is useless, but that it is not sufficient. That matters for the EU AI Act’s Article 4 AI-literacy duty too: literacy is necessary — and legally required — but it is the floor of oversight, not the whole of it.
This combination — systems that confidently fabricate, and people who increasingly defer to them — is what turns a model error into a business incident. And as assistants give way to AI agents that execute multi-step work on their own, the same combination operates with fewer human checkpoints in the loop.
When Nobody Catches It: Three Documented Incidents
These failure modes are not hypothetical. Three well-documented cases — each confirmed by court records or primary reporting — show what happens when an AI error travels all the way to the outside world.
Mata v. Avianca (USA, 2023) — the canonical hallucinated-citations case. A New York lawyer used ChatGPT for legal research, and the resulting court filing cited six judicial opinions that did not exist — complete with fabricated quotes and citations. When challenged, ChatGPT had even assured him the cases “can be found in reputable legal databases such as LexisNexis and Westlaw.” In June 2023, Judge P. Kevin Castel of the Southern District of New York found the two attorneys involved had acted in bad faith and imposed a $5,000 penalty, jointly and severally, on the lawyers and their firm — and ordered them to send the sanctions opinion to their client and to every real judge who had been falsely identified as the author of a fake opinion. A modest fine; a career-defining public record. Legal-research trackers have since catalogued hundreds of subsequent court decisions worldwide involving AI-fabricated material in filings.
Moffatt v. Air Canada (Canada, 2024) — your chatbot’s word is your word. Air Canada’s website chatbot told a bereaved customer he could apply for the airline’s bereavement fare retroactively — contradicting the actual policy stated elsewhere on the same website. When he claimed the discount, the airline refused, and he took it to the British Columbia Civil Resolution Tribunal. Air Canada argued, in the tribunal’s words, that the chatbot was “a separate legal entity that is responsible for its own actions” — which the tribunal called “a remarkable submission.” The February 2024 decision found negligent misrepresentation: the airline “did not take reasonable care to ensure its chatbot was accurate,” and it made no difference “whether the information comes from a static page or a chatbot.” The award was small — CA$812.02 — but the principle was not: the company owns what its AI says.
Deloitte Australia (2025) — hallucinations in a paid deliverable. Deloitte delivered an independent assurance review — a contract worth roughly AU$440,000 — to Australia’s Department of Employment and Workplace Relations. After publication in July 2025, a University of Sydney researcher flagged fabricated academic references attributed to real academics and a fabricated quote from a Federal Court judgment. The corrected version disclosed that a generative AI toolchain (Azure OpenAI GPT-4o) had been used in preparing the report, and Deloitte repaid the final instalment under the contract — just over AU$97,000 — while the department maintained the report’s substantive recommendations stood.
Notice the shared pattern: in all three cases, the error was caught outside the organization that deployed the AI — by a judge, a customer with a tribunal claim, an external academic. Internal review either did not exist or did not work. That is the gap the next section is about.
What Actually Catches AI Errors
The backdrop to those incidents is that adoption keeps outrunning oversight. In McKinsey’s global State of AI survey (fielded mid-2025, ~2,000 respondents across 105 countries), 88% of organizations reported regular AI use in at least one business function — while in the preceding edition of the same survey, only 27% of organizations using gen AI said employees review all gen-AI-generated content before it is used. And Stanford’s AI Index, drawing on the AI Incident Database, counted 233 reported AI incidents in 2024 and 362 in 2025 — records in consecutive years.
The encouraging news in the research is that AI error rates are not fate — they respond to engineering and process. Four patterns have evidence behind them.
1. Engineer the workflow, not just the prompt. In the same npj Digital Medicine study, iterative prompt engineering, structured note templates, function calls and JSON-structured outputs reduced major hallucinations by roughly 75% between early and refined experiments (in small absolute numbers: from 4 to 1 per 25 notes). The error rate turned out to be a property of the configuration — templates, structure, constraints — not just of the model choice. The same study also shows the limit: the run with the fewest hallucinations still had 10 major omissions. Low hallucinations do not mean low errors.
2. Don’t outsource detection to another AI — anchor evaluation on humans. Automated hallucination detectors are themselves unreliable on hard cases: on the deliberately challenging FaithBench set, the best-performing detector managed 66.7% balanced accuracy, and some current detection methods — including LLM classifiers — performed close to chance. (Notably, that finding comes from Vectara researchers evaluating their own detector.) Detectors are useful as a first-pass filter on easier material; they are not a substitute for human-anchored evaluation of the outputs that matter.
3. Design oversight as a system, not a heroic reviewer. Because automation bias affects trained experts, effective oversight is structural: sampling-based review of AI outputs, verification steps for consequential actions, escalation paths, and monitoring in production. This is exactly the shape of the established frameworks — the voluntary NIST AI Risk Management Framework (January 2023) organizes this into four functions — Govern, Map, Measure, Manage — and its Generative AI Profile (NIST AI 600-1, July 2024) names confabulation as a risk unique to or exacerbated by generative AI. In the EU, Article 14 of the EU AI Act makes effective human oversight a design requirement for high-risk systems — the system must be built so a human can understand, monitor, and intervene.
4. Start from an inventory — you can only oversee the AI you know you have. Every pattern above presupposes something so basic it is routinely skipped: a current list of which AI systems are in use, for what, by whom, and with what risk. Review workflows can’t cover a chatbot nobody registered; evaluation can’t measure a departmental tool nobody disclosed. That is why both NIST’s Map function and the EU AI Act’s obligations effectively begin with an inventory. If your organization runs Jira, Model Inventory for Jira gives you that register inside the tool you already have — every AI system recorded with owners, risk classification, and an audit trail, so the oversight patterns above have something to attach to.
Frequently Asked Questions (FAQ)
Does RAG eliminate hallucinations?
No. Retrieval-augmented generation reduces hallucinations but does not eliminate them. The FaithJudge benchmark (Vectara researchers with the University of Waterloo, EMNLP 2025) measured hallucination rates of 6.6% to 16.4% across frontier models on grounded RAG tasks — even when the model was given the relevant source material. An independent Stanford benchmark found RAG-based legal research tools hallucinating in 17–34% of test queries.
How often does AI hallucinate?
There is no single hallucination rate — it depends heavily on the task and configuration. Measured rates range from 1.47% of sentences in clinical note generation (npj Digital Medicine, 2025) through 6.6–16.4% on grounded RAG tasks (FaithJudge, 2025) and 17–34% of legal research queries (Stanford, tools as of May 2024), up to 27–81% on short factual Q&A for 2024-generation models (HalluLens, Meta AI, ACL 2025). Always ask: measured on which task, in which configuration, and when.
Does AI training protect people from AI errors?
It helps, but it is not sufficient. In a randomized clinical trial published in NEJM AI, physicians who had completed 20 hours of AI training — including critical evaluation of AI outputs — still lost an adjusted 14 percentage points of diagnostic accuracy when exposed to deliberately erroneous chatbot recommendations. Decades of human-factors research reach the same conclusion: training and experience reduce automation bias but do not eliminate it, so processes must be designed to catch errors, not just people trained to spot them.
Who is responsible when an AI tool gives wrong information?
In practice, the organization that deploys it. In Moffatt v. Air Canada (2024), a Canadian tribunal held the airline responsible for incorrect fare information given by its own website chatbot and rejected the argument that the chatbot was a separate entity. Courts have also sanctioned lawyers who filed AI-fabricated citations, and Deloitte partially refunded the Australian government for a report containing AI-hallucinated references.
What is the first step to overseeing AI at work?
An AI inventory. Every oversight practice — review workflows, evaluation, monitoring, incident response — presupposes that you know which AI systems are in use, for what, and by whom. Governance frameworks such as the NIST AI RMF start from exactly this kind of mapping, and the EU AI Act’s human oversight requirement (Article 14) can only be met for systems you have identified. You can only oversee the AI you know you have.
Sources
- Stanford RegLab/HAI — Magesh et al., “Hallucination-Free? Assessing the Reliability of Leading AI Legal Tools” (benchmark run May 2024; peer-reviewed in the Journal of Empirical Legal Studies, 2025)
- npj Digital Medicine (2025) — clinician-annotated evaluation of GPT-4 clinical notes; authors employed by clinical-AI vendor Tortus AI, incl. its CEO
- FaithJudge (EMNLP 2025 Industry Track) — Vectara researchers with the University of Waterloo; hallucination rates on grounded RAG tasks
- FaithBench (NAACL 2025 Findings) — benchmark of hallucination detectors on challenging cases
- HalluLens (ACL 2025) — Meta AI hallucination benchmark; task-dependent rates, 2024-generation models
- Kalai, Nachum, Vempala & Zhang — “Why Language Models Hallucinate” (preprint; three of four authors at OpenAI)
- Parasuraman & Manzey (2010) — “Complacency and Bias in Human Use of Automation,” Human Factors
- Lee et al. (CHI 2025) — Microsoft Research & Carnegie Mellon survey of GenAI and critical thinking in knowledge work
- Qazi et al. — randomized clinical trial of physicians exposed to erroneous LLM recommendations (published in NEJM AI; preprint on medRxiv)
- Mata v. Avianca, Inc., No. 22-cv-1461 (S.D.N.Y.) — sanctions opinion of June 22, 2023 (678 F.Supp.3d 443), court docket; news coverage: CBS News, June 23, 2023
- Moffatt v. Air Canada, 2024 BCCRT 149 — BC Civil Resolution Tribunal decision, February 14, 2024; news coverage: CBC News
- Fortune, October 7, 2025 — Deloitte Australia partial refund over AI-hallucinated report content; refund amount: CFO Dive, October 21, 2025
- Damien Charlotin — AI Hallucination Cases database (court decisions involving AI-fabricated material)
- NIST — AI Risk Management Framework (AI RMF 1.0, January 2023) and NIST AI 600-1 — Generative AI Profile (July 2024), listing confabulation among generative AI risks
- McKinsey — The State of AI (November 2025 edition; survey fielded June–July 2025, 1,993 participants in 105 nations) and March 2025 edition (review-of-gen-AI-output figure)
- Stanford HAI — AI Index Report 2026, Responsible AI chapter (AI incident counts via the AI Incident Database)
All figures are benchmark- and era-specific: the legal benchmark tested tool versions of May 2024, the clinical study used a 2023-era GPT-4, and HalluLens covers 2024-generation models. None of them should be read as the performance of mid-2026 frontier models — or of your configuration. That, in a sentence, is why you measure your own.