# AI Hallucinations: Limiting Risk in Production

Author: Marin Niehues, Niehues Consulting · Published: 2025-12-15 · Updated: 2026-11-16 · AI Adoption & Operations
Source: https://www.niehuesconsulting.com/blog/ai-hallucinations-in-production/

Hallucinations can be limited to a manageable level in operation when every answer is tied to sources and passes a review graded by risk. A language model generates text by probability and does not automatically check it against facts. The result is statements that sound plausible and are wrong.

For the executive team this is a question of delivery capability: which measures must an organization put in place so that an AI output is reliable enough in daily work? Incorrect information also raises questions of liability. This article provides an overview and does not constitute legal advice.

**The error rate of an AI system is an operational metric. Like lead time or schedule reliability, it is measured, given a target value and assigned to a person.**

## What are AI hallucinations?

A hallucination is an output of a language model that is linguistically convincing and factually wrong or unsupported. It arises because the model produces the most probable continuation of a text without any built-in fact check.

In a pilot it carries little weight, because experts read and assess every answer. In regular operation the situation changes. Answers flow into offers and customer letters, into reports and decision papers, and the number of users grows. The guiding question thus shifts from "Does the model reach the right answer?" to "What is the error rate in the process, and who catches the errors?"

## How often do language models hallucinate?

How often a language model hallucinates is determined mainly by the type of task: answers from the model's own knowledge on specialist questions are far more error-prone than summaries of documents provided to it. There is therefore no universal rate, but there are robust orders of magnitude.

A study by Stanford University tested widely used language models with verifiable questions about US court decisions. Depending on the model, 69 to 88 percent of answers were wrong. A follow-up study tested specialized legal research tools that tie their answers to a case-law database. They, too, produced false or unsupported statements in 17 to 33 percent of cases, markedly less often than a general model without source grounding, but far from zero.

**Hallucinations in figures**
69 to 88 % | Share of incorrect answers from general language models on verifiable questions about US court decisions
17 to 33 % | Share of false or unsupported answers from specialized research tools with source grounding in the same domain

Three rules follow for practice. Source grounding reduces the error rate considerably, but does not eliminate it. Specialist knowledge from the model's memory is unsuitable for operation as soon as errors have consequences. And every company measures its own rate on its own use case, because values from other domains only indicate direction. The article [Large language models compared](/blog/large-language-models-comparison/) covers differences between models.

## Examples of AI hallucinations with consequences for companies

Two publicly decided cases show how a hallucination turns into damage.

**Two cases with consequences**
Air Canada: The airline's chatbot told a customer about a bereavement refund rule that did not exist in that form. A Canadian tribunal ruled in 2024 that the company is responsible for its chatbot's information like any other information on its website and awarded the customer the difference.

Mata v. Avianca: In 2023, lawyers filed a brief with a federal court in New York that had been drafted with a language model and cited invented judgments. The court imposed sanctions on the lawyers and their firm.

Both cases share the same structure. The answer went to third parties without review, and the organization could not hide behind the tool. The damage lay less in the amount than in the loss of trust. The article [Liability for AI errors](/blog/ai-liability-for-executives/) sets out who in the company is answerable for incorrect information given to customers.

## Four error types with different damage

Classifying by error type makes it easier to choose the countermeasure. In practice, four patterns occur:

- Invented facts: The model cites figures or sources that do not exist.
- Wrong attribution: Correct information is applied to the wrong matter, such as the terms of a different customer.
- Outdated information: The model answers according to its training data, while the process requires the current state.
- Overreach: The model answers a question for which no basis exists instead of naming the gap.

The damage results from the error type and the use of the answer. An invented figure in an internal draft costs correction time. The same figure in a customer offer costs trust and margin. Every use case therefore needs a classification of potential damage before review depth and budget are decided.

## Four measures along the delivery chain

Limitation succeeds along the entire delivery chain, from scoping the task to measurement in operation. Four points determine the result.

First, the task. The more narrowly it is defined, the less often the model deviates. The instruction specifies the output format. It requires source references and explicitly permits the answer "No information is available on this". This permission reduces the number of invented answers, because without it a model tends to answer every question.

Second, the knowledge. [Retrieval augmented generation](/blog/retrieval-augmented-generation-enterprise/) connects the model with approved company documents. The answer is based on retrieved passages, and the source reference makes checking quick. The quality of answers then depends on data quality and search, both tasks for the organization.

Third, the review. Automated checks compare figures with the source system, verify quotations against the original text and block output in the wrong format. Human review applies where the potential damage requires it. The next section arranges it by risk level.

Fourth, measurement. A test set of questions with known correct answers makes the error rate visible before every release, and samples in ongoing operation show how it develops. Every change to model, instructions or knowledge base triggers the test again. The article on [the quality of AI results](/blog/ensuring-ai-output-quality/) describes the procedure.

**Metrics for steering the error rate**
- Error rate per use case, in the test set and in operation
- Share of answers with a verifiable source reference
- Share of outputs corrected by a review
- Time to correct a reported error

## How do you detect hallucinations in AI answers?

Hallucinations are detected most reliably by checking every statement against an approved source. Three checks have proven effective. The source check verifies whether the cited passage actually contains the statement. Data matching compares figures, names and dates with the leading system, such as the customer master or the price list. Repeated querying asks the same question several times; if the answers differ, the statement is uncertain and goes to human review.

Warning signs for experts are very precise details without a source reference, quotations that cannot be found, and an answer to a question for which the documents contain no basis.

## Setting review depth by risk level

Review depth follows the potential damage. Three levels suffice as a framework:

- Level 1, internal drafts: Use has minor consequences. Samples and ongoing measurement of the error rate are sufficient.
- Level 2, results with external impact: A second person reviews before sending, supported by automated matching with the source system.
- Level 3, results with high financial, legal or personnel significance: The system delivers a proposal with sources, and a named expert makes the decision.

For systems that the EU AI Act classifies as high-risk, human oversight is mandatory in any case. The levels prevent two undesirable developments. Blanket review ties up experts in cases that need no review. Missing review at level 3 turns a productivity project into a risk.

**Three ways of dealing with hallucinations**
Criterion | Switching models | Individual tests | With Niehues Consulting
Approach | Expectation of a better model | Samples before launch | Source grounding and review rules by potential damage of the application
Accountability | Open | With the project team | Named owners per use case
Effect in operation | Fluctuates with every model version | Ends with the project | Error rate remains manageable and reportable
Adoption in the business unit | Random | Tied to individuals | Secured through enablement and clear review processes

## How Niehues Consulting delivers such initiatives

[Niehues Consulting](/) supports initiatives to limit hallucinations from analysis to implementation. Marin Niehues works as a management consultant from a delivery perspective. He builds organizations that deliver and brings AI into operations, in projects ranging from VC-funded scale-ups to DAX 40 corporations, with assignments in 20+ countries across six continents.

The approach follows the Niehues Delivery Model with the phases Understand, Build and Implement. For error management, three review questions matter most. Who decides, who delivers, who uses? Does the prototype prove the benefit in the real workflow with real data? Who owns operations from day one? In Understand, this produces the classification of use cases by potential damage, combined with requirements and the business case. In Build, a prototype in the real workflow shows which error rate is achievable with source grounding and review rules. In Implement, Niehues Consulting defines review processes and responsibilities, enables the business units and hands operations over to internal or external teams.

The error rate thus gets an owner and a target value, and it appears in reporting before the system is rolled out widely. The article [From pilot to scale](/blog/from-pilot-to-scale-ai-operations/) describes the full path from use case to regular operation. The free 30-minute initial consultation clarifies which use cases should be classified first ([book an appointment](/#kontakt)).

**What decision-makers should settle**
Four decisions turn a quality risk into a metric that management tracks in reporting:
- the potential damage per use case
- the grounding of answers in sources
- the review depth per risk level
- the owner of the error rate

**Sources**
- Dahl, Magesh, Suzgun, Ho: Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models, Journal of Legal Analysis, 2024
- Magesh et al.: Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, Stanford University, 2024

## Frequently asked questions

### What are AI hallucinations?

Hallucinations are outputs of a language model that are convincingly worded and yet false or unsupported. They arise because the model generates probable word sequences and performs no fact check. In a company they become a risk as soon as answers flow into decisions or customer communication without review.

### How often do LLMs hallucinate?

Frequency is determined mainly by the type of task. In studies on legal questions, general models produced 69 to 88 percent incorrect answers, and tools with source grounding 17 to 33 percent. When summarizing documents provided to the model the rate is markedly lower, and it is measured on one's own use case.

### Why does AI hallucinate?

AI hallucinates because a language model generates the most probable continuation of a text and has no knowledge of truth. If a suitable basis is missing, a fluent answer is produced anyway. Instructions that permit admitting a knowledge gap reduce this behavior.

### Can hallucinations be prevented completely?

Hallucinations cannot be prevented completely, because language models generate text by probability. The aim is a residual rate the process can absorb. It is achieved through narrow tasks, source grounding, graded review and ongoing measurement.

### Does retrieval augmented generation help against hallucinations?

Retrieval augmented generation reduces the error rate considerably, because the model relies on approved documents and cites sources. Errors still arise when the search retrieves the wrong passages or the documents are outdated. Data maintenance and measurement are therefore part of the initiative.

### How can you detect hallucinations in AI answers?

Hallucinations are detected by checking every statement against an approved source and against the data of the leading system. In addition, repeated querying reveals uncertain statements, because the answers then differ. Precise details without a source reference are a warning sign.

### What are examples of AI hallucinations with business consequences?

Well-known examples are the Air Canada chatbot, for whose incorrect refund information a Canadian tribunal held the company liable, and the case of Mata v. Avianca, in which lawyers cited invented judgments and were sanctioned. In both cases the answer went to third parties unreviewed.

### Who in the company is responsible for the error rate?

Responsibility lies with the business unit that uses the result. IT provides technology and measurement, while the business unit sets the target value and review depth. This assignment belongs in the operating model of the use case before the system goes into regular operation.
