# Ensuring the quality of AI outputs: testing, evaluation, monitoring

Author: Marin Niehues, Niehues Consulting · Published: 2025-12-08 · Updated: 2026-11-16 · AI Adoption & Operations
Source: https://www.niehuesconsulting.com/blog/ensuring-ai-output-quality/

A company ensures the quality of AI outputs by defining quality before launch, testing every change against a test set and measuring continuously in operation. A language model does not always give the same answer to the same question, and providers release new model versions at short intervals and retire old ones. Quality therefore emerges from a control loop with clear ownership.

This article describes that loop from a delivery perspective: what must an organization put in place so that an AI application keeps producing reliable results? The article [From pilot to scale](/blog/from-pilot-to-scale-ai-operations/) sets out the full path from the first use case to regular operation.

**The quality of an AI application is a management requirement with a measurement rule and a threshold. It is defined by the business unit, approved before the first test and checked again with every change.**

## Define quality before testing

A test needs a yardstick. In conventional software the specification provides it; for AI applications the business unit has to formulate it. The first step is therefore a quality profile, owned by the business unit and approved by delivery management. For each application it defines what makes a good result and from what point a result is no longer acceptable.

Five dimensions have proven effective:

- Factual accuracy against a verified source
- Completeness measured against the task
- Compliance with format and tone as well as company guidelines
- Behavior with unclear or inadmissible requests
- Response time and cost per case

Each dimension receives a measurement rule and a threshold. The threshold follows the risk of the use case: an assistant for internal research tolerates different margins from an application whose output goes to customers unchecked. For critical errors, such as a wrong deadline or a wrong amount, a separate threshold of zero occurrences in the test set applies.

## How do you build a test set for AI?

A test set for AI is a collection of real requests from the business unit, each with an expected answer or an assessment standard, created and assessed by subject-matter experts. It contains typical and rare cases as well as deliberately difficult ones, such as ambiguous wording, contradictory sources or requests outside the application's remit.

The composition follows real demand. If complaints account for 40 percent of requests, they also make up about 40 percent of the test set, supplemented by a fixed group of difficult cases. Subject-matter experts create the test set, the development team runs it. That way nobody can adjust the yardstick to their own solution. Part of the cases is also held back and used only for approval, so that instructions are not optimized towards the known test cases.

## How large does a test set for evaluation need to be?

For the start, 50 to 200 carefully selected cases are sufficient; for applications whose results go to customers unchecked, several hundred. The size follows a statistical consideration: the smaller the test set, the greater the uncertainty of the measured hit rate.

**Worked calculation: how much a test set tells you**
At a 95 percent confidence level, the uncertainty of a measured hit rate p with n test cases is about 1.96 times the square root of p times (1 minus p) divided by n.

100 cases, measured hit rate 90 percent: the uncertainty is around 6 percentage points, so the true rate lies between about 84 and 96 percent.

400 cases, same hit rate: the uncertainty falls to around 3 percentage points.

Question for the executive team: how large is the gap between the measured rate and the threshold? If it is smaller than the uncertainty, the test set needs more cases before approval.

The same applies when comparing two versions. If two versions differ by two percentage points, a test set of 100 cases does not show this difference reliably. Because language models do not answer deterministically, each case also runs several times, and the rate across all runs is what counts.

## Three methods of assessment

Organizations combine three methods of assessment. Automated checks capture everything that can be determined by rules, such as format, mandatory information or the presence of specific amounts.

The second method is called LLM-as-a-judge: a second language model assesses the answers against a fixed assessment guide. A study by researchers at UC Berkeley and other universities showed that a strong model acting as judge agreed with human preferences in over 80 percent of cases, about as often as humans agree with each other. The same study described typical biases, such as favoring the answer presented first or the longer one. The method therefore scales the assessment of content and tone, but is calibrated against human judgment before it feeds into approval.

Assessment by subject-matter experts remains the reference standard and continues permanently as a sample.

**Assessment by a second model**
over 80 % | Agreement of a strong language model acting as judge with human preferences in a comparative study, on a par with agreement between humans

Approval is granted when all dimensions reach their thresholds. A single average is not enough, because it hides weak areas and rare but critical errors.

## Testing every change: the regression run

An AI application changes more often than conventional software. The provider releases a new model, a team adjusts an instruction, the knowledge base grows. Each of these changes can improve results in one place and worsen them in another. The regression run therefore checks every change against the approved reference state: internal changes before approval, provider model changes before switching to the new version. For this, the application uses a pinned model version instead of an automatically updated model. The run reports deviations per dimension.

**Triggers for a test run**
- Switch to a new version of the language model
- Change to instructions or parameters
- Extension or clean-up of the knowledge base
- New user group or new use case

The product owner decides on approval on the basis of the run. A change without a test run counts as not approved. The rule belongs in release management, so that it applies without case-by-case decisions.

## Which tools exist for evaluating AI applications?

There are three categories of tools for evaluating AI applications: open testing frameworks, platforms for traces and evaluation, and the evaluation features of the major cloud providers. Open frameworks such as promptfoo run test sets automatically and compare versions; for applications with a connected knowledge base there are specialized libraries such as Ragas. Platforms such as Langfuse or MLflow connect pre-launch evaluation with the recording of real requests in operation.

The selection follows the quality profile. First the dimensions and thresholds are settled, then the company decides which tool can measure them. A tool without a quality profile measures what is easy to measure and leaves out what matters.

## Monitoring in operation

Even a passed test set only reflects the requests the team anticipated. In operation, applications encounter inputs nobody planned for and content that changes over time. Monitoring closes this gap with four groups of signals:

- User feedback
- Spot checks by subject-matter experts
- Behavioral indicators such as correction and abandonment rates
- Technical metrics such as response time and cost

The difference between evaluation and monitoring lies in timing and data basis. Evaluation tests before approval against a fixed test set with known answers. Monitoring observes real requests in operation, for which no answer is known in advance. The two are connected: conspicuous cases from monitoring move into the test set as new test cases.

The escalation logic is decisive. Each indicator has a threshold and an accountable role, together with a response time. If quality falls below the threshold, a defined fallback applies, such as routing cases back to manual processing. The results flow monthly into reporting on the AI portfolio, so that the executive team sees quality and cost side by side. [LLMOps: running AI applications](/blog/llmops-running-ai-applications/) describes technical operations.

## Ownership: who is accountable for quality

In practice, clarity of roles determines whether quality assurance succeeds. Four roles cover the control loop completely:

- Business unit: defines the quality profile, assesses the test set and checks samples
- Product owner: owns the test set and approval as well as the prioritization of improvements
- Technical team: runs test runs and operates monitoring
- Delivery management: steers the control loop and reports to the executive team

This assignment belongs in the operational handover of the application. It thereby remains stable when staff change.

**Approaches to quality assurance compared**
Criterion | One-off acceptance | Ad hoc checks | With Niehues Consulting
Yardstick | Set by the project team | Set case by case | Quality profile of the business unit with thresholds
Test set | A few examples from development | Changing individual cases | Representative test set, sized by calculation
Changes | Adopted unchecked | Checked by sample | Regression run before every approval
Operation | No ongoing measurement | Individual feedback | Monitoring with escalation and fallback
Accountability | Ends with the project | Dispersed | Named roles, anchored in operations

## How Niehues Consulting supports companies

[Niehues Consulting](/) sets up this control loop along the Niehues Delivery Model with the phases Understand, Build and Implement. Marin Niehues works as a management consultant from a delivery perspective and brings AI into operations, in projects ranging from VC-funded scale-ups to DAX 40 corporations, with assignments in 20+ countries across six continents.

For quality assurance, three of the model's review questions matter most. What is the baseline, and how do we measure success? Does the prototype prove the benefit in the real workflow with real data? Are users enabled and processes adapted? In Understand, this produces the business case and quality profile for each application. In Build, prototypes are tested in the real workflow against the test set, so that the benefit is proven before investment in operations. In Implement, the business units take over roles and monitoring, and Niehues Consulting steers the control loop as delivery manager until responsibilities are anchored in the organization. [Delivering AI solutions](/blog/ai-solution-delivery-prototype-to-product/) describes the transfer from prototype to operation.

**What decision-makers should settle**
The quality of AI outputs is managed like any other delivery performance: with a yardstick and approval, ongoing measurement and clear escalation. Companies that settle these elements for each application can switch models and extend functions without losing reliability. The next step is a quality profile for the application with the greatest benefit. The [free initial consultation](/#kontakt) offers a conversation on this.

**Sources**
- Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS, 2023

## Frequently asked questions

### How do you test AI outputs?

AI outputs are tested with a test set of real requests from the business unit, assessed by subject-matter experts. Automated checks and assessment by a second model complement the procedure. Approval is granted only when all quality dimensions reach their thresholds.

### How do you measure the quality of a language model?

The quality of a language model is measured through dimensions such as factual accuracy and completeness, each with a measurement rule and threshold. An average hides weaknesses, so results are reported per dimension.

### How large should an evaluation test set be?

For the start, 50 to 200 carefully selected cases are enough; for critical applications, several hundred. With 100 cases and a 90 percent hit rate, the statistical uncertainty is around 6 percentage points, with 400 cases around 3. If the threshold is closer to the measured rate, the test set needs more cases.

### What is LLM-as-a-judge?

LLM-as-a-judge refers to the assessment of AI answers by a second language model against a fixed assessment guide. The method scales the review of content and tone, but shows known biases, for example in favor of longer answers. It is therefore calibrated against the judgment of subject-matter experts.

### What is the difference between evaluation and monitoring?

Evaluation tests against a fixed test set with known answers before approval, while monitoring observes real requests in operation. Conspicuous cases from monitoring become new test cases. Both belong to the same control loop.

### Which tools are used for LLM evaluation?

Common choices are open testing frameworks such as promptfoo, specialized libraries such as Ragas for applications with a knowledge base, and platforms such as Langfuse or MLflow. The evaluation features of the major cloud providers complement them. The selection follows the application's quality profile.

### How often does an AI application need to be retested?

An AI application is retested with every change to model, instructions or knowledge base. In operation, monitoring also runs continuously, with a monthly review for steering.

### Who is responsible for the quality of an AI application?

The business unit defines quality, and the product owner owns the test set and approval. Delivery management steers the control loop as a whole and reports to the executive team.
