Ensuring the quality of AI outputs: testing, evaluation, monitoring

A company ensures the quality of AI outputs by defining quality before launch, testing every change against a test set and measuring continuously in operation. A language model does not always give the same answer to the same question, and providers release new model versions at short intervals and retire old ones. Quality therefore emerges from a control loop with clear ownership.
This article describes that loop from a delivery perspective: what must an organization put in place so that an AI application keeps producing reliable results? The article From pilot to scale sets out the full path from the first use case to regular operation.
Define quality before testing
A test needs a yardstick. In conventional software the specification provides it; for AI applications the business unit has to formulate it. The first step is therefore a quality profile, owned by the business unit and approved by delivery management. For each application it defines what makes a good result and from what point a result is no longer acceptable.
Five dimensions have proven effective:
- Factual accuracy against a verified source
- Completeness measured against the task
- Compliance with format and tone as well as company guidelines
- Behavior with unclear or inadmissible requests
- Response time and cost per case
Each dimension receives a measurement rule and a threshold. The threshold follows the risk of the use case: an assistant for internal research tolerates different margins from an application whose output goes to customers unchecked. For critical errors, such as a wrong deadline or a wrong amount, a separate threshold of zero occurrences in the test set applies.
How do you build a test set for AI?
A test set for AI is a collection of real requests from the business unit, each with an expected answer or an assessment standard, created and assessed by subject-matter experts. It contains typical and rare cases as well as deliberately difficult ones, such as ambiguous wording, contradictory sources or requests outside the application's remit.
The composition follows real demand. If complaints account for 40 percent of requests, they also make up about 40 percent of the test set, supplemented by a fixed group of difficult cases. Subject-matter experts create the test set, the development team runs it. That way nobody can adjust the yardstick to their own solution. Part of the cases is also held back and used only for approval, so that instructions are not optimized towards the known test cases.
How large does a test set for evaluation need to be?
For the start, 50 to 200 carefully selected cases are sufficient; for applications whose results go to customers unchecked, several hundred. The size follows a statistical consideration: the smaller the test set, the greater the uncertainty of the measured hit rate.
The same applies when comparing two versions. If two versions differ by two percentage points, a test set of 100 cases does not show this difference reliably. Because language models do not answer deterministically, each case also runs several times, and the rate across all runs is what counts.
Three methods of assessment
Organizations combine three methods of assessment. Automated checks capture everything that can be determined by rules, such as format, mandatory information or the presence of specific amounts.
The second method is called LLM-as-a-judge: a second language model assesses the answers against a fixed assessment guide. A study by researchers at UC Berkeley and other universities showed that a strong model acting as judge agreed with human preferences in over 80 percent of cases, about as often as humans agree with each other. The same study described typical biases, such as favoring the answer presented first or the longer one. The method therefore scales the assessment of content and tone, but is calibrated against human judgment before it feeds into approval.
Assessment by subject-matter experts remains the reference standard and continues permanently as a sample.
Approval is granted when all dimensions reach their thresholds. A single average is not enough, because it hides weak areas and rare but critical errors.
Testing every change: the regression run
An AI application changes more often than conventional software. The provider releases a new model, a team adjusts an instruction, the knowledge base grows. Each of these changes can improve results in one place and worsen them in another. The regression run therefore checks every change against the approved reference state: internal changes before approval, provider model changes before switching to the new version. For this, the application uses a pinned model version instead of an automatically updated model. The run reports deviations per dimension.
The product owner decides on approval on the basis of the run. A change without a test run counts as not approved. The rule belongs in release management, so that it applies without case-by-case decisions.
Which tools exist for evaluating AI applications?
There are three categories of tools for evaluating AI applications: open testing frameworks, platforms for traces and evaluation, and the evaluation features of the major cloud providers. Open frameworks such as promptfoo run test sets automatically and compare versions; for applications with a connected knowledge base there are specialized libraries such as Ragas. Platforms such as Langfuse or MLflow connect pre-launch evaluation with the recording of real requests in operation.
The selection follows the quality profile. First the dimensions and thresholds are settled, then the company decides which tool can measure them. A tool without a quality profile measures what is easy to measure and leaves out what matters.
Monitoring in operation
Even a passed test set only reflects the requests the team anticipated. In operation, applications encounter inputs nobody planned for and content that changes over time. Monitoring closes this gap with four groups of signals:
- User feedback
- Spot checks by subject-matter experts
- Behavioral indicators such as correction and abandonment rates
- Technical metrics such as response time and cost
The difference between evaluation and monitoring lies in timing and data basis. Evaluation tests before approval against a fixed test set with known answers. Monitoring observes real requests in operation, for which no answer is known in advance. The two are connected: conspicuous cases from monitoring move into the test set as new test cases.
The escalation logic is decisive. Each indicator has a threshold and an accountable role, together with a response time. If quality falls below the threshold, a defined fallback applies, such as routing cases back to manual processing. The results flow monthly into reporting on the AI portfolio, so that the executive team sees quality and cost side by side. LLMOps: running AI applications describes technical operations.
Ownership: who is accountable for quality
In practice, clarity of roles determines whether quality assurance succeeds. Four roles cover the control loop completely:
- Business unit: defines the quality profile, assesses the test set and checks samples
- Product owner: owns the test set and approval as well as the prioritization of improvements
- Technical team: runs test runs and operates monitoring
- Delivery management: steers the control loop and reports to the executive team
This assignment belongs in the operational handover of the application. It thereby remains stable when staff change.
How Niehues Consulting supports companies
Niehues Consulting sets up this control loop along the Niehues Delivery Model with the phases Understand, Build and Implement. Marin Niehues works as a management consultant from a delivery perspective and brings AI into operations, in projects ranging from VC-funded scale-ups to DAX 40 corporations, with assignments in 20+ countries across six continents.
For quality assurance, three of the model's review questions matter most. What is the baseline, and how do we measure success? Does the prototype prove the benefit in the real workflow with real data? Are users enabled and processes adapted? In Understand, this produces the business case and quality profile for each application. In Build, prototypes are tested in the real workflow against the test set, so that the benefit is proven before investment in operations. In Implement, the business units take over roles and monitoring, and Niehues Consulting steers the control loop as delivery manager until responsibilities are anchored in the organization. Delivering AI solutions describes the transfer from prototype to operation.
Frequently asked questions
How do you test AI outputs?
AI outputs are tested with a test set of real requests from the business unit, assessed by subject-matter experts. Automated checks and assessment by a second model complement the procedure. Approval is granted only when all quality dimensions reach their thresholds.
How do you measure the quality of a language model?
The quality of a language model is measured through dimensions such as factual accuracy and completeness, each with a measurement rule and threshold. An average hides weaknesses, so results are reported per dimension.
How large should an evaluation test set be?
For the start, 50 to 200 carefully selected cases are enough; for critical applications, several hundred. With 100 cases and a 90 percent hit rate, the statistical uncertainty is around 6 percentage points, with 400 cases around 3. If the threshold is closer to the measured rate, the test set needs more cases.
What is LLM-as-a-judge?
LLM-as-a-judge refers to the assessment of AI answers by a second language model against a fixed assessment guide. The method scales the review of content and tone, but shows known biases, for example in favor of longer answers. It is therefore calibrated against the judgment of subject-matter experts.
What is the difference between evaluation and monitoring?
Evaluation tests against a fixed test set with known answers before approval, while monitoring observes real requests in operation. Conspicuous cases from monitoring become new test cases. Both belong to the same control loop.
Which tools are used for LLM evaluation?
Common choices are open testing frameworks such as promptfoo, specialized libraries such as Ragas for applications with a knowledge base, and platforms such as Langfuse or MLflow. The evaluation features of the major cloud providers complement them. The selection follows the application's quality profile.
How often does an AI application need to be retested?
An AI application is retested with every change to model, instructions or knowledge base. In operation, monitoring also runs continuously, with a monthly review for steering.
Who is responsible for the quality of an AI application?
The business unit defines quality, and the product owner owns the test set and approval. Delivery management steers the control loop as a whole and reports to the executive team.
Let’s talk about your initiative.
In 30 minutes we clarify where your organization stands and which step will pay off most now.
Request a consultation30 minutes · free · no obligation



