AI Adoption & Operations

Human in the Loop and Human on the Loop: How Much Human Oversight AI Needs in Operation

Marin Niehues27 April 202612 min read

Marin Niehues with a headset, pointing to a presentation slide in front of a seated audience.

Human in the loop means that a person reviews and approves every AI output before it takes effect. With human on the loop the system acts on its own, while a person monitors operation and intervenes when something looks wrong. The choice of form follows the consequences of errors and their reversibility, along with volume. This article provides an overview and does not constitute legal advice.

What does human in the loop mean?

Human in the loop means that an AI system produces proposals and a person decides before any effect whether to accept, amend or reject them. The AI prepares, the person decides and is accountable.

Typical examples are a draft reply in customer service that an agent reviews before sending, a pre-filled invoice entry that an accountant confirms, or an anomaly in an inspection report that an expert assesses. In each case no output reaches customers or systems without human approval.

The term comes from control engineering and the debate on autonomous systems, which distinguishes three levels: the human in the loop, the human on the loop and the human out of the loop. For companies it turns the question of autonomy into a question of roles.

What is the difference between human in the loop and human on the loop?

The difference lies in when the human decides: with human in the loop the person decides before every single effect, with human on the loop the system acts itself and the person monitors ongoing operation with the right and the means to intervene at any time.

Human on the loop suits high-volume processes in which individual approval would consume the benefit. One example is the automatic classification and routing of incoming emails. A specialist monitors metrics such as error rate and escalations, reviews samples and stops the system when it deviates. With human out of the loop the system runs without ongoing monitoring. This is only justifiable where errors have minor consequences and are easy to correct.

Automation research adds an important distinction. A widely cited model describes four functions a system can take over: acquiring information, analyzing information, selecting a decision and implementing the action. The degree of automation can be set separately for each function, so an AI can prioritize cases on its own while the decision remains with the person.

Why a human in the loop alone does not protect

A human in the loop only protects if the person actually reviews, and research shows that people systematically follow a system's proposals too readily. This effect is called automation bias.

A study with 27 radiologists who assessed mammograms with the support of an AI system shows the scale. When the system suggested the correct category, inexperienced readers were right in around 80 percent of cases. When it suggested a wrong category, their accuracy fell below 20 percent. Very experienced readers were affected too: their accuracy dropped from 82 to 46 percent. A systematic review of 74 studies on decision support names the drivers: time pressure and high workload amplify the effect, as do complex tasks and high trust in the system.

Exhibit 1

A wrong AI suggestion cut even very experienced readers' accuracy from 82 to 46 percent

A wrong AI suggestion cut even very experienced readers' accuracy from 82 to 46 percentA wrong AI suggestion cut even very experienced readers' accuracy from 82 to 46 percent. Before and after comparison. Very experienced readers: 82% → 46%.AI suggestion correctAI suggestion wrongVery experienced readers82%46%A wrong AI suggestion cut even very experienced readers' accuracy from 82 to 46 percentA wrong AI suggestion cut even very experienced readers' accuracy from 82 to 46 percent. Before and after comparison. Very experienced readers: 82% → 46%.AI suggestion correctAI suggestion wrongVery experienced readers82%46%
Source: Dratsch et al., Automation Bias in Mammography, Radiology, 2023
Note: Inexperienced readers were right in around 80 percent of cases with a correct suggestion and in under 20 percent with a wrong one
Data and download
Download
Data as table
LabelAI suggestion correctAI suggestion wrongChange
Very experienced readers82%46%−36%

Chart: Niehues Consulting, CC BY 4.0. Data: Dratsch et al., Automation Bias in Mammography, Radiology, 2023; the source's terms apply.

A field experiment with recruiters shows a second side. Those working with a particularly good AI system invested less attention in their own assessment and followed its recommendations unchecked more often. Automation research has long described this pattern: people who only monitor lose practice and notice deviations later.

For decision-makers this yields a clear rule. Approval by a single click is not oversight. It becomes effective through design: review time built into the workflow, a duty to give reasons for approvals with significant impact, and measurement of whether reviewers actually deviate when the system is wrong. How to assure the quality of the outputs themselves is covered in Assuring the quality of AI outputs.

Which form of oversight suits which AI process?

The right form of oversight follows from four criteria: consequences of errors, reversibility, volume and reviewability. The more severe and lasting an error, the closer the decision moves to the human.

Clear thresholds apply to the classification. For decisions about people with legal or similarly significant effects, such as on job applications or loans, human in the loop applies with a documented individual decision. For processes with external effects that cannot be recalled, such as customer letters containing commitments or triggered payments, human in the loop applies until the error rate stays stably below the agreed threshold for several months. For high-volume internal processes that can be corrected quickly, such as routing or tagging, human on the loop with sampling is sufficient.

What do the AI Act and the GDPR require of human oversight?

The European AI Act requires effective human oversight for high-risk systems, and the General Data Protection Regulation protects individuals from decisions based solely on automated processing with legal or similarly significant effects. Both frameworks presuppose oversight that can actually intervene.

The AI Act describes what effective oversight must achieve: the persons responsible understand the capabilities and limits of the system, interpret its outputs correctly and can override or halt it. Notably, the regulation explicitly names the tendency to rely excessively on a system's outputs. Companies deploying high-risk systems assign oversight to persons with the necessary competence and authority. Which systems count as high-risk is covered in High-risk AI under the AI Act.

The GDPR gives individuals the right not to be subject to a decision based solely on automated processing that significantly affects them, with narrowly defined exceptions. Human involvement only counts if the person reviews the output on its merits and has the authority to decide differently. A formal confirmation without review does not meet this requirement. How responsibility and liability are distributed when errors occur is covered in Liability for AI errors.

What does human oversight cost?

Human oversight costs review time, and this review time belongs in the business case of every AI application. Leaving it out overstates the benefit.

The calculation is simple. Number of cases times review rate times review minutes per case gives the annual review time. Multiplied by the fully loaded hourly rate, this gives the review cost, which is deducted from the gross benefit.

Exhibit 2 · Schematic

With a five percent sample, review cost in the example falls from 120,000 to 12,000 euros

With a five percent sample, review cost in the example falls from 120,000 to 12,000 eurosWith a five percent sample, review cost in the example falls from 120,000 to 12,000 euros. Bar chart. 2 values. Highest: Human in the loop, every output reviewed (120,000 euros a year). Lowest: Human on the loop, 5% sample (12,000 euros a year). Schematic illustration, not measured data.Human in the loop, every outputreviewed120,000 euros a yearHuman on the loop, 5% sample12,000 euros a yearWith a five percent sample, review cost in the example falls from 120,000 to 12,000 eurosWith a five percent sample, review cost in the example falls from 120,000 to 12,000 euros. Bar chart. 2 values. Highest: Human in the loop, every output reviewed (120,000 euros a year). Lowest: Human on the loop, 5% sample (12,000 euros a year). Schematic illustration, not measured data.Human in the loop, every output reviewed120,000 euros a yearHuman on the loop, 5% sample12,000 euros a year
Schematic illustration, not measured data.
Source: Worked example in this article
Note: Calculation with assumed values from the article, 60,000 cases a year, excluding ongoing monitoring
Data and download
Download
Data as table
LabelValue
Human in the loop, every output reviewed120,000 euros a year
Human on the loop, 5% sample12,000 euros a year

Graphic and data: Niehues Consulting, CC BY 4.0. Free to use with attribution and a link.

In high-volume processes, the move from individual approval to monitored operation often decides whether an application pays off, provided the risk classification permits it.

Autonomy is earned step by step

The autonomy of an AI application grows in stages, and each stage is approved on the basis of measured quality. A maturity model with four stages makes this progression manageable.

Moving to the next stage is a decision with a named decision-maker and a threshold fixed in advance. The way back matters just as much: if the error rate rises, for instance after a model change or a change in input data, the application automatically falls back to the previous stage. This principle applies with particular rigour to AI agents that perform multi-step tasks and access tools. The overall steering of risk and portfolio is set out in the article AI management: what executive leadership must steer.

Niehues Consulting as a partner for oversight in operation

Niehues Consulting defines the human oversight of an AI application before the build and works according to the Niehues Delivery Model with the phases Understand, Build and Implement. Each phase ends with fixed review questions that decide on the next step.

Three of them matter for the form of oversight. In Understand: who decides, who delivers, who uses? In Build: which roles, decision paths and interfaces does operation require? In Implement: who owns operations from the first day? The answers produce the classification per process, the review roles with authority to stop and the metrics that show whether oversight is effective.

Founder Marin Niehues is a management consultant who works from the delivery perspective. He builds organizations that deliver and brings AI into operation, as delivery manager, project manager or product owner. His projects range from VC-funded scale-ups to DAX 40 corporations, in 20+ countries on six continents, with his home base in the DACH region. In a free 30-minute initial consultation you can establish with him which form of oversight your applications need.

Frequently asked questions

What is an example of human in the loop?

A typical example is a customer service agent who reviews and amends an AI-generated reply before sending it. Other examples are an accountant confirming AI-extracted invoice data or an expert assessing an anomaly flagged by a model. In each case nothing takes effect without human approval.

What does human out of the loop mean?

Human out of the loop means that an AI system works without ongoing human monitoring. People set rules and limits but do not intervene during operation. This form is only justifiable where errors have minor consequences and are easy to correct.

Does the EU AI Act require human in the loop?

The AI Act requires effective human oversight for high-risk systems but does not prescribe individual approval of every output. Oversight must be able to understand outputs, override them and halt the system. How it is designed follows the risk and the intended purpose.

Is clicking “approve” enough human oversight?

A click without substantive review is not sufficient oversight. Approval is effective when the reviewer has time to review, has the authority to decide differently and actually deviates when the system is wrong. The override rate makes this measurable.

What is the override rate?

The override rate is the share of AI outputs that reviewers amend or reject. If it is close to zero although samples reveal errors, reviewers are not reviewing effectively. It therefore belongs in the reporting of every application with human in the loop.

When can an AI application move from human in the loop to human on the loop?

A move is possible when the error rate stays stably below a threshold fixed in advance for several months and the process involves no decisions about people with significant effects. A named person decides on the move. If the error rate rises again, the application falls back to individual approval.

Let’s talk about your initiative.

In 30 minutes we clarify where your organization stands and which step will pay off most now.

Request a consultation

30 minutes · free · no obligation