Delivering Agentic AI: From Agent Prototype to Reliable Operation

An agent prototype becomes reliable operation when its task and ownership are defined before rollout and the agent proves its effect in the live process. Agentic AI refers to AI systems that plan steps themselves and call enterprise systems until the result is in place. That makes the prototype quick to demonstrate and operation demanding. This article sets out a four-step approach with which executives and business unit heads move an agent from trial into regular operation.
Where the article touches on obligations under the AI Act: This article provides an overview and does not constitute legal advice.
Why do AI agents fail between prototype and production?
AI agents fail between prototype and production because the prototype runs under conditions that do not exist in operation. It processes selected inputs under the eyes of its developers. In operation, unknown inputs meet changing users, systems change their interfaces, and nobody sees every item.
With an agent this gap widens, because it executes several steps in sequence. An error in the first step carries through the entire chain and often shows only in the result. If each of ten steps succeeds with a probability of 95 percent, the success rate of the chain is around 60 percent. Four organizational causes also recur in agent initiatives:
- The task is scoped too broadly and has no verifiable end.
- There is no success measure, so the benefit can be neither proven nor defended.
- Nobody in the business unit is accountable for the result.
- The running cost per item was never calculated.
An agent's ability to deliver is therefore decided in the organization around it. The model provides the ability to plan and act. Whether that becomes a result the business unit uses day to day is determined by how task and accountability are scoped and how control is organized. Where agents pay off first and how they differ from assistants and RPA is described in the article Agentic AI in the enterprise.
Steps 1 and 2: Scope the task narrowly, release autonomy in stages
The first step defines what the agent actually does. A suitable task has a recognizable end and a verifiable result that a business unit uses. Pre-qualifying incoming requests meets these requirements. General relief for case handlers does not, because neither the end nor the result is defined. Every task has a success measure fixed before the build, such as processing time per item or the share of items without rework.
The second step governs how much independence the agent receives. Autonomy is released in stages, with measurable criteria at each transition.
Promotion to the next level follows fixed criteria, such as an agreed error rate over at least four weeks and a falling share of escalations. Every agent starts at level 1 or 2, even if the prototype ran independently. The level depends on the damage an error causes in the process. Where an item moves money or reaches customers directly, human approval stays in place for longer.
Step 3: Measure quality before load arrives
The third step makes quality measurable. The business unit provides a test set of real cases with the expected result. For a narrowly scoped task, 100 to 200 cases are a sound start, deliberately including the difficult special cases. Every interim step of the agent and the overall result are measured, because only then can an error in the chain be located. How testing and monitoring fit together is shown in the article Assuring the quality of AI results.
The same test set runs again with every change, for example a model switch or an adjusted instruction. Providers release new model versions at short intervals and retire older ones. Anyone using an alias or forced to switch gets an agent that behaves differently from the one tested yesterday. Without a regression test, the deviation only surfaces in live operation, at worst with the customer.
In operation, monitoring takes over the same task continuously. Four metrics are enough to start:
- Success rate: share of items without rework
- Escalation rate: share of items passed to a person
- Lead time per item
- Cost per item including model usage
What does an AI agent cost per transaction?
The cost per item is the model usage plus the human review time for escalated cases. Model costs are usually the smaller share and review time the larger. A falling escalation rate therefore cuts costs more than a cheaper model does.
Exhibit 1 · Schematic
In the example an item handled by an AI agent costs around 1.06 euros, manual processing 12 euros
Source: worked example in the article
Note: example with assumed values from the article, excluding operation, monitoring and test-set maintenance
Insert the values from your own prototype, the assumptions in the box serve only as illustration. How to manage the running costs of AI applications over time is covered in the article LLMOps: running AI applications.
Step 4: Define operation and ownership
The fourth step names who runs the agent. An agent in regular operation needs a business owner accountable for objectives and results, and a technical team that looks after the platform and changes. In practice the business role is taken by a product owner who steers priorities and further development. The decision paths between the two are set down in writing, as is the escalation path for incidents.
Readiness for operation includes four safeguards:
- Versioning: model and instructions are versioned, and every change runs through the test set.
- Kill switch: the agent can be switched off at any time, and a manual fallback process is described and rehearsed.
- Log: every action of the agent is recorded traceably. For high-risk systems the AI Act explicitly requires this logging.
- Handover: an operations manual and the enablement of teams are in place as soon as internal or external teams take over operation.
This step is easily skipped while the prototype is showing results. Yet it decides whether the effect lasts once the development team turns to new tasks.
What are the risks of AI agents in production?
The two largest operational risks of an agent are wrong actions and prompt injection. A wrong action occurs when the agent acts on a faulty interim result, for example booking a credit note or sending an email to the wrong recipient. Prompt injection refers to attacks in which instructions are hidden in content the agent reads, such as an incoming email or a web page. The agent then treats them like instructions from its principal.
In 2023 Germany's Federal Office for Information Security warned that indirect prompt injection is a fundamental weakness of applications that connect language models with external content. In the OWASP Foundation's list of the ten most critical risks for language model applications, prompt injection ranks first. The Federal Office describes the weakness as inherent in the way the models work. Defense therefore focuses on the possible damage.
The technical defences are described in detail in the article AI security: prompt injection and new attack surfaces.
How long does it take to move from prototype to production?
For a narrowly scoped task with existing interfaces, the first level of operation can be reached within eight to twelve weeks. This planning value applies when task, success measure and test set are fixed at the outset. If interfaces are missing or the task is scoped broadly, the path lengthens by the time needed for integration and alignment. Plan the promotion to higher autonomy levels separately, as it follows the measured criteria. How the individual agent fits into the overall AI program is described in the article From pilot to scale.
How Niehues Consulting brings agents into operation
Niehues Consulting supports agent initiatives using the Niehues Delivery Model with its phases understand, build and introduce. The understand phase covers business analysis and the business case, as well as selecting the task on which an agent is first to prove what it can do. Three of the model's test questions decide readiness for operation. Does the prototype prove its value in the real process with real data? Is the business case confirmed with real values, including the cost per item? Who owns operation from day one?
The perspective throughout is that of delivery. Marin Niehues steers initiatives as delivery manager, project manager or product owner and transfers the agent into operation with internal or external teams, including processes and responsibilities as well as the enablement of those involved. Projects range from VC-funded scale-ups to DAX 40 groups, with assignments in 20+ countries on six continents. In the free 30-minute initial consultation you can clarify which task is the right place to start.
Frequently asked questions
What is agentic AI?
Agentic AI refers to AI systems that plan tasks across several steps on their own, call tools and systems to do so and deliver a result. The difference from an assistant lies in the capacity to act. The agent executes steps and does not merely formulate answers.
What is the difference between an AI agent and a chatbot?
A chatbot answers questions in dialogue, while an AI agent completes tasks and accesses enterprise systems to do so. The chatbot ends with an answer, the agent with a result in the process, such as a checked invoice. The agent therefore needs access rights, tests and an accountable role in the business unit.
How long does it take to move an AI agent from prototype to production?
For a narrowly scoped task with existing interfaces, the first level of operation can be reached within eight to twelve weeks. The precondition is that task, success measure and test set are fixed at the outset. Missing interfaces extend the path by the integration time.
How do you test an AI agent?
An AI agent is tested with a set of real cases from the business unit and the expected result for each. Every interim step and the overall result are measured. The same test set runs again as a regression test with every change to model or instructions.
What does an AI agent cost per transaction in production?
The cost per item consists of model usage and human review time for escalated cases. Review time is usually the larger item. A falling escalation rate therefore cuts costs more than a cheaper model does.
Which metrics show whether an agent works in production?
The success rate and the escalation rate are the informative metrics, complemented by lead time and cost per item. They are defined before rollout and reported continuously in operation.
Who is accountable for an AI agent in a company?
A business owner in the business unit is accountable for the result of an AI agent, in practice a product owner. A technical team looks after the platform and changes. Both roles and the escalation path are set down in writing before rollout.
How much autonomy should an AI agent get at the start?
An AI agent starts at level 1 or 2, meaning it proposes or executes only after human approval. Promotion follows measurable criteria such as an agreed error rate over at least four weeks. Where money moves or customers are reached, approval stays in place for longer.
Let’s talk about your initiative.
In 30 minutes we clarify where your organization stands and which step will pay off most now.
Request a consultation30 minutes · free · no obligation



