# AI Organization: Central Team, Center of Excellence or Decentralized

Author: Marin Niehues, Niehues Consulting · Published: 2025-11-03 · Updated: 2026-11-16 · AI Adoption & Operations
Source: https://www.niehuesconsulting.com/blog/ai-organisation-central-team-centre-of-excellence/

An AI organization should be as central as standards and platform require, and as decentralized as proximity to the business unit demands. Where that line falls determines the speed and impact of every AI initiative in the company. This article describes the basic models, the criteria for choosing between them and the way a model proves itself in delivery.

The organizational question is one component of AI management as a whole. How it fits with accountability and portfolio steering is set out in the article [AI management: what executive leadership must steer](/blog/ai-management-what-executives-must-steer/).

## Which organizational models for AI exist?

Internationally, four organizational models for AI are distinguished: centralized, hub-and-spoke, federated and decentralized. They differ in where development takes place and who sets standards.

- Centralized: one team pools all AI expertise and builds and runs applications for the whole company.
- Hub-and-spoke, usually called a center of excellence: a small core unit sets methods and platform, while applications are built in the business units.
- Federated: several functions have their own AI teams, connected through a shared platform and common metrics.
- Decentralized: the functions develop independently, each with its own budget and specialists.

The centralized model quickly creates uniform standards and makes full use of scarce specialists. Its risk lies in distance. The team knows the workflow in sales or maintenance only from interviews, and the business units experience it as a queue.

The center of excellence deploys a small core unit that sets methods and platform and supports the business units during implementation. Applications are largely built where the benefit arises. The model combines standardization with proximity, but requires clear agreements on what the core prescribes and what the business unit decides itself. In practice the term is also used for purely central teams. Clarify in every discussion, therefore, whether the core builds itself or enables others.

The decentralized model creates high speed in individual cases and close proximity to the workflow. Without a shared platform, however, parallel solutions and duplicate licenses emerge alongside inconsistent quality, and the company loses sight of its AI portfolio. The federated model is the mature form between both poles: the functions deliver themselves, while the center steers the overall system.

**Every model shifts the burden between standardization and proximity to the business unit. The choice depends on which burden the company can handle better at its level of maturity.**

## Five criteria for choosing the model

The decision can be anchored in five questions that the executive team answers together with the function heads.

- Number and spread of use cases: the more cases arise across different functions, the greater the benefit of shared standards.
- Availability of specialists: where specialists are scarce, pooling is justified.
- Data ownership: if the data sits in the functions and belongs to them, development needs direct access on site.
- Maturity of the business units: functions with their own product and analytics capability can deliver themselves.
- Risk profile: applications with high demands on quality and control belong under shared review procedures.

The answers are rarely uniform. Sales may be mature enough for its own development while the finance function requires shared controls. The result is then a hybrid model, and the task is to draw its boundaries precisely. A rule of thumb from delivery practice: if fewer than two business units have their own product and data capability, start centrally. From three enabled functions with their own use cases, the move to a center of excellence pays off.

**The models at a glance**
Criterion | Purely central | Purely decentralized | With Niehues Consulting
Speed | Queue as demand grows | High in individual cases, slowed by duplication | Business units deliver themselves, the core removes obstacles
Standards and quality | Uniform | Inconsistent | Binding, set by the core and applied in the business unit
Proximity to the workflow | Low | High | High, through a product owner in the business unit
Portfolio overview | High | Low | High, through shared metrics
Scaling | Limited by team size | Limited by coordination effort | Grows with the number of enabled business units

## The model follows the company's maturity

In the entry phase there are few use cases, knowledge is scarce and every experience is valuable. A small central team delivers the first applications and in doing so builds platform and standards that later benefit all functions. The business units are involved from the outset through a product owner, so that the benefit is tested in the real workflow.

In the growth phase, demand rises faster than the team's capacity. Now the central team turns into a center of excellence. It sets methods and platform, while the business units deploy their own developers or external partners. The transition succeeds when the core has defined in advance which building blocks it provides and which decisions lie with the business unit.

In the mature phase, several functions work with their own AI teams that remain connected through platform and metrics as well as a fixed rhythm of portfolio steering. This is the federated model. The central unit then steers fewer individual projects and more the overall system. It owns the portfolio and the quality standards, and it decides which applications move into regular operation.

**When a change of model is due**
Three signals show that the organization is reaching its limit. The business units' waiting list grows over several quarters. Different functions buy the same capability from different vendors. Applications run without anyone being named responsible for their operation. Each of these signals can be captured with a few metrics and belongs in the executive team's regular reporting.

## Which roles does an AI team need?

An AI team needs six roles: lead, product owner, data engineering, machine learning engineering, platform and operations, and governance. In the entry phase a few people share these roles; in a center of excellence the product owner moves permanently into the business unit.

- Lead: owns portfolio and standards and reports to the responsible member of the executive team.
- Product owner: comes from the business unit, owns the benefit of the application and prioritizes requirements.
- Data engineering: opens up data sources and secures data quality.
- Machine learning or AI engineering: develops models and language model applications and takes them into production.
- Platform and operations: provides shared tooling and monitors applications in regular operation, often under the label MLOps or LLMOps.
- Governance: runs the approval procedure and the risk register and involves the relevant specialist functions.

Without a product owner, technically sound applications emerge that nobody uses. Without platform and operations, applications go unmaintained after the pilot. In delivery practice these are the two most common gaps.

## Should the AI team sit in IT or in the business?

The core unit belongs where the mandate over prioritization and benefit lies, and the platform belongs where integration and operations lie. In practice this usually means the core unit reports directly to a member of the executive team, the platform sits in IT and the product owner in the business unit.

Two criteria govern where the core unit sits. If the use cases focus on changing business processes, such as customer service or order handling, it belongs close to operations, with the COO or a Chief AI Officer. If the focus is integration into existing systems and a shared platform, IT under the CIO is the right home, with named product owners from the business units. Placing it two levels below the executive team works poorly, because it then lacks the mandate to set priorities across functions.

## Which AI operating model suits a mid-sized company?

For most mid-sized companies, a small central team with product owners in the business units and external partners for specialist tasks is the right fit. Official statistics show why the question is pressing: according to Germany's Federal Statistical Office, around 20 percent of companies with ten or more employees used AI in 2024, and 48 percent of those with 250 or more. Use is growing precisely where a company's own specialists are scarcest.

A mid-sized company with one line of business pools its scarce specialists centrally and buys in specialist expertise project by project. The roles described above are spread across two to four people, complemented by one product owner per business unit with an active initiative. Where the company has several independent business units with their own IT, a lean center of excellence that sets platform and policy is the better fit. A purely decentralized model rarely suits a mid-sized company, because it spreads scarce specialists too thinly.

## Clear accountability determines whether a model delivers

Even the best model fails if nobody knows who decides what. Overall accountability lies with the CEO, because companies where AI oversight sits at the top report the highest bottom-line contribution from their AI initiatives. Operationally, a member of the executive team with a mandate over budget and prioritization leads, depending on the set-up the COO, the CIO, a Chief Digital Officer or a Chief AI Officer. Below that, three determinations are needed in every variant.

- Prioritization: who decides which use cases receive budget? A standing portfolio forum chaired by the executive team ranks cases by value, feasibility and risk and concentrates the budget on a few business domains with the greatest leverage. The procedure is described in [KI-Use-Cases priorisieren](/blog/ai-use-case-prioritisation/).
- Platform: who is responsible for selecting and running the shared technology, and who approves deviations?
- Operations: who accepts an application, who supports it after launch, and who decides to switch it off?

These responsibilities belong in lean governance with clear decision paths. How to build it without it becoming a brake is shown in [KI-Governance aufbauen](/blog/ai-governance-framework/).

From a delivery perspective, what ultimately counts is whether an application is used in the business unit's workflow. Every model therefore needs a product owner in the business unit who owns the benefit, and a fixed handover point into operations. Two metrics reveal whether the organization delivers: the number of applications in regular operation and the usage rate in each business unit. An organization that reports many prototypes and few applications in operation has a handover problem at the interface to operations.

## Niehues Consulting as a partner for the AI organization

[Niehues Consulting](/) designs the AI organization from a delivery perspective. Founder Marin Niehues builds organizations that deliver and brings AI into operations. His projects range from VC-funded scale-ups to DAX 40 corporations, with assignments in 20+ countries on six continents.

The approach follows the Niehues Delivery Model with its phases Understand, Build and Introduce. For the AI organization, three of its review questions matter most. In Understand: who decides, who delivers, who uses? In Build: which roles, decision paths and interfaces does operation require? In Introduce: who owns operation from day one? The answers lead to the choice of model with the right metrics. In parallel, AI prototypes deliver benefit in the real workflow, so the model matures on real initiatives. Delivery management and organization design ensure that decisions reach day-to-day business.

The entry point is the free 30-minute initial consultation. It clarifies which model fits your company's starting position: [book an initial consultation](/#kontakt).

**Recommendation for the executive team**
Start with a small central core that defines platform and roles early. Shift development into the business units as soon as they can apply the standards. Measure success by the applications in regular operation. The model may change as the company matures, while responsibilities remain unambiguous at all times.

**Sources**
- Federal Statistical Office of Germany (Destatis): One in five companies uses artificial intelligence, press release, 2024

## Frequently asked questions

### What is the difference between a center of excellence and a central AI team?

A central AI team builds and runs the applications itself, whereas a center of excellence sets methods and platform and supports business units that build their own. The central team scales with its own capacity, the center of excellence with the number of business units it enables.

### What is a hub-and-spoke model for AI?

A hub-and-spoke model connects a central core unit, the hub, with AI capability in the business units, the spokes. The hub sets standards and runs the platform, while the spokes build applications close to the workflow. It is what most companies mean by a center of excellence.

### When does a decentralized AI model make sense?

A decentralized model fits when business units have their own development capability and data and their use cases are independent of one another. Even then, platform and portfolio steering should be organized jointly to avoid duplication and license sprawl.

### Who should own AI on the executive team?

Overall accountability lies with the CEO, while a member of the executive team with a mandate over budget and prioritization leads operationally. Depending on the set-up, that is the COO, the CIO, a Chief Digital Officer or a Chief AI Officer. What matters is that this person answers for the result in operation.

### Does a company need a Chief AI Officer?

A dedicated Chief AI Officer pays off from around ten parallel AI initiatives across several functions. Below that, a named member of the executive team with a small central team is sufficient. Mandate and metrics of the role are described in [Die Rolle des Chief AI Officer](/blog/chief-ai-officer-role/).

### How large should an AI team be?

An AI team is as large as the portfolio of running initiatives requires and starts with a few people covering the core roles. With every business unit that delivers itself, capacity grows in the functions rather than in the core. Sizing in detail is covered in the article on building a center of excellence.

### How can a company tell that its model no longer fits?

The clearest sign is a waiting list from the business units that grows over several quarters. Added to this are parallel purchases of the same capability and applications without a named operations owner. Tracking these indicators quarterly reveals the tipping point early and allows a planned adjustment.
