# AI in Software Development: Assessing Productivity Realistically

Author: Marin Niehues, Niehues Consulting · Published: 2025-08-11 · Updated: 2026-11-16 · AI Adoption & Operations
Source: https://www.niehuesconsulting.com/blog/ai-in-software-development-productivity/

AI assistants speed up individual tasks in software development, but whether a company delivers faster only shows when the entire delivery flow is measured. Many organizations still judge the effect by activity data such as the acceptance rate of code suggestions. That data describes how a tool is used and says little about business value. This article reviews the research, presents a four-level measurement framework and describes the typical shift of bottlenecks.

## How much more productive does AI make software developers?

On clearly defined tasks, developers with AI assistants work considerably faster, less of that gain survives across the whole delivery flow, and for experienced developers in large, familiar codebases the effect can even turn negative. The research shows this range clearly.

In a controlled experiment, developers solved a well-defined programming task around 56 percent faster with a code assistant than the control group. Field experiments with several thousand developers at three companies found around 26 percent more completed tasks in daily work, with the largest gains among less experienced staff. A randomised study of experienced open-source developers working on their own large projects, by contrast, measured 19 percent longer completion times, although participants believed they had been faster.

**Exhibit 1: Code assistants have the strongest effect on well-defined tasks, and in large codebases of one's own the effect reverses**

| Label | Value |
| --- | --- |
| Well-defined task, controlled experiment | 56% faster |
| Daily work, field experiments at three companies | 26% more tasks |
| Experienced developers, own large codebases | 19% more time |

Source: Peng, Kalliamvakou, Cihon, Demirer, arXiv, 2023; Cui, Demirer, Jaffe, Musolff, Peng, Salz, Working Paper, 2024; Becker, Rush, Barnes, Rein, METR, 2025
Note: The studies measure different things: speed on one task, completed tasks in daily work, completion time; negative means more time with AI
Graphic: [SVG](https://www.niehuesconsulting.com/blog/ai-in-software-development-productivity/grafik-1.svg) · [CSV](https://www.niehuesconsulting.com/blog/ai-in-software-development-productivity/grafik-1.csv) · [JSON](https://www.niehuesconsulting.com/blog/ai-in-software-development-productivity/grafik-1.json) · Niehues Consulting, [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/deed.en). Data: Peng, Kalliamvakou, Cihon, Demirer, arXiv, 2023; the source's terms apply.

This gives executives a rule. The clearer the task and the less context it requires, the larger the effect. The more the work depends on knowledge of a mature system, the smaller it becomes. Only your own baseline shows your own value.

## Which AI tools are there for software development?

AI tools for software development fall into three classes: autocompletion in the editor, chat assistants with access to the code, and agents that carry out changes across several files on their own. Widely used examples are GitHub Copilot, the Cursor editor and Claude Code as an agent in the terminal. Many products now cover several classes.

The classes differ in how much steering they need. Autocompletion changes individual lines that the developer sees immediately. Agents generate entire change sets whose review demands more time and experience. The greater the autonomy, the more important review rules and automated tests become.

Code assistants have the clearest effect on tasks with a defined scope and a familiar pattern:

- Boilerplate code and scaffolding for new components
- Test cases for existing code
- Documentation and explanation of unfamiliar codebases
- Migrations between versions or frameworks

An organization that wants to assess the value therefore first defines the task classes for which it uses the tools. The overall effect results from the effect per task class and its share of a team's actual work.

Vibe coding is a different matter. The term describes an approach in which software is created largely through natural-language instructions and the generated code is barely read. For prototypes and throwaway internal tools it is a fast route. For software that goes into production, review and tests with clear ownership still apply.

## Is GitHub Copilot worth it for companies?

A code assistant license pays off for a company as soon as it saves more working time per developer per month than it costs, and at usual license prices that threshold is a few hours or less. The real question is therefore whether the time saved reaches the delivery flow.

**Calculation for the license decision**
- Step 1: determine license costs per developer per month, including rollout and training spread over twelve months.
- Step 2: set the fully loaded cost of a developer hour, meaning salary with on-costs and workplace.
- Step 3: calculate the threshold: license costs divided by hourly rate gives the hours that must be saved each month.
- Step 4: measure in the pilot whether lead time and throughput improve while the change failure rate stays stable.
- Step 5: subtract additional costs for longer reviews and rework before extrapolating the benefit.

The calculation shows that the license is rarely the risk. What becomes expensive is use that generates more code than the team can review, pushing defects into operations.

## Why activity metrics distort the assessment

The first evaluation usually relies on data the tool itself supplies: the acceptance rate of suggestions and the number of lines generated. These metrics are immediately available and seem plausible. They measure activity, however. A high acceptance rate can indicate useful suggestions, and just as well a cursory review. More generated code enlarges the codebase that later has to be read, tested and maintained. A team can double its line output and still deliver later if reviews take longer and rework rises.

Measurement belongs in the systems where the work is tracked: the ticket system, the repository and the release process. How to set it up is described in the article [Steering delivery performance: lead time, schedule reliability, quality](/blog/managing-delivery-performance/).

## A measurement framework with four levels

A robust judgment combines four levels. The higher the level, the closer it is to the business result and the more meaningful it is for an investment decision.

**The four levels of assessment**
- Usage: how many developers use the tool regularly, and for which task classes? This level shows adoption, not yet impact.
- Flow: how do lead time for changes, waiting time before review and deployment frequency develop?
- Quality: how do the change failure rate and the effort for rework change?
- Impact: how quickly do features reach users, and what do they contribute to the business result?

Flow and quality always belong together, as the DORA metrics intend. A shorter lead time with a rising change failure rate shifts effort into operations. This is exactly the pattern described by the annual DORA research on software delivery, which measured declining delivery stability as AI use increased. Conversely, a falling failure rate with longer lead times shows that review has become the bottleneck.

**Productivity in software development shows in lead time and quality, measured through to the effect on users. Usage data complements this view and does not replace it.**

## The shift of bottlenecks in the delivery flow

Writing code is only part of development work. The larger part goes into requirements, coordination, review, testing and operations. When AI assistants speed up code generation, the constraint moves to these points in the flow.

A thought experiment makes the mechanism visible. A team submits twice as many changes for review as before but has the same number of experienced reviewers. The waiting time before review grows, the lead time per change rises, and the gain in generation speed evaporates before the next step.

This gives steering a sequence. First the current bottleneck is identified, then capacity or automation is built up there, and only then does further acceleration of generation pay off. Typical measures are smaller change units, clear review ownership and automated tests.

**Two ways of assessing**
Criterion | Activity measurement | Flow measurement | With Niehues Consulting
Data source | Tool telemetry | Ticket and repository data | Linked data from both sources with a defined baseline
Value for the investment decision | Limited, shows usage | High, shows impact in the process | Decision paper with a business case per task class
Risk of mis-steering | High acceptance rate counts as success | Bottlenecks become visible | Bottlenecks are resolved before they consume the gain

## Approach: from baseline to decision

A robust assessment starts before the broad rollout and ends with a decision on the scope and sequence of further investment. The overall route from pilot to operations is set out in the article [From pilot to scale](/blog/from-pilot-to-scale-ai-operations/).

- Establish a baseline: lead time, deployment frequency and change failure rate over recent quarters form the comparison value. If this data is missing, the baseline is built in the first weeks of the pilot.
- Define task classes: teams specify what they use the tools for, so that the effect remains attributable.
- Form comparison groups: a pilot team with the tool and a comparable team without it show which part of the change is due to AI and which to other factors such as staff changes.
- Measure for long enough: with an existing baseline, a first picture emerges after six to eight weeks, and a robust decision after two to three quarters, once defects after release are visible.
- Decide: based on the four levels, the business case per task class is drawn up and expansion or adjustment is decided.

## Niehues Consulting as a partner for assessment and introduction

[Niehues Consulting](/) takes AI in software development from first trial to reliable regular operations. Business analysis produces the business case with a baseline and the selection of task classes. Prototypes prove the value in the real development workflow, and the transfer into operations follows with internal or external teams and enabled participants. In delivery management and organization design, Niehues Consulting sets the review paths and decision rights that stop the bottleneck from merely moving.

The approach follows the Niehues Delivery Model with its phases of understanding, building and introducing. Three of its review questions matter most for assessing AI in development. What is the baseline, and how do we measure success? Has the business case been confirmed with real values from the pilot? Which metric shows after 90 days that the solution is landing? Marin Niehues builds organizations that deliver, in projects ranging from VC-funded scale-ups to DAX 40 corporations and in 20+ countries on six continents. The entry point is a free initial consultation of 30 minutes, which you can book via [this contact](/#kontakt).

**Recommendation for the executive team**
Assess AI in software development by lead time and quality across the entire delivery flow. Establish a baseline before scaling up, compare teams with and without the tool and resolve the current bottleneck before accelerating generation further. This way, the investment decision rests on your own data.

**Sources**
- Becker, Rush, Barnes, Rein: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, METR, 2025
- Cui, Demirer, Jaffe, Musolff, Peng, Salz: The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers, Working Paper, 2024
- Peng, Kalliamvakou, Cihon, Demirer: The Impact of AI on Developer Productivity: Evidence from GitHub Copilot, arXiv, 2023

## Frequently asked questions

### How much does AI increase developer productivity?

On well-defined tasks, studies measure time savings of up to around 56 percent, in daily work around 26 percent more completed tasks, but also a slowdown for experienced developers in their own large codebases. Less of the gain survives across the whole delivery flow than at task level. Your own effect shows in comparing lead time and quality before and after introduction.

### Which metrics are suitable for assessing AI coding assistants?

Suitable metrics come from the delivery flow: lead time to production, waiting time before review, deployment frequency and change failure rate. Usage data such as the acceptance rate complements this view. Flow and quality are always considered together.

### How long does it take to measure the effect reliably?

With an existing baseline, a first picture emerges after six to eight weeks, and a robust decision after two to three quarters. The benefit develops as teams gain practice, and quality effects only become visible after release. Without a baseline, measurement takes longer by the time needed to build one.

### What are the best AI coding tools?

Widely used AI coding tools are GitHub Copilot, the AI editor Cursor and Claude Code as an agent in the terminal. They range from autocompletion to autonomous changes across several files. What matters in selection is how well a tool fits the company's review process and permissions model.

### Is GitHub Copilot worth it for businesses?

GitHub Copilot is worth it for businesses when the time saved reaches the delivery flow, because the license costs are already covered by a few hours saved per developer per month. Review capacity and a stable change failure rate are decisive. A pilot with a comparison group shows whether both are in place.

### What is vibe coding?

Vibe coding is an approach in which software is created largely through natural-language instructions to an AI and the generated code is barely read. The term was coined in early 2025 by the AI researcher Andrej Karpathy. It is a fast route for prototypes, while review and tests remain mandatory for production software.

### Will AI replace software developers?

AI replaces individual activities in software development, while responsibility for architecture and operations stays with people. As the share of generated code grows, so does the review effort, and that requires experienced developers.
