AI Adoption & Operations

AI in Software Development: Assessing Productivity Realistically

Marin Niehues11 August 202511 min read

Welding robots on a car body

AI assistants speed up individual tasks in software development, but whether a company delivers faster only shows when the entire delivery flow is measured. Many organizations still judge the effect by activity data such as the acceptance rate of code suggestions. That data describes how a tool is used and says little about business value. This article reviews the research, presents a four-level measurement framework and describes the typical shift of bottlenecks.

How much more productive does AI make software developers?

On clearly defined tasks, developers with AI assistants work considerably faster, less of that gain survives across the whole delivery flow, and for experienced developers in large, familiar codebases the effect can even turn negative. The research shows this range clearly.

In a controlled experiment, developers solved a well-defined programming task around 56 percent faster with a code assistant than the control group. Field experiments with several thousand developers at three companies found around 26 percent more completed tasks in daily work, with the largest gains among less experienced staff. A randomised study of experienced open-source developers working on their own large projects, by contrast, measured 19 percent longer completion times, although participants believed they had been faster.

Exhibit 1

Code assistants have the strongest effect on well-defined tasks, and in large codebases of one's own the effect reverses

Code assistants have the strongest effect on well-defined tasks, and in large codebases of one's own the effect reversesCode assistants have the strongest effect on well-defined tasks, and in large codebases of one's own the effect reverses. Bar chart. 3 values. Highest: Well-defined task, controlled experiment (56% faster). Lowest: Experienced developers, own large codebases (19% more time).Well-defined task, controlledexperiment56% fasterDaily work, field experiments at threecompanies26% more tasksExperienced developers, own largecodebases19% more timeCode assistants have the strongest effect on well-defined tasks, and in large codebases of one's own the effect reversesCode assistants have the strongest effect on well-defined tasks, and in large codebases of one's own the effect reverses. Bar chart. 3 values. Highest: Well-defined task, controlled experiment (56% faster). Lowest: Experienced developers, own large codebases (19% more time).Well-defined task, controlled experiment56% fasterDaily work, field experiments at threecompanies26% more tasksExperienced developers, own largecodebases19% more time
Source: Peng, Kalliamvakou, Cihon, Demirer, arXiv, 2023; Cui, Demirer, Jaffe, Musolff, Peng, Salz, Working Paper, 2024; Becker, Rush, Barnes, Rein, METR, 2025
Note: The studies measure different things: speed on one task, completed tasks in daily work, completion time; negative means more time with AI
Data and download
Download
Data as table
LabelValue
Well-defined task, controlled experiment56% faster
Daily work, field experiments at three companies26% more tasks
Experienced developers, own large codebases19% more time

Chart: Niehues Consulting, CC BY 4.0. Data: Peng, Kalliamvakou, Cihon, Demirer, arXiv, 2023; the source's terms apply.

This gives executives a rule. The clearer the task and the less context it requires, the larger the effect. The more the work depends on knowledge of a mature system, the smaller it becomes. Only your own baseline shows your own value.

Which AI tools are there for software development?

AI tools for software development fall into three classes: autocompletion in the editor, chat assistants with access to the code, and agents that carry out changes across several files on their own. Widely used examples are GitHub Copilot, the Cursor editor and Claude Code as an agent in the terminal. Many products now cover several classes.

The classes differ in how much steering they need. Autocompletion changes individual lines that the developer sees immediately. Agents generate entire change sets whose review demands more time and experience. The greater the autonomy, the more important review rules and automated tests become.

Code assistants have the clearest effect on tasks with a defined scope and a familiar pattern:

  • Boilerplate code and scaffolding for new components
  • Test cases for existing code
  • Documentation and explanation of unfamiliar codebases
  • Migrations between versions or frameworks

An organization that wants to assess the value therefore first defines the task classes for which it uses the tools. The overall effect results from the effect per task class and its share of a team's actual work.

Vibe coding is a different matter. The term describes an approach in which software is created largely through natural-language instructions and the generated code is barely read. For prototypes and throwaway internal tools it is a fast route. For software that goes into production, review and tests with clear ownership still apply.

Is GitHub Copilot worth it for companies?

A code assistant license pays off for a company as soon as it saves more working time per developer per month than it costs, and at usual license prices that threshold is a few hours or less. The real question is therefore whether the time saved reaches the delivery flow.

The calculation shows that the license is rarely the risk. What becomes expensive is use that generates more code than the team can review, pushing defects into operations.

Why activity metrics distort the assessment

The first evaluation usually relies on data the tool itself supplies: the acceptance rate of suggestions and the number of lines generated. These metrics are immediately available and seem plausible. They measure activity, however. A high acceptance rate can indicate useful suggestions, and just as well a cursory review. More generated code enlarges the codebase that later has to be read, tested and maintained. A team can double its line output and still deliver later if reviews take longer and rework rises.

Measurement belongs in the systems where the work is tracked: the ticket system, the repository and the release process. How to set it up is described in the article Steering delivery performance: lead time, schedule reliability, quality.

A measurement framework with four levels

A robust judgment combines four levels. The higher the level, the closer it is to the business result and the more meaningful it is for an investment decision.

Flow and quality always belong together, as the DORA metrics intend. A shorter lead time with a rising change failure rate shifts effort into operations. This is exactly the pattern described by the annual DORA research on software delivery, which measured declining delivery stability as AI use increased. Conversely, a falling failure rate with longer lead times shows that review has become the bottleneck.

The shift of bottlenecks in the delivery flow

Writing code is only part of development work. The larger part goes into requirements, coordination, review, testing and operations. When AI assistants speed up code generation, the constraint moves to these points in the flow.

A thought experiment makes the mechanism visible. A team submits twice as many changes for review as before but has the same number of experienced reviewers. The waiting time before review grows, the lead time per change rises, and the gain in generation speed evaporates before the next step.

This gives steering a sequence. First the current bottleneck is identified, then capacity or automation is built up there, and only then does further acceleration of generation pay off. Typical measures are smaller change units, clear review ownership and automated tests.

Approach: from baseline to decision

A robust assessment starts before the broad rollout and ends with a decision on the scope and sequence of further investment. The overall route from pilot to operations is set out in the article From pilot to scale.

  • Establish a baseline: lead time, deployment frequency and change failure rate over recent quarters form the comparison value. If this data is missing, the baseline is built in the first weeks of the pilot.
  • Define task classes: teams specify what they use the tools for, so that the effect remains attributable.
  • Form comparison groups: a pilot team with the tool and a comparable team without it show which part of the change is due to AI and which to other factors such as staff changes.
  • Measure for long enough: with an existing baseline, a first picture emerges after six to eight weeks, and a robust decision after two to three quarters, once defects after release are visible.
  • Decide: based on the four levels, the business case per task class is drawn up and expansion or adjustment is decided.

Niehues Consulting as a partner for assessment and introduction

Niehues Consulting takes AI in software development from first trial to reliable regular operations. Business analysis produces the business case with a baseline and the selection of task classes. Prototypes prove the value in the real development workflow, and the transfer into operations follows with internal or external teams and enabled participants. In delivery management and organization design, Niehues Consulting sets the review paths and decision rights that stop the bottleneck from merely moving.

The approach follows the Niehues Delivery Model with its phases of understanding, building and introducing. Three of its review questions matter most for assessing AI in development. What is the baseline, and how do we measure success? Has the business case been confirmed with real values from the pilot? Which metric shows after 90 days that the solution is landing? Marin Niehues builds organizations that deliver, in projects ranging from VC-funded scale-ups to DAX 40 corporations and in 20+ countries on six continents. The entry point is a free initial consultation of 30 minutes, which you can book via this contact.

Frequently asked questions

How much does AI increase developer productivity?

On well-defined tasks, studies measure time savings of up to around 56 percent, in daily work around 26 percent more completed tasks, but also a slowdown for experienced developers in their own large codebases. Less of the gain survives across the whole delivery flow than at task level. Your own effect shows in comparing lead time and quality before and after introduction.

Which metrics are suitable for assessing AI coding assistants?

Suitable metrics come from the delivery flow: lead time to production, waiting time before review, deployment frequency and change failure rate. Usage data such as the acceptance rate complements this view. Flow and quality are always considered together.

How long does it take to measure the effect reliably?

With an existing baseline, a first picture emerges after six to eight weeks, and a robust decision after two to three quarters. The benefit develops as teams gain practice, and quality effects only become visible after release. Without a baseline, measurement takes longer by the time needed to build one.

What are the best AI coding tools?

Widely used AI coding tools are GitHub Copilot, the AI editor Cursor and Claude Code as an agent in the terminal. They range from autocompletion to autonomous changes across several files. What matters in selection is how well a tool fits the company's review process and permissions model.

Is GitHub Copilot worth it for businesses?

GitHub Copilot is worth it for businesses when the time saved reaches the delivery flow, because the license costs are already covered by a few hours saved per developer per month. Review capacity and a stable change failure rate are decisive. A pilot with a comparison group shows whether both are in place.

What is vibe coding?

Vibe coding is an approach in which software is created largely through natural-language instructions to an AI and the generated code is barely read. The term was coined in early 2025 by the AI researcher Andrej Karpathy. It is a fast route for prototypes, while review and tests remain mandatory for production software.

Will AI replace software developers?

AI replaces individual activities in software development, while responsibility for architecture and operations stays with people. As the share of generated code grows, so does the review effort, and that requires experienced developers.

Let’s talk about your initiative.

In 30 minutes we clarify where your organization stands and which step will pay off most now.

Request a consultation

30 minutes · free · no obligation