# Retrieval Augmented Generation: Connecting AI to Enterprise Knowledge

Author: Marin Niehues, Niehues Consulting · Published: 2025-05-26 · Updated: 2026-11-16 · AI Adoption & Operations
Source: https://www.niehuesconsulting.com/blog/retrieval-augmented-generation-enterprise/

Retrieval Augmented Generation (RAG) connects a language model to enterprise knowledge: before each answer, the system searches for suitable sources and has the model formulate its reply from them. For executives and business heads, delivery is the central question. Which knowledge should the system use, who is accountable for its quality, and how does the organization know that answers are reliable? This article explains the architecture, distinguishes RAG from long context and fine-tuning, sets out cost and duration and describes an approach through to regular operation. This article provides an overview and does not constitute legal advice.

**A RAG system is only as reliable as the knowledge base behind it and the accountability for maintaining it. The technology is one building block; the organization determines the benefit.**

## How does retrieval augmented generation work?

RAG supplements a language model with relevant passages from the company's content for every query, so that it can answer on knowledge that was not in its training data. Internal policies, current pricing logic or maintenance manuals are not part of that training. RAG closes the gap in three steps. First, documents are prepared and stored in a searchable index. When a query arrives, the system retrieves the most relevant passages. The model formulates its answer on the basis of these passages, and a well-built system points to the sources it used.

The term was coined by Lewis et al. in 2020 in a Facebook AI Research paper presented at NeurIPS. Today's enterprise systems use the principle without training of their own: a search component and an off-the-shelf language model are combined. Most systems work with semantic search over vector representations of text, often complemented by classic keyword search. The operational advantage lies in currency: once a document is replaced, the system answers on the basis of the new version after the next indexing run, without any model having to be retrained.

**The building blocks of a RAG system**
- Knowledge sources: documents, wikis, ticketing systems or databases with a clearly named owner.
- Preparation: splitting content into sections and enriching it with metadata such as version and validity.
- Embedding: converting each section into a numerical vector that represents its meaning. Sections with similar content sit close together in vector space, even when they use different words.
- Vector database: a store for these vectors that finds the closest sections to a question within milliseconds, filtered by the access rights of the person asking.
- Language model: formulating the answer with references to the sources.
- Quality assurance: a catalog of test questions plus the analysis of incorrect answers.

## When RAG is the right architecture

Three routes lead to enterprise knowledge. The model can be retrained on the company's own data. Small sets of documents can be pasted directly into the prompt. Or the system retrieves knowledge on demand from an index. The decision follows four criteria: how often does the knowledge change, must every answer be traceable to a source, do different access rights apply, and how large is the document set?

**Three routes to enterprise knowledge compared**
Criterion | Retrain the model | Paste documents into the prompt | With Niehues Consulting
Currency | New training run with every change | Current as long as documents are inserted manually | Continuous index updates with defined ownership
Source attribution | Hard to trace | Possible, but manual | Every answer with a reference to its source
Access rights | Rights cannot be represented in the model | Held by the individual user | Retrieval based on the permissions of the person asking
Scalability | High effort per change | Limited by prompt length | Entire document estate, maintainable by business units

For most operational questions about policies and product documentation, RAG is the natural architecture. Retraining is appropriate when a model must reliably master a particular style or narrow specialist vocabulary. The approaches can be combined, while the order of decisions stays the same: first access to verifiable knowledge, then fine-tuning.

## Do you need RAG when models can process very long contexts?

Yes, as soon as the knowledge base is larger than a single dossier or different access rights apply. Large language models can now process text the length of several books in one query, which is sufficient for analyzing a single set of contracts. For operations, three reasons favor RAG. First, effort per query grows with the volume of text, so every question put to the entire estate multiplies cost and waiting time. Second, models use information in the middle of long contexts less well than at the beginning or end, as a Stanford study has shown. Third, permissions can only be enforced if the system selects, before answering, which sections the person asking may see.

The rule of thumb for decision-makers: for a bounded document set with uniform rights, long context is sufficient. For an estate that changes continuously or is separated by role, RAG belongs in the architecture. The two complement each other, since a long context allows more retrieved sections to be passed to the model.

## Quality is decided by the knowledge base

An index does not make contradictory documents consistent. If three versions of the same travel expense policy sit in different repositories, the system finds all three and, in the worst case, returns the oldest. The work therefore begins before the technology: selecting sources, removing duplicates and deciding which version is authoritative. The article on [data quality](/blog/data-quality-for-ai-success/) sets out the prerequisites in detail.

Then there is accountability. Every knowledge source needs a person in the business unit who answers for its content and currency. Without one, the content ages faster than the system can use it. RAG's connection to the wider topic of AI for knowledge management is therefore above all this: it reveals where knowledge maintenance was never properly organized.

## What does a RAG system cost, and how long does implementation take?

A RAG system for one use case costs above all working time for the knowledge base, prototype and integration, and implementation through to regular operation takes three to six months. The prototype with real questions from the business unit is ready after four to eight weeks. Operationalisation with permission checks, system integration and monitoring follows over a further two to four months. External support is typically billed in the German market at around 1,000 to 2,500 euros per consultant day.

Costs fall into three blocks: one-off build, ongoing operation of model and infrastructure, and maintenance of the knowledge base by the business units. Many plans overlook the third block, even though it decides the quality of the answers.

**Worked example with assumed values**
- Build: 40 external consultant days at 1,500 euros come to 60,000 euros, plus 30 internal days at 600 euros, or 18,000 euros.
- Operation: model usage, vector database and hosting at 1,000 euros a month come to 12,000 euros a year.
- Maintenance: two content owners with two days a month each come to 48 days, or 28,800 euros a year.
- Benefit: if the system saves 300 employees 20 minutes a week each, that is 4,500 hours over 45 working weeks. At 60 euros an hour this equals 270,000 euros a year, before applying the usage rate.

The calculation shows where the business case tips. If actual usage falls below a third, the benefit in the example barely covers operation and maintenance. The usage rate therefore belongs in every calculation, and user adoption is part of the budget.

## A four-step approach

The first step sharpens the use case. A business case names which decision or task should become faster or more reliable, such as answering customer inquiries or searching project files. The knowledge sources and the metric for success follow from this.

The second step builds a prototype in the real workflow. A business unit asks real questions, and a test catalog records the expected answers. This makes it possible to measure early whether the search finds the right sections and whether the answers meet the requirements.

The third step operationalises the prototype: integration with systems, permission checks, monitoring and handover to an internal or external operations team. For data protection, two questions shape the design: which documents contain personal data and whether they may appear in the index, and where the language model runs. The basics are covered in the article on [data protection and AI](/blog/gdpr-compliant-use-of-language-models/).

The fourth step is roll-out. Adapted processes and user enablement determine whether the system is used day to day. Without them, pilots never reach operation.

In operation, the system belongs in the AI portfolio. Three metrics are enough to start: the share of answers with the correct source, the share of queries without a match and the time saved per case. Incorrect answers flow back into the catalog as test cases.

## Niehues Consulting as a partner for RAG initiatives

[Niehues Consulting](/) supports RAG initiatives from the delivery perspective according to the Niehues Delivery Model, with the phases Understand, Build and Roll Out. Understand produces business analysis, requirements and the business case, and the test question is: what is the baseline, and how will success be measured? In Build, the prototype must prove its benefit in the real process with real data, measured against the test catalog. In Roll Out, the system moves into operation with internal or external teams, and the test question is: are users enabled and processes adapted? The full path is described in the article [From pilot to scale](/blog/from-pilot-to-scale-ai-operations/).

Marin Niehues leads initiatives of this kind as delivery manager, project manager or product owner, in projects ranging from VC-funded scale-ups to DAX 40 corporations. The entry point is a [free initial consultation](/#kontakt) of 30 minutes.

**Key points for decision-makers**
RAG offers the shortest route to verifiable answers from enterprise knowledge. Success depends on four decisions: a clear use case, a maintained knowledge base with named owners, a prototype in the real workflow and operation steered by metrics. Following this sequence moves the pilot into regular operation.

**Sources**
- Bundesverband Deutscher Unternehmensberater: Honorare in der Unternehmensberatung, 2022

## Frequently asked questions

### What is retrieval augmented generation in simple terms?

RAG is a method in which a language model retrieves relevant passages from a company's content before answering. The model formulates its answer on that basis and points to the sources. It can therefore use current internal knowledge that was not part of its training.

### What is the difference between RAG and fine-tuning?

Fine-tuning changes the model itself through additional training and suits style and specialist vocabulary. RAG leaves the model unchanged and supplies it with suitable sources for each query. For changing knowledge that must be evidenced, RAG is the right choice, because updates take effect without retraining.

### What is a vector database?

A vector database stores text as numerical vectors that represent its meaning and finds the sections most similar in content to a question. It is the search store of a RAG system. Many existing databases and search systems now offer this function as an extension, so not every initiative needs a new product.

### Does RAG prevent hallucinations completely?

No. RAG reduces hallucinations because the model is tied to sources. What matters is the quality of the search, the knowledge base and testing answers against test cases. For high-risk applications, expert sign-off by people remains part of the process.

### Which documents are suitable for a RAG system?

Suitable documents are maintained, clearly versioned and have a clear owner, such as policies or project reports. Documents with personal or confidential content require alignment with data protection and information security before they enter the index.

### What is the difference between RAG and an AI agent?

RAG answers questions on the basis of retrieved sources, whereas an AI agent also carries out work steps, such as creating a case in a business system. Many agents use RAG as a tool to retrieve the knowledge they need before acting. Agents therefore also need rules for approvals and limits on what they may do.

### Is RAG GDPR-compliant?

RAG can be operated in compliance with data protection law if it is clear which personal data enters the index, who may retrieve it and where the model processes the data. Permission checks at retrieval are the central building block. Case-by-case assessment belongs with data protection officers and legal advisers.
