AI Adoption & Operations

Retrieval Augmented Generation: Connecting AI to Enterprise Knowledge

Marin Niehues26 May 202510 min read

Technician checking a server rack

Retrieval Augmented Generation (RAG) connects a language model to enterprise knowledge: before each answer, the system searches for suitable sources and has the model formulate its reply from them. For executives and business heads, delivery is the central question. Which knowledge should the system use, who is accountable for its quality, and how does the organization know that answers are reliable? This article explains the architecture, distinguishes RAG from long context and fine-tuning, sets out cost and duration and describes an approach through to regular operation. This article provides an overview and does not constitute legal advice.

How does retrieval augmented generation work?

RAG supplements a language model with relevant passages from the company's content for every query, so that it can answer on knowledge that was not in its training data. Internal policies, current pricing logic or maintenance manuals are not part of that training. RAG closes the gap in three steps. First, documents are prepared and stored in a searchable index. When a query arrives, the system retrieves the most relevant passages. The model formulates its answer on the basis of these passages, and a well-built system points to the sources it used.

The term was coined by Lewis et al. in 2020 in a Facebook AI Research paper presented at NeurIPS. Today's enterprise systems use the principle without training of their own: a search component and an off-the-shelf language model are combined. Most systems work with semantic search over vector representations of text, often complemented by classic keyword search. The operational advantage lies in currency: once a document is replaced, the system answers on the basis of the new version after the next indexing run, without any model having to be retrained.

When RAG is the right architecture

Three routes lead to enterprise knowledge. The model can be retrained on the company's own data. Small sets of documents can be pasted directly into the prompt. Or the system retrieves knowledge on demand from an index. The decision follows four criteria: how often does the knowledge change, must every answer be traceable to a source, do different access rights apply, and how large is the document set?

For most operational questions about policies and product documentation, RAG is the natural architecture. Retraining is appropriate when a model must reliably master a particular style or narrow specialist vocabulary. The approaches can be combined, while the order of decisions stays the same: first access to verifiable knowledge, then fine-tuning.

Do you need RAG when models can process very long contexts?

Yes, as soon as the knowledge base is larger than a single dossier or different access rights apply. Large language models can now process text the length of several books in one query, which is sufficient for analyzing a single set of contracts. For operations, three reasons favor RAG. First, effort per query grows with the volume of text, so every question put to the entire estate multiplies cost and waiting time. Second, models use information in the middle of long contexts less well than at the beginning or end, as a Stanford study has shown. Third, permissions can only be enforced if the system selects, before answering, which sections the person asking may see.

The rule of thumb for decision-makers: for a bounded document set with uniform rights, long context is sufficient. For an estate that changes continuously or is separated by role, RAG belongs in the architecture. The two complement each other, since a long context allows more retrieved sections to be passed to the model.

Quality is decided by the knowledge base

An index does not make contradictory documents consistent. If three versions of the same travel expense policy sit in different repositories, the system finds all three and, in the worst case, returns the oldest. The work therefore begins before the technology: selecting sources, removing duplicates and deciding which version is authoritative. The article on data quality sets out the prerequisites in detail.

Then there is accountability. Every knowledge source needs a person in the business unit who answers for its content and currency. Without one, the content ages faster than the system can use it. RAG's connection to the wider topic of AI for knowledge management is therefore above all this: it reveals where knowledge maintenance was never properly organized.

What does a RAG system cost, and how long does implementation take?

A RAG system for one use case costs above all working time for the knowledge base, prototype and integration, and implementation through to regular operation takes three to six months. The prototype with real questions from the business unit is ready after four to eight weeks. Operationalisation with permission checks, system integration and monitoring follows over a further two to four months. External support is typically billed in the German market at around 1,000 to 2,500 euros per consultant day.

Costs fall into three blocks: one-off build, ongoing operation of model and infrastructure, and maintenance of the knowledge base by the business units. Many plans overlook the third block, even though it decides the quality of the answers.

The calculation shows where the business case tips. If actual usage falls below a third, the benefit in the example barely covers operation and maintenance. The usage rate therefore belongs in every calculation, and user adoption is part of the budget.

A four-step approach

The first step sharpens the use case. A business case names which decision or task should become faster or more reliable, such as answering customer inquiries or searching project files. The knowledge sources and the metric for success follow from this.

The second step builds a prototype in the real workflow. A business unit asks real questions, and a test catalog records the expected answers. This makes it possible to measure early whether the search finds the right sections and whether the answers meet the requirements.

The third step operationalises the prototype: integration with systems, permission checks, monitoring and handover to an internal or external operations team. For data protection, two questions shape the design: which documents contain personal data and whether they may appear in the index, and where the language model runs. The basics are covered in the article on data protection and AI.

The fourth step is roll-out. Adapted processes and user enablement determine whether the system is used day to day. Without them, pilots never reach operation.

In operation, the system belongs in the AI portfolio. Three metrics are enough to start: the share of answers with the correct source, the share of queries without a match and the time saved per case. Incorrect answers flow back into the catalog as test cases.

Niehues Consulting as a partner for RAG initiatives

Niehues Consulting supports RAG initiatives from the delivery perspective according to the Niehues Delivery Model, with the phases Understand, Build and Roll Out. Understand produces business analysis, requirements and the business case, and the test question is: what is the baseline, and how will success be measured? In Build, the prototype must prove its benefit in the real process with real data, measured against the test catalog. In Roll Out, the system moves into operation with internal or external teams, and the test question is: are users enabled and processes adapted? The full path is described in the article From pilot to scale.

Marin Niehues leads initiatives of this kind as delivery manager, project manager or product owner, in projects ranging from VC-funded scale-ups to DAX 40 corporations. The entry point is a free initial consultation of 30 minutes.

Frequently asked questions

What is retrieval augmented generation in simple terms?

RAG is a method in which a language model retrieves relevant passages from a company's content before answering. The model formulates its answer on that basis and points to the sources. It can therefore use current internal knowledge that was not part of its training.

What is the difference between RAG and fine-tuning?

Fine-tuning changes the model itself through additional training and suits style and specialist vocabulary. RAG leaves the model unchanged and supplies it with suitable sources for each query. For changing knowledge that must be evidenced, RAG is the right choice, because updates take effect without retraining.

What is a vector database?

A vector database stores text as numerical vectors that represent its meaning and finds the sections most similar in content to a question. It is the search store of a RAG system. Many existing databases and search systems now offer this function as an extension, so not every initiative needs a new product.

Does RAG prevent hallucinations completely?

No. RAG reduces hallucinations because the model is tied to sources. What matters is the quality of the search, the knowledge base and testing answers against test cases. For high-risk applications, expert sign-off by people remains part of the process.

Which documents are suitable for a RAG system?

Suitable documents are maintained, clearly versioned and have a clear owner, such as policies or project reports. Documents with personal or confidential content require alignment with data protection and information security before they enter the index.

What is the difference between RAG and an AI agent?

RAG answers questions on the basis of retrieved sources, whereas an AI agent also carries out work steps, such as creating a case in a business system. Many agents use RAG as a tool to retrieve the knowledge they need before acting. Agents therefore also need rules for approvals and limits on what they may do.

Is RAG GDPR-compliant?

RAG can be operated in compliance with data protection law if it is clear which personal data enters the index, who may retrieve it and where the model processes the data. Permission checks at retrieval are the central building block. Case-by-case assessment belongs with data protection officers and legal advisers.

Let’s talk about your initiative.

In 30 minutes we clarify where your organization stands and which step will pay off most now.

Request a consultation

30 minutes · free · no obligation