# Information Security for Generative AI

Author: Marin Niehues, Niehues Consulting · Published: 2025-01-27 · Updated: 2026-11-16 · AI Law, Risk & Governance
Source: https://www.niehuesconsulting.com/blog/generative-ai-information-security/

Information security for generative AI protects confidential information against leakage and manipulation as soon as language models work with documents and business systems. A language model processes instructions and data in the same stream of text and increasingly accesses company systems through interfaces. This creates an attack surface that conventional security concepts only partly cover. For the executive team, the question is which measures must be in place before rollout and who owns them.

**Generative AI needs a security concept that follows the data flow, from input through the model and its connections to the output. Companies that settle this concept before rollout implement faster, because business units and information security use the same approval logic.**

## What are the information security risks of generative AI?

Generative AI brings four main risks: leakage of inputs to external services, manipulation of the model through injected instructions, disclosure of documents to unauthorised users and unintended actions by agents with excessive permissions.

The reason lies in the architecture. Conventional applications separate commands and data technically, for example through parameterised queries. A language model fundamentally cannot make this separation. The user's instruction, the content of an attached document and the system prompt reach the model as a single text. A manipulated document can therefore contain commands that the model follows without any human ever having read them. This pattern is called prompt injection.

The OWASP Foundation classifies these risks in the "OWASP Top 10 for LLM Applications". Prompt injection heads the list, followed by sensitive information disclosure. Excessive agency of applications and agents adds to them. Germany's Federal Office for Information Security classifies the same threats in its publication on generative AI models and updates it continuously. Both sources lead to the same conclusion: a language model with access to company data is an application with its own attack surface and belongs in the company's information security management system. Where this article touches on obligations under NIS2 and the AI Act: this article provides an overview and does not constitute legal advice.

## The protection model: four layers along the data flow

A robust concept assigns every measure to a point in the data flow. This shows where protection is missing and who owns it.

**The four layers**
- Input: which information may employees enter into which tool? The basis is the classification of data.
- Model and operations: where does the model run, who sees the logs, and are inputs used for training?
- Connections: which document repositories and business systems does the application access, and with which rights?
- Output: who checks results before they feed into decisions or actions?

The layers can be built up independently. Most companies start with input, because that is where leakage becomes visible. The greatest risks lie in the connections, as soon as an assistant uses company knowledge or triggers actions. Plan the connections from the first prototype onwards, even if input is regulated first.

## Which data may go into ChatGPT or Microsoft Copilot?

Public and internal information may go into ChatGPT or Microsoft Copilot, but only under business licenses with commitments on training and retention. Process strictly confidential data exclusively in environments under your own control.

A policy that demands "no confidential data" fails on the question of what counts as confidential. What works is an allocation by data class that builds on the existing information classification. Each class receives a clear list of permitted tools.

**Data classes and permitted tools**
Criterion | Personal version | Business license | With Niehues Consulting
Public information | Permitted | Permitted | Approval list per tool, visible to all
Internal information | Not permitted | Permitted with commitments on training and retention | Commitments reviewed and documented in the register
Strictly confidential information | Not permitted | Not permitted, except in a specially secured environment | Dedicated environment with passed-through permissions
Visibility for the company | None | Logs and administration | Review on a fixed cycle

The article [AI policy for employees](/blog/ai-policy-for-employees/) shows how to word and introduce the rules for staff. A ban without an approved alternative shifts usage to personal accounts and removes all visibility for the organization. The article [Detecting and governing shadow AI](/blog/shadow-ai-detect-and-govern-unsanctioned-use/) describes how such usage can be identified.

## How do you prevent data leakage from AI assistants?

You prevent data leakage from AI assistants with three levels: an approved business license, technical controls at the hand-over to the provider and labeling of confidential documents that the assistant respects.

The first level creates the alternative to the personal account. The second level works at the network. Data loss prevention tools detect account numbers, customer numbers or health details in inputs and block or mask them before the request leaves the company. An intermediate gateway replaces such details with placeholders and restores them in the response. Access to unapproved AI services is blocked at the proxy or redirected to the business license. The third level works on the document: sensitivity labels determine which content an assistant may read at all and use in responses.

No level is sufficient on its own. Pattern matching checks free text only partially, and a label works only if documents are labeled. The combination lowers the risk far enough for internal information to be approved broadly.

## Connections: passing permissions through

The greatest risk arises as soon as an assistant uses company knowledge. When document repositories are connected through retrieval augmented generation, the system must know and apply the access rights of the person asking. An assistant that runs under a service account with full access answers any question from any document, even if the person asking could never open those documents. The user's rights must therefore be passed through to the search in the repository. This exposes historically grown permission structures with open folders, and cleaning them up is part of the project.

With agents that trigger actions themselves, the scope of authority adds to this. The principle of least privilege applies: an agent receives only the interfaces its task requires, read access before write access. Every action with external effect, such as a payment or an email to customers, goes through human approval.

## Model and operations: reviewing providers, monitoring usage

With external providers, contractual terms and technical assurances determine the level of protection. The location of processing and the retention period of inputs need to be clarified. Added to this are whether inputs are used for training and the contractual form of data processing. ISO/IEC 27001 certification indicates the provider's security management; the specific AI services require their own review.

In operation, every application needs logs of requests and access that are evaluated on a fixed cycle. Before approval and after every major change to the model or instructions, tests with targeted attack attempts follow, known as red teaming. An incident process defines who decides in the event of data leakage or manipulation. The NIST AI Risk Management Framework divides these tasks into four functions: Govern, Map, Measure and Manage. It thus provides a structure that can be applied to the portfolio of AI applications.

## What obligations do NIS2 and the AI Act create?

For companies within the scope of the NIS2 Directive, its risk management obligations also apply to AI applications, and the management body must approve the measures and oversee their implementation.

NIS2 distinguishes essential and important entities by sector and size. For both, an AI application with access to company systems is part of the network and information systems and falls under the same requirements as any other system. These include risk analysis, incident handling and supply chain security, which covers the review of the AI provider. The directive also requires members of management bodies to undergo training. The EU AI Act adds requirements for high-risk systems. It demands an appropriate level of accuracy, robustness and cybersecurity and expressly names attacks such as poisoned training data and manipulated inputs. A four-layer protection model provides the evidence that an audit under either regime requires.

## How Niehues Consulting builds information security into rollout

Approvals for AI projects often fail at the end, when security questions are raised for the first time. [Niehues Consulting](/) therefore builds the safeguards into the Niehues Delivery Model with the phases Understand, Build and Roll out. Three of its checkpoint questions are decisive for information security. Does the prototype prove its value in the real process with real data, including the real permissions? Which roles, decision paths and interfaces does operation require? And who is accountable for operation from day one?

In Understand, a picture emerges of actual usage and data classes, together with a sequence of planned applications by value and risk. In Build, prototypes contain the safeguards from the start, such as passed-through permissions and limited agent authority. In Roll out, the solution moves into operation with clear responsibilities between business unit and information security and with users enabled. The result is an organization in which every new application follows a known path to approval. The article [From pilot to scale](/blog/from-pilot-to-scale-ai-operations/) describes the overall process.

Founder Marin Niehues is a management consultant who brings AI into operation, in projects ranging from VC-funded scale-ups to DAX 40 corporations. A [free initial consultation of 30 minutes](/#kontakt) assesses where your company stands.

**Recommendation for the executive team**
Before the next rollout, define data classes with permitted tools and require passed-through permissions for every connection. Give agents only minimal rights and name an accountable function for incidents. This turns bans and one-off approvals into a procedure in which new applications reach approval predictably.

**Sources**
- OWASP Foundation: OWASP Top 10 for LLM Applications 2025, risk list, 2024
- Bundesamt für Sicherheit in der Informationstechnik: Generative KI-Modelle: Chancen und Risiken für Industrie und Behörden, publication, 2024
- National Institute of Standards and Technology: Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, 2023
- European Parliament and Council: Directive (EU) 2022/2555 (NIS2), directive, 2022
- ISO and IEC: ISO/IEC 27001 Information security management systems, standard, 2022

## Frequently asked questions

### What are the security risks of generative AI?

Four risks stand out. Inputs leak to external services, and injected instructions manipulate the model. Through connections, users see documents they are not entitled to, and agents with excessive rights trigger unintended actions.

### Can employees enter confidential data into ChatGPT or Microsoft Copilot?

Employees may enter internal information, but only under business licenses with commitments on training and retention. Strictly confidential data belongs exclusively in environments under the company's own control. Personal accounts are off limits for company data of any kind.

### What is the OWASP Top 10 for LLM applications?

The OWASP Top 10 for LLM Applications is a list of the most important security risks in applications built on large language models, published by the OWASP Foundation. Prompt injection and sensitive information disclosure head the list, with excessive agency of agents added. Development teams and information security use it as a common review basis.

### How do you prevent data leakage with AI assistants?

You prevent data leakage with an approved business license, technical controls at the hand-over to the provider and labeled documents. Data loss prevention tools detect sensitive details in inputs and block or mask them. The proxy blocks unapproved services.

### What does NIS2 require for AI applications?

NIS2 treats AI applications like any other information system: risk analysis, incident handling and supply chain security apply to them too. The management body approves the measures and oversees implementation. The AI provider therefore belongs in the supplier review.

### Who is responsible for information security in generative AI?

The executive team owns the framework. The business unit owns the individual application, information security sets requirements and grants approval, and IT runs the solution. Assigning these roles in writing prevents responsibilities from being settled only during an incident.

### Which standards help secure generative AI?

ISO/IEC 27001 describes the information security management system, and ISO/IEC 42001 adds requirements for AI management systems. The NIST AI Risk Management Framework provides a structure for risk management. The OWASP list and the publications of Germany's Federal Office for Information Security name the technical threats.
