AI Adoption & Operations

Multimodal AI: Text, Image and Speech in Everyday Business

Marin Niehues30 April 202612 min read

Marin Niehues speaking with a microphone and presentation clicker in front of a conference banner.

Multimodal AI processes images and spoken language alongside text, in current systems within a single model, and so opens up processes that used to fail at media breaks. A photo record and a voice note can be analyzed together with the scanned delivery note, without a person translating between media. For the executive team, the key question is where in the value chain employees currently transfer information from one medium to another. This article maps the fields of application, sets out selection criteria and costs, and describes how a demonstration becomes a process that runs reliably in operation.

What does multimodal AI change in operations?

Multimodal AI extends the use of AI from desk work to every activity in which photos or spoken language are produced. Classic language models work with text, so scans and conversations first needed character recognition or transcription. Multimodal models take in the media directly. A model can read the photo of a machine nameplate, classify the related fault report and answer a technician's spoken query.

This makes preliminary stages dispensable wherever handwriting or visual context matters. For large volumes of uniform documents, specialized recognition combined with a language model often remains the more accurate and cheaper choice, so the decision is made per field of application on the basis of a test series.

Examples of multimodal models are the current model families GPT from OpenAI, Gemini from Google and Claude from Anthropic, alongside open models with image understanding that can run in a company's own data center. What counts for selection is performance on your own material.

Multimodal and generative AI overlap, yet they describe different properties. Generative AI creates new content such as text or images. A model is called multimodal when it processes several types of media as input or output. Most current language models are both: they read a photo and write a report on it.

As soon as photos of people or voice recordings come into play, the use touches on data protection and, where employees are recorded, on co-determination. This article provides an overview and does not constitute legal advice. How multimodal applications fit into an orderly AI rollout is described in the article From pilot to scale.

Fields of application along the information flow

A robust organizing principle follows the path of a piece of information: capture, understand, act. Each stage has typical fields of application, and the effect is greatest where a process currently loses time at a media break.

The capture stage deals with material that is still transferred into a system by hand. Typical scenarios are incoming invoices with handwritten notes and delivery notes in varying layouts, as well as inspection reports as photos. The model reads and structures the data and passes it to the leading system. How this field is designed in detail is described in the article on document processing with AI.

At the understand stage, the model combines several sources into an assessment, for example photos of a claim with the customer's description. The value lies in preparing the decision: the case handler sees an ordered proposal with evidence and checks it, instead of assembling the case from scratch.

At the act stage, the model triggers follow-up steps, such as a query to a supplier or a spoken reply to a customer. This is where requirements for control and approval rise most, because the result directly affects operations. The organization decides in advance which steps run automatically and which require human approval.

What are the advantages and disadvantages of multimodal AI?

The greatest advantage of multimodal AI is that processes with mixed material no longer need manual transfer; the greatest disadvantage is fluctuating quality when the source material is poor. Both sides carry equal weight in the decision to deploy.

On data protection: faces and voices are personal data, and processing them requires a legal basis under the GDPR. Where a face or voice is processed to uniquely identify a person, it counts as biometric data with particularly strict requirements. With external providers, a data processing agreement and the question of storage location must be settled. Where recordings capture employees, an existing works council must be involved. The responsible data protection and co-determination functions therefore belong in the planning from the start, which shortens the path into operation because later corrections to the set-up fall away. The article on data protection and AI covers the use of language models in more depth.

Selection criteria for fields of application and providers

Prioritization follows the same axes as any AI initiative: value and feasibility as well as risk. The article on prioritizing AI use cases describes the assessment in detail. For multimodal applications, three review questions on material quality are added.

Five criteria decide the choice of model:

  • accuracy on your own material
  • response time in the process
  • the mode of operation
  • ability to integrate with the existing system landscape
  • price per processing run relative to the benefit

A test with a representative sample from your own operations, assessed by the specialists who will later use the result, is what provides meaningful evidence.

How much does multimodal AI cost to run?

The ongoing usage costs of multimodal AI are low per transaction; the larger share of costs comes from rollout and integration. Cloud providers bill usage by volume processed, for images and document pages via the number of tokens derived from them, and for audio usually per minute. Depending on the model, the cost of one document page ranges from a fraction of a cent to a few cents.

The worked example shows the typical picture: profitability is decided by the processing time saved and the effort for integration and review, while the price per processing run only becomes significant at very large volumes or with audio and video. How to build a robust business case is shown in the article on the AI business case.

From prototype to stable operation

Most multimodal initiatives begin with a convincing demonstration. The demonstration shows what the model can do. It says little about how it behaves day to day with poor photos and dialects. A robust approach is therefore divided into three phases.

In the Understand phase, the process is mapped, the field of application defined and the business case calculated. This includes current processing time and error rate per transaction as well as the number of transactions per year.

In the Build phase, a prototype in the real process with real material proves the benefit with measured values, comparing processing time and error rate with the baseline over a sample large enough that outliers do not distort the picture.

In the Introduce phase, the prototype moves into operation with defined accountabilities, ongoing monitoring of result quality, escalation paths and user enablement. Adoption by the business unit ultimately decides whether the benefit materializes in daily work.

Niehues Consulting as a partner for multimodal initiatives

Niehues Consulting helps companies take AI applications with multimodal material from the field of application into regular operation. In AI management, analysis and the business case are developed along with a sequence of applications and metrics for managing the portfolio. Prototypes prove the benefit in the real process, and the transfer into operation takes place with internal or external teams.

The approach follows the Niehues Delivery Model with its phases Understand, Build and Introduce. Three of its review questions matter most for multimodal initiatives. What is the baseline, and how do we measure success? Does the prototype prove the benefit in the real process with real data? Who is accountable for operations from day one? Marin Niehues builds organizations that deliver and brings AI into operation, on projects from VC-funded scale-ups to DAX 40 corporations and in 20+ countries on six continents. The entry point is a free 30-minute initial consultation that identifies the right field of application and the next step: book an initial consultation.

Frequently asked questions

What is multimodal AI?

Multimodal AI refers to models that process several types of media together, such as images and audio alongside text. Preliminary stages such as character recognition become dispensable wherever handwriting or visual context matters.

What are examples of multimodal AI models?

Well-known examples are the model families GPT from OpenAI, Gemini from Google and Claude from Anthropic, which read images and documents directly. In addition there are open models with image understanding for operation in your own data center. Selection should be decided by measured accuracy on your own material.

What is the difference between multimodal AI and generative AI?

Generative AI creates new content, while multimodal AI processes several types of media such as images and speech. The terms describe different properties and are not mutually exclusive. Most current language models are generative and multimodal at the same time.

What are the pros and cons of multimodal AI?

The main advantage is fewer system breaks, because one component replaces several preliminary stages and opens up new fields of application with camera and microphone. The main disadvantage is dependence on material quality, combined with the risk of misread or invented content. Every deployment therefore needs a quality check against reference cases.

Is multimodal AI GDPR compliant for photos and voice recordings?

Multimodal AI can be used in compliance with data protection law once the legal basis and data processing agreement are settled. Faces and voices are personal data, and when processed for unique identification they are biometric data with stricter requirements. Recordings of employees additionally require the involvement of an existing works council.

How much does multimodal AI cost?

Depending on the model, usage costs range from a fraction of a cent to a few cents per document page and are billed by volume processed. The larger share of costs comes from integration and ongoing quality checks. External consulting is typically billed in the German market at around 1,000 to 2,500 euros per consultant day.

Which fields of application pay off first?

Processes with high transaction volumes and mixed material pay off fastest, provided a clear quality check is possible. These include capturing documents and inspection records as well as preparing claims and maintenance cases. The sequence follows from assessing the individual use cases.

How can the accuracy of multimodal models be assessed?

Accuracy is measured on your own material. The business unit assembles a reference set of real cases with a known correct result and compares the model's outputs against it. The sample also includes difficult cases, such as poor photos and unusual layouts.

How should a company get started with multimodal AI?

The entry point is a clearly defined field of application with a measurable baseline. A short analysis is followed by a prototype in the real process whose benefit is proven with measured values. Only then is the transfer into operation with fixed accountabilities and ongoing quality monitoring worthwhile.

Let’s talk about your initiative.

In 30 minutes we clarify where your organization stands and which step will pay off most now.

Request a consultation

30 minutes · free · no obligation