Multimodal AI

What multimodal means for the request you send

A multimodal model takes more than one kind of input. You send an image with your question, or an audio clip, or a frame from a video, and it answers about what it received.

The distinction worth holding is between a model that does this and a pipeline that fakes it. Running an image through a captioning service and pasting the caption into a text prompt gives you what the captioner noticed. A multimodal model reads the image against your question, so asking "which row has the discrepancy" looks at the table rather than at a description of it.

Getting media into the request

Three ways, and they are close enough across providers to describe once.

A URL the provider fetches. A base64 data URL inline in the message. Or a file identifier from a files API, where you upload first and reference afterward.

The third is the one to reach for with anything substantial. Inlining a large video as base64 means sending it on every request in the conversation, and both OpenAI and Google provide a files API for exactly this reason.

Media rides in the message alongside your text. There is no separate vision endpoint to call and no second request to correlate.

Why media counts against your token budget

This is the fact that changes how you design.

OpenAI states that "Vision models convert image inputs into billable input tokens." Images count toward cost and toward your tokens-per-minute rate limit. They are not attachments riding alongside the prompt; they are part of it.

So an image competes with your retrieved documents and your conversation history for the same window. A support tool that attaches three screenshots to a twelve-message thread is spending its context budget on pixels, and the context layer that decides what goes in has to account for them.

A detail parameter controls fidelity, with values including low, high, original, and auto, and it changes the token count. There is a trap in the naming: the documentation notes cases where low consumes more tokens than high on a given model, because the budgets are model-specific. The parameter names describe fidelity, not cost. Measure the cost on the model you are calling rather than inferring it from the word.

What vision models read badly

OpenAI publishes the limitations, and the list is more specific than the usual warning about hallucination:

• Small text. Enlarging the image before sending helps.
• Rotated or upside-down text.
• Precise spatial localization, with chess positions given as the example.
• Graphs or text where colors or styles vary.
• Counting. You get approximate counts of objects.
• Medical images and non-Latin text, both flagged as unreliable.

Two of these bite document pipelines hardest. Small text covers most scanned invoices and forms, which is exactly the content people point vision models at first. And approximate counting means "how many line items are on this receipt" is a question you should not trust the answer to, however confident it sounds.

Test on your own documents at the resolution you will send. A model that reads your sample PDF cleanly may fail on the phone photographs your users upload.

Text inside an image is text in the prompt

Follow the first fact to its conclusion. If an image becomes tokens in the prompt, then writing in the image becomes text the model reads.

So an instruction photographed onto a page reaches the model the same way an instruction typed into a message does. Everything true of prompt injection applies to images, and a system that treats uploaded files as data while treating typed text as potentially hostile has the boundary in the wrong place.

The defense is the same one. Decisions belong in your code, checked against your rules, not in a sentence of the prompt asking the model to disregard instructions it finds in attachments.

Further reading

• OpenAI, Images and vision: input methods, token billing, the detail parameter, and the limitations list.
• Google Gemini: one provider's take on multimodal-first design.
• Image understanding: depth on the visual modality specifically.