Multimodal Use Cases

Where another modality earns its cost

Multimodal AI covers how media reaches a request and why it competes for your token budget. This page is the prior question: which problems are better solved with an image, an audio track or a video than with the text you could have extracted from them.

The test is whether the modality carries information the text does not. Often it does not, and a cheaper pipeline wins.

The cases that work

Documents where layout is the meaning. Invoices, forms, receipts, statements, tables. Text extraction throws away the spatial relationship that tells you which number is the total, so a vision model reading the page beats an OCR string. Ask for structured output so the answer lands in fields you can validate.

Screenshots of software. Bug reports, support tickets, "why is this button greyed out". Nobody is going to describe a UI in enough detail in prose, and the screenshot already exists. The cost per image is low and the alternative is a back-and-forth with a user.

Meetings and calls. A recording becomes a transcript, and the transcript becomes decisions, actions and follow-ups. The value is in the second step, which is text work; the audio is just how the input arrives. Speech to text covers getting that step right.

Accessibility. Alt text, described scenes, read-aloud versions. The requirement is real, the failure is graceful, and nobody expects a description to be perfect. One of the few categories where a wrong answer costs almost nothing.

Triage over user-supplied media. Is this a photo of a receipt or of a cat. Is this audio clear enough to transcribe. A cheap classifying pass in front of an expensive pipeline saves more than it costs.

Long video where you do not know what you are looking for. Security footage, lecture recordings, gameplay. The alternative is a person watching all of it. Note that this is the most expensive modality per minute, so clip to the interval that matters; video understanding covers the parameters that decide the bill.

The cases that disappoint

Anything requiring measurement. Counting objects, reading exact positions, judging distances, comparing sizes. The documented weaknesses cluster here, and the answers come back plausible and wrong, which is worse than an error. If you need a count, use something that counts.

Text you already have. A PDF with a text layer does not need a vision model. Extract the text, spend the tokens on the question instead. This is the most common unnecessary use of the modality.

Small print at the edge of legibility. Downscaling happens before the model sees the image, so the fine print you can just about read is not what arrives. Image understanding covers the resolution arithmetic behind that.

High-volume, low-value classification. If a hand-built classifier reaches 95% on your images, a multimodal model that reaches 96% at a hundred times the cost per item is not an upgrade.

Anything where a mistake is expensive and nothing checks. Medical imaging is explicitly out of scope in the providers' own documentation. So is anything where the answer goes straight into a decision with no review step.

Three questions before you build

What would a person need to see to answer this? If the answer is "the picture", the modality is right. If it is "the text on the picture", extract the text.

What does one item cost, times your volume? Images, and video more so, carry a per-item cost large enough to change the economics of a feature. Work it out before the design, not after the invoice.

What happens when it is wrong? The successful cases above all have either a low error cost or a human in the loop. The disappointing ones mostly do not.

Further reading

• Multimodal AI: the mechanics, and what vision models read badly.
• Image understanding and video understanding: what each modality costs.
• Speech to text: the transcription path, and measuring it on the fields you care about.
• Structured output: how to get an answer you can check.