Multimodal AI
Listed inMultimodal AIMultimodal applicationson
Images are billed as input tokens and share the context budget. Plus the provider’s own list of what vision reads badly.
Guide
Beginner4 min#multimodal
Images, audio, voice and documents, as input and as output.
Sign in to track your progress across sections.
14 articles
Listed inMultimodal AIMultimodal applicationson
Images are billed as input tokens and share the context budget. Plus the provider’s own list of what vision reads badly.
Listed inText to SpeechMultimodal applicationson
Uncompressed formats start playing sooner, streams fail after playback begins, and disclosing that the voice is synthetic is a policy requirement.
Listed inAudio ProcessingMultimodal applicationson
Transcribing first meters you by file size and sending the audio meters you by tokens. An hour of audio is about 115,000 of them.
Listed inLangChain for Multimodal AppsMultimodal applicationson
Message content becomes a list of typed blocks. The shapes for images, audio and files, and when the abstraction is worth its dependency.
Listed inImage GenerationMultimodal applicationson
Rewriting the prompt gives you a different picture, not an edit of the one you had. Since no provider promises the same prompt returns the same image, store the result along with the prompt, the model and the interaction ID.
Listed inNano Banana APIMultimodal applicationson
The nickname is the documented product name for Gemini's native image generation. Multi-turn conversation is the recommended way to change an image, reference budgets are capped by category and in total, and every result carries a SynthID watermark.
Listed inMultimodal Use CasesMultimodal applicationson
The test for another modality is whether the picture or the audio carries information its transcript does not. Document layout and screenshots pass it; counting objects and re-reading a PDF's text layer do not.
Listed inOpenAI Vision APIMultimodal applicationson
OpenAI bills an image as a patch count against a per-model budget, so four times the area costs under two and a half times as much. The published ceilings sit far outside what a latency budget tolerates first.
Listed inSpeech to TextMultimodal applicationson
The first transcription decision is whether anyone is waiting, because streaming is a different application rather than a faster call. Word error rate then measures the wrong thing, penalising a misheard filler as heavily as a wrong account number.
Listed inLlamaIndex for Multimodal AppsMultimodal applicationson
LlamaIndex approaches multimodal work through the index rather than the request, so the choice is captioning an image into text or embedding it jointly with CLIP. One is inspectable and lossy; the other is neither.
Listed inVideo UnderstandingMultimodal applicationson
Gemini samples video at one frame per second and charges roughly 100 tokens for every second at low resolution, 300 at high. Clipping to the interval that matters is the largest saving available and the one most often skipped.
Listed inWhisper APIMultimodal applicationson
Whisper is the transcription model you can also download, which is what earns it a page separate from speech to text. Hosted as whisper-1 it remains the one carrying timestamps and subtitles, and self-hosting buys data residency rather than a lower bill.
Listed inImage UnderstandingMultimodal applicationson
Claude bills an image as visual tokens, one per 28-pixel patch, capped by a resolution tier. Moving to a high-resolution model can triple what the same screenshot costs, and oversized images are downscaled rather than refused.
Listed inDALL·E APIMultimodal applicationson
DALL·E is the name people use; gpt-image is what the models are called, split between an editing-precision model and a fast one. Custom dimensions must be multiples of 16, and a blocked request tells you whether the input or the output failed.