LangChain for Multimodal Apps

Content is a list of blocks, not a string

If you have used a chat API for text, you have sent messages that look like {"role": "user", "content": "summarize this"}. Content is a string.

For anything other than text, it stops being a string and becomes a list of typed blocks. One block holds your instruction, another holds the image, and the model receives both as parts of one turn.

{"role": "user", "content": [
    {"type": "text", "text": "What is in this picture?"},
    {"type": "image", "url": "https://example.com/path/to/image.jpg", "mime_type": "image/jpeg"},
]}

A picture is not an attachment to a message. It is a sibling of the text, and everything below follows from that.

The reason the shape matters more than the syntax is that providers disagree about the syntax. They all landed on typed blocks and they all named the fields differently, which is the problem a framework's common format is there to solve.

Building a message with an image

LangChain standardizes those blocks. The image form, from its documentation:

{"type": "image", "url": "https://example.com/path/to/image.jpg", "mime_type": "image/jpeg"}

The type field is required and names the media category. The rest says where the bytes are.

Put that in a list beside a text block and you have a multimodal turn. Send it to any chat model LangChain supports that accepts images, and the framework translates the block into whatever that provider's API wants. Switching providers is a model change rather than a rewrite of how you build messages, which is what you are buying.

URL or base64, and when the mime type is required

Two ways to supply a file, and one asymmetry worth committing to memory.

{"type": "image", "base64": "AAAAIGZ0eXBtcDQy...", "mime_type": "image/jpeg"}

mime_type is documented as "Required for base64 data." The URL form omits it, because a URL fetch reveals the type and raw bytes do not.

Beyond that requirement, the choice has practical consequences. A URL means the provider fetches the file, so it must be reachable from their network, which rules out anything behind your VPN, on localhost, or in a private bucket without a signed link. Base64 means you send the bytes, which always works and inflates the payload by roughly a third.

The usual shape: URLs for public assets, base64 for anything a user uploaded, since you are holding those bytes already and a round trip through public storage to make them fetchable is work you do not need.

Audio and files in the same shape

The pattern generalizes, which is the point of having a pattern.

{"type": "audio", "base64": "AAAAIGZ0eXBtcDQy...", "mime_type": "audio/wav"}
{"type": "file",  "url": "https://example.com/path/to/document.pdf"}
{"type": "file",  "base64": "AAAAIGZ0eXBtcDQy...", "mime_type": "application/pdf"}

Same three fields, same rule about mime_type. Learn the image case and you know the others.

For audio specifically, note that sending it to a chat model is one of two possible designs, audio processing covers the other, transcribing first, and why the choice affects both cost and what you can ask.

What LangChain's common format does not do

A shared message shape normalizes syntax. It does not confer capability.

Send an audio block to a model that only accepts text and images and it will fail, because the model cannot do the thing. The framework has no way to make it possible; the block is a request, not a promise. The failure will be a provider error surfacing through the framework, which is usually less legible than the same error direct from the API.

I should be precise about the basis for that. The block shapes above are documented; that a common format cannot add a missing capability is an inference from what the abstraction is, a translation layer, rather than a sentence I found in the documentation. It is a safe inference, but the practical advice stands either way: check the model's own page for which modalities it accepts, and do not treat framework support for a block type as evidence that your chosen model supports it.

The version caveat is real too. LangChain has reorganized its message interfaces more than once, and package names have moved between releases. The shapes here come from the Python documentation as of 2026-09-16. Verify against the version you have installed before assuming an example from anywhere, including this article, still applies.

Whether the LangChain abstraction is worth it

The answer depends on how many providers you have.

One provider, and no plans to change. The framework is a layer between you and an API you could call directly, and multimodal is the case where the layer is thinnest. You are constructing a dictionary either way. Calling the provider directly means fewer dependencies, error messages that match the vendor's documentation, and same-day access to new features rather than waiting for an adapter. Multimodal AI covers the concepts without any framework at all.

More than one, or expecting to switch. The normalization earns its place. Building provider-specific message construction twice, then maintaining both as the APIs drift, is more work than the dependency, and switching to compare becomes a model string rather than a branch in your code.

Already using LangChain for other reasons. Then the question is settled; use the blocks it gives you and move on.

LlamaIndex is the neighboring framework and makes a similar trade with a different center of gravity. The reasoning is the same: count your providers, then decide.

Further reading

• LangChain, Messages: every content block shape above, and the mime type rule.
• Multimodal AI: what these models are doing with an image in the first place.
• Audio processing: the other way to get audio into a model, and when to prefer it.
• LlamaIndex: the neighboring framework.