LlamaIndex approaches multimodal work through the index rather than the request, so the choice is captioning an image into text or embedding it jointly with CLIP. One is inspectable and lossy; the other is neither.
A retrieval framework's answer is about the index
LangChain for multimodal apps covers that framework's contribution: a common message format, so an image block looks the same whichever provider receives it.
LlamaIndex is a different kind of tool, and its multimodal story follows from that. LlamaIndex is built around indexing and retrieval, so the question it answers is not how to send an image but how to retrieve one, and how to search a corpus that is part text and part pictures.
It extends the RAG shape rather than the request shape: input can be text or image, and the stored knowledge base can be either.
Two ways to make an image retrievable, and they are not equivalent
This is the architectural decision, and it is worth understanding before picking a framework feature.
Caption, then index the text. Run the image through a multimodal model, store the description, embed the description. Retrieval is then ordinary text retrieval. It works with any vector store you already run, it is cheap to query, and it is debuggable, because you can read what was indexed.
The cost is that the caption is now the document. Anything the captioning model did not mention is unfindable, and you will not know what was dropped.
Joint embeddings. A model such as CLIP puts images and text in the same vector space, so a text query can retrieve an image directly with no description in between. Nothing is lost to a caption.
The costs are a second embedding model, a store that holds those vectors, and retrieval you cannot inspect by reading. When it returns the wrong image there is no text to examine.
LlamaIndex supports both, and the framework's value here is that switching between them is a change of index rather than a rewrite.
The patterns worth knowing
Multimodal RAG, in the general case: a store holding text and images, a query in either, results from either.
Retrieval-augmented captioning, which is the neatest of them. Caption the image with a multimodal model, then retrieve against a text corpus to refine that caption. The model describes a product photo in generic terms, and your catalogue copy then corrects it, so the caption ends up in your vocabulary rather than a borrowed one.
Structured output from an image, by specifying a Pydantic object, so an image yields typed fields rather than prose. This is the pattern most document work wants; the same argument as structured output generally.
Image-to-image retrieval, via CLIP, for "find me more like this one".
Evaluate the retriever separately
LlamaIndex ships evaluation for multimodal RAG that assesses retrievers and generators separately, and that separation is the part to use.
A wrong answer from a multimodal RAG system has two possible causes and they need different fixes. Either the right image was never retrieved, or it was retrieved and read badly. Without separate numbers you cannot tell, and evaluation metrics makes the same argument for the text case.
Multimodal makes this worse than usual, because a captioning step sits in the middle and can fail on its own. Three stages, three places to measure.
Check the documentation's age before you commit
One practical caution. The multimodal documentation names models from an earlier generation, including GPT-4V, LLaVa-13B, Fuyu-8B and MiniGPT-4. Some of those are retired.
That does not mean the patterns are wrong; the caption-or-joint-embedding choice is durable and the evaluation split is right. It does mean the integration list is not a current survey, so confirm that the model you intend to use is still supported rather than inferring it from a page that lists its ancestors.
This is a general hazard with framework documentation for a fast-moving capability, and it is the strongest argument in LlamaIndex's own trade-off section: an abstraction over providers is only as current as its maintenance.
Further reading
- LlamaIndex, Multimodal use cases: the patterns, integrations and evaluation tooling.
- LlamaIndex: the framework, and what the abstraction costs.
- LangChain for multimodal apps: the other framework's angle, on the request rather than the index.
- Multimodal AI: what you are indexing, and what it costs to send.
Knowledge check
Question 1 of 3
Sign in to save your progress and pick up where you left off.