Cohere

Where Cohere fits when you already have a generation model

Most vendor articles ask you to switch something. This one does not have to.

Cohere sells three families (Command for generation, Embed for vectors, and Rerank for search) and the middle of those three is described in a way that tells you the company's angle. Rerank is "the fastest way to inject the intelligence of a language model into an existing search system."

Existing. The proposition is additive: keep the model you generate with, keep the vector store you built, and insert one call. For a reader with a working retrieval pipeline that is nearly good enough, that is a more interesting offer than another frontier model.

Reranking is the piece a working pipeline adopts first, so it gets most of the space below.

The three product families

In the vendor's own terms.

Command is "the text-generation LLMs powering conversational agents, summarization, copywriting, and similar use cases", through a Chat endpoint with retrieval-augmented generation supported directly.

Embed "improves the accuracy of search, classification, clustering, and RAG results", powering both the Embed and Classify endpoints. It competes with the OpenAI embeddings API and with the open models in embedding models on Hugging Face, and the same warning applies: changing embedding models means re-embedding your corpus.

Rerank takes a query and a list of documents and "indexes the documents from most to least semantically relevant to the query."

Current rerank models are rerank-v4.0-pro and rerank-v4.0-fast, with rerank-v3.5 and the 3.0 pair still listed, trained "for performance across 100+ languages." Names move; check the docs rather than this paragraph.

What reranking does that vector similarity cannot

The difference is when the query and the document meet.

In vector search, they never do. The document was embedded months ago, alone, into a fixed vector. The query is embedded now, alone, into another. Comparing them compares two independent summaries. That is what makes search fast, the document vectors were computed in advance, and it is also what makes it approximate. The document's vector had to be a good summary of it for every possible future query, because it did not know which one was coming.

A reranker reads the query and the document together and scores that pair. It can weigh the part of the document relevant to this question, notice that a qualifier changes the answer, and see that a document mentioning your terms is about something else.

The cost follows from the same fact. Nothing can be precomputed, because the score does not exist until the query arrives. Vector search over a million documents is one embedding plus an index lookup; reranking a million documents is a million scored pairs. That is why it is a second pass rather than a first one.

Where the rerank pass goes in a pipeline

Between retrieval and the prompt. Retrieve generously, rerank, then send the survivors.

query → embed → vector search (top 100) → rerank → top 5 → prompt

The numbers are the design. Vector search casts wide, because at that stage you are trying not to miss the right document rather than to rank it, which is recall over precision. The reranker then does precision over a list small enough to afford, and the model sees five documents instead of a hundred.

This shape fixes a specific, common complaint. If your pipeline returns roughly the right documents but the best one is seventh, and you are sending the top three, you are losing answers you already retrieved. Reordering recovers them without touching your chunking, your embeddings, or your index. That is the cheapest quality improvement available in a retrieval pipeline, which is why it appears in so many of them.

It also improves what goes into the prompt in a second way. Context engineering argues that more context is not better; five well-chosen documents beat a hundred mediocre ones both in cost and in the model's ability to use them.

When a second pass is worth the latency

It is a network call on the path to every answer, so it is not free.

The overview page I read documents no per-request latency figure, no document-count ceiling, and no token limit, only a top_n parameter for how many results come back. No latency figure is published and none was measured here. Treat the extra call as a real addition to your response time and measure it on your own data before committing.

It earns its place when the ranking is the problem. Signals: the right document is usually retrieved but not in the top few; your corpus contains many near-duplicates that vector similarity cannot separate; your queries are long or contain qualifiers that a single embedding flattens.

It does not earn its place when the right document is not being retrieved at all. Reranking cannot promote what was never in the list, and a pipeline missing documents has a chunking or embedding problem, chunking strategies covers the first, and re-reading your retrieval evaluation is the honest next step. Context evaluation is how to tell the two cases apart rather than guessing, and it is worth doing before adding a call.

Running the models inside your own network

The other distinctive thing about this vendor is where its models can run, which decides whether it is usable at all for a team whose prompts cannot leave their network.

Model Vault runs models "on dedicated, single-tenant infrastructure managed by Cohere", with optional "confidential computing, so inference runs fully encrypted." Beyond that, models "can be deployed privately in most virtual private cloud (VPC) environments", and fully on-premise deployment is offered for organizations with sensitive data. Both of the latter are contact-sales arrangements rather than something you sign up for.

If your blocker is that prompts cannot leave your network, that is the relevant sentence in this entire article. Self-hosted models makes the general case for running models yourself; what is being offered here is the middle position, where someone else operates the model but it sits inside your boundary.

Further reading

• Cohere, The Cohere platform: the three families and the deployment options, in the vendor's words.
• Cohere, Rerank overview: model names, language coverage, and the request shape.
• Storing in a vector DB: the pipeline a rerank pass slots into.
• Context evaluation: how to find out whether ranking is your problem before you add a call.