Embedding Models on Hugging Face

Why this choice is stickier than a generation model choice

Swapping generation models is a string change. Swapping embedding models is a migration.

The reason is that vectors from different models are not comparable. They have different dimensions, and even at the same dimension count the coordinates mean different things, because each model learned its own arrangement of the space. A query embedded with model B and compared against a corpus embedded with model A does not return poor results. It returns noise.

So changing embedding models means re-embedding everything: every document, every chunk, every stored vector, plus whatever downtime or dual-write scheme keeps search working while you do it. On a large corpus that is a project.

Which makes this the one model decision worth slowing down for, and it is usually the one made fastest, copied from a tutorial in an afternoon and discovered eighteen months later.

Reading the leaderboard as a shortlist

The MTEB leaderboard ranks embedding models across a large set of tasks, and it is the obvious place to start. It is a good place to start and a bad place to stop.

Sentence Transformers, whose own library these models run in, says so with its own emphasis: "models that perform well on the leaderboard do not necessarily do well on your tasks, it is crucial to experiment with various promising models." It still recommends the leaderboard, as "an inspiration of strong Sentence Transformer models."

Inspiration, not answer. The mismatch is structural rather than a flaw in the benchmark. It aggregates across many tasks and many domains, so the top model is the best generalist. Your corpus is not a generalist problem. It is support tickets, or case law, or product descriptions in one industry with its own vocabulary, and a model ranked eighth overall may be first on text like yours.

Use it to get to five candidates. Decide between them on your own data, the way choosing the right model describes for generation models. Fifty real queries with known correct answers will separate five candidates faster than any amount of reading.

Four things to check on an embedding model card

The leaderboard column you care about is not the score. It is the metadata beside it, and each of these has a consequence that is expensive to discover later.

Dimensions. How many numbers per vector. It sets your storage bill and your index size, and larger is not automatically better. It is more memory and more comparison work per query, for a quality difference that may be small on your data. Check that your vector store supports the number.

Maximum sequence length. How much text the model reads before truncating, and truncating is what it does, silently. A model with a 512-token limit handed a 2,000-token chunk embeds the first quarter and discards the rest, with no error. Your chunk size and this number have to be decided together, which is why chunking strategies belongs next to this decision.

Prompts and instructions. Covered below, because it is the one that fails quietly.

License. Not every model on the hub is usable commercially. Some are research-only, some inherit restrictions from their training data, and the consequence of getting it wrong arrives from a lawyer rather than from a test suite.

The last two of these come from the model's own card rather than from a leaderboard row. Models on Hugging Face covers reading a card in general; what changes here is which fields you are reading it for.

The prompt prefix some embedding models require

Some embedding models expect the text to be prefixed before encoding, and they do not tell you when you forget.

The INSTRUCTOR family is the clearest example, being models "trained with instructions in mind" that require specific prompts during encoding. Others use a short prefix distinguishing a query from a document: query: on one side, passage: on the other, so the model can place a question and an answer near each other even though they are written nothing alike.

The failure mode is what makes this worth its own section. Skip the prefix and nothing errors. You get vectors. Search works. It is worse than it should be, by an amount you have no baseline to notice, and you will attribute it to chunking or to the corpus.

Read the model card's usage example and copy it in full, including the strings that look decorative. If the card shows a prefix, it is load-bearing.

Running an embedding model locally

Once chosen, the library is the easy part:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-small-en-v1.5")
vectors = model.encode(["a support ticket about a refund"])

What that hides is a download and a model in memory. Sentence Transformers' practical advice for the choosing stage is to "filter away the large models that might not be feasible without excessive hardware", and its tables report throughput as queries per second on GPU and CPU, which is the number that decides whether this is viable in your deployment.

Two things to size before you commit. Indexing is a bulk job: embedding a corpus on CPU can run for hours where a GPU runs for minutes, and it is a cost you pay again on every re-embedding. Query-time embedding is in your latency budget, since every search embeds the query first, and on CPU that is tens of milliseconds you are adding to every request.

Against that, the argument for running your own: no per-token bill, no rate limit, no third party seeing your corpus, and a model that does not change underneath you. That last one matters more here than elsewhere. A hosted embedding model that is updated is a corpus that needs re-embedding on someone else's schedule. Self-hosted models makes the general case; embeddings are where it is strongest.

Further reading

• Sentence Transformers, Pretrained models: the leaderboard caveat, normalization, instruction-tuned models, and throughput tables.
• MTEB, the embedding leaderboard: the shortlist.
• Models on Hugging Face: reading a model card and its license.
• OpenAI embeddings API: the hosted alternative, and what you give up by not running your own.