Local Embedding Models: Choosing One for Search
The embedding model decides which passages a private search system can find at all. Here is what to check on the model card and how to test two candidates against real questions.

A popular small embedding model, all-MiniLM-L6-v2, reads at most 256 tokens of each input by default and silently drops the rest, according to its model card. Feed it a 600-token contract clause and the obligations in the second half never reach the index. No error appears; those passages just never come back in search. That one detail explains why choosing among local embedding models is mostly about reading the card carefully and then testing on your own questions, not about picking the top name on a leaderboard.
What an embedding model actually does
An embedding model turns a piece of text into a fixed-length list of numbers, a vector. Texts with similar meaning land near each other, so "how much annual leave do new hires get" sits close to a handbook paragraph about "vacation entitlement in the first year", even though the two share almost no words.
In a private search or RAG system the model runs twice. At indexing time it embeds every chunk of every document and the vectors go into a store. At query time it embeds the question, and the store returns the chunks whose vectors are nearest. The language model that writes the answer only ever sees what this step retrieves. If the right paragraph is not in the top results, no amount of prompt work recovers it.
Two consequences follow:
- The model is part of your data format. Vectors from different models are not comparable, so changing models means re-embedding the whole corpus.
- Embedding models are small next to chat models. Many run comfortably on a CPU for modest volumes, and on a GPU they can share the card with the main model. The full architecture is covered in the overview of private RAG over internal documents.
The properties that matter on a model card
Five lines on a model card decide most of the outcome. Here they are for several open models, as stated on their Hugging Face cards in October 2026:
| Model | Dimensions | Max input | Languages | License |
|---|---|---|---|---|
| all-MiniLM-L6-v2 | 384 | 256 tokens | English | Apache-2.0 |
| nomic-embed-text-v1.5 | 768 (down to 64) | 8192 tokens | English | Apache-2.0 |
| multilingual-e5-large | 1024 | 512 tokens | About 100 | MIT |
| BGE-M3 | 1024 | 8192 tokens | 100+ | MIT |
| Qwen3-Embedding-0.6B | up to 1024 (from 32) | 32K tokens | 100+ | Apache-2.0 |
The table is a starting point for a shortlist, not a ranking.
Dimensions and storage
Dimensions set the size of each vector, which sets storage and memory. At 32-bit floats each number takes 4 bytes. For one million chunks:
- 384 dimensions: 1,000,000 x 384 x 4 = 1.54 GB
- 768 dimensions: 1,000,000 x 768 x 4 = 3.07 GB
- 1024 dimensions: 1,000,000 x 1024 x 4 = 4.10 GB
Index structures add overhead on top of this. More dimensions can hold finer distinctions, but the gain is not automatic, and a small office archive of 50,000 chunks is only about 0.2 GB even at 1024 dimensions. Some models are trained so the vector can be cut short with modest loss (Matryoshka training). The nomic card lists 768, 512, 256, 128 and 64 as supported sizes and reports a lower benchmark score at each step down, and Qwen3-Embedding accepts output sizes from 32 to 1024. That lets you trade accuracy for storage after testing, without changing models.
Maximum input length
This is the property that bites quietly. multilingual-e5-large truncates inputs past 512 tokens, MiniLM at 256, while BGE-M3 and nomic accept 8192. The limit must be larger than your chunk size, with room to spare. If chunks are 400 tokens, a 256-token model is wrong for the job no matter how well it scores elsewhere. Long limits are not a reason to embed whole documents either: one vector for a 30-page report blurs every topic in it together. Chunk sizes are a separate decision, covered in chunking documents for RAG.
Languages
An English-only model will still produce vectors for German or Spanish text, and they will be poor. If your archive or your staff mix languages, shortlist only models whose cards claim those languages, then test with real mixed-language questions. A useful check: ask a question in one language whose answer exists only in another, and see whether the right passage comes back.
Prefixes and instructions
Several models expect a marker on every input, and the card says so in plain words:
- multilingual-e5-large wants
query:before questions andpassage:before documents, and its card warns of degraded performance if they are left out. - nomic-embed-text-v1.5 requires
search_query:andsearch_document:for retrieval. - Qwen3-Embedding asks for a one-sentence task instruction in front of queries only. Its authors say leaving it out typically costs about 1% to 5% on retrieval.
Forgetting these is one of the most common reasons a well-reviewed model underperforms in a private deployment. Whatever library or runtime does the embedding, check that it adds the prefix or add it yourself, the same way at indexing and query time.
License
Every model in the table above carries a permissive license (MIT or Apache-2.0) per its card. Not every embedding model does; some are released under non-commercial terms. Read the license on the card before any testing, because a model that wins the comparison but cannot be used commercially wastes the effort.
Size, speed and where it runs
Embedding cost is front-loaded. Indexing a large archive once is the heavy job; after that, each question is a single short embedding. For a back-of-envelope estimate, measure throughput on your own hardware with a sample of real chunks and divide. If a test run embeds 2,000 chunks in 100 seconds, that is 20 chunks per second, and 400,000 chunks take 400,000 / 20 = 20,000 seconds, about 5.6 hours. That is an overnight job, done once, plus small increments for new documents.
Larger models such as the 4B and 8B Qwen3-Embedding variants (2560 and 4096 dimensions per their cards) may retrieve better on hard queries, but they multiply indexing time and storage. Test whether the gain shows up on your questions before paying for it.
How to compare two candidates on your own queries
Public leaderboards such as MTEB average results across many datasets that are not yours. Legal clauses, part numbers and internal jargon behave differently from Wikipedia and web forums. A small private test settles the question for your data.
- Collect 50 real questions from staff, the kind they would type, including vague ones.
- For each question, mark the chunk or chunks that contain the answer. This is the slow part, and the most valuable.
- Chunk the corpus once. Use identical chunks for both models so only the model changes.
- Embed the corpus with model A and model B into two separate indexes, applying each model's prefixes correctly.
- Run every question against both and record where the first correct chunk appears in the ranked results.
Then compute two numbers per model.
Recall at 5 is the share of questions where a correct chunk appears in the top five. If model A finds it for 41 of 50 questions, that is 41 / 50 = 82%. If model B manages 36 of 50, that is 72%.
Mean reciprocal rank (MRR) rewards putting the answer near the top. Each question scores 1 divided by the rank of the first correct chunk, or 0 if it is missing. For three questions answered at rank 1, rank 3, and not found, MRR is (1 + 1/3 + 0) / 3 = 0.44. Higher is better, and 1.0 means the right chunk was always first.
With 50 questions, a gap of one or two questions is noise. A gap of five or more is worth taking seriously. Read the misses for both models too: if both fail on the same questions, the problem is likely the chunking or the questions themselves, not the embedder.
Keep the question set. When a newer model appears, the same 50 questions give a comparison in an afternoon, and the set grows into the broader test described in evaluating a RAG system before staff rely on it.
A simple decision rule: pick the smallest model whose recall at 5 is within a couple of points of the best candidate, whose input limit clears your chunk size, and whose license fits. Spend the saved storage and indexing time on better chunking and a reranker if results still fall short.
Frequently asked questions
Can the embedding model be changed later without re-indexing?
No. Each model produces vectors in its own space, so a query embedded with a new model cannot be matched against documents embedded with the old one. Plan a full re-embedding run whenever the model changes, and keep the original text of every chunk so the rerun is possible.
Should the embedding model come from the same family as the chat model?
There is no need. The embedder only decides which passages are retrieved; the chat model reads plain text afterwards and never sees the vectors. Choose each on its own test results.
Which similarity measure should the vector store use?
Use the one the model card's example code uses. Most of these cards normalize vectors to unit length and compare them with cosine similarity, and for normalized vectors cosine and dot product give the same ranking. A store set to plain Euclidean distance on unnormalized vectors can return a different order without any error, so set the metric once and keep it identical for indexing and queries.
What is a reranker and is it a replacement for a better embedder?
A reranker is a second model that reads the question and each of the top candidates together and reorders them. It improves ordering within what was retrieved but cannot recover a passage the embedder missed entirely, so it complements a good embedding model rather than replacing one.
Ready to Modernize Your Business?
Get your AI automation roadmap in minutes, not months.
Analyze Your Workflows →