How do I search content that is in a different language from my query?
Embed the query and the content into one shared vector space, so that a sentence and its translation land in nearly the same place. A multilingual embedding model does this by construction: it is trained so "red sports car", "voiture de sport rouge" and "赤いスポーツカー" produce nearly identical vectors, which means a similarity search run in one language retrieves matching content in all the others without translating anything at query time.
The alternative is to translate first, either the query or the whole corpus, and run ordinary same-language search. That is simpler to reason about and it is the right answer in narrower cases than people expect. Which one you want depends on how many languages you have, whether you know a document's language at write time, and whether you can afford to be wrong about it.
Which approach should I use?
| Approach | How it works | Strength | Where it breaks |
| Multilingual embeddings | One model embeds every language into a shared space | One index, no query-time translation, handles unknown languages | Quality varies sharply by language; weakest on low-resource ones |
| Translate the query | Translate the query into each corpus language, search each | Reuse a strong monolingual model per language | One translation call per language per query; translation errors become recall failures |
| Translate the corpus | Translate everything into one pivot language at ingest | Search is monolingual and simple | Pay translation on the whole corpus; re-translate on model change; the original wording is gone |
| Per-language indexes | Detect language, route to that language's index | Best per-language quality | Needs reliable language detection, and cross-language recall is lost by design |
What is a shared embedding space, and why does it work?
A multilingual encoder is trained so that meaning, not surface form, determines where a vector lands. The usual mechanism is a contrastive objective over translation pairs: the model sees a sentence and its translation, and is pushed to place them close together while pushing unrelated sentences apart. Do that across enough pairs and enough languages, and the geometry ends up organised by meaning with language becoming almost a nuisance dimension.
The practical consequence is the thing that surprises people: you never detect the query's language, never branch on it, and never translate. The vector for a French query is simply near the vectors of relevant English documents, so an ordinary nearest-neighbour search finds them.
The limits follow from the same mechanism. Alignment quality tracks how much parallel data existed for a language pair, which is why these models are strong on the well-resourced European and East Asian languages and noticeably weaker on low-resource ones. Idiom, domain jargon and named entities align worst, because they are exactly what appears least often in parallel corpora.
Which multilingual embedding models are worth considering?
| Model | Languages | Notable property | Licence |
| BAAI/bge-m3 | 100+ | Dense, sparse and multi-vector retrieval from one model | MIT |
| multilingual-e5-large-instruct | 100+ | Instruction-tuned; expects query/passage prefixes | MIT |
| LaBSE | 109 | Built specifically for translation-pair alignment | Apache 2.0 |
| jina-embeddings-v3 | Multilingual | Task-specific LoRA adapters | CC BY-NC 4.0, non-commercial |
The instruction-prefix detail is the most common silent failure with the E5 family: it expects "query:" and "passage:" prefixes, and omitting them degrades results quietly rather than erroring, so the system looks like it works and simply retrieves worse than it should.
Does cross-lingual search work for images, video and audio?
Yes, and it is where the idea earns the most, because the content has no language at all.
An image does not need translating. A vision-language model already embeds images and text into a shared space, so if the text side of that model is multilingual, you get cross-lingual image search for free: a Spanish query retrieves the picture directly, with nothing in between. The same holds for video through sampled frames, since a frame is an image.
Audio and documents are different, because they carry real language. Speech becomes text through transcription, and that text has a language you did not choose; scanned documents become text through OCR with the same property. So a mixed archive usually ends up cross-lingual whether or not anyone planned it: the video is language-free, its transcript is Portuguese, and the slide deck attached to it is English.
That is the case that defeats per-language routing. There is no single language to route on, because one asset has several.
How do I evaluate cross-lingual retrieval before trusting it?
Measure per language pair, never in aggregate. A single averaged recall number hides the failure mode that matters, because a model can look strong overall while being unusable for one language that happens to be a small share of your test set and a large share of your users.
Build the evaluation set from your own content. Public benchmarks tell you about general alignment and not about your domain, and domain vocabulary is where alignment is weakest.
Include the case where query and document are the same language. Cross-lingual capability sometimes costs monolingual quality compared to a strong single-language model, and that trade is worth knowing before you make it rather than after.
Watch for the asymmetry: retrieving English documents with a French query and retrieving French documents with an English query are different operations and frequently score differently on the same model. If your users query in one direction, evaluate that direction.
Frequently Asked Questions
Is translating the query as good as a multilingual embedding model?
For a small number of languages it often is, and it has a real advantage: you can use the strongest available monolingual model for the target language, which usually beats a multilingual model on that language alone. What it costs is a translation call in the query path, which adds latency and a failure mode, and it scales linearly with the number of languages you search. It also inherits translation errors directly into recall, and a mistranslated technical term simply returns nothing with no indication that anything went wrong.
Do I need to detect the language of my documents?
Not with a multilingual embedding model, and that is most of its value. The encoder places a document by meaning regardless of language, so nothing branches on a detected language and nothing breaks when detection is wrong. You do need detection for per-language routing or corpus translation, which is why those designs are more brittle: language detection is unreliable on short text, on mixed text, and on the code-switched content that is common in real archives.
Why do results get worse for some languages than others?
Because alignment quality follows the parallel data the model was trained on. Well-resourced languages have vast translation corpora, so their vectors sit tightly aligned with English; low-resource languages have far less, so their vectors sit in roughly the right region rather than precisely on target, and precision at the top of the ranking suffers first. This is a property of the training data rather than a bug, and no amount of tuning at query time fixes it. Measure the languages you actually serve rather than trusting a headline count of supported languages.
Can I mix languages inside a single index?
Yes, and with a multilingual model that is the intended design. One index, one model, every language, and a query in any of them searches all of the content. The rule that breaks people is the same one that governs any vector index: every vector in it must come from the same model with the same preprocessing. Mixing outputs from two embedding models in one index produces nonsense similarity scores, and switching multilingual models later means re-embedding the whole corpus.
How does cross-lingual search interact with keyword search?
Poorly, and it is worth planning around. Lexical retrieval such as BM25 matches tokens, so it has no cross-lingual capability whatsoever: a French query scores zero against an English document that means exactly the same thing. In a hybrid system the dense half carries every cross-language match and the sparse half contributes only when query and document share a language. That is not a reason to drop the sparse half, since it is what catches exact identifiers, product codes and names, and it does mean your fusion weights behave differently on cross-language queries than on same-language ones.
What about searching video in another language?
The visual content needs no translation, so a multilingual vision-language model handles it directly: embed sampled frames, query in any language, retrieve matching moments. The complication is that most video search is not purely visual. Transcripts, on-screen text and attached documents all carry real language, so a production video search usually blends a language-free visual signal with several language-bearing text signals. Keeping those as separate retrieval stages rather than one blended score is what lets you tell whether a miss came from the vision side or the transcript side.
Key Takeaways
Where to go next
Models that do this: BAAI/bge-m3 · multilingual-e5-large-instruct · best multimodal embedding models · best self-hosted embedding models
The retrieval machinery around it: hybrid search: BM25 and vector fusion · cross-encoder reranking · best rerankers
Mixpeek indexes unstructured files in object storage at the token level, so a video becomes scenes, frames, a transcript and any attached documents, each embedded and individually searchable. That separation is what makes a mixed-language archive tractable: the visual signal never needed a language, and the text signals can carry several without one index having to pick. See the vector store or pricing.