MTEB: The Massive Text Embedding Benchmark

MTEB is the benchmark you consult to choose an embedding model. It tests one model, frozen, across eight task types: retrieval, classification, clustering, reranking, pair classification, semantic similarity, summarisation scoring, and cross-lingual bitext mining. The breadth is the point, and it is also the trap: the aggregate rank mixes jobs your pipeline may never do, so the honest read is the task sub-score that matches your workload.

MTEB, introduced by Muennighoff et al. in 2022, is a multi-task benchmark for text-embedding models. Where earlier evaluations tested embeddings on one job (usually semantic similarity or a single retrieval set), MTEB runs a single frozen model across a wide spread of tasks and reports how it does on each. The original English core spanned eight task types across 58 datasets and 112 languages, which was a step-change in coverage at the time and quickly became the default reference for embedding-model selection.

The defining choice is that the embedding model is not fine-tuned per task. A model produces vectors; a lightweight task head (a logistic-regression probe for classification, a cosine-similarity comparison for STS and retrieval) turns those vectors into predictions. This isolates the quality of the embedding space itself rather than the capacity of a downstream classifier, which is what you want when the question is "which embedding model should I put in my pipeline".

That breadth is why MTEB became the canonical embedding benchmark, and also why it is easy to misread. A model with a high aggregate MTEB rank is good across many jobs on average; it is not necessarily the best at the one job your pipeline actually runs. For retrieval-augmented generation, the retrieval sub-score is the number that matters, and it can diverge sharply from the aggregate rank.

Each MTEB task type has its own primary metric, chosen to match the task. Because the metrics are not on a common scale, MTEB cannot meaningfully average raw scores across task types; the leaderboard aggregates by rank instead.

For a RAG workload, retrieval and reranking are the load-bearing rows; for a deduplication or clustering job, clustering and pair classification matter more; for a cross-lingual application, bitext mining is decisive. Reading the task table against your own workload is the whole skill of using MTEB well.

The original MTEB was English-centric in practice despite covering 112 languages, and the benchmark has since been expanded substantially. MMTEB (Massive Multilingual Text Embedding Benchmark), published at ICLR 2025 by Enevoldsen and 85 co-authors, grew the suite to over 500 quality-controlled evaluation tasks across more than 250 languages. This is now the broadest embedding evaluation in public use.

The expansion changed how the leaderboard works. Results are organised into named benchmark splits rather than a single global table (an English v2 split, a multilingual split, and language- or domain-specific tracks), and the default ranking uses a Borda count: models are ranked per task and rewarded for placing consistently well across many tasks rather than topping a handful. This matters when you quote a number. An "MTEB score" without its split is ambiguous, because an English-v2 rank and a full-multilingual rank measure different things and a model can lead one while trailing the other.

MTEB and BEIR are quoted together on almost every embedding model card and are easy to conflate. BEIR is retrieval-only: 18 datasets, one job, headline metric nDCG@10. MTEB is the superset benchmark, covering eight task types including retrieval but also classification, clustering, reranking, semantic similarity, summarisation and bitext mining. MTEB's retrieval track incorporates BEIR datasets directly, so the two overlap on retrieval and diverge everywhere else.

The practical rule mirrors the BEIR page: use BEIR when the question is narrow (how good is this retriever at finding relevant documents across domains), and use MTEB when the question is broad (how good is this embedding model for every job a production pipeline runs, from retrieval to reranking to deduplication). For RAG specifically, the MTEB retrieval sub-score and the BEIR average are measuring nearly the same thing; read them together and treat a divergence between them as a signal to look at the per-dataset breakdown.

Three caveats travel with an MTEB rank. First, benchmark-directed training: MTEB is influential enough that some embedding models are trained with its task types and datasets in view, so a high rank can reflect optimisation toward MTEB as much as general embedding quality. This is the embedding-model form of the wider contamination problem: a model scored on datasets it was tuned on is not being tested cleanly. Second, saturation and noise: several classification and STS datasets are close to solved for frontier models, so top-of-table gaps compress and one-point differences stop being meaningful. Third, aggregate-versus-task mismatch: the headline rank mixes task types, so it can rank a model highly for a workload it is actually mediocre at.

None of these break MTEB as the reference embedding benchmark; they are reasons to read the task sub-scores rather than the aggregate rank. The wider point that benchmarks miss production behaviour applies to embeddings too: MTEB says nothing about latency, index size, or how an embedding behaves on your specific corpus, all of which matter as much as leaderboard rank in a live pipeline.

Reach for MTEB first when you are selecting an embedding model and you care about performance across more than one job. Read the task sub-scores that match your workload rather than the aggregate rank, confirm which split (English v2 or multilingual) the number is on, and check the model was not fine-tuned on the exact datasets it is scored on. For a RAG pipeline, quote the retrieval sub-score, pair it with BEIR for the retrieval cross-check, and add an end-to-end benchmark such as RAGAS or a golden dataset scored with LLM-as-judge to check generation faithfulness. See our RAG benchmark comparison for how the embedding, retrieval and end-to-end numbers fit together.