e5-omni-7B

Name: e5-omni-7B
Author: Haon-Chen

by Haon-Chen

State-of-the-art omnimodal embedding with explicit cross-modal alignment

261dl/month

~9Bparams

HuggingFace Run it on your own data, free

Identifiers

Model ID

Haon-Chen/e5-omni-7B

Feature URI

mixpeek://image_extractor@v1/haon_chen_e5_omni_7b_v1

Overview

E5-Omni is Microsoft's omnimodal embedding model that achieves state-of-the-art on the MMEB-V2 benchmark across text, image, audio, and video tasks. Built on Qwen2.5-Omni-7B, it introduces modality-aware temperature calibration, controllable negative curriculum learning, and batch whitening for cross-modal alignment.

On Mixpeek, E5-Omni delivers the highest-quality cross-modal embeddings available: its explicit alignment techniques mean that similarity scores between different modalities (e.g., text query vs. audio clip) are more reliable than models trained with simple contrastive objectives.

Architecture

Qwen2.5-Omni-7B backbone with three alignment components: (1) modality-aware temperature calibration, (2) controllable negative curriculum that progressively masks easy negatives, (3) batch whitening and covariance alignment. ~9B total parameters. Unified embedding space for all modalities.

Mixpeek SDK Integration

// No extractor parameter takes a Hugging Face model id (checked against
// GET /v1/discovery/extractors, which returns 13), so e5-omni-7B runs
// on your side and the output is upserted through POST
// /v1/namespaces/{namespace_id}/documents/upsert. On Enterprise the other
// path is to upload the weights instead: POST /v1/namespaces/{id}/models
// accepts the huggingface format and a custom plugin loads them.
const res = await fetch(
  "https://api.mixpeek.com/v1/namespaces/ns_your_namespace/documents/upsert",
  {
    method: "POST",
    headers: {
      Authorization: "Bearer API_KEY",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      collection_id: "col_your_collection",
      documents: [
        {
          document_id: "asset-00412",
          // The vector name has to match a vector index on the collection.
          vectors: { "multimodal-embedding": yourVector },
          payload: { source_key: "archive/2026/asset-00412" },
        },
      ],
    }),
  },
);

// Managed alternative, if this exact model is not the requirement:
// multimodal_extractor@v2 runs google/gemini-embedding-2
// (3072-d) over a bucket, with no inference of your own.

Capabilities

SOTA on MMEB-V2 benchmark (66.4 overall across 78 tasks)
Best audio retrieval among omnimodal models (37.7 Recall@1 on AudioCaps)
Unified text, image, audio, and video embeddings
Explicit cross-modal alignment for reliable similarity scores
Outperforms 3B models by 15+ points on MMEB-V2

Use Cases on Mixpeek

Cross-modal retrieval: find audio clips matching a text description

Multimedia RAG: unified retrieval across all content types

Audio-visual search: query meetings by both spoken content and visual slides

Research libraries: embed papers, presentations, and recorded talks together

Benchmarks

Dataset	Metric	Score	Source
MMEB-V2 (78 tasks)	Overall	66.4	Chen et al., 2025: arxiv,2601.03666
MMEB-V2 Image (36 tasks)	Hit@1	71.2	Chen et al., 2025: arxiv,2601.03666
AudioCaps	Recall@1	37.7	Chen et al., 2025: arxiv,2601.03666