gme-Qwen2-VL-2B-Instruct

Name: gme-Qwen2-VL-2B-Instruct
Author: Alibaba-NLP

by Alibaba-NLP

Unified multimodal embedding: text, images, and visual documents in one vector space

2.2Bparams

HuggingFace Run it on your own data, free

Identifiers

Model ID

Alibaba-NLP/gme-Qwen2-VL-2B-Instruct

Feature URI

Overview

GME (General Multimodal Embedding) turns a vision-language model, Qwen2-VL-2B-Instruct, into a single-vector embedding model. Unlike a CLIP-style dual encoder that keeps separate towers for text and images, GME feeds text, an image, or an interleaved image+text pair through the same VLM and pools the final hidden states into one dense vector. Because every modality lands in the same space, you can retrieve any-to-any: text→image, image→text, text→text, and (image+text)→(image+text).

The practical payoff is visual-document retrieval. A slide, a scanned invoice, or a chart-heavy PDF page is encoded as an image, so layout, figures, and text are embedded together without a brittle OCR-then-embed pipeline. On Mixpeek, GME powers a single multimodal_embedding index where an agent's text query reaches document pages, product photos, and screenshots side by side.

Architecture

Built on Qwen2-VL-2B-Instruct (~2.2B params). The VLM encodes the input; the last-layer hidden states are mean-pooled into one dense vector (instruction-tunable via a task prompt). Inherits Qwen2-VL's dynamic-resolution vision encoder, so images are processed at native aspect ratio without forced 224×224 cropping: important for dense document pages. Trained with contrastive learning over text, image, and fused image-text pairs plus hard negatives.

Mixpeek SDK Integration

// No extractor parameter takes a Hugging Face model id (checked against
// GET /v1/discovery/extractors, which returns 13), so gme-Qwen2-VL-2B-Instruct runs
// on your side and the output is upserted through POST
// /v1/namespaces/{namespace_id}/documents/upsert. On Enterprise the other
// path is to upload the weights instead: POST /v1/namespaces/{id}/models
// accepts the huggingface format and a custom plugin loads them.
const res = await fetch(
  "https://api.mixpeek.com/v1/namespaces/ns_your_namespace/documents/upsert",
  {
    method: "POST",
    headers: {
      Authorization: "Bearer API_KEY",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      collection_id: "col_your_collection",
      documents: [
        {
          document_id: "asset-00412",
          // The vector name has to match a vector index on the collection.
          vectors: { "multimodal-embedding": yourVector },
          payload: { source_key: "archive/2026/asset-00412" },
        },
      ],
    }),
  },
);

// Managed alternative, if this exact model is not the requirement:
// multimodal_extractor@v2 runs google/gemini-embedding-2
// (3072-d) over a bucket, with no inference of your own.

Capabilities

Single shared space for text, image, and interleaved image-text inputs
Any-to-any retrieval (text↔image, image↔image, fused queries)
Strong visual-document retrieval (ViDoRe): embeds page layout + figures, no OCR step
Dynamic image resolution inherited from Qwen2-VL (no fixed crop)
Instruction-conditioned embeddings via a task prompt

Use Cases on Mixpeek

Multimodal RAG over corpora that mix text passages and images in one index

Visual-document search: slides, scanned PDFs, charts, and screenshots by text query

Product search where the query or catalog item is image + text together

Agent retrieval where queries and documents span modalities without per-modality pipelines

Benchmarks

Dataset	Metric	Score	Source
UMRB (Universal Multimodal Retrieval Benchmark)	introduced + evaluated	-	Introduced alongside the GME paper; the 2B model is a top unified-embedding performer for its size at release. See paper for full table.
ViDoRe (visual document retrieval)	nDCG@5	-	Competitive screenshot/document retrieval without an OCR stage.