colpali-v1.2

Name: colpali-v1.2
Author: vidore

by vidore

Late-interaction visual document retrieval: search PDFs as images, not extracted text

9Kdl/month

112likes

3Bparams

HuggingFace Run it on your own data, free

Identifiers

Model ID

vidore/colpali-v1.2

Feature URI

mixpeek://image_extractor@v1/vidore_colpali_v1

Overview

ColPali retrieves documents by embedding the page image directly, skipping OCR and layout parsing entirely. Instead of one vector per page it produces a grid of patch embeddings and scores a query against all of them with ColBERT-style late interaction, so a match can come from a chart, a table, a diagram or a stamp that text extraction would have flattened or never seen.

This matters because the standard document pipeline, OCR then chunk then embed the text, silently loses exactly the content people search for in technical and financial documents. ColPali treats the page as what it is, a picture, and lets the retrieval model decide what is salient.

The cost is index size: many vectors per page rather than one. That trade is the central design decision in visual document retrieval.

Architecture

PaliGemma-3B vision-language backbone with a LoRA adapter, producing multi-vector page embeddings. Query-document scoring uses ColBERT-style late interaction (MaxSim over patch embeddings) rather than a single dot product, which is what preserves region-level detail.

Mixpeek SDK Integration

// No extractor parameter takes a Hugging Face model id (checked against
// GET /v1/discovery/extractors, which returns 13), so colpali-v1.2 runs
// on your side and the output is upserted through POST
// /v1/namespaces/{namespace_id}/documents/upsert. On Enterprise the other
// path is to upload the weights instead: POST /v1/namespaces/{id}/models
// accepts the huggingface format and a custom plugin loads them.
const res = await fetch(
  "https://api.mixpeek.com/v1/namespaces/ns_your_namespace/documents/upsert",
  {
    method: "POST",
    headers: {
      Authorization: "Bearer API_KEY",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      collection_id: "col_your_collection",
      documents: [
        {
          document_id: "asset-00412",
          // The model produces text, so it lands in payload. Give the
          // collection a text vector index and embed that text to make it
          // searchable rather than only filterable.
          payload: { extracted_text: modelOutput, source_key: "archive/2026/asset-00412" },
          vectors: { "text-embedding": embeddingOfModelOutput },
        },
      ],
    }),
  },
);

// Managed alternative, if this exact model is not the requirement:
// document_graph_extractor@v1 runs intfloat/multilingual-e5-large-instruct
// (1024-d) over a bucket, with no inference of your own.