RAG Prepare

RAG Prepare stage showing document formatting for LLM context

The RAG Prepare stage formats search results for LLM consumption by managing token budgets, formatting documents, and adding citations. This is a preparation stage that does NOT call an LLM - it prepares content for downstream LLM stages or external LLM calls.

Stage Category: APPLYTransformation:

single_context mode: N documents → 1 combined context document
formatted_list mode: N documents → N formatted documents

When to Use

Use Case	Description
Before LLM generation	Prepare context for summarization or Q&A
Token budget management	Fit multiple docs into context window
Citation tracking	Enable source attribution in responses
Consistent formatting	Standardize document format for LLM input

When NOT to Use

Scenario	Recommended Alternative
Want LLM to generate summary	`summarize` stage (calls LLM)
Don’t need token management	Pass documents directly
Simple pass-through	Skip this stage

Parameters

Parameter	Type	Default	Description
`max_tokens`	integer	`8000`	Maximum tokens for combined output
`tokenizer`	string	`cl100k_base`	Tokenizer to use (GPT-4 compatible)
`truncation_strategy`	string	`priority_truncate`	How to handle token overflow
`output_mode`	string	`single_context`	Output format
`document_template`	string	`[{{CONTEXT.INDEX}}] {{DOC.content}}\n\n`	Template for each document
`content_field`	string	`content`	Field to extract content from
`separator`	string	`\n`	Separator between documents
`citation`	object	`{style: "numbered"}`	Citation configuration

Truncation Strategies

Strategy	Behavior
`priority_truncate`	Include docs in score order, truncate last to fit
`proportional`	Give each doc proportional token budget
`drop_last`	Include complete docs until limit, drop remaining

Output Modes

Mode	Output	Use Case
`single_context`	1 document with combined `context` string	Direct LLM input
`formatted_list`	N documents with `formatted_content` field	Custom processing

Configuration Examples

{
  "stage_name": "rag_prepare",
  "stage_type": "apply",
  "config": {
    "stage_id": "rag_prepare",
    "parameters": {
      "max_tokens": 8000,
      "output_mode": "single_context"
    }
  }
}

{
  "stage_name": "rag_prepare",
  "stage_type": "apply",
  "config": {
    "stage_id": "rag_prepare",
    "parameters": {
      "max_tokens": 4000,
      "document_template": "[{{CONTEXT.INDEX}}] {{DOC.metadata.title}}\n{{DOC.content}}\n\n",
      "truncation_strategy": "priority_truncate",
      "citation": {
        "style": "numbered",
        "include_title": true
      }
    }
  }
}

{
  "stage_name": "rag_prepare",
  "stage_type": "apply",
  "config": {
    "stage_id": "rag_prepare",
    "parameters": {
      "max_tokens": 32000,
      "tokenizer": "cl100k_base",
      "truncation_strategy": "proportional"
    }
  }
}

{
  "stage_name": "rag_prepare",
  "stage_type": "apply",
  "config": {
    "stage_id": "rag_prepare",
    "parameters": {
      "output_mode": "formatted_list",
      "document_template": "Source: {{DOC.metadata.source}}\n{{DOC.content}}"
    }
  }
}

{
  "stage_name": "rag_prepare",
  "stage_type": "apply",
  "config": {
    "stage_id": "rag_prepare",
    "parameters": {
      "max_tokens": 8000,
      "document_template": "Document {{CONTEXT.INDEX}}:\nTitle: {{DOC.metadata.title}}\nURL: {{DOC.metadata.url}}\n\n{{DOC.content}}\n\n---\n",
      "citation": {
        "style": "bracketed",
        "include_url": true
      }
    }
  }
}

Template Placeholders

Placeholder	Description
`{{CONTEXT.INDEX}}`	1-based position in result set (1, 2, 3…)
`{{CONTEXT.CITATION}}`	Citation marker based on citation.style
`{{DOC.*}}`	Any document field (e.g., `{{DOC.content}}`, `{{DOC.metadata.title}}`)

Citation Styles

Style	Output	Example
`numbered`	`[1]`, `[2]`, `[3]`	Default, clean
`bracketed`	`[doc_id]`	Document ID references
`footnote`	Superscript numbers	Academic style
`none`	No citations	When not needed

Output Schema

single_context Mode

{
  "rag_context": "[1] First document content...\n\n[2] Second document content...",
  "citations": [
    {"index": 1, "title": "Document Title", "document_id": "doc_123"},
    {"index": 2, "title": "Another Title", "document_id": "doc_456"}
  ]
}

formatted_list Mode

Each document gets:

{
  "document_id": "doc_123",
  "formatted_content": "[1] Title\nContent here...",
  "original_content": "Content here...",
  "metadata": {...}
}

Citing Timestamped Source Clips (Video / Audio)

The default citation object is {index, title, document_id} — enough to name the source, but it does not inline the timestamp or a link back to the source clip. For investigative / RAG use cases where every claim must resolve to a timestamped, clickable source (e.g. “watch this at 03:12”), surface the anchor yourself — the data is already on every document, so this needs no new stage or endpoint:

Video/audio segment documents carry start_time_s / end_time_s (see Universal Extractor → Output).
Every document carries _internal.lineage.root_object_id pointing back to the original source object (see Lineage Traversal).

Reference those fields directly in document_template so the timestamp travels into the LLM context (and into any answer the LLM cites):

Timestamped clip citations

{
  "stage_name": "rag_prepare",
  "stage_type": "apply",
  "config": {
    "stage_id": "rag_prepare",
    "parameters": {
      "max_tokens": 8000,
      "output_mode": "single_context",
      "document_template": "[{{CONTEXT.INDEX}}] {{DOC.metadata.title}} @ {{DOC.start_time_s}}s–{{DOC.end_time_s}}s (source: {{DOC._internal.lineage.root_object_id}})\n{{DOC.text}}\n\n",
      "citation": { "style": "numbered", "include_title": true }
    }
  }
}

To render a clickable clip link, hydrate the document_id returned in the citations[] array (or the root_object_id above) into a playable URL with GET /v1/documents/{id}?expand=root_object — see Lineage Traversal. Do this in the app layer after execute; the timestamp and source object are preserved end-to-end in the document, they are simply not inlined into the structured citations[] array by default.

Citations attach at the document level. rag_prepare (and summarize’s source_document_ids) cite the set of source documents behind the context, not individual sentences — per-claim/per-sentence attribution is up to the downstream LLM prompt. Ask the LLM to emit the [n] marker inline with each claim so the numbered citation resolves back to the timestamped clip above.

Performance

Metric	Value
Latency	< 10ms
Token counting	Uses tiktoken (accurate)
No LLM calls	Pure formatting

This stage does NOT call an LLM. It only formats content for LLM consumption. Use the summarize stage if you want LLM-generated summaries.

Common Pipeline Patterns

Search + Prepare + External LLM

[
  {
    "stage_name": "semantic_search",
    "stage_type": "filter",
    "config": {
      "stage_id": "feature_search",
      "parameters": {
        "searches": [
          { "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 50 }
        ],
        "final_top_k": 50
      }
    }
  },
  {
    "stage_name": "rerank",
    "stage_type": "sort",
    "config": {
      "stage_id": "rerank",
      "parameters": {
        "inference_name": "BAAI__bge_reranker_v2_m3",
        "top_k": 10
      }
    }
  },
  {
    "stage_name": "rag_prepare",
    "stage_type": "apply",
    "config": {
      "stage_id": "rag_prepare",
      "parameters": {
        "max_tokens": 8000,
        "output_mode": "single_context",
        "citation": {"style": "numbered"}
      }
    }
  }
]

The output rag_context can then be passed to an external LLM call.

vs Summarize Stage

Feature	rag_prepare	summarize
Calls LLM	No	Yes
Output	Formatted context	Generated summary
Latency	< 10ms	500-2000ms
Cost	Free	LLM API costs
Use case	Prepare for external LLM	End-to-end RAG

Summarize - LLM-powered summarization
JSON Transform - Custom document formatting
LLM Enrich - Extract structured data with LLM

Get started

Connect your data

Extract features

Build retrievers

Enrich & organize

Integrate & operate

Resources

When to Use

When NOT to Use

Parameters

Truncation Strategies

Output Modes

Configuration Examples

Template Placeholders

Citation Styles

Output Schema

single_context Mode

formatted_list Mode

Citing Timestamped Source Clips (Video / Audio)

Performance

Common Pipeline Patterns

Search + Prepare + External LLM

vs Summarize Stage

​When to Use

​When NOT to Use

​Parameters

​Truncation Strategies

​Output Modes

​Configuration Examples

​Template Placeholders

​Citation Styles

​Output Schema

​single_context Mode

​formatted_list Mode

​Citing Timestamped Source Clips (Video / Audio)

​Performance

​Common Pipeline Patterns

​Search + Prepare + External LLM

​vs Summarize Stage

​Related

When to Use

When NOT to Use

Parameters

Truncation Strategies

Output Modes

Configuration Examples

Template Placeholders

Citation Styles

Output Schema

single_context Mode

formatted_list Mode

Citing Timestamped Source Clips (Video / Audio)

Performance

Common Pipeline Patterns

Search + Prepare + External LLM

vs Summarize Stage

Related