> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# JSON Transform
> Transform document structure using Jinja2 templates for API payloads or custom schemas
The JSON Transform stage applies a Jinja2 template to each document, rendering the template with full document context and replacing the document with the parsed JSON output. Use this to reformat documents for external APIs or reshape data for downstream consumers.
**Stage Category**: APPLY (1-1 Transformation)
**Transformation**: N documents → N documents (or fewer with `fail_on_error=False`)
## When to Use
| Use Case | Description |
| --------------------------- | ------------------------------------------------- |
| **External API formatting** | Format documents for webhook payloads |
| **Response optimization** | Remove unused fields to reduce bandwidth |
| **Schema adaptation** | Convert internal format to client-specific format |
| **Conditional outputs** | Include fields based on document properties |
| **Array flattening** | Transform nested structures to flat arrays |
| **Field renaming** | Rename or reorganize document fields |
## When NOT to Use
| Scenario | Recommended Alternative |
| ----------------------- | ---------------------------------- |
| Filtering documents | `attribute_filter` or `llm_filter` |
| Sorting documents | `sort_by_field` or `rerank` |
| Enriching with new data | `document_enrich` or `api_call` |
| Joining external data | `taxonomy_enrich` |
## Parameters
| Parameter | Type | Default | Description |
| --------------- | ------- | ---------- | ---------------------------------------------- |
| `template` | string | *Required* | Jinja2 template that must render to valid JSON |
| `fail_on_error` | boolean | `false` | Fail entire pipeline on transformation error |
## Template Context
Templates have access to the full retriever execution context:
| Namespace | Description | Example |
| --------------------- | ----------------------------------------- | ---------------------------- |
| `DOC` / `doc` | Current document fields and metadata | `{{ DOC.document_id }}` |
| `INPUT` / `inputs` | Original query inputs from search request | `{{ INPUT.query }}` |
| `CONTEXT` / `context` | Execution context (namespace\_id, etc.) | `{{ CONTEXT.namespace_id }}` |
| `STAGE` / `stage` | Current stage execution data | `{{ STAGE.name }}` |
| `SECRET` / `secret` | Vault secrets (API keys, credentials) | `{{ SECRET.api_key }}` |
Both uppercase and lowercase namespace formats work identically (`DOC` == `doc`).
### Built-in Functions
These numeric helpers are available directly in templates — useful for clamping
and rounding computed values (e.g. ranking scores):
| Function | Description | Example |
| ---------------- | ----------------------------------- | --------------------------------- |
| `max` / `min` | Largest / smallest of the arguments | `{{ max(0, 1 - days_old / 14) }}` |
| `abs` | Absolute value | `{{ abs(DOC.delta) }}` |
| `round` | Round to nearest (optional digits) | `{{ round(score, 4) }}` |
| `ceil` / `floor` | Round up / down to an integer | `{{ ceil(DOC.count / 10) }}` |
Read tunable inputs with a default using `INPUT.get`, so callers can override a
weight per request but omit it otherwise: `{{ INPUT.get('recency_weight', 0.3) }}`.
## Template Features
### Jinja2 Syntax
| Feature | Syntax | Description |
| ------------ | ------------------------- | ------------------- |
| Variables | `{{ DOC.field }}` | Output field values |
| Conditionals | `{% if %}...{% endif %}` | Conditional content |
| Loops | `{% for item in items %}` | Iterate over arrays |
| Filters | `{{ value \| tojson }}` | Transform values |
| Comments | `{# comment #}` | Template comments |
### Useful Filters
| Filter | Description | Example |
| ---------------- | ----------------------- | -------------------------------------- |
| `tojson` | JSON-safe encoding | `{{ DOC.data \| tojson }}` |
| `length` | Get array/string length | `{{ DOC.tags \| length }}` |
| `default` | Fallback value | `{{ DOC.optional \| default('N/A') }}` |
| `first` / `last` | Array element | `{{ DOC.items \| first }}` |
| `join` | Join array | `{{ DOC.tags \| join(', ') }}` |
## Configuration Examples
```json Simple Field Selection theme={null}
{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\"id\": \"{{ DOC.document_id }}\", \"content\": \"{{ DOC.text }}\", \"score\": {{ DOC.score }}}"
}
}
}
```
```json With JSON Escaping theme={null}
{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\"id\": \"{{ DOC.document_id }}\", \"content\": {{ DOC.text | tojson }}, \"metadata\": {{ DOC.metadata | tojson }}}"
}
}
}
```
```json Conditional Field Inclusion theme={null}
{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\"workflow_name\": \"process-asset\", \"inputs\": [{\"name\": \"id\", \"value\": \"{{ DOC.id }}\"}{% if DOC.asset_type == \"VIDEO\" %}, {\"name\": \"video\", \"value\": {\"src\": \"{{ DOC.url }}\"}}{% endif %}]}"
}
}
}
```
```json Array Iteration theme={null}
{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\"title\": \"{{ DOC.title }}\", \"tags\": [{% for tag in DOC.tags %}\"{{ tag }}\"{% if not loop.last %}, {% endif %}{% endfor %}]}"
}
}
}
```
```json Nested Field Access theme={null}
{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\"user_id\": \"{{ DOC.metadata.user_id }}\", \"category\": \"{{ DOC.metadata.category }}\", \"raw_data\": {{ DOC.metadata.raw | tojson }}}"
}
}
}
```
```json Strict Mode (Fail on Error) theme={null}
{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\"required_field\": \"{{ DOC.must_exist }}\", \"value\": {{ DOC.number }}}",
"fail_on_error": true
}
}
}
```
```json External Workflow API Format theme={null}
{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\"workflow\": \"{{ DOC.workflow_name }}\", \"inputs\": [{\"name\": \"variant_id\", \"value\": \"{{ DOC.variant_id }}\"}{% if DOC.asset_type == \"VIDEO\" %}, {\"name\": \"video\", \"value\": {\"src\": \"{{ DOC.asset_url }}\"}}{% endif %}]}"
}
}
}
```
## Error Handling
| Setting | Behavior |
| -------------------------------- | ------------------------------------------------------- |
| `fail_on_error: false` (default) | Skip failed documents with warning, continue processing |
| `fail_on_error: true` | Fail entire retrieval on first transformation error |
**Common failure causes:**
* Invalid template syntax
* Template rendering errors (missing fields)
* Invalid JSON output from template
* Document missing required fields
Use `fail_on_error: false` for public APIs where partial results are acceptable. Use `fail_on_error: true` for internal workflows where data integrity is critical.
## Performance
| Metric | Value |
| -------------- | ------------------------------------- |
| **Latency** | \< 1ms per document |
| **Processing** | Sequential (fast, no caching needed) |
| **Schema** | Output completely defined by template |
## Multi-line Templates
For complex templates, use HEREDOC syntax in the API call:
```bash theme={null}
curl -X POST "$MP_API_URL/v1/retrievers" \
-H "Authorization: Bearer $MP_API_KEY" \
-d '{
"stages": [{
"stage_name": "json_transform",
"stage_type": "apply",
"config": {
"stage_id": "json_transform",
"parameters": {
"template": "{\n \"id\": \"{{ DOC.document_id }}\",\n \"title\": {{ DOC.title | tojson }},\n \"items\": [\n {% for item in DOC.items %}{\n \"name\": \"{{ item.name }}\",\n \"value\": {{ item.value }}\n }{% if not loop.last %},{% endif %}\n {% endfor %}\n ]\n}"
}
}
}]
}'
```
## Common Patterns
### Drop Unused Fields
```json theme={null}
{
"template": "{\"id\": \"{{ DOC.document_id }}\", \"title\": \"{{ DOC.title }}\", \"url\": \"{{ DOC.url }}\"}"
}
```
### Flatten Nested Metadata
```json theme={null}
{
"template": "{\"doc_id\": \"{{ DOC.document_id }}\", \"user_id\": \"{{ DOC.metadata.user_id }}\", \"category\": \"{{ DOC.metadata.category }}\", \"score\": {{ DOC.score }}}"
}
```
### Add Query Context
```json theme={null}
{
"template": "{\"query\": \"{{ INPUT.query }}\", \"result_id\": \"{{ DOC.document_id }}\", \"score\": {{ DOC.score }}}"
}
```
### Compute a custom ranking signal (recency boost + used-demotion)
`json_transform` is the general-purpose way to compute a **custom ranking score**
at query time: read the relevance score and any document fields, blend in your own
signals, emit the result as a `score`, then order by it with
[`sort_relevance`](/docs/retrieval/stages/sort-relevance). Because every weight is read
from `INPUT`, the same retriever is tunable per request — including an instant
off-switch — with no redeploy.
The pipeline is three stages:
```
feature_search → json_transform → sort_relevance
(relevance) (blend the score) (order by "score")
```
A worked example: boost fresh documents, and demote any document that already
carries a `used_in_ad` edge. The blended score is
```
score = relevance × (1 + recency_weight × recency_factor) × (1 − used_demotion if used)
```
where `recency_factor` is a gentle linear decay `max(0, 1 − days_old / freshness_days)`.
The template reads each weight from `INPUT` with a default, so callers override only
what they want. This is the exact template that runs in production, prettified for
readability (the verbatim single-line version is in the block below):
```jinja theme={null}
{% set md = DOC['_internal']['metadata'] %}
{% set d = (md.get('date_created') or '1970-01-01')[:10] %}
{# pseudo-ordinal day count: Y*372 + M*31 + D — monotonic, months≈31d, ±1d vs true days #}
{% set ord_doc = (d[:4]|int)*372 + (d[5:7]|int)*31 + (d[8:10]|int) %}
{% set asof = INPUT.get('as_of') or '2026-07-15' %}
{% set ord_asof = (asof[:4]|int)*372 + (asof[5:7]|int)*31 + (asof[8:10]|int) %}
{% set days_old = ord_asof - ord_doc %}
{# tunables — `x if x is not none else default` also catches a client-passed null #}
{% set fw = (INPUT.get('freshness_days') or 14) | int %}
{% set rw = (INPUT.get('recency_weight') if INPUT.get('recency_weight') is not none else 0.3) | float %}
{% set ud = (INPUT.get('used_demotion') if INPUT.get('used_demotion') is not none else 0.4) | float %}
{% set on = INPUT.get('boost_enabled') if INPUT.get('boost_enabled') is not none else true %}
{# edge-driven, not a flag: does this doc carry a used_in_ad edge? #}
{% set used = ((DOC.get('edges') or []) | selectattr('type', 'equalto', 'used_in_ad') | list | length) > 0 %}
{% set rf = max(0.0, 1.0 - (days_old / fw)) %}
{% set base = DOC.get('score') or 0 %}
{% set final = (base * (1 + rw * rf) * (1 - (ud if used else 0))) if on else base %}
{
"document_id": {{ DOC.document_id | tojson }},
"title": {{ md.get('title') | tojson }},
"brand": {{ md.get('brand') | tojson }},
"job_id": {{ md.get('job_id') | tojson }},
"date_created": {{ d | tojson }},
"days_old": {{ days_old }},
"base_score": {{ base | round(4) }},
"recency_factor": {{ rf | round(4) }},
"is_used_in_ad": {{ used | tojson }},
"new_badge": {{ (days_old <= fw) | tojson }},
"score": {{ final | round(4) }}
}
```
A few things worth noticing in the real template:
* **User metadata is read from `_internal.metadata`** here (`md`), the path managed-pipeline
documents use. If your documents carry metadata at the top level or under `metadata`
instead, adjust the path to match how they were ingested.
* **The default idiom guards `null`, not just missing.** `INPUT.get('recency_weight') if
INPUT.get('recency_weight') is not none else 0.3` falls back both when the caller omits
the input **and** when it passes an explicit `null` — `INPUT.get('x', 0.3)` alone would
keep a client-sent `null`.
* **`days_old` is a pseudo-ordinal** (`Y*372 + M*31 + D`), monotonic and accurate to about
±1 day versus true calendar days — deliberately cheap, since exact date math isn't needed
to rank by freshness.
The production template is authored on one line (newlines are optional in Jinja).
This is byte-for-byte what ran in prod:
```jinja theme={null}
{% set md = DOC['_internal']['metadata'] %}{% set d = (md.get('date_created') or '1970-01-01')[:10] %}{% set ord_doc = (d[:4]|int)*372 + (d[5:7]|int)*31 + (d[8:10]|int) %}{% set asof = INPUT.get('as_of') or '2026-07-15' %}{% set ord_asof = (asof[:4]|int)*372 + (asof[5:7]|int)*31 + (asof[8:10]|int) %}{% set days_old = ord_asof - ord_doc %}{% set fw = (INPUT.get('freshness_days') or 14) | int %}{% set rw = (INPUT.get('recency_weight') if INPUT.get('recency_weight') is not none else 0.3) | float %}{% set ud = (INPUT.get('used_demotion') if INPUT.get('used_demotion') is not none else 0.4) | float %}{% set on = INPUT.get('boost_enabled') if INPUT.get('boost_enabled') is not none else true %}{% set used = ((DOC.get('edges') or []) | selectattr('type','equalto','used_in_ad') | list | length) > 0 %}{% set rf = max(0.0, 1.0 - (days_old / fw)) %}{% set base = DOC.get('score') or 0 %}{% set final = (base * (1 + rw * rf) * (1 - (ud if used else 0))) if on else base %}{"document_id": {{ DOC.document_id | tojson }}, "title": {{ md.get('title') | tojson }}, "brand": {{ md.get('brand') | tojson }}, "job_id": {{ md.get('job_id') | tojson }}, "date_created": {{ d | tojson }}, "days_old": {{ days_old }}, "base_score": {{ base | round(4) }}, "recency_factor": {{ rf | round(4) }}, "is_used_in_ad": {{ used | tojson }}, "new_badge": {{ (days_old <= fw) | tojson }}, "score": {{ final | round(4) }}}
```
Then sort by the computed field (it is the default, shown here for clarity):
```json theme={null}
{
"stage_name": "sort_relevance",
"stage_type": "sort",
"config": {
"stage_id": "sort_relevance",
"parameters": { "score_field": "score", "direction": "desc" }
}
}
```
`json_transform` **replaces** each document with the template's JSON output, so
emit every field you want downstream — including the `score` that `sort_relevance`
reads. Pass the template to the stage as its `template` string (escape it as JSON,
or use the heredoc form shown under [Multi-line Templates](#multi-line-templates)).
Tunable inputs (declare them in the retriever's `input_schema` as optional):
| Input | Default | Effect |
| ---------------- | ------- | ------------------------------------------------------------------------------------------- |
| `recency_weight` | `0.3` | How strongly freshness lifts the score. Raise to let recent docs dominate. |
| `used_demotion` | `0.4` | Fraction to subtract when the doc has a `used_in_ad` edge. |
| `freshness_days` | `14` | Decay window, and the cutoff for the `new_badge` flag. |
| `as_of` | today | Reference date the age is measured from. |
| `boost_enabled` | `true` | Set `false` to bypass all boosting and return pure relevance order — an instant off-switch. |
```bash theme={null}
# default ranking
curl -X POST "$MP_API_URL/v1/retrievers/$RET_ID/execute" -H "Authorization: Bearer $MP_API_KEY" \
-d '{"inputs": {"query": "footage scene"}}'
# heavier recency, wider window
curl ... -d '{"inputs": {"query": "footage scene", "recency_weight": 2.0, "freshness_days": 30}}'
# off-switch — pure relevance
curl ... -d '{"inputs": {"query": "footage scene", "boost_enabled": false}}'
```
## Related
* [Sort Relevance](/docs/retrieval/stages/sort-relevance) - Order by a score, including one computed here
* [RAG Prepare](/docs/retrieval/stages/rag-prepare) - Format documents for LLM context
* [LLM Enrich](/docs/retrieval/stages/llm-enrich) - Extract structured data with LLMs
* [API Call](/docs/retrieval/stages/api-call) - Format for external API calls