JSON Transform

JSON Transform stage showing Jinja2 template transformation of documents

The JSON Transform stage applies a Jinja2 template to each document, rendering the template with full document context and replacing the document with the parsed JSON output. Use this to reformat documents for external APIs or reshape data for downstream consumers.

Stage Category: APPLY (1-1 Transformation)Transformation: N documents → N documents (or fewer with fail_on_error=False)

When to Use

Use Case	Description
External API formatting	Format documents for webhook payloads
Response optimization	Remove unused fields to reduce bandwidth
Schema adaptation	Convert internal format to client-specific format
Conditional outputs	Include fields based on document properties
Array flattening	Transform nested structures to flat arrays
Field renaming	Rename or reorganize document fields

When NOT to Use

Scenario	Recommended Alternative
Filtering documents	`attribute_filter` or `llm_filter`
Sorting documents	`sort_by_field` or `rerank`
Enriching with new data	`document_enrich` or `api_call`
Joining external data	`taxonomy_enrich`

Parameters

Parameter	Type	Default	Description
`template`	string	Required	Jinja2 template that must render to valid JSON
`fail_on_error`	boolean	`false`	Fail entire pipeline on transformation error

Template Context

Templates have access to the full retriever execution context:

Namespace	Description	Example
`DOC` / `doc`	Current document fields and metadata	`{{ DOC.document_id }}`
`INPUT` / `inputs`	Original query inputs from search request	`{{ INPUT.query }}`
`CONTEXT` / `context`	Execution context (namespace_id, etc.)	`{{ CONTEXT.namespace_id }}`
`STAGE` / `stage`	Current stage execution data	`{{ STAGE.name }}`
`SECRET` / `secret`	Vault secrets (API keys, credentials)	`{{ SECRET.api_key }}`

Both uppercase and lowercase namespace formats work identically (DOC == doc).

Built-in Functions

These numeric helpers are available directly in templates — useful for clamping and rounding computed values (e.g. ranking scores):

Function	Description	Example
`max` / `min`	Largest / smallest of the arguments	`{{ max(0, 1 - days_old / 14) }}`
`abs`	Absolute value	`{{ abs(DOC.delta) }}`
`round`	Round to nearest (optional digits)	`{{ round(score, 4) }}`
`ceil` / `floor`	Round up / down to an integer	`{{ ceil(DOC.count / 10) }}`

Read tunable inputs with a default using INPUT.get, so callers can override a weight per request but omit it otherwise: {{ INPUT.get('recency_weight', 0.3) }}.

Template Features

Jinja2 Syntax

Feature	Syntax	Description
Variables	`{{ DOC.field }}`	Output field values
Conditionals	`{% if %}...{% endif %}`	Conditional content
Loops	`{% for item in items %}`	Iterate over arrays
Filters	`{{ value \| tojson }}`	Transform values
Comments	`{# comment #}`	Template comments

Useful Filters

Filter	Description	Example
`tojson`	JSON-safe encoding	`{{ DOC.data \| tojson }}`
`length`	Get array/string length	`{{ DOC.tags \| length }}`
`default`	Fallback value	`{{ DOC.optional \| default('N/A') }}`
`first` / `last`	Array element	`{{ DOC.items \| first }}`
`join`	Join array	`{{ DOC.tags \| join(', ') }}`

Configuration Examples

{
  "stage_name": "json_transform",
  "stage_type": "apply",
  "config": {
    "stage_id": "json_transform",
    "parameters": {
      "template": "{\"id\": \"{{ DOC.document_id }}\", \"content\": \"{{ DOC.text }}\", \"score\": {{ DOC.score }}}"
    }
  }
}

{
  "stage_name": "json_transform",
  "stage_type": "apply",
  "config": {
    "stage_id": "json_transform",
    "parameters": {
      "template": "{\"id\": \"{{ DOC.document_id }}\", \"content\": {{ DOC.text | tojson }}, \"metadata\": {{ DOC.metadata | tojson }}}"
    }
  }
}

{
  "stage_name": "json_transform",
  "stage_type": "apply",
  "config": {
    "stage_id": "json_transform",
    "parameters": {
      "template": "{\"workflow_name\": \"process-asset\", \"inputs\": [{\"name\": \"id\", \"value\": \"{{ DOC.id }}\"}{% if DOC.asset_type == \"VIDEO\" %}, {\"name\": \"video\", \"value\": {\"src\": \"{{ DOC.url }}\"}}{% endif %}]}"
    }
  }
}

{
  "stage_name": "json_transform",
  "stage_type": "apply",
  "config": {
    "stage_id": "json_transform",
    "parameters": {
      "template": "{\"title\": \"{{ DOC.title }}\", \"tags\": [{% for tag in DOC.tags %}\"{{ tag }}\"{% if not loop.last %}, {% endif %}{% endfor %}]}"
    }
  }
}

{
  "stage_name": "json_transform",
  "stage_type": "apply",
  "config": {
    "stage_id": "json_transform",
    "parameters": {
      "template": "{\"user_id\": \"{{ DOC.metadata.user_id }}\", \"category\": \"{{ DOC.metadata.category }}\", \"raw_data\": {{ DOC.metadata.raw | tojson }}}"
    }
  }
}

{
  "stage_name": "json_transform",
  "stage_type": "apply",
  "config": {
    "stage_id": "json_transform",
    "parameters": {
      "template": "{\"required_field\": \"{{ DOC.must_exist }}\", \"value\": {{ DOC.number }}}",
      "fail_on_error": true
    }
  }
}

{
  "stage_name": "json_transform",
  "stage_type": "apply",
  "config": {
    "stage_id": "json_transform",
    "parameters": {
      "template": "{\"workflow\": \"{{ DOC.workflow_name }}\", \"inputs\": [{\"name\": \"variant_id\", \"value\": \"{{ DOC.variant_id }}\"}{% if DOC.asset_type == \"VIDEO\" %}, {\"name\": \"video\", \"value\": {\"src\": \"{{ DOC.asset_url }}\"}}{% endif %}]}"
    }
  }
}

Error Handling

Setting	Behavior
`fail_on_error: false` (default)	Skip failed documents with warning, continue processing
`fail_on_error: true`	Fail entire retrieval on first transformation error

Common failure causes:

Invalid template syntax
Template rendering errors (missing fields)
Invalid JSON output from template
Document missing required fields

Use fail_on_error: false for public APIs where partial results are acceptable. Use fail_on_error: true for internal workflows where data integrity is critical.

Performance

Metric	Value
Latency	< 1ms per document
Processing	Sequential (fast, no caching needed)
Schema	Output completely defined by template

Multi-line Templates

For complex templates, use HEREDOC syntax in the API call:

curl -X POST "$MP_API_URL/v1/retrievers" \
  -H "Authorization: Bearer $MP_API_KEY" \
  -d '{
    "stages": [{
      "stage_name": "json_transform",
      "stage_type": "apply",
      "config": {
        "stage_id": "json_transform",
        "parameters": {
          "template": "{\n  \"id\": \"{{ DOC.document_id }}\",\n  \"title\": {{ DOC.title | tojson }},\n  \"items\": [\n    {% for item in DOC.items %}{\n      \"name\": \"{{ item.name }}\",\n      \"value\": {{ item.value }}\n    }{% if not loop.last %},{% endif %}\n    {% endfor %}\n  ]\n}"
        }
      }
    }]
  }'

Common Patterns

Drop Unused Fields

{
  "template": "{\"id\": \"{{ DOC.document_id }}\", \"title\": \"{{ DOC.title }}\", \"url\": \"{{ DOC.url }}\"}"
}

Flatten Nested Metadata

{
  "template": "{\"doc_id\": \"{{ DOC.document_id }}\", \"user_id\": \"{{ DOC.metadata.user_id }}\", \"category\": \"{{ DOC.metadata.category }}\", \"score\": {{ DOC.score }}}"
}

Add Query Context

{
  "template": "{\"query\": \"{{ INPUT.query }}\", \"result_id\": \"{{ DOC.document_id }}\", \"score\": {{ DOC.score }}}"
}

Compute a custom ranking signal (recency boost + used-demotion)

json_transform is the general-purpose way to compute a custom ranking score at query time: read the relevance score and any document fields, blend in your own signals, emit the result as a score, then order by it with sort_relevance. Because every weight is read from INPUT, the same retriever is tunable per request — including an instant off-switch — with no redeploy. The pipeline is three stages:

feature_search  →  json_transform  →  sort_relevance
  (relevance)      (blend the score)   (order by "score")

A worked example: boost fresh documents, and demote any document that already carries a used_in_ad edge. The blended score is

score = relevance × (1 + recency_weight × recency_factor) × (1 − used_demotion if used)

where recency_factor is a gentle linear decay max(0, 1 − days_old / freshness_days). The template reads each weight from INPUT with a default, so callers override only what they want. This is the exact template that runs in production, prettified for readability (the verbatim single-line version is in the block below):

{% set md = DOC['_internal']['metadata'] %}
{% set d  = (md.get('date_created') or '1970-01-01')[:10] %}

{# pseudo-ordinal day count: Y*372 + M*31 + D — monotonic, months≈31d, ±1d vs true days #}
{% set ord_doc  = (d[:4]|int)*372 + (d[5:7]|int)*31 + (d[8:10]|int) %}
{% set asof     = INPUT.get('as_of') or '2026-07-15' %}
{% set ord_asof = (asof[:4]|int)*372 + (asof[5:7]|int)*31 + (asof[8:10]|int) %}
{% set days_old = ord_asof - ord_doc %}

{# tunables — `x if x is not none else default` also catches a client-passed null #}
{% set fw = (INPUT.get('freshness_days') or 14) | int %}
{% set rw = (INPUT.get('recency_weight') if INPUT.get('recency_weight') is not none else 0.3) | float %}
{% set ud = (INPUT.get('used_demotion')  if INPUT.get('used_demotion')  is not none else 0.4) | float %}
{% set on = INPUT.get('boost_enabled')   if INPUT.get('boost_enabled')  is not none else true %}

{# edge-driven, not a flag: does this doc carry a used_in_ad edge? #}
{% set used = ((DOC.get('edges') or []) | selectattr('type', 'equalto', 'used_in_ad') | list | length) > 0 %}

{% set rf    = max(0.0, 1.0 - (days_old / fw)) %}
{% set base  = DOC.get('score') or 0 %}
{% set final = (base * (1 + rw * rf) * (1 - (ud if used else 0))) if on else base %}
{
  "document_id": {{ DOC.document_id | tojson }},
  "title": {{ md.get('title') | tojson }},
  "brand": {{ md.get('brand') | tojson }},
  "job_id": {{ md.get('job_id') | tojson }},
  "date_created": {{ d | tojson }},
  "days_old": {{ days_old }},
  "base_score": {{ base | round(4) }},
  "recency_factor": {{ rf | round(4) }},
  "is_used_in_ad": {{ used | tojson }},
  "new_badge": {{ (days_old <= fw) | tojson }},
  "score": {{ final | round(4) }}
}

A few things worth noticing in the real template:

User metadata is read from _internal.metadata here (md), the path managed-pipeline documents use. If your documents carry metadata at the top level or under metadata instead, adjust the path to match how they were ingested.
The default idiom guards null, not just missing. INPUT.get('recency_weight') if INPUT.get('recency_weight') is not none else 0.3 falls back both when the caller omits the input and when it passes an explicit null — INPUT.get('x', 0.3) alone would keep a client-sent null.
days_old is a pseudo-ordinal (Y*372 + M*31 + D), monotonic and accurate to about ±1 day versus true calendar days — deliberately cheap, since exact date math isn’t needed to rank by freshness.

Verbatim single-line template (paste as the template string)

The production template is authored on one line (newlines are optional in Jinja). This is byte-for-byte what ran in prod:

{% set md = DOC['_internal']['metadata'] %}{% set d = (md.get('date_created') or '1970-01-01')[:10] %}{% set ord_doc = (d[:4]|int)*372 + (d[5:7]|int)*31 + (d[8:10]|int) %}{% set asof = INPUT.get('as_of') or '2026-07-15' %}{% set ord_asof = (asof[:4]|int)*372 + (asof[5:7]|int)*31 + (asof[8:10]|int) %}{% set days_old = ord_asof - ord_doc %}{% set fw = (INPUT.get('freshness_days') or 14) | int %}{% set rw = (INPUT.get('recency_weight') if INPUT.get('recency_weight') is not none else 0.3) | float %}{% set ud = (INPUT.get('used_demotion') if INPUT.get('used_demotion') is not none else 0.4) | float %}{% set on = INPUT.get('boost_enabled') if INPUT.get('boost_enabled') is not none else true %}{% set used = ((DOC.get('edges') or []) | selectattr('type','equalto','used_in_ad') | list | length) > 0 %}{% set rf = max(0.0, 1.0 - (days_old / fw)) %}{% set base = DOC.get('score') or 0 %}{% set final = (base * (1 + rw * rf) * (1 - (ud if used else 0))) if on else base %}{"document_id": {{ DOC.document_id | tojson }}, "title": {{ md.get('title') | tojson }}, "brand": {{ md.get('brand') | tojson }}, "job_id": {{ md.get('job_id') | tojson }}, "date_created": {{ d | tojson }}, "days_old": {{ days_old }}, "base_score": {{ base | round(4) }}, "recency_factor": {{ rf | round(4) }}, "is_used_in_ad": {{ used | tojson }}, "new_badge": {{ (days_old <= fw) | tojson }}, "score": {{ final | round(4) }}}

Then sort by the computed field (it is the default, shown here for clarity):

{
  "stage_name": "sort_relevance",
  "stage_type": "sort",
  "config": {
    "stage_id": "sort_relevance",
    "parameters": { "score_field": "score", "direction": "desc" }
  }
}

json_transform replaces each document with the template’s JSON output, so emit every field you want downstream — including the score that sort_relevance reads. Pass the template to the stage as its template string (escape it as JSON, or use the heredoc form shown under Multi-line Templates).

Tunable inputs (declare them in the retriever’s input_schema as optional):

Input	Default	Effect
`recency_weight`	`0.3`	How strongly freshness lifts the score. Raise to let recent docs dominate.
`used_demotion`	`0.4`	Fraction to subtract when the doc has a `used_in_ad` edge.
`freshness_days`	`14`	Decay window, and the cutoff for the `new_badge` flag.
`as_of`	today	Reference date the age is measured from.
`boost_enabled`	`true`	Set `false` to bypass all boosting and return pure relevance order — an instant off-switch.

# default ranking
curl -X POST "$MP_API_URL/v1/retrievers/$RET_ID/execute" -H "Authorization: Bearer $MP_API_KEY" \
  -d '{"inputs": {"query": "footage scene"}}'
# heavier recency, wider window
curl ... -d '{"inputs": {"query": "footage scene", "recency_weight": 2.0, "freshness_days": 30}}'
# off-switch — pure relevance
curl ... -d '{"inputs": {"query": "footage scene", "boost_enabled": false}}'

Sort Relevance - Order by a score, including one computed here
RAG Prepare - Format documents for LLM context
LLM Enrich - Extract structured data with LLMs
API Call - Format for external API calls

Get started

Connect your data

Extract features

Build retrievers

Enrich & organize

Integrate & operate

Resources

When to Use

When NOT to Use

Parameters

Template Context

Built-in Functions

Template Features

Jinja2 Syntax

Useful Filters

Configuration Examples

Error Handling

Performance

Multi-line Templates

Common Patterns

Drop Unused Fields

Flatten Nested Metadata

Add Query Context

Compute a custom ranking signal (recency boost + used-demotion)

​When to Use

​When NOT to Use

​Parameters

​Template Context

​Built-in Functions

​Template Features

​Jinja2 Syntax

​Useful Filters

​Configuration Examples

​Error Handling

​Performance

​Multi-line Templates

​Common Patterns

​Drop Unused Fields

​Flatten Nested Metadata

​Add Query Context

​Compute a custom ranking signal (recency boost + used-demotion)

​Related

When to Use

When NOT to Use

Parameters

Template Context

Built-in Functions

Template Features

Jinja2 Syntax

Useful Filters

Configuration Examples

Error Handling

Performance

Multi-line Templates

Common Patterns

Drop Unused Fields

Flatten Nested Metadata

Add Query Context

Compute a custom ranking signal (recency boost + used-demotion)

Related