Moment Group

Moment Group stage merging frame-level intervals into consolidated video moments

The Moment Group stage takes frame-level or chunk-level documents with temporal metadata and merges contiguous intervals into consolidated video moments (time ranges), grouped by parent object. Instead of returning individual frames that matched a query, you get precise start/end time ranges pinpointing where in a video the match occurs.

Stage Category: REDUCE (Merges intervals into moments)Transformation: N frame/chunk documents → M consolidated moments per parent video

When to Use

Use Case	Description
Sub-scene localization	Pinpoint exactly where in a video a query matches
Moment extraction	Return time-range clips instead of individual frames
Video search results	Provide start/end timestamps for player seek
Highlight reels	Extract the top-scoring moments across a video library

When NOT to Use

Scenario	Recommended Alternative
Grouping by non-temporal fields	`group_by`
Time-window aggregations (hour/day/week)	`temporal`
Results without temporal metadata	`aggregate`
Semantic clustering of results	`cluster`

Requirements

Input documents must have temporal intervals — either query_chunks with start_ms/end_ms (attached by feature_search preprocessing) or document-level time fields.
The parent_field must exist on the documents so results can be grouped by source video/object.

Text query vs file query — pick the right time_field. The default time_field: "query_chunks" is only populated when you search with a file input (via query preprocessing). For a plain text query, the matched video segments carry top-level start_time/end_time — set time_field: "start_time" (the stage derives the end field by swapping start→end). Leaving query_chunks with a text query yields empty moments — the most common silent failure here. See the Video Moment Localization recipe.

Parameters

Parameter	Type	Default	Description
`parent_field`	string	`"source_object_id"`	Field to group results by parent video/object. Common values: `source_object_id` (standard ingest lineage), `root_object_id` (top-level ancestor), or any custom field.
`time_field`	string	`"query_chunks"`	Source of temporal intervals. `"query_chunks"` reads the array `feature_search` attaches (each entry has `start_ms`, `end_ms`, `score`). For document-level timestamps, use a dot-path like `"metadata.start_ms"` — the stage derives the end field by replacing `"start"` with `"end"`.
`merge_tolerance_ms`	integer	`2000`	Maximum gap in milliseconds between intervals before they are split into separate moments. `0` = exact-overlap only. Range: 0–60000.
`max_moments_per_parent`	integer	`10`	Maximum moments returned per parent, sorted by score (highest first). Range: 1–100.
`score_strategy`	string	`"max"`	How to aggregate scores across merged intervals: `"max"`, `"avg"`, or `"sum"`.
`min_score`	float	`null`	Drop moments scoring below this threshold. Range: 0.0–1.0.
`output_mode`	string	`"annotated"`	`"annotated"` keeps the best-scoring document per parent and attaches a `moments` array (preserves all original document fields). `"moments_only"` emits standalone moment documents with IDs like `moment_{parent}_{start_ms}`.

Configuration Examples

{
  "stage_name": "moment_group",
  "stage_type": "reduce",
  "config": {
    "stage_id": "moment_group",
    "parameters": {
      "parent_field": "source_object_id",
      "time_field": "query_chunks",
      "merge_tolerance_ms": 2000,
      "max_moments_per_parent": 5,
      "score_strategy": "max",
      "output_mode": "annotated"
    }
  }
}

{
  "stage_name": "moment_group",
  "stage_type": "reduce",
  "config": {
    "stage_id": "moment_group",
    "parameters": {
      "parent_field": "source_object_id",
      "time_field": "query_chunks",
      "merge_tolerance_ms": 1500,
      "max_moments_per_parent": 3,
      "score_strategy": "max",
      "min_score": 0.6,
      "output_mode": "annotated"
    }
  }
}

{
  "stage_name": "moment_group",
  "stage_type": "reduce",
  "config": {
    "stage_id": "moment_group",
    "parameters": {
      "parent_field": "root_object_id",
      "time_field": "query_chunks",
      "merge_tolerance_ms": 3000,
      "max_moments_per_parent": 10,
      "score_strategy": "avg",
      "output_mode": "moments_only"
    }
  }
}

{
  "stage_name": "moment_group",
  "stage_type": "reduce",
  "config": {
    "stage_id": "moment_group",
    "parameters": {
      "parent_field": "source_object_id",
      "time_field": "metadata.start_ms",
      "merge_tolerance_ms": 2000,
      "max_moments_per_parent": 5,
      "score_strategy": "max",
      "output_mode": "annotated"
    }
  }
}

Output Schema

Annotated Mode (default)

In annotated mode, the best-scoring document per parent is preserved with a moments array attached:

{
  "document_id": "doc_abc123",
  "source_object_id": "video_001",
  "score": 0.92,
  "content": "...",
  "metadata": { "..." : "..." },
  "moments": [
    {
      "start_ms": 12000,
      "end_ms": 18500,
      "score": 0.92,
      "match_count": 4,
      "document_ids": ["doc_abc123", "doc_abc124", "doc_abc125", "doc_abc126"]
    },
    {
      "start_ms": 45000,
      "end_ms": 51000,
      "score": 0.78,
      "match_count": 2,
      "document_ids": ["doc_abc130", "doc_abc131"]
    }
  ]
}

Moments-Only Mode

In moments_only mode, standalone moment documents are emitted:

{
  "document_id": "moment_video_001_12000",
  "start_ms": 12000,
  "end_ms": 18500,
  "score": 0.92,
  "match_count": 4,
  "document_ids": ["doc_abc123", "doc_abc124", "doc_abc125", "doc_abc126"]
}

Moment Object Fields

Field	Type	Description
`start_ms`	integer	Start of the moment in milliseconds
`end_ms`	integer	End of the moment in milliseconds
`score`	float	Aggregated score (per `score_strategy`)
`match_count`	integer	Number of source intervals merged into this moment
`document_ids`	array	IDs of the original documents that contributed to this moment

Performance

Metric	Value
Latency	5-50ms
Memory	O(N) where N = input documents
Cost	Free
Scalability	Efficient — runs in the API layer, no engine call

Common Pipeline Patterns

Video Search with Moment Localization

The canonical use case: search across video frames, then consolidate matches into seekable moments.

[
  {
    "stage_name": "feature_search",
    "stage_type": "filter",
    "config": {
      "stage_id": "feature_search",
      "parameters": {
        "searches": [
          {
            "feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
            "query": "{{INPUT.query}}",
            "top_k": 200
          }
        ],
        "final_top_k": 200
      }
    }
  },
  {
    "stage_name": "moment_group",
    "stage_type": "reduce",
    "config": {
      "stage_id": "moment_group",
      "parameters": {
        "parent_field": "source_object_id",
        "time_field": "start_time",
        "merge_tolerance_ms": 1500,
        "max_moments_per_parent": 3,
        "score_strategy": "max",
        "min_score": 0.6,
        "output_mode": "annotated"
      }
    }
  }
]

Moment Extraction with Reranking

Search, rerank for precision, then group into moments:

[
  {
    "stage_name": "feature_search",
    "stage_type": "filter",
    "config": {
      "stage_id": "feature_search",
      "parameters": {
        "searches": [
          {
            "feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
            "query": "{{INPUT.query}}",
            "top_k": 500
          }
        ],
        "final_top_k": 500
      }
    }
  },
  {
    "stage_name": "rerank",
    "stage_type": "sort",
    "config": {
      "stage_id": "rerank",
      "parameters": {
        "inference_name": "BAAI__bge_reranker_v2_m3",
        "query": "{{INPUT.query}}",
        "top_k": 100
      }
    }
  },
  {
    "stage_name": "moment_group",
    "stage_type": "reduce",
    "config": {
      "stage_id": "moment_group",
      "parameters": {
        "parent_field": "source_object_id",
        "time_field": "start_time",
        "merge_tolerance_ms": 2000,
        "max_moments_per_parent": 5,
        "score_strategy": "max",
        "output_mode": "annotated"
      }
    }
  }
]

Error Handling

Error	Behavior
Missing `parent_field` on document	Document skipped
Missing temporal metadata	Document skipped
No intervals pass `min_score`	Parent omitted from output
Empty input	Empty results returned

Temporal - Time-window aggregations with drift detection
Group By - Group documents by any field value
Aggregate - Statistical aggregations
Deduplicate - Remove duplicate documents

Get started

Connect your data

Extract features

Build retrievers

Enrich & organize

Integrate & operate

Resources

When to Use

When NOT to Use

Requirements

Parameters

Configuration Examples

Output Schema

Annotated Mode (default)

Moments-Only Mode

Moment Object Fields

Performance

Common Pipeline Patterns

Video Search with Moment Localization

Moment Extraction with Reranking

Error Handling

​When to Use

​When NOT to Use

​Requirements

​Parameters

​Configuration Examples

​Output Schema

​Annotated Mode (default)

​Moments-Only Mode

​Moment Object Fields

​Performance

​Common Pipeline Patterns

​Video Search with Moment Localization

​Moment Extraction with Reranking

​Error Handling

​Related

When to Use

When NOT to Use

Requirements

Parameters

Configuration Examples

Output Schema

Annotated Mode (default)

Moments-Only Mode

Moment Object Fields

Performance

Common Pipeline Patterns

Video Search with Moment Localization

Moment Extraction with Reranking

Error Handling

Related