Aggregate

Aggregate stage showing statistical computations on search results

The Aggregate stage computes statistical aggregations across your search results, including counts, sums, averages, min/max values, and custom metrics. This is useful for analytics, faceted search, and understanding result distributions.

Stage Category: REDUCE (Aggregates results)Transformation: N documents → aggregation results + optional documents

Aggregating a whole collection? The collection-level endpoint POST /v1/collections/{collection_id}/aggregate runs the same functions over every matching document (not just the current pipeline’s results) and returns exact counts at any collection size — count, count_distinct, sum, avg, min, and max stream without a row cap. Pair it with the is_null filter operator for field-presence / data-validity checks, e.g. count how many documents are missing an ancestor (from_collection) field across 180k+ assets.

When to Use

Use Case	Description
Faceted search	Count documents by category
Analytics	Compute metrics across results
Price ranges	Min/max/avg calculations
Distributions	Understand result characteristics

When NOT to Use

Scenario	Recommended Alternative
Grouping with full docs	`group_by`
Simple counting	Use search facets
Per-document stats	`code_execution`
LLM-based analysis	`summarize`

Parameters

Parameter	Type	Default	Description
`aggregations`	array	Required	List of aggregation operations
`group_by`	string	none	Field to group aggregations by
`include_documents`	boolean	`false`	Include original documents in output

Aggregation Types

Type	Description	Example
`count`	Number of documents	Total results
`sum`	Sum of field values	Total revenue
`avg`	Average value	Average price
`min`	Minimum value	Lowest price
`max`	Maximum value	Highest price
`cardinality`	Unique values count	Unique authors
`percentile`	Percentile values	P50, P95
`histogram`	Value distribution	Price buckets
`stddev`	Standard deviation (sample)	Score spread
`variance`	Variance (sample)	Price volatility
`frequency`	Value frequency distribution	Top categories
`co_occurrence`	Co-occurrence of two fields	Brand + category pairs
`correlation`	Pearson correlation of two fields	Price vs. rating

Configuration Examples

{
  "stage_name": "aggregate",
  "stage_type": "reduce",
  "config": {
    "stage_id": "aggregate",
    "parameters": {
      "aggregations": [
        {"type": "count", "name": "total"},
        {"type": "avg", "field": "metadata.price", "name": "avg_price"},
        {"type": "min", "field": "metadata.price", "name": "min_price"},
        {"type": "max", "field": "metadata.price", "name": "max_price"}
      ]
    }
  }
}

{
  "stage_name": "aggregate",
  "stage_type": "reduce",
  "config": {
    "stage_id": "aggregate",
    "parameters": {
      "aggregations": [
        {"type": "count", "name": "count"},
        {"type": "avg", "field": "metadata.rating", "name": "avg_rating"}
      ],
      "group_by": "metadata.category"
    }
  }
}

{
  "stage_name": "aggregate",
  "stage_type": "reduce",
  "config": {
    "stage_id": "aggregate",
    "parameters": {
      "aggregations": [
        {"type": "count", "name": "total"},
        {"type": "cardinality", "field": "metadata.brand", "name": "brand_count"},
        {"type": "histogram", "field": "metadata.price", "interval": 50, "name": "price_ranges"}
      ],
      "include_documents": true
    }
  }
}

{
  "stage_name": "aggregate",
  "stage_type": "reduce",
  "config": {
    "stage_id": "aggregate",
    "parameters": {
      "aggregations": [
        {"type": "percentile", "field": "score", "percentiles": [25, 50, 75, 95], "name": "score_distribution"},
        {"type": "avg", "field": "score", "name": "avg_score"}
      ]
    }
  }
}

{
  "stage_name": "aggregate",
  "stage_type": "reduce",
  "config": {
    "stage_id": "aggregate",
    "parameters": {
      "aggregations": [
        {"type": "count", "name": "total_docs"},
        {"type": "sum", "field": "metadata.views", "name": "total_views"},
        {"type": "avg", "field": "metadata.engagement_rate", "name": "avg_engagement"},
        {"type": "cardinality", "field": "metadata.author_id", "name": "unique_authors"}
      ]
    }
  }
}

Output Schema

Without Group By

{
  "aggregations": {
    "total": 150,
    "avg_price": 49.99,
    "min_price": 9.99,
    "max_price": 199.99
  },
  "documents": []  // if include_documents: false
}

With Group By

{
  "aggregations": {
    "electronics": {
      "count": 45,
      "avg_rating": 4.2
    },
    "clothing": {
      "count": 62,
      "avg_rating": 4.5
    },
    "books": {
      "count": 43,
      "avg_rating": 4.7
    }
  }
}

Histogram Output

{
  "aggregations": {
    "price_ranges": {
      "buckets": [
        {"key": "0-50", "count": 45},
        {"key": "50-100", "count": 62},
        {"key": "100-150", "count": 28},
        {"key": "150-200", "count": 15}
      ]
    }
  }
}

Performance

Metric	Value
Latency	5-50ms
Memory	O(groups × aggregations)
Cost	Free
Scalability	Efficient for large result sets

Common Pipeline Patterns

[
  {
    "stage_name": "semantic_search",
    "stage_type": "filter",
    "config": {
      "stage_id": "feature_search",
      "parameters": {
        "searches": [
          { "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 1000 }
        ],
        "final_top_k": 1000
      }
    }
  },
  {
    "stage_name": "aggregate",
    "stage_type": "reduce",
    "config": {
      "stage_id": "aggregate",
      "parameters": {
        "aggregations": [
          {"type": "count", "name": "total"},
          {"type": "histogram", "field": "metadata.category", "name": "categories"},
          {"type": "histogram", "field": "metadata.price", "interval": 25, "name": "price_ranges"}
        ],
        "include_documents": true
      }
    }
  }
]

Analytics Pipeline

[
  {
    "stage_name": "structured_filter",
    "stage_type": "filter",
    "config": {
      "stage_id": "attribute_filter",
      "parameters": {
        "conditions": {
          "AND": [
            {"field": "metadata.date", "operator": "gte", "value": "2024-01-01"},
            {"field": "metadata.status", "operator": "eq", "value": "published"}
          ]
        }
      }
    }
  },
  {
    "stage_name": "aggregate",
    "stage_type": "reduce",
    "config": {
      "stage_id": "aggregate",
      "parameters": {
        "aggregations": [
          {"type": "count", "name": "total_published"},
          {"type": "sum", "field": "metadata.views", "name": "total_views"},
          {"type": "avg", "field": "metadata.engagement", "name": "avg_engagement"},
          {"type": "percentile", "field": "metadata.views", "percentiles": [50, 90, 99], "name": "view_distribution"}
        ],
        "group_by": "metadata.author"
      }
    }
  }
]

E-Commerce Product Analytics

[
  {
    "stage_name": "hybrid_search",
    "stage_type": "filter",
    "config": {
      "stage_id": "feature_search",
      "parameters": {
        "searches": [
          { "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 500 }
        ],
        "final_top_k": 500
      }
    }
  },
  {
    "stage_name": "structured_filter",
    "stage_type": "filter",
    "config": {
      "stage_id": "attribute_filter",
      "parameters": {
        "conditions": {
          "field": "metadata.in_stock",
          "operator": "eq",
          "value": true
        }
      }
    }
  },
  {
    "stage_name": "aggregate",
    "stage_type": "reduce",
    "config": {
      "stage_id": "aggregate",
      "parameters": {
        "aggregations": [
          {"type": "count", "name": "available_products"},
          {"type": "min", "field": "metadata.price", "name": "lowest_price"},
          {"type": "max", "field": "metadata.price", "name": "highest_price"},
          {"type": "avg", "field": "metadata.rating", "name": "avg_rating"},
          {"type": "cardinality", "field": "metadata.brand", "name": "brand_count"}
        ],
        "group_by": "metadata.category",
        "include_documents": true
      }
    }
  }
]

Aggregation Details

Count

{"type": "count", "name": "total"}

Counts documents. No field required.

Sum / Avg / Min / Max

{"type": "avg", "field": "metadata.price", "name": "average_price"}

Requires numeric field.

Cardinality

{"type": "cardinality", "field": "metadata.author", "name": "unique_authors"}

Counts unique values (approximate for large sets).

Percentile

{"type": "percentile", "field": "score", "percentiles": [25, 50, 75, 95], "name": "score_percentiles"}

Returns specified percentile values.

Histogram

{"type": "histogram", "field": "metadata.price", "interval": 50, "name": "price_buckets"}

Groups values into buckets.

Statistical Aggregation Examples

{
  "stage_name": "aggregate",
  "stage_type": "reduce",
  "config": {
    "stage_id": "aggregate",
    "parameters": {
      "aggregations": [
        {"function": "percentile", "field": "score", "alias": "median_score", "percentile_value": 50},
        {"function": "percentile", "field": "score", "alias": "p90_score", "percentile_value": 90},
        {"function": "stddev", "field": "score", "alias": "score_spread"},
        {"function": "variance", "field": "score", "alias": "score_variance"}
      ]
    }
  }
}

{
  "stage_name": "aggregate",
  "stage_type": "reduce",
  "config": {
    "stage_id": "aggregate",
    "parameters": {
      "aggregations": [
        {"function": "frequency", "field": "category", "alias": "top_categories", "top_k": 10},
        {"function": "frequency", "field": "brand", "alias": "top_brands", "top_k": 5}
      ]
    }
  }
}

{
  "stage_name": "aggregate",
  "stage_type": "reduce",
  "config": {
    "stage_id": "aggregate",
    "parameters": {
      "aggregations": [
        {"function": "co_occurrence", "field": "brand", "field_b": "category", "alias": "brand_category_pairs"},
        {"function": "correlation", "field": "price", "field_b": "rating", "alias": "price_rating_corr"}
      ]
    }
  }
}

Statistical Output Examples

Frequency returns value counts with percentages:

{
  "top_categories": [
    {"value": "electronics", "count": 45, "percent": 30.0},
    {"value": "clothing", "count": 38, "percent": 25.3},
    {"value": "books", "count": 22, "percent": 14.7}
  ]
}

Co-occurrence returns field pair counts:

{
  "brand_category_pairs": [
    {"field_a": "Nike", "field_b": "footwear", "count": 12, "percent": 8.0},
    {"field_a": "Nike", "field_b": "apparel", "count": 8, "percent": 5.3}
  ]
}

Correlation returns the Pearson coefficient (-1 to 1):

{
  "price_rating_corr": 0.342156
}

Error Handling

Error	Behavior
Missing field	Skip document for that aggregation
Non-numeric field	Error for numeric aggregations
Empty results	Return zero/empty aggregations
Invalid type	Stage fails
Insufficient data for stddev/correlation	Returns null (need 2+ values)

Group By - Group with full documents
Sample - Statistical sampling
Summarize - LLM-powered analysis

Get started

Connect your data

Extract features

Build retrievers

Enrich & organize

Integrate & operate

Resources

When to Use

When NOT to Use

Parameters

Aggregation Types

Configuration Examples

Output Schema

Without Group By

With Group By

Histogram Output

Performance

Common Pipeline Patterns

Search with Facets

Analytics Pipeline

E-Commerce Product Analytics

Aggregation Details

Count

Sum / Avg / Min / Max

Cardinality

Percentile

Histogram

Statistical Aggregation Examples

Statistical Output Examples

Error Handling

​When to Use

​When NOT to Use

​Parameters

​Aggregation Types

​Configuration Examples

​Output Schema

​Without Group By

​With Group By

​Histogram Output

​Performance

​Common Pipeline Patterns

​Search with Facets

​Analytics Pipeline

​E-Commerce Product Analytics

​Aggregation Details

​Count

​Sum / Avg / Min / Max

​Cardinality

​Percentile

​Histogram

​Statistical Aggregation Examples

​Statistical Output Examples

​Error Handling

​Related

When to Use

When NOT to Use

Parameters

Aggregation Types

Configuration Examples

Output Schema

Without Group By

With Group By

Histogram Output

Performance

Common Pipeline Patterns

Search with Facets

Analytics Pipeline

E-Commerce Product Analytics

Aggregation Details

Count

Sum / Avg / Min / Max

Cardinality

Percentile

Histogram

Statistical Aggregation Examples

Statistical Output Examples

Error Handling

Related