HTTP API

Each item in the API response becomes a bucket object. The item’s JSON body is stored as the object blob.

Overview

Mixpeek polls a JSON endpoint on a schedule. On each run it:

Calls the URL in source_path with the configured method and headers.
Reads the array of items at items_path.
Creates one bucket object per item, keyed on item_id_field.
Skips items whose ID already exists, unless you set skip_duplicates to false.

Prerequisites

An endpoint that returns JSON or newline-delimited JSON.
An array of items, at the root or at a nested path.
A field on each item that is unique and stable.

Configuration

Connection-level fields

Field	Required	Description
`item_id_field`	Yes	Field on each item used as the deduplication ID
`credentials.headers`	No	HTTP headers, for example `Authorization` or `X-API-Key`
`http_method`	No	Defaults to `GET`
`request_body`	No	JSON body for `POST` requests
`items_path`	No	Dot-notation path to the array, for example `data.results`
`item_modified_field`	No	Timestamp field for incremental sync (ISO 8601 or Unix epoch)
`response_content_type`	No	`json` (default) or `jsonl`
`user_agent`	No	Defaults to `Mixpeek HTTP API Sync/1.0`
`request_timeout`	No	Seconds, defaults to `30`

Sync-level fields

Field	Required	Description
`source_path`	Yes	The full API URL to call
`polling_interval_seconds`	No	Defaults to `300`
`batch_size`	No	Defaults to `50`

Leave items_path unset when the response is a bare array. Set it to hits for {"hits": [...]}.

Setup

Create the storage connection

from mixpeek import Mixpeek

client = Mixpeek(api_key="your-mixpeek-api-key")

connection = client.organizations.connections.create(
    name="Internal Orders API",
    provider_type="http_api",
    provider_config={
        "credentials": {
            "type": "http_header",
            "headers": {"Authorization": "Bearer your-api-token"},
        },
        "items_path": "data.orders",
        "item_id_field": "order_id",
        "item_modified_field": "updated_at",
    },
)
print(f"Created connection: {connection['connection_id']}")

import { Mixpeek } from 'mixpeek-sdk'

const client = new Mixpeek({ apiKey: 'your-mixpeek-api-key' })

const connection = await client.organizations.connections.create({
  name: 'Internal Orders API',
  provider_type: 'http_api',
  provider_config: {
    credentials: {
      type: 'http_header',
      headers: { Authorization: 'Bearer your-api-token' },
    },
    items_path: 'data.orders',
    item_id_field: 'order_id',
    item_modified_field: 'updated_at',
  },
})
console.log('Created connection:', connection.connection_id)

curl -X POST https://api.mixpeek.com/v1/organizations/connections \
  -H "Authorization: Bearer YOUR_MIXPEEK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Internal Orders API",
    "provider_type": "http_api",
    "provider_config": {
      "credentials": {
        "type": "http_header",
        "headers": {"Authorization": "Bearer your-api-token"}
      },
      "items_path": "data.orders",
      "item_id_field": "order_id",
      "item_modified_field": "updated_at"
    }
  }'

Create a sync on your bucket

sync = client.buckets.syncs.create(
    bucket_id="bkt_your_bucket_id",
    connection_id=connection["connection_id"],
    source_path="https://api.example.com/v2/orders",
    polling_interval_seconds=900,
    batch_size=100,
)
print(f"Sync created: {sync['sync_config_id']}")

curl -X POST https://api.mixpeek.com/v1/buckets/bkt_your_bucket_id/syncs \
  -H "Authorization: Bearer YOUR_MIXPEEK_API_KEY" \
  -H "X-Namespace: ns_your_namespace_id" \
  -H "Content-Type: application/json" \
  -d '{
    "connection_id": "conn_your_connection_id",
    "source_path": "https://api.example.com/v2/orders",
    "polling_interval_seconds": 900,
    "batch_size": 100
  }'

Verify the connection

cURL

curl -X POST https://api.mixpeek.com/v1/organizations/connections/conn_your_connection_id/test \
  -H "Authorization: Bearer YOUR_MIXPEEK_API_KEY"

Incremental sync

Set item_modified_field to a timestamp field. Mixpeek stores the highest value it has seen and requests only newer items on the next run. Without it, every run reads the full response. Deduplication still applies, so repeated items do not create repeated objects.

RSS

Sync RSS and Atom feeds.

Sync configuration

Sync modes, reconciliation, and scheduling.

Get started

Connect your data

Extract features

Build retrievers

Enrich & organize

Integrate & operate

Resources

Overview

Prerequisites

Configuration

Connection-level fields

Sync-level fields

Setup

Incremental sync

RSS

Sync configuration

​Overview

​Prerequisites

​Configuration

​Connection-level fields

​Sync-level fields

​Setup

​Incremental sync

​Related

RSS

Sync configuration

Overview

Prerequisites

Configuration

Connection-level fields

Sync-level fields

Setup

Incremental sync

Related