Collections
Turn raw objects into a searchable set of documents, derived from sources you never have to re-upload
Why do anything?
Uploaded media is not searchable. A two-hour video is one opaque file until something segments it, transcribes it, reads its on-screen text and embeds the result.
Why now?
Teams accumulate media faster than they can catalogue it, and every extractor change would otherwise mean re-uploading terabytes rather than re-running over what is already stored.
Why this feature?
A collection declares its sources and its extractors, then derives documents from them. Because the derivation is declared rather than hand-built, changing the configuration and re-running rebuilds the documents without touching the original objects.
How It Works
A collection is derived state. Sources hold what arrived; the collection holds what was computed from it, which is what queries actually run against.
Source resolution
Resolve the declared sources, applying any per-source field map
Extraction
Run the named extractors over objects in batches
Document write
Write documents with signals, vectors and lineage
Index update
Make the new documents queryable through the namespace indexes
Why This Approach
Keeping sources immutable and the collection derived means an extractor or model change is a re-run rather than a re-ingest, which is the difference between minutes and moving terabytes.
Where This Is Used
Integration
collection = client.collections.create(collection_name="catalogue", source={"type": "bucket", "bucket_ids": ["bkt_ads"]})