Deployment

• API Layer – FastAPI + Celery + Redis connection (HTTP endpoints, task orchestration, webhooks).
• Engine Layer – Ray cluster + Ray Serve (extractors, inference, clustering, taxonomy runs).

​

Local Development

./start.sh api      # FastAPI + Celery
./start.sh celery   # Celery Beat
./start.sh engine   # Ray head + workers

• mongodb – metadata (mongodb://localhost:27017)
• mvs – vector storage (MVS)
• redis – task queue/cache (redis://localhost:6379)
• localstack – S3 emulator (http://localhost:4566)

​

Production Topology (Kubernetes)

Namespace: mixpeek-api
  ├─ fastapi-deployment (ReplicaSet + HPA)
  ├─ celery-worker-deployment (process tasks)
  └─ celery-beat-deployment (1 replica scheduler)

Namespace: mixpeek-engine
  ├─ ray-head (StatefulSet)
  ├─ ray-worker-cpu (Autoscaled Deployment)
  └─ ray-worker-gpu (Autoscaled Deployment)

Namespace: mixpeek-data
  ├─ mongodb (StatefulSet, replica set)
  ├─ mvs (StatefulSet or distributed cluster)
  └─ redis (Deployment or Redis Cluster)

• API nodes – general purpose (e.g., t3.xlarge), scale FastAPI/Celery horizontally.
• CPU workers – compute-optimized (e.g., c5.4xlarge) for text extraction, clustering.
• GPU workers – GPU instances (e.g., p3.2xlarge) for embeddings, rerankers, video processing.

​

Managed Ray (Anyscale / Ray Service)

• Deploy the Engine layer via a managed Ray service.
• Point the API layer to the Ray cluster using ENGINE_API_URL and Ray job submission credentials.
• Managed Ray handles autoscaling, node health, and GPU provisioning; you manage API + data stores.

​

Core Environment Variables

​

Health & Verification

• Endpoint: GET /v1/health – checks Redis, MongoDB, MVS, Celery, Engine, ClickHouse (if enabled).
• Smoke test: create namespace → bucket → collection → upload object → submit batch → execute retriever.
• Tasks: ensure Celery workers process webhook events, cache invalidations, and maintenance tasks.

​

Scaling Guidelines

​

Deployment Checklist

1. Provision MongoDB, MVS, Redis, and S3/GCS buckets (with IAM roles).
2. Deploy Ray cluster (head + workers) and confirm job submission works.
3. Deploy FastAPI + Celery services; configure environment variables to point to Ray + data stores.
4. Configure ingress/HTTPS, secrets, and network policies.
5. Run health checks and quickstart workflow to verify end-to-end functionality.
6. Set up observability (logs, metrics, webhooks) and configure backups for MongoDB.

​

References

• Architecture – full system design
• Observability – metrics, logs, dashboards
• Security – tenancy, auth, secret management
• Webhooks – event processing pipeline