Reliability and safety
Two lines of defense: an ordered set of admission guards on the way into a job, and a circuit breaker around the model calls inside it. Both sit on top of a multi-tenant core where isolation is an invariant.
Multi-tenant isolation#
Every row, artifact, and queue entry is scoped to a tenant. Tenancy is denormalized into even leaf tables, and no query is allowed to join across tenants, so a bug in one query cannot leak another tenant's data. API keys are stored only as a one-way hash.
Request admission#
Every job submission passes the same ordered gates before it is accepted, shown in the diagram above: authentication, then SSRF screening of the source URL, then the idempotency check, then the per-tenant cost cap. A request that fails any gate is rejected with a specific error code and never starts a job.
Operational safeguards#
Per-tenant daily cost cap
Each tenant has a daily cost ceiling. When a tenant exceeds it, new jobs are rejected with a COST_QUOTA_EXCEEDED error, so a runaway or a leaked key cannot accrue unbounded model spend.
SSRF screening
Source URLs on POST /v1/jobs are screened before the job runs. The check rejects literal private-range IPs and hostnames that resolve (via DNS-over-HTTPS) into private CIDRs, with an allowlist for trusted platforms. A blocked source returns SOURCE_BLOCKED.
Per-model circuit breaker
The reasoning and Pass A model calls are wrapped in a per-model circuit breaker (closed, open, half-open) backed by D1 state shared across Worker invocations. When a model is degraded the breaker opens, so requests fail fast or fall back instead of piling onto a failing provider and dragging the whole pipeline down with it.
Idempotency
POST /v1/jobs accepts an Idempotency-Key header with a 24-hour TTL. Retrying with the same key returns the original job rather than creating a duplicate, with the write made race-safe by an insert-or-ignore.
Signed webhooks with secret rotation
Webhook deliveries are signed with HMAC-SHA256 and carry an X-Lirovo-Signature header. Signing secrets can be rotated via POST /v1/destinations/{id}/rotate-signing-secret, with an optional grace window during which deliveries are dual-signed so you can cut over without dropping events.
Weekly D1 backups
A weekly cron dumps every D1 table as JSON plus a manifest to R2 under a dated prefix, giving a point-in-time snapshot to recover from, with a documented restore procedure.
Trace id and Sentry
Every request gets a trace id at API ingress that propagates through the Workflow, the Container, and the result write. It is returned on every response as x-lirovo-request-id. Logs are structured (pino), and errors are captured in Sentry with the trace id attached, so a single id locates a request across the whole pipeline.