lirovo/ docs
Get startedAPI referenceSDKMCPArchitecture
Architecture

The extraction pipeline

A job runs as one Cloudflare Workflow: a durable, ordered sequence of stages that resumes from the last successful step. Media stages run in a Container; the model and graph stages run in the Worker.

Lirovo · architecture / pipelinelirovo.aisource videoContainer · mediaingestyt-dlpnormalizeffmpeg remuxscene-detectpHash dedupmediaWorker · HTTP + JSONASRvoxtralvisiongemini VLMPass A · knowledge graphfuse tracks · attach evidencereasontyped JSONwrite artifactsR2 + D1deliveroptionalaudioframestranscriptvisiongraphresultdata flowindirectentry point
Media stages run in the Container; transcription, vision, graph build, and reasoning run in the Worker. The audio and visual branches run in parallel and rejoin at Pass A.

The stages#

Stages run in order, with one fork: after normalization the audio and visual tracks run in parallel and rejoin at the graph build. The label on each step is the runtime it executes in.

  1. 1

    Ingest (Container)

    The Container fetches the source with yt-dlp and writes it to R2. The source URL is screened before the job starts (see Reliability and safety).

  2. 2

    Normalize (Container)

    FFmpeg remuxes the media and normalizes frame rate, emitting an audio track and a normalized video track to R2. It uses a stream-copy remux rather than a full re-encode where possible, which is the difference between minutes and seconds.

  3. 3

    Scene detection (Container)

    FFmpeg scene-cut detection samples representative frames to R2 with a manifest. This is the input to deduplication.

  4. 4

    Perceptual-hash dedup (Container)

    A perceptual hash drops near-duplicate frames so the vision model only sees visually distinct moments. Fewer frames means lower vision cost with no loss of coverage.

  5. 5

    Vision (Worker, parallel with ASR)

    The Worker describes the deduplicated frames with the vision model through AI Gateway. R2 reads and model batches fan out in parallel. Output is a per-frame description set in R2.

  6. 6

    ASR (Worker, parallel with vision)

    The Worker transcribes the audio track with Voxtral into a time-aligned transcript in R2. ASR and vision are independent branches that rejoin at the next stage.

  7. 7

    Pass A: knowledge graph (Worker)

    Pass A fuses the transcript and the vision descriptions into a compact temporal knowledge graph: nodes and edges, each node carrying a pointer to its source span and modality. The Workflow writes the canonical graph and a compact view to R2.

  8. 8

    Reasoning (Worker)

    The reasoning step reads the compact graph and emits typed JSON that conforms to the job's schema. Reasoning over the graph rather than the raw transcript is cheaper and more accurate.

  9. 9

    Write artifacts (Worker)

    The result and every intermediate land in R2. The Workflow mirrors the graph nodes, edges, and the evidence chain into D1 so they are queryable with SQL.

  10. 10

    Delivery (Worker, optional)

    If the job has a destination, the validated result is delivered in its own Workflow instance: HMAC-signed and retried with backoff, decoupled from the job status.

Two runtimes, one boundary#

The split is not arbitrary. The Workers runtime cannot run arbitrary binaries or long CPU-bound work, so anything that shells out to yt-dlp or FFmpeg, or hashes frames, has to run in the Container. Everything that is HTTP plus JSON logic (ASR, vision, reasoning, Pass A, validation) runs in the Worker, where Sentry, the request logger, and AI Gateway metering can observe every call.

The Container is stateless
The Container holds no job state. It reads its inputs from R2 by key and writes its outputs back to R2 by key, so it can be scaled, retried, or replaced without coordination. All durable state lives in the Workflow, R2, and D1.

Job lifecycle#

A job is created in queued, moves to running when the Workflow picks it up, and ends in exactly one terminal state. The typed result is fetched separately once the job has succeeded.

Lirovo · architecture / job lifecyclelirovo.aiPOST /v1/jobsqueuedrunningsucceededresult readyfailederror_codecancelledpicked upokerrorcancelGET resultdata flowentry point
A job is created queued, runs, and lands in one terminal state. The result is fetched separately. Cancel is cooperative from queued or running.
  • succeeded. The result is in R2 and readable at GET /v1/jobs/{id}/result.
  • failed. The job carries an error_code and error_message explaining why.
  • cancelled. Cancellation is cooperative: a cancel request marks the job, and the Workflow stops at its next checkpoint rather than being killed mid-step.

Durability and retries#

Because each stage is a Workflow step, the engine checkpoints after every one. A transient failure or a Worker restart resumes from the last successful step instead of re-running the whole job. The model-touching steps (the graph build and reasoning) carry an explicit retry policy on top, and webhook delivery retries independently with exponential backoff.

Re-extraction is cheap
Every intermediate is stored, so re-running the same video against a new schema reuses the media, frames, transcript, vision, and graph from R2 and only re-runs the reasoning step.

Failure modes and degradation#

The pipeline prefers a partial answer to no answer. If the visual branch fails, the job degrades to audio-only and still produces a result from the transcript rather than failing outright. If a model provider is degraded, the per-model circuit breaker opens so the pipeline fails fast or falls back instead of piling onto an unhealthy dependency. A hard failure surfaces as a terminal failed status with an error code, never a silent or partial write.

Latency#

End-to-end latency is dominated by media normalization and the model calls, not by orchestration overhead. The stream-copy remux and the parallel vision plus ASR branches bring a 90-minute video to roughly three to four minutes end to end. The tail is set by model-provider capacity rather than by Lirovo: the graph build is fast at the median but can spike when the model is under load, which is exactly what the circuit breaker and retries are there to absorb.