Worker architecture

Aegra uses a worker-based execution model for production deployments. Runs are dispatched through a Redis job queue and executed by concurrent asyncio workers across any number of instances. Lease-based crash recovery ensures no run is lost, even if an instance dies mid-execution. In dev mode (aegra dev), none of this is needed — runs execute as simple in-process asyncio tasks.

​

Run lifecycle overview

How a single run flows through the system — from client request to completion: View fullscreen ↗ The happy path of a single run, step by step:

1. Client sends a run request to any instance via the load balancer.
2. FastAPI validates, persists the run to PostgreSQL (with serialized execution params), and pushes the run ID onto the Redis job queue.
3. Worker picks up the job via BLPOP (blocking pop — instant delivery, no polling).
4. Worker acquires an exclusive lease in PostgreSQL, executes the graph, heartbeats every 10s, and finalizes the run (status, output, release lease).
5. Worker publishes each event to Redis Pub/Sub for real-time streaming.
6. SSE Broker subscribes to Redis Pub/Sub and relays events.
7. Client receives events over an SSE connection.

Behind the scenes, the Lease Reaper scans PostgreSQL every 15 seconds for runs with expired leases (crashed workers) and re-enqueues them. Workers also send OpenTelemetry traces to Langfuse, Phoenix, or any OTLP collector.

​

Horizontal scaling

Every instance is stateless — add more instances behind a load balancer to increase capacity. All instances share the same Redis and PostgreSQL backends. View fullscreen ↗ Each instance runs multiple worker loops (default 3), and each worker loop handles up to N_JOBS_PER_WORKER concurrent runs (default 10) via an asyncio semaphore. Total capacity per instance is WORKER_COUNT x N_JOBS_PER_WORKER — 30 concurrent runs by default. A job enqueued by Instance A can be picked up by Instance B — any worker can execute any run.

​

Job lifecycle

The detailed sequence — every Redis command, database query, and semaphore operation involved in executing a single run: View fullscreen ↗ Key details:

1. Execution params are stored in PostgreSQL alongside the run, so any worker can reconstruct the full job context (user identity, config, trace metadata, interrupt settings, etc.).
2. Lease acquisition is atomic — UPDATE ... WHERE claimed_by IS NULL ensures only one worker wins the race.
3. Heartbeats extend the lease every 10 seconds. If a worker crashes, the lease expires and the run becomes eligible for recovery.
4. OpenTelemetry trace context propagates across the Redis queue boundary, so traces span the full request lifecycle.

​

Streaming run with crash recovery

The full picture — a streaming run shows how the client SSE connection, worker execution, and reaper recovery all interact: View fullscreen ↗

​

Crash recovery

Every instance runs a LeaseReaper background task that scans for runs with expired leases. When a worker crashes (OOM, kill signal, network partition), its heartbeat stops and the lease expires. The reaper resets the run to pending and re-enqueues it. The new worker resumes from the last checkpoint. Once retries are exhausted, the reaper marks the run error, clears ownership, and marks the thread error in the same transaction when no other run is active. View fullscreen ↗ The reaper runs on every instance (default every 15 seconds). This is safe — the atomic lease acquisition prevents duplicate execution.

​

Graceful shutdown

On SIGTERM (rolling deploy, pod reschedule, node drain) the executor waits up to WORKER_DRAIN_TIMEOUT (default 30s) for in-flight runs to finish. Runs still executing when the window closes are handed back to the queue, not finalized: the run is reset to pending with its lease cleared and re-enqueued, so another instance resumes it from the last checkpoint — the same guarantee the crash path provides. Clients streaming the run stay connected and pick up events from the resuming worker.

• A user-initiated cancel that races the shutdown still wins: explicitly cancelled runs are finalized as interrupted, never resurrected.
• If Redis is unreachable during the hand-off, the rows are already pending and unclaimed, so a surviving instance’s reaper re-enqueues them after STUCK_PENDING_THRESHOLD_SECONDS.
• Drain hand-offs do not consume the crash retry budget (BG_JOB_MAX_RETRIES).

Set your orchestrator’s grace period a few seconds above WORKER_DRAIN_TIMEOUT (on Kubernetes: terminationGracePeriodSeconds), so the requeue completes before the pod is killed. Runs longer than the drain window survive the deploy either way; the timeout only decides how long the old pod waits before handing them off.

​

Cross-instance cancellation

Cancel requests can arrive at any instance, but the run may be executing on a different one. The API first tries an ownership-safe database update that succeeds only when the run is active and has no owner or an expired lease. That path interrupts the run, clears its lease, and marks the thread idle in one transaction when no other run is active. The API still broadcasts a best-effort cancellation after this transaction because an expired worker may be delayed rather than dead. It emits the terminal stream event itself so connected clients always receive a definitive end event. When the run has a live owner, the API leaves the database state unchanged and uses Redis pub/sub to ask that worker to cancel its task and finalize the run. Requests for runs that are already terminal are no-ops, so a late cancellation cannot overwrite a successful or failed result. Worker finalization is also conditional on the run remaining active, which prevents a delayed worker from overwriting a terminal state committed by the API. The response contains the currently persisted state; set wait=1 to wait for a live worker to finalize the run before returning.

​

Dev vs production mode

​

Redis data layout

​

Configuration

All worker settings are configured via environment variables. See the environment variables reference for the full list.

​

File map