Operate durable workflow workers

The worker model

A durable Workflow definition describes deterministic steps. A Workflow Task is persisted execution state for one definition, input, version, tenant, and checkpoint identity. Workers are replaceable processes that claim ready Tasks, execute bounded work, and commit state through the Store.

Do not keep correctness only in a process-local future. The process may stop after an external operation but before it reports completion.

Task lifecycle

A Task moves through queued, leased, waiting, terminal, or cancelled states. The Store owns authoritative timestamps and transitions. A Worker claims a bounded lease, reconstructs the Workflow from registered name and version, resumes from checkpoint state, and either commits progress, enters a durable wait, or records a terminal result.

Definition identity must be stable. Deploying new code must not reinterpret old checkpoints as a different graph.

Run a bounded supervisor

After constructing the WorkflowWorker shown in the durable workflow guide, host it with bounded concurrency and graceful cancellation:

use std::{sync::Arc, time::Duration};
 
use runifold::{
    CancellationToken, WorkflowSupervisor, WorkflowSupervisorConfig,
};
 
let config = WorkflowSupervisorConfig::new(8)?
    .with_backoff(Duration::from_millis(25), Duration::from_secs(5))?;
let supervisor = WorkflowSupervisor::new(Arc::new(worker), config);
let shutdown = CancellationToken::new();
let signal = shutdown.clone();
 
tokio::spawn(async move {
    if tokio::signal::ctrl_c().await.is_ok() {
        signal.cancel();
    }
});
 
let report = supervisor.run(&shutdown).await;
println!("{report:?}");

The concurrency value bounds simultaneous claim-and-execute cycles in this process. It does not replace per-tenant admission, database pool limits, model concurrency, or downstream rate limits. During shutdown, the supervisor stops scheduling replacement cycles and drains those already started so their lease protocol can finish.

Fenced leases

A lease names tenant, Task, Worker, expiration, and monotonically increasing fencing token. Heartbeats extend live ownership. After expiration, another Worker may claim with a higher token.

Every state-changing write validates ownership and fencing. The old process cannot commit after takeover, even if its network call returns late. Database time is the authority for distributed expiration; process clocks are not.

Checkpoint recovery

Checkpoints store semantic progress, not serialized Rust futures. Recovery decides the next step from durable phase and Effect evidence. Completed Effects can replay their recorded result; an ambiguous non-idempotent Effect fails closed for application reconciliation.

Parallel branches reserve shared budget before work. First-success races cancel losers only when doing so is semantically safe; cancellation does not undo already completed external writes.

Worker operations

Operate Workers with:

• bounded claim batches and concurrency;
• heartbeat intervals safely below lease duration;
• graceful shutdown that stops new claims and preserves owned progress;
• metrics for queue delay, lease loss, execution duration, waits, and failures;
• version registration for every recoverable definition;
• Store health and clock behavior included in incident response.

Scale by adding Workers only after the Store and downstream systems can absorb the extra concurrency. A larger fleet does not repair a hot tenant, missing idempotency, or an unbounded Workflow.

Deployment runbook

1. deploy schema changes and all still-active Workflow definitions;
2. start new Workers with a unique stable WorkerId per process;
3. confirm claims, heartbeats, and completions before increasing concurrency;
4. stop old Workers gracefully and wait at least one lease window before assuming every old owner is gone;
5. verify queue age, retry count, definition-unavailable count, and lease loss;
6. remove an old definition only after no queued, leased, waiting, or recoverable Task references it.

If lease loss rises, reduce concurrency and check database pool wait, transaction latency, heartbeat timing, and runtime stalls. If the queue grows while workers are idle, check tenant policy, due timers, definition registration, and whether tasks are repeatedly deferred by budget admission.