Make workflows durable

Durability model

A durable workflow persists semantic progress, then lets any compatible worker continue it. Process uptime is not part of correctness.

Use durability for work that survives deploys, waits for people or timers, coordinates costly effects, or must recover after a worker disappears.

Run one durable task locally

Enable SQLite for a single-machine durable worker. Use PostgreSQL instead when multiple processes or hosts must claim work concurrently.

[dependencies]
runifold = { version = "=0.9.0", features = ["sqlite-bundled"] }
serde_json = "1"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

Assuming workflow is the Workflow built in the Workflow guide, this creates the durable control plane, registers the exact definition, enqueues one task, and runs one claim cycle:

use std::{sync::Arc, time::Duration};
 
use runifold::{
    Budget, CapabilitySet, LeaseDuration, WorkerId, WorkflowDefinition,
    WorkflowRegistry, WorkflowStore, WorkflowTask, WorkflowWorker,
    sqlite::SqliteWorkflowStore,
};
use serde_json::json;
 
let store = Arc::new(SqliteWorkflowStore::open("runifold-workflows.db")?);
let mut registry = WorkflowRegistry::new();
registry.register(WorkflowDefinition::new(
    Arc::new(workflow),
    Budget::default(),
    CapabilitySet::new(),
))?;
 
store
    .enqueue(WorkflowTask::new(
        "order-intake",
        1,
        json!({ "order_id": "ord_42" }),
    )?)
    .await?;
 
let worker = WorkflowWorker::new(
    store,
    registry,
    WorkerId::parse("worker-local-1")?,
    LeaseDuration::new(Duration::from_secs(30))?,
    Duration::from_secs(10),
)?;
let outcome = worker.run_once().await?;
println!("{outcome:?}");

run_once() returns Idle, Completed, Retried, Suspended, Failed, DefinitionUnavailable, or LeaseLost. Production services normally wrap the worker in a bounded supervisor; use run_once() in tests and when integrating with an existing job loop.

Checkpoints and revisions

Checkpoints record the workflow definition, current phase, completed step outputs, budget state, and recovery metadata. Revisions prevent two workers from committing incompatible progress.

Keep step outputs serializable, compact, and versioned. Store large artifacts outside the checkpoint and reference them by immutable ID.

Timers, signals, and review

A wait persists intent without holding a thread or process. Timers wake after a deadline; signals wake on an external event such as approval.

Signal names and payloads are application contracts. Authenticate the sender, deduplicate by signal ID, and retain enough history to explain who resumed the workflow and why.

Workers and leases

Workers claim tasks using finite leases. A healthy worker renews its lease; another worker may recover the task after expiry.

The SQLite store is useful for local durable execution. Use the PostgreSQL workflow feature for distributed workers and multi-process coordination.

Recovery policy

Recovery must distinguish a safely repeatable step from an effect whose result is unknown. Configure resume policy per boundary and reconcile ambiguous effects before proceeding.

Deploy compatible workflow definitions while old tasks exist. Treat step IDs, serialized state, and signal names like a database schema.

Recovery drill

Before production, run this drill against the real store:

1. enqueue a task with a known checkpoint ID;
2. terminate the worker while a deterministic step is in flight;
3. wait beyond the lease duration and start a different worker identity;
4. verify completed steps are not repeated and usage never decreases;
5. repeat at an external write boundary and confirm the result becomes replayed or ambiguous according to Effect evidence—not guessed from a timeout;
6. deploy definition version 2 while a version 1 task is waiting and verify both definitions remain registered.

Common failures

See workers, waits and signals, checkpoint recovery, and workflow versioning for the corresponding control-plane details.