Recover safely from checkpoints

Checkpoint envelope

A Checkpoint has stable ID, owning Run, monotonic revision, namespaced kind, schema version, payload, and update time. CheckpointStore::compare_and_swap creates only at revision zero and updates only from the exact current revision to the next one.

This is optimistic concurrency, not a distributed lease. It prevents an old writer from overwriting newer semantic state; Workflow ownership adds separate fencing where processes execute concurrently.

Write-ahead phases

Persist intent before external work. An Agent checkpoint distinguishes ReadyForTurn, TurnInFlight, and Completed. A Workflow checkpoint records the active step or branch state. Stable state is written only after every result required to explain the transition is durable.

A process stopping after an external call but before the stable checkpoint creates ambiguity. The store must not rewrite that phase as success or failure.

Resume policy

Completed state is replayed without model, Tool, or Workflow execution. Stable state continues from the next transition. In-flight state is rejected under the conservative policy.

An explicit retry policy acknowledges that model cost, Tool calls, delegation, or custom steps may repeat. Pair it with the same Effect Store so completed idempotent work can replay recorded results. Restored usage never decreases.

Schema and versioning

Checkpoint payloads are provider-neutral but versioned. Match Agent, model, Workflow name, and definition version before resuming. Reject unsupported schemas instead of guessing how an old branch or phase maps to new code.

Migration should be explicit, offline or transactionally fenced, reversible where possible, and covered by fixtures from every supported version.

Privacy and operations

Unlike redacted default Journal events, checkpoints may contain transcripts, generated content, Tool results, Workflow input, and provider extensions. Apply encryption, tenant authorization, backup, retention, deletion, and access audit.

Monitor revision conflicts, ambiguous recovery, schema mismatch, resume age, and checkpoint size. Test process death at every write-ahead boundary, not only clean restart.

Commit one revision safely

let initial = Checkpoint::initial(
    CheckpointId::new(),
    run.run_id(),
    "acme.order",
    1,
    json!({ "phase": "ready" }),
);
store.compare_and_swap(&initial, None)?;
 
let loaded = store.load(initial.id)?;
let next = loaded.next(json!({ "phase": "validated" }))?;
match store.compare_and_swap(&next, Some(loaded.revision)) {
    Ok(()) => println!("committed revision {}", next.revision),
    Err(error) if error.kind == CheckpointErrorKind::Conflict => {
        // Another writer won. Reload; do not overwrite its decision.
        let current = store.load(initial.id)?;
        println!("current revision is {}", current.revision);
    }
    Err(error) => return Err(error.into()),
}

Use None only for create-only revision zero. Persist an in-flight phase before external work and a stable phase only after the result required to explain the transition is durable. A crash between them is intentionally ambiguous and must be resolved by Resume Policy plus Effect evidence.