Run multi-tenant workflow infrastructure

Tenant is a security boundary

WorkflowTenantId participates in queueing, claims, waits, Signals, budgets, Task lookup, retention, and audit. It is not decorative metadata. Every Store operation must scope by tenant before it observes or mutates workflow identity.

Normalize cross-tenant lookup to not-found where revealing existence would leak information. Authenticate the caller before translating application identity into a Workflow tenant.

Admission and fairness

Admission policy bounds queued and active work per tenant before a Task enters the system. Claim order should prevent one hot tenant from monopolizing every Worker. Batch size and concurrency remain bounded globally and per tenant.

Backpressure is a product behavior. Return a typed overload or quota result; do not accept work that cannot be durably represented and hope Workers catch up.

Durable budget ledger

In-memory Run budgets bound one causal execution. Multi-process workflow budgets also need durable reservations and immutable audit facts. Reserve before dispatch, reconcile actual usage, and release or forfeit under fenced ownership.

Projection and supervision can turn the ledger into metrics, but metrics are not the ledger. A missed scrape must not restore spending authority.

Retention control plane

Physical cleanup handles terminal Tasks only after operator-selected retention. Cleanup ownership is itself leased and fenced per tenant. Candidate selection, tombstone insertion, dependent-state deletion, and Task deletion must be atomic.

An immutable tombstone preserves tenant, workflow identity, terminal status, and lifecycle timestamps after ordinary lookup returns not found. Removing that audit evidence requires a separate policy.

Operational invariants

• tenant identity is present in every storage and authorization path;
• stale claim or cleanup owners cannot write;
• queued, leased, and waiting Tasks are never retention candidates;
• budget reservation and Task transition commit atomically where required;
• Task and Signal IDs never grant authority alone;
• fairness, quota, cleanup, and projection have bounded batch sizes;
• audit facts survive the mutable operational record.

Configure tenant admission

let tenant = WorkflowTenantId::parse("tenant-acme")?;
store
    .set_tenant_policy(
        tenant.clone(),
        WorkflowTenantPolicy::new(
            1_000, // maximum non-terminal Tasks
            20,    // maximum concurrent leases
        )?,
    )
    .await?;
 
let task = WorkflowTask::new("order-review", 1, input)?
    .with_tenant(tenant);
store.enqueue(task).await?;

Derive WorkflowTenantId from authenticated server-side identity; never accept it directly from a model or untrusted request body. Configure policy before admitting work, return a typed overload/quota response on denial, and measure queue age and active leases by bounded tenant classes rather than tenant IDs in metric labels.