Read reliability claims precisely

Claim policy

Read every reliability claim as a statement with scope and evidence. “Supports provider X” may mean compilation, deterministic protocol tests, live smoke tests, or production observation; those are not equivalent.

Runifold is pre-alpha. Expect API evolution, review migrations, and pin the exact crate version used by your application.

Evidence matrix

For each critical path, record:

Claim	Required evidence
Correct model encoding	deterministic contract tests
Real endpoint compatibility	dated live smoke test
Cancellation propagation	fault-injection test
Durable recovery	worker-loss and lease-expiry test
Safe external effect	idempotency or reconciliation test

Keep evidence near the release artifact so a future reader can reproduce it.

Establish a release baseline

Pin the exact pre-1.0 version and keep only the features the application uses:

Cargo.toml

[dependencies]
runifold = { version = "=0.9.0", features = ["sqlite-bundled"] }
runifold-providers = { version = "=0.9.0", features = ["openai"] }

Run the deterministic baseline on every change:

cargo fmt --check
cargo check --locked --all-targets
cargo test --locked
cargo clippy --locked --all-targets -- -D warnings

Then add application evidence the compiler cannot provide: one redacted provider cassette per protocol path, budgeted live smoke tests for exact model IDs, forced cancellation, worker-loss recovery, and Effect reconciliation. A green compile proves type compatibility, not endpoint or behavioral compatibility.

Benchmarks

A benchmark is meaningful only with workload, hardware, runtime, concurrency, features, and percentile definition. Provider network latency usually dominates microbenchmarks of local orchestration.

Measure what matters to users: time to first event, total completion latency, resource use, recovery time, and cost. Compare versions under the same fixture.

Pre-alpha expectations

Before production adoption:

pin dependencies and review release notes;
run your own provider and platform verification;
set budgets, deadlines, and explicit capabilities;
test cancellation, worker loss, and ambiguous effects;
keep a rollback path and inspect journals during incidents.

Open an issue with a minimal reproduction when a documented contract and observed behavior differ.

Production-readiness worksheet

Question	Evidence to retain	Release blocker
does the exact model support requested tools/streaming/schema?	dated live smoke result	unsupported or unknown capability
does cancellation stop local work and bound remote exposure?	injected disconnect/deadline test	unbounded continuation
can a task resume after process loss?	killed-worker recovery fixture	repeated committed step or lost usage
can external writes be retried safely?	idempotency/reconciliation test	ambiguous state treated as failure
can operators explain one run?	correlated journal + trace sample	no terminal evidence
can the release be rolled back?	tested binary/schema/definition plan	old tasks require removed code

Record the crate lockfile, Rust version, target, features, provider/model IDs, test timestamp, and fixture revision with the result. Re-run the affected row when any of those inputs changes.

Minimal reproduction template

When behavior differs from the documented contract, reduce it to one pinned Cargo.toml, one source file, exact command, expected result, actual typed error, target/runtime, and whether the test uses a cassette or live endpoint. Remove credentials, prompts, customer content, provider request IDs, and signed URLs.