Test and benchmark provider adapters

Evidence layers

Provider confidence comes from complementary layers:

1. unit tests for encoding and local state;
2. deterministic TCP or HTTP cassettes for wire behavior;
3. conformance reports for shared invariants;
4. fault injection for timeout, truncation, response loss, and cancellation;
5. pinned browser tests for CORS, Fetch, WebSocket, and WebRTC;
6. opt-in live canaries for the remaining external boundary;
7. benchmarks for measured performance, never correctness.

One successful live request proves almost none of the failure contract.

Protocol conformance

Verify provider identity, exact request shape, fragmented stream ordering, terminal completion, reasoning separation, usage accounting, Tool argument assembly, structured errors, retry safety, redaction, and concurrent isolation.

runifold-provider-testkit produces machine-readable acceptance evidence. Support documentation should name which checks passed for the exact adapter and model family.

Fault testing

Inject body timeout, connection reset, truncated stream, malformed JSON, oversized frame, invalid event order, cancellation race, lost successful response, and downstream unavailability. Assert error kind, retry safety, committed stream state, and absence of credentials in diagnostics.

Use real disposable dependencies where correctness relies on their transaction, clock, locking, browser, or Object Lock behavior.

Benchmark contract

Measure success rate, throughput, total latency percentiles, time to first model output, empty-output successes, concurrency, and environment metadata. Timing includes request construction, signing, transport, routing, queueing, decoding, and validation.

Fair comparisons use the same release mode, machine, Rust version, runtime, model, endpoint, request, limits, concurrency, telemetry, connection reuse, and retry policy. Alternate order and retain every raw report.

Claim policy

Do not publish a single-run winner. Use repeated paired rounds, medians, confidence intervals, and both success and latency. Separate loopback framework overhead from Provider-network behavior.

Phrase claims as “verified under these conditions,” link the artifact, and state what remains unverified.

Local verification ladder

Run the cheapest deterministic layer first and make live access an explicit, credentialed final step:

cargo test -p runifold-providers
cargo test -p runifold-provider-testkit
cargo test -p runifold-providers --test openai_control_http
cargo test --workspace --all-features

For every failing cassette, record the adapter, model family, protocol event, expected normalized error kind, retry-safety decision, and whether output was already committed. A live canary should use a dedicated low-privilege project, a fixed small request, a hard spend limit, and a test name that makes network access obvious. Never make live credentials a prerequisite for ordinary cargo test.