Test without the network

Testing layers

Use a small test pyramid:

1. pure unit tests for policy, schemas, and workflow conditions;
2. deterministic Agent tests with scripted models and tools;
3. protocol contract tests with redacted cassettes;
4. a narrow set of live provider smoke tests;
5. versioned quality evaluations.

Most application correctness should not depend on a live model or network.

Scripted models

Add the runifold-testkit development dependency and use ScriptedModel to return a known sequence of model responses. Assert the final outcome, transcript, tool calls, events, and usage.

[dev-dependencies]
anyhow = "1"
runifold = "=0.9.0"
runifold-testkit = "=0.9.0"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

The following test makes no network request and needs no API key:

use std::{collections::BTreeMap, sync::Arc};
 
use runifold::{
    Agent, ContentPart, FinishReason, ModelRef, ModelStreamEvent,
};
use runifold_testkit::ScriptedModel;
 
#[tokio::test]
async fn agent_returns_the_scripted_answer() -> anyhow::Result<()> {
    let model = ScriptedModel::new();
    model.enqueue([
        ModelStreamEvent::ResponseStarted {
            id: Some("response-1".into()),
            model: ModelRef::new("test", "scripted"),
        },
        ModelStreamEvent::ContentPartCompleted {
            index: 0,
            part: ContentPart::text("approved"),
        },
        ModelStreamEvent::ResponseCompleted {
            finish_reason: FinishReason::Stop,
            provider_metadata: BTreeMap::new(),
        },
    ]);
    let observed = model.clone();
    let agent = Agent::builder(
        "reviewer",
        Arc::new(model),
        ModelRef::new("test", "scripted"),
    )
    .system("Return one review decision.")
    .build()?;
 
    let answer = agent.prompt_text("Review order 42").await?;
 
    assert_eq!(answer, "approved");
    assert_eq!(observed.recorded_requests().len(), 1);
    Ok(())
}

Run only this integration test with cargo test --test agent. Clone the scripted model before passing it to the Agent when you need to inspect captured requests or call contexts after execution.

Script both the happy path and boundaries: malformed tool arguments, denied capability, budget exhaustion, cancellation, provider refusal, and maximum turns.

Provider cassettes

Cassettes verify HTTP encoding and decoding without making CI depend on a provider. Redact authorization, cookies, request IDs, and user content before committing them.

Treat cassette format and endpoint versions as test fixtures. Refresh them deliberately when protocol behavior changes, not automatically during tests.

Quality evaluation

Model behavior is probabilistic, so evaluate observable product properties: correct classification, grounded citations, refusal behavior, tool choice, latency, and cost.

Keep prompts, datasets, graders, model identifiers, and acceptance thresholds versioned together. Inspect failures rather than trusting one aggregate score.

CI gates

Run formatting, clippy, unit tests, docs examples, protocol contracts, and offline evaluations on every change. Run live smoke tests in a protected environment with strict budgets.

Block release when a required capability becomes unknown or a verified provider combination loses its evidence.

What to assert at each boundary

Reproduce faults and recovery

runifold-testkit 0.9 includes productized disconnect, named Tool failure, runtime reconstruction and normalized golden-trace boundaries:

use runifold_testkit::{FaultScenario, RecoveryHarness};
 
let faults = FaultScenario::new()
    .disconnect_after_tool_call()
    .fail_tool_on_invocation("charge", 2, injected_error);
let model = faults.model(scripted_model);
let mut runtime = RecoveryHarness::new(runtime_factory, faults.clone());
 
runtime.restart();
faults.assert_tool_executed_exactly("charge", 1)?;

For 0.9 review gates, kill the process separately at review-ready, review-in-flight and approved-plan checkpoints. Assert that a ready candidate is not regenerated, a previously approved Tool plan is replayed exactly, and an ambiguous in-flight review fails until the application grants explicit retry authority. Golden traces remove generated IDs and timestamps while preserving the first causal divergence.

Test failure checklist

If ScriptedModel reports that no invocation is queued, count how many calls the Agent path should make—retries, tool follow-up turns, and fallback routes each consume another script. If the stream fails to complete, include exactly one terminal ResponseCompleted event. If a live test is flaky, move protocol correctness to a cassette test and keep the live test as a budgeted canary.