Route across models without duplicate output

Logical and physical models

A router presents one logical ModelRef to the Agent and selects among named physical routes. This keeps prompts, Tools, RunContext, and downstream observability stable when provider topology changes.

Routing is not a claim that models are behaviorally interchangeable. Validate output quality and feature support for every route in the same logical pool.

Build a router

use std::{sync::Arc, time::Duration};
use runifold::{
    Agent, CircuitBreakerConfig, ModelFallbackPolicy, ModelRef,
    ModelRouter,
};
 
let logical = ModelRef::new("router", "assistant");
let router = ModelRouter::builder(logical.clone())
    .route(
        "primary",
        primary_model,
        ModelRef::new("openai", "gpt-5"),
    )
    .route(
        "backup",
        backup_model,
        ModelRef::new("anthropic", "claude-sonnet"),
    )
    .fallback_policy(ModelFallbackPolicy::safe_only())
    .circuit_breaker(CircuitBreakerConfig::new(
        3,
        Duration::from_secs(30),
    )?)
    .build()?;
 
let agent = Agent::builder("assistant", Arc::new(router), logical)
    .build()?;

Use stable route names for metrics and operations. Do not encode credentials, tenant IDs, or ephemeral hostnames into them.

Retry and fallback safety

Retry repeats the same physical route. Fallback selects another route. Both can duplicate cost if a provider may have accepted the first request.

The default fallback policy accepts only errors explicitly marked retry-safe. Allowing an error kind with unknown safety is a deliberate decision to risk another charge. Respect provider Retry-After, the invocation deadline, and the shared run budget.

Cancellation never retries or falls back.

Stream locking

Once the first canonical stream event is visible, Runifold locks the selected route. Switching providers after partial output would duplicate text, tool calls, or reasoning in ways the consumer cannot reliably reconcile.

This means a mid-stream failure is returned as a failure, not hidden behind a fresh answer from another model. Design the user interface to preserve partial progress while making terminal status clear.

Operate the router

Circuit breakers count failures per physical route. After the threshold, a route is skipped until cooldown; exactly one half-open request probes recovery.

Export route_health() into readiness diagnostics and operational dashboards. Track selection count, failure classification, retry attempts, breaker state, latency, tokens, and quality by route.

Routing checklist:

• keep one logical pool limited to models that satisfy the same product contract;
• set explicit retry and fallback policies;
• test rate limits, connection loss before send, response loss, and stream interruption;
• use effects for non-idempotent tools—the model router cannot make writes safe;
• run evaluation gates before adding or promoting a backup model.