Handle errors and retries safely
Distinguish build failures, execution failures, retry-safe transport errors, and ambiguous side effects.
Error layers
Handle failures at the boundary that understands them:
| Layer | Examples | Typical response |
|---|---|---|
| Build | duplicate tool, invalid policy | fail at startup |
| Model | rate limit, bad response | route or retry by classification |
| Tool | invalid input, denied capability | return a safe tool result |
| Run | deadline, budget, cancellation | terminate the run tree |
| Workflow | lease loss, stale revision | recover from durable state |
Do not flatten all of these into one string. Preserve the typed source and a safe message for the caller.
Inspect the typed cause
Agent::prompt_text returns AgentError. Match the layer before converting it
to your application's HTTP, queue, or CLI error. The wildcard keeps the code
forward-compatible because public error enums are non-exhaustive.
use runifold::{Agent, AgentError};
async fn answer(agent: &Agent, prompt: &str) -> anyhow::Result<String> {
match agent.prompt_text(prompt).await {
Ok(text) => Ok(text),
Err(AgentError::Model(error)) => {
eprintln!(
"model kind={:?} retry_safety={:?} provider={:?}",
error.kind, error.retry_safety, error.provider
);
Err(error.into())
}
Err(AgentError::Budget(error)) => {
eprintln!("budget stopped the run: {error}");
Err(error.into())
}
Err(AgentError::AmbiguousCheckpoint { turn }) => {
anyhow::bail!("turn {turn} may already have produced an external result")
}
Err(error) => Err(error.into()),
}
}Expose a stable application error code to callers, but retain the typed source and Run ID in controlled diagnostics. Never send provider bodies, prompts, tool arguments, or credentials back as an error detail.
Safe retry
Retry only when the adapter classifies the failure as safe. A connection error before a request was sent differs from a timeout after the provider may have accepted and billed it.
Use capped exponential backoff with jitter. Retry budgets should be smaller than the request's overall deadline and resource budget.
Circuit breakers
runtime() composes conservative retry and circuit-breaker defaults around a
provider. Inspect route_health() for operational status and route away from
a failing edge before every request pays the full timeout.
Circuit breakers protect capacity; they do not prove that a provider is healthy. Combine them with health checks and outcome metrics.
Ambiguous effects
Never blindly retry a non-idempotent external effect. Use an idempotency key when the destination supports one, or Runifold's write-ahead effect boundary to record intent before execution.
If completion cannot be proven, mark the effect ambiguous and require reconciliation. “Probably failed” is not safe enough for payments, email, or destructive tools.
Retry decision table
| Observation | Retry automatically? | Action |
|---|---|---|
| local validation or unsupported feature | no | fix input, feature policy, or model selection |
| cancelled or deadline exceeded | no | return the terminal state; a caller may start new work |
| transport failure classified safe | within policy | use capped backoff and remaining deadline |
| provider rate limit classified safe | within policy | honor provider delay and global budget |
| malformed provider protocol | normally no | capture safe diagnostics and route away |
| tool input rejected | no identical retry | let the model correct arguments within turn budget |
| idempotent write with confirmed key | policy-dependent | replay through the Effect boundary |
| non-idempotent or ambiguous write | no | reconcile with the destination first |
Operational debugging sequence
- Find the root Run ID and terminal error kind.
- Check cancellation, deadline, and budget before investigating the provider.
- Inspect
runtime.route_health()for open or half-open model routes. - Correlate the failed attempt with journal events and telemetry without exposing content.
- Confirm whether an Effect intent, completion, or ambiguous state was persisted.
- Retry only through the same policy boundary; do not add an ad-hoc loop around
prompt_text.