Operate Runifold with SLOs

Operational signals

Operate Runifold as a causal execution system, not only as an HTTP client. Measure root Run success and latency, model duration and first chunk, Agent turns, Tool and delegation failures, Workflow queue and lease behavior, MCP stages, budget exhaustion, and durable Store health.

Every terminal failure should retain a normalized kind and correlation to the Run tree. “Provider error” is too broad for an actionable objective.

Instrumentation layers

OtelModel instruments canonical model calls and routing. OtelJournal exports semantic Run evidence. OtelRuntime connects the configured OpenTelemetry boundary. Workflow budget and Task governance modules add projection, supervision, cleanup, and retention metrics.

Instrument at stable library boundaries. Avoid wrapping only the outer HTTP handler, which hides slow Tools, repeated turns, fallback attempts, and wait time.

Default objectives

The bundled runbook starts with:

These are conservative examples, not universal promises. Create separate objectives for direct model traffic, durable Workflow completion, queue delay, and Realtime sessions when those are user-visible products.

Incident workflows

For elevated failures, split by normalized error kind, follow an exemplar into the Run tree, then inspect provider, Tool, child, Store, and Effect boundaries. For latency, separate queueing, model time, first chunk, turns, and callable duration. For budget pressure, compare resource dimensions before raising caps.

Never respond to an outage by enabling broad retries. Confirm retry safety, stream commit state, idempotency, remaining deadline, and downstream capacity.

Cardinality and content

Prometheus labels should contain bounded status, error kind, stage, and route class. Run IDs, invocation IDs, Agent or Tool names, request IDs, tenants, and user extensions belong in traces or controlled logs, not labels.

OpenTelemetry capture is redacted by default. Enable prompt or output capture only with explicit purpose, access control, retention, deletion, and incident review. The observability system must not become a second ungoverned data store.

Install the bundled assets

Export the SLO assets from the exact Runifold version used by the service, so dashboard queries and emitted metric names cannot drift:

use std::fs;
 
fs::create_dir_all("artifacts/observability")?;
fs::write(
    "artifacts/observability/prometheus-rules.yaml",
    runifold::otel::slo::PROMETHEUS_RULES,
)?;
fs::write(
    "artifacts/observability/grafana-dashboard.json",
    runifold::otel::slo::GRAFANA_DASHBOARD,
)?;

Validate the Prometheus rules before deployment, import the dashboard into a staging data source, and fire one synthetic success, timeout, Tool failure, and budget exhaustion. If a panel is empty, first verify instrumentation and label names; do not weaken the alert. Record the chosen SLO targets, burn-rate action, owner, and rollback condition beside the deployed assets.