Observe the complete run tree

Three kinds of history

Keep three records separate:

the transcript is model-visible conversation state;
the journal is the runtime's semantic execution record;
telemetry is an operational projection for traces, metrics, and logs.

They have different retention, privacy, and correctness requirements. Reusing one as another creates accidental data exposure and weak recovery.

Structured journals

Journal stable domain events with run and parent IDs, phase, counters, and safe metadata. Consumers should tolerate new event variants and process events idempotently.

The journal is useful for audit and recovery, but it is not a dump of prompts, chain-of-thought, or provider payloads.

OpenTelemetry

Enable the otel feature to export GenAI-oriented traces and metrics through OpenTelemetry. Carry trace context across model, tool, child-Agent, and workflow boundaries.

Set service and deployment resource attributes at startup. Use sampling and attribute allowlists before data reaches an exporter.

Enable both your provider and the otel feature. Runifold emits through the global OpenTelemetry providers; your application still owns SDK, sampler, resource, exporter, batching, and shutdown configuration.

Cargo.toml

[dependencies]
runifold = { version = "=0.9.0", features = ["otel"] }
runifold-providers = { version = "=0.9.0", features = ["openai"] }

use runifold::ProviderModelExt;
use runifold_providers::openai::OpenAiClient;
 
let runtime = OpenAiClient::from_api_key(std::env::var("OPENAI_API_KEY")?)?
    .runtime("gpt-5")?
    .with_otel();
let agent = runtime
    .agent("support-triage")
    .system("Classify the request and explain the decision.")
    .build()?;
 
let outcome = agent.prompt("My order has not arrived", &run).await?;
println!("run={} tokens={:?}", run.run_id(), outcome.usage);

Initialize the OpenTelemetry SDK before constructing this runtime and flush the provider during graceful shutdown. Create the runtime once and clone it into handlers; rebuilding it per request also resets shared circuit-breaker health.

For custom model and journal wiring, construct one OtelRuntime, then use model(...) and journal(...) from the same instance so causal correlation is shared. Content capture and provider error-message capture are disabled by default; enabling them is a deliberate privacy decision.

Production signals

Track at least:

success, cancellation, deadline, and budget-exhaustion rates;
latency by provider, model, Agent, and callable;
tokens, cost, turns, tool calls, and delegations;
retry, circuit-breaker, lease-recovery, and ambiguous-effect counts;
structured-output and retrieval-quality failures.

Keep tenant IDs out of high-cardinality metric labels. Use traces or controlled logs for per-run investigation.

Investigation workflow

When a request fails, start from the application request ID and Run ID. Locate the terminal Run event, then follow child Runs to model and callable attempts. Use the trace to answer where time was spent, the journal to answer which semantic transitions committed, and the transcript only when authorized to inspect model-visible content.

Symptom	First signal	Next check
slow first token	model span latency	route health, queue wait, provider latency
high total latency	child span waterfall	tool/retrieval latency and retry count
unexpected cost	usage events	turns, fallback routes, losing race reservations
repeated write	Effect journal state	idempotency key and reconciliation evidence
missing trace	SDK/exporter health	global provider initialized before runtime
cardinality spike	metric attributes	remove Run, request, tenant, and document IDs

Safe production defaults

Keep model content disabled, sample at the SDK boundary, allowlist attributes, and set retention separately for traces and journals. Alert on rates and percentiles, not individual tenant labels. Test exporter outage: telemetry backpressure must be bounded and must not silently change execution correctness.

Inspect runs without executing effects

Runifold 0.9 ships a separate read-only operations CLI. It reads exported events or canonical SQLite/PostgreSQL journals without loading Provider credentials, running migrations, or re-executing Effects:

cargo install runifold-cli --version 0.9.0 --locked
runifold run inspect --events events.json
runifold run tail --events events.json --limit 50
runifold run inspect --sqlite runifold.db --run-id 019...
runifold run replay --events events.json --output replay-evidence.json
runifold checkpoint diff before.json after.json
runifold budget explain budget.json usage.json
runifold doctor --events events.json

Checkpoint diffs report JSON Pointers and change kinds without printing values. Replay produces causal evidence only; actual Effect execution remains behind the runtime's explicit recovery policy. Use doctor first when an exported run appears incomplete, then follow its normalized finding into traces and the durable journal.