Evaluate quality and prevent regressions

What to evaluate

Unit tests verify deterministic contracts. Evaluations measure product behavior that can change with a prompt, model, provider, retrieval index, or Tool set.

Start with observable requirements:

• classification and structured-field correctness;
• grounded answers and citation accuracy;
• refusal and escalation behavior;
• Tool selection and argument quality;
• retrieval recall and irrelevant-context rate;
• latency, token use, and cost.

Do not use “sounds good” as a release criterion. Turn it into a case, scorer, and threshold.

Build a versioned dataset

Add the testkit as a development dependency:

cargo add --dev runifold-testkit
cargo add --dev serde_json

use runifold_testkit::{EvaluationCase, EvaluationDataset};
use serde_json::json;
 
let dataset = EvaluationDataset::new(
    "support-triage",
    "2026-07-30",
    vec![
        EvaluationCase::new("payment-urgent", json!("charged twice"))?
            .with_expected(json!({
                "category": "payment",
                "urgent": true
            })),
    ],
)?;

Case IDs are stable; dataset versions change when cases or labels change. Keep inputs representative, consented, minimized, and free of live secrets.

Choose scorers

Prefer deterministic scorers where the contract permits:

• JsonExactMatchScorer for exact structured results;
• JSON rule scorers for required fields and ranges;
• token overlap for bounded lexical similarity;
• retrieval scorers for recall and ranking.

Use a model judge only for qualities that cannot be expressed deterministically. Version the rubric and judge model, require rationale, and calibrate it against human labels.

One aggregate score is not enough. Inspect per-case failures and pass rates by low-cardinality tags.

Compare with a baseline

Run the same dataset against a pinned baseline and the candidate. Regression policy can bound mean-score drop, pass-rate drop, and execution failures.

Absolute gates answer “is this good enough?” Relative gates answer “did this change make it worse?” Production releases usually need both.

Record candidate identity as the combination that affects behavior: application revision, prompt version, model ID, provider configuration, and retrieval index version.

CI and data safety

Evaluation reports intentionally store scores, safe failures, metrics, and Run correlation—not model outputs. Persist sensitive case data separately with stricter access and retention.

In CI:

1. run deterministic offline evaluations on every change;
2. shard larger datasets deterministically;
3. cap concurrency, cost, and total duration;
4. retry only infrastructure failures, never silently discard bad cases;
5. compare with the approved baseline and publish the report;
6. require review before replacing the baseline.

Live evaluation needs protected credentials and explicit budgets. A provider outage should appear as execution failure, not as a zero-quality model score.