Run reproducible evaluations in CI

The CLI contract

runifold-eval keeps application-level process execution and CI policy out of runtime library dependencies. run executes a Candidate for every case; compare applies gates to existing reports; experiment measures repeated samples; merge reconstructs sharded evidence.

Exit status is stable: 0 passes, 1 means invalid configuration or artifact, and 2 means evaluation completed but a quality gate failed.

Dataset and candidate

Each JSONL line contains case ID, input, expected value, and tags. Dataset name and version form identity. Empty datasets, duplicate IDs, invalid tags, or mixed identities fail before execution.

The Candidate is launched directly without a shell and receives case ID, input, and tags—but never the reference answer. It must produce one bounded JSON object. Timeout, non-zero exit, oversized output, invalid JSON, unknown fields, or invalid metrics become isolated target failures.

Quality gates

Absolute gates require every selected scorer and case to meet its threshold. Relative gates compare mean score, pass rate, and execution success against a matching baseline. Missing scores or target failures fail closed.

Reports exclude raw prompts, inputs, references, Candidate output, and stderr. JSON is the canonical artifact; JUnit and Markdown are derived for CI and human review.

Experiments

One run cannot distinguish improvement from sampling noise. Experiments repeat cases with stable sample indices and derived seeds, then calculate mean, standard deviation, 95% Student's t confidence interval, and flaky-case rate.

Cache identity includes dataset content, Candidate version and arguments, scorer, seed, shard, timeout, and limits. Corrupt or contradictory cache entries fail explicitly. Deterministic case hashing enables independent shards and evidence-preserving merge.

CI design

Pin dataset and Candidate versions, build the Candidate before evaluation, use an absolute gate for minimum quality, add a baseline gate for regressions, and upload canonical reports even on gate failure. Use experiments for release decisions where model variance is material.

Do not make a flaky external Provider your only pull-request signal. Separate fast deterministic contract tests, offline quality gates, cassette integration, and opt-in live canaries.

Run a release gate

Install the CLI, build your Candidate, and keep reports even when the gate fails. The command after -- is executed directly, so pass each argument separately and do not depend on shell expansion.

cargo install runifold-eval-cli --version 0.9.0 --locked
cargo build --release --bin my-eval-candidate
runifold-eval run \
  --dataset evals/support.jsonl \
  --dataset-name support \
  --dataset-version 2026-08-06 \
  --candidate-version prompt-v2 \
  --output artifacts/evaluation.json \
  --junit artifacts/evaluation.xml \
  --markdown artifacts/evaluation.md \
  -- ./target/release/my-eval-candidate

In CI, distinguish exit 1 from exit 2: the first means the evidence itself is invalid, while the second is a valid result that failed policy. Upload the JSON, JUnit, and Markdown artifacts in both cases. Reproduce a failure locally with the exact dataset version and Candidate binary before changing a gate.