Evals as code, with traces and regressions.

from aieval import dataset, run, scorer
from aieval.scorers import llm_judge, rouge_l


@scorer
def length_under_120_words(prediction: str, _expected: str) -> float:
    return 1.0 if len(prediction.split()) <= 120 else 0.0


faithful = llm_judge(
    rubric="Reward summaries faithful to the source that omit nothing important.",
    provider="sarmalink",
    model="smart",
    name="faithfulness",
)


if __name__ == "__main__":
    run(
        name="summarisation",
        dataset=dataset.jsonl("examples/summarisation/dataset.jsonl"),
        scorers=[rouge_l, length_under_120_words, faithful],
        provider="sarmalink",
        model="smart",
    )

from aieval import scorer

# Declare 'example', 'model', 'provider' or 'prompt' as keyword-only
# parameters and the runner fills them in via invoke_scorer.

@scorer
def grounded_in_prompt(
    prediction: str,
    _expected: str,
    *,
    prompt: str | None = None,
) -> float:
    return 1.0 if prompt and prompt.split()[0].lower() in prediction.lower() else 0.0

# The common two-argument case is unchanged.
# The extra parameters are supplied only when you declare them.

uv run aieval list                              # find run ids
uv run aieval diff <run_a> <run_b>              # per-scorer mean deltas + movers
uv run aieval pairwise <run_a> <run_b> \
    --confidence 0.95 --iterations 2000         # paired bootstrap, winner column
uv run aieval ci <run_id> --threshold 0.05      # exits non-zero on regression

from aieval import compare, pairwise
from aieval.backends import get_backend

backend = get_backend()

# Regression diff
report = compare(backend.get_results(run_a), backend.get_results(run_b))
for d in report.scorer_deltas:
    print(d.scorer, d.delta)

# Paired bootstrap with a 95% confidence interval
for r in pairwise(backend.get_results(run_a), backend.get_results(run_b)):
    print(r.scorer, r.mean_diff, (r.ci_low, r.ci_high), r.winner)
    # winner reads: 'a', 'b', or 'tie'

- run: uv run aieval run my_eval.py
- run: |
    RUN_ID=$(uv run python -c \
      "from aieval.backends import get_backend; print(get_backend().list_runs()[0]['id'])")
    uv run aieval ci "$RUN_ID" --threshold 0.05

IDE support, real debuggers, and no leaky abstraction between what you want to measure and how you express it. A scorer is any callable returning a float in 0.0 to 1.0. Scorers that need more signal declare example, model, provider, or prompt as keyword-only parameters and the runner fills them via invoke_scorer. This is what lets the LLM judge see the original request without changing the common two-argument case.

It resamples the per-example differences between two runs with replacement many times, recomputes the mean difference on each resample, and reports a percentile confidence interval. The winner column reads b when the interval sits above zero, a when it sits below zero, or tie when the interval straddles zero. The generator is seeded so results are reproducible.

Each run is tagged with the git SHA and the dataset content version. The dataset version is an order-independent SHA-256 over the canonical JSON of every row, so reordering rows does not invalidate a comparison, but editing a row does. Comparing runs over different datasets is possible but rarely meaningful, since examples no longer line up by index.

aieval ci compares the candidate run against a baseline and exits non-zero when any scorer mean drops by more than the threshold. By default the baseline is the previous run of the same name; pass --baseline <run_id> to pin it. If there is no baseline the gate passes, so the first run on a new eval never blocks.

No. SQLite and DuckDB are both file-based. Set AIEVAL_BACKEND and AIEVAL_DB_PATH to switch. SQLite is the default for portability; DuckDB is there when you want ad-hoc analytics on traces.

The grading model is prompted to answer with a single JSON object containing score and reason. The verdict is parsed strictly. A malformed verdict scores 0.0 rather than crashing the run, so a flaky judge does not bring the eval down.

Capture lives in aieval/core/telemetry.py behind an optional extra. A span() context manager yields a SpanHandle whose set() method records attributes once they are known. The captured attribute dict is available in both modes, so the same code path works whether or not OpenTelemetry is installed, and tests assert on captured attributes without standing up a collector.

It can, but the design lets you fix the grading provider and model independently of the model under evaluation. That means you can grade a cheap model with a stronger judge, which usually surfaces more meaningful signal than a model marking its own homework.