
1 / 3
【Overview】 EvalCore is snapshot testing for AI behavior. It records how an LLM app or agent responds, then replays those recordings in CI so every prompt, model, or dependency change is checked for regressions offline, deterministically, and at $0. The project is free and open source under Apache-2.0. The tool is built for teams that run LLM apps and agents and want eval results gated on every pull request, for engineers who need repeatable offline reruns without API spend, and for developers comparing models or watching for model drift over time. Instead of calling live endpoints during CI, EvalCore stores cacheable calls in local cassettes and replays them, so test runs do not depend on network access or API keys. 【Key features】 • Record / replay cassettes: Saves every call to a cacheable target in a local SQLite cassette keyed by a hash of the canonical request, then replays it in CI with no network and no API keys. • Scorers: Grades each case with contains, exact, regex, subprocess, trajectory, judge, json-schema, and similarity scorers. • Gates and baselines: Sets absolute suite-level floors such as pass_rate >= 0.95 or a minimum judge mean score, and can gate a run against a saved baseline so only regressions fail. • Matrix model comparison: Runs the whole suite once per named target and prints a side-by-side table with per-case winners and per-target cost. • Any-language targets: Evaluates anything that speaks HTTP or shell, including openai-compatible endpoints, Anthropic Messages, Gemini, plain HTTP APIs, and OTel/OpenInference traces. • HTML and JUnit reports: Writes a self-contained HTML report as one file with no assets or server, and emits JUnit XML for GitLab, Jenkins, and other CI systems. 【Use cases】 1. Gate pull requests on eval regressions by running with --cache replay --baseline main. 2. Compare models by running one suite across several targets with --matrix to see per-case winners and per-target cost. 3. Detect model drift with a scheduled nightly job using --cache live. 【Pros and cons】 Pros: - Teams that want an eval gate on every pull request get a deterministic check that does not need network access or API keys during CI. - Because recorded calls replay offline, engineers can iterate on prompts and rerun suites repeatedly without paying for model calls each time. - The baseline and matrix options fit developers who need to catch drift or compare models over time, not just pass a single fixed suite. Cons: - Shell targets are never cached, so those cases run local code whose behavior can change without the config changing. - A case passes only when every scorer passes, so one failing scorer blocks the case. - With --baseline, failures already accepted into the baseline are tolerated per case but can still sink a pass_rate gate. - A matrix run is mutually exclusive with target selection and baselines. - Streaming responses are unsupported. - JUnit output does not include gate outcomes. 【FAQ】 Q: What does EvalCore record and replay? A: It records every call to a cacheable target into a local SQLite cassette keyed on a hash of the canonical request, then replays it in CI with no network and no API keys. Q: Which targets can EvalCore evaluate? A: Anything that speaks HTTP or shell, including openai-compatible endpoints, Anthropic Messages, Gemini, plain HTTP APIs, and OTel/OpenInference traces. Streaming responses are unsupported. Q: How do gates and baselines work? A: You can set absolute suite-level floors such as pass_rate >= 0.95 or a minimum judge mean score, and gate a run against a saved baseline so only regressions fail.
free
WPS AI is an AI assistant built into WPS Office that helps users write documents, generate presentation outlines and…
Baidu Comate is an AI coding assistant built by Baidu on its ERNIE model. It goes beyond simple autocomplete: it…
UniConverter is an AI media conversion and enhancement tool for video, audio, and images. It combines everyday…
Added
10/22/2026
Views
618
Clicks
3
6 similar tools to consider instead of EvalCore