Regressionsyour dashboards don't show.

Power analysis sets the size, not folklore. For a primary metric where you need to detect a 2% regression with 80% power, a calibrated LLM-as-judge typically needs 1,500–3,000 paired examples. We start there, then add segments - at least 200–500 examples per critical segment (enterprise tier, regulated languages, high-stakes intents). Most production suites land between 2K and 10K examples, run as smoke (200–500) on PR and full nightly.

Three filters before anything pages a human. Statistical significance - bootstrap CIs and McNemar's where applicable, no flag without p-value. Semantic equivalence - embedding-based and rubric-based checks catch outputs that are textually different but mean the same thing. And cluster size - single-trace anomalies route to a low-priority review queue while clustered patterns trigger gates. Review-queue-to-real-regression ratio typically lands between 2:1 and 4:1.

Yes - that's one of the main use cases. We run continuous evaluation against a held-out canary set with fixed prompts and fixed input distribution. When the vendor updates, output distribution shifts, judge scores move, and our change-point detector flags the date and the metric. Combined with input/output embedding drift monitors, we typically catch silent updates within hours.

Hard gates for safety regressions (refusal, policy violation, prompt injection) and clear quality breaches (>5% regression with high statistical confidence). Human review for ambiguous cases - small regressions, mixed segment effects, novel failure modes. Split is typically 80/20 automation/human, and human decisions get logged as labels so the auto-classifier improves over time.

We integrate with whatever you use - Langfuse, PromptLayer, LaunchDarkly, custom Git-based - and tag every eval run with prompt version, model version, and config hash. When a regression appears in production, you can trace it back to the exact change that introduced it. If you don't have prompt versioning yet, we set it up as part of the engagement.

Diff-based eval on PRs and a baseline regression suite typically lands in 2–3 weeks. Behavior-cluster analysis adds another 2 weeks. Production change-point detection and human review queues - 6–8 weeks total for a production-grade setup. We work in increments so each milestone delivers usable capability before the deeper layers ship.

Agent regressions are harder to surface because a step-3 tool-call failure shows up only as a drop in end-task success rate - per-call quality scores look fine. We instrument at three levels: per-step (each LLM call and tool invocation gets its own trace and score), per-trajectory (common agent paths get golden-set comparison), and end-to-end (task completion rate as the headline metric). Behavior-cluster analysis groups failures by the step they originate at, so a reviewer sees 'step 3 tool-call parse failure: 23 cases' instead of 70 individual agent traces. Change-point detection runs on both per-step scores and end-task success rate so silent vendor updates that only manifest through cascaded agent failures still surface within hours.