// Production / LLM Observability & Reliability
Custom LLM evals, in CI.
Off-the-shelf harnesses - Inspect AI, ragas, lm-eval-harness, DeepEval, promptfoo - get you to the starting line. We build the domain-specific layer on top, calibrate the judges, and wire the result into CI as a deploy gate.
// What we see
Public benchmarks pass. Real customers complain anyway.
01
MMLU going up tells you nothing about your product
The team upgrades the model, MMLU and HumanEval hold, the dashboard says ship. Two days later customer success lights up about tone, citation accuracy, or refusal rate.
02
There's no single right answer to compare against
How do you grade a customer-facing summary? The ground truth is a region of acceptable answers - exact-match metrics score a flat zero on outputs your senior reviewer would accept.
03
Structured outputs fail where public benchmarks don't look
Schema compliance under noisy inputs, field-level accuracy, completeness against your form, refusal calibration on edge cases - the metrics that matter for B2B products, and the ones no public leaderboard tracks.
// Case Study
Production LLM Processing at Surfer Scale
We helped Surfer handle massive content generation workloads with a reliable, cost-optimized LLM pipeline built for scale.
300B+
tokens processed
100k+
credits sold in 6 months
5 months
from concept to full product release
// What we do
Three layers, from off-the-shelf to deploy gate.
Reuse what's already good. Build the domain-specific layer that doesn't exist yet. Wire the result to your CI so a regression on a metric your customers care about blocks the merge.
Start from open harnesses, not from scratch
Off-the-shelf frameworks solved the runner, parallelism, and result schema. We pick the one that fits your stack and start there.
- Inspect AI, ragas, lm-eval-harness, DeepEval, promptfoo
- Langfuse / LangSmith / Phoenix for trace-based evals
- Weights & Biases or MLflow for experiment tracking
- Toolchain chosen for your stack, not vendor preference
Build the domain layer that doesn't exist yet
This is the engagement - golden sets from your production traces, rubrics that match how your senior reviewer actually grades.
- Golden sets stratified by tenant, intent, and tail behavior
- Win-rate evals with paired bootstrap and McNemar's test
- Rubric scoring with explicit criteria for structured outputs
- Calibrated LLM-as-judge with human-vs-judge agreement tracked over time
Wire it to CI as a deploy gate
An eval that runs once a quarter is documentation. The version your team trusts blocks a deploy when it fails.
- GitHub Actions / GitLab CI / Buildkite integration with PR comments
- Smoke (50–200) on PR, full (1–5K) nightly, deep (10K+) weekly
- Per-metric thresholds with explicit rollback criteria
- Sharded execution to keep PR runs under 15 minutes
// Method fit
You can't ship what you can't grade.
skip it if
It's a research prototype with no users
A notebook with twenty test cases is enough while iterating. Don't pay for CI integration before there's a CI to integrate with.
For anything that touches a real customer, treat the eval suite as load-bearing.
use it if
Almost everything else. If real users will judge your output, you need evals to judge it first.
Most agencies sell "AI integration" with a smoke test on five hand-picked prompts. That's how teams learn about regressions from customer-success tickets instead of from a failed CI run.
Off-the-shelf harness wired week one (Inspect AI / ragas / promptfoo). Domain golden sets from your traces by week three. Calibrated LLM-as-judge and CI gates by week six.
Without this layer, every prompt change is a coin flip and every model upgrade is a leap of faith.
// How we work
Reuse what works. Build what doesn't exist. Hand off the deploy gates.
Start from a working off-the-shelf harness within days. Build the domain layer over weeks. Hand off CI integration plus the runbook for adding cases and re-calibrating judges as the system evolves.
01
Stand up an off-the-shelf harness against your system
Inspect AI for capability suites, ragas for RAG, promptfoo for prompt diff, lm-eval-harness for academic baselines - connected to your model, prompts, and trace store. Working dashboard inside the first week.
02
Build the domain golden sets and rubrics
Production traces stratified by tenant, intent, and complaint signal - a few hundred to a few thousand examples labeled with your domain experts. Rubrics for structured outputs, citation accuracy, and tone.
03
Calibrate LLM-as-judge, wire to CI, hand off the runbook
100–500 human labels per metric to calibrate judges, agreement tracked over time. Tiered suites in CI - smoke on PR, full nightly, deep weekly. Runbook covers new cases and judge recalibration.
// Expert insight
“The first version of an eval suite is always wrong. The second version, after you've watched it disagree with your senior reviewer twenty times, is the one that earns the right to gate a deploy. The teams that get this don't think of evals as a checkbox - they treat the rubric as a living artifact, the same way they treat their tests. The teams that don't, ship a model upgrade because MMLU went up and find out from customer success three days later.”
Karol Gawron
Head of R&D @ bards.ai
// Why bards.ai
Has your eval suite ever blocked a deploy?
Your model vendor hands you leaderboard numbers. We build evals that grade what your customers care about - and tell you whether the upgrade ships or rolls back.
CLARIN-PL research lineage
Spun out of academic NLP. We come from a tradition where benchmarks are scrutinized, not blindly trusted.
1B+ tokens/day in production
We've built eval pipelines that gate real deploys for real customers - not toy notebooks against public benchmarks.
Statistics-first methodology
Bootstrap CIs, paired tests, power analysis, and McNemar's on every comparison. Eyeballing leaderboards is how teams ship regressions.
16+ open-source models on Hugging Face
80K+ monthly downloads. We've evaluated more checkpoints than most teams have prompts.
On-prem & air-gapped capable
Eval pipelines that run inside your perimeter on your data - including environments where outbound traffic is blocked.
Judge calibration is a practiced skill here
Everyone on your evals has built golden sets from production traces, calibrated LLM-as-judge against human labels, and gated real deploys.
// FAQ
Common questions about LLM evaluation frameworks
Nothing - until you ship. MMLU, HumanEval, GSM8K, MT-Bench, AlpacaEval are useful for cross-model triage and for catching capability regressions on a model upgrade. They're useless for telling you whether your contract analyzer extracted the right clause, whether your chatbot's tone matches your brand, or whether your RAG cited the right source. Public benchmarks are a sanity floor; domain evals are the ceiling that matters for your product.
Three approaches, often combined. Win-rate evals (your output vs. baseline, judged pairwise with paired bootstrap and McNemar's for significance) work for tone, persuasiveness, helpfulness. Rubric scoring breaks the answer into measurable criteria - citations present, schema compliant, no unsupported claims, on-brand voice - each scored independently. And reference-bag matching (acceptable answers as a set, not a single string) for cases where multiple specific outputs are correct.
Three layers. Calibration: judge-vs-human agreement on 100–500 labeled examples per metric. Bias correction: position swapping for pairwise, length normalization, self-preference penalties when the judge is also a candidate. Drift tracking: judge-vs-judge agreement across model versions, so when a vendor silently updates the model, we catch the metric shift before it corrupts a deploy decision.
Whenever the metric is genuinely subjective and your business cares - brand voice, persuasiveness, expert correctness in regulated domains, edge-case safety. Hybrid pipelines work best: human labels on the calibration set and ambiguous cases, LLM-as-judge for breadth. The split usually lands around 5% human, 95% automated, with disagreements flagged for review.
Tiered suites - fast smoke (50–200 examples) on every PR, full (1–5K) nightly, deep (10K+) weekly or pre-release. Aggressive judge-call caching. Stratified subsamples where statistical power allows. Smaller calibrated judges where they agree with the larger model. Eval bills typically land at 1–5% of inference spend.
Depends on traffic drift. Stable use cases (legal doc analysis, fixed-format extraction) - quarterly. Consumer-facing products with rapid intent drift - monthly. We set up data-drift monitors on production embeddings; when the distribution shifts past threshold, the system flags the golden set for review.
No. We build on top of existing harnesses where they fit (Inspect AI, lm-eval-harness, ragas, promptfoo) and extend them where they stop short. The framework, golden sets, rubrics, and judge configs are all your code in your repo. Swap the harness later and the data and rubrics carry over.
// Related services
Adjacent problems we solve
- Learn more
LLM Observability & Reliability
LLM Observability & Monitoring
Per-tenant cost attribution and streaming-aware latency (TTFT, inter-token) in the Grafana, Datadog, or Langfuse stack you already run.
- Learn more
LLM Observability & Reliability
LLM Regression Testing & Drift Detection
A prompt edit silently breaks 8% of traces. Diff-based paired evals and drift detection catch it before customer support does.
- Learn more
LLM Observability & Reliability
LLM Canary & Shadow Deployment
Shadow traffic, canary stages with statistical gates, and auto-rollback - a bad prompt change reverts in 30 seconds.
// Let's ship it
Build evals that earn the right to gate your deploys.
Send your application, failure modes, and the metrics your customers care about - you'll get an eval design and a number back, usually within a business day.
Karol Gawron
Head of R&D @ bards.ai