// Production / Inference Optimization

Distillation. Smaller models, same answers.

Teacher-student training that takes a 70B model and gives you a 7B that holds up on your evals - at a fraction of the cost. We tell you when distillation is the wrong tool.

Book a meeting See related case study

distill.py

def distill_loss(student, teacher, x, y):
  s_logits = student(x)
  with torch.no_grad():
    t_logits = teacher(x)

  kl = F.kl_div(
    F.log_softmax(s_logits / T, dim=-1),
    F.softmax(t_logits / T, dim=-1),
    reduction="batchmean",
  ) * (T * T)

  ce = F.cross_entropy(s_logits, y)
  return alpha * kl + (1 - alpha) * ce

# teacher: Llama-3-70B
# student: Llama-3.2-3B
# T=4.0, alpha=0.7

70B → 3B · 23x cheaper-2.1 pts on eval

// What we see

The frontier API bill is the symptom. The task is too narrow to justify it.

GPT-5 pricing on a classification step

Retrieval-rerankers, intent classifiers, domain extractors run thousands of times a day. Teams reach for the frontier model and pay $40K/month for a task a 3B fine-tune nails at $2K.

Latency the product team sold that the model can't hit

The product commits to sub-500ms; the 70B model takes 2 seconds on a loaded cluster. Swapping to a well-distilled 7B often closes that gap without touching the application.

Provider dependency baked into the unit economics

Every pricing change from OpenAI or Anthropic reprices the P&L, and every model update is a regression risk nobody tested. Teams discover this during a vendor negotiation.

// Case Study

Fine-tuned a small model to frontier quality - 50× cheaper at high volume

Customer's frontier-API entity-extraction pipeline worked but the per-token bill was eating margin at the volume they wanted to ship at. We split the task into hybrid retrieval + two fine-tuned Gemini 2.5 Flash Lite models. 98.3% F1 retention on the customer's existing eval suite, ~50× cheaper per 1000 requests, ~3× faster - without touching the prompt or the eval.

50×
cheaper per 1000 requests
3×
lower end-to-end latency
98.3%
F1 retention vs. frontier-API baseline

Read the case study

Fine-tuned a small model to frontier quality - 50× cheaper at high volume

// What we deliver

Distillation pipelines, end to end.

From teacher selection to deployed student, with the eval harness checked into your repo so you can repeat it next quarter.

Teacher and student selection

The teacher sets the ceiling, the student sets the floor - we profile both before training starts.

Open-weight teachers: Llama, Qwen, DeepSeek, Mistral
Closed-model distillation via API logs and synthetic generation
Student sized to your latency budget, not arbitrary parameter count
Architecture matching where it helps, divergence where it pays off

Distillation losses that work

KL divergence on logits is the textbook answer. Production wants attention transfer, hidden-state matching, and task-specific objectives.

Soft-target KL divergence with temperature tuning
MSE on logits for sequence-level transfer
Hidden state and attention matrix alignment
Combined CE + distillation losses with curriculum scheduling

Distillation dataset curation

We mine your traffic, generate from the teacher, and filter aggressively - teacher mistakes become student dogma.

Traffic mining and PII-safe replay from production
Teacher generation with self-consistency filtering
Difficulty-aware sampling so the student learns the hard cases
Domain coverage audits against your taxonomy

Eval harness and regression reports

Every distilled model goes through a custom eval suite tied to deploy gates.

Task-specific eval suites, not just MMLU and GSM8K
Per-segment regression analysis on your real traffic
Pairwise judge models for open-ended tasks
Cost-per-quality curves so the tradeoff is explicit

Distillation vs alternatives

Sometimes quantization or a sharper prompt with a smaller stock model wins. We tell you which lever to pull.

Quantization (AWQ, GPTQ, FP8) where the model is already small
Pruning and structured sparsity where compute is the bottleneck
LoRA and adapter routing instead of a full distillation
Honest recommendation when no compression is the answer

Production rollout and watch

Distilled models behave differently under real traffic. We instrument the rollout so regressions don't surface in customer tickets.

Shadow deployment alongside the teacher for direct comparison
Canary rollout gated on live eval metrics
Drift detection on input distribution and output quality
Easy rollback path with the teacher kept warm during ramp

// Method fit

Distillation earns its cost when the task is narrow and high-volume.

skip it if

Quantization gets you close enough
If the cost gap is modest, AWQ INT4 or FP8 quantization gets 30-50% savings at near-zero engineering cost. Run the benchmark first.
The task keeps shifting
Distillation locks in a snapshot. Pre-PMF, you'll retrain before the deployment pays back - ship on the frontier API, then distill what stabilizes.
Your teacher isn't reliable on the task
If the teacher hallucinates 10% on your domain, the student learns to hallucinate more efficiently. Fix the teacher first, then distill.
The payback period is longer than 3 months
Engagement cost vs. monthly inference savings: if ROI takes longer than 3 months, it rarely makes sense - models change too fast.

use it if

You have a high-volume, narrow task - extraction, classification, structured output, domain Q&A - where a frontier model is 5-20x more expensive than the task warrants.

Your teacher is consistently correct on that task (>90% eval score on your data) and you have enough production traffic to mine for training examples.

You want a reproducible training pipeline you can re-run as the task evolves. The pipeline is the deliverable; the first student is just the first run.

// How we work

Eval first. Teach on filtered data. Shadow before you swap.

The number we're moving is 'student quality within X points of teacher at Y cost.' Without a task-specific eval in place before training starts, distillation is guesswork dressed up as engineering.

Eval design + teacher profiling (week one)

Task-specific eval suite built from your production traces. We profile the teacher against it - per-segment accuracy, hallucination rate, tail failures. The teacher gap sets the student's quality bar.

Dataset construction + distillation run

Traffic mined with PII-safe replay, teacher generation with self-consistency filtering, reward-model scoring (20-40% kept). Losses chosen per task. Pilot on 10K examples validates the recipe before the full run.

Shadow deployment + handoff

Student deployed alongside the teacher on real traffic, canary ramp once shadow metrics hold. Handoff: training pipeline as code in your repo, eval suite in CI, and a retraining runbook.

// Expert insight

“The mistake we see is teams distilling against academic benchmarks and then deploying for a product that looks nothing like MMLU. The eval has to be your eval - built from your traffic, judged the way your customers judge - or the distilled model will pass the test and fail the launch.”

Karol Gawron

Head of R&D @ bards.ai

See our open-source work

// Why bards.ai

Researchers who train. Engineers who deploy.

Distillation is half ML research, half production engineering. We do both, and we don't hand off between them.

16+ open-source models on Hugging Face

80K+ monthly downloads. We've trained, distilled, and shipped models people actually use - not just papers about them.

CLARIN-PL spinoff, NLP research roots

We came out of one of Europe's strongest NLP research labs. Distillation losses and curriculum scheduling are not new territory for us.

Production training at scale

Multi-node training on H100s, FSDP, mixed precision, and the boring infrastructure that makes long runs not blow up overnight.

Eval harnesses we ship with

Every project includes a task-specific eval harness in your repo - so the next distillation, fine-tune, or prompt change is measurable too.

On-prem & air-gapped capable

We can train on your hardware, with your data, behind your firewall - including environments where outbound network is blocked.

Every engineer has run the full loop

Dataset curation, multi-node training runs, eval harnesses, shadow rollout - the people scoping your distillation have done each step on production models before.

// FAQ

Common questions about model distillation

On task-specific evals, a well-distilled student typically lands within 1-3 percentage points of a teacher 10x its size - sometimes closer when the task is narrow. On open-ended general evals the gap is larger. We measure this on your evals before promising anything, and we'll walk away if the regression is unacceptable.

They solve different problems. Quantization is cheapest when your model is already small enough - drop to FP8 or AWQ INT4 and ship. Pruning helps when compute, not memory, is the bottleneck. Distillation is the right choice when you need to drop one or two orders of magnitude in size and quantizing the big model isn't enough. Often the answer is a combination.

Yes - via API output distillation. We replay or generate prompts, capture teacher outputs, and train an open-weight student on the resulting dataset. Quality depends heavily on dataset coverage and filtering. Check your provider's terms of service first; some prohibit using outputs to train competing models.

Less than you'd think for narrow tasks - tens of thousands of high-quality teacher-labeled examples can be enough for a focused student. Broader behavior transfer wants hundreds of thousands to millions of examples, often generated and filtered. We typically combine real traffic, teacher synthetic generation, and curated public data.

We build a task-specific eval suite from your traffic and your acceptance criteria - not just MMLU and HellaSwag. For open-ended outputs we use pairwise judge models, often a separate strong LLM, validated against human ratings on a sample. The harness is checked into your repo so you can re-run it next quarter when something changes.

End-to-end distillation projects typically run 4 to 8 weeks. The first two weeks are dataset construction and eval harness; the next two to three are training and iteration; the rest is rollout and monitoring. We work in weekly increments with a measurable artifact at the end of each week.

Yes - we routinely train on customer infrastructure, including on-prem H100 clusters and air-gapped environments. We bring the training stack (FSDP, mixed precision, eval harness, checkpointing) and operate it on your boxes. Your data and your weights never leave your perimeter.

// Related services

Adjacent problems we solve

// Let's ship it

Cut your inference bill without cutting your quality.

A distillation plan, an eval design, and a number - usually within a business day. That's what comes back once you share your teacher, task, and latency budget.

Book a meeting hello@bards.ai