// Research / Custom Fine-tuning
Synthetic data that survives eval.
Teacher distillation from Llama-405B, Qwen-72B, or DeepSeek-V3. Self-instruct, evol-instruct, persona-based generation. Then MinHash deduplication, reward-model filtering, n-gram and embedding contamination checks against your evals. The pipeline that throws 80% away is the deliverable.
// What we see
Synthetic data is mostly a filtering problem, not a generation problem.
01
Teams generate 1M and train on all of it
Raw synthetic output is mostly noise - teacher hallucinations, repetitive prompts, format breaks, off-task responses. The kept fraction after filtering - usually 10–25% of raw - is what trains the model.
02
Single-teacher monoculture
Distill only from GPT-5 or only from Claude and the student inherits that teacher's quirks - refusal patterns, hallucination shapes, style tics. Multi-teacher ensembling and persona-based generation widen the distribution.
03
Contamination silently inflates eval scores
Teachers regurgitate benchmark questions verbatim. Without n-gram and embedding-space contamination checks, your benchmark numbers are lies - we've seen 5–20% leakage on uncurated synthetic mixes against MMLU and HumanEval.
// Case Study
GDPR-compliant PII redaction - 24 EU languages, drop-in
Every GDPR-bound team we worked with was hand-rolling their own preprocessing - regex stacks, per-language NER, hours of pipeline work per dataset, and coverage that stopped at the languages they had labelers for. We shipped `bardsai/eu-pii-anonimization-multilang` as the drop-in library that replaces all of it: one import covers 24 EU languages with a GDPR-aware tagging schema, recall-tuned because false negatives leak. 0.890 F2 on the Gretel benchmark; hours of pipeline work collapse to seconds per dataset.
0.890
F2 on Gretel PII benchmark
24
EU languages with one model
Hours → s
preprocessing per dataset
// What we do
Three layers, from generation to provenance.
Generate broadly, filter aggressively, document everything. The deliverable is a reproducible pipeline you can re-run as the task evolves - not a one-off dataset handed back as a tarball.
Generation - distillation, self-instruct, evol-instruct
Open-weight frontier teachers for license clarity; closed models when ToS permits and quality justifies.
- Teacher distillation: Llama-405B, Qwen-72B, DeepSeek-V3, Mixtral-8x22B
- Self-Instruct + Evol-Instruct depth/breadth mutations for difficulty curriculum
- Persona-conditioned generation for diverse voice and style coverage
- Multi-teacher ensembling to dilute single-model quirks
Filtering - reward scoring, dedup, format gates
Aggressive multi-stage filtering - the pipeline that throws 75–90% of raw output away.
- Reward-model scoring with quality thresholds calibrated against held-out humans
- MinHash / SimHash dedup on prompts and responses (typical: Jaccard < 0.8)
- Length, format, language, and structural gates with explicit removal logging
- LLM-as-judge with bias correction (position swap, length norm)
Contamination + license-aware provenance
Contamination checks against every eval set you care about, plus per-record license provenance.
- 13-gram overlap checks vs MMLU, HumanEval, GSM8K, MATH, custom evals
- Embedding-space proximity for paraphrase contamination (cosine threshold tuned)
- Per-eval-set decontamination with documented removal counts
- Teacher-license tracking (Llama, Qwen, Mistral CLAs) - audit-ready provenance
// Method fit
Synthetic data fits when labels don't exist or won't scale.
skip it if
You have plentiful, high-quality real labels
If your team can produce 50K+ well-labeled examples at acceptable cost, real data wins on the long tail of edge cases.
License terms forbid teacher-derived training
OpenAI, Anthropic, and Google ToS restrict training on their outputs. Open-weight teachers - Llama, Qwen, Mistral, DeepSeek - sidestep this; we default to them.
Your task is high-stakes safety-critical
Medical, legal, financial advice - where teacher hallucinations could land you in a deposition. The long tail needs human-labeled data and judgment.
You don't have a base model to train
Synthesizing data without a downstream training plan is just generating tokens. We do both halves - this is the data layer.
Supervised Fine Tuning (SFT)
use it if
You want to train a smaller specialist (7B-13B) on a task where a frontier model already does it well - distill the capability down at 1/100th the inference cost.
You have a narrow task with no labeled data and labeling would take quarters. Self-instruct + evol-instruct + filter pipelines can bootstrap a 50K-example training set in days.
You want a reproducible synthesis pipeline as a deliverable, so your team can re-generate and re-filter as the task drifts.
// How we work
Seed broadly. Filter ruthlessly. Document everything.
Generate more than you'll keep. Filter to the fraction that survives quality and contamination checks. Track provenance per record so license audits are a non-event.
01
Seed prompt design + teacher selection
Seed prompts mined from user logs, support tickets, or domain experts. Teacher choice driven by license clarity and pilot quality. Mixture strategy decided up front - pure synthetic vs. 80/20 synthetic-real.
02
Generate, filter, contamination-check
Self-Instruct + Evol-Instruct + persona conditioning to widen the distribution. Reward-model scoring, MinHash dedup at Jaccard < 0.8, n-gram and embedding contamination checks with documented removal rates.
03
Pilot SFT + handoff
Pilot SFT on a 10K subset checks eval lift before the full run. Handoff as code: seed prompts, teacher configs, filtering thresholds, contamination scripts, provenance manifest.
// Expert insight
“Teams generate a million synthetic examples and train on all of them. We generate ten million, throw eight million away, and train on the remaining two - and it almost always wins. Synthetic data is a filtering problem, not a generation problem.”
Karol Gawron
Head of R&D @ bards.ai
// Why bards.ai
We've trained models from scratch on synthetic data.
Our open-source Polish-language models on Hugging Face - 80K+ monthly downloads - were trained partly on synthetic data we built and curated. We know what survives contact with reality.
16+ open-source models on Hugging Face
Including models trained on substantial synthetic-data components. We've shipped, served, and learned from each one.
Filtering infrastructure that scales
MinHash on tens of millions of documents, embedding-based dedup, reward-model scoring at scale. The throw-away ratio is the deliverable.
10+ peer-reviewed publications
CLARIN-PL spinoff. We've published on data quality, instruction tuning, and Polish NLP - synthetic data sits at the intersection.
Contamination paranoia, by default
13-gram + embedding overlap checks against every eval set. Held-out fresh splits. Documented removal rates.
License-aware provenance
Per-record license tracking, teacher ToS verified before generation. Every record's source documented for audits.
Mixture studies, not folklore
Pure synthetic vs 80/20 vs 50/50 vs curriculum mixing - we run the ablations to find your sweet spot.
// FAQ
Common questions about synthetic data pipelines
For a focused task on a strong base model, 10K–50K well-filtered examples is often the right zone. Past 100K you're paying for marginal gains unless the task is broad. The kept fraction matters more than raw generation volume - generating 1M and keeping 50K after filtering routinely beats generating 100K and keeping 80K.
Yes - it's a real risk and we treat it explicitly. Distillation transfers both the teacher's strengths and its quirks (hallucination patterns, refusal behaviors, factual errors). We use multi-teacher ensembling, reward-model filtering, and red-team prompts to catch the worst inheritances. For domain-critical tasks we mix in real data to anchor the student against teacher drift.
13-gram overlap between generated prompts/responses and the eval sets we care about (MMLU, HumanEval, GSM8K, MATH, your custom evals), plus embedding-space proximity for paraphrase contamination. Anything that flags gets removed and we document removal rates per benchmark. We also keep held-out 'fresh' eval splits the synthesis pipeline never sees, so leakage shows up in the gap.
Depends on the source. OpenAI, Anthropic, and Google ToS historically restrict using their outputs to train competing models - that's a legal call your counsel needs to make. Open-weight teachers (Llama, Qwen, Mistral, DeepSeek) come with permissive licenses for downstream training. We document provenance for every record so you can answer license audits without drama, and we default to open-weight teachers unless there's a specific reason not to.
For domains where the teacher model is weak or wrong (specialized medical, legal, niche technical), for high-stakes safety-critical evaluations, and for the final preference layer where human judgment is the ground truth. Synthetic data is excellent for breadth and coverage; human labels are still better for the long tail of edge cases and for calibration of reward models.
Reward-model score distributions, length and format statistics, diversity metrics (n-gram entropy, embedding clustering), and small pilot SFT runs on subsets to read the loss curve. We don't train on a million examples without first validating the recipe on 10K and seeing that the pilot model improves on a held-out eval. The pilot saves the engagement when the recipe is wrong.
A pipeline, ideally. Reproducible code, dataset versioning, hash-tracked configs, provenance manifest, W&B / MLflow runs. Your team can re-generate and re-filter as the task evolves - and the synthesis pipeline is usually where most of the IP lives. A static dataset goes stale; a pipeline doesn't.
// Related services
Adjacent problems we solve
- Learn more
Custom Fine-tuning
Supervised Fine Tuning (SFT)
Reproducible fine-tuning with correct chat templates and eval gates - a QLoRA 7B on 50K examples lands in 12-24h and $30-60 of compute.
- Learn more
Custom Fine-tuning
Preference Optimization (DPO / KTO / GRPO)
DPO, KTO, or GRPO chosen by your data shape - for the quality bar SFT alone can't reach.
- Learn more
Custom Fine-tuning
Reinforcement Learning for Agents
PPO with self-play leagues on JAX - thousands of parallel environments per GPU for games and multi-agent decision problems.
// Let's generate it
Bootstrap the dataset, with the filtering that makes it work.
A synthesis recipe, a teacher recommendation, and a number back within a business day. All we need: the task, the base model, and the data you have.
Karol Gawron
Head of R&D @ bards.ai