// Research / Custom Fine-tuning
Self-play RL for game-theoretic agents.
PPO + self-play + league play on Jaxpot, our open-source JAX-native RL framework. Thousands of parallel games per GPU, snapshot leagues, Hydra-configured experiments. Built for board games, hidden-information games, and multi-agent decision problems.
// What we see
When demonstrations don't exist, self-play is the only option that scales.
01
Imitation hits a ceiling on game-theoretic problems
SFT against expert play caps at the expert's level - for adversarial games the expert pool barely exists. Self-play sidesteps the data problem: the agent generates its own training data.
02
Generic RL frameworks aren't tuned for self-play throughput
RLlib and Stable-Baselines3 weren't designed around JAX rollouts and league management. Jaxpot runs thousands of parallel games per GPU on pgx environments - Dark Hex 7×7 trains overnight on one H100.
03
Reward shaping + checkpoint discipline is the actual engagement
PPO works. The harder problem: what to score, when to archive, when to swap opponents, proving genuine improvement. ELO ladders, scripted baselines, and held-out adversarial probes are the discipline.
// What we do
Three layers, all on top of Jaxpot.
Self-play training on our open-source JAX-native RL framework. PPO + league play + vectorized environments. Open code, reproducible Hydra configs, your engineering team can audit and extend the same stack we ship.
Self-play training loops
PPO with GAE, vectorized JAX rollouts, snapshot opponents so the policy can't collapse against its current self.
- PPO + self-play + GAE on the Jaxpot training loop
- Vectorized pgx environments - 1000s of parallel games per GPU
- ResNet (b6/c128 typical) or MLP backbones, configurable per env
- Snapshot opponent archive with configurable archive cadence
League play + multi-agent training
Multiple distinct policies training against each other concurrently - for when single-policy self-play collapses to a narrow strategy.
- League management with configurable opponent sampling strategies
- Concurrent training of N policies with cross-policy rollouts
- Population-based selection for non-transitive game structure
- Automatic baseline eval (random, scripted, prior checkpoints)
Evaluation, ELO, reproducibility
Self-play loss curves lie - ELO ladders, scripted baselines, and held-out probes verify genuine improvement.
- ELO / TrueSkill ladders against frozen checkpoint pools
- Win-rate vs random + scripted + archived adversaries
- Held-out probes for non-transitive overfitting
- Hydra config + W&B / TensorBoard / local - full reproducibility
// Method fit
RL for agents fits when the problem is game-theoretic.
skip it if
You want to RL fine-tune an LLM
RL fine-tuning of LLMs - DPO, KTO, GRPO, RLHF - is a different engagement with different reward signals, infra, and failure modes.
Preference Optimization (DPO / KTO / GRPO)Your problem isn't multi-agent or game-theoretic
Single-agent control, prediction, classification - usually better solved with supervised learning, imitation, or offline RL. Self-play needs an opponent.
You don't have a simulator (or can't afford to build one)
Self-play needs a fast environment - real-world data collection at RL rates is impractical. Without a simulator, the engagement is simulator-building first.
You're at the prototype stage with no clear target
Self-play needs a measurable target - beat the best agent, hit ELO X. Exploration is cheaper in a Colab on toy environments.
use it if
Your problem is game-theoretic, multi-agent, or has adversarial dynamics - board games, hidden-info card games, debate / negotiation, market making against other agents, robotic policies trained in self-play simulation.
You have a simulator or are willing to build one - pgx-compatible board games work out of the box on Jaxpot; custom environments need to expose a `step / reset / observation` interface.
You want an open-source stack you can audit and extend - Jaxpot is permissively licensed and the code is yours to read.
// How we work
Environment first. Self-play next. League play and eval to close it out.
Every engagement starts with the environment and reward - the two things that decide whether self-play training will produce something useful. Then a baseline PPO + self-play run on Jaxpot. Then league play and evaluation discipline before handoff.
01
Environment, reward, and a working baseline
Wrap (or build) the environment in pgx-compatible form. Sparse terminal reward or shaped intermediate signals; adversarial probes hand-crafted before training. Scripted and random baseline opponents from day one.
02
PPO + self-play on Jaxpot
Hydra-configured experiment, JAX rollouts (1000s of parallel envs per GPU), ResNet/MLP backbone chosen per environment. Snapshot opponent archive so the policy plays past selves. W&B + TensorBoard logging from iteration one.
03
League play, evaluation, handoff
League play if single-policy self-play collapses. ELO / TrueSkill ladder against archived checkpoints; win-rate vs scripted and random baselines as a sanity floor. Handoff: Jaxpot config, checkpoints, W&B project, runbook.
// Expert insight
“Every successful self-play run we've shipped follows the same curve. First few hours: nothing. The reward is bouncing around the floor and the win-rate vs scripted baselines is at random. Then a few rollouts produce decent strategies - purely by chance - and the policy learns from them. Quality compounds exponentially. The unintuitive part isn't the algorithm. It's keeping the team's nerve through the warmup and trusting that exponentials start invisibly.”
Karol Gawron
Head of R&D @ bards.ai
// Why bards.ai
Want to audit the stack first? Jaxpot is on GitHub.
Self-play at production quality needs vectorized JAX environments, league management, and eval discipline. We built Jaxpot because existing tools didn't ship that combination - and we ship it open.
Jaxpot - our open-source RL framework
JAX-native, pgx-compatible environments, PPO + self-play + league play, Hydra configs, W&B/TensorBoard logging. Permissive license, public at github.com/bards-ai/Jaxpot.
JAX-native vectorized environments
1000s of parallel games per GPU via pgx, multi-GPU out of the box - the throughput that decides days vs weeks.
League play + population-based training
Concurrent training of N policies, configurable opponent sampling (latest, random archived, ELO-weighted), automatic baseline eval. Prevents self-play collapse on non-transitive games.
ELO + held-out probe evaluation
ELO ladders against frozen checkpoint pools, win-rate vs scripted opponents, and held-out adversarial probes catch circular self-play before it ships.
10+ peer-reviewed publications
CLARIN-PL spinoff. RL papers reviewed in NeurIPS / ICML cycles; the AlphaZero family and modern PPO variants reproduced.
We tell you when self-play is the wrong tool
Many decision-making problems aren't game-theoretic. Imitation learning, behavioral cloning, or offline RL fits better. We say so before you spend GPU budget.
// FAQ
Common questions about RL for agents
RLlib and SB3 are general-purpose - they support a long list of algorithms but their abstractions weren't designed around vectorized JAX rollouts and league management for self-play. Jaxpot is narrower (PPO + self-play + league play) but optimized for that combination: pgx-compatible vectorized envs, JAX throughput, snapshot archives, ELO eval as first-class. Spinning Up is for learning, not for shipping. If you want to extend RLlib to do what Jaxpot does, you can - we just shipped the result.
Anything implementing the `pgx.core.Env` interface - Go (9×9 demonstrated), Dark Hex (7×7 example), Tic-Tac-Toe (the tutorial), and the rest of pgx's library (chess, shogi, backgammon, various card games). Custom environments need to expose roughly a `step / reset / observation` interface - straightforward to add for any deterministic or stochastic-but-simulable game environment. For complex non-game environments (robotic sim, market sim), wrapping is part of the engagement.
Tic-tac-toe converges in minutes on CPU with ~2,000 PPO iterations - fastest sanity check we use. Dark Hex 7×7 trains overnight on a single H100 to a checkpoint that beats every scripted baseline. Go 9×9 is in the 1–3 day range on multi-GPU depending on the depth target. Production-quality strategy in larger games (Hex 11×11, complex multi-agent) takes serious clusters and 1–2 weeks of training time. The bottleneck is rollout throughput; vectorized JAX makes that the lever that matters.
Self-play loss curves are not a reliable signal - the agent can train for a week and end up at a strategy that only beats its current self. We track three things continuously. ELO / TrueSkill against a frozen pool of past checkpoints. Win-rate vs scripted baselines (random opponent, simple heuristics, prior production agent if applicable). And held-out adversarial probes - hand-crafted situations where a healthy agent should make a specific decision. Self-play that looks like it's improving but only against itself is the most common failure mode; these checks catch it.
For two-player zero-sum games with full information (Hex, Go, chess), the math says it should converge to a Nash equilibrium given enough time and the right algorithm. In practice, modern PPO + self-play + league play gets within striking distance of that for most game sizes the engagement is realistic for. Non-transitive games (rock-paper-scissors-shaped dynamics) need population training to avoid cycles. Hidden-information games (Dark Hex, poker variants) need careful handling of belief states - the engagement is more involved but the patterns are well-studied.
Yes - most engagements involve some custom environment work. Pgx-compatible games are the easiest path; for non-game environments (robotic sim, market sim, scheduling, supply-chain) we adapt the rollout loop and reward hooks. The constraint is environment speed: if a single env step takes 100ms and the network forward is 1ms, you're rollout-bound and we'll spend time speeding the env up before we touch the algorithm.
Jaxpot is open-source under a permissive license - free to use, audit, fork. The engagement is everything around it: environment design, reward shaping, training infra (multi-GPU, league management, eval discipline), debugging the runs that didn't converge, and handing off a reproducible Hydra config + checkpoint + runbook so your team can iterate after we leave. The code is the same code we ship in OSS; the engineering judgment around it is what you're paying for.
// Related services
Adjacent problems we solve
- Learn more
Custom Fine-tuning
Supervised Fine Tuning (SFT)
Reproducible fine-tuning with correct chat templates and eval gates - a QLoRA 7B on 50K examples lands in 12-24h and $30-60 of compute.
- Learn more
Custom Fine-tuning
Synthetic Data Pipelines
Teacher distillation plus the part that matters: dedup, reward-model filtering, and contamination checks that discard the 80% of synthetic data that would hurt you.
- Learn more
Custom Fine-tuning
Preference Optimization (DPO / KTO / GRPO)
DPO, KTO, or GRPO chosen by your data shape - for the quality bar SFT alone can't reach.
// Let's run it
Ship a self-play agent that actually wins.
What's the environment, the reward signal, and the target you'd accept? Answer those and we'll come back with an algorithm choice and a number within a business day.
Karol Gawron
Head of R&D @ bards.ai