DPO and GRPO for last-mile quality.

GRPO (DeepSeek-R1) drops the value model and uses group-relative advantages - dramatically less memory, often ~50% reduction vs PPO, and runs on a single GPU with vLLM colocated rollouts. It's mathematically equivalent to RLOO up to a scaling constant. The original GRPO has known issues at scale (entropy collapse, length bias, zero-gradient batches) that DAPO and Dr.GRPO patch. We default to DAPO-style clip-higher + dynamic sampling + token-level loss; the tradeoff is more knobs but a more stable curve.

DAPO (ByteDance, Mar 2025) hits AIME 2024 = 50 in half the steps of DeepSeek-R1-Zero-Qwen-32B by adding clip-higher (asymmetric ε to keep low-prob exploration tokens alive), dynamic sampling (drop prompts where every rollout agrees - zero gradient), token-level loss instead of sequence-level (fixes length bias), and overlong reward shaping (don't penalize truncated answers). Dr.GRPO removes the std-dev normalization in the advantage and the per-sequence length normalization - both were sources of bias. Plain GRPO often shows response length growing after rewards plateau; the patches stop that.

Three layers. (1) Multiple complementary rewards - a single reward is easy to game; three weighted rewards aren't. (2) Adversarial probes designed up front - we hand-craft outputs that score high but read terrible; if the model produces those, the reward function is the bug. (3) Track length, entropy, training reward, and held-out validation simultaneously per Cameron Wolfe's playbook. KL regularization helps for some setups; for verifiable-reward GRPO runs (DAPO and follow-ups), most groups now drop the KL penalty entirely - it isn't necessary and can hurt. The 2025 Anthropic paper on natural emergent misalignment from reward hacking is required reading; we treat reward hacking as a safety surface, not just a quality bug.

Less than people assume. Argilla/HF shipped Zephyr-141B with 7k pairs via ORPO. SmolLM3 used DPO/APO with HelpSteer-style mixes. At the larger end, Tülu 3 used 337–360k preference pairs for the 70B/405B mixes - and the key finding was that *unique prompts* matter; duplicating prompts with different responses doesn't help. Start small, evaluate on held-out preferences, scale up only when the eval shows the data is the bottleneck. Human preferences run $5–20 per sample; calibrated AI preferences (RLAIF) run under $0.01.

LoRA, in most cases, more aggressively than people assume. Thinking Machines published recent work showing LoRA matches full fine-tuning even at rank-1 for RL - because policy gradients carry only ~1 bit per episode of information vs ~1000× more for SFT tokens. Optimal LoRA learning rate is roughly 10× the full-FT rate. LoRA underperforms when (a) the dataset is large enough to look like pre-training, (b) batch size is huge, or (c) it's not applied to all linear layers (especially MLPs and MoE). Lewis Tunstall's framing: "LoRA forgets less, which is why DPO with LoRA often works really well - it acts as a regularizer."

SFT bootstrap on 8B with ~100K examples on 8×H100: ~20h, ~$620. DPO on the same scale: single H100, 24-48h, often under $50. GRPO depends entirely on rollout cost - multi-day on 8×H100 is normal for production-quality runs, with vLLM/SGLang as the rollout engine being the lever that matters. Engagement timeline: eval + reward function design week one, SFT bootstrap week two, DPO or GRPO iteration weeks three through six, total 6–10 weeks for production-grade engagements with multiple reward components and tight quality bars. Every week ships a measurable improvement, not a status update.