The chapter derives the policy gradient from trajectory probabilities before connecting it to prompt-and-completion training, which makes the log-derivative trick feel like a working tool rather than a formula to memorize. I also like the clarification that many RLHF implementations treat a whole completion as one action; it helps explain why the chapter sets the discount factor to 1.

Interesting course. i have been enjoying their setup. example - Averaging fine-tuning runs can improve accuracy without increasing inference time, while averaging independently trained models can produce a nonsensical model.

I’d start with the learning-rate section: LoRA typically needs about 10x the full fine-tuning rate, so carrying over the old rate can make it look worse than it is. I like the distinction between RL, where the primer says small ranks match full fine-tuning, and large supervised datasets, where LoRA can lose ground; I’d treat the token-count rule as a rough sizing heuristic, not a guarantee.

I’m cautious about generalizing the Dimuon recommendation, but the page gives a clear reason to test it: in Tinker’s SFT runs, it reached lower loss in fewer steps at similar step time, while RL efficiency was similar to AdamW. The roughly 15× learning-rate ratio for SFT versus 60× for RL makes clear that switching optimizers also means retuning; I’d treat those ratios as starting points and validate on my own evaluations.

I’m not sure this helps choose between objectives, since it explains data layout rather than comparing losses. I like the shift-by-one walkthrough because it makes the masking distinction explicit: SFT zeros prompt weights, while the RL example zero-pads advantages to exclude prompt positions.

If you’re trying to tell whether an agent improves training data or just changes the training recipe, CurateBench makes that boundary unusually clear: the optimizer, recipe, and evaluations are fixed, leaving data as the only intervention. DataSmith beats Claude Code by 5.1 percentage points on average across 12 matched comparisons, but this is Datology’s benchmark on three base models and four evaluations, so I’d treat the result as evidence for this setup rather than a general harness ranking.

A 1B SPT model outperforming a 3B standard-pretrained model on domains far from web text is the standout result; mixing domain tokens into pretraining seems to reduce the representational jump left to finetuning. I like that they also fit overfitting scaling laws to choose the mix for a compute budget, though evidence across chemistry, music, and proofs leaves me cautious about treating that recipe as settled.

I like the case for using class tokens instead of adding a classification head: it keeps the model on the standard fine-tuning and inference path. The interesting claim is that fine-tuning can make renormalizing over just the class tokens unnecessary, even though the softmax covers the full vocabulary; tokenization can still distort the mapping, so I’d keep calibration separate from simply predicting the right label.

The strongest part is the ceiling argument: fixed teacher data can only carry SFT so far, while OPD samples from the student’s own states and still uses teacher token-level feedback. That makes the comparison more precise than “OPD beats RL”: OPD can reach teacher quality efficiently, while verifier-based RL can aim past it; the reported 9-30× compute gap is intriguing, though I’d want benchmark and teacher-call details before generalizing it.

If you’re comparing post-training methods, this framing is worth a read: SFT pulls toward a fixed dataset, while on-policy distillation matches a teacher on the student’s own samples. The surprising result is that students distilled from SFT and RL teachers performed similarly on minimal code editing and forgot less than the SFT teacher; I’d keep the implication that on-policy sampling matters more than teacher choice tentative, since it comes from one coding-task setup.

Cameron Wolfe’s analysis on applying reinforcement learning for ongoing post-training adaptation without catastrophic forgetting.

• Explores replay buffers, KL regularizers, and distribution shift mitigation.