I’d start at the on-policy section, where the student generates its own tokens and a stronger teacher scores them, so training targets mistakes the student actually makes. I like the arc from compression to behavior transfer because it makes clear how distillation’s goal has changed; the claim that distillation, supervised fine-tuning, RL, and synthetic data are becoming the same pipeline needs sharper boundaries than this short history gives.

I like this because it separates ordinary teacher-to-student compression from a newer pattern: same-size, domain-specialized RL checkpoints guide a student on its own rollouts. The contrast between token-level teacher feedback and RL’s single reward for an attempt makes the appeal clear, though the model-report examples don’t establish when distillation actually wins.

I like this because it makes the mismatch between training on “flawless teacher prefixes” and generating from the student’s own outputs the organizing idea, then sorts methods by objectives, signal sources, and training dynamics. The DAgger-style reduction from quadratic to linear compounding error is a good motivation, but I wouldn’t take it as proof that every white-box, black-box, and self-distillation setup gets that benefit; the survey’s account of conditions and failures like diversity collapse is what makes the comparison worth reading.

I’d start with the contrast between SFT and OPD: the student generates the trajectory, then the teacher gives token-level feedback on that same path. The walkthrough makes the idea of the “teacher’s distribution over the student’s trajectory” easier to follow, including a sampled-token, reverse-KL variant; I like that it distinguishes OPD from both imitation and sparse-reward RL, though it doesn’t show whether this approach works better in practice.

The strongest part is the ceiling argument: fixed teacher data can only carry SFT so far, while OPD samples from the student’s own states and still uses teacher token-level feedback. That makes the comparison more precise than “OPD beats RL”: OPD can reach teacher quality efficiently, while verifier-based RL can aim past it; the reported 9-30× compute gap is intriguing, though I’d want benchmark and teacher-call details before generalizing it.

If you’re comparing post-training methods, this framing is worth a read: SFT pulls toward a fixed dataset, while on-policy distillation matches a teacher on the student’s own samples. The surprising result is that students distilled from SFT and RL teachers performed similarly on minimal code editing and forgot less than the SFT teacher; I’d keep the implication that on-policy sampling matters more than teacher choice tentative, since it comes from one coding-task setup.

On-policy distillation trains a student using contexts sampled from its own evolving policy while learning from a stronger teacher, addressing the distribution mismatch that limits conventional offline supervised fine-tuning. The approach offers a compelling bridge between imitation learning and reinforcement learning, but its benefits depend heavily on objective design, teacher quality, and stable optimization.

• Student-generated states expose the teacher to the errors and failure modes the student actually encounters, reducing the train–inference mismatch of teacher-forced data.
• Distillation objectives must carefully manage token-level probability matching, sequence-level quality, and exploration; naïve KL optimization can over-constrain the student or propagate teacher biases.
• Because the student’s sampling distribution changes during training, evaluation, rollout quality, and optimization stability become central engineering concerns rather than implementation details.

Literature review outlining sequence-level distillation techniques across MiniLLM, GKD, and f-divergence generalizations.

• Analyzes teacher guidance strategies during autoregressive sampling.

A unified mathematical formulation connecting supervised fine-tuning (forward KL), RLHF (reverse KL with reward shaping), and on-policy distillation.

• Clarifies why on-policy student generations prevent mode collapse and exposure bias.