I’m not sure these benchmark gains transfer beyond the tested protocols, but the ACOS result makes the training-evaluation mismatch hard to ignore: matching the evaluator’s hundreds of category-sentiment questions and 0.56% positive rate took the score from 27.2 to 61.6, while oversampling only reached 30.1. I also like the distillation result: the one-pass student scored 62.47 against 64.81 for the reasoning model, then v1.1 passed it at more than 100 times lower mean latency. Their attempts with pause tokens and layer reuse didn’t beat the plain approach, which makes the case for soft targets from teachers that reason more interesting.

What stands out to me is that the teacher can guide the student along its own attempt with dense token-level predictions, without writing a replacement answer; that turns feedback into a richer training signal than a pass-or-fail reward. The reported OPSD comparison is worth a closer look because it gives both the 8-12× token-efficiency result and the setup behind it, including 1,024-token student responses versus 16,384 for GRPO; the broader case for preserving capabilities in continual learning still seems unsettled.

I’d start at the on-policy section, where the student generates its own tokens and a stronger teacher scores them, so training targets mistakes the student actually makes. I like the arc from compression to behavior transfer because it makes clear how distillation’s goal has changed; the claim that distillation, supervised fine-tuning, RL, and synthetic data are becoming the same pipeline needs sharper boundaries than this short history gives.

I like this because it separates ordinary teacher-to-student compression from a newer pattern: same-size, domain-specialized RL checkpoints guide a student on its own rollouts. The contrast between token-level teacher feedback and RL’s single reward for an attempt makes the appeal clear, though the model-report examples don’t establish when distillation actually wins.

I like this because it makes the mismatch between training on “flawless teacher prefixes” and generating from the student’s own outputs the organizing idea, then sorts methods by objectives, signal sources, and training dynamics. The DAgger-style reduction from quadratic to linear compounding error is a good motivation, but I wouldn’t take it as proof that every white-box, black-box, and self-distillation setup gets that benefit; the survey’s account of conditions and failures like diversity collapse is what makes the comparison worth reading.

I’d start with the contrast between SFT and OPD: the student generates the trajectory, then the teacher gives token-level feedback on that same path. The walkthrough makes the idea of the “teacher’s distribution over the student’s trajectory” easier to follow, including a sampled-token, reverse-KL variant; I like that it distinguishes OPD from both imitation and sparse-reward RL, though it doesn’t show whether this approach works better in practice.

The strongest part is the ceiling argument: fixed teacher data can only carry SFT so far, while OPD samples from the student’s own states and still uses teacher token-level feedback. That makes the comparison more precise than “OPD beats RL”: OPD can reach teacher quality efficiently, while verifier-based RL can aim past it; the reported 9-30× compute gap is intriguing, though I’d want benchmark and teacher-call details before generalizing it.

If you’re comparing post-training methods, this framing is worth a read: SFT pulls toward a fixed dataset, while on-policy distillation matches a teacher on the student’s own samples. The surprising result is that students distilled from SFT and RL teachers performed similarly on minimal code editing and forgot less than the SFT teacher; I’d keep the implication that on-policy sampling matters more than teacher choice tentative, since it comes from one coding-task setup.

On-policy distillation trains a student using contexts sampled from its own evolving policy while learning from a stronger teacher, addressing the distribution mismatch that limits conventional offline supervised fine-tuning. The approach offers a compelling bridge between imitation learning and reinforcement learning, but its benefits depend heavily on objective design, teacher quality, and stable optimization.

• Student-generated states expose the teacher to the errors and failure modes the student actually encounters, reducing the train–inference mismatch of teacher-forced data.
• Distillation objectives must carefully manage token-level probability matching, sequence-level quality, and exploration; naïve KL optimization can over-constrain the student or propagate teacher biases.
• Because the student’s sampling distribution changes during training, evaluation, rollout quality, and optimization stability become central engineering concerns rather than implementation details.

Literature review outlining sequence-level distillation techniques across MiniLLM, GKD, and f-divergence generalizations.

• Analyzes teacher guidance strategies during autoregressive sampling.

A unified mathematical formulation connecting supervised fine-tuning (forward KL), RLHF (reverse KL with reward shaping), and on-policy distillation.

• Clarifies why on-policy student generations prevent mode collapse and exposure bias.