The unusual move is giving the model unrestricted edits to a context file, rather than adding another fixed compaction or retrieval action to the harness. I like that this lets context strategies emerge while making the serving cost visible: on BrowseComp-Plus, the paper reports 11.4% higher accuracy with 21.5% fewer prefix-reuse FLOPs than its strongest baseline. Arbitrary edits can force re-prefilling, so Suffix Cache Reuse is an important part of the story; I’d want to see how well that cache result holds beyond the reported setup.

The unusual move is to treat the weights as a species and sampled policies as individuals, then ask what survives across varied tasks. I like that this connects generalization and instruction-reward mismatch to the same selection story, though the claims about honesty and deception are explicitly speculative, not demonstrated results.

I’d start with the harness design: the agent maintains a curated result set as it searches, so running out of turns still leaves something gradable. I like that clear connection between episode limits and evaluation, but the early runs use only 256 training queries and treat supporting facts like answers, so they don’t establish performance on the canonical task.

I like that this treats reward modeling as a recurring inference-cost problem, not just a one-time RLHF expense: TinyRM uses bidirectional 150M- and 400M-parameter models with FLAN-style cloze prompts, DoRA, and layer freezing, and reports rivaling models over 175 times larger on reasoning and safety tasks. The result is domain-specific, with conversational preference modeling still a limitation, and the paper leaves ablations to future work, so it doesn’t establish which design choice is doing the work.

I like that the experiment treats query-plan scoring as noisy in practice: it builds a measurement rig to reduce Linux page-cache contention and uses a custom GRPO variant for noisy rollouts. The reported 44.7% latency reduction across 113 join-heavy queries is interesting, but I’d want the 81%-faster title result reconciled with that aggregate before reading it as a general win over Postgres.

I like the fixed-position board encoding and the 1,968 move tokens: they give a decoder-only model a much cleaner chess interface than raw move text, while keeping the route to WebGPU deployment open. The 2004 Elo comes from a full recipe with a strong teacher, some RL, and a lot of search, so I wouldn’t read it as the strength of the 230M model alone.

I’m not convinced Atari 100k makes the case for real-world viability; the leap from a game benchmark to robotics or healthcare is still unshown here. I like the paper’s diagnosis of what limits sample-efficient MuZero-style learning, especially its end-to-end value prefix to reduce compounding error, and the 109% median human score against DQN’s 96% with 500 times less data makes the result worth inspecting.

The 100-0 result is striking, but the part I like is the training loop: MCTS turns the current network into a stronger policy target, then self-play outcomes train its value estimate, with no human games or rollouts. It’s a clean demonstration of search and learning improving each other; 4.9 million self-play games and 1,600 simulations per move also make the compute demands hard to ignore, and Go is a much tidier setting than open-ended agent work.

I’d start with the hidden-state design: MuZero predicts reward, policy, and value without reconstructing observations or recovering the environment’s true state. I like this focus on what planning needs, and the results are striking: state of the art across 57 Atari games and AlphaZero-level performance in Go, chess, and shogi without game rules. Those benchmarks don’t show whether the approach transfers to open-ended real-world dynamics.

I’d start with the shift from bits per sample to signal-to-noise on the task objective: a binary reward carries little information, but points the update at success, while next-token loss also spends gradient on incidental tokens. I like this as a way to frame why RL can beat iterative SFT, though the broader SNR and loss-landscape explanation is explicitly speculative and needs tests that distinguish objective alignment from other causes.

I’d start at the on-policy section, where the student generates its own tokens and a stronger teacher scores them, so training targets mistakes the student actually makes. I like the arc from compression to behavior transfer because it makes clear how distillation’s goal has changed; the claim that distillation, supervised fine-tuning, RL, and synthetic data are becoming the same pipeline needs sharper boundaries than this short history gives.

If you’re trying to understand why agent RL has become an infrastructure problem, I’d point you here. The post separates the task and verifier, the agent’s interaction contract, the sandbox machine, and the rollout system, then lays out the white-box versus black-box harness tradeoff; the survey is necessarily a map of disclosed practice, since some major labs say little about what they train in.

“If you can execute a check, you have a reward function” is a strong anchor for this tour of frontier-lab reports, especially the example of Kimi K2 training a critic on rollouts from verifiable-reward prompts. I like that it also traces how GRPO has become a family of modified objectives; the caveat is that collected reports show what labs describe, not which variant performs best.

I like this because it separates ordinary teacher-to-student compression from a newer pattern: same-size, domain-specialized RL checkpoints guide a student on its own rollouts. The contrast between token-level teacher feedback and RL’s single reward for an attempt makes the appeal clear, though the model-report examples don’t establish when distillation actually wins.

A single overview can flatten the important details, so I’d treat this as a map rather than a substitute for the deeper sources it links. I like the choice to start with the standard agent, action, and policy setup, then frame the LLM as a stochastic policy; it connects familiar RL machinery to LLM training without making the basics feel like a special case.

The interesting move is treating the hand-rated reference pool as part of the reward: the pairwise judge compares paintings against it, so “the pool defines taste here.” I like that the open implementation compares three mixes of that judge and HPSv3, though 60- and 110-step runs and model-based taste proxies leave the strength of the result uncertain.

I’m wary of treating the leaderboard as a clean model comparison: there are only 20 samples per model, the closed models use different sampling conditions, and structure scores 1.000 on 136 of 139 outputs. I’d still read this for the scorer design: source-level gates catch image and text cheats before geometry and vision calls, and the blind caption called only two of the seven median drawings pelicans, making the judge’s semantic read a real limitation rather than a footnote.

The strongest part, to me, is the explanation that Universe’s basic loop wasn’t the missing piece: pretrained models, tasks within reach of sparse-reward RL, and interfaces beyond raw pixels arrived later. The benchmark-to-training transition adds an operational wrinkle, especially Terminal-Bench’s warning about benchmark data in training; I’d read the timeline as a causal sketch rather than proof of which change mattered most.

I like that this treats scaling as a question about reusable learning, not just lower loss: the Dyna-2 example follows human-video pretraining into robot task scores under the same post-training setup, rising from 20% to 53% as the data grows from 1,000 to 1 million hours. That transfer is the interesting part, though data composition and the video prediction objective matter too, so I wouldn’t read the curve as a clean scale-only law.

I like that TailRL turns continuous rewards into a family of binary success events, then optimizes their log-probabilities instead of only the mean. The harmonic mixture of Best-of-k gradients makes the connection to inference-time sampling easy to see, though I’m not sure how sensitive the method is to choosing thresholds uniformly over [0,1].

I like this because it makes the mismatch between training on “flawless teacher prefixes” and generating from the student’s own outputs the organizing idea, then sorts methods by objectives, signal sources, and training dynamics. The DAgger-style reduction from quadratic to linear compounding error is a good motivation, but I wouldn’t take it as proof that every white-box, black-box, and self-distillation setup gets that benefit; the survey’s account of conditions and failures like diversity collapse is what makes the comparison worth reading.

The strongest part is the ceiling argument: fixed teacher data can only carry SFT so far, while OPD samples from the student’s own states and still uses teacher token-level feedback. That makes the comparison more precise than “OPD beats RL”: OPD can reach teacher quality efficiently, while verifier-based RL can aim past it; the reported 9-30× compute gap is intriguing, though I’d want benchmark and teacher-call details before generalizing it.

If you’re comparing post-training methods, this framing is worth a read: SFT pulls toward a fixed dataset, while on-policy distillation matches a teacher on the student’s own samples. The surprising result is that students distilled from SFT and RL teachers performed similarly on minimal code editing and forgot less than the SFT teacher; I’d keep the implication that on-policy sampling matters more than teacher choice tentative, since it comes from one coding-task setup.

I like this because each task includes a second, discoverable bug, so passing requires more than fixing the defect named in the prompt. On 10 held-out fontTools tasks, SFT goes from 0/100 attempts to 52/100, and RL adds five points; the SQLGlot transfer is intriguing, but it is also only 10 tasks, and SFT increases token use there. I’d read this as a promising recipe for repository adaptation, not broad evidence that the gains transfer.

This treats large-scale code search as a tool-use policy to post-train, pairing a small model with precomputed BM25 and dense indexes. I like that the open-ended task scores precision and efficiency while requiring code citations; the LLM judge and constructed tasks leave transfer to real queries uncertain.

If you’re working on long-horizon agent training, I like that this spends real attention on the unglamorous part: “Steps 1 to 3 are de-risking,” from harness bugs and token accounting to rollout failures, before significant compute goes into ablations. The reported 70% relative Pass@1 gain on held-out APEX-Agents is worth examining, though it’s still evidence on one benchmark; releasing the training script, weights, and eval traces makes the recipe easier to inspect.

Taking the log of the trajectory probability turns the product into a sum, and the environment-transition terms vanish when differentiated with respect to policy parameters. I like this clear account of how REINFORCE learns from sampled trajectories without differentiating through the environment; the connection to variance and trajectory length keeps the cost of that choice in view.

I trust the variance argument as far as it goes: the score has zero conditional mean, so a baseline can depend on state without biasing the gradient, as long as it doesn’t depend on the action or future rewards. The claimed drop in the variance bound from O(T^3) to O(T^2) explains why baselines matter, though it doesn’t tell us how much variance falls in a particular training setup.

I’d start with the RLVR framing: it separates where the reward comes from, a verifier rather than a preference model, from which optimizer produces the policy update. I like the focus on vanilla GRPO’s subtle problems at scale, though the portion available here is still laying foundations, so I can’t assess whether the promised training tricks are convincing.

On-policy distillation trains a student using contexts sampled from its own evolving policy while learning from a stronger teacher, addressing the distribution mismatch that limits conventional offline supervised fine-tuning. The approach offers a compelling bridge between imitation learning and reinforcement learning, but its benefits depend heavily on objective design, teacher quality, and stable optimization.

• Student-generated states expose the teacher to the errors and failure modes the student actually encounters, reducing the train–inference mismatch of teacher-forced data.
• Distillation objectives must carefully manage token-level probability matching, sequence-level quality, and exploration; naïve KL optimization can over-constrain the student or propagate teacher biases.
• Because the student’s sampling distribution changes during training, evaluation, rollout quality, and optimization stability become central engineering concerns rather than implementation details.

A first-principles-to-frontier survey of how reinforcement learning is adapted to language models, explaining the progression from classical RL concepts through RLHF and newer reasoning-oriented training methods.

• Frames an LLM as a policy that generates token sequences, with rewards assigned to outputs or trajectories rather than individual actions.
• Decomposes the RLHF pipeline into supervised fine-tuning, preference-based reward modeling, and policy optimization, including the role of KL regularization in preventing drift from the reference model.
• Connects modern approaches such as verifiable rewards and group-relative policy updates to the practical challenge of improving reasoning without relying exclusively on human preference labels.

Explores how generative environment simulators allow agents to simulate actions and consequences in latent space before committing real-world actions.

• Essential reading for model-based RL and forward simulation in software engineering agents.

Luke Huang on the challenges of asynchronous, distributed policy gradient methods at frontier model scale.

• Evaluates staleness penalties, off-policy correction heuristics, and gradient synchronization tradeoffs.

Cameron Wolfe’s analysis on applying reinforcement learning for ongoing post-training adaptation without catastrophic forgetting.

• Explores replay buffers, KL regularizers, and distribution shift mitigation.

NVIDIA Technical Blog overview of training reasoning agents on multi-step environment trajectories.

• Covers reward shaping, credit assignment over long tool horizons, and trajectory curation.

A unified mathematical formulation connecting supervised fine-tuning (forward KL), RLHF (reverse KL with reward shaping), and on-policy distillation.

• Clarifies why on-policy student generations prevent mode collapse and exposure bias.