Humanize RL applied to a specific domain. but nice read.

I’m not sure how much stability‑score centering helps across different mismatch regimes, but I like that this explanation traces instability to the sampler and trainer token probabilities diverging. As always, Cameron has done a nice writeup.

I like the failure analysis: generic rubrics reward a fluent “mini-paper” and penalize the human author’s selective omissions, so a judge can prefer model prose for the wrong reasons. Interesting approach, might try in my humanize rl experiment

If you’re starting with RL, I like that this guide pairs a structured learning path with a PyTorch Deep Q-Network project, so there’s a clear way to move from concepts to implementation. It’s from 2021 and centers on a classic method, so I’d use it as an entry point rather than a guide to the field as it stands today.

The chapter derives the policy gradient from trajectory probabilities before connecting it to prompt-and-completion training, which makes the log-derivative trick feel like a working tool rather than a formula to memorize. I also like the clarification that many RLHF implementations treat a whole completion as one action; it helps explain why the chapter sets the discount factor to 1.

I like the argument that online weight updates could stop agents from repeatedly spending context fighting their own learned behaviors, while offline refinement can use completed traces, delayed feedback, and replay.

I like the division of labor: SFT on user and tool outputs models the environment, while RL on assistant tokens trains decisions.

“There is nothing explicit in a task reward that incentivises working in parallel” is the sharpest point here: the training setup has to make faster work pay if it wants to elicit coordination.

Pure sci-fi stuff. If you’re curious whether an LLM can do more than describe a policy, I’d point you here. I like that the example exposes the actual weight assignments, with explicit patterns for ball position, bounce handling, and context-gated comparisons; that makes the proposed weight-space programming unusually inspectable.

If you’re working on tool-using RL, this is worth a look: decoding and re-encoding can change token IDs, and inserting one byte can shift BPE merges on both sides, so the training sequence may no longer match inference. The Rust-to-Lean path through Aeneas is an interesting way to address that correctness gap; I’d want to know exactly which renderer and tokenizer behaviors fall inside the proof boundary before treating it as end-to-end protection.

I trust this one because it holds the training setup fixed while changing the judge, so the comparison gets at how judge preferences shape a model’s design. The reward combines group-pick comparisons with brief fulfillment, but the task set leans heavily toward landing pages, which keeps any broader claim about design taste narrow.

I like this because it makes a choice that often gets buried in annotation pipelines explicit: whether labels should capture different views or apply one prescribed belief. The descriptive/prescriptive split clarifies why disagreement can be signal in subjective safety data, while also showing the tradeoff: aggregation gets easier with a single standard, but guidelines and training are hard to get right.

I’d start with the learning-rate section: LoRA typically needs about 10x the full fine-tuning rate, so carrying over the old rate can make it look worse than it is. I like the distinction between RL, where the primer says small ranks match full fine-tuning, and large supervised datasets, where LoRA can lose ground; I’d treat the token-count rule as a rough sizing heuristic, not a guarantee.

I’m cautious about generalizing the Dimuon recommendation, but the page gives a clear reason to test it: in Tinker’s SFT runs, it reached lower loss in fewer steps at similar step time, while RL efficiency was similar to AdamW. The roughly 15× learning-rate ratio for SFT versus 60× for RL makes clear that switching optimizers also means retuning; I’d treat those ratios as starting points and validate on my own evaluations.

I’m not sure this helps choose between objectives, since it explains data layout rather than comparing losses. I like the shift-by-one walkthrough because it makes the masking distinction explicit: SFT zeros prompt weights, while the RL example zero-pads advantages to exclude prompt positions.

The criterion I keep coming back to is that a synthetic world should reliably elicit the target failure mode; a bug in a cloned travel site shouldn’t be mistaken for weak planning. I like the proposed loop of solver agents, a verifier agent, and hackers searching for reward-gaming solutions, though the piece offers a design recipe rather than evidence that the loop improves task quality.

I trust this one because it asks whether a learned revision procedure transfers, not just whether a harness can be optimized for one task: the proposer uses execution feedback to edit code while both proposer and solver stay frozen at test time. The trained 4B proposer outperforms its 35B teacher on average for single-step revision, and the reported transfer across reasoning and multi-hop QA is encouraging; I’d keep the generalization claim bounded to those settings.

The sharpest move here is to treat any ungraded instruction as a lie: the Triton example shows how a fast, correct kernel can still violate a ban the grader never checks. I also find the case for accepting environment awareness and prioritizing sandbox security over realism interesting, though the post argues for this tradeoff without showing how it affects learned behavior.

The reported Pareto position is plotted against decoded text per second per GPU, not just parameter count; I like that serving-cost axis for a 78.1B MoE with 3.46B active parameters. I’d read the report for its 24T-token training pipeline and 1.2M-task RL setup, though the comparison uses each model’s fastest configuration on eight B200 GPUs, with 4k contexts for base models and 16k for post-trained ones.

The unusual move is giving the model unrestricted edits to a context file, rather than adding another fixed compaction or retrieval action to the harness. I like that this lets context strategies emerge while making the serving cost visible: on BrowseComp-Plus, the paper reports 11.4% higher accuracy with 21.5% fewer prefix-reuse FLOPs than its strongest baseline. Arbitrary edits can force re-prefilling, so Suffix Cache Reuse is an important part of the story; I’d want to see how well that cache result holds beyond the reported setup.

Another jev clone. Nice writeup.

What stands out to me is that the teacher can guide the student along its own attempt with dense token-level predictions, without writing a replacement answer; that turns feedback into a richer training signal than a pass-or-fail reward. The reported OPSD comparison is worth a closer look because it gives both the 8-12× token-efficiency result and the setup behind it, including 1,024-token student responses versus 16,384 for GRPO; the broader case for preserving capabilities in continual learning still seems unsettled.

The unusual move is to treat the weights as a species and sampled policies as individuals, then ask what survives across varied tasks. I like that this connects generalization and instruction-reward mismatch to the same selection story, though the claims about honesty and deception are explicitly speculative, not demonstrated results.

I’d start with the harness design: the agent maintains a curated result set as it searches, so running out of turns still leaves something gradable. I like that clear connection between episode limits and evaluation, but the early runs use only 256 training queries and treat supporting facts like answers, so they don’t establish performance on the canonical task.

I like that this treats reward modeling as a recurring inference-cost problem, not just a one-time RLHF expense: TinyRM uses bidirectional 150M- and 400M-parameter models with FLAN-style cloze prompts, DoRA, and layer freezing, and reports rivaling models over 175 times larger on reasoning and safety tasks. The result is domain-specific, with conversational preference modeling still a limitation, and the paper leaves ablations to future work, so it doesn’t establish which design choice is doing the work.

I like that the experiment treats query-plan scoring as noisy in practice: it builds a measurement rig to reduce Linux page-cache contention and uses a custom GRPO variant for noisy rollouts. The reported 44.7% latency reduction across 113 join-heavy queries is interesting, but I’d want the 81%-faster title result reconciled with that aggregate before reading it as a general win over Postgres.

I like the fixed-position board encoding and the 1,968 move tokens: they give a decoder-only model a much cleaner chess interface than raw move text, while keeping the route to WebGPU deployment open. The 2004 Elo comes from a full recipe with a strong teacher, some RL, and a lot of search, so I wouldn’t read it as the strength of the 230M model alone.

I’m not convinced Atari 100k makes the case for real-world viability; the leap from a game benchmark to robotics or healthcare is still unshown here. I like the paper’s diagnosis of what limits sample-efficient MuZero-style learning, especially its end-to-end value prefix to reduce compounding error, and the 109% median human score against DQN’s 96% with 500 times less data makes the result worth inspecting.

The 100-0 result is striking, but the part I like is the training loop: MCTS turns the current network into a stronger policy target, then self-play outcomes train its value estimate, with no human games or rollouts. It’s a clean demonstration of search and learning improving each other; 4.9 million self-play games and 1,600 simulations per move also make the compute demands hard to ignore, and Go is a much tidier setting than open-ended agent work.

I’d start with the hidden-state design: MuZero predicts reward, policy, and value without reconstructing observations or recovering the environment’s true state. I like this focus on what planning needs, and the results are striking: state of the art across 57 Atari games and AlphaZero-level performance in Go, chess, and shogi without game rules. Those benchmarks don’t show whether the approach transfers to open-ended real-world dynamics.

I’d start with the shift from bits per sample to signal-to-noise on the task objective: a binary reward carries little information, but points the update at success, while next-token loss also spends gradient on incidental tokens. I like this as a way to frame why RL can beat iterative SFT, though the broader SNR and loss-landscape explanation is explicitly speculative and needs tests that distinguish objective alignment from other causes.

I’d start at the on-policy section, where the student generates its own tokens and a stronger teacher scores them, so training targets mistakes the student actually makes. I like the arc from compression to behavior transfer because it makes clear how distillation’s goal has changed; the claim that distillation, supervised fine-tuning, RL, and synthetic data are becoming the same pipeline needs sharper boundaries than this short history gives.

If you’re trying to understand why agent RL has become an infrastructure problem, I’d point you here. The post separates the task and verifier, the agent’s interaction contract, the sandbox machine, and the rollout system, then lays out the white-box versus black-box harness tradeoff; the survey is necessarily a map of disclosed practice, since some major labs say little about what they train in.

“If you can execute a check, you have a reward function” is a strong anchor for this tour of frontier-lab reports, especially the example of Kimi K2 training a critic on rollouts from verifiable-reward prompts. I like that it also traces how GRPO has become a family of modified objectives; the caveat is that collected reports show what labs describe, not which variant performs best.

I like this because it separates ordinary teacher-to-student compression from a newer pattern: same-size, domain-specialized RL checkpoints guide a student on its own rollouts. The contrast between token-level teacher feedback and RL’s single reward for an attempt makes the appeal clear, though the model-report examples don’t establish when distillation actually wins.

A single overview can flatten the important details, so I’d treat this as a map rather than a substitute for the deeper sources it links. I like the choice to start with the standard agent, action, and policy setup, then frame the LLM as a stochastic policy; it connects familiar RL machinery to LLM training without making the basics feel like a special case.

The interesting move is treating the hand-rated reference pool as part of the reward: the pairwise judge compares paintings against it, so “the pool defines taste here.” I like that the open implementation compares three mixes of that judge and HPSv3, though 60- and 110-step runs and model-based taste proxies leave the strength of the result uncertain.

I’m wary of treating the leaderboard as a clean model comparison: there are only 20 samples per model, the closed models use different sampling conditions, and structure scores 1.000 on 136 of 139 outputs. I’d still read this for the scorer design: source-level gates catch image and text cheats before geometry and vision calls, and the blind caption called only two of the seven median drawings pelicans, making the judge’s semantic read a real limitation rather than a footnote.

The strongest part, to me, is the explanation that Universe’s basic loop wasn’t the missing piece: pretrained models, tasks within reach of sparse-reward RL, and interfaces beyond raw pixels arrived later. The benchmark-to-training transition adds an operational wrinkle, especially Terminal-Bench’s warning about benchmark data in training; I’d read the timeline as a causal sketch rather than proof of which change mattered most.

I like that this treats scaling as a question about reusable learning, not just lower loss: the Dyna-2 example follows human-video pretraining into robot task scores under the same post-training setup, rising from 20% to 53% as the data grows from 1,000 to 1 million hours. That transfer is the interesting part, though data composition and the video prediction objective matter too, so I wouldn’t read the curve as a clean scale-only law.

I like that TailRL turns continuous rewards into a family of binary success events, then optimizes their log-probabilities instead of only the mean. The harmonic mixture of Best-of-k gradients makes the connection to inference-time sampling easy to see, though I’m not sure how sensitive the method is to choosing thresholds uniformly over [0,1].

I like this because it makes the mismatch between training on “flawless teacher prefixes” and generating from the student’s own outputs the organizing idea, then sorts methods by objectives, signal sources, and training dynamics. The DAgger-style reduction from quadratic to linear compounding error is a good motivation, but I wouldn’t take it as proof that every white-box, black-box, and self-distillation setup gets that benefit; the survey’s account of conditions and failures like diversity collapse is what makes the comparison worth reading.

The strongest part is the ceiling argument: fixed teacher data can only carry SFT so far, while OPD samples from the student’s own states and still uses teacher token-level feedback. That makes the comparison more precise than “OPD beats RL”: OPD can reach teacher quality efficiently, while verifier-based RL can aim past it; the reported 9-30× compute gap is intriguing, though I’d want benchmark and teacher-call details before generalizing it.

If you’re comparing post-training methods, this framing is worth a read: SFT pulls toward a fixed dataset, while on-policy distillation matches a teacher on the student’s own samples. The surprising result is that students distilled from SFT and RL teachers performed similarly on minimal code editing and forgot less than the SFT teacher; I’d keep the implication that on-policy sampling matters more than teacher choice tentative, since it comes from one coding-task setup.

I like this because each task includes a second, discoverable bug, so passing requires more than fixing the defect named in the prompt. On 10 held-out fontTools tasks, SFT goes from 0/100 attempts to 52/100, and RL adds five points; the SQLGlot transfer is intriguing, but it is also only 10 tasks, and SFT increases token use there. I’d read this as a promising recipe for repository adaptation, not broad evidence that the gains transfer.

This treats large-scale code search as a tool-use policy to post-train, pairing a small model with precomputed BM25 and dense indexes. I like that the open-ended task scores precision and efficiency while requiring code citations; the LLM judge and constructed tasks leave transfer to real queries uncertain.

If you’re working on long-horizon agent training, I like that this spends real attention on the unglamorous part: “Steps 1 to 3 are de-risking,” from harness bugs and token accounting to rollout failures, before significant compute goes into ablations. The reported 70% relative Pass@1 gain on held-out APEX-Agents is worth examining, though it’s still evidence on one benchmark; releasing the training script, weights, and eval traces makes the recipe easier to inspect.

Taking the log of the trajectory probability turns the product into a sum, and the environment-transition terms vanish when differentiated with respect to policy parameters. I like this clear account of how REINFORCE learns from sampled trajectories without differentiating through the environment; the connection to variance and trajectory length keeps the cost of that choice in view.

I trust the variance argument as far as it goes: the score has zero conditional mean, so a baseline can depend on state without biasing the gradient, as long as it doesn’t depend on the action or future rewards. The claimed drop in the variance bound from O(T^3) to O(T^2) explains why baselines matter, though it doesn’t tell us how much variance falls in a particular training setup.

I’d start with the RLVR framing: it separates where the reward comes from, a verifier rather than a preference model, from which optimizer produces the policy update. I like the focus on vanilla GRPO’s subtle problems at scale, though the portion available here is still laying foundations, so I can’t assess whether the promised training tricks are convincing.

On-policy distillation trains a student using contexts sampled from its own evolving policy while learning from a stronger teacher, addressing the distribution mismatch that limits conventional offline supervised fine-tuning. The approach offers a compelling bridge between imitation learning and reinforcement learning, but its benefits depend heavily on objective design, teacher quality, and stable optimization.

• Student-generated states expose the teacher to the errors and failure modes the student actually encounters, reducing the train–inference mismatch of teacher-forced data.
• Distillation objectives must carefully manage token-level probability matching, sequence-level quality, and exploration; naïve KL optimization can over-constrain the student or propagate teacher biases.
• Because the student’s sampling distribution changes during training, evaluation, rollout quality, and optimization stability become central engineering concerns rather than implementation details.

A first-principles-to-frontier survey of how reinforcement learning is adapted to language models, explaining the progression from classical RL concepts through RLHF and newer reasoning-oriented training methods.

• Frames an LLM as a policy that generates token sequences, with rewards assigned to outputs or trajectories rather than individual actions.
• Decomposes the RLHF pipeline into supervised fine-tuning, preference-based reward modeling, and policy optimization, including the role of KL regularization in preventing drift from the reference model.
• Connects modern approaches such as verifiable rewards and group-relative policy updates to the practical challenge of improving reasoning without relying exclusively on human preference labels.

Explores how generative environment simulators allow agents to simulate actions and consequences in latent space before committing real-world actions.

• Essential reading for model-based RL and forward simulation in software engineering agents.

Luke Huang on the challenges of asynchronous, distributed policy gradient methods at frontier model scale.

• Evaluates staleness penalties, off-policy correction heuristics, and gradient synchronization tradeoffs.

Cameron Wolfe’s analysis on applying reinforcement learning for ongoing post-training adaptation without catastrophic forgetting.

• Explores replay buffers, KL regularizers, and distribution shift mitigation.

NVIDIA Technical Blog overview of training reasoning agents on multi-step environment trajectories.

• Covers reward shaping, credit assignment over long tool horizons, and trajectory curation.

A unified mathematical formulation connecting supervised fine-tuning (forward KL), RLHF (reverse KL with reward shaping), and on-policy distillation.

• Clarifies why on-policy student generations prevent mode collapse and exposure bias.