I’d start with the Pokémon section: Sonnet 3.7 wrote 413 memory files and earned one badge, while Fable 5 used nine files plus global.md and earned four, favoring durable strategies over game state it could see on-screen. I like the framing of memory as a prediction about what a future session will need; with one run per model, though, the comparison illustrates the design point more than it isolates its cause.

I like the failure analysis: generic rubrics reward a fluent “mini-paper” and penalize the human author’s selective omissions, so a judge can prefer model prose for the wrong reasons. Interesting approach, might try in my humanize rl experiment

I like the distinction between matching the human score distribution and actually ranking stories as people do: one in-context example brought GLM 5.3 Flash closer to the human baseline, but the extremes collapsed and story-level correlation still included zero. The pairwise reconstruction found no detectable preference for LLM-written stories, though the evidence is specific to one model setup and a subjective writing dataset.

I trust this one because it holds the training setup fixed while changing the judge, so the comparison gets at how judge preferences shape a model’s design. The reward combines group-pick comparisons with brief fulfillment, but the task set leans heavily toward landing pages, which keeps any broader claim about design taste narrow.

I’m not sure these benchmark gains transfer beyond the tested protocols, but the ACOS result makes the training-evaluation mismatch hard to ignore: matching the evaluator’s hundreds of category-sentiment questions and 0.56% positive rate took the score from 27.2 to 61.6, while oversampling only reached 30.1. I also like the distillation result: the one-pass student scored 62.47 against 64.81 for the reasoning model, then v1.1 passed it at more than 100 times lower mean latency. Their attempts with pause tokens and layer reuse didn’t beat the plain approach, which makes the case for soft targets from teachers that reason more interesting.

The unusual part is how much of the work is inspectable: training checkpoints, item-by-item evaluation outputs, the chat-tuning data, and even a failed RL run. I like that they also say Laghu cleared none of its three pre-registered categories; the $146 pretraining run is a striking detail, but the release’s real value is making the results easier to check, with some dataset slices available only as rebuild pointers.

I’d start with the paired ablation: Luna goes from 36.9% to 37.1% on all 480 SWE-bench Multimodal tasks without images, while its MMMU-Pro accuracy falls from 79.7% to 36.9%.

Rubric-based judging is often treated as prompt design; Autorubric makes the measurement choices explicit, with independent scores across binary, ordinal, and nominal criteria and psychometric reliability checks.

A higher benchmark score does not automatically mean an agent is better, especially when tasks are subjective and runs vary. I’d read this for its three-part approach, pairing human evaluation with offline evaluation and production monitoring, and for the attention to uncertainty and task selection.

The unusual move is using a decision-only judge’s label probabilities as an escalation gate, with the threshold frozen before held-out evaluation. I find the test persuasive: on 1,610 pairs, the cascade escalates 31%, beats GPT-6 alone by 0.9 points, and costs 41% as much. Confidence weakens on style-adversarial pairs and reference-free prose, so I wouldn’t carry a threshold to a new workload without local labels.

I’d start with the threshold-tuning section: a model can separate failures from passes perfectly and still catch none at the default 0.50 threshold. I like that the example chooses thresholds on training folds and measures them on held-out folds.

The criterion I keep coming back to is that a synthetic world should reliably elicit the target failure mode; a bug in a cloned travel site shouldn’t be mistaken for weak planning. I like the proposed loop of solver agents, a verifier agent, and hackers searching for reward-gaming solutions, though the piece offers a design recipe rather than evidence that the loop improves task quality.

The post turns a port of ALE’s 105-task Linux CLI subset into an audit, finding hidden output requirements, answer leakage, and broken tasks. I’d read it for the reminder that standardized rollout traces make comparisons easier, but can’t rescue defective tasks or scoring; the task-level findings matter more than another aggregate score.

I’d read this for the distinction between choosing quickly and knowing whether the choice is right: masking the vocabulary to A-E cuts the example from 11 decoding passes to one, but the model’s 99.78% for Cave on an ambiguous bat question shows how misleading those probabilities can be. The CommonsenseQA results, 59.38% accuracy before a quick fine-tune and 62.41% after, make this a grounded demonstration; I wouldn’t treat the apparent confidence as calibrated without stronger evidence.

Task-design advice often centers on the prompt; this treats the grader and environment as part of the task, which is the useful shift.

I’d read this for the scoring setup: verdict macro-F1, attack-type macro-F1, and attack recall at 95% precision make the tradeoff between catching attacks and keeping false alarms rare visible in one number. Jev and the OpenAI Decisions API sit on the cost-versus-score Pareto frontier, but the 1,985-email set is synthetic and 40% attacks, so its high-precision results are easier to reach than in a real inbox.

The sharpest move here is to treat any ungraded instruction as a lie: the Triton example shows how a fast, correct kernel can still violate a ban the grader never checks. I also find the case for accepting environment awareness and prioritizing sandbox security over realism interesting, though the post argues for this tradeoff without showing how it affects learned behavior.

If you’re trying to tell whether an agent improves training data or just changes the training recipe, CurateBench makes that boundary unusually clear: the optimizer, recipe, and evaluations are fixed, leaving data as the only intervention. DataSmith beats Claude Code by 5.1 percentage points on average across 12 matched comparisons, but this is Datology’s benchmark on three base models and four evaluations, so I’d treat the result as evidence for this setup rather than a general harness ranking.

I trust this one because it separates the method-level benchmark from SID-only comparisons with the backbone and training protocol held fixed; that makes its finding that codebook balance doesn’t reliably predict recommendation quality more informative than a leaderboard alone. The results are non-monotonic: RQ-VAE and OPQ vary across datasets, while OPQ and RQ-Kmeans preserve different aspects of local neighborhoods, so the study doesn’t settle on a universal design.

The reported Pareto position is plotted against decoded text per second per GPU, not just parameter count; I like that serving-cost axis for a 78.1B MoE with 3.46B active parameters. I’d read the report for its 24T-token training pipeline and 1.2M-task RL setup, though the comparison uses each model’s fastest configuration on eight B200 GPUs, with 4k contexts for base models and 16k for post-trained ones.

Another jev clone. Nice writeup.

I’m not sure a guided workflow can keep evals from drifting toward what the current model is bad at, but the warning about measuring a model’s “failure fingerprint” is exactly the trap to watch for. I like the safeguards: choose hard cases because people can explain why they’re hard, validate the grader on sample outputs, and use held-out examples while changing one thing at a time; the post lays out the process but doesn’t show how much it improves applications.

The interesting move is treating the hand-rated reference pool as part of the reward: the pairwise judge compares paintings against it, so “the pool defines taste here.” I like that the open implementation compares three mixes of that judge and HPSv3, though 60- and 110-step runs and model-based taste proxies leave the strength of the result uncertain.

I’m wary of treating the leaderboard as a clean model comparison: there are only 20 samples per model, the closed models use different sampling conditions, and structure scores 1.000 on 136 of 139 outputs. I’d still read this for the scorer design: source-level gates catch image and text cheats before geometry and vision calls, and the blind caption called only two of the seven median drawings pelicans, making the judge’s semantic read a real limitation rather than a footnote.

The strongest part, to me, is the explanation that Universe’s basic loop wasn’t the missing piece: pretrained models, tasks within reach of sparse-reward RL, and interfaces beyond raw pixels arrived later. The benchmark-to-training transition adds an operational wrinkle, especially Terminal-Bench’s warning about benchmark data in training; I’d read the timeline as a causal sketch rather than proof of which change mattered most.

I’m not sure one six-hour attack on the lab’s own harness says much about transfer, but the method is worth a look: 89 escapes in 153 attempts became cases the revised harness had to block on replay. I especially like the separation between checking the eval and fixing the decision rule before results arrive; a missing run makes the evidence invalid, not a pass.

A clean training loop can still be wrong, and I like that this recipe treats silent failure as the central problem rather than a tooling nuisance. It builds trust in stages, from inspecting examples to a tiny end-to-end model, with checks like loss at initialization and an input-independent baseline; the insistence on explicit hypotheses is a good counterweight to piling on augmentation and model complexity.

I trust this one because it makes the case for a “single trace for each request” that links retrieval, tool choices, model calls, and user outcomes; prompt-response logs alone can’t show which step led to a bad answer. It’s a clear guide to what teams might instrument, though it reads as a vendor’s taxonomy rather than evidence that a platform can reliably catch those failures.

Netflix frames an LLM judge as a continuously managed production system rather than a static benchmark artifact. The lifecycle spans benchmark creation, rubric-oriented training, deployment, and human-supervised monitoring as recommendation data and model behavior evolve.

• Curated human-labeled datasets define multiple evaluation criteria; Reasoning-Aligned Rubric Tuning (RART) uses a meta-judge over reasoning traces to refine judge rubrics.
• A single judge supports both quality gating and reflective explanation generation, evaluating hundreds of thousands of show-level explanations weekly.
• Human-in-the-loop drift monitoring triggers re-tuning behind a review gate; a five-week A/B test across tens of millions of members increased novel-content viewing and successful browse-to-play sessions without quality-related takedowns.

Self-Harness presents a model-specific, self-improvement loop in which an LLM agent mines its own execution failures, proposes minimal harness changes, and validates them through regression testing. Across nine model–benchmark combinations, the approach improved both held-in and held-out performance, with relative gains of up to 132%.

• Weakness Mining: extracts recurring failure patterns from execution traces rather than relying on hand-authored diagnoses.
• Harness Proposal: generates diverse, minimal modifications targeted at bottlenecks such as artifact handling, runtime control, patch verification, and application-state retrieval.
• Proposal Validation: retains edits only when they pass regression tests, enabling iterative improvement without stronger external agents or human engineers.

A critical breakdown of why statistical AI text classifiers suffer from high false-positive rates on formal, non-native, and technical English.

• Explains why perplexity and burstiness metrics fail in real-world adversarial settings.

Detailed taxonomy of autonomous scientific research agents and the meta-harnesses required to validate hypotheses, run experiments, and synthesize literature.

• Bridges theoretical agent architectures with practical laboratory workflows.

Lilian Weng’s comprehensive guide on building robust evaluation harnesses that allow AI agents to iteratively improve their own performance.

• Focuses on environment feedback loops, automated failure analysis, and benchmark integrity.

Lee Han Chung examining the operational friction of testing LLM agents in CI/CD.

• Addresses sandbox state contamination, flaky non-deterministic tool mockings, and cost bloat.

Eugene Yan on designing automated evaluation environments for offensive and defensive AI security agents.

• Emphasizes capturing exploit mechanics, deterministic scoring sandboxes, and defense-in-depth metrics.

Cameron Wolfe’s extensive breakdown on constructing agent benchmarks.

• Evaluates unit test assertions, trajectory fidelity, tool argument correctness, and human-in-the-loop rubrics.