I’m not sure a guided workflow can keep evals from drifting toward what the current model is bad at, but the warning about measuring a model’s “failure fingerprint” is exactly the trap to watch for. I like the safeguards: choose hard cases because people can explain why they’re hard, validate the grader on sample outputs, and use held-out examples while changing one thing at a time; the post lays out the process but doesn’t show how much it improves applications.

I trust this one because it keeps the caveat beside the phone-built production changes: the work was low-risk, and its business logic was covered by automated tests. The more interesting claim is that coding agents make typing less central while review, testing, and product judgment matter more; the evidence here is a run of practitioners’ “a-ha” stories, not proof that the shift is universal.

The interesting move is treating the hand-rated reference pool as part of the reward: the pairwise judge compares paintings against it, so “the pool defines taste here.” I like that the open implementation compares three mixes of that judge and HPSv3, though 60- and 110-step runs and model-based taste proxies leave the strength of the result uncertain.

I like this because each task includes a second, discoverable bug, so passing requires more than fixing the defect named in the prompt. On 10 held-out fontTools tasks, SFT goes from 0/100 attempts to 52/100, and RL adds five points; the SQLGlot transfer is intriguing, but it is also only 10 tasks, and SFT increases token use there. I’d read this as a promising recipe for repository adaptation, not broad evidence that the gains transfer.

This treats large-scale code search as a tool-use policy to post-train, pairing a small model with precomputed BM25 and dense indexes. I like that the open-ended task scores precision and efficiency while requiring code citations; the LLM judge and constructed tasks leave transfer to real queries uncertain.

I can’t judge the reported 12.5% accuracy lift without the evaluation details, but the two-index explanation is worth reading: semantic search handles intent, while sparse n-grams handle exact patterns, and each covers a gap in the other. The more unusual detail is training embeddings from agent-session traces, with an LLM ranking what should have surfaced earlier; the author’s suggested contrastive objective is clearly marked as inference, not a confirmed implementation detail.