Code review
The model gets a real pull request and reviews it.
It passes if it finds the specific problem that a later commit had to fix, and doesn't raise known false alarms.
Public benchmarks tell me how models do on someone else's problems. These evals use my own: real tasks from my work, each with a right answer that was settled before any model saw it.
The suite has tasks. Each one is a piece of work I did on a production TypeScript product (a Node.js API, an Angular web app and an Ionic mobile app) where the outcome is now known: a bug a later commit proved, a fix that shipped with a test, a ticket whose merged change came with specs, a CI failure whose cause was found. That known outcome is what makes a task gradable.
Tasks fall into five kinds of work. Overall weights each kind by roughly how much of my time it takes.
The model gets a real pull request and reviews it.
It passes if it finds the specific problem that a later commit had to fix, and doesn't raise known false alarms.
The model gets the ticket text and the codebase as it was before the work started, and implements it.
It passes if the tests that shipped with the real change pass against its code. Some tickets also require nearby existing tests to keep passing.
The model gets a window of recent commits, one of which introduced a real bug, and is asked to find bugs, prove each with a failing test and fix it.
It passes if its test fails on the buggy code and passes once the real fix is applied.
The model gets a failing CI run or API test, with the logs, and has to find the root cause.
It passes if it names the actual mechanism and says correctly whether the code, the test or the environment needs to change.
The model gets a request that sounds reasonable but, taken literally, is wrong for the product.
It passes if it works out what was actually needed, which sometimes means no change at all.
The tasks stay private. Once a task is public it can end up in training data, and the code it comes from isn't mine to publish. The results page numbers the tasks instead of naming them.
Every attempt follows the same steps, and each one starts from scratch.
A fresh copy of the repository is made at the commit just before the real work. Later history is not in it, so the fix can't be found with git log. Dependencies are installed in advance.
The task's material (ticket text, pull request diff, CI log) is saved with the task and copied into the checkout. Nothing is fetched live, and later comments that hint at the answer are removed.
Each model runs headless through its vendor's coding agent (Claude Code, Codex CLI and so on) with full permissions inside the checkout. It can read code, run tests and edit files. It is told not to use the internet, GitHub or issue trackers. Nobody answers questions mid-task, and there is a time limit.
Scripts grade whatever can be checked by running code. A judge model grades written answers against a fixed answer key. Details below.
Every model gets attempts at every task, so a lucky or unlucky run counts for less.
Tickets are graded by the tests that shipped with the real change. The model never sees them; they are added to its checkout after it finishes. Bug hunts are graded by running the model's own test twice in fresh checkouts: once on the buggy commit, where it must fail, and once with the real fix applied, where it must pass.
Reviews, triage and judgement tasks produce a written answer. Each has an answer key: the specific points a correct answer must make, known wrong conclusions it must not reach, and sometimes the expected verdict (approve or request changes). A fixed judge model reads the answer and decides, point by point, whether each one is present, quoting the evidence. Pass or fail is then worked out from those decisions: every required point found, no wrong conclusion, the right verdict. The judge never gives a score of its own.
A task only enters the suite once its grading is shown to tell right from wrong:
Tasks that stop telling models apart, or that turn out to punish valid alternative solutions, are retired.
The weights roughly follow how much of my working time each kind of task takes: