Real-work evals by Gui Politta
Methodology

Tasks I've already solved, graded the same way every time.

Public benchmarks tell me how models do on someone else's problems. These evals use my own: real tasks from my work, each with a right answer that was settled before any model saw it.

The tasks

The suite has tasks. Each one is a piece of work I did on a production TypeScript product (a Node.js API, an Angular web app and an Ionic mobile app) where the outcome is now known: a bug a later commit proved, a fix that shipped with a test, a ticket whose merged change came with specs, a CI failure whose cause was found. That known outcome is what makes a task gradable.

Tasks fall into five kinds of work. Overall weights each kind by roughly how much of my time it takes.

Code review

The model gets a real pull request and reviews it.

It passes if it finds the specific problem that a later commit had to fix, and doesn't raise known false alarms.

Ticket

The model gets the ticket text and the codebase as it was before the work started, and implements it.

It passes if the tests that shipped with the real change pass against its code. Some tickets also require nearby existing tests to keep passing.

Bug hunt

The model gets a window of recent commits, one of which introduced a real bug, and is asked to find bugs, prove each with a failing test and fix it.

It passes if its test fails on the buggy code and passes once the real fix is applied.

Triage

The model gets a failing CI run or API test, with the logs, and has to find the root cause.

It passes if it names the actual mechanism and says correctly whether the code, the test or the environment needs to change.

Judgement

The model gets a request that sounds reasonable but, taken literally, is wrong for the product.

It passes if it works out what was actually needed, which sometimes means no change at all.

The tasks stay private. Once a task is public it can end up in training data, and the code it comes from isn't mine to publish. The results page numbers the tasks instead of naming them.

How a model runs

Every attempt follows the same steps, and each one starts from scratch.

  1. A clean checkout from before the answer existed

    A fresh copy of the repository is made at the commit just before the real work. Later history is not in it, so the fix can't be found with git log. Dependencies are installed in advance.

  2. Frozen inputs

    The task's material (ticket text, pull request diff, CI log) is saved with the task and copied into the checkout. Nothing is fetched live, and later comments that hint at the answer are removed.

  3. The model works in its own agent CLI

    Each model runs headless through its vendor's coding agent (Claude Code, Codex CLI and so on) with full permissions inside the checkout. It can read code, run tests and edit files. It is told not to use the internet, GitHub or issue trackers. Nobody answers questions mid-task, and there is a time limit.

  4. Grading

    Scripts grade whatever can be checked by running code. A judge model grades written answers against a fixed answer key. Details below.

  5. Repeat

    Every model gets attempts at every task, so a lucky or unlucky run counts for less.

Grading

Tests decide when they can

Tickets are graded by the tests that shipped with the real change. The model never sees them; they are added to its checkout after it finishes. Bug hunts are graded by running the model's own test twice in fresh checkouts: once on the buggy commit, where it must fail, and once with the real fix applied, where it must pass.

A judge with a checklist for written answers

Reviews, triage and judgement tasks produce a written answer. Each has an answer key: the specific points a correct answer must make, known wrong conclusions it must not reach, and sometimes the expected verdict (approve or request changes). A fixed judge model reads the answer and decides, point by point, whether each one is present, quoting the evidence. Pass or fail is then worked out from those decisions: every required point found, no wrong conclusion, the right verdict. The judge never gives a score of its own.

Every task is checked before it counts

A task only enters the suite once its grading is shown to tell right from wrong:

  • the hidden tests fail on the old code and pass on the real fix;
  • a bug hunt's real fix comes with a test that fails without it;
  • an answer key passes a reference answer and fails plausible but lazy answers.

Tasks that stop telling models apart, or that turn out to punish valid alternative solutions, are retired.

Scoring

task   = attempts passed / attempts
kind   = average of its tasks
overall = weighted average of the kinds

The weights roughly follow how much of my working time each kind of task takes:

  • The interval on the leaderboard comes from resampling the tasks within each kind thousands of times and recomputing Overall. With tasks, a few points between two models is usually noise.
  • Solved every attempt counts tasks a model passed all times. It's the number I care about most when deciding whether to trust a model with a kind of work.
  • Each model's latest run counts. When a model is re-run (a new CLI version, different settings), its new results replace its old ones. Models are compared on the same suite version only.

Keeping it fair

  • No looking things up. Transcripts are scanned for GitHub, issue tracker and web lookups, and any such attempt is flagged.
  • No peeking at answers. An attempt that reads the eval's own files, where the answer keys live, is disqualified and scored as a fail.
  • Harness failures don't count against a model. If a checkout, install or grader breaks, the attempt is thrown out and run again.
  • Same task, same prompt. Every model gets identical instructions and inputs, and runs at the same reasoning effort where the CLI allows it.

Limits

  • It's small. tasks is enough to see large differences, not small ones. Read the intervals.
  • It's one engineer's work. One product, one stack (TypeScript across an API, a web app and a mobile app), one person's idea of a good answer. Your work may rank models differently.
  • It measures model plus agent. Each model runs in its own vendor's CLI, so the score covers the tools and system prompt as well as the model.
  • The judge is a Claude model. Its checklist is narrow (is this specific point present?), but answer-key tasks may still lean slightly towards Claude's writing style.
  • Ticket grading can be strict. The real change's tests sometimes check details the ticket doesn't spell out, so a correct alternative fix can fail. Tasks where this happens are retired.
  • Cost isn't always known. Only some agent CLIs report what a run cost.
  • You can't rerun it. Keeping the tasks private keeps them out of training data, but it also means you have to take the methodology on trust.

See the results →