Real-work evals by Gui Politta
Updated

How good are coding agents at my actual job?

tasks from my day-to-day work as a software engineer, each with a right answer that is already known. Every model gets attempts at each task, in its own agent CLI, with no internet. How it works.

0models
0private tasks
0kinds of work
0graded attempts

Leaderboard

Share of attempts that passed. Overall weights each kind of work by how much of my time it takes.

    Strengths at a glance

    Pass rate for each kind of work.

    Every task, every attempt

    Each column is one task and each stripe one attempt. Within each kind of work, tasks run from most to least often solved. Tasks are numbered, not named: they stay private so they can't leak into training data.

    Passed Failed Hover a task for details

    Consistency, speed and cost

    A model that solves a task once in three tries is less useful than one that solves it every time.

    Time is wall-clock per attempt, including the agent running tests. Cost is what the agent CLI reports; the Codex CLI does not report one.