How good are coding agents at my actual job?
tasks from my day-to-day work as a software engineer, each with a right answer that is already known. Every model gets attempts at each task, in its own agent CLI, with no internet. How it works.
Leaderboard
Share of attempts that passed. Overall weights each kind of work by how much of my time it takes.
Strengths at a glance
Pass rate for each kind of work.
Every task, every attempt
Each column is one task and each stripe one attempt. Within each kind of work, tasks run from most to least often solved. Tasks are numbered, not named: they stay private so they can't leak into training data.
Consistency, speed and cost
A model that solves a task once in three tries is less useful than one that solves it every time.
Time is wall-clock per attempt, including the agent running tests. Cost is what the agent CLI reports; the Codex CLI does not report one.