How we measure

Methodology

MarginLab runs AI coding agents on the same 50 real software tasks every day and checks whether their results have changed. This page explains what runs, how each task is scored, and how we decide that a drop is real rather than day-to-day noise. Every change to this method is listed in the changelog; the most recent was on Nov 14, 2026.

Current setup

What runs every day

Each tracker runs one agent on a fixed set of 50 tasks from SWE-Bench Pro, a benchmark of real issues from open-source repositories that was built to resist training-data contamination. The task list does not change from day to day, so every result can be compared with the same tasks on earlier days.

  • Each task is attempted once per day.
  • Agents run in their own command-line tools (Claude Code, Codex CLI) with no custom harness, using the latest release of each tool. Every run records the tool's version, so a change can be traced to a model or a release.
  • Runs start at 06:00 UTC. Codex runs first and Claude Code right after it; results are published once both finish.

How each task is scored

A task passes when the agent's changes make the task's tests pass. A task that runs but fails its tests counts as a failure. A task that produces no test result because of an infrastructure problem (a container error, the agent process failing, an unknown runner failure, or the task hitting its time limit) is left out of the pass rate, so outages are not counted against the agent. It still counts among the tasks run that day.

A day's run is published only if at least 45 of the 50 tasks produced a test result and none were canceled.

Pass rates and uncertainty

The pass rate is passes divided by tasks with a test result. We report it for the latest day and pooled over the last 7 and 30 calendar days, each with a 95% Wilson score interval.

Because the same tasks run every day, most of them pass every day or fail every day; on stable stretches only about 13 to 16 of the 50 change from day to day. A day's pass rate therefore moves about 2 to 4 points, roughly half of what a model of 50 independent coin flips would predict (5 to 7 points). The same-task check and the sustained-drop detector below use that structure. The window tests do not, which makes them conservative.

Baselines and model changes

A baseline is the pass rate a tracker is expected to hold for its current model. When the model changes, results are no longer comparable with the old baseline, so we collect a new one. While that happens, detection is paused, pass rates and intervals stay visible, and the tracker says so at the top of its page. Charts mark the day each model's runs began, and earlier models stay on the historical performance pages.

How we detect a drop

Each day, four kinds of check run against the latest results.

1. Window tests (1, 7 and 30 days)

Every graded task in the window is pooled and compared with the baseline pass rate using an exact one-sided binomial test in the direction of the change. The threshold drawn on the Change Overview is roughly the smallest change that would reach p < 0.05 for that window's sample size, capped at 15 points. These tests are computed by the daily pipeline.

2. Same tasks, today versus earlier runs

Today's results are compared with the same tasks in up to 14 earlier runs of the current model; at least 7 are needed. Each task's pass rate in those runs is estimated as (passes + 0.5) ÷ (attempts + 1), which leaves a small chance of failure for a task that always passed. If nothing has changed, today's number of passes follows a Poisson-binomial distribution with those rates, and the p-value is the chance of a result as low as today's or lower. This catches a sudden drop on the day it happens.

3. Sustained drop (CUSUM)

When a baseline is set, each task's pass rate over the baseline days becomes the reference (at least 7 days). For every later day we compute a drop score, z = (expected passes − observed passes) ÷ standard deviation, from those per-task rates. A cumulative sum adds the scores up:

S(today) = max(0, S(yesterday) + z(today) − 0.5)

A sustained drop is flagged while S is above 3. Small shortfalls that would not stand out on any single day add up, and the score falls back toward zero as results recover. The baseline days are shown on each tracker page once detection is on.

4. Correction for multiple comparisons

Every window test and same-task check on every tracker with detection on is corrected together each day using the Benjamini–Hochberg procedure. A result is significant when its adjusted q-value is below 0.05, which keeps the expected share of false alarms among flagged results at or below 5%, however many agents we track. The procedure stays valid when tests are positively related, as overlapping windows are. Tracker pages show both the raw p-value and the adjusted q-value.

The verdict

A tracker shows Degradation detected when any of these is true:

  • a window test finds a drop with adjusted q below 0.05;
  • the same-task check finds a drop with adjusted q below 0.05;
  • the sustained-drop score is above 3.

Otherwise it shows Nominal, or Collecting baseline while detection is paused.

Tested on our own history

Before turning these checks on, we replayed them on the published results from January to September 2026.

  • On Claude Code's May 2026 drop, both the same-task check (p < 0.001) and the sustained-drop detector flagged May 23, the first day of the drop.
  • Across six earlier model periods with no reported regression (about 250 monitored days), the sustained-drop detector raised no alarms. Across nine periods, the same-task check fell below p = 0.05 on 3 of 343 days.
  • In simulations that removed passes from those periods, a sustained 5-point drop was flagged within 20 days 90% of the time (median 6 days), and a 10-point drop within about 2 days.

Independence and corrections

MarginLab is independent of the companies whose agents it tests. No one can pay to change, delay or preview a published result. We report what we measure, show the uncertainty, and correct mistakes in public; corrections and method changes go in the changelog.

Limitations

  • One attempt per task per day is a small daily sample; the checks above are built for it.
  • Results cover 50 SWE-Bench Pro tasks. They show changes on those tasks, not an agent's overall quality.
  • Agents run on their latest releases, so a change can come from the model or from the tool. The recorded version helps tell them apart.
  • Tasks that hit their time limit are currently left out rather than counted as failures. They are rare (2 of about 2,000 Claude Code attempts and none for Codex in the 40 runs before November 14, 2026), and we plan to count them as failures; that change will appear in the changelog.
  • All runs come from one machine. A change of machine or network can shift results, so we list such changes in the changelog.

Code and data

The daily runner and the window tests are open source in Margin-Lab/evals (ops/daily-runs). The same-task check, the sustained-drop detector and the correction are computed when this site is built, from the per-task results published with each run, using exactly the formulas above.