How we measure

Changelog

Every change to how the trackers are run, scored and judged, newest first. Model changes and baseline updates are listed too, because they change what a result is compared with. How the checks work today is on the methodology page.

November 2026

  1. Method Claude CodeCodex

    Three detection checks added; methodology and changelog published

    A same-task check now compares each day with the same tasks in up to 14 earlier runs of the current model. A sustained-drop detector (CUSUM, allowance 0.5, alarm above 3) adds up daily shortfalls against a fixed per-task baseline. Every window test and same-task check on every tracker is corrected together with the Benjamini–Hochberg procedure, and verdicts and "Statistically Significant" labels use the corrected q-values. Detection is paused on both trackers, so these take effect as each new baseline is set.

  2. Site Claude CodeCodex

    Charts mark model changes; old baselines no longer compared during collection

    The 30-day chart marks the day each model's runs began and leaves a gap on days with no published run. While a new baseline is collected, the Change Overview no longer compares results with the previous model's baseline.

  3. Model Codex

    GPT-6.1 Sol at high reasoning effort; new baseline collection

    Daily Codex runs resumed with GPT-6.1 Sol at high effort (Codex CLI 0.167.0 on the first day). Detection is paused while the new baseline is collected. From this date, runs use a different machine than earlier runs.

  4. Operations Claude Code

    Daily runs resumed on a new machine

    Claude Code runs resumed with Opus 5.5 at high effort and continue its baseline collection. From this date, runs use a different machine than earlier runs.

October 2026

  1. Method Claude CodeCodex

    Runner defaults set to GPT-6.1 Sol and Opus 5.5 at high effort

    The daily runner's configurations were set to GPT-6.1 Sol for Codex and Opus 5.5 for Claude Code, both at high reasoning effort. No runs used these settings until November 8.

    Margin-Lab/evals 6648d296

September 2026

  1. Operations Claude CodeCodex

    No runs published

    Neither tracker ran from September 30 through November 7, 2026. Charts show the gap.

  2. Model Codex

    GPT-6 Sol at high reasoning effort; baseline collection began

    Detection was paused for the new model. Six daily runs (September 24–29) were published before the gap.

  3. Model Claude Code

    Opus 5.5 at high reasoning effort; baseline collection began

    Detection was paused for the new model.

July 2026

  1. Method Claude CodeCodex

    Runs with up to five infrastructure failures can be published

    A run counts if at least 45 of its 50 tasks produced a test result and none were canceled. Tasks with no test result because of an infrastructure failure are left out of the pass rate rather than counted as failures. The limit was one such task from July 26.

    Margin-Lab/evals b57dfe92
  2. Method Claude CodeCodex

    Failure rule recorded with every result; publishing automated

    Statistics now record how infrastructure failures are treated ("exclude": left out of the pass rate). Earlier statistics did not record it but used the same rule. Daily results are published to this site automatically once both trackers finish.

    Margin-Lab/evals db241042
  3. Model Claude Code

    Opus 5.0

    Claude Code runs moved to Opus 5.0.

  4. Baseline Codex

    Baseline set to 83.4% for gpt-5.6-sol

    The baseline pass rate was set from 613 passes in 735 graded attempts of gpt-5.6-sol (previously 56%).

  5. Model Codex

    gpt-5.6-sol

    Codex runs moved to gpt-5.6-sol.

May 2026

  1. Model Claude Code

    Opus 4.8; baseline set to 73%

    Claude Code runs moved to Opus 4.8, and the baseline pass rate was raised from 65% to 73%.

April 2026

  1. Baseline Claude Code

    Baseline raised to 65%

    The baseline pass rate was raised from 56% to 65%.

  2. Model Codex

    gpt-5.5 at xhigh reasoning effort

    Codex runs moved to gpt-5.5-xhigh, released that day.

  3. Model Claude Code

    Opus 4.7

    Claude Code runs moved to Opus 4.7.

March 2026

  1. Model Codex

    gpt-5.4 at xhigh reasoning effort

    Codex runs moved to gpt-5.4-xhigh, released March 5.

February 2026

  1. Model Codex

    gpt-5.3-codex

    Codex runs moved to gpt-5.3-codex, released February 5.

  2. Model Claude Code

    Opus 4.6

    Claude Code runs moved to Opus 4.6.

January 2026

  1. Model Codex

    Codex tracking began

    Daily runs began with gpt-5.2 and a baseline pass rate of 56%.

  2. Model Claude Code

    Claude Code tracking began

    Daily runs began with Opus 4.5 and a baseline pass rate of 56%.