Winthropic

Methodology :: how the readings are taken

Same battery. Same shape. Every day.

Each day at 07:30 Eastern, every tracked Claude model receives the same versioned battery of tasks spanning arithmetic, extraction, logic, classification, instruction-following, JSON emission, and code. Code answers are executed in a sandbox against test vectors; everything else is scored deterministically (exact match, regex, numeric tolerance, JSON equality). No LLM judges anything.

The request shape is frozen: no system prompt, fixed token limit, temperature 0 where the API accepts it. Claude 5 models reject the temperature parameter and always sample, so their traces carry natural day-to-day variance. That is signal, not noise: the published record is what separates a model's normal variance band from a genuine regression.

Batteries are versioned and never edited in place. A task change means a new battery version and a new trace; readings are only compared within one version. Runs where every call fails (an outage, an auth change) are recorded separately and never overwrite a good day's reading.

Honesty about scale: today's battery is a barometer, not a physics lab. A dozen tasks per model measures direction and stability, not intelligence. The battery grows over time, by versioned addition, and the raw per-task results, including every failure's actual output, are published for each run.