Roster is an agent that runs performance reviews for your AI coding agents.
It finds the setup that's quietly making your codebase worse, tests a better one on your own past tasks, and ships it only if nothing regresses.
The result
| Only the new setup passed | 11 |
| Only the old setup passed | 1 |
| Both passed | 20 |
| Neither passed | 3 |
| Valid pairs | 35 |
| McNemar exact test | p = 0.006 |
| Call | Verdict | Why |
|---|---|---|
| Original gate (v1) | promote | tie on hidden_pass 0; candidate cheaper $0.4385 < $2.5181raw reasonstie on hidden_pass 0; candidate cheaper $0.4385 < $2.5181 |
| Revised gate (v2), same tasks | reject | fewer hidden-test passes than the incumbent (0 vs 1); 1 regression; 1 scope violationraw reasonshidden_pass 0 < incumbent 1; 1 regression(s) after reruns: ea846f90-734e-4fb0-bd01-0ce8658c6b0c; 1 scope violation(s): 8d77769a-b6f0-44cb-b3d4-928acc638e1b |
| Forward test of v1's call | 11 vs 1 passed, p = 0.006 |
Our own AI team's review level A
| Setup | Sessions | Spend | Unattended |
|---|---|---|---|
| Claude Code · claude-opus-5-56a200cd990a7 | 21 | $23.54 | 100% |
| Claude Code · claude-sonnet-5-5 (rules 243fa2)97433f7b4446 | 15 | $6.37 | 53% |
| Claude Code · claude-sonnet-5-5 (rules dabdd0)a9544a002085 | 60 | $10.79 | 58% |
| Claude Code · claude-haiku-5-5c8d6907735ec | 18 | ≥ $2.07 | 56% |
| Claude Code · deepseek-flashf656e13e2af0 | 40 | $1.26 | 63% |
| All setups | 154 | ≥ $44.03 |
How it works
Roster reads every recorded agent session and scores what happened to its change: kept, reverted, or reworked by a person. This points at a suspect setup but proves nothing on its own.
Roster reruns the suspect's real past tasks at their parent commit, under the current setup and under candidates, and grades every attempt with hidden tests. A candidate is promoted only if nothing regresses.
New tasks, written down before the decision, run through the old and the new setup side by side. This checks that the decision holds on work that did not exist yet.
The experiment
Roster built itself with several agent setups, and its first review of this repo picked which setup to promote. Before that review ran, we froze a backlog of new tasks and their acceptance tests and published their hashes in FREEZE.txt. The hypotheses, tests and stop rules are fixed in PREREGISTRATION.md. Results get published whatever they show.
Built with Entire, neatlogs and cfo.ai
- Entire records every coding agent session with its prompts, tokens and commit, and Roster reads those records as its input.
- neatlogs traces Roster's runs, from ingest to each replay attempt and gate decision. The first and second gate calls on the same tasks are traces
820af0a5dd1c888b70af931671d9cbb8and38f71c40ceb5fca5b7046542fa9e17f2, re-emitted from the recorded results, so span timings are emission times. - cfo.ai is where Roster books the AI team's spend per setup as a department in the company plan. Read-only: business plan, AI workforce book.