Summary / The 27 components
18 Validation and patch acceptance
Deciding whether a patch fixed the problem without breaking anything else.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
What is built now
accept.py: a two-part verdict, fixed (the reproduction tests now pass) and unbroken (the regression tests still pass), with inconclusive when either side was not run. About 500 acceptance rows. A tests-as-selector picker (picker-v41, see A6) was also built with gold-leak gates.
What is missing
Required evidence level by change class, and the demote-only semantic veto.
Reviewer Claude, 2026-10-04; close-out verdicts 2026-10-23
Reviewer's read: Passes check (10-23), after a sandbox fix
Validated on real data. On the regression set the gate bundle rejected 1 of 30 patches, and it was a true positive. Of the guard's first four false positives, three were the cwd bug in 11 and are fixed. The 09-07 review's point still applies to anything layered on top: a verdict is only as good as the tests the model cannot see, which is why gold tests never reach the selector (see A6).
Close-out, 10-23 (pre-registered check, Claude): the acceptance tests (a real fix is accepted, a regression rejected) pass after the same fix; grader canary gold 5/5, empty 0/5. Passes check. The fix: since the 10-16 move to the new machine, the harness venv's python is a link to an interpreter outside the venv, and the sandbox did not mount it, so every sandboxed run of that python failed (6 tests in execution, test running and acceptance). The sandbox now mounts that interpreter tree read-only; nothing else in the home directory is visible. The boxed bench never used this path, so no score is affected.
How it is measured passes check (10-23, fixed)
Harness verdict against the official SWE-bench grader, per cell: false accepts and false rejects.
Adoption rules shared by every arm: How the tests work.
Research & attempts
Score history
No scores yet. The first entry is the baseline for suite v1.
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.