Summary / Harness arms
A12 hx (close-out bundle)
h0b plus four stages built for the 10-21 close-out: S1 localize pre-pass (03-08), S2 repro-first with failure digest (13, 12), S4 repro-triggered reflect and repair up to 2 rounds (14, 15), S3 independent critic with one revision (22, 21).
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
What is built now
Built 10-21 by Claude in piperharness/closeout.py, two hooks in the driver, default-off: every existing arm runs byte-identical. Scratch files never enter the patch; the critic pins the change with a commit, never git stash (worktrees share one stash). Smoke on 3 tasks: every stage fired, 3/3 repro scripts valid, critic approved 3/3, box held (curl, pip download and searches for the answer store all failed).
What is missing
Nothing: closed.
Reviewer Claude, 2026-10-04; close-out verdicts 2026-10-23
Reviewer's read: No detectable gain (10-22)
Run 10-21 22:14 to 10-22 11:47, boxed, same model as h0b. Verified-hard 65/114 against h0b's 55; pro-hard 38/60 against 39. Over both sets +3.0 tasks of 58, CI [−1.0, +7.0], 4 regressions on tasks h0b always solved, $0.064 per cell against $0.022. The verified gain did not carry to pro. The smoke showed why it was likely: the model's own reproduction usually passes once its patch is in, while the hidden tests check something the issue text does not show, the same limit c14 hit on 10-12.
How it is compared no detectable gain (10-22)
Pre-registered (closeout-20261021): paired bootstrap against h0b's 10-18 boxed rows over both sets; KEEP needs CI excluding 0, ≥ +4.2 tasks, ≤ 2 always-solved regressions, gold-free.
Adoption rules shared by every arm: How the tests work.
Research & attempts
No documents yet.
Score history
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
| Date | Suite | Type | Score | ± | Live use | Cost | Notes |
|---|---|---|---|---|---|---|---|
| 2026-10-22 | v4p | sealed | 63.3 | — | — | $3.96 | hx BOXED on pro-hard, k=3: 38/60. |
| 2026-10-22 | v4 | sealed | 57 | — | — | $7.14 | hx BOXED on verified-hard, k=3: 65/114. Close-out: with pro-hard, +3.0 tasks vs h0b, CI [−1.0, +7.0], NO DETECTABLE GAIN. |
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.