Summary / The 27 components
12 Runtime instrumentation and traces
Seeing variable values, execution paths and state at run time, beyond the error text.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
What is built now
Nothing.
What is missing
Everything, and first a definition of the "passing run" a trace would be compared against.
Reviewer Claude, 2026-10-04; close-out verdicts 2026-10-23
Reviewer's read: Closed 10-22: built as the hx repro-first stage; no detectable gain
The spec's first-divergence diff against a passing run is a good idea with a missing input: most SWE-bench failures have no passing sibling run on the same code. Until that is defined, there is nothing to build.
Close-out, 10-22 (pre-registered, boxed, Claude): built for real as stage S2 (repro-first + error digest, lite) of the close-out bundle arm hx (A12): h0b + localize pre-pass + repro-first with failure digest + repro-triggered reflect/repair (2 rounds) + critic with one revision. hx against h0b, verified-hard 38 + pro-hard 20, 3 seeds: 103 against 94 cells (verified 65/114 against 55, pro 38/60 against 39). Gain +3.0 tasks of 58, paired bootstrap CI [−1.0, +7.0]; 4 regressions on tasks h0b always solved (limit 2); 2.9× h0b cost per cell. No detectable gain at this model. The pre-registered rule closes every bundled component together, with no halving round. Terminal: not re-run.
How it is measured no detectable gain (hx, 10-22)
Not before 13 exists to use it.
Adoption rules shared by every arm: How the tests work.
Research & attempts
Score history
No scores yet. The first entry is the baseline for suite v1.
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.