Piper · Coding harness
Log in

Coding harness

Closed 10-23. Piper's coding-harness programme (27 components of a coding agent, each with her paper and spec, plus the benches and harness arms built to measure them) is finished: every component has a final verdict. 9 pass their checks (01, 09, 11, 17, 18, 19, 23, 24, 27), 17 showed no detectable gain on a pre-registered A/B, and 3 closed without a bench run (16 not measured, 20 not measurable, 08a native to pi). Nothing beat the plain single-session arm h0b, which stays her default. The harness is frozen at 0.9.0 under version control.

How the close-out ran (10-21 to 10-23). The operator asked for the programme to be put to bed: Claude ran it under one pre-registration, a $100 cap, and the rule that no detectable gain at this model is a final verdict. Yardstick: boxed verified-hard 38 + pro-hard 20, 3 seeds, paired against h0b's 10-18 rows; KEEP needs a CI excluding 0, at least +4.2 tasks of 58, no more than 2 regressions on tasks h0b always solves, and no gold. The gate bundle hg (02, 10, 23a, 25, 26) scored +2.7 tasks, CI [−1.3, +7.0]. The 13 components that had no implementation were built for real as four stages and run together as hx (localize 03-08, repro-first 12/13, repair and reflection 14/15, critic 21/22): +3.0 tasks, CI [−1.0, +7.0], at 2.9 times the cost. It gained 10 cells on verified and lost 1 on pro. Both came out no detectable gain, and both had 4 regressions on tasks h0b always solves, so neither is kept and there is no halving round. Gold-free selection (16) was pre-registered to run only on a kept arm, so it closes not measured; memory (20) is not measurable on this bench; compaction (08a) is native to pi. The infrastructure components (01, 09, 11, 17, 18, 19, 23, 24, 27) pass their checks (10-23): test suite green, hostile corpus 40/40 with all 40 checks going red when disabled, grader canary gold 5/5 and empty 0/5, box selftest 7/7. The check found one real fault: since the move to the new machine the sandbox could not start the harness's own python, which broke six tests in execution, test running and acceptance. It is fixed; the boxed bench never used that path. Bench spend for the whole push: $15.73. The programme is closed: every component has a final verdict, the harness is frozen at 0.9.0 under version control, and her default stays h0b, the plain single-session arm.

Before the close-out. Piper's coding-harness programme: 27 components of a coding agent, each with a paper and a spec she wrote in September, and the benches and harness arms built to measure them. The rule was paper → spec → build → A/B: nothing is kept unless a pre-registered comparison on hidden tests says it helps. It got as far as the measurement rig, the workflow driver, a proven sandbox for the harness's own runs, a security plane in observe mode on her real workers, and one adopted change, a three-session vote (+11 points on dev, confirmed on held-out), now unconfirmed: a 10-18 audit found the bench tools could reach the internet and the answer store (h1-v41 leaked in 61% of its draws), and the first boxed re-run on 10-19 gave h1-v41 21/38 against h0b's 18/38 per seed, +7.5 points with CI [−3.1, +18.0], at about nine times the cost per task: the pre-registered rule says undetermined. Both arms ran the same model: h0b names deepseek-v4-flash, which is only an alias of deepseek-flash (DeepSeek retired V4 Flash on 09-10; both ids now return the same model, V4.1-Flash-1017 since 10-17, checked 10-19). A 10-19 note here said otherwise and was wrong. The vote does filter its pool on the graded PASS_TO_PASS list, so the +7.5 is not gold-free. A gold-free re-test (h1c-v41 against h0-v41) was pre-flighted on 10-19 and came back NO-GO: its nearby-test check was unusable on all 5 tasks, so the mechanism never ran. It was on hold from 09-27 until 10-10, when iterative repair (14) was built and given its first A/B. That run did not count (a parser bug and a gold-list verifier). The fixed re-run on 10-11 (h14b against h0b) is not kept: +5 cells on dev but −2 on the regression set, and the repair loop started in only 1 of 68 cells, so seeded repair is still effectively unmeasured (see 14). On 10-12 Claude took over 14 with a reproduction-script trigger. It is precise (0 false fires) and fixed every failure it caught, but on a harder set the first draws almost never failed, so the paired run stopped as inconclusive. A 10-14 check on tasks rated 1-4 hours found the same: 5 of 6 solved on the first try. Component 14 is parked until there is a task source this model fails. Each page below says what is built, what is missing, an independent review, and how it is measured.

Pages scored10/47
Suites retired9
Interventions0
Bench spend$50.86

How the tests work →

Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.

"Reviewer read" is a one-line judgement, not a score, revised after each test round. The detail is on each component's page.

The 27 components 29

Piper's paper-and-spec series, one per component (plus 08a and 23a). Numbers follow her specs.

#ComponentReviewer readSuiteBaselineLatestChangeLive useTrend
01 Coding harness (REPAIR and primitives) Passes check (10-23): full test suite green — — — — — not scored
02 Tool selection and tool calling Closed 10-22: no detectable gain (hg gate bundle) — — — — — not scored
03 Repository reconnaissance Closed 10-22: built as the hx localize stage; no detectable gain — — — — — not scored
04 Code and context retrieval Closed 10-22: built as the hx localize stage; no detectable gain — — — — — not scored
05 Issue and fault localization Closed 10-22: built as the hx localize stage; no detectable gain — — — — — not scored
06 Dependency and impact analysis Closed 10-22: built as the hx localize stage; no detectable gain — — — — — not scored
07 Task decomposition and planning Closed 10-22: built as the hx localize stage; no detectable gain — — — — — not scored
08 Context construction Closed 10-22: built as the hx localize stage; no detectable gain — — — — — not scored
08a Context compaction Closed: in base (pi's own compaction) — — — — — not scored
09 Patch and code generation Passes check (10-23): gold patches 5/5, empty 0/5 — — — — — not scored
10 Editing mechanism Closed 10-22: no detectable gain (hg gate bundle) — — — — — not scored
11 Code execution Passes check (10-23), after a sandbox fix — — — — — not scored
12 Runtime instrumentation and traces Closed 10-22: built as the hx repro-first stage; no detectable gain — — — — — not scored
13 Error interpretation and debugging Closed 10-22: built as the hx repro-first stage; no detectable gain — — — — — not scored
14 Iterative repair Closed 10-22: built as the hx repair stage; no detectable gain — — — — — not scored
15 Reflection after failure Closed 10-22: built as the hx reflect stage; no detectable gain — — — — — not scored
16 Candidate generation and competing hypotheses Closed: not measured (no kept arm to select over) — — — — — not scored
17 Test execution Passes check (10-23), after a sandbox fix — — — — — not scored
18 Validation and patch acceptance Passes check (10-23), after a sandbox fix — — — — — not scored
19 Regression protection Passes check (10-23), after a sandbox fix — — — — — not scored
20 Episodic coding memory Closed: not measurable on this bench — — — — — not scored
21 Multi-agent development Closed 10-22: built as the hx critic stage; no detectable gain — — — — — not scored
22 Independent reviewer (critic) Closed 10-22: built as the hx critic stage; no detectable gain — — — — — not scored
23 Workflow and state machine Passes check (10-23): drove all 348 close-out cells, 0 errors — — — — — not scored
23a Recovery-transition agent loop Closed 10-22: no detectable gain (hg gate bundle) — — — — — not scored
24 Execution sandbox Passes check (10-23): box 7/7, hostile 40/40 — — — — — not scored
25 Permissions and least privilege Closed 10-22: no detectable gain (hg gate bundle) — — — — — not scored
26 Agent security and prompt-injection resistance Closed 10-22: no detectable gain (hg gate bundle) — — — — — not scored
27 Benchmarking and objective scoring Passes check (10-23): grader and hostile canaries hold — — — — — not scored

Benches 6

What the harness is measured on.

#BenchReviewer readSuiteBaselineLatestChangeLive useTrend
B1 SWE-bench Verified rig Trustworthy and saturated — — — — — not scored
B2 piperbench Small; used for smoke only — — — — — not scored
B3 Hostile corpus All green; scope is the mechanisms v1 100 100 0 —
B4 Coding holdout At ceiling; useful only for regressions v1 1 run 100 — —
B5 Replay bench The right yardstick for her own work v1 1 run 50 — —
B6 RSI loop graded set Live and unsaturated v1 1 run 76.2 — —

Harness arms 12

Each arm is one configuration run through the benches and compared with its parent.

#ArmReviewer readSuiteBaselineLatestChangeLive useTrend
A1 baseline-flash Control; numbers from the old model v1 1 run 78.3 — —
A2 h0 (plumbing) Neutral, as intended v4p +4 earlier 1 run 65 — —
A3 hg (gate bundle) Closed 10-22: no detectable gain v4p +2 earlier 1 run 68.3 ▲ +3.3 vs h0 —
A4 h1 (three-session vote) Adopted by rule on both pre-box sets; unconfirmed on boxed rows (10-19) v4 +2 earlier 1 run 55.3 ▲ +7.1 vs h0 —
A5 h2p (repair prompt) Registered, never run — — — — — not scored
A6 picker-v41 (tests as selector) Tested; the vote won v2 1 run 91.7 — —
A7 oracle_k (keep every patch) Selection is nearly solved — — — — — not scored
A8 Diversity probes (pk4–pk7) No headroom to show anything — — — — — not scored
A9 Worker gates in production Observing cleanly — — — — — not scored
A10 h14 (seeded retry) Keep rule not met; the loop barely ran — — — — — not scored
A11 h14b (seeded retry, scope verifier) Not kept: regression −2; the loop ran once — — — — — not scored
A12 hx (close-out bundle) No detectable gain (10-22) v4p +1 earlier 1 run 63.3 — —