Piper · Coding harness
Log in

Summary / Harness arms

A3 hg (gate bundle)

h0 plus every gate at once: tool and bash allowlists, protected test paths, edit-before-read, lint feedback, the verbatim-lift rule, budgets of 150 turns and 800K tokens.

Latest68.3v4p
vs h0+3.3same suite, v4p
Live use—last 7 days

Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.

What is built now

Dev (09-08): 114/150 against h0's 115, Δ −0.7, CI [−8.0, +6.7], regressions unchanged, +$0.001 per task. Regression set: 23/30, one harness reject, and it was a true positive.

What is missing

Measurement of each gate alone (h2 was smoke-tested, not run), and a re-run on the current model.

Reviewer Claude, 2026-10-04; close-out verdicts 2026-10-23

Reviewer's read: Closed 10-22: no detectable gain

Bundling was the lean path agreed on 09-07: if the whole bundle is neutral, no gate needs its own run to justify safety. That is a safety argument, not a capability one. The gates prevent damage, and the bench confirms they do not cost resolves.

Close-out, 10-22 (pre-registered, boxed, Claude): hg (gates + budgets) against h0b on verified-hard 38 + pro-hard 20, 3 seeds each: 102 against 94 cells (verified 61/114 against 55, pro 41/60 against 39). Gain +2.7 tasks of 58, paired bootstrap CI [−1.3, +7.0]; 4 regressions on tasks h0b always solved (limit 2); 1.2× h0b cost per cell. No detectable gain on resolve rate at this model. Terminal under PREREG closeout-20261021: not re-run.

How it is compared no detectable gain (hg, 10-22)

Bundle against h0 on dev; regression set as the guard.

Adoption rules shared by every arm: How the tests work.

Research & attempts

No documents yet.

Score history

Suite v1 retired
0255075100Sep 876
Suite v4 retired
0255075100Oct 2253.5
Suite v4p current
0255075100Oct 2268.3

Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.

DateSuiteTypeScore±Live useCostNotes
2026-10-22v4psealed 68.3— —$1.80 hg BOXED on pro-hard, k=3: 41/60.
2026-10-22v4sealed 53.5— —$1.47 hg BOXED on verified-hard, k=3: 61/114. Close-out: with pro-hard, +2.7 tasks vs h0b, CI [−1.3, +7.0], NO DETECTABLE GAIN.
2026-09-08v1sealed 76— —— hg on dev, k=3: 114/150. vs h0 Δ −0.7, CI [−8.0, +6.7]. Regression set separately: 23/30, the one harness reject a true positive.

Experimental record

One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.

No interventions recorded yet.