Piper · Coding harness
Log in

Summary / The 27 components

02 Tool selection and tool calling

When the model should reach for a tool instead of continuing to reason, and how a tool request is validated.

Latest—not scored
Baseline— 
Live use—last 7 days

Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.

What is built now

One trigger: after every edit or write, the harness compiles the file and appends any syntax error to the same tool result, so the model sees it on its next turn.

What is missing

Request validation with a stated reason, first-class ask_user and stop tools, and the symbol trigger (dotted names in a tool result prefetch up to three symbol lookups).

Reviewer Claude, 2026-10-04; close-out verdicts 2026-10-23

Reviewer's read: Closed 10-22: no detectable gain (hg gate bundle)

Pi already handles tool choice and schema validation, so most of this spec had nothing to replace. Its loop was removed in the reconciliation and its counterfactual metric dropped. What is left is a handful of small triggers, each of which has to earn its place as an ablation. The lint trigger shipped inside the gate bundle (hg), which measured as costing nothing (Δ −0.7, CI [−8.0, +6.7]), but it was never measured on its own.

Close-out, 10-22 (pre-registered, boxed, Claude): hg (gates + budgets) against h0b on verified-hard 38 + pro-hard 20, 3 seeds each: 102 against 94 cells (verified 61/114 against 55, pro 41/60 against 39). Gain +2.7 tasks of 58, paired bootstrap CI [−1.3, +7.0]; 4 regressions on tasks h0b always solved (limit 2); 1.2× h0b cost per cell. No detectable gain on resolve rate at this model. Terminal under PREREG closeout-20261021: not re-run.

How it is measured no detectable gain (hg, 10-22)

Each trigger as its own arm against h0-v41; the symbol trigger kept only if the ablation shows a gain.

Adoption rules shared by every arm: How the tests work.

Research & attempts

Score history

No scores yet. The first entry is the baseline for suite v1.

Experimental record

One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.

No interventions recorded yet.