Summary / The 27 components
02 Tool selection and tool calling
When the model should reach for a tool instead of continuing to reason, and how a tool request is validated.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
What is built now
One trigger: after every edit or write, the harness compiles the file and appends any syntax error to the same tool result, so the model sees it on its next turn.
What is missing
Request validation with a stated reason, first-class ask_user and stop tools, and the symbol trigger (dotted names in a tool result prefetch up to three symbol lookups).
Reviewer Claude, 2026-10-04; close-out verdicts 2026-10-23
Reviewer's read: Closed 10-22: no detectable gain (hg gate bundle)
Pi already handles tool choice and schema validation, so most of this spec had nothing to replace. Its loop was removed in the reconciliation and its counterfactual metric dropped. What is left is a handful of small triggers, each of which has to earn its place as an ablation. The lint trigger shipped inside the gate bundle (hg), which measured as costing nothing (Δ −0.7, CI [−8.0, +6.7]), but it was never measured on its own.
Close-out, 10-22 (pre-registered, boxed, Claude): hg (gates + budgets) against h0b on verified-hard 38 + pro-hard 20, 3 seeds each: 102 against 94 cells (verified 61/114 against 55, pro 41/60 against 39). Gain +2.7 tasks of 58, paired bootstrap CI [−1.3, +7.0]; 4 regressions on tasks h0b always solved (limit 2); 1.2× h0b cost per cell. No detectable gain on resolve rate at this model. Terminal under PREREG closeout-20261021: not re-run.
How it is measured no detectable gain (hg, 10-22)
Each trigger as its own arm against h0-v41; the symbol trigger kept only if the ablation shows a gain.
Adoption rules shared by every arm: How the tests work.
Research & attempts
- When to Act: A Tool-Calling Policy for a Weak-Model Coding Harness Paper 2026-09-05
- Build Spec 02 — Tool-Calling Policy (the ≈80% fraction) Spec 2026-09-04
Score history
No scores yet. The first entry is the baseline for suite v1.
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.