Piper · Amplification
Log in

Summary / Component 5 of 24

5 Outcome attribution

Determining whether the change actually caused the observed improvement.

Latest93.8v2
Baseline93.8Oct 13
Live use—Nov 16
Re-read87.5Oct 23 · not official

Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.

Layer 1 — Improvement loop mechanics. Can Piper reliably complete and learn from a self-improvement cycle?

What her system has now

A change ledger (state/change_ledger.py 1.0.0, 10-12). Every change to her own machinery records the file, versions before and after, why, the predicted effect, a resolve-by date and how to revert. She built it after noticing that six changes landed on 10-12 and not one carried a prediction anything would ever check.

A resolver (state/change_ledger_resolver.py 1.0.1). For each entry it reads the signal source the prediction names and returns RESOLVED, WRONG or NOT-RESOLVABLE-YET, always with a row count and a path, never from the prediction's own wording, and never on fewer than 3 rows. It ran for the first time at 00:21 on 10-13: 3 entries due, 1 settled (WRONG), 2 honestly left open with what they still need.

Rules set before the measurement. The ingest switch went on at 20:36 on 10-12 with a stated rule (median lookup latency within 1.2× of before). When the reading came in over it, she reverted at 00:25 and wrote the numbers down. Nothing was softened.

Since 10-13: resolver 1.0.3 builds the ingest entry's blocks from the row marker (true off / on / off-again windows); 1.2.0 (10-23) settles rows from check specs attached as data. attribution.py 1.1.0 (10-26) can label rows from a recorded switch history, but nothing calls it yet.

What is missing

  • Most evaluators check that the mechanism fires, not that it helped. Five of the seven ask whether a log line or skip reason appears. That is change execution (#4). One (transfer.py 1.4.2) predicts a drop in empty lookups but can never settle, because the log does not record the signal it names; the resolver says so itself. Only the ingest entry compares a before and an after.
  • That comparison is unpaired. It sets lookups in the hours before the change against lookups in the hours after, with nothing to separate the change from time of day, workload or another change landing nearby.
  • One hand-written evaluator per entry. Seven entries, seven functions. A new change cannot be judged until someone writes code for it.
  • No restart or version markers in the logs, so the transfer.ts entry cannot settle ("inert before restart" and "kills unrecorded" look the same).
  • Manual. It runs when a seat task asks it to; nothing is scheduled.
(versions updated to live on 2026-10-23 where the file was found)

Reviewer Claude, 2026-10-28

Reviewer's read, before testing: Judges its own changes on pre-set rules; the resolver separates on/off blocks only where a run marker exists, and no switch history is recorded

The discipline is real and rare: a prediction written before the change, a rule fixed before the reading, a revert when the rule said so, and the numbers kept. The weak point is the attribution itself, and the one settled entry shows it.

The ingest entry was settled WRONG on a 1.3× median ratio. Re-reading the same log (state/transfer/transfer.jsonl, real lookups only, bench rows excluded, transfer.py 1.4.2 throughout; bootstrap 95% intervals on the ratio of medians):

WindowLookupsMedian
Before (10-12 evening, ingest off)443.3 s
Ingest on (20:34 → 00:26)794.3 s
Ingest off again (00:26 → 08:50)683.9 s

Three measurements of the same change now disagree: the 10-11 bench read 1.64× (on transfer.py 1.4.0, before the 1.4.1 fix cut median lookup time from about 16 s to about 3 s, so a different program); her paired replay of 22 stored inputs on 10-12 read 0.97×; the live before/after read 1.3×. The revert followed her rule correctly. Whether ingest actually costs 20% latency is not established. This is the gap #5 measures: the ledger can say the number moved, not that the change moved it.

Update 2026-10-26 — attribution.py 1.1.0: switch-history labelling, built for a defect she had already fixed. Nothing calls it

What she did (deliverable d-ff457b37, $0.02). The operator asked which components she could enhance to open new paths. She picked #5, quoting the 10-13 card on this page: the resolver hands attribution a before/after wall, so 404 of 483 "on" rows really ran with ingest off. She added load_switch_history, switch_state_at, label_rows, blocks_from_labels and an --report CLI to foundation/core/attribution.py (1.1.0). Rows with no recorded state come back unknown and are dropped from both sides instead of guessed. Her new test passes 12 of 12 (Claude re-ran it), and the 1.0.0 self-test still passes 4 of 4. The design is right: no guessing, and the row's own marker wins over the history.

What checking found (read-only, $0):

Score. Unchanged: v1 75, v2 official 93.8. No bench was re-run; the seat suite tests her reading of latency logs, which this change does not touch. The live measure (settled rows whose verdict rests on a drift-controlled design) is unchanged too, because nothing settled through the new code.

What would count. A real switch history (ingest, the attention card, the card holdout flips) written when a switch flips, and the resolver using it for every switched change rather than for one hand-coded evaluator.

Update 2026-10-23 — 74 overdue predictions graded from her logs: mechanism claims hold, effect claims mostly don't, and 1 of 74 controls for drift

Why. Her change ledger held 836 rows with 27 settled. The resolver runs four times a day but has no evaluator for a free-text prediction, so all 74 predicted rows past their resolve-by date sat at "no evaluator registered". A result nobody settles can't be attributed, right or wrong.

What was done (read-only, $0 on her books): five independent graders read the source each prediction names, graded it as written, and named any confounder. Claude reviewed the grades and moved 3 low-confidence or confounded RIGHTs to cannot-tell.

Prediction claimedRightWrongCannot tellNot evaluable
A mechanism (a test passes, a refusal fires, a field exists)29341
An effect, read as a before/after count131292
Paired or controlled design1000

Reading. When she predicts that something will work, she is usually right (29 of 36). When she predicts how much a number will move, she is right about a third of the time (13 of 36), and 9 more cannot be separated from drift: a second change in the same window, a hand fix in the same minute, a project reopened, a window redefined. Only 1 of 74 predictions used a design that controls for drift. That is the #5 gap in her own record: attribution.py exists and works, but her predictions are not written in a form it can test.

The grades are not written into her ledger. They are held as a blind check on her own settles (next card on #6): when she settles a row herself, it is compared with the independent grade. Agreement on the effect rows is the live #5 measure. The 10-13 ingest row already reads cannot-tell with baseline drift named. No score changed.

Update 2026-10-13 — attribution.py built and wired into the resolver; the module works, the wiring does not use it

What she did (deliverables d-8a0442, d-129bd677). foundation/core/attribution.py 1.0.0 compares like-for-like blocks (on/off/on, or a paired replay) instead of a before/after wall, bootstraps the ratio, returns caused, no-effect or cannot-tell, and names the confounder when the baseline itself moved. This is the design the review above asked for. Its self-test passes 4 of 4 (re-run 10-13 evening), including the reviewer's pre-flight case: the off-again window posed as "after" now reads cannot-tell with baseline drift named, where the old resolver read a slowdown. The resolver (change_ledger_resolver.py 1.0.2) now settles the ingest entry through it. Ledger row cl-20261014-attribution-1.0.0 says honestly that it was written after the code landed (a write-set refusal), so its prediction is not pre-registered.

v2 bench: she submitted it through the amp-bench API at 13:57. Result 93.8 ± 12.2 against the accepted 93.8: INCONCLUSIVE, no change. Expected, because the seat suite tests her own reading of latency logs, and that was already strong before the module (one miss: an overcautious null call).

What I found checking it (read-only, $0). Run through the resolver without --apply, the ingest entry now reads cannot-tell, 1.21× [1.13, 1.25], which is what her ledger row predicted. But the resolver feeds the module only two blocks: before, and every lookup since the change. Ingest was switched back off at 00:26, so 404 of those 483 "on" rows ran with ingest off. All 75 rows with a register shortlist fall in the real on-window, 20:34 to 00:26. The verdict is right but the reason is wrong, and no confounder is named. The same module given the true blocks (off 44 / on 79 / off-again 404) says:

So the tool is right and the wiring undoes it: the resolver still builds a before/after wall and hands it over. The fix is small: build blocks from the transfer.json switch history (or the n_short field) rather than "since the change". The stored ingest row still says WRONG (settled 10-13); nothing has re-settled it. The resolver still runs only when asked.

Test suite built · amp-bench v2, sealed items through her seat, about $0.40 · official 93.8 · re-read 87.5 (10-23) · no KEEP possible at this n (smallest detectable gain 17.3 > headroom 6.2)

Pre-flight ($0, about 10 minutes): run the resolver's ingest evaluator with the off-again window posing as "after". If it reports a slowdown for a change that never happened (it should: 1.18×), the test can fail. Then check that a paired or switch-back design (on/off/on/off in short blocks, or replay of stored inputs) does not.

Items: change histories built from her real logs where the signal moves for a reason other than the change: a version bump in the same hour, a workload shift, overnight versus daytime traffic, a second change landing nearby. Some items have a real effect, some have none. Counted: her verdict against the known truth, and whether she names the confounder. Judged (blind, other model family): whether her proposed re-measure would actually separate the two.

Live use: share of ledger entries settled in the last 7 days whose verdict rests on a design that controls drift (paired, switch-back or a concurrent control). Today: 0 of 1.

Scoring rules shared by all 24 suites: How the tests work.

Research & attempts

No documents yet.

Score history

Suite v1 retired
0255075100Oct 1375
Suite v2 current
0255075100Oct 1393.8

Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.

DateSuiteTypeScore±Live useCostNotes
2026-10-23v2reread 87.516 —$0.31 RE-READ 10-23 (Claude; not her submit, the official 93.8 stays). n=8. Failed categories: overcautious / wrong null call 2.
2026-10-15v2reread 90.612.9 —$0.10 RE-READ 10-15 (Claude, full refresh; not her submit, accepted 93.8 stays). n=8. Failed categories: overcautious / wrong null call 2.
2026-10-13v2trial 93.812.2 —$0.31 INCONCLUSIVE via the amp-bench API (cl-20261014-attribution-1.0.0). n=8. Failed categories: overcautious / wrong null call 1.
2026-10-13v2sealed 93.812.2 —$0.11 BASELINE via the amp-bench API (set by Claude at install). n=8. Failed categories: overcautious / wrong null call 1.
2026-10-13v1sealed 7512.5 —$0.00 v1 summary score, UPDATED 10-13 evening ($0, was 50): E exists 25, P in her default path 12.5, M measured 25, W measure shows it working 12.5. ± 12.5 = one rubric step. attribution.py 1.0.0 compares like-for-like blocks and names the confounder (self-test 4/4) and the resolver now calls it (+M: the v2 seat bench measures #5, 93.8 twice). Still partly: the resolver runs only when asked and feeds the module before vs everything-since (404 of 483 'on' rows were ingest-off), so the right verdict comes for the wrong reason; the v2 score did not move. Was: read off the 10-13 review, no bench run. Rubric = 4 checks x 25 (yes 25 / partly 12.5 / no 0): E exists 25, P in her default path 12.5, M measured 12.5, W measure shows it working 0. ± 12.5 = one rubric step. Ledger plus resolver judge changes on pre-set rules, but the resolver runs only on request, just 1 of 7 rows is settled, and the one settled verdict can't separate the change from drift.

v2 bench LIVE (2026-10-13, amp-bench API: Piper submits a change and gets KEEP / REVERT / INCONCLUSIVE / SATURATED back; v2 rows below): #5 Outcome attribution, v2 (seat suite, about $0.50 a run). 8 sealed latency logs from a lookup service after a change meant to make it faster, answered blind through your Day seat with your current code and tools. Some carry a real effect, shown by a balanced switch-back or a counterbalanced paired replay. Some carry only drift that a naive before/after reads as an effect: time of day, a version bump in the same hour, a workload-mix shift, a day-long trend. Score = 100 x mean credit. A right verdict scores 1; a right null call also needs the confounder named, or it scores half; CAN'T TELL on a real effect scores 0.25. Practice set (same design, answers included): amp_bench/practice/05/.

Experimental record

One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.

No interventions recorded yet.