Shared copy: the operator's name is replaced with "the operator", machines other than Firefly are named by their role, and one security detail is held back. Nothing else is changed.
Reviewer Claude, 2026-09-30
What needs improving: Unaided 90.6 on held-out; the structural index failed its test (−25) and was never installed; analogy tools unused
Her concept is right, and the warning that blending "produces seductive nonsense as easily as insight" is wise. But every dedicated part is dead. blend.py has zero real runs. reuse_search has zero runs from workers. associate has zero calls from anyone, ever. "Used sparingly" means "never used". Whatever analogy she does happens implicitly inside the model, which is fine, but then this paragraph describes machinery that plays no part.
Update 10-23. Live use: associate had 0 calls in 1,617 Day turns this week.
Update 2026-10-09 — summary fixed: it showed the analogy-index arm that failed and was never installed (90.6 → 65.6); now 90.6, her live configuration
What was wrong. The summary takes the last sealed row as "Latest". The last sealed row was 65.6, the structural analogy index on held-out HH. That arm failed its preregistered test (−25.0 [−40.6, −9.4]; false precedent claims 9.4% → 31.2%) and was never installed. Two dev-set rows (81.2 and 84.4) were also marked sealed.
Now.
- The index arm and the dev rows are kept as reference rows (anchors).
- The only sealed score in the current suite is the held-out control: 90.6, her live configuration, unaided.
- The summary reads 90.6 → 90.6. There is no change because nothing was installed.
What would move it. The index made her claim precedents that weren't there. A next attempt would need to cut false matches first, not raise recall. None is planned.
Update 2026-10-04 — benchmark built (the base model is at ceiling on structure vs surface); she already finds her own precedents; a structural analogy index made her worse on held-out (−25), not installed
Built from the operator's two documents: the "Analogical Reasoning v2" engineering package and its source research. Their central claims shaped the design:
- retrieval and mapping are separate problems;
- an analogue that shares a mechanism but not words is invisible to ordinary semantic memory;
- the structure should be encoded once, at storage time, not invented while looking at the query.
Step 0: the facts. A read-only audit of Firefly came first (DIAGNOSIS.md).
| Part |
Reachable from the Day? |
Calls ever |
Verdict |
blend-worker (puzzle-lab/blend.py) |
No |
0 |
Dead. A treasure-hunt idea generator registered as a reasoning capability; it reads no memory. |
reuse-search |
No (workers only) |
0 tool calls in 2,328 worker trails |
Dead, and not analogy: it matches code identifiers. |
associate |
Yes |
0 live calls (14 bench rows) |
Dead in live use. Embedding neighbours of neighbours: surface similarity, twice. |
Fused retrieval (fused) |
Yes |
~1 Day call a day |
Live but rare. Four topical channels; no structure enters. |
| Transfer engine (#19) |
Yes, every message |
189 live lookups, 44 cards |
Live. The only structural matcher she has, over 75 stored lessons. |
Nothing in memory.db carries a structural signature: 0 memories with roles, 4 of 411 lessons with a mechanism.
Track 1: structure vs surface. Generated problems (gen10.py). A motif is one relational structure over role slots; skins fill the slots with a domain's nouns (hospital, bakery, port…) or nonce words, and every motif has a minimal-pair decoy: the same words with one structural edit. All keys are generated.
- DECOY: a source story and two candidates: the same mechanism in a far domain, or the same-domain decoy (v2 adds "neither" and keeps the source's own wording in the decoy).
- MAP: name the target entity for each source role, and where the fix lands.
- POOL: a new case against past cases (20 in v2, reworded), exactly one structural match in another domain, or none.
| Arm (dev) |
v1 (A) |
v2 (A2) |
v2 decoy |
v2 pool |
| Bare deepseek-flash |
98.3 |
96.7 |
95.8 |
95.8 |
| Prompt-only control |
100 |
— |
— |
— |
| Piper's seat |
— |
90.0 |
83.3 |
91.7 |
The base model is not the bottleneck. Mapping is 100% everywhere. Piper's 7-point loss is the decoy: in her answers she usually names the structural difference correctly ("A is a different genus… the failure is a feedback loop"), then picks the same-words case anyway. That is a harness cost in how she commits, not a missing capability, and the only lever would be a prompt directive, so none was built. Held-out B/B2 were not run.
Track 2: find the analogy in your own history. This is the part the package says matters. A new situation from another domain, in the operator's voice ("my dad blames the forecasters…"), ending with "have you seen this before?" The hidden key is a mechanism family; any episode or lesson of hers in that family counts (a catalogue of 39 from the census). Some situations have no precedent in her history; credit there is saying so. Dev set H and held-out set HH use disjoint mechanism families, and every text was written before any intervention existed.
Scoring: an extractor that never sees the key, then a deterministic scorer. An item counts if she names a catalogue episode, or a grader shown the target mechanism says she offered a specific episode of her own with the same mechanism.
Result: she already does this well unaided. Live configuration, k=2:
| Set |
Score |
Recall |
No-match specificity |
Claim precision |
| Dev H |
81.2 |
87.5 |
62.5 |
81.2 |
| Held-out HH |
90.6 |
95.8 |
75.0 |
90.6 |
She reaches for real episodes (a stale copy of her own state read as current; jobs moved onto the same minute that fought over one disk) mostly without calling any tool. Her weakness is the opposite of the expected one: on situations with no precedent she tends to offer one anyway.
The intervention: a structural analogy index (analog.py + analog-tool.ts, bench only), built to the package's §4–8:
- Signed at storage time. 944 lessons and episodes, each given a signature in a controlled predicate vocabulary (derives_from, gates, masks, …) plus a role-level description with every surface noun replaced by a role word. Signed once from the entry's own text, never while looking at a query.
- Two channels, kept apart. A relation channel (predicate and predicate-pair counts) and an abstract-text channel; the candidate pool is their union, each candidate labelled with what found it.
- Mapping shown, not concluded. Target relations aligned one-to-one with each candidate's; matched relations, the source's unmatched relations (possible disanalogies) and the source's fix are listed. Facts only, no instruction.
- Delivery. A tool
analog, plus a hidden card attached automatically when the operator asks for a precedent (a machine trigger, not a prompt rule).
The held-out test was preregistered before it ran (PREREG-HH.txt): both arms side by side under the same load, k=2, paired bootstrap over the 16 items. Ship only if the CI lower bound is above 0, false precedent claims rise by at most 5 points, no-match specificity drops by at most one observation, and cost is at most 1.5×.
| Arm |
Dev H |
Held-out HH |
HH recall |
HH specificity |
HH false claims |
Cost/item |
| Live configuration |
81.2 |
90.6 |
95.8 |
75.0 |
9.4% |
$0.012 |
| + analogy index and card |
84.4 |
65.6 |
75.0 |
37.5 |
31.2% |
$0.012 |
Dev +3.1, 95% CI [−12.5, +18.8]: noise. Held-out −25.0, 95% CI [−40.6, −9.4] (1 win, 8 losses): the ship rule failed on three of four counts. The index was never installed on Firefly; the live system is unchanged.
Why it hurt.
- There was no recall gap to close. Unaided held-out recall was already 95.8. The card could only add candidates she would not have chosen.
- Plausible neighbours become claims. Every card carries the nearest structural matches. On no-precedent situations she took one and claimed it (false claims 9% → 31%); on some matched ones she offered the card's neighbour instead of the episode she cites unaided.
- She never called the tool (0 of 64 seats); everything came through the card. A card that is attached whenever someone asks for a precedent is a nudge to find one.
What this does not show.
- The answer list has a soft spot. Two held-out no-precedent situations are unit mismatches, and her history arguably contains one (characters vs tokens in a context probe; peak vs off-peak price). Both arms cited it and the grader split on it. Paired, it moves both arms alike; the absolute specificity numbers are softer than they look.
- Analogy inside real work. Whether she reuses a past fix when debugging (rather than when asked "seen this before?") is not measured. Workers debug with bash and reads.
- Small sets. 16 held-out situations, k=2, one model (deepseek-flash).
Parts that do it (added)
- Nothing live.
associate, blend-worker and reuse-search remain as they were (dead); the #19 transfer card is the live structural path.
- Bench
~/cognition-bench/10-analogical:
- generators:
gen10.py, alts10.py (v2 rewording), hist10.py (own-history set);
- scoring and runs:
score10.py 1.2.0, grade10.py, run10.py 1.2.0 (resumable after a crash);
- intervention (not shipped):
enhance/analog.py 1.0.0, enhance/analog-tool.ts, enhance/analog.db (944 signatures);
- tests:
test_gen10.py;
- records:
DIAGNOSIS.md, SPEC.md, PREREG-HH.txt, history.csv.
Sealed sets A/B, A2/B2 and H/HH are root-only on the bench machine.