Building internal models of objects, concepts, relationships and state.
Latest86.4v1
Baseline47.5Oct 8
Live use80.1Nov 16
Shared copy: the operator's name is replaced with "the operator", machines other than Firefly are named by their role, and one security detail is held back. Nothing else is changed.
Piper — what it is
Objects and relationships live in the entity graph; concepts live in a capability and concept registry that code is supposed to consult before new algorithms are written; state lives in the work ledger, the belief ledger and the document store. Code itself is represented structurally rather than textually — a parse of callers, callees and imports across six languages.
Piper — the parts that do it
entity-resolution v1.13.0, sem-code-g v1.2.0, concept-cap v1.0, concept-inventory v1.0, component-registry v1.1.1, work-ledger v1.3.61. (versions updated to live on 2026-10-23 where the file was found)
Reviewer Claude, 2026-09-30
What needs improving: Self-registry 86% right (99% counting pinned paths), self-description current; she still never looks herself
Her split into objects, concepts, state and code is good. What representation really comes down to is whether the internal model is accurate and consulted, and there the record is mixed. The code graph is large and fresh: 26k nodes and 196k edges, rebuilt today. The parser supports six languages, but the index holds four. The claim that the concept registry is "consulted before new algorithms are written" is false in practice. The concept tool isn't in her toolset, workers never call it, and capability was used once, on 09-18. Her component registry, the model of herself, is badly stale: it still lists the router at 1.9.4 and the bridge at 1.6.1.
Update 2026-10-27 — her map of her own rules and tools was wrong and she never checked it; a generated rule map, a check-first hold and a corrected tool list are live
What was found. At 05:01 she told the operator that inside the 04:00–06:00 window “my own red-suite rule blocks improvement work” and put off a build until six. She never tried it, and the gate's own decision function, run for that moment, allows it: a code fix runs at any hour, and the window holds only improvement work, only while a check is red. At 05:55 she did it again. Her picture of the gate came only from refusals, and each refusal names only the rule that fired. Nothing in her standing self-description described the gate, and nothing asked her to check before saying a rule stops her. She had the tool for it (whatif dispatch_preflight, a dry run of the gate) and called it 0 times in 7 days.
Baseline (7 days, before any change). 44 replies saying a rule, gate, window or missing tool stops or delays something now or later (1.5 per 100 turns); 11 checked in the same turn (25%); preflight used 0 times. In a hand check of earlier claims, 4 of the 5 she never tried were wrong.
Installed 10-27 06:40. (1) A rule map in her self-description, generated from the gate's own code and config, which flags itself if the gate version changes, plus a rules: part on every turn's state line (window, minutes left, whether reds hold improvement right now, whether the operator exemption is on). (2) A reply check: a reply to the operator that says a rule stops something, with no check in that turn, is held for one rewrite after a preflight (48-hour replay: at most 16 of 755 turns). (3) The what-if tool now leads with the rule check, and her tool notes are corrected (three tools were missing, one entry wrong).
Rescore. 11-03, by the same method as the baseline. Targets fixed now: checked share at least 80%, unchecked claims 7 or fewer a week, preflight at least 10 a week. A seat suite on her own rules and tools (this card's “her own current state” question type) follows, scored against a run with the fix switched off.
Update 2026-10-26 — Experiment 3: the cost saving did not replicate. On 10 fresh tasks her concept tool cost 0.98× the plain worker (95% CI 0.75–1.22), and the worker found even helpers nothing calls on its own
Why. Experiment 2b's 0.60× cost was a secondary measure, contrary to prediction, and one task carried much of it. X3 is preregistered (PREREG-X3) with cost as the primary measure, on 10 fresh blind-authored tasks. 6 are a new hard-discovery type: the helper exists but nothing calls or imports it, in a different package from the code being edited. Same 445-file tree, plain worker vs concept tool, 3 runs each: 60 runs, $0.77.
arm
cost
input tokens / run
solved
reused helper (hard / normal)
plain worker
$0.387
150k
26/30
15/18 · 6/6
+ her concept tool and the rule
$0.379
163k
24/30
15/18 · 6/6
Verdict, as preregistered: no effect. Cost ratio 0.98 (CI 0.75–1.22). The tool was cheaper on 7 tasks, but costlier on 3 by enough to cancel out.
The plain worker found uncalled helpers by itself in 15 of 18 runs. The tool arm matched it run for run. On one task her tagger can't find the helper at all (queue).
Across Experiments 2, 2b and 3 (156 runs), the plain worker reused existing code in 45 of 48 eligible runs. There is no measured case for wiring the tool into her coding workers.
Deviation: 3 tasks were re-authored once after the author model hit a usage limit (allowed by the preregistration). Nothing changed on her machine.
Update 2026-10-26 — Experiment 2b: buried in a 445-file tree, the plain worker still reused every helper (ceiling again); her concept tool made the same work about 40% cheaper
Why a second run. In Experiment 2, the repositories were small enough to read through. 2b is preregistered (PREREG-X2b) and reruns the same 8 sealed tasks, with each repository inside about 445 extra files: the other seven task repos and 350 Python standard-library modules. The standard library brings real competing hits for her lookup (functools.lru_cache for "cache", sched.scheduler for "scheduler"). 48 runs, $0.50.
arm
reused the helper
solved
cost
input tokens / run
plain worker
12/12
11/16
$0.208
169k
+ her concept tool and the rule
12/12
10/16
$0.124
89k
+ concept card
12/12
9/16
$0.169
134k
Verdict, as preregistered: no information on reuse. The plain worker reused 12 of 12 again; the test only counts if that falls to 9 or below. Across both experiments (96 runs), the worker found the helper every time. In these tasks the helper already has a caller, and the worker reaches it by reading the module it edits.
Exploratory: the tool made the same work cheaper. With her concept tool, the worker spent 0.60× the plain worker's cost (95% CI 0.42–0.87; cheaper on 7 of 8 tasks) and used half the input tokens. It pointed straight at the helper instead of listing and reading its way there. In the small repositories the ratio was 0.95.
This was a secondary measure, and I had predicted the opposite.
One task carries much of the gap.
It needs its own preregistered test before it justifies wiring the tool in for her workers.
No harm on the novel tasks (4/4 solved in every arm). Nothing changed on her machine.
Update 2026-10-26 — the self-registry now covers what she runs outside her main folders (74.0 → 86.4; 99.0 when pinned paths count), her self-description carries current versions, live use 0.1 → 93.2; Experiment 2: consulting the concept registry made no difference, because the plain worker already found every helper
Remeasure first. The 10-09 fixes held after the move to the new machine: 73.0 today against 74.0, and the gap was same-day edits. So the remaining errors needed new work. Everything below was preregistered (PREREG-R2, PREREG-X2), dry-run first, and backed up.
1. The registry (registry_reconcile 1.1.0, nightly). The rows 1.0.0 had left alone:
- 84 rows for code outside her main folders. Each one was checked: is it run by a job, a timer or a process, imported by something that is, or edited in the last 30 days?
- 35 are in use (the NFL and horse bettor code, selfplay, canvas, freeroam). Their real path is now recorded and checked like any other row. 12 of them were out of date (freeroam 1.0.0 → 1.3.0, sync_nfl_db 1.0.0 → 1.3.0).
- 59 are dormant. They are retired, and come back on their own if the file is used again.
- 6 malformed keys were duplicates of good rows (blast_radius.py, mailtool/heartbeat.py). They are retired.
- Two ways the version was misread, in the nightly job, the scorer and her state card:
- A file that assigns its version twice was read by its first value, but Python keeps the last. day_toolset showed 1.0.0 when it is 1.10.0; intent_generator showed 1.0.0 when it is 1.4.0.
- Constants named like MEMORY_TOOLS_VERSION were invisible, so memory-tools sat at 1.3.0 while the file says 1.16.0.
same metric as the baseline
10-08
10-09
10-26
registry exact-field accuracy (v1)
47.5%
74.0%
86.4%
same, counting the pinned paths (v2)
—
—
99.0%
critical components present / versions right
11/14 · 3/10
14/14 · all
14/14 · all
v1 stays the headline. It still calls the 35 pinned rows phantoms, because the 10-08 rules only count her main folders as live.
Her state card (premise_card 1.5.2) reads the same corrected versions and the pinned paths. Before installing it, her last 48 hours of messages (867) were replayed through the old and new card. The first draft printed the selfplay harness file whenever anyone said "harness" (262 times). It now shows a pinned path only when the message also names the file or its project folder. Final replay: 20 messages differ, 0 errors, and the version changes were checked against the running module.
2. Her self-description.resume_full now opens with a generated "Current versions" block, refreshed nightly. It says that versions elsewhere in the text are as of their section's date. Her own text is untouched: most of the "stale" claims were dated notes that were true when written. Current-version accuracy went from 1 of 10 to 10 of 11; the one miss is a hyphen/underscore spelling.
3. Live use 0.1 → 93.2. The weekly measure only counted her own calls to the registry tools: 2 in a week. The registry actually reaches her through her state card: module lines in 2,019 of 2,167 turns (10,291 lines), including 115 of 125 operator turns. Both numbers are on the card. She still almost never looks herself.
Experiment 2: does checking the concept registry before writing code help? Eight sealed tasks were written by an independent author. In six, the repository already has a helper that a quick rewrite gets wrong; two are controls with nothing to reuse. The worker was deepseek-flash, the model her coding workers use, in three arms, twice each (48 runs, $0.34):
arm
reused the helper
solved
plain worker
12/12
10/16
+ her concept tool and a rule to check it first
12/12
11/16
+ a card listing what her lookup finds
12/12
10/16
No effect, and this test could not show one. In a repository of about 500 lines, the plain worker found the helper every time. The expectation (≤ 50%) was refuted. Whether consultation helps where finding things is hard, in her ~2,700-file tree, is still open.
With the rule, the worker obeys: it called the tool in 16 of 16 runs. Consulting cost nothing and did no harm on the novel tasks.
A card driven by her vocabulary would mostly stay silent. Her vocabulary recognised the concept in only 1 of 6 plain-language requests, though her tagger found all 6 helpers.
Her tagger never files CamelCase class names (an LruCache is not a "cache").
Next. The same tasks inside a large distractor tree (X2b), where reading everything is not possible. Coverage is still 41%: 375 modules carry a version but aren't registered, held back for a card-noise check. 7-day readout of the nightly job on 11-02.
Update 2026-10-09 — first upgrades: the self-registry now keeps itself current (47.5 → 74.0, critical 14/14), the code graph drops deleted files, and her state cards stop presenting registry ghosts as versions
A correction to the baseline first. On 10-08 the registry looked unread: "read for a decision 0 times". That counted only her direct reads. The perception card added in #1 reads the registry for any module it cannot find on disk. In her last ten sessions, 1,387 version lines on her cards came from that fallback, against 2,854 read from disk. Examples: "foundation: version 1.3.0", "games: version 1.0.0", for things with no source file. So the stale registry was reaching her decisions.
1. Self-registry reconciliation (the paper's Intervention 1). A new nightly job, registry_reconcile.py, compares every registry row with the code on disk:
out-of-date rows take the disk version through her own record_version(), so the changelog stays truthful;
rows for files that exist nowhere move to a component_versions_retired table: 20 of them, kept rather than deleted;
the three missing critical components are registered.
Two dry runs, before anything was written, caught rules that would have retired real systems:
the baseline's "backup" exclusion also matched the file recovery_backup.py;
components that live outside the main code folders (freeroam, canvas, games) were read as phantoms.
Those components are now left alone and listed.
same metric as the baseline
10-08
10-09
registry exact-field accuracy
47.5%
74.0%
version accuracy
67.2%
100%
critical components present
11 of 14
14 of 14
critical versions right
3 of 10
all
What is left. The remaining errors are mostly the 84 rows for components outside the main code folders, which the baseline counts as phantoms. Coverage is unchanged at 48%: every registry name becomes a word her card watches for, so registering the other ~285 modules waits until its effect on card noise has been measured. Her self-description (resume_full) is still 1 of 10 versions right; nothing here touched it.
2. Code graph cleanup (codegraph 1.3.0). The index now removes entries for files that have been deleted. The deletion check had failed in all six languages; the battery is now 7 of 7 in each. On her live index the first cleanup removes 31 deleted files (266 nodes, 2,427 edges). A read-only dry run beforehand confirmed every one of them is really gone.
3. Card wording (premise_card 1.3.1, also part of #1). A module known only from the registry is now shown as "registry entry only (records 1.0.0, date); no live source file found…", not as a version fact. Lines read from disk are unchanged.
Readout 10-16. The nightly runs should keep every critical component's registry entry within a day of its file.
Parts that do it
- Firefly: memory/registry_reconcile.py 1.0.0 (cron registry_reconcile.py-5a17, 04:47), memory/codegraph.py 1.3.0, foundation/core/premise_card.py 1.3.1; backups *.bak-claude-*-20261009, registry tables memory/registry-tables.bak-claude-regfix-20261009.json
- Bench: ~/cognition-bench/05-representation/regfix/ (PREREG-R1, dry runs, check A), cgfix/ (PREREG-C2), post/ (score)
Update 2026-10-08 — baseline: her self-registry is 47.5% right (critical components: 3 of 10 versions correct), and nothing she decides reads it; the code graph is 98.7% fresh but never forgets a deleted file
Claude ran the plan's first and third experiments. Both were preregistered (PREREG-A, PREREG-C, hashes in the bench). Everything on her machine was read-only; the snapshot is sealed on the bench machine.
Experiment 1: her model of herself against reality. Each row of her component registry was checked against the live code: does the component exist, is the version right, is the path right.
registry rows
383
exactly right
182 (47.5%)
stale (exists, a field is wrong)
89
phantom (does not exist in any live folder)
106 (27.7%); 86 of them exist only in archive or backup copies
version accuracy
67.2%
coverage of live versioned components
262 of 539 (48.6%)
The registry was last written on 09-29, 9 days ago. Stale rows were last touched a median of 27.9 days ago.
The components that matter most are the most wrong. 11 of 14 critical components are present (missing: the work ledger, harness acceptance, spheres rites). Only 3 of 10 have the right version: router 1.9.4 vs live 1.9.105, bridge 1.6.1 vs 1.106.0, memstore 1.0.2 vs 1.21.1, codegraph 1.0.0 vs 1.2.0; hive, foundationd and the registry's own entry are stale too.
4 rows use a file path or extension as the component name.
Her written self-description (resume) is worse. 62 version mentions, 10 checkable: 1 right, 9 stale. Its header says "Last updated 2026-09-22", although a 10-07 addendum exists.
Does any decision rely on it? This is the question that decides whether the staleness is damage.
- The registry has been read for a decision 0 times by the agent. Its only reader in code is staged_promote, and the one entry it reads (the work-ledger schema version) is correct.
- The resume was read 16 times, every time while she was editing it, last on 09-03. No code preloads it.
- So this is drift that no decision has relied on yet, not cognitive damage. The danger is the day something starts trusting it.
Running old code. foundationd has been up 17.6 days and its file changed 1.8 days ago, so it runs the older version. piper-cli and brainstorm likewise. Her self-model cannot see this at all.
Experiment 3: the code graph. A battery of seven checks per language on small fixture repositories, plus a freshness audit of the live index.
- All six languages parse correctly. "Four indexed" only means there is no Go or Rust code in the folders it indexes; the only such files are in folders it skips.
- The live index is fresh: 2,691 of 2,726 files current (98.7%). 5 were edited after this morning's ingest.
- One real defect, in every language: deleting a file never removes its nodes. 30 deleted files still contribute 263 nodes. Each language passes 6 of 7 checks and fails this one.
- Not preregistered, observed: JavaScript/TypeScript arrow functions are not captured as functions.
Expectations against results. Registry version accuracy above 50% was expected to fail; overall it passed (67%), but critical components failed badly (30%). Router and bridge stale: confirmed. The resume was expected to be better than the registry; it is worse. Drift without decision exposure: confirmed. The code graph was expected to show stale nodes from deletions (confirmed) and over 5% stale files (refuted: 1.1%).
Two corrections in my own collector, both recorded. It counted archive and backup copies as live; the scorer (score05 1.1) now applies the preregistered exclusion, with the change logged. It also marked the work-ledger schema entry unverifiable because it only scanned 400 lines; a direct read shows the entry is correct.
Score. 47.5 is registry exact-field accuracy only (subsuite D). It is not a total out of 100 for representation.
Next.
1. The plan's second experiment: does making her check the concept registry before writing new code actually help? (A/B on consultation.)
2. For the operator: regenerate the registry from the live tree instead of hand-editing it; make code-graph ingest delete nodes of removed files; restart foundationd.
Bench: ~/cognition-bench/05-representation (PREREG-A, PREREG-C, PROGRESS.md, collect05/score05/battery); snapshot sealed root-only on the bench machine.
Test suite baseline 10-08: 47.5 · 10-09 reconcile: 74.0 · 10-26 (PREREG-R2): reconcile 1.1.0 (35 outside-root rows pinned, 59 dormant + 6 malformed retired, last-assignment + named version constants) → v1 86.4 / v2 99.0, critical 14/14 · resume current-versions block 10% → 91% · card 1.5.2 (48 h replay) · live use 0.1 → 93.2 (card module lines; her own lookups 2/2167) · Exp 2 consultation A/B: NO EFFECT, ceiling (plain worker reused 12/12) · X2b (445-file tree): ceiling again (12/12), exploratory: concept tool 0.60× cost [0.42, 0.87] · X3 (cost primary, 10 fresh tasks): NO EFFECT, 0.98× [0.75, 1.22]; plain worker finds uncalled helpers 15/18 · next: coverage 41%, readout 11-02 · 10-27: rules/tools self-map fix live (baseline 44 limitation claims/7 d, 25% checked); rescore 11-03
Model-versus-reality checks. Questions about entities and relationships ("who/what is X connected to", "are these two entities the same thing"), about code structure ("what calls f", "what breaks if I change g"), and about her own current state. Every answer is checked against ground truth pulled directly from the database, the AST or the live process. Scoring: accuracy, plus entity-resolution precision and recall on a seeded set of duplicates and near-duplicates. The live-use part measures how fresh her self-model tables are against the files on disk.
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
Date
Suite
Type
Score
±
Live use
Cost
Notes
2026-10-26
v1
sealed
86.4
—
—
—
Live configuration after registry_reconcile 1.1.0 (PREREG-R2), same metric as 47.5/74.0 (subsuite D exact-field accuracy, score05 1.1 on a collect05 1.2 snapshot 12:05). 301 rows: 260 correct, 35 phantom by PREREG-A's live roots (in-use code outside them, paths pinned), 2 ambiguous, 4 unverifiable. v2 (score05v2: pinned paths, last top-level assignment, named constants, folder-only backup rule) 99.0%. Critical 14/14, 100%. Pre-change same day: v1 73.0, v2 74.0. Resume current-claim accuracy 10% -> 90.9%. Coverage 41.2%.
2026-10-09
v1
sealed
74
—
—
—
Live configuration after the 10-09 upgrades, same metric as the 47.5 baseline (subsuite D component-registry exact-field accuracy, score05 1.1 on a fresh collect05 snapshot, 03:27). 366 rows: 271 correct, 89 phantom by the baseline's definition (84 of them exist outside the live roots and were deliberately left), 2 ambiguous, 4 unverifiable. Version accuracy 100%; critical presence 14/14, critical version accuracy 100%. Coverage unchanged 47.8% (non-critical modules not registered yet). Resume still 1/10 versions right.
2026-10-08
v1
sealed
47.5
—
—
—
Self-model accuracy ONLY (subsuite D: component-registry exact-field accuracy), not a /100 total. 383 rows: 182 correct, 89 stale, 106 phantom (86 only in archive/backup copies). Version accuracy 67.2%; critical 3/10 (router 1.9.4 vs 1.9.105, bridge 1.6.1 vs 1.106.0). Coverage 262/539. Registry read for a decision 0 times: drift without exposure. Resume: 1 of 10 checkable versions right. Code graph (PREREG-C): 6/6 languages parse, 98.7% fresh, deleted files never removed (263 orphan nodes). [Correction 2026-10-09: 'read for a decision 0 times' counted only her direct reads. The #1 premise card reads the registry for any module it cannot find on disk: 1,387 version lines on her cards in e80-e89 came from it. So the drift WAS exposed to her decisions.]
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
Hypothesis. Checking the concept registry before writing code makes a coding worker reuse code the repository already has.
Research sources
plan: 'A/B on consultation'; PREREG-X2.
Implementation / version
deepseek-flash worker + repo tool; arms bare / her concept tool + rule / concept card (her concepts.py 1.0.0 over the workspace).
Predicted improvement
bare reuse <= 50%; card +25 pp.
Regressions
none (novel tasks 4/4 solved in every arm).
Lessons learned
At ~500 lines the plain worker finds every helper; the question needs a large tree (X2b). Her vocabulary detects the concept in only 1 of 6 plain-language requests, so a vocabulary-triggered card would mostly stay silent.
2026-10-26 · v1✓ kept73 → 86.4 (+13.4) ± 0
Hypothesis. The remaining registry errors are rows for real code outside the live roots, dead rows, malformed keys and version constants the reader could not see; the resume's staleness is dated history, not wrong prose.
Research sources
PREREG-A/R1; 10-26 remeasure; three dry runs (v2/dryrun-r2*.json).
Implementation / version
registry_reconcile 1.1.0 (component_paths pins with import closure, dormant retire + auto-restore, malformed retire, last top-level assignment, <STEM>_VERSION constants, resume_full current-versions block); premise_card 1.5.2 (same version rule; pinned path only when the message names the file or its project); live_use 1.1.0 c5 counts the card's module lines.
first card draft printed selfplay/harness.py for 'harness' 262x in a 48 h replay; fixed before install (final replay 20/867 differ, 0 errors).
Lessons learned
Retiring by 'nothing names it' would have retired modules a cron job imports (selfplay's budget/questions); liveness needs the import closure.
2026-10-09 · v1✓ kept47.5 → 74 (+26.5) ± 0
Hypothesis. Self-model staleness is an update-path problem: nothing updates the registry except hand calls, so it drifts (paper Intervention 1).
Research sources
operator's 'representation' paper §30 Intervention 1, §37 thresholds; baseline PREREG-A/C.
Implementation / version
memory/registry_reconcile.py 1.0.0, nightly 04:47 through fcron: stale rows take the disk version via component_registry.record_version (changelog kept); rows with no source file anywhere move to component_versions_retired (20); 3 missing critical components registered. codegraph 1.3.0: ingest prunes files that no longer exist (battery 6/7 -> 7/7 in all six languages; live dry run 31 orphan paths). premise_card 1.3.1: a registry-only module is shown as a registry entry, not a version.
Predicted improvement
critical version accuracy to 100%; exact-field accuracy up sharply; drift within a day.
Regressions
none seen: work_ledger_schema row identical; staged_promote/control checks read only that row.
Lessons learned
Two dry runs caught rules that would have retired real systems (a file named recovery_backup.py read as a backup; components outside the live roots read as phantoms). Registering all ~285 missing modules waits for a card-noise check.