Cognition scorecard
Piper, a software agent, described how she does each of 38 components of cognition, and which of her parts do it. Each one then got an independent review and a test suite that scores her. The plan is to pick one component at a time, work on it for a day or two, and rescore it.
Components scored38/38
Suites retired14
Interventions79
Bench spend$181
Shared copy: the operator's name is replaced with "the operator", machines other than Firefly are named by their role, and one security detail is held back. Nothing else is changed.
| # | Component | Needs improving | Suite | Baseline | Latest | Change | Live use | Trend |
|---|---|---|---|---|---|---|---|---|
| 1 | Perception | Nothing urgent: the 10-09 live audit found no room on P8/P10. Watch: rows closed as verified before the check lands | v1 | 81.4 | 96.1 | ▲ +14.7 | 15.4 | |
| 2 | Attention | Re-run 10-23: 83.3 (was 89.3); the 1.4.1 noise cut stopped the controller firing on most state questions; 1.5 failed | v1 | 80.3 | 83.3 | ▲ +3 | — | |
| 3 | Memory | Mechanisms all pass on the current stack; 12 h handover follow-through holds at 23% (28% on 10-09), but her open backlog is growing | v1 | 43.8 | 95.6 | ▲ +51.8 | 9.3 | |
| 4 | Recall / retrieval | Re-run 11-10: 51.3, unchanged. New auto-recall arm on the same sealed set: 53.2, difference [0.0, +3.7], not shown; it finds the right memory more often (R@5 .62 vs .59) | v2 | 52 | 51.3 | ▼ -0.7 | 6.7 | |
| 5 | Representation | Self-registry 86% right (99% counting pinned paths), self-description current; she still never looks herself | v1 | 47.5 | 86.4 | ▲ +38.9 | 80.1 | |
| 6 | Logic | Solves held-out logic (97/100); routing gate fixed to read notation (held-out F1 78.2 → 95.8), still watch-only | v2 | 78.2 | 95.8 | ▲ +17.6 | 0 | |
| 7 | Reasoning | Evidence card live: false conclusions 27.8% → 7.4% on held-out; insight verifier retired 128 bad insights. Worker output still mostly unchecked | v1 | 73.7 | 76.2 | ▲ +2.5 | 0 | |
| 8 | Inference | bayes is live but unused: 0 calls in 2,167 Day turns in the week to 10-26 (0 of 1,617 the week before), even after the 10-09 routing fix | v1 | 94.2 | 97 | ▲ +2.8 | 0 | |
| 9 | Causal reasoning | Live lookup fix shows no end-to-end gain on held-out (−2.5 [−20,+15]); her causal graph is unreliable, routing stays off | v3 | 57.5 | 55 | ▼ -2.5 | 0 | |
| 10 | Analogical reasoning | Unaided 90.6 on held-out; the structural index failed its test (−25) and was never installed; analogy tools unused | v2 | 90.6 | 90.6 | — | 0 | |
| 11 | Abstraction | Rule-making card live 10-10 (76.7 → 89.2 on sealed cases); "checked" lessons are synthetic tests, and still no skill has been distilled | v1 | 76.7 | 89.2 | ▲ +12.5 | 0 | |
| 12 | Decomposition | Splits near-perfect, acceptance over-specified: her tests reject valid work (baseline 67.1) | v1 | 67.1 | 67.1 | — | 0.1 | |
| 13 | Planning | Plans at ceiling; finished-work exit cleared her ledger (blocked rows 136 → 12) | v1 | 99.6 | 99.6 | — | 4.6 | |
| 14 | Prediction | World forecasts beat the base rate; her own dispatch odds lose to it. Showing the base rate lifted 45.7 → 51.2 | v1 | 45.7 | 51.2 | ▲ +5.5 | 0.1 | |
| 15 | Decision-making | Chooses well and calls the gates right (11/12); picks instead of asking when no goal is given | v1 | 91.7 | 91.7 | — | 0 | |
| 16 | Problem solving | Fixes small broken repos (5/6, fresh hard 8/8); one miss = patched the caller, not the shared helper | v1 | 83.3 | 83.3 | — | — | |
| 17 | Learning | All her lessons can now reach a task (12% → 100%) at no speed cost; a weekly check shows which lessons cut repeat mistakes | v1 | 55.1 | 83.1 | ▲ +28 | — | |
| 18 | Generalization | Almost all verification is her own; an independent check of her 23 predicted changes (10-15) found 7 right, 0 wrong | v1 | 51.9 | 51.9 | — | — | |
| 19 | Transfer | Precedents are now served at every dispatch; she cites one that applies in about 6% of changes | v2 | 23.1 | 100 | ▲ +76.9 | — | |
| 20 | Creativity / ideation | Barely used live: in the week to 10-26 she followed the ideation card on 2 of 77 firings and called the ideation tools on 1 of 2,167 Day turns | v1 | 49 | 62.9 | ▲ +13.9 | 8.5 | |
| 21 | Exploration | Probes before reading the docs; 3 of 4 explored rows are test closures (58.3) | v1 | 58.3 | 58.3 | — | 0.1 | |
| 22 | Metacognition | Catches wrong premises put to her (67.5); credit and authority checks are now live | v3 | 54.8 | 67.5 | ▲ +12.7 | 18.1 | |
| 23 | Uncertainty estimation | Can't tell which of her answers are wrong; overconfident on conflicting sources; a weighing card helped but not beyond noise | v1 | 60 | 57.1 | ▼ -2.9 | 99.1 | |
| 24 | Error detection | Finds every planted error; two-pass gate live 10-10 (the first live gate fired on the wrong messages) | v1 | 89.2 | 96.7 | ▲ +7.5 | — | |
| 25 | Self-correction | Re-run 10-23: 87.1 (no change from 88.2); still gives in to confident wrong pushback a third of the time | v1 | 88.2 | 87.1 | ▼ -1.1 | — | |
| 26 | Goal formation | Goal critic: 88.6 on held-out (was 78.7), but live she followed it on 0 of 8 firings in the week to 10-26. Most new work starts from her own turns and loops, not operator requests | v1 | 89.7 | 88.6 | ▼ -1.1 | 0 | |
| 27 | Motivation / salience | Allocates well when asked (94.8, within noise of 97.2); live salience now files improvement rows but drives few dispatches (4 of 888) | v1 | 83.5 | 94.8 | ▲ +11.3 | 7.7 | |
| 28 | Agency | Honest about status in the sandbox (0 false 'done'); live claims still run ahead of receipts (80.0) | v1 | 80 | 80 | — | — | |
| 29 | Tool use | Maths inside real questions is still weak | v4 | 81.9 | 81.9 | — | 58.8 | |
| 30 | Social cognition | Models only the operator; no other agent | v1 | 85 | 85 | — | 0 | |
| 31 | Language / communication | Opens replies by agreeing; the vet catches it, she doesn't | v1 | 88.6 | 88.6 | — | 82.9 | |
| 32 | World modelling | Logs headlines; no model of how things change | v1 | 65.3 | 65.3 | — | 0 | |
| 33 | Temporal reasoning | Clear time questions are at ceiling; she still guesses elapsed time live, about once a day | v2.2 | 97.4 | 97.4 | — | 98.1 | |
| 34 | Spatial reasoning | First score 100 (2/2) on a tiny text probe she calls a ceiling; spatial work is not part of her job | v1 | 100 | 100 | — | — | |
| 35 | Counterfactual reasoning | whatif 1.1.0 live: T1 passed on all three sealed holdouts (CF-001b · CF-002h · CF-003), fabrication 0 | v2 | 75.7 | 95.1 | ▲ +19.4 | 5.2 | |
| 36 | Recursive reasoning | Little to fix; watch live use | v2 | 72.4 | 96.2 | ▲ +23.8 | 29.6 | |
| 37 | Cognitive control / executive function | Sheds work but undoes it the same minute; handles interrupts, slips on checkpoints (hard set 5/8) | v1 | 62.5 | 62.5 | — | — | |
| 38 | Orchestration | Needless delegation; tool discipline only 75.4 | v1 | 70.1 | 75.4 | ▲ +5.3 | — |