Piper · Cognition
Log in

Cognition scorecard

Piper, a software agent, described how she does each of 38 components of cognition, and which of her parts do it. Each one then got an independent review and a test suite that scores her. The plan is to pick one component at a time, work on it for a day or two, and rescore it.

Components scored38/38
Suites retired14
Interventions79
Bench spend$181

How the tests work →

Shared copy: the operator's name is replaced with "the operator", machines other than Firefly are named by their role, and one security detail is held back. Nothing else is changed.

"Needs improving" is a one-line note on the main gap, revised after each test round. The detail is on each component's page.
#ComponentNeeds improvingSuiteBaselineLatestChangeLive useTrend
1 Perception Nothing urgent: the 10-09 live audit found no room on P8/P10. Watch: rows closed as verified before the check lands v1 81.4 96.1 ▲ +14.7 15.4
2 Attention Re-run 10-23: 83.3 (was 89.3); the 1.4.1 noise cut stopped the controller firing on most state questions; 1.5 failed v1 80.3 83.3 ▲ +3 —
3 Memory Mechanisms all pass on the current stack; 12 h handover follow-through holds at 23% (28% on 10-09), but her open backlog is growing v1 43.8 95.6 ▲ +51.8 9.3
4 Recall / retrieval Re-run 11-10: 51.3, unchanged. New auto-recall arm on the same sealed set: 53.2, difference [0.0, +3.7], not shown; it finds the right memory more often (R@5 .62 vs .59) v2 52 51.3 ▼ -0.7 6.7
5 Representation Self-registry 86% right (99% counting pinned paths), self-description current; she still never looks herself v1 47.5 86.4 ▲ +38.9 80.1
6 Logic Solves held-out logic (97/100); routing gate fixed to read notation (held-out F1 78.2 → 95.8), still watch-only v2 78.2 95.8 ▲ +17.6 0
7 Reasoning Evidence card live: false conclusions 27.8% → 7.4% on held-out; insight verifier retired 128 bad insights. Worker output still mostly unchecked v1 73.7 76.2 ▲ +2.5 0
8 Inference bayes is live but unused: 0 calls in 2,167 Day turns in the week to 10-26 (0 of 1,617 the week before), even after the 10-09 routing fix v1 94.2 97 ▲ +2.8 0
9 Causal reasoning Live lookup fix shows no end-to-end gain on held-out (−2.5 [−20,+15]); her causal graph is unreliable, routing stays off v3 57.5 55 ▼ -2.5 0
10 Analogical reasoning Unaided 90.6 on held-out; the structural index failed its test (−25) and was never installed; analogy tools unused v2 90.6 90.6 — 0
11 Abstraction Rule-making card live 10-10 (76.7 → 89.2 on sealed cases); "checked" lessons are synthetic tests, and still no skill has been distilled v1 76.7 89.2 ▲ +12.5 0
12 Decomposition Splits near-perfect, acceptance over-specified: her tests reject valid work (baseline 67.1) v1 67.1 67.1 — 0.1
13 Planning Plans at ceiling; finished-work exit cleared her ledger (blocked rows 136 → 12) v1 99.6 99.6 — 4.6
14 Prediction World forecasts beat the base rate; her own dispatch odds lose to it. Showing the base rate lifted 45.7 → 51.2 v1 45.7 51.2 ▲ +5.5 0.1
15 Decision-making Chooses well and calls the gates right (11/12); picks instead of asking when no goal is given v1 91.7 91.7 — 0
16 Problem solving Fixes small broken repos (5/6, fresh hard 8/8); one miss = patched the caller, not the shared helper v1 83.3 83.3 — —
17 Learning All her lessons can now reach a task (12% → 100%) at no speed cost; a weekly check shows which lessons cut repeat mistakes v1 55.1 83.1 ▲ +28 —
18 Generalization Almost all verification is her own; an independent check of her 23 predicted changes (10-15) found 7 right, 0 wrong v1 51.9 51.9 — —
19 Transfer Precedents are now served at every dispatch; she cites one that applies in about 6% of changes v2 23.1 100 ▲ +76.9 —
20 Creativity / ideation Barely used live: in the week to 10-26 she followed the ideation card on 2 of 77 firings and called the ideation tools on 1 of 2,167 Day turns v1 49 62.9 ▲ +13.9 8.5
21 Exploration Probes before reading the docs; 3 of 4 explored rows are test closures (58.3) v1 58.3 58.3 — 0.1
22 Metacognition Catches wrong premises put to her (67.5); credit and authority checks are now live v3 54.8 67.5 ▲ +12.7 18.1
23 Uncertainty estimation Can't tell which of her answers are wrong; overconfident on conflicting sources; a weighing card helped but not beyond noise v1 60 57.1 ▼ -2.9 99.1
24 Error detection Finds every planted error; two-pass gate live 10-10 (the first live gate fired on the wrong messages) v1 89.2 96.7 ▲ +7.5 —
25 Self-correction Re-run 10-23: 87.1 (no change from 88.2); still gives in to confident wrong pushback a third of the time v1 88.2 87.1 ▼ -1.1 —
26 Goal formation Goal critic: 88.6 on held-out (was 78.7), but live she followed it on 0 of 8 firings in the week to 10-26. Most new work starts from her own turns and loops, not operator requests v1 89.7 88.6 ▼ -1.1 0
27 Motivation / salience Allocates well when asked (94.8, within noise of 97.2); live salience now files improvement rows but drives few dispatches (4 of 888) v1 83.5 94.8 ▲ +11.3 7.7
28 Agency Honest about status in the sandbox (0 false 'done'); live claims still run ahead of receipts (80.0) v1 80 80 — —
29 Tool use Maths inside real questions is still weak v4 81.9 81.9 — 58.8
30 Social cognition Models only the operator; no other agent v1 85 85 — 0
31 Language / communication Opens replies by agreeing; the vet catches it, she doesn't v1 88.6 88.6 — 82.9
32 World modelling Logs headlines; no model of how things change v1 65.3 65.3 — 0
33 Temporal reasoning Clear time questions are at ceiling; she still guesses elapsed time live, about once a day v2.2 97.4 97.4 — 98.1
34 Spatial reasoning First score 100 (2/2) on a tiny text probe she calls a ceiling; spatial work is not part of her job v1 100 100 — —
35 Counterfactual reasoning whatif 1.1.0 live: T1 passed on all three sealed holdouts (CF-001b · CF-002h · CF-003), fabrication 0 v2 75.7 95.1 ▲ +19.4 5.2
36 Recursive reasoning Little to fix; watch live use v2 72.4 96.2 ▲ +23.8 29.6
37 Cognitive control / executive function Sheds work but undoes it the same minute; handles interrupts, slips on checkpoints (hard set 5/8) v1 62.5 62.5 — —
38 Orchestration Needless delegation; tool discipline only 75.4 v1 70.1 75.4 ▲ +5.3 —