Piper: Skill Compilation
Turning work she has already solved into procedures she no longer has to reason through
Research programme proposal · v0.1.0 · 3 November 2026
Shared copy. Machines other than Firefly are named by their role. Nothing else differs from the full copy.
| Version | Date | What changed |
|---|---|---|
| v0.1.0 | 3 Nov 2026 | First version. §1–9 are the plan, checked against Piper's records on Firefly on 3 November. Appendix A is the external reviewer's assessment that prompted it, kept verbatim. |
Status: queued, not built. Nothing in this paper is running and Piper has not been told. Stage 0 (mining her records) is offline and read-only, so it can run before the freeze ends on 8 November. Nothing is wired into Piper before the surprise-why loop goes in on 9 November (§8).
1. The idea in one paragraph
Piper remembers what worked (episodic memory) and is starting to predict what will work (the change ledger, foresight priors, the JEPA proposal in the companion paper Predictive Self-Improvement Engine). What she does not do is stop paying for work she has already figured out. Each time a familiar task comes back she reasons through it from scratch, at full model cost, and often makes the same mistake she made last time. Skill compilation turns a task family she has solved many times into a small, tested, versioned program with explicit conditions for when it applies. The model is kept for the parts that are actually new. The question the programme asks is the reviewer's:
Can I solve a problem today in a way that permanently reduces the amount of intelligence needed to solve similar problems tomorrow?
2. What she already has (checked 3 Nov 2026)
A skills folder, all prose. Firefly ~/.pi/agent/skills/ holds 21 skills (change-ledger-resolve, selfstate-query, launch-detached-status, dbq, docstore-get/set/search, control-regression-selfcheck and others). Every one is a SKILL.md instruction note. None has executable code. She still reasons through each step when she uses one. Her own skills/README.md already sets two rules that this programme needs: evaluate a skill against a no-skill baseline and record the reasoning behind every skill edit, not just the edit (from WikiSkill). The plan below builds on her conventions and does not replace them.
A clear repeat-failure family: writing acceptance bars. Every dispatch she sends carries an acceptance command (the "bar"). The pre-send precheck in delegate_bridge.py tests the bar before the work goes out. In the 8 days to 3 November it logged 4,567 checks:
| Precheck verdict | All rows | Without the 3 repeat loops* |
|---|---|---|
| BAR_PRESENCE_ONLY (bar only tests that a file exists) | 2,006 | 739 |
| ALREADY_GREEN (bar passes before any work, so it can't show the work happened) | 1,784 | 827 |
| CANNOT_PASS (bar calls a command that isn't there) | 733 | 356 |
| BAR_UNEVALUABLE / EMPTY_TARGET / DEMOTED | 44 | 34 |
* Two deliverables (d-budgettest 923 rows, d-81bff16c 894) and rows with no deliverable (794) are mostly the same check firing over and over. With them removed there are still 1,956 checks over 131 distinct objectives and 162 distinct deliverable-and-problem pairs. Three mistakes account for 98% of them. This is the clearest case in her records of a solved problem that gets re-solved, badly, at full cost: the gate knows exactly what a good bar looks like and she keeps writing bad ones.
High-reuse tools. Over the same window dispatch_reuse_lint saw 1,504 dispatches. The most-used tools were work-ledger 191, selfstate 152, harness-acceptance 114, control-regression 80, launch-detached 72, cache-price-probe 67 and retrieval-events 53. These are the next candidate families after bars.
Trace sources for mining. delegate_pre_send_precheck.jsonl, delegate_failures.jsonl (864 rows), delegate_bar_refusal.jsonl, dispatch_reuse_lint.jsonl, the change ledger, worker sessions, cost_ledger.jsonl.
3. Corrections to the reviewer's note
- "63 against a bar of 16". These were not tests that fired 63 times. They were 63 refused envelopes in the week after prediction 9 (12–19 October, mostly C21 blast-radius, 42). "Read and not used" means she kept sending envelopes that tripped the same codes. The reviewer's question, how many decisions would have differed if the checks had not run?, has a simple answer here: every refused envelope was stopped by construction. The real question is why she didn't learn from it, and that is a procedural-learning failure. So the data supports compiling the behaviour that passes the check (writing a good bar or envelope). It does not support compiling the check itself, which is already deterministic code. This is the reviewer's own point in §2: compile the successful behaviour, not the rule.
- The cited papers were not checked. The reviewer could not find the ATHENA paper. Lifting Traces to Logic (ICML 2026) and Autonomous Skill Development (Oct 2026) are the reviewer's citations and have not been read for this version.
- Rigour level. The reviewer's Phase A–C (30–50 tasks, a hidden split, matched comparison) is proof-grade. After 8 November the operator's standing direction is quick before/after reads around a change, with no pre-registered windows (§7). This paper keeps the parts that protect correctness (broken-implementation tests, preconditions checked before a skill runs) and drops the parts that only serve proof.
4. Design
4.1 A compiled skill is a folder she already knows how to use
Each compiled skill extends the existing skill folder layout. It does not add a new system:
~/.pi/agent/skills/<name>/
SKILL.md # existing convention: when to use, near-miss boundaries, the WHY
run.py # the compiled procedure: explicit inputs -> outputs, no model calls
skill.json # contract + lifecycle (below)
tests/ # positive cases, negative cases, broken-implementation mutants
HISTORY.md # why each version exists (her WikiSkill rule)
skill.json holds:
- preconditions, checked by
run.pybefore it does anything. If one fails, it exits withABSTAINand she reasons the task through as normal. A learned habit can't harden into a rule. - postconditions: what a successful run guarantees.
- depends_on: hashes of the code it relies on (for example the precheck gate's version). If a dependency changes, the skill goes to probation automatically.
- state:
candidate/active/probation/retired. - provenance: who built it (operator, Claude, Piper, or the compiler), from which traces, at what compile cost.
- stats: invocations, abstentions, successes, failures, and the model calls/tokens saved per use.
4.2 The loop
task arrives
-> matching active skill? -- no --> she deliberates as now --> trace logged
| yes
-> run.py checks preconditions -- fail --> ABSTAIN --> deliberate
| pass
-> run, verify postconditions -- fail --> count failure; 2 in 10 -> probation
| pass
-> done, stats updated
offline, daily: compiler reads traces -> finds families -> drafts candidate
-> tests (incl. mutants) -> break-even check -> promote / reject
4.3 Lifecycle
- candidate → active: passes its tests, catches every mutant, and its expected reuse covers its compile cost (§6).
- active → probation: 2 failures in its last 10 runs, a
depends_onhash changes, or a newer method beats it on the shadow sample. - probation → active/retired: re-tested. Retired skills stay on disk for history and are never run automatically.
- Shadow sample: about 1 in 10 eligible tasks still goes the deliberative way, so skill drift shows up. (The reviewer's "reserve some executions for the original process.")
5. Stages
Stage 0 · Mine (offline, read-only, may run before 8 Nov). A trace miner on a copy of the logs ranks task families by runs per week × model cost per run × failure rate. Output: a ranked family list with cost per run measured from cost_ledger.jsonl. The bar family is expected to top it. If it doesn't, the miner's top pick goes first.
Stage 1 · Skill #1: acceptance-bar (two versions). Input: objective, deliverable, target paths. Output: a bar that (a) resolves in the gate's own shell, (b) fails before the work and (c) tests content, not just that a file exists. The gate's precheck is the oracle, so the verifier already exists and is deterministic.
- 1a, hand-written by us (Claude with the operator). This is the reviewer's "simple manual procedure" baseline.
- 1b, compiled by the first compiler from her successful and refused bar traces.
- Both are tested on the same held-back traces plus mutants (a bar that would pass on an empty file, a bar that passes before the change). Whichever does better goes active. If 1a wins, that's still useful: it tells us the compiler needs work before she gets it.
Stage 2 · Wire it in. One line in her dispatch path: before writing a bar, try acceptance-bar/run.py. On ABSTAIN she writes it herself as now. No gate is changed, loosened or bypassed. The skill only proposes bars, and the precheck still judges every one.
Stage 3 · Hand her the compiler. Piper runs the compiler on the next family from the Stage 0 list (likely work-ledger or selfstate queries) and promotes her own skill through the same tests. From here the skills are hers to build, per the scorecard split (we spec and score, she builds her own system).
Stage 4 · The recursive step. When she changes the compiler, run the old and new compiler on the same trace set. If the new one finds a skill the old one can't, and that skill goes active and pays for itself, that is a concrete improvement chain: she improved the thing that improves her procedures, and it produced a capability the earlier version couldn't. This costs one extra offline compiler run, not a freeze.
6. What we measure (quick reads, not windows)
For each skill, before and after it goes active:
| Measure | Want |
|---|---|
| First-try precheck pass rate on her dispatches (skill #1) | up, from today's figure |
| Model calls / tokens per task in the family | down |
| Re-sends per deliverable | down |
| Wrong invocations (ran when it shouldn't have, caught by postconditions or the shadow sample) | ~0 |
| Abstention rate | reported; very high means preconditions too tight |
| Break-even: compile cost ÷ saving per use, against observed uses per week | pays back inside ~2 weeks, or it isn't promoted |
Reading the numbers: the reviewer's example ($2 compile, $0.048 saved per use, 42 uses to break even) is illustrative. Stage 0 replaces it with her real per-run cost. A family that runs 1,500 times a week clears break-even almost regardless of cost. A family that runs twice a week doesn't get compiled.
Capability, not just cost. Fewer refused bars means less wasted work and fewer silent false greens (an ALREADY_GREEN bar can't show the work happened). That counts toward the capability objective, not just the budget.
7. Guard rails, kept short
- No gate touched. Skills propose; existing gates still judge. Root-owned piper-gates stay propose-only.
- Preconditions run before execution, never left to the model's judgement alone.
- Probation on dependency change. This matters because the simplification sphere (§8) may change the very rules a skill was built against.
- Premature compilation (a one-off workaround turned into permanent procedure): promotion needs repeated evidence across ≥5 distinct deliverables, mutants caught, and the break-even met.
- Not a reason to hold back. After 8 November the operator talks to her, tinkers and codes with her freely. Operator or Claude involvement in a skill is labelled in provenance, never treated as contamination.
8. Where it sits in the queue
| When | What |
|---|---|
| now – 8 Nov | Stage 0 mining (offline, read-only); draft skill 1a; build test and mutant sets |
| 8 Nov | freeze ends |
| 9 Nov 00:05 | surprise-why loop wires in (already scheduled) |
| ~10 Nov | model decision. Compiled skills are plain code and survive a model swap, which is one reason to start here |
| from ~10 Nov | Stage 1 (1a vs 1b), then Stage 2 wiring |
| ≥ 12 Nov | simplification sphere (≥3 days after surprise-why). Skill #1's depends_on the precheck gate, so a gate change there sends it to probation, not to a silent break |
| after Stage 2 reads well | Stage 3 (she compiles family #2), then Stage 4 |
The order is proposed, not fixed. Unlike the JEPA work, skill compilation needs no new dataset or model training, so it can go first.
Relation to the JEPA proposal. JEPA asks what will this change do? Skill compilation asks have I already learned how to do this? They don't run in sequence on every task. A familiar low-risk task goes straight to a skill. A new one is deliberated, and JEPA helps there once it exists. Compiled skills also produce clean, labelled (state, action, outcome) traces with explicit preconditions, which makes JEPA's dataset cheaper to build later.
9. Kill condition
If after 3 weeks of Stage 2 skill #1's first-try pass rate hasn't moved, or its wrong-invocation rate is above 2%, retire it and write down why. If Stage 3 produces no skill of hers that reaches break-even within 4 weeks, the compiler goes back to being an operator tool and this paper records the negative result.
Appendix A. External reviewer's assessment (3 November 2026), verbatim
Kept unchanged, as received from the operator. Corrections are in §3.
Skill Compilation for Piper: I like this more than JEPA as a near-term project
My assessment: 9/10 as an architectural direction, but approximately 6/10 for the specific implementation you've described.
I think skill compilation could address one of Piper's most persistent problems: she accumulates experience, but doesn't necessarily become more efficient at performing tasks she's already solved.
The critical idea is:
Piper should not have to pay the full reasoning cost of solving the same problem twice.
That is different from simply remembering how she solved it.
And potentially much more valuable.
One qualification: I couldn't independently locate the exact January 2026 ATHENA paper from the title and quotations you provided. Several unrelated projects use that name. I'm therefore treating your three-tier description as the proposed architecture, rather than attributing unverified findings to that paper.
The broader concept is supported by related research. For example, Lifting Traces to Logic (ICML 2026) demonstrates an approach to converting agent interaction traces into modular programs with explicit conditional logic. That's closely related to what you're proposing for Piper. Proceedings of Machine Learning Research
1. Why this fits Piper particularly well
Consider her existing architecture.
Piper subsystem Current function Skill compilation opportunity Trinity / persistent memory Retains accumulated experience Identify recurring successful procedures Dispatch system Executes engineering work Reuse validated execution patterns Acceptance tests Evaluate completed work Compile recurring verification procedures Change ledger Records modifications and outcomes Supply evidence for skill induction Salience / concern ranking Prioritizes attention Identify opportunities for applying skills RSI machinery Attempts self-improvement Improve how future modifications are performed The distinction is between three forms of learning.
Episodic learning: Piper remembers that a particular modification worked.
Predictive learning: Piper estimates whether a similar modification will work again.
Procedural learning: Piper possesses a validated procedure for performing the modification without reasoning through every step.
The third form is particularly interesting because it changes how Piper performs future work.
Example: Piper encounters a recurring dispatch problem
Currently, the process might look like this:
- Piper identifies a task.
- She reasons about the applicable dispatch rules.
- She examines the acceptance-test requirements.
- She creates an execution plan.
- She performs the task.
- She verifies the result.
Suppose much of that process requires repeated LLM calls.
With skill compilation, Piper could eventually recognize:
"This is the same class of problem I've successfully handled eleven times."
She then invokes an executable procedure that performs the established steps.
The LLM remains responsible for genuinely novel decisions.
The objective isn't to remove reasoning. It's to reserve reasoning for situations that require it.
2. The part I'd change: Don't compile rules. Compile successful behaviours.
This is my biggest criticism of your proposed implementation.
You suggest targeting her dispatch contract and pre-send tests because they fire frequently without producing the desired effect.
That's a reasonable place to look, but it risks repeating an existing architectural problem.
Piper already has considerable governance and validation machinery.
If you compile more checks into deterministic procedures without changing what they accomplish, you could simply make her existing inefficiencies execute faster.
I would instead distinguish three possible compilation targets.
Skill type Example Value Verification skill Execute a known test sequence and validate its result Reliability Execution skill Perform a repeatable engineering procedure Efficiency Cognitive skill Apply a proven method for choosing or improving an engineering approach Potential improvement in decision quality The third category interests me most, although it's also the most difficult to compile safely.
For example, suppose Piper repeatedly succeeds when she follows this pattern:
- Identify two overlapping rules.
- Compare their actual operating conditions.
- Construct a unified replacement.
- Test the replacement against historical cases.
- Remove the redundant rules.
- Verify that no previously permitted behaviour is blocked.
This is directly relevant to your earlier idea about consolidating redundant rules and gates.
Piper could compile the repeated structure into a parameterized procedure.
The LLM would still identify which rules might overlap and judge whether consolidation is appropriate.
But the mechanical comparison, regression testing and verification could become reusable executable components.
That's considerably more useful than simply adding another mandatory check.
3. The architecture I'd build: A procedural learning pipeline
I'd keep the three-tier concept, but add a compiler and a promotion system.
PIPER | Novel problem? | +----------+----------+ | | YES NO | | Tier 3: WILL Search skill library LLM deliberation | | Skill found? | / \ | YES NO | | | | Tier 1: SKILL | | Execute | | | | +----------------+----------+ | Verify outcome | Tier 2: HABITS Store experience | SKILL COMPILER | Find regularities | Generate candidate | Test against history | Independent validation | Promote / reject | Tier 1: SKILLSThe important addition is that a successful execution trace doesn't automatically become a skill.
It becomes evidence from which a reusable skill might be constructed.
That distinction is essential.
The skill compiler should perform five operations
1. Discover: Find recurring patterns across previously successful work.
2. Generalize: Separate what is common across executions from task-specific details.
3. Compile: Convert the common procedure into an executable routine with explicit inputs, outputs and conditions.
4. Verify: Test the new routine on historical cases and independent withheld cases, including failures.
5. Promote: Add it to Piper's active skill library only after it demonstrates acceptable reliability and cost savings.
Each compiled skill should also contain:
- Preconditions specifying when it is applicable.
- Postconditions specifying what successful execution guarantees.
- Explicit failure and abstention conditions.
- A version and provenance record.
- A rollback or recovery procedure.
- A mechanism for detecting when it becomes obsolete.
I would use versioned Python functions or tightly constrained declarative workflows for the first implementation.
They are inspectable, testable and relatively inexpensive to execute.
There's related research worth reviewing here. Autonomous Skill Development (October 2026) describes agents creating verifiable symbolic skills, storing them in versioned libraries and reusing them for deterministic computation. HEVS Publications
4. The innovation I'd add: Skill promotion AND skill demotion
Your earlier interest in getting Piper to eliminate redundant or obsolete rules applies directly here.
A skill library that only grows will eventually become another accumulation problem.
Piper should be able to recognize when a compiled procedure is no longer useful.
I'd establish four states:
State Meaning Candidate Discovered pattern, not yet validated Active Validated procedure available for execution Probation Evidence of degradation, uncertainty or changed conditions Retired No longer eligible for automatic execution Skills could enter probation following repeated failures, dependency changes, conflicting results or degradation against a newer method.
A retired skill would remain available for historical analysis and reproducibility.
Crucially, the skill's applicability conditions should be checked before the procedure runs, not left entirely to the LLM.
If those conditions are not satisfied, Piper returns to deliberation.
That prevents a learned habit from becoming an inflexible rule.
5. The relationship between JEPA, JEV and skill compilation
This is where I think the broader architecture becomes interesting.
Assuming JEV is your proposed decision-selection mechanism, you could organize the three systems as follows:
System Question answered JEPA What consequences should I expect? JEV Which action should I choose? Skill compiler Have I learned how to perform this efficiently? But I would avoid making them three mandatory stages for every decision.
For a familiar, low-risk task, the system might go straight to a validated skill.
For an unfamiliar or high-risk task, it should deliberate, consider predictions, choose an action and verify the outcome.
The skill compiler works primarily afterward, studying the traces and determining whether a reusable procedure exists.
A good architecture reduces unnecessary deliberation rather than running every cognitive subsystem on every task.
This also creates a useful division between two kinds of uncertainty.
JEPA can estimate uncertainty about an action's consequences.
The skill system can determine whether a previously validated procedure applies to the current situation.
If either uncertainty is excessive, Piper should deliberate rather than execute automatically.
6. The experiment I would run first
I would not immediately turn the compiler loose on Piper's dispatch contract.
Instead, I'd identify one narrow, repetitive task family with reliable historical evidence.
The repeated acceptance-test workflow would be a reasonable candidate, provided you first address the vacuous-test problem identified in the earlier proposal.
Experiment: Can Piper compile a reusable verification procedure?
Phase A — Baseline
Select approximately 30–50 representative, independent verification tasks.
Measure Piper performing them through her existing process.
Record total tokens, LLM calls, elapsed time, verification accuracy and failures.
Phase B — Compilation
Give the compiler several successful and unsuccessful traces from a separate development set.
Have it construct a reusable verification procedure.
Validate the resulting procedure against hidden positive and negative cases.
In particular, use deliberately broken implementations to confirm the procedure detects failures.
Phase C — Comparison
Run the original and compiled approaches on matched tasks.
Evaluate:
Metric Desired result LLM token consumption Lower Execution time Lower Verification accuracy Equal or better Missed regressions No increase Incorrect skill invocation Near zero Recovery from unexpected inputs Equal or better Total cost, including compilation Lower over repeated use Also compare against a simple, manually written procedure.
Otherwise, you might demonstrate only that deterministic code is cheaper than LLM deliberation, rather than that Piper has learned to produce useful procedures herself.
A critical measurement: Break-even point
Compiling a skill has a cost.
Suppose:
- Discovering, generating and validating a skill costs $2.
- Each ordinary execution costs $0.05.
- Each compiled execution costs $0.002.
The skill saves $0.048 per invocation.
It would take approximately 42 invocations to recover the initial compilation cost.
That means a skill used twice might not be worth compiling, even if it works perfectly.
I'd make expected reuse frequency part of the promotion decision.
For Piper, whose operating budget matters, this is especially relevant.
7. The connection to RSI is stronger than it first appears
Consider three progressively more interesting outcomes.
Level 1: Piper compiles a skill
Piper discovers that she repeatedly performs the same verification procedure.
She creates a validated executable version.
Future verification becomes cheaper.
Result: Procedural learning and efficiency improvement.
Useful, but not necessarily recursive.
Level 2: Piper compiles a skill that improves her engineering
Piper discovers that a particular method of debugging repeatedly produces successful results.
She compiles its reusable components.
Future engineering tasks become more efficient, enabling more attempts within the same budget.
Result: An improvement to her engineering process.
This could produce persistent capability gains, but would still require evidence that those gains actually occur.
Level 3: Piper improves her own skill compiler
Now imagine Piper notices that her compiler struggles to recognize equivalent procedures with different variable names.
She independently modifies its generalization mechanism.
The modification passes independent evaluation.
The improved compiler subsequently discovers a useful skill that the previous compiler couldn't generate.
That new skill improves another part of her engineering system.
Now you have a plausible candidate for a causally supported recursive improvement chain.
You would still need to establish the counterfactual: would the additional skill have been discovered without the compiler modification?
But the chain is much more concrete than a general claim that Piper is becoming smarter.
8. One concern: The 63-versus-16 measurement
You mentioned that Piper's pre-send tests fired 63 times against a bar of 16, but were read rather than used.
I would be careful with that interpretation.
A high firing count establishes exposure, not effectiveness.
It doesn't establish whether Piper ignored the tests, whether they provided no new information, or whether they prevented mistakes that would otherwise have occurred.
Before compiling those checks into procedures, I'd measure:
How many decisions would have differed if the tests had not run?
Then measure whether those differences would have improved or worsened outcomes.
If the answer is essentially zero, the appropriate intervention might be eliminating redundant checks rather than compiling them.
This distinction matters because your goal is not to increase procedural activity.
It's to improve outcomes while reducing unnecessary computation.
9. My biggest concern: Premature compilation
Here's a failure mode I would specifically warn against.
Piper encounters a difficult problem, eventually solves it, and decides the solution should become a reusable skill.
But perhaps the solution worked only because of unusual circumstances.
If she compiles it too aggressively, she could turn a one-off workaround into a permanent procedure.
The resulting system might become faster but less adaptable.
This is why I'd require:
Repeated evidence + generalization tests + explicit applicability conditions before promotion.
I would also reserve some recurring executions for the original deliberative process.
That gives Piper an ongoing comparison against which to detect skill degradation.
The worst outcome would be a system that becomes increasingly efficient at repeating obsolete solutions.
10. Where I'd place this among Piper's potential improvements
Project Primary benefit Implementation difficulty Near-term measurability JEPA predictive world model Anticipating consequences High Moderate Skill compilation Reducing repeated reasoning Moderate High Automated rule consolidation Reducing architectural complexity Moderate High Counterfactual decision engine Improving action selection High Moderate I'd put skill compilation ahead of JEPA in the development queue.
Not because it has greater theoretical potential, but because it appears easier to implement, validate and evaluate with Piper's existing data.
It could also help make JEPA development cheaper later.
A successful skill compiler might automate portions of dataset validation, evaluation, experiment setup and regression testing.
However, I would initially keep it separate from changes to Piper's governing mechanisms.
That preserves the integrity of the current RSI measurements.
My recommendation
Build a self-compiling procedural memory system, not merely another collection of reusable functions.
The difference is that Piper herself should eventually be able to:
- Recognize repeated successful behaviour.
- Identify which parts are generalizable.
- Produce a candidate executable procedure.
- Verify that procedure independently.
- Promote it into her active skill library.
- Recognize when it becomes obsolete and replace or retire it.
The first few skills may be mundane.
That's fine.
The objective is to establish a reliable mechanism through which experience can be transformed into enduring, reusable capability.
The question I'd ultimately want Piper to answer
"Can I solve a problem today in a way that permanently reduces the amount of intelligence required to solve similar problems tomorrow?"
That's a meaningful form of cumulative procedural learning.
And if she eventually learns how to improve the machinery performing that compilation, you have another potential route toward recursive self-improvement.
Bottom line: I consider skill compilation a stronger near-term engineering investment than JEPA. I'd run it as an isolated experiment after November 8, beginning with one repetitive verification task family and requiring evidence of both correctness and net cost savings before integrating it into Piper's normal workflow.