← Piper
Piper: Predictive Self-Improvement Engine
updated 2026-11-03 15:01
Markdown All papers

Piper: Predictive Self-Improvement Engine

JEPA Integration and Causal Evaluation Roadmap

Research programme proposal · v0.2.1 · 3 November 2026

Shared copy. Machines other than Firefly are named by their role. Nothing else differs from the full copy.

Version Date What changed
v0.2 3 Nov 2026 Operator's proposal as received. Kept unchanged here.
v0.2.1 3 Nov 2026 Sections 1–10 unchanged. Added the status note below and §11, an external reviewer's note on why this project is worth doing and how to start it.

Status: queued, not built. Nothing in this paper is running. Building the live parts waits until the freeze ends on 8 November and two other changes already queued for the period after it are installed first. The offline pieces (the sandbox, the state-transition dataset, the baseline predictors and the benches in §4 and §8) may be built before then, but nothing will be connected to Piper until the queued changes are done.

Version: 0.2
Status: Proposed — not implemented
Classification: Operator-originated research programme
Primary objective: Improve Piper's ability to predict the consequences of her own engineering decisions.


1. Research hypothesis

Piper's ability to improve herself may be constrained partly by an inability to accurately predict the consequences of proposed modifications.

A learned, action-conditioned predictive model may improve her selection of engineering actions by estimating likely benefits, risks and costs before implementation.

Primary hypothesis:

Access to an independently validated predictive model will increase Piper's rate of verified, retained engineering improvements per unit of compute expenditure.

Secondary hypothesis:

A JEPA-style latent transition model will outperform simpler predictive methods when evaluating unfamiliar engineering modifications.

Exploratory RSI hypothesis:

Improvements to Piper's prediction and decision machinery may increase her subsequent ability to improve that same machinery, creating measurable causal depth.

These hypotheses must be evaluated separately.

A successful predictor does not establish RSI.


2. Architecture

The proposed system consists of five components.

Component A — State encoder

Encodes Piper's relevant operational state before an engineering action.

Inputs may include:

Only information available at decision time may be used.

Post-outcome information must be excluded from prediction inputs.

Component B — Action encoder

Represents the proposed engineering modification.

Inputs include:

Actions must be represented consistently enough to permit comparisons between candidate modifications.

Component C — Predictive dynamics model

Learns a function:

Predicted next state = F(current state, proposed action)

The model predicts a latent representation of the relevant future system state.

Observable prediction heads estimate:

A JEPA-style implementation is one candidate architecture, not a mandatory dependency.

Component D — Decision evaluator

Compares alternative actions using predictions from Component C.

It may recommend an action, request additional information or abstain when uncertainty is excessive.

The evaluator must not bypass existing safety, verification, rollback or governance gates.

Initially, recommendations are advisory only.

Component E — Learning and verification loop

After an action is completed:

  1. Capture the actual resulting state.
  2. Independently verify the outcome.
  3. Compare observed outcomes with predictions.
  4. Record prediction errors.
  5. Add eligible records to the training dataset.
  6. Retrain or update the predictive model according to a registered schedule.
  7. Evaluate the new model on untouched data before promotion.

A model update is retained only if it passes the registered acceptance criteria.


3. Dataset and validity requirements

Historical dispatch logs may be used for initial development.

However, the training dataset must distinguish:

Completed must never be treated as equivalent to verified success.

Required protections

Temporal integrity: Features must represent information available before the action.

Leakage prevention: Related attempts, duplicated tasks and later revisions must not cross training and evaluation boundaries in ways that expose outcomes.

Model identity: Every record must identify the LLM and relevant configuration used.

Environment identity: Hardware migrations, harness changes and major infrastructure revisions must be recorded.

Outcome quality: Unverified historical results must be marked as such and excluded from the primary verified-success target.

Selection bias: Historical action outcomes must not be interpreted as proof of what would have happened under unchosen actions.

Training, validation and test periods must be established before model selection.

The final test period must remain untouched until the candidate models and scoring rules are frozen.


4. Competing predictive models

The first implementation must compare:

  1. Constant base-rate predictor
  2. Logistic regression
  3. Gradient-boosted decision trees
  4. Small supervised neural network
  5. Action-conditioned JEPA
  6. JEPA using pretrained text representations

Each model receives equivalent decision-time information.

Models are evaluated using identical splits and outcome definitions.

Primary prediction metrics:

Additional metrics:

The existing cognition scorecard remains a separate reporting instrument.

Its score of 51.2 must not be treated as equivalent to a success probability or standard accuracy score without confirming its exact definition.

A model advances only if it provides an independently measured advantage over the registered baseline.

JEPA must additionally demonstrate a useful advantage over the strongest simpler model to justify its added complexity.


5. Experimental stages

Stage 0 — Preregistration

Before training or inspecting evaluation results:

Changes after registration must be recorded as amendments.

The final evaluation set must not be repeatedly reused for model development.

Pass: Complete, timestamped and reproducible experimental registration.

Failure: Missing outcome definitions, unprotected evaluation data or unresolved leakage.

Stage 1 — Offline prediction

Train the candidate models using historical records.

Compare each candidate against simple baselines using held-out future observations.

Evaluate calibration, outcome prediction and useful action ranking.

Use independently verified outcomes as the primary success target.

Pass: At least one candidate demonstrates a registered, measurable improvement over the strongest baseline.

Failure: Apparent gains disappear after correcting leakage, imbalance, uncertainty or verification quality.

A failed JEPA does not terminate the entire predictive-modelling programme if a simpler approach succeeds.

Stage 2 — Shadow-mode operation

Run the best qualifying predictor alongside Piper's normal engineering process.

The predictor produces forecasts but cannot influence decisions.

For every eligible action:

This stage establishes whether offline performance survives deployment.

Pass: Prospective predictions retain useful calibration and discriminative performance.

Failure: Performance deteriorates beyond preregistered limits, or the model's uncertainty is not trustworthy.

No autonomous decision authority is granted during this stage.

Stage 3 — Advisory decision support

Expose predictions to Piper during engineering planning.

Piper may use the predictions when selecting actions.

Every recommendation must record:

Model calls and predictions must be logged even when Piper ignores them.

Usage alone is not considered success.

Pass: Evidence that forecasts meaningfully influence choices without increasing regressions or violating existing gates.

Failure: Predictions are unused, misleading, or consistently overridden without demonstrated benefit.

Stage 4 — Controlled decision experiment

Evaluate three policies:

A: Existing engineering process

Piper selects actions using her established machinery.

B: Simple predictive assistance

Piper receives recommendations from the strongest qualifying conventional predictor.

C: JEPA assistance

Piper receives recommendations from the action-conditioned latent model.

Randomly assign comparable tasks to the conditions.

Use independent verification, equivalent resource limits and the same underlying LLM configuration.

Primary outcome:

Number of independently verified, retained improvements per fixed engineering budget.

Secondary outcomes:

A pilot establishes feasibility and variance.

A later confirmatory experiment must use prespecified statistical criteria and sufficient sample size.

Pass: Predictive assistance demonstrates a credible improvement in engineering effectiveness.

Failure: The observed advantage is absent, unreliable or explained by additional resources.

Stage 5 — Persistent capability transfer

Determine whether predictive assistance produces improvements that survive the removal of the assisting mechanism.

Procedure:

  1. Establish a baseline for Piper.
  2. Enable the predictive decision system.
  3. Permit eligible engineering modifications.
  4. Independently verify and retain successful modifications.
  5. Disable the predictive decision system.
  6. Re-evaluate Piper under her original decision machinery.
  7. Compare against matched controls.

The experiment must distinguish improvements to task-specific code from improvements to Piper's general engineering capabilities.

Pass: A persistent, independently verified capability improvement remains after predictive assistance is removed.

Failure: All measured gains disappear when assistance is withdrawn.

Persistent improvement is not automatically recursive improvement.

Stage 6 — Recursive improvement experiment

Evaluate whether Piper can improve the predictive system itself and subsequently benefit from that improvement.

A valid candidate chain requires:

  1. Piper independently detects a weakness in her predictive or decision machinery.
  2. Piper proposes and implements a modification.
  3. Independent evaluation confirms the modification improves the machinery.
  4. The improved machinery changes a later engineering decision.
  5. That later decision produces an independently verified improvement that would not have occurred under the earlier machinery.
  6. Appropriate ablation or counterfactual testing supports the causal relationship.

Successful completion constitutes evidence of a causally supported improvement chain.

Claims of recursive amplification require additional evidence of repeated improvement to improvement capability.

The experiment must not equate one successful causal chain with sustained RSI.


6. Critical failure conditions

The experiment must explicitly detect:

Representation collapse: Learned latent states become uninformative or nearly constant.

Outcome leakage: Models learn details unavailable at decision time.

Verification contamination: Models learn unreliable success labels.

Distribution shift: Predictions fail after changes in models, hardware or harness architecture.

Policy-induced shift: Recommendations change Piper's action distribution, invalidating historical assumptions.

False confidence: The model makes confident predictions outside its validated operating range.

Goodhart effects: Piper optimizes predicted success instead of producing valuable improvements.

Adoption without effectiveness: The tool is used but produces no measurable improvement.

Operator contamination: Externally suggested modifications are misclassified as independently initiated.

Uncontrolled model changes: A fallback LLM or provider upgrade changes engineering performance during an experiment.

Excessive overhead: Prediction costs outweigh engineering benefits.

Every failure must be recorded alongside successful results.


7. Relationship to Piper's RSI measurement programme

This project is a candidate improvement to Piper's improvement machinery.

It does not change Piper's existing RSI classification merely by being built.

Evidence should be recorded separately for:

The initial project remains operator-originated.

Future modifications independently initiated by Piper may be evaluated separately against the existing RSI autonomy criteria.

An operator-originated predictive system can still produce useful engineering improvements without qualifying as autonomous RSI.


8. Minimum viable implementation

The first build should avoid unnecessary complexity.

Recommended initial scope:

No live decision authority is required for the first release.

No changes to Piper's existing RSI gates are required to establish offline predictive usefulness.


9. Success criteria

This programme is successful if predictive assistance demonstrably improves Piper's ability to select and execute valuable engineering modifications.

The strongest outcome would be evidence that:

  1. Piper learns predictive relationships from her own experience.
  2. Those predictions improve her future engineering choices.
  3. The resulting improvements persist.
  4. Piper independently improves her own predictive machinery.
  5. Those improvements enable further independently verified improvements.

The first three establish useful learning and decision support.

The fourth and fifth create potential evidence toward recursive self-improvement.

The claims must remain separate.


10. Governing principle

The purpose of the predictive world model is not to predict whether Piper will succeed.

Its purpose is to help Piper understand which actions are likely to change her capabilities, how those changes might propagate through her system, and which experiments are worth performing next.

Success must be measured by better real-world engineering decisions, not by model complexity, training loss or the number of predictions generated.


11. External reviewer's note (3 November 2026)

The note below was written by an external reviewer after reading v0.2. It is included as written.

JEPA would be one candidate implementation within the larger system.

The reason is strategic.

Piper already has considerable infrastructure for taking action, evaluating results, remembering outcomes and selecting tasks.

What appears comparatively underdeveloped is the ability to anticipate the consequences of her actions before attempting them.

That potentially affects almost everything she does.

And unlike another rule, gate or evaluation harness, it introduces a genuinely different capability.

Where I think the greatest potential lies

It's not necessarily in predicting dispatch outcomes.

It's in allowing Piper to learn relationships such as:

"The last four times I changed this kind of rule, I introduced regressions in these other components."

Or:

"Memory-retrieval improvements tend to help my engineering performance when the task involves previous failed attempts."

Or, eventually:

"Improving my ability to evaluate proposed modifications has increased the effectiveness of my subsequent modification attempts."

The first two are potentially learnable predictive relationships.

The third is the kind of causal relationship your RSI programme needs to investigate.

My recommendation: pursue the project, but don't commit to JEPA until it proves it can outperform simpler predictors.

And don't build the full decision system immediately. Start with reliable state-transition data and prospective shadow predictions.

I would preserve the November 8 freeze. Once it ends, this is a credible research direction for Piper, particularly because it targets a weakness already identified in her own measurements rather than adding another layer of governance.

The strongest version of this project is not about giving Piper a sophisticated neural network.

It's about giving her the beginnings of an experimentally grounded ability to anticipate how changing herself will affect what she can do next.