← Piper
Piper: Predictive Self-Improvement Engine (v0.2)
updated 2026-11-03 15:01
Markdown All papers

Piper: Predictive Self-Improvement Engine

JEPA Integration and Causal Evaluation Roadmap

Version: 0.2
Status: Proposed — not implemented
Classification: Operator-originated research programme
Primary objective: Improve Piper's ability to predict the consequences of her own engineering decisions.


1. Research hypothesis

Piper's ability to improve herself may be constrained partly by an inability to accurately predict the consequences of proposed modifications.

A learned, action-conditioned predictive model may improve her selection of engineering actions by estimating likely benefits, risks and costs before implementation.

Primary hypothesis:

Access to an independently validated predictive model will increase Piper's rate of verified, retained engineering improvements per unit of compute expenditure.

Secondary hypothesis:

A JEPA-style latent transition model will outperform simpler predictive methods when evaluating unfamiliar engineering modifications.

Exploratory RSI hypothesis:

Improvements to Piper's prediction and decision machinery may increase her subsequent ability to improve that same machinery, creating measurable causal depth.

These hypotheses must be evaluated separately.

A successful predictor does not establish RSI.


2. Architecture

The proposed system consists of five components.

Component A — State encoder

Encodes Piper's relevant operational state before an engineering action.

Inputs may include:

Only information available at decision time may be used.

Post-outcome information must be excluded from prediction inputs.

Component B — Action encoder

Represents the proposed engineering modification.

Inputs include:

Actions must be represented consistently enough to permit comparisons between candidate modifications.

Component C — Predictive dynamics model

Learns a function:

Predicted next state = F(current state, proposed action)

The model predicts a latent representation of the relevant future system state.

Observable prediction heads estimate:

A JEPA-style implementation is one candidate architecture, not a mandatory dependency.

Component D — Decision evaluator

Compares alternative actions using predictions from Component C.

It may recommend an action, request additional information or abstain when uncertainty is excessive.

The evaluator must not bypass existing safety, verification, rollback or governance gates.

Initially, recommendations are advisory only.

Component E — Learning and verification loop

After an action is completed:

  1. Capture the actual resulting state.
  2. Independently verify the outcome.
  3. Compare observed outcomes with predictions.
  4. Record prediction errors.
  5. Add eligible records to the training dataset.
  6. Retrain or update the predictive model according to a registered schedule.
  7. Evaluate the new model on untouched data before promotion.

A model update is retained only if it passes the registered acceptance criteria.


3. Dataset and validity requirements

Historical dispatch logs may be used for initial development.

However, the training dataset must distinguish:

Completed must never be treated as equivalent to verified success.

Required protections

Temporal integrity: Features must represent information available before the action.

Leakage prevention: Related attempts, duplicated tasks and later revisions must not cross training and evaluation boundaries in ways that expose outcomes.

Model identity: Every record must identify the LLM and relevant configuration used.

Environment identity: Hardware migrations, harness changes and major infrastructure revisions must be recorded.

Outcome quality: Unverified historical results must be marked as such and excluded from the primary verified-success target.

Selection bias: Historical action outcomes must not be interpreted as proof of what would have happened under unchosen actions.

Training, validation and test periods must be established before model selection.

The final test period must remain untouched until the candidate models and scoring rules are frozen.


4. Competing predictive models

The first implementation must compare:

  1. Constant base-rate predictor
  2. Logistic regression
  3. Gradient-boosted decision trees
  4. Small supervised neural network
  5. Action-conditioned JEPA
  6. JEPA using pretrained text representations

Each model receives equivalent decision-time information.

Models are evaluated using identical splits and outcome definitions.

Primary prediction metrics:

Additional metrics:

The existing cognition scorecard remains a separate reporting instrument.

Its score of 51.2 must not be treated as equivalent to a success probability or standard accuracy score without confirming its exact definition.

A model advances only if it provides an independently measured advantage over the registered baseline.

JEPA must additionally demonstrate a useful advantage over the strongest simpler model to justify its added complexity.


5. Experimental stages

Stage 0 — Preregistration

Before training or inspecting evaluation results:

Changes after registration must be recorded as amendments.

The final evaluation set must not be repeatedly reused for model development.

Pass: Complete, timestamped and reproducible experimental registration.

Failure: Missing outcome definitions, unprotected evaluation data or unresolved leakage.

Stage 1 — Offline prediction

Train the candidate models using historical records.

Compare each candidate against simple baselines using held-out future observations.

Evaluate calibration, outcome prediction and useful action ranking.

Use independently verified outcomes as the primary success target.

Pass: At least one candidate demonstrates a registered, measurable improvement over the strongest baseline.

Failure: Apparent gains disappear after correcting leakage, imbalance, uncertainty or verification quality.

A failed JEPA does not terminate the entire predictive-modelling programme if a simpler approach succeeds.

Stage 2 — Shadow-mode operation

Run the best qualifying predictor alongside Piper's normal engineering process.

The predictor produces forecasts but cannot influence decisions.

For every eligible action:

This stage establishes whether offline performance survives deployment.

Pass: Prospective predictions retain useful calibration and discriminative performance.

Failure: Performance deteriorates beyond preregistered limits, or the model's uncertainty is not trustworthy.

No autonomous decision authority is granted during this stage.

Stage 3 — Advisory decision support

Expose predictions to Piper during engineering planning.

Piper may use the predictions when selecting actions.

Every recommendation must record:

Model calls and predictions must be logged even when Piper ignores them.

Usage alone is not considered success.

Pass: Evidence that forecasts meaningfully influence choices without increasing regressions or violating existing gates.

Failure: Predictions are unused, misleading, or consistently overridden without demonstrated benefit.

Stage 4 — Controlled decision experiment

Evaluate three policies:

A: Existing engineering process

Piper selects actions using her established machinery.

B: Simple predictive assistance

Piper receives recommendations from the strongest qualifying conventional predictor.

C: JEPA assistance

Piper receives recommendations from the action-conditioned latent model.

Randomly assign comparable tasks to the conditions.

Use independent verification, equivalent resource limits and the same underlying LLM configuration.

Primary outcome:

Number of independently verified, retained improvements per fixed engineering budget.

Secondary outcomes:

A pilot establishes feasibility and variance.

A later confirmatory experiment must use prespecified statistical criteria and sufficient sample size.

Pass: Predictive assistance demonstrates a credible improvement in engineering effectiveness.

Failure: The observed advantage is absent, unreliable or explained by additional resources.

Stage 5 — Persistent capability transfer

Determine whether predictive assistance produces improvements that survive the removal of the assisting mechanism.

Procedure:

  1. Establish a baseline for Piper.
  2. Enable the predictive decision system.
  3. Permit eligible engineering modifications.
  4. Independently verify and retain successful modifications.
  5. Disable the predictive decision system.
  6. Re-evaluate Piper under her original decision machinery.
  7. Compare against matched controls.

The experiment must distinguish improvements to task-specific code from improvements to Piper's general engineering capabilities.

Pass: A persistent, independently verified capability improvement remains after predictive assistance is removed.

Failure: All measured gains disappear when assistance is withdrawn.

Persistent improvement is not automatically recursive improvement.

Stage 6 — Recursive improvement experiment

Evaluate whether Piper can improve the predictive system itself and subsequently benefit from that improvement.

A valid candidate chain requires:

  1. Piper independently detects a weakness in her predictive or decision machinery.
  2. Piper proposes and implements a modification.
  3. Independent evaluation confirms the modification improves the machinery.
  4. The improved machinery changes a later engineering decision.
  5. That later decision produces an independently verified improvement that would not have occurred under the earlier machinery.
  6. Appropriate ablation or counterfactual testing supports the causal relationship.

Successful completion constitutes evidence of a causally supported improvement chain.

Claims of recursive amplification require additional evidence of repeated improvement to improvement capability.

The experiment must not equate one successful causal chain with sustained RSI.


6. Critical failure conditions

The experiment must explicitly detect:

Representation collapse: Learned latent states become uninformative or nearly constant.

Outcome leakage: Models learn details unavailable at decision time.

Verification contamination: Models learn unreliable success labels.

Distribution shift: Predictions fail after changes in models, hardware or harness architecture.

Policy-induced shift: Recommendations change Piper's action distribution, invalidating historical assumptions.

False confidence: The model makes confident predictions outside its validated operating range.

Goodhart effects: Piper optimizes predicted success instead of producing valuable improvements.

Adoption without effectiveness: The tool is used but produces no measurable improvement.

Operator contamination: Externally suggested modifications are misclassified as independently initiated.

Uncontrolled model changes: A fallback LLM or provider upgrade changes engineering performance during an experiment.

Excessive overhead: Prediction costs outweigh engineering benefits.

Every failure must be recorded alongside successful results.


7. Relationship to Piper's RSI measurement programme

This project is a candidate improvement to Piper's improvement machinery.

It does not change Piper's existing RSI classification merely by being built.

Evidence should be recorded separately for:

The initial project remains operator-originated.

Future modifications independently initiated by Piper may be evaluated separately against the existing RSI autonomy criteria.

An operator-originated predictive system can still produce useful engineering improvements without qualifying as autonomous RSI.


8. Minimum viable implementation

The first build should avoid unnecessary complexity.

Recommended initial scope:

No live decision authority is required for the first release.

No changes to Piper's existing RSI gates are required to establish offline predictive usefulness.


9. Success criteria

This programme is successful if predictive assistance demonstrably improves Piper's ability to select and execute valuable engineering modifications.

The strongest outcome would be evidence that:

  1. Piper learns predictive relationships from her own experience.
  2. Those predictions improve her future engineering choices.
  3. The resulting improvements persist.
  4. Piper independently improves her own predictive machinery.
  5. Those improvements enable further independently verified improvements.

The first three establish useful learning and decision support.

The fourth and fifth create potential evidence toward recursive self-improvement.

The claims must remain separate.


10. Governing principle

The purpose of the predictive world model is not to predict whether Piper will succeed.

Its purpose is to help Piper understand which actions are likely to change her capabilities, how those changes might propagate through her system, and which experiments are worth performing next.

Success must be measured by better real-world engineering decisions, not by model complexity, training loss or the number of predictions generated.