Piper: Predictive Self-Improvement Engine
JEPA Integration and Causal Evaluation Roadmap
Version: 0.2
Status: Proposed — not implemented
Classification: Operator-originated research programme
Primary objective: Improve Piper's ability to predict the consequences of her own engineering decisions.
1. Research hypothesis
Piper's ability to improve herself may be constrained partly by an inability to accurately predict the consequences of proposed modifications.
A learned, action-conditioned predictive model may improve her selection of engineering actions by estimating likely benefits, risks and costs before implementation.
Primary hypothesis:
Access to an independently validated predictive model will increase Piper's rate of verified, retained engineering improvements per unit of compute expenditure.
Secondary hypothesis:
A JEPA-style latent transition model will outperform simpler predictive methods when evaluating unfamiliar engineering modifications.
Exploratory RSI hypothesis:
Improvements to Piper's prediction and decision machinery may increase her subsequent ability to improve that same machinery, creating measurable causal depth.
These hypotheses must be evaluated separately.
A successful predictor does not establish RSI.
2. Architecture
The proposed system consists of five components.
Component A — State encoder
Encodes Piper's relevant operational state before an engineering action.
Inputs may include:
- Cognitive performance measurements
- Recent engineering history
- Active failures and unresolved defects
- Memory retrieval performance
- Available compute and token budget
- Current model identity
- Relevant rules and gates
- Existing subsystem dependencies
Only information available at decision time may be used.
Post-outcome information must be excluded from prediction inputs.
Component B — Action encoder
Represents the proposed engineering modification.
Inputs include:
- Target subsystem
- Intended change
- Expected benefit
- Anticipated risks
- Affected files and dependencies
- Alternative candidate actions
- Expected execution cost
Actions must be represented consistently enough to permit comparisons between candidate modifications.
Component C — Predictive dynamics model
Learns a function:
Predicted next state = F(current state, proposed action)
The model predicts a latent representation of the relevant future system state.
Observable prediction heads estimate:
- Probability of independently verified success
- Expected magnitude of improvement
- Probability of regression
- Expected resource consumption
- Likely effects on related subsystems
- Uncertainty in each prediction
A JEPA-style implementation is one candidate architecture, not a mandatory dependency.
Component D — Decision evaluator
Compares alternative actions using predictions from Component C.
It may recommend an action, request additional information or abstain when uncertainty is excessive.
The evaluator must not bypass existing safety, verification, rollback or governance gates.
Initially, recommendations are advisory only.
Component E — Learning and verification loop
After an action is completed:
- Capture the actual resulting state.
- Independently verify the outcome.
- Compare observed outcomes with predictions.
- Record prediction errors.
- Add eligible records to the training dataset.
- Retrain or update the predictive model according to a registered schedule.
- Evaluate the new model on untouched data before promotion.
A model update is retained only if it passes the registered acceptance criteria.
3. Dataset and validity requirements
Historical dispatch logs may be used for initial development.
However, the training dataset must distinguish:
- Attempted actions
- Completed actions
- Independently verified successes
- Regressions
- Refusals
- Timeouts
- Inconclusive outcomes
- Vacuous verification tests
Completed must never be treated as equivalent to verified success.
Required protections
Temporal integrity: Features must represent information available before the action.
Leakage prevention: Related attempts, duplicated tasks and later revisions must not cross training and evaluation boundaries in ways that expose outcomes.
Model identity: Every record must identify the LLM and relevant configuration used.
Environment identity: Hardware migrations, harness changes and major infrastructure revisions must be recorded.
Outcome quality: Unverified historical results must be marked as such and excluded from the primary verified-success target.
Selection bias: Historical action outcomes must not be interpreted as proof of what would have happened under unchosen actions.
Training, validation and test periods must be established before model selection.
The final test period must remain untouched until the candidate models and scoring rules are frozen.
4. Competing predictive models
The first implementation must compare:
- Constant base-rate predictor
- Logistic regression
- Gradient-boosted decision trees
- Small supervised neural network
- Action-conditioned JEPA
- JEPA using pretrained text representations
Each model receives equivalent decision-time information.
Models are evaluated using identical splits and outcome definitions.
Primary prediction metrics:
- Brier score for binary outcomes
- Log loss
- Calibration error
- Discrimination between successful and unsuccessful actions
Additional metrics:
- Prediction error for measured performance changes
- Regression-risk detection
- Performance on unfamiliar task families
- Predictive uncertainty
- Computational cost
The existing cognition scorecard remains a separate reporting instrument.
Its score of 51.2 must not be treated as equivalent to a success probability or standard accuracy score without confirming its exact definition.
A model advances only if it provides an independently measured advantage over the registered baseline.
JEPA must additionally demonstrate a useful advantage over the strongest simpler model to justify its added complexity.
5. Experimental stages
Stage 0 — Preregistration
Before training or inspecting evaluation results:
- Freeze the dataset schema.
- Define eligible outcomes.
- Record historical verification limitations.
- Establish training, validation and test boundaries.
- Specify candidate architectures.
- Define primary metrics and acceptance thresholds.
- Define stopping conditions.
- Establish resource limits.
- Register model-selection and promotion rules.
Changes after registration must be recorded as amendments.
The final evaluation set must not be repeatedly reused for model development.
Pass: Complete, timestamped and reproducible experimental registration.
Failure: Missing outcome definitions, unprotected evaluation data or unresolved leakage.
Stage 1 — Offline prediction
Train the candidate models using historical records.
Compare each candidate against simple baselines using held-out future observations.
Evaluate calibration, outcome prediction and useful action ranking.
Use independently verified outcomes as the primary success target.
Pass: At least one candidate demonstrates a registered, measurable improvement over the strongest baseline.
Failure: Apparent gains disappear after correcting leakage, imbalance, uncertainty or verification quality.
A failed JEPA does not terminate the entire predictive-modelling programme if a simpler approach succeeds.
Stage 2 — Shadow-mode operation
Run the best qualifying predictor alongside Piper's normal engineering process.
The predictor produces forecasts but cannot influence decisions.
For every eligible action:
- Record the forecast before execution.
- Record candidate alternatives.
- Record confidence and predicted risks.
- Record the actual outcome.
- Measure prospective prediction quality.
This stage establishes whether offline performance survives deployment.
Pass: Prospective predictions retain useful calibration and discriminative performance.
Failure: Performance deteriorates beyond preregistered limits, or the model's uncertainty is not trustworthy.
No autonomous decision authority is granted during this stage.
Stage 3 — Advisory decision support
Expose predictions to Piper during engineering planning.
Piper may use the predictions when selecting actions.
Every recommendation must record:
- Available candidate actions
- Predicted outcomes
- Selected action
- Whether the prediction influenced selection
- Actual verified result
Model calls and predictions must be logged even when Piper ignores them.
Usage alone is not considered success.
Pass: Evidence that forecasts meaningfully influence choices without increasing regressions or violating existing gates.
Failure: Predictions are unused, misleading, or consistently overridden without demonstrated benefit.
Stage 4 — Controlled decision experiment
Evaluate three policies:
A: Existing engineering process
Piper selects actions using her established machinery.
B: Simple predictive assistance
Piper receives recommendations from the strongest qualifying conventional predictor.
C: JEPA assistance
Piper receives recommendations from the action-conditioned latent model.
Randomly assign comparable tasks to the conditions.
Use independent verification, equivalent resource limits and the same underlying LLM configuration.
Primary outcome:
Number of independently verified, retained improvements per fixed engineering budget.
Secondary outcomes:
- Improvement magnitude
- Regression rate
- Cost per retained improvement
- Time per retained improvement
- Prediction calibration
- Performance on unfamiliar tasks
- Failed attempts avoided
A pilot establishes feasibility and variance.
A later confirmatory experiment must use prespecified statistical criteria and sufficient sample size.
Pass: Predictive assistance demonstrates a credible improvement in engineering effectiveness.
Failure: The observed advantage is absent, unreliable or explained by additional resources.
Stage 5 — Persistent capability transfer
Determine whether predictive assistance produces improvements that survive the removal of the assisting mechanism.
Procedure:
- Establish a baseline for Piper.
- Enable the predictive decision system.
- Permit eligible engineering modifications.
- Independently verify and retain successful modifications.
- Disable the predictive decision system.
- Re-evaluate Piper under her original decision machinery.
- Compare against matched controls.
The experiment must distinguish improvements to task-specific code from improvements to Piper's general engineering capabilities.
Pass: A persistent, independently verified capability improvement remains after predictive assistance is removed.
Failure: All measured gains disappear when assistance is withdrawn.
Persistent improvement is not automatically recursive improvement.
Stage 6 — Recursive improvement experiment
Evaluate whether Piper can improve the predictive system itself and subsequently benefit from that improvement.
A valid candidate chain requires:
- Piper independently detects a weakness in her predictive or decision machinery.
- Piper proposes and implements a modification.
- Independent evaluation confirms the modification improves the machinery.
- The improved machinery changes a later engineering decision.
- That later decision produces an independently verified improvement that would not have occurred under the earlier machinery.
- Appropriate ablation or counterfactual testing supports the causal relationship.
Successful completion constitutes evidence of a causally supported improvement chain.
Claims of recursive amplification require additional evidence of repeated improvement to improvement capability.
The experiment must not equate one successful causal chain with sustained RSI.
6. Critical failure conditions
The experiment must explicitly detect:
Representation collapse: Learned latent states become uninformative or nearly constant.
Outcome leakage: Models learn details unavailable at decision time.
Verification contamination: Models learn unreliable success labels.
Distribution shift: Predictions fail after changes in models, hardware or harness architecture.
Policy-induced shift: Recommendations change Piper's action distribution, invalidating historical assumptions.
False confidence: The model makes confident predictions outside its validated operating range.
Goodhart effects: Piper optimizes predicted success instead of producing valuable improvements.
Adoption without effectiveness: The tool is used but produces no measurable improvement.
Operator contamination: Externally suggested modifications are misclassified as independently initiated.
Uncontrolled model changes: A fallback LLM or provider upgrade changes engineering performance during an experiment.
Excessive overhead: Prediction costs outweigh engineering benefits.
Every failure must be recorded alongside successful results.
7. Relationship to Piper's RSI measurement programme
This project is a candidate improvement to Piper's improvement machinery.
It does not change Piper's existing RSI classification merely by being built.
Evidence should be recorded separately for:
- Prediction performance
- Operational adoption
- Improvement-decision effectiveness
- Persistence of resulting capability
- Agent initiation and autonomy
- Causal improvement depth
- Recursive amplification
The initial project remains operator-originated.
Future modifications independently initiated by Piper may be evaluated separately against the existing RSI autonomy criteria.
An operator-originated predictive system can still produce useful engineering improvements without qualifying as autonomous RSI.
8. Minimum viable implementation
The first build should avoid unnecessary complexity.
Recommended initial scope:
- Structured event and state-transition logging
- Independent outcome-label validation
- Baseline predictive models
- One small action-conditioned JEPA candidate
- Temporal holdout evaluation
- Shadow-mode predictions
- Complete prediction and outcome provenance
No live decision authority is required for the first release.
No changes to Piper's existing RSI gates are required to establish offline predictive usefulness.
9. Success criteria
This programme is successful if predictive assistance demonstrably improves Piper's ability to select and execute valuable engineering modifications.
The strongest outcome would be evidence that:
- Piper learns predictive relationships from her own experience.
- Those predictions improve her future engineering choices.
- The resulting improvements persist.
- Piper independently improves her own predictive machinery.
- Those improvements enable further independently verified improvements.
The first three establish useful learning and decision support.
The fourth and fifth create potential evidence toward recursive self-improvement.
The claims must remain separate.
10. Governing principle
The purpose of the predictive world model is not to predict whether Piper will succeed.
Its purpose is to help Piper understand which actions are likely to change her capabilities, how those changes might propagate through her system, and which experiments are worth performing next.
Success must be measured by better real-world engineering decisions, not by model complexity, training loss or the number of predictions generated.