spec: Self-Improving Training Loop — 9 enrichments, eval-as-training

Every epoch's validation backtest feeds back into the next epoch: E1: Q-value reality check (bias correction, replaces drift penalty) E2: Adaptive epsilon (performance-driven, replaces schedule) E3: Dynamic gamma (from trade duration, replaces manual annealing) E4: Trade autopsy (per-branch LR scaling from error rates) E5: Ensemble agreement tuning (auto-tune epistemic gate) E6: Winner distillation (boost top 10% trades in replay) E7: Hindsight optimal labels (correct actions for losers) E8: Curriculum weights (oversample failure regimes) E9: State confidence (tradability scores from eval) Zero hardcoded schedules. The model adapts from its own performance. ~215 lines Rust, no new CUDA kernels. ~15s overhead per epoch. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 17:30:01 +02:00
parent caf0c07121
commit b111713330
1 changed files with 381 additions and 0 deletions
--- a/docs/superpowers/specs/2026-04-16-self-improving-training-loop-design.md
+++ b/docs/superpowers/specs/2026-04-16-self-improving-training-loop-design.md
@@ -0,0 +1,381 @@
+# Self-Improving Training Loop — Design Spec
+
+**Goal:** Transform the validation backtest from a passive measurement into an active training signal. Every epoch's greedy backtest produces ground truth (actual trades, P&L, branch errors, regime performance) that automatically tunes the next epoch's training configuration — epsilon, gamma, LR per branch, replay weights, Q-value calibration, ensemble gating. Zero hardcoded schedules. The model adapts from its own performance.
+
+**Problem:** The current training loop runs 100 epochs with static configurations (epsilon schedule, gamma annealing, fixed lambdas, uniform replay sampling). These configurations are hand-tuned guesses that are wrong for many states/regimes. Training Sharpe oscillates around zero because the model fights static rules that contradict its learned behavior. The validation backtest already computes rich evaluation data every epoch but throws it away after logging val_Sharpe.
+
+**Architecture:** Post-validation enrichment phase added to the existing epoch loop. After the greedy backtest runs (already ~12s), 9 enrichment steps extract training signals from the eval data and apply them to the next epoch. All on GPU, ~15 seconds additional per epoch. Zero external tools — everything inside the training loop.
+
+**Companion to:** OOS Performance Enhancement (training dynamics), OOS Generalization Enhancement (capacity compression), Learned Risk Management (5th branch). This spec makes ALL of those components adaptive instead of static.
+
+---
+
+## The Epoch Structure
+
+```
+Epoch N:
+  Phase 1: Experience Collection (existing, ~0.3s)
+  Phase 2: Fused Training — C51 + Adam + auxiliary (existing, ~88s)
+  Phase 3: Validation Backtest — greedy on val window (existing, ~12s)
+  Phase 4: Self-Improving Enrichment (NEW, ~15s)
+    ├── E1: Q-Value Reality Check
+    ├── E2: Adaptive Epsilon
+    ├── E3: Dynamic Gamma
+    ├── E4: Trade Autopsy (per-branch LR)
+    ├── E5: Ensemble Agreement Tuning
+    ├── E6: Winner Distillation
+    ├── E7: Hindsight Optimal Labels
+    ├── E8: Curriculum Weights
+    └── E9: State Confidence Update
+  Phase 5: Apply enrichments (instant — write to pinned scalars)
+```
+
+Total epoch time: ~88s → ~103s (+17%). The enrichment phase uses data already on GPU from Phase 3.
+
+---
+
+## E1: Q-Value Reality Check
+
+### Problem
+Q-mean drifts to +4-7 over 20 epochs. The Q-values the model uses for action selection don't match actual returns. Penalty-based fixes (quadratic drift, homeostatic) fight the symptom but not the cause.
+
+### Solution
+Compare predicted Q-values to actual returns from the validation backtest. Compute systematic bias. Apply a correction factor to Q-values before the next epoch's experience collection.
+
+```
+For each (state, action, actual_return) from val backtest:
+    predicted_q = Q(state, action)  // from forward pass during eval
+    bias = mean(predicted_q - actual_return) across all eval trades
+
+// Apply correction to experience collection Q-values:
+q_correction_factor = -bias  // additive correction
+// Written to pinned scalar, read by experience collector
+```
+
+### Implementation
+- During Phase 3 (val backtest), save per-trade (Q_predicted, actual_return) pairs
+- After backtest: compute bias = mean(Q - actual)
+- Write correction to `q_correction_pinned` device-mapped scalar
+- Experience collector reads correction: `Q_adjusted = Q + q_correction_factor`
+- Replaces: Q-mean drift penalty, homeostatic Q-mean observable
+- ~20 lines Rust in validation path + 2 lines in experience collector
+
+---
+
+## E2: Adaptive Epsilon
+
+### Problem
+Epsilon follows a fixed schedule (0.05 → 0.02 over epochs). But the model's exploration needs depend on its PERFORMANCE, not its age. A model that found an edge should exploit. A struggling model should search harder.
+
+### Solution
+```
+if eval_sharpe > 2.0:
+    epsilon *= 0.8    // model is working → exploit
+elif eval_sharpe > 0.5:
+    epsilon *= 0.95   // model is promising → gentle exploitation
+elif eval_sharpe < -0.5:
+    epsilon *= 1.2    // model is struggling → explore more
+// else: hold current epsilon
+
+epsilon = epsilon.clamp(0.01, 0.15)
+```
+
+### Implementation
+- Read val_Sharpe from Phase 3
+- Multiply epsilon by scale factor
+- Clamp to [0.01, 0.15] safety range
+- Write to existing epsilon config
+- ~10 lines Rust at epoch boundary
+
+---
+
+## E3: Dynamic Gamma from Trade Duration
+
+### Problem
+Gamma is manually annealed (0.90 → 0.95) based on atom utilization. But the correct gamma depends on the model's actual TRADING BEHAVIOR — how long it holds profitable trades.
+
+### Solution
+```
+During val backtest, track per-trade holding duration:
+    profitable_trades: collect holding_bars where P&L > 0
+    mean_hold = mean(profitable_holding_bars)
+    gamma_optimal = 1.0 - 1.0 / max(mean_hold, 5.0)
+    
+// Smooth update (don't jump):
+adaptive_gamma = 0.9 * adaptive_gamma + 0.1 * gamma_optimal
+adaptive_gamma = adaptive_gamma.clamp(0.85, 0.98)
+```
+
+### Implementation
+- During Phase 3, accumulate holding durations for profitable trades (already tracked in portfolio sim)
+- Compute mean → gamma_optimal
+- EMA blend with current gamma
+- Write to `adaptive_gamma` field
+- Replaces: gamma annealing based on atom utilization
+- ~15 lines Rust
+
+---
+
+## E4: Trade Autopsy (Per-Branch LR Scaling)
+
+### Problem
+All 4 branches share the same learning rate. But some branches are accurate and others are not. The magnitude branch might be oversizing (61% error) while the order branch is fine (30% error). Uniform LR wastes gradient budget on accurate branches and under-trains inaccurate ones.
+
+### Solution
+For each losing trade in the val backtest, determine which branch was wrong:
+
+```
+For each trade with P&L < 0:
+    actual_dir = (price_change > 0) → Long, else → Short
+    model_dir = trade.direction
+    dir_error += (model_dir != actual_dir)
+    
+    // Magnitude: Full on a small move = error, Small on a big move = error
+    actual_mag_need = |price_change| / avg_price_change
+    mag_error += (trade.magnitude > actual_mag_need + 0.5)  // oversized
+    
+    // Similar for order and urgency
+
+// Per-branch LR scale:
+branch_lr_scale[d] = 1.0 + 0.5 * (error_rate[d] - mean_error_rate)
+// Higher error → higher LR. Clamped to [0.5, 2.0].
+```
+
+### Implementation
+- During Phase 3, classify each losing trade's error source (which branch was wrong)
+- Compute per-branch error rates
+- Scale branch LR proportionally
+- Write to per-branch LR multipliers (applied in the Adam step via the existing per-branch gradient budget)
+- ~30 lines Rust in validation analysis
+
+---
+
+## E5: Ensemble Agreement Tuning
+
+### Problem
+The epistemic gate threshold (var_ema) is a slow-moving EMA that doesn't respond to actual trading performance. The threshold might be too loose (letting bad trades through) or too tight (blocking good trades).
+
+### Solution
+During val backtest, compute TWO Sharpe ratios:
+- Full Sharpe: all trades
+- Agreement Sharpe: only trades where ensemble variance < current threshold
+
+```
+if agreement_sharpe > full_sharpe * 1.5:
+    // Agreement filter IS helping → tighten threshold
+    epistemic_threshold *= 0.9
+elif agreement_sharpe < full_sharpe:
+    // Agreement filter is HURTING → loosen threshold
+    epistemic_threshold *= 1.1
+// Clamp to [0.01, 10.0]
+```
+
+### Implementation
+- During Phase 3, record ensemble variance per trade (already computed in forward pass)
+- Partition trades by variance < threshold
+- Compute Sharpe for each partition
+- Adjust threshold
+- Write to `var_ema_pinned`
+- ~20 lines Rust
+
+---
+
+## E6: Winner Distillation
+
+### Problem
+The replay buffer contains exploration noise — actions taken with 5% random probability. These noisy samples dilute the learning signal.
+
+### Solution
+After val backtest, extract top 10% most profitable trades. Boost their PER priority in the replay buffer:
+
+```
+For each trade from val backtest, sorted by P&L descending:
+    top_10pct = trades[:len/10]
+    for trade in top_10pct:
+        // Find corresponding experience in replay buffer (by bar index)
+        replay_buffer.boost_priority(trade.entry_bar, boost=5.0)
+```
+
+### Implementation
+- After Phase 3, sort trades by P&L
+- For top 10%, look up their bar indices in the replay buffer
+- Multiply PER priority by 5.0
+- Uses existing PER infrastructure (priority update is already GPU-side)
+- ~15 lines Rust
+
+---
+
+## E7: Hindsight Optimal Labels
+
+### Problem
+The model only learns from its OWN actions. It never sees what the CORRECT action would have been for losing trades.
+
+### Solution
+For each losing trade in val backtest, compute the optimal action using future data:
+
+```
+For each trade with P&L < 0:
+    // What direction would have been profitable?
+    optimal_dir = if price_went_up { Long } else { Short }
+    // What magnitude was appropriate?
+    optimal_mag = clamp(|price_change| / avg_price_change, Small, Full)
+    // Counterfactual P&L
+    optimal_pnl = abs(price_change) * optimal_mag - costs
+    
+    // Create synthetic experience
+    synthetic = Experience {
+        state: trade.entry_state,
+        action: encode(optimal_dir, optimal_mag, trade.order, trade.urgency),
+        reward: optimal_pnl,
+        priority: 3.0  // medium-high priority
+    }
+    replay_buffer.insert(synthetic)
+```
+
+### Implementation
+- After Phase 3, process losing trades
+- Compute hindsight-optimal actions (trivial: look at sign of price change)
+- Insert synthetic experiences into replay buffer
+- Cap at 10% of buffer size (don't overwhelm real experiences)
+- ~25 lines Rust
+
+---
+
+## E8: Curriculum Weights
+
+### Problem
+Walk-forward training samples uniformly from all time ranges. But the model fails on some ranges and succeeds on others. Uniform sampling wastes capacity on regimes the model already handles.
+
+### Solution
+Weight experience sampling by inverse eval performance:
+
+```
+// Divide val window into sub-regions (10 segments)
+for segment in val_segments:
+    segment_sharpe = sharpe(trades_in_segment)
+    // Inverse weight: bad segments get more training
+    weight[segment] = 1.0 / max(segment_sharpe, 0.1)
+    
+// Normalize to sum=1
+weights = normalize(weights)
+
+// Apply to replay buffer sampling
+// Experiences from poor-performing time ranges get higher PER priority
+for exp in replay_buffer:
+    segment_idx = exp.bar_index / segment_size
+    exp.priority *= weights[segment_idx]
+```
+
+### Implementation
+- Segment the val window into 10 equal parts
+- Compute per-segment Sharpe from Phase 3 trades
+- Inverse-weight
+- Multiply into PER priorities (same mechanism as E6)
+- ~20 lines Rust
+
+---
+
+## E9: State Confidence Update
+
+### Problem
+The model has no way to know if the current state is "tradeable" — similar to states where it historically profits, or similar to states where it historically loses.
+
+### Solution
+Maintain a running confidence map: (h_s2_centroid, mean_P&L) pairs. Updated every epoch from val backtest data.
+
+```
+// K-means on h_s2 activations from val backtest (k=16 centroids)
+centroids = kmeans(eval_h_s2_vectors, k=16)
+for each centroid:
+    nearby_trades = trades where ||h_s2 - centroid|| < radius
+    centroid.confidence = mean(nearby_P&L) / std(nearby_P&L)
+
+// During next epoch's experience collection:
+// Add distance-to-nearest-profitable-centroid as auxiliary feature
+// The model sees: "am I in a state space region that's historically profitable?"
+```
+
+### Implementation
+- K-means on GPU (simple kernel or use existing clustering infrastructure)
+- 16 centroids × SH2 floats = 4KB
+- Update every epoch from eval h_s2 + P&L data
+- Inject as auxiliary signal (scale the risk branch or add as state feature)
+- ~40 lines CUDA kernel + 20 lines Rust
+- This is the most complex enrichment — can be deferred to phase 2
+
+---
+
+## Safety & Robustness
+
+All enrichments have safety guards:
+
+| Enrichment | Guard |
+|-----------|-------|
+| Q-value correction | Clamped to ±2.0 (can't flip Q-values) |
+| Epsilon | Clamped to [0.01, 0.15] |
+| Gamma | Clamped to [0.85, 0.98], EMA smoothed |
+| Branch LR scale | Clamped to [0.5, 2.0] |
+| Epistemic threshold | Clamped to [0.01, 10.0] |
+| Winner boost | Max 5× priority, max 10% of buffer |
+| Hindsight labels | Max 10% of buffer, flagged as synthetic |
+| Curriculum weights | Normalized to sum=1, min weight=0.1 |
+| State confidence | Updated via EMA (alpha=0.3), not replaced |
+
+If any enrichment produces NaN or Inf, it's skipped with a warning. The training loop continues with unchanged parameters.
+
+---
+
+## What This Replaces
+
+| Static Component | Replaced By |
+|---|---|
+| epsilon schedule (0.05→0.02) | E2: eval Sharpe drives epsilon |
+| gamma annealing (0.90→0.95) | E3: eval trade duration drives gamma |
+| Q-mean drift penalty (λ=0.1) | E1: Q-value reality check (bias correction) |
+| epistemic gate threshold (var_ema) | E5: ensemble agreement from eval |
+| uniform replay sampling | E6+E7+E8: winner/hindsight/curriculum weights |
+| fixed per-branch LR | E4: trade autopsy per-branch LR scaling |
+| cost curriculum (sigmoid) | E2: eval profitability controls exploration rate |
+
+---
+
+## Implementation Order
+
+1. E1 (Q-value reality check) — highest impact, simplest (20 lines)
+2. E2 (adaptive epsilon) — 10 lines, immediate behavior change
+3. E3 (dynamic gamma) — 15 lines, replaces manual annealing
+4. E4 (trade autopsy) — 30 lines, targeted branch improvement
+5. E5 (ensemble agreement) — 20 lines, auto-tune existing gate
+6. E6 (winner distillation) — 15 lines, boost replay quality
+7. E7 (hindsight labels) — 25 lines, learn from mistakes
+8. E8 (curriculum weights) — 20 lines, focus on weaknesses
+9. E9 (state confidence) — 60 lines, most complex, defer if needed
+
+Total: ~215 lines of Rust (no new CUDA kernels — all enrichments are CPU-side post-processing of GPU eval data)
+
+---
+
+## Success Criteria
+
+| Metric | Current (static) | Target (self-improving) |
+|--------|-----------------|------------------------|
+| Training Sharpe after 25 epochs | oscillating ±0.5 | sustained > 1.0 |
+| Epochs to plateau | 22 (frozen Q-gap) | > 50 or no plateau |
+| Greedy OOS Sharpe (val_Sharpe) | 14-25 (one window) | > 5 across K windows |
+| val/OOS gap | unknown (wrong metric) | < 15% |
+| Trade count (greedy) | ~60K | < 15K (quality > quantity) |
+| Hardcoded hyperparameters | 7+ (epsilon, gamma, LR, λ, threshold...) | 1 (learning rate) |
+| Per-epoch overhead | 0s | ~15s (17% increase) |
+
+---
+
+## Risks
+
+| Risk | Mitigation |
+|------|-----------|
+| Feedback loop amplifies errors | All corrections clamped with safety bounds |
+| Enrichment phase too slow | All CPU-side on data already in host memory from DtoH readback |
+| Hindsight labels create overfit | Capped at 10% of buffer, labeled as synthetic |
+| Adaptive gamma oscillates | EMA smoothed (alpha=0.1), clamped [0.85, 0.98] |
+| Curriculum weights collapse to one segment | Min weight 0.1, normalized |
+| Winner distillation creates echo chamber | Max 10% boost, decays naturally via PER aging |