foxhunt/docs/superpowers/specs/2026-04-16-self-improving-training-loop-design.md

Self-Improving Training Loop — Design Spec

Goal: Transform the validation backtest from a passive measurement into an active training signal. Every epoch's greedy backtest produces ground truth (actual trades, P&L, branch errors, regime performance) that automatically tunes the next epoch's training configuration — epsilon, gamma, LR per branch, replay weights, Q-value calibration, ensemble gating. Zero hardcoded schedules. The model adapts from its own performance.

Problem: The current training loop runs 100 epochs with static configurations (epsilon schedule, gamma annealing, fixed lambdas, uniform replay sampling). These configurations are hand-tuned guesses that are wrong for many states/regimes. Training Sharpe oscillates around zero because the model fights static rules that contradict its learned behavior. The validation backtest already computes rich evaluation data every epoch but throws it away after logging val_Sharpe.

Architecture: Post-validation enrichment phase added to the existing epoch loop. After the greedy backtest runs (already ~12s), 9 enrichment steps extract training signals from the eval data and apply them to the next epoch. All on GPU, ~15 seconds additional per epoch. Zero external tools — everything inside the training loop.

Companion to: OOS Performance Enhancement (training dynamics), OOS Generalization Enhancement (capacity compression), Learned Risk Management (5th branch). This spec makes ALL of those components adaptive instead of static.

---

The Epoch Structure

Epoch N:
  Phase 1: Experience Collection (existing, ~0.3s)
  Phase 2: Fused Training — C51 + Adam + auxiliary (existing, ~88s)
  Phase 3: Validation Backtest — greedy on val window (existing, ~12s)
  Phase 4: Self-Improving Enrichment (NEW, ~15s)
    ├── E1: Q-Value Reality Check
    ├── E2: Adaptive Epsilon
    ├── E3: Dynamic Gamma
    ├── E4: Trade Autopsy (per-branch LR)
    ├── E5: Ensemble Agreement Tuning
    ├── E6: Winner Distillation
    ├── E7: Hindsight Optimal Labels
    ├── E8: Curriculum Weights
    └── E9: State Confidence Update
  Phase 5: Apply enrichments (instant — write to pinned scalars)

Total epoch time: ~88s → ~103s (+17%). The enrichment phase uses data already on GPU from Phase 3.

---

E1: Q-Value Reality Check

Problem

Q-mean drifts to +4-7 over 20 epochs. The Q-values the model uses for action selection don't match actual returns. Penalty-based fixes (quadratic drift, homeostatic) fight the symptom but not the cause.

Solution

Compare predicted Q-values to actual returns from the validation backtest. Compute systematic bias. Apply a correction factor to Q-values before the next epoch's experience collection.

For each (state, action, actual_return) from val backtest:
    predicted_q = Q(state, action)  // from forward pass during eval
    bias = mean(predicted_q - actual_return) across all eval trades

// Apply correction to experience collection Q-values:
q_correction_factor = -bias  // additive correction
// Written to pinned scalar, read by experience collector

Implementation

• During Phase 3 (val backtest), save per-trade (Q_predicted, actual_return) pairs
• After backtest: compute bias = mean(Q - actual)
• Write correction to q_correction_pinned device-mapped scalar
• Experience collector reads correction: Q_adjusted = Q + q_correction_factor
• Replaces: Q-mean drift penalty, homeostatic Q-mean observable
• ~20 lines Rust in validation path + 2 lines in experience collector

---

E2: Adaptive Epsilon

Problem

Epsilon follows a fixed schedule (0.05 → 0.02 over epochs). But the model's exploration needs depend on its PERFORMANCE, not its age. A model that found an edge should exploit. A struggling model should search harder.

Solution

if eval_sharpe > 2.0:
    epsilon *= 0.8    // model is working → exploit
elif eval_sharpe > 0.5:
    epsilon *= 0.95   // model is promising → gentle exploitation
elif eval_sharpe < -0.5:
    epsilon *= 1.2    // model is struggling → explore more
// else: hold current epsilon

epsilon = epsilon.clamp(0.01, 0.15)

Implementation

• Read val_Sharpe from Phase 3
• Multiply epsilon by scale factor
• Clamp to [0.01, 0.15] safety range
• Write to existing epsilon config
• ~10 lines Rust at epoch boundary

---

E3: Dynamic Gamma from Trade Duration

Problem

Gamma is manually annealed (0.90 → 0.95) based on atom utilization. But the correct gamma depends on the model's actual TRADING BEHAVIOR — how long it holds profitable trades.

Solution

During val backtest, track per-trade holding duration:
    profitable_trades: collect holding_bars where P&L > 0
    mean_hold = mean(profitable_holding_bars)
    gamma_optimal = 1.0 - 1.0 / max(mean_hold, 5.0)
    
// Smooth update (don't jump):
adaptive_gamma = 0.9 * adaptive_gamma + 0.1 * gamma_optimal
adaptive_gamma = adaptive_gamma.clamp(0.85, 0.98)

Implementation

• During Phase 3, accumulate holding durations for profitable trades (already tracked in portfolio sim)
• Compute mean → gamma_optimal
• EMA blend with current gamma
• Write to adaptive_gamma field
• Replaces: gamma annealing based on atom utilization
• ~15 lines Rust

---

E4: Trade Autopsy (Per-Branch LR Scaling)

Problem

All 4 branches share the same learning rate. But some branches are accurate and others are not. The magnitude branch might be oversizing (61% error) while the order branch is fine (30% error). Uniform LR wastes gradient budget on accurate branches and under-trains inaccurate ones.

Solution

For each losing trade in the val backtest, determine which branch was wrong:

For each trade with P&L < 0:
    actual_dir = (price_change > 0) → Long, else → Short
    model_dir = trade.direction
    dir_error += (model_dir != actual_dir)
    
    // Magnitude: Full on a small move = error, Small on a big move = error
    actual_mag_need = |price_change| / avg_price_change
    mag_error += (trade.magnitude > actual_mag_need + 0.5)  // oversized
    
    // Similar for order and urgency

// Per-branch LR scale:
branch_lr_scale[d] = 1.0 + 0.5 * (error_rate[d] - mean_error_rate)
// Higher error → higher LR. Clamped to [0.5, 2.0].

Implementation

• During Phase 3, classify each losing trade's error source (which branch was wrong)
• Compute per-branch error rates
• Scale branch LR proportionally
• Write to per-branch LR multipliers (applied in the Adam step via the existing per-branch gradient budget)
• ~30 lines Rust in validation analysis

---

E5: Ensemble Agreement Tuning

Problem

The epistemic gate threshold (var_ema) is a slow-moving EMA that doesn't respond to actual trading performance. The threshold might be too loose (letting bad trades through) or too tight (blocking good trades).

Solution

During val backtest, compute TWO Sharpe ratios:

• Full Sharpe: all trades
• Agreement Sharpe: only trades where ensemble variance < current threshold

if agreement_sharpe > full_sharpe * 1.5:
    // Agreement filter IS helping → tighten threshold
    epistemic_threshold *= 0.9
elif agreement_sharpe < full_sharpe:
    // Agreement filter is HURTING → loosen threshold
    epistemic_threshold *= 1.1
// Clamp to [0.01, 10.0]

Implementation

• During Phase 3, record ensemble variance per trade (already computed in forward pass)
• Partition trades by variance < threshold
• Compute Sharpe for each partition
• Adjust threshold
• Write to var_ema_pinned
• ~20 lines Rust

---

E6: Winner Distillation

Problem

The replay buffer contains exploration noise — actions taken with 5% random probability. These noisy samples dilute the learning signal.

Solution

After val backtest, extract top 10% most profitable trades. Boost their PER priority in the replay buffer:

For each trade from val backtest, sorted by P&L descending:
    top_10pct = trades[:len/10]
    for trade in top_10pct:
        // Find corresponding experience in replay buffer (by bar index)
        replay_buffer.boost_priority(trade.entry_bar, boost=5.0)

Implementation

• After Phase 3, sort trades by P&L
• For top 10%, look up their bar indices in the replay buffer
• Multiply PER priority by 5.0
• Uses existing PER infrastructure (priority update is already GPU-side)
• ~15 lines Rust

---

E7: Hindsight Optimal Labels

Problem

The model only learns from its OWN actions. It never sees what the CORRECT action would have been for losing trades.

Solution

For each losing trade in val backtest, compute the optimal action using future data:

For each trade with P&L < 0:
    // What direction would have been profitable?
    optimal_dir = if price_went_up { Long } else { Short }
    // What magnitude was appropriate?
    optimal_mag = clamp(|price_change| / avg_price_change, Small, Full)
    // Counterfactual P&L
    optimal_pnl = abs(price_change) * optimal_mag - costs
    
    // Create synthetic experience
    synthetic = Experience {
        state: trade.entry_state,
        action: encode(optimal_dir, optimal_mag, trade.order, trade.urgency),
        reward: optimal_pnl,
        priority: 3.0  // medium-high priority
    }
    replay_buffer.insert(synthetic)

Implementation

• After Phase 3, process losing trades
• Compute hindsight-optimal actions (trivial: look at sign of price change)
• Insert synthetic experiences into replay buffer
• Cap at 10% of buffer size (don't overwhelm real experiences)
• ~25 lines Rust

---

E8: Curriculum Weights

Problem

Walk-forward training samples uniformly from all time ranges. But the model fails on some ranges and succeeds on others. Uniform sampling wastes capacity on regimes the model already handles.

Solution

Weight experience sampling by inverse eval performance:

// Divide val window into sub-regions (10 segments)
for segment in val_segments:
    segment_sharpe = sharpe(trades_in_segment)
    // Inverse weight: bad segments get more training
    weight[segment] = 1.0 / max(segment_sharpe, 0.1)
    
// Normalize to sum=1
weights = normalize(weights)

// Apply to replay buffer sampling
// Experiences from poor-performing time ranges get higher PER priority
for exp in replay_buffer:
    segment_idx = exp.bar_index / segment_size
    exp.priority *= weights[segment_idx]

Implementation

• Segment the val window into 10 equal parts
• Compute per-segment Sharpe from Phase 3 trades
• Inverse-weight
• Multiply into PER priorities (same mechanism as E6)
• ~20 lines Rust

---

E9: State Confidence Update

Problem

The model has no way to know if the current state is "tradeable" — similar to states where it historically profits, or similar to states where it historically loses.

Solution

Maintain a running confidence map: (h_s2_centroid, mean_P&L) pairs. Updated every epoch from val backtest data.

// K-means on h_s2 activations from val backtest (k=16 centroids)
centroids = kmeans(eval_h_s2_vectors, k=16)
for each centroid:
    nearby_trades = trades where ||h_s2 - centroid|| < radius
    centroid.confidence = mean(nearby_P&L) / std(nearby_P&L)

// During next epoch's experience collection:
// Add distance-to-nearest-profitable-centroid as auxiliary feature
// The model sees: "am I in a state space region that's historically profitable?"

Implementation

• K-means on GPU (simple kernel or use existing clustering infrastructure)
• 16 centroids × SH2 floats = 4KB
• Update every epoch from eval h_s2 + P&L data
• Inject as auxiliary signal (scale the risk branch or add as state feature)
• ~40 lines CUDA kernel + 20 lines Rust
• This is the most complex enrichment — can be deferred to phase 2

---

Safety & Robustness

All enrichments have safety guards:

Enrichment	Guard
Q-value correction	Clamped to ±2.0 (can't flip Q-values)
Epsilon	Clamped to [0.01, 0.15]
Gamma	Clamped to [0.85, 0.98], EMA smoothed
Branch LR scale	Clamped to [0.5, 2.0]
Epistemic threshold	Clamped to [0.01, 10.0]
Winner boost	Max 5× priority, max 10% of buffer
Hindsight labels	Max 10% of buffer, flagged as synthetic
Curriculum weights	Normalized to sum=1, min weight=0.1
State confidence	Updated via EMA (alpha=0.3), not replaced

If any enrichment produces NaN or Inf, it's skipped with a warning. The training loop continues with unchanged parameters.

---

What This Replaces

Static Component	Replaced By
epsilon schedule (0.05→0.02)	E2: eval Sharpe drives epsilon
gamma annealing (0.90→0.95)	E3: eval trade duration drives gamma
Q-mean drift penalty (λ=0.1)	E1: Q-value reality check (bias correction)
epistemic gate threshold (var_ema)	E5: ensemble agreement from eval
uniform replay sampling	E6+E7+E8: winner/hindsight/curriculum weights
fixed per-branch LR	E4: trade autopsy per-branch LR scaling
cost curriculum (sigmoid)	E2: eval profitability controls exploration rate

---

Implementation Order

1. E1 (Q-value reality check) — highest impact, simplest (20 lines)
2. E2 (adaptive epsilon) — 10 lines, immediate behavior change
3. E3 (dynamic gamma) — 15 lines, replaces manual annealing
4. E4 (trade autopsy) — 30 lines, targeted branch improvement
5. E5 (ensemble agreement) — 20 lines, auto-tune existing gate
6. E6 (winner distillation) — 15 lines, boost replay quality
7. E7 (hindsight labels) — 25 lines, learn from mistakes
8. E8 (curriculum weights) — 20 lines, focus on weaknesses
9. E9 (state confidence) — 60 lines, most complex, defer if needed

Total: ~215 lines of Rust (no new CUDA kernels — all enrichments are CPU-side post-processing of GPU eval data)

---

Success Criteria

Metric	Current (static)	Target (self-improving)
Training Sharpe after 25 epochs	oscillating ±0.5	sustained > 1.0
Epochs to plateau	22 (frozen Q-gap)	> 50 or no plateau
Greedy OOS Sharpe (val_Sharpe)	14-25 (one window)	> 5 across K windows
val/OOS gap	unknown (wrong metric)	< 15%
Trade count (greedy)	~60K	< 15K (quality > quantity)
Hardcoded hyperparameters	7+ (epsilon, gamma, LR, λ, threshold...)	1 (learning rate)
Per-epoch overhead	0s	~15s (17% increase)

---

Risks

Risk	Mitigation
Feedback loop amplifies errors	All corrections clamped with safety bounds
Enrichment phase too slow	All CPU-side on data already in host memory from DtoH readback
Hindsight labels create overfit	Capped at 10% of buffer, labeled as synthetic
Adaptive gamma oscillates	EMA smoothed (alpha=0.1), clamped [0.85, 0.98]
Curriculum weights collapse to one segment	Min weight 0.1, normalized
Winner distillation creates echo chamber	Max 10% boost, decays naturally via PER aging