diff --git a/docs/superpowers/specs/2026-04-14-cross-pollinated-branch-intelligence-design.md b/docs/superpowers/specs/2026-04-14-cross-pollinated-branch-intelligence-design.md index 5261126ba..6981c8a67 100644 --- a/docs/superpowers/specs/2026-04-14-cross-pollinated-branch-intelligence-design.md +++ b/docs/superpowers/specs/2026-04-14-cross-pollinated-branch-intelligence-design.md @@ -239,6 +239,68 @@ L_sel = BCE(selectivity, clamp(loss_target, 0, 1)) // binary cross-e --- +## Component 6: Trajectory Backtracking (single GPU) + +### Problem +Every training run hits the Q-gap plateau at epoch ~22. The current approach (spread gradient, Q-gap momentum, liquid tau) tries to PREVENT the plateau. But prevention can fail. There is no RECOVERY mechanism — once stuck, the model trains for 178 more frozen epochs, wasting compute. + +Shrink-and-perturb perturbs FORWARD from the stuck state. But the stuck state's optimizer momentum (Adam m/v buffers) points in a dead-end direction. Perturbing weights while keeping bad momentum rarely escapes. + +### Solution +Lazy trajectory backtracking: save checkpoints ranked by improvement rate, detect plateau, REWIND to the best pre-plateau checkpoint, apply informed perturbation, retry. Single GPU, sequential exploration. + +``` +Normal: epoch 1→18 (improving, checkpoint saved) → 19→22 (plateau detected) +Rewind: restore epoch 18 checkpoint +Route 1: reset Adam + LR×2 → train 10 epochs → still stuck? +Route 2: restore epoch 18 + shrink-perturb(σ=0.1) + temp boost → train 10 epochs → improving? KEEP +Total: 42 epochs instead of 200 frozen epochs +``` + +### Checkpoint Ring Buffer +```rust +struct TrajectoryCheckpoint { + epoch: usize, + improvement_rate: f32, // d(val_Sharpe)/d(epoch) at save time + params_host: Vec, // DtoH copy of params_buf (~2MB) + target_host: Vec, // DtoH copy of target_params_buf (~2MB) + adam_step: u32, // optimizer step counter + ema_state: EmaSnapshot, // q_gap_ema, eval_q_mean_ema, etc. +} +// Top-3 by improvement rate, minimum parameter distance between entries +``` + +### Plateau Detection +Uses existing liquid tau per-branch velocity. Plateau = all 4 branch velocities ≈ 0 for N consecutive Q-stats updates (N=50, ~5 epochs at 10 updates/epoch). + +### Informed Perturbation Cycle +Each rewind attempt uses a different strategy, cycling through: + +| Route | Strategy | Rationale | +|-------|----------|-----------| +| 1 | Reset Adam (m=0, v=0) | Fresh gradient estimates from good weights | +| 2 | Shrink-perturb (σ=0.1) on branch heads only | Diversify action evaluation, keep trunk features | +| 3 | Temperature boost (2× Boltzmann + SARSA tau for 5 epochs, then cool) | Force re-exploration from good weights | +| 4 | LR×2 for 10 epochs then restore | Stronger gradient signal to break saddle point | + +### Properties +- ~8MB per checkpoint (params + target + scalars), 3 checkpoints = 24MB CPU RAM +- Zero CUDA kernel changes — all Rust in training_loop.rs +- Max 3 rewinds per run, min 10 epochs between rewinds +- Max 4 routes per rewind (cycle through perturbation strategies) +- Depth limit prevents infinite loops +- Replay buffer NOT restored (transitions are still valid data) +- Liquid tau EMAs reset on rewind (stale velocity from pre-plateau) +- Compatible with CUDA graph (params_buf pointer unchanged, only data changes via HtoD) + +### Implementation +- `BacktrackingState` struct in `crates/ml/src/trainers/dqn/trainer/mod.rs` +- Checkpoint save/restore methods in `training_loop.rs` +- Plateau detection reads liquid tau velocity (already computed) +- ~150 lines total, no new files + +--- + ## Dead Code Cleanup Remove before building new system: