spec: add Component 6 — Trajectory Backtracking (single GPU)

Checkpoint ring buffer ranked by improvement rate, plateau detection
via liquid tau velocity, informed perturbation cycle (Adam reset,
shrink-perturb, temp boost, LR×2), depth-limited rewind. ~150 lines
in training_loop.rs, zero CUDA changes. Recovers from plateaus that
prevention mechanisms (spread gradient, liquid tau) fail to avoid.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-04-14 23:38:05 +02:00
parent cc232f644d
commit f5985cff14

View File

@@ -239,6 +239,68 @@ L_sel = BCE(selectivity, clamp(loss_target, 0, 1)) // binary cross-e
---
## Component 6: Trajectory Backtracking (single GPU)
### Problem
Every training run hits the Q-gap plateau at epoch ~22. The current approach (spread gradient, Q-gap momentum, liquid tau) tries to PREVENT the plateau. But prevention can fail. There is no RECOVERY mechanism — once stuck, the model trains for 178 more frozen epochs, wasting compute.
Shrink-and-perturb perturbs FORWARD from the stuck state. But the stuck state's optimizer momentum (Adam m/v buffers) points in a dead-end direction. Perturbing weights while keeping bad momentum rarely escapes.
### Solution
Lazy trajectory backtracking: save checkpoints ranked by improvement rate, detect plateau, REWIND to the best pre-plateau checkpoint, apply informed perturbation, retry. Single GPU, sequential exploration.
```
Normal: epoch 1→18 (improving, checkpoint saved) → 19→22 (plateau detected)
Rewind: restore epoch 18 checkpoint
Route 1: reset Adam + LR×2 → train 10 epochs → still stuck?
Route 2: restore epoch 18 + shrink-perturb(σ=0.1) + temp boost → train 10 epochs → improving? KEEP
Total: 42 epochs instead of 200 frozen epochs
```
### Checkpoint Ring Buffer
```rust
struct TrajectoryCheckpoint {
epoch: usize,
improvement_rate: f32, // d(val_Sharpe)/d(epoch) at save time
params_host: Vec<f32>, // DtoH copy of params_buf (~2MB)
target_host: Vec<f32>, // DtoH copy of target_params_buf (~2MB)
adam_step: u32, // optimizer step counter
ema_state: EmaSnapshot, // q_gap_ema, eval_q_mean_ema, etc.
}
// Top-3 by improvement rate, minimum parameter distance between entries
```
### Plateau Detection
Uses existing liquid tau per-branch velocity. Plateau = all 4 branch velocities ≈ 0 for N consecutive Q-stats updates (N=50, ~5 epochs at 10 updates/epoch).
### Informed Perturbation Cycle
Each rewind attempt uses a different strategy, cycling through:
| Route | Strategy | Rationale |
|-------|----------|-----------|
| 1 | Reset Adam (m=0, v=0) | Fresh gradient estimates from good weights |
| 2 | Shrink-perturb (σ=0.1) on branch heads only | Diversify action evaluation, keep trunk features |
| 3 | Temperature boost (2× Boltzmann + SARSA tau for 5 epochs, then cool) | Force re-exploration from good weights |
| 4 | LR×2 for 10 epochs then restore | Stronger gradient signal to break saddle point |
### Properties
- ~8MB per checkpoint (params + target + scalars), 3 checkpoints = 24MB CPU RAM
- Zero CUDA kernel changes — all Rust in training_loop.rs
- Max 3 rewinds per run, min 10 epochs between rewinds
- Max 4 routes per rewind (cycle through perturbation strategies)
- Depth limit prevents infinite loops
- Replay buffer NOT restored (transitions are still valid data)
- Liquid tau EMAs reset on rewind (stale velocity from pre-plateau)
- Compatible with CUDA graph (params_buf pointer unchanged, only data changes via HtoD)
### Implementation
- `BacktrackingState` struct in `crates/ml/src/trainers/dqn/trainer/mod.rs`
- Checkpoint save/restore methods in `training_loop.rs`
- Plateau detection reads liquid tau velocity (already computed)
- ~150 lines total, no new files
---
## Dead Code Cleanup
Remove before building new system: