spec: add Component 6 — Trajectory Backtracking (single GPU)
Checkpoint ring buffer ranked by improvement rate, plateau detection via liquid tau velocity, informed perturbation cycle (Adam reset, shrink-perturb, temp boost, LR×2), depth-limited rewind. ~150 lines in training_loop.rs, zero CUDA changes. Recovers from plateaus that prevention mechanisms (spread gradient, liquid tau) fail to avoid. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -239,6 +239,68 @@ L_sel = BCE(selectivity, clamp(loss_target, 0, 1)) // binary cross-e
|
||||
|
||||
---
|
||||
|
||||
## Component 6: Trajectory Backtracking (single GPU)
|
||||
|
||||
### Problem
|
||||
Every training run hits the Q-gap plateau at epoch ~22. The current approach (spread gradient, Q-gap momentum, liquid tau) tries to PREVENT the plateau. But prevention can fail. There is no RECOVERY mechanism — once stuck, the model trains for 178 more frozen epochs, wasting compute.
|
||||
|
||||
Shrink-and-perturb perturbs FORWARD from the stuck state. But the stuck state's optimizer momentum (Adam m/v buffers) points in a dead-end direction. Perturbing weights while keeping bad momentum rarely escapes.
|
||||
|
||||
### Solution
|
||||
Lazy trajectory backtracking: save checkpoints ranked by improvement rate, detect plateau, REWIND to the best pre-plateau checkpoint, apply informed perturbation, retry. Single GPU, sequential exploration.
|
||||
|
||||
```
|
||||
Normal: epoch 1→18 (improving, checkpoint saved) → 19→22 (plateau detected)
|
||||
Rewind: restore epoch 18 checkpoint
|
||||
Route 1: reset Adam + LR×2 → train 10 epochs → still stuck?
|
||||
Route 2: restore epoch 18 + shrink-perturb(σ=0.1) + temp boost → train 10 epochs → improving? KEEP
|
||||
Total: 42 epochs instead of 200 frozen epochs
|
||||
```
|
||||
|
||||
### Checkpoint Ring Buffer
|
||||
```rust
|
||||
struct TrajectoryCheckpoint {
|
||||
epoch: usize,
|
||||
improvement_rate: f32, // d(val_Sharpe)/d(epoch) at save time
|
||||
params_host: Vec<f32>, // DtoH copy of params_buf (~2MB)
|
||||
target_host: Vec<f32>, // DtoH copy of target_params_buf (~2MB)
|
||||
adam_step: u32, // optimizer step counter
|
||||
ema_state: EmaSnapshot, // q_gap_ema, eval_q_mean_ema, etc.
|
||||
}
|
||||
// Top-3 by improvement rate, minimum parameter distance between entries
|
||||
```
|
||||
|
||||
### Plateau Detection
|
||||
Uses existing liquid tau per-branch velocity. Plateau = all 4 branch velocities ≈ 0 for N consecutive Q-stats updates (N=50, ~5 epochs at 10 updates/epoch).
|
||||
|
||||
### Informed Perturbation Cycle
|
||||
Each rewind attempt uses a different strategy, cycling through:
|
||||
|
||||
| Route | Strategy | Rationale |
|
||||
|-------|----------|-----------|
|
||||
| 1 | Reset Adam (m=0, v=0) | Fresh gradient estimates from good weights |
|
||||
| 2 | Shrink-perturb (σ=0.1) on branch heads only | Diversify action evaluation, keep trunk features |
|
||||
| 3 | Temperature boost (2× Boltzmann + SARSA tau for 5 epochs, then cool) | Force re-exploration from good weights |
|
||||
| 4 | LR×2 for 10 epochs then restore | Stronger gradient signal to break saddle point |
|
||||
|
||||
### Properties
|
||||
- ~8MB per checkpoint (params + target + scalars), 3 checkpoints = 24MB CPU RAM
|
||||
- Zero CUDA kernel changes — all Rust in training_loop.rs
|
||||
- Max 3 rewinds per run, min 10 epochs between rewinds
|
||||
- Max 4 routes per rewind (cycle through perturbation strategies)
|
||||
- Depth limit prevents infinite loops
|
||||
- Replay buffer NOT restored (transitions are still valid data)
|
||||
- Liquid tau EMAs reset on rewind (stale velocity from pre-plateau)
|
||||
- Compatible with CUDA graph (params_buf pointer unchanged, only data changes via HtoD)
|
||||
|
||||
### Implementation
|
||||
- `BacktrackingState` struct in `crates/ml/src/trainers/dqn/trainer/mod.rs`
|
||||
- Checkpoint save/restore methods in `training_loop.rs`
|
||||
- Plateau detection reads liquid tau velocity (already computed)
|
||||
- ~150 lines total, no new files
|
||||
|
||||
---
|
||||
|
||||
## Dead Code Cleanup
|
||||
|
||||
Remove before building new system:
|
||||
|
||||
Reference in New Issue
Block a user