Surgical gradient projection that makes overfitting mathematically
impossible. At each training step:
1. Save g_train (from CUDA Graph forward+backward)
2. Run SEPARATE non-graph forward+backward on vaccine batch → g_val
3. Compute dot(g_train, g_val) and |g_val|² via GPU reduction kernel
4. If dot < 0 (gradients DISAGREE), project out conflicting component:
g_train -= (dot/|g_val|²) * g_val
5. Adam only sees gradient directions where train AND val agree
Two new CUDA kernels:
- gradient_dot_and_norm: parallel reduction with warp+block+atomicAdd
Computes both dot product and norm² in single pass (bandwidth optimal)
- gradient_project: conditional SAXPY (skips when dot >= 0, no branch divergence)
The vaccine runs OUTSIDE the CUDA Graph (between replay_forward and
replay_adam) because it needs conditional logic. The non-graph vaccine
forward+backward reuses the same cuBLAS/loss/backward infrastructure.
Vaccine batch is sampled from the replay buffer alongside the training
batch and passed via FusedTrainingCtx::pending_vaccine_batch.
Config: enable_gradient_vaccine=false (default). Enable for mathematical
guarantee that no gradient update makes the model worse on held-out data.
Together with bottleneck (#31): "you can only remember 2 numbers, and
those 2 numbers must work on data you haven't trained on."
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
09b6bd6eaa