diff --git a/docs/superpowers/specs/2026-04-17-unified-cublas-training-graph-design.md b/docs/superpowers/specs/2026-04-17-unified-cublas-training-graph-design.md index 1f19a08cd..55e4dc33d 100644 --- a/docs/superpowers/specs/2026-04-17-unified-cublas-training-graph-design.md +++ b/docs/superpowers/specs/2026-04-17-unified-cublas-training-graph-design.md @@ -700,3 +700,96 @@ It's a reward engineering change, not a graph/GPU change: 3. Add config parameters 4. Propagate to backtest evaluator 5. Validate: train 50 epochs, compare trade frequency and val_Sharpe vs Phase 1 + +--- + +## Phase 1 Results (2026-04-18) — Measured on H100 PCIe + +### What was delivered + +- **cuGraphClone eliminated**: `std::mem::forget` takes ownership of cudarc's CudaGraph handles directly. cuGraphClone corrupted cublasLtMatmul HMMA tile scheduling on Hopper. +- **5 child graphs**: spectral + forward + ddqn + aux + adam — launched individually (parent composition deferred). +- **All features inside graph**: temporal ops, ISV, attention (12 cuBLAS GEMMs), curiosity inference (2 cuBLAS GEMMs), ensemble (2 fwd + KL gradient + trunk backward), IQL×2, IQN — all captured in child graphs. +- **Zero atomicAdd on gradient paths**: curiosity training, KAN backward, DT kernels, recursive_confidence all rewritten to per-sample/per-block output + deterministic reduce. +- **Zero cuStreamSynchronize in training loop**: IQN loss, Q-stats, DQN loss/grad_norm all use pinned device-mapped memory. Q-stats moved to epoch boundary only. +- **Zero cuMemcpyHtoDAsync in graph capture**: IQN/IQL/Attention adam_step use pinned device-mapped. +- **bf16 naming purged**: all `bf16_*()` alias calls replaced with f32 intrinsics across 4 crates. + +### Measured performance + +| Metric | Before (pre-cuBLAS) | After Phase 1 | +|--------|---------------------|---------------| +| Epoch time | 685s | ~568s (GPU-bound) | +| Per-step CPU time | 3537ms (sync-bound) | 0.1ms (fully async) | +| Per-step GPU time | ~3.5s (ungraphed) | ~4.0s (graph replay, all features) | +| Graph replay launch | N/A | 0ms (async) | +| Q-stats overhead | 4s/step (15 kernel re-run + sync) | 0 (epoch boundary only) | +| Write lock overhead | ~4s/step (async yield) | 0 (acquired once/epoch) | +| Attention | disabled (wrong cubin) | enabled (12 GEMMs, LayerNorm arg fix) | +| Curiosity inference | per-sample matmul | 2 cuBLAS GEMMs | +| Training steps/epoch | 178 | 178 | + +### Bottleneck analysis + +The CPU pipeline is now fully async — zero overhead. The epoch time is 100% GPU compute: + +``` +178 steps × 4.0s per graph replay = 712s ++ step 0 ungraphed: 3.5s ++ experience collection: 2s ++ epoch boundary (validation + sync): ~1s += ~718s total (GPU compute bound) +``` + +**The 4.0s per graph replay is the floor on single-stream.** The graph replay time +matches the ungraphed time — graph eliminates CPU launch overhead but the GPU compute +is unchanged. cuBLAS replaced manual matmul but with all new features enabled (attention, +curiosity cuBLAS, ensemble, temporal), the total kernel count is higher than before. +Each replay executes ~177 kernel nodes: +- spectral_child: ~12 kernels (spectral norm on 10 weight tensors) +- forward_child: ~60 kernels (cuBLAS trunk fwd + temporal + ISV + loss + backward) +- ddqn_child: ~15 kernels (Double DQN pass 3) +- aux_child: ~80 kernels (attention + IQL×2 + IQN + ensemble + CQL + HER + EMA) +- adam_child: ~10 kernels (mamba2 bwd + pruning + grad_norm + Adam + ISV update) + +### What Phase 2 multi-stream should target + +Based on measurements, the parallelism opportunities are: + +1. **forward_child branch parallelism** (Section 2a in spec): + - Trunk (shared): h_s1 → h_s2 is SERIAL (cuBLAS SGEMM on [B, state_dim] → [B, SH2]) + - Branches (independent): 4 branches (direction, magnitude, order, urgency) can fork after h_s2 + - Each branch: 2 cuBLAS GEMMs + bias+ReLU + 2 cuBLAS backward GEMMs + - Estimated branch time: ~25% of forward_child + - **Expected speedup**: ~1.3x on forward_child (trunk is serial, branches parallelize) + +2. **aux_child trainer parallelism** (Section 2b in spec): + - IQL high, IQL low, IQN, Attention are fully independent + - Each does: ~16-20 cuBLAS GEMMs (forward + backward + adam) + - Currently serial: ~80 kernels × sequential launch + - **Expected speedup**: ~3-4x on aux_child (4 trainers run in parallel) + +3. **Kernel profiling needed** (nsys): + - Which cuBLAS GEMMs dominate? Trunk [B, 256] × [256, 256] or branch [B, 128] × [128, 52]? + - Are small kernels (ISV, bias_add, ReLU) creating SM idle gaps? + - Is the spectral_child + adam_child contribution significant or negligible? + +### Revised targets + +| Phase | Target epoch time | Speedup | Method | +|-------|-------------------|---------|--------| +| Phase 1 (current) | 718s | 1.0x baseline | single-stream graph replay | +| Phase 2a: branch parallelism | ~550s | 1.3x | 4 branch streams in forward_child | +| Phase 2b: aux parallelism | ~250s | 2.9x | 4 aux streams (IQL×2 + IQN + Attention) | +| Phase 2a+2b combined | ~200s | 3.6x | all independent ops parallel | +| Phase 2c: double-buffer experience | ~198s | marginal | overlap collection with training | +| nsys-guided kernel optimization | ~120s | target | fuse small kernels, reduce node count | + +**Priority order**: 2b (aux parallelism) > 2a (branch parallelism) > nsys profiling > 2c. +aux_child contains ~80 of 177 kernels and all 4 trainers are fully independent — +this is the highest-impact parallelism opportunity. + +**nsys profiling** should run BEFORE implementing 2a/2b to validate assumptions: +- Are the aux kernels actually SM-bound (parallelism helps) or memory-bound (it won't)? +- What fraction of the 4s/step is trunk (serial) vs branches+aux (parallelizable)? +- Are there unexpected bottlenecks (cuBLAS workspace allocation, small kernel gaps)?