docs: update unified graph spec with Phase 1 H100 measurements
Phase 1 results: 4.0s/step GPU compute (graph replay = ungraphed, cuBLAS replaced manual matmul but feature count increased). CPU pipeline fully async (0.1ms/step). Total ~718s/epoch — 100% GPU bound. Updated Phase 2 targets: aux parallelism (2b) is highest priority — 80/177 kernels with 4 independent trainers. nsys profiling should run first to validate SM vs memory-bound assumptions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -700,3 +700,96 @@ It's a reward engineering change, not a graph/GPU change:
|
||||
3. Add config parameters
|
||||
4. Propagate to backtest evaluator
|
||||
5. Validate: train 50 epochs, compare trade frequency and val_Sharpe vs Phase 1
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 Results (2026-04-18) — Measured on H100 PCIe
|
||||
|
||||
### What was delivered
|
||||
|
||||
- **cuGraphClone eliminated**: `std::mem::forget` takes ownership of cudarc's CudaGraph handles directly. cuGraphClone corrupted cublasLtMatmul HMMA tile scheduling on Hopper.
|
||||
- **5 child graphs**: spectral + forward + ddqn + aux + adam — launched individually (parent composition deferred).
|
||||
- **All features inside graph**: temporal ops, ISV, attention (12 cuBLAS GEMMs), curiosity inference (2 cuBLAS GEMMs), ensemble (2 fwd + KL gradient + trunk backward), IQL×2, IQN — all captured in child graphs.
|
||||
- **Zero atomicAdd on gradient paths**: curiosity training, KAN backward, DT kernels, recursive_confidence all rewritten to per-sample/per-block output + deterministic reduce.
|
||||
- **Zero cuStreamSynchronize in training loop**: IQN loss, Q-stats, DQN loss/grad_norm all use pinned device-mapped memory. Q-stats moved to epoch boundary only.
|
||||
- **Zero cuMemcpyHtoDAsync in graph capture**: IQN/IQL/Attention adam_step use pinned device-mapped.
|
||||
- **bf16 naming purged**: all `bf16_*()` alias calls replaced with f32 intrinsics across 4 crates.
|
||||
|
||||
### Measured performance
|
||||
|
||||
| Metric | Before (pre-cuBLAS) | After Phase 1 |
|
||||
|--------|---------------------|---------------|
|
||||
| Epoch time | 685s | ~568s (GPU-bound) |
|
||||
| Per-step CPU time | 3537ms (sync-bound) | 0.1ms (fully async) |
|
||||
| Per-step GPU time | ~3.5s (ungraphed) | ~4.0s (graph replay, all features) |
|
||||
| Graph replay launch | N/A | 0ms (async) |
|
||||
| Q-stats overhead | 4s/step (15 kernel re-run + sync) | 0 (epoch boundary only) |
|
||||
| Write lock overhead | ~4s/step (async yield) | 0 (acquired once/epoch) |
|
||||
| Attention | disabled (wrong cubin) | enabled (12 GEMMs, LayerNorm arg fix) |
|
||||
| Curiosity inference | per-sample matmul | 2 cuBLAS GEMMs |
|
||||
| Training steps/epoch | 178 | 178 |
|
||||
|
||||
### Bottleneck analysis
|
||||
|
||||
The CPU pipeline is now fully async — zero overhead. The epoch time is 100% GPU compute:
|
||||
|
||||
```
|
||||
178 steps × 4.0s per graph replay = 712s
|
||||
+ step 0 ungraphed: 3.5s
|
||||
+ experience collection: 2s
|
||||
+ epoch boundary (validation + sync): ~1s
|
||||
= ~718s total (GPU compute bound)
|
||||
```
|
||||
|
||||
**The 4.0s per graph replay is the floor on single-stream.** The graph replay time
|
||||
matches the ungraphed time — graph eliminates CPU launch overhead but the GPU compute
|
||||
is unchanged. cuBLAS replaced manual matmul but with all new features enabled (attention,
|
||||
curiosity cuBLAS, ensemble, temporal), the total kernel count is higher than before.
|
||||
Each replay executes ~177 kernel nodes:
|
||||
- spectral_child: ~12 kernels (spectral norm on 10 weight tensors)
|
||||
- forward_child: ~60 kernels (cuBLAS trunk fwd + temporal + ISV + loss + backward)
|
||||
- ddqn_child: ~15 kernels (Double DQN pass 3)
|
||||
- aux_child: ~80 kernels (attention + IQL×2 + IQN + ensemble + CQL + HER + EMA)
|
||||
- adam_child: ~10 kernels (mamba2 bwd + pruning + grad_norm + Adam + ISV update)
|
||||
|
||||
### What Phase 2 multi-stream should target
|
||||
|
||||
Based on measurements, the parallelism opportunities are:
|
||||
|
||||
1. **forward_child branch parallelism** (Section 2a in spec):
|
||||
- Trunk (shared): h_s1 → h_s2 is SERIAL (cuBLAS SGEMM on [B, state_dim] → [B, SH2])
|
||||
- Branches (independent): 4 branches (direction, magnitude, order, urgency) can fork after h_s2
|
||||
- Each branch: 2 cuBLAS GEMMs + bias+ReLU + 2 cuBLAS backward GEMMs
|
||||
- Estimated branch time: ~25% of forward_child
|
||||
- **Expected speedup**: ~1.3x on forward_child (trunk is serial, branches parallelize)
|
||||
|
||||
2. **aux_child trainer parallelism** (Section 2b in spec):
|
||||
- IQL high, IQL low, IQN, Attention are fully independent
|
||||
- Each does: ~16-20 cuBLAS GEMMs (forward + backward + adam)
|
||||
- Currently serial: ~80 kernels × sequential launch
|
||||
- **Expected speedup**: ~3-4x on aux_child (4 trainers run in parallel)
|
||||
|
||||
3. **Kernel profiling needed** (nsys):
|
||||
- Which cuBLAS GEMMs dominate? Trunk [B, 256] × [256, 256] or branch [B, 128] × [128, 52]?
|
||||
- Are small kernels (ISV, bias_add, ReLU) creating SM idle gaps?
|
||||
- Is the spectral_child + adam_child contribution significant or negligible?
|
||||
|
||||
### Revised targets
|
||||
|
||||
| Phase | Target epoch time | Speedup | Method |
|
||||
|-------|-------------------|---------|--------|
|
||||
| Phase 1 (current) | 718s | 1.0x baseline | single-stream graph replay |
|
||||
| Phase 2a: branch parallelism | ~550s | 1.3x | 4 branch streams in forward_child |
|
||||
| Phase 2b: aux parallelism | ~250s | 2.9x | 4 aux streams (IQL×2 + IQN + Attention) |
|
||||
| Phase 2a+2b combined | ~200s | 3.6x | all independent ops parallel |
|
||||
| Phase 2c: double-buffer experience | ~198s | marginal | overlap collection with training |
|
||||
| nsys-guided kernel optimization | ~120s | target | fuse small kernels, reduce node count |
|
||||
|
||||
**Priority order**: 2b (aux parallelism) > 2a (branch parallelism) > nsys profiling > 2c.
|
||||
aux_child contains ~80 of 177 kernels and all 4 trainers are fully independent —
|
||||
this is the highest-impact parallelism opportunity.
|
||||
|
||||
**nsys profiling** should run BEFORE implementing 2a/2b to validate assumptions:
|
||||
- Are the aux kernels actually SM-bound (parallelism helps) or memory-bound (it won't)?
|
||||
- What fraction of the 4s/step is trunk (serial) vs branches+aux (parallelizable)?
|
||||
- Are there unexpected bottlenecks (cuBLAS workspace allocation, small kernel gaps)?
|
||||
|
||||
Reference in New Issue
Block a user