The backward branch parallelism allocated 4 × 32MB = 128MB for per-branch
cuBLAS workspaces, pushing H100 VRAM over 80GB → OOM. Forward and backward
are in the same child graph (sequential) — workspaces never conflict.
Now backward receives forward's branch_workspace_ptrs as a parameter.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Separates buffer prep (decode_actions, DtoD copies) from training pipeline
so prep runs on main stream and execute_training_pipeline runs on iqn_stream.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
grad_norm_standalone was captured in BOTH forward_child (d_logits clipping)
and adam_child (gradient norm). Same CUfunction in two captured graphs
corrupts kernel state on Hopper — the GPU replayed a corrupted node for
3120ms per step (88% of total step time).
Fix: adam_child uses grad_norm_kernel (the regular handle, not captured
elsewhere) instead of grad_norm_standalone. Each CUfunction is now captured
in exactly one child graph.
Expected: adam_child drops from 3120ms to ~30ms. Total step from ~3550ms
to ~460ms. Epoch from ~700s to ~80s.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add optional override_stream and override_workspace parameters to
execute_training_pipeline (GpuIqnHead), and forward/backward/adam_step
(GpuAttention). All existing callers pass None, None — behaviour is
unchanged. The parallel path can now dispatch these ops to dedicated
CUDA streams with separate cuBLAS workspaces.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The child graph breakdown showed adam=-1.0ms because it was measuring
elapsed(backward_end, adam_end) where adam_end is from the OLD phase
timing system, not the adam child graph. Added proper adam_child_end
event recorded after the adam_child launch.
This will reveal the actual adam_child GPU time — currently suspected
to be 3117ms (73% of step time) based on total - measured children.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Records cuEventRecord between each of the 5 child graph launches.
At epoch end, cuEventElapsedTime reports per-child GPU time:
spectral, forward, ddqn, aux, adam.
This tells us exactly which child dominates the 4s/step graph replay
cost, informing Phase 2 multi-stream parallelism priorities.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1 results: 4.0s/step GPU compute (graph replay = ungraphed, cuBLAS
replaced manual matmul but feature count increased). CPU pipeline fully
async (0.1ms/step). Total ~718s/epoch — 100% GPU bound.
Updated Phase 2 targets: aux parallelism (2b) is highest priority —
80/177 kernels with 4 independent trainers. nsys profiling should run
first to validate SM vs memory-bound assumptions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Per-step reduce_current_q_stats created GPU pipeline bubbles every 50 steps.
Each sync waited for the GPU to drain all pending graph replays before
the CPU could continue. Q-stats are now computed at epoch boundary only
(process_epoch_boundary already calls reduce_current_q_stats).
Training loop is now fully async: 178 graph launches queue in <1ms,
GPU executes back-to-back with zero pipeline stalls.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
q_stats_kernel now writes directly to q_readback_dev_ptr (pinned device-mapped).
CPU reads previous step's values from q_readback_pinned — one-step lag, zero sync.
This eliminates the LAST cuStreamSynchronize from the per-step training path.
The training loop is now fully async: graph launch → CPU returns immediately →
next step starts while GPU still executes → zero pipeline bubbles.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The function re-ran 15+ kernels (expected_q, q_mean_center, q_anchoring,
risk_budget, branch_confidence, epistemic_gate, temporal_consistency,
branch_independence, q_attention, graph_message_pass, q_denoise) outside
the graph every step — duplicating work the graph already did.
Now: q_out_buf is populated by graph replay (populate_q_out in aux_child).
reduce_current_q_stats just runs q_stats_kernel (1 kernel) + DtoH sync.
Runs every step since cost is now ~1ms, not ~4s.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
reduce_current_q_stats ran 15+ GPU kernels (expected_q, q_mean_center,
q_anchoring, risk_budget, branch_confidence, epistemic_gate, temporal_consistency,
branch_independence, q_attention, graph_message_pass, q_denoise, q_stats)
PLUS cuStreamSynchronize for DtoH readback — EVERY training step.
This added ~4s per step (matching graph replay cost), making epochs 700s
instead of ~35s. Running every 50 steps reduces Q-stats overhead to <1%
of epoch time while still tracking Q-range changes within 200 steps.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Ensemble diversity (2 extra head forward passes + KL gradient + trunk backward)
was running OUTSIDE the CUDA graph on every training step. Now runs inside
submit_aux_ops, captured in the aux_child graph for zero launch overhead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
reduce_current_q_stats() re-ran 10 kernels (mamba2_step, predictive_coding,
regime_dropout, ISV forward/gate/route, recursive_confidence, trade_plan,
plan_noise, risk_budget) every training step OUTSIDE the graph replay.
These ops already executed inside the graph — the logit buffers are current.
The redundant execution added ~4s of GPU compute per step (matching the
graph replay cost), doubling total epoch time.
Fix: compute expected_q directly from the existing logit buffers.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The async write lock (self.agent.write().await) was acquired inside the
per-step training loop, causing 696s/epoch overhead from async yield on
every iteration — 178 steps × ~4s per lock acquisition. Actual GPU compute
was only 3.6s (graph replay at 2ms/step).
Fix: acquire the lock once before the loop. The training loop is the only
writer during this phase — no contention, no need to release between steps.
Expected: epoch time drops from 700s to ~5s.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Logs step_ms for first 5 steps and every 50th step thereafter.
If steps aren't completing, the graph is hanging.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Forward: removed spurious ln_save_xnorm arg (kernel has 9 params, code passed 10).
The extra arg shifted D and B, causing CUDA_ERROR_ILLEGAL_ADDRESS on H100.
Backward: added missing residual (projected_buf) and save_mean args (kernel has
11 params, code passed 9). Without these, gamma/rstd/d_x/d_gamma/d_beta were all
shifted to wrong positions.
Both bugs existed since the cuBLAS attention rewrite but were latent because
attn_sdp_fwd was loaded from the wrong cubin (attention_kernel.cubin instead of
attention_backward_kernel.cubin), silently disabling attention.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
attn_sdp_fwd and attn_layer_norm_fwd were loaded from attention_kernel.cubin
(which only contains the legacy multihead_feature_attention) instead of
attention_backward_kernel.cubin where they actually live. This caused
CUDA_ERROR_NOT_FOUND on H100, silently disabling attention.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
IQN read_total_loss() called cuStreamSynchronize + cuMemcpyDtoH every training
step, blocking the CPU until all GPU work completed. This serialized the entire
async pipeline, causing 3537ms/step instead of <10ms.
Fix: pinned device-mapped memory for IQN total_loss (same pattern as DQN trainer).
GPU writes via device pointer, CPU reads via host pointer, zero sync.
Also re-applies the cuGraphClone elimination (was lost when agents modified
fused_training.rs) and adds recursive_confidence_reduce for deterministic
gradient accumulation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- capture_child_graph: use std::mem::forget to take ownership of cudarc's
CudaGraph handles directly. cuGraphClone corrupts cublasLtMatmul HMMA tile
scheduling state on Hopper, causing graph replay to hang with temporal ops.
- create_eval_forward_exec: instantiate directly from original graph (no clone).
Multiple execs from the same CUgraph is safe per NVIDIA docs.
- recursive_confidence_backward: replace atomicAdd into grad_buf with per-block
partial output + deterministic reduce kernel. Zero atomicAdd on gradient path.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Split into two kernels: kan_gate_backward writes per-element basis
contributions to intermediate buffers (no cross-element reduction),
then kan_grad_reduce deterministically accumulates them with one
thread per output parameter — fully eliminating atomicAdd from the
KAN spline gradient path.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three atomicAdd anti-patterns removed from Decision Transformer kernels:
1. dt_linear_backward_kernel (critical): Split into dt_linear_backward_kernel
(input gradient only, per-sample deterministic) + dt_linear_grad_kernel
(one thread per weight, loop over samples — zero atomics).
2. dt_causal_attention_kernel: Replace cross-head atomicAdd output projection
with per-head output buffer [B, H, T, E] + separate dt_sum_heads_kernel
that sums heads in fixed order.
3. dt_cross_entropy_kernel: Remove atomicAdd for total_loss, add
dt_reduce_loss_kernel (single-thread sequential sum for full determinism).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- gpu_experience_collector: add curiosity_weight param to new(); conditionally
initialize GpuCuriosityTrainer when weight > 0.001 (was always None)
- training_loop: pass hyperparams.curiosity_weight to GpuExperienceCollector::new()
- gpu_experience_collector: replace misleading 'curiosity disabled' log with
weight-conditional message; zero-init comment explains trainer populates weights
- attention_backward_kernel.cu: remove dead per-sample attention_backward_kernel
and attn_weight_grad_reduce (not loaded by any Rust code since cuBLAS rewrite);
remove helpers (f32_shfl, f32_block_max/sum_bwd) and defines only used by them;
update file header to describe current cuBLAS element-wise kernel contents
- c51_loss_kernel.cu: fix stale 'atomicAdd accumulator' comment on q_divergence
param — it is written by the reduction kernel, not via atomicAdd
Audit (task 4): curiosity_training_kernel.cu uses warp-reduced atomicAdd in
curiosity_forward_backward and curiosity_fused_zero_fwd_bwd_adam. These run
outside CUDA graphs (experience collection between epochs), so atomicAdd is
acceptable; existing DETERMINISM NOTE comments document this accurately.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
mamba2_step: replace memcpy_dtod_async (not captured in CUDA Graph) with
a mamba2_copy_enriched kernel that IS captured — fixes stale data on replay.
predictive_coding_loss: replace cuMemsetD8Async + atomicAdd with per-sample
output + c51_loss_reduce — eliminates both the uncaptured memset and the
non-deterministic atomicAdd, matching the existing loss pipeline pattern.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pod logs are now archived to MinIO before pod GC. Historical logs
accessible via `argo logs <workflow>` after pod completion.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
cudaGraphAddChildGraphNode parent replay hangs on sm_90.
Individual child launches (5 per step, ~10µs overhead) work.
Each child is instantiated (cuGraphInstantiateWithFlags) and
launched (cuGraphLaunch) sequentially on the training stream.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CLI arg was removed in cost-driven hold timing but Argo template
still passed it, causing exit code 2 on H100 deploy.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Tests had hardcoded state_dim=45/53 from before OFI was always enabled.
PPO tests use raw 45-dim data (no OFI), DQN tests use 65-dim.
899 unit tests pass, 20 GPU smoke tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
17 commits: IQL/IQN/Attention rewritten to batched cuBLAS GEMMs,
child graph architecture (5 sub-graphs in single parent launch),
temporal pipeline wired into training forward, ISV signal update,
cost-driven hold timing (no min_hold_bars), Q-spread opportunity cost,
hardcoded dimension cleanup, dead code removal.
Target: <80s epochs (from 685s), all features active.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
# Conflicts:
# docs/superpowers/plans/2026-04-17-unified-cublas-training-graph.md
# docs/superpowers/specs/2026-04-17-unified-cublas-training-graph-design.md
Adds opportunity cost to experience_env_step reward: when the model is
flat, subtract |q_gap| * opp_cost_scale from the reward. q_gap is the
model's own conviction signal (max_long_Q - max_short_Q), already
computed by experience_action_select and stored in q_gaps_buf[N].
- experience_kernels.cu: new opp_cost_scale kernel param; apply penalty
after churn penalty, before plan conviction scaling
- backtest_env_kernel.cu: matching opp_cost_scale param in both kernels;
set to 0.0 during backtest (no Q-gap available post-hoc)
- DQNHyperparameters: opportunity_cost_scale field, default 0.001
- ExperienceCollectorConfig: opportunity_cost_scale field, wired through
training_loop.rs; backtest configs default to 0.0
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removed: enforce_hold(), min_hold_bars from config/kernels/backtest.
Added: holding_cost_rate (inventory penalty), churn_threshold_bars +
churn_penalty_scale (graduated flip penalty) to reward in
experience_env_step and backtest kernels.
The model learns optimal hold timing from cost signals:
- Per-trade tx cost prevents churning (existing)
- Inventory penalty makes large positions expensive to hold
- Churn penalty graduates cost for rapid flips
- Temporal attention learns when holding cost > expected profit
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Inventory penalty makes flat optimal — model learns to do nothing.
Instead: penalize inaction proportional to model's own predicted edge
(Q-value spread). Creates virtuous cycle: better temporal attention →
higher self-imposed penalty for missed trades → more trading on signal.
No hindsight bias (uses predicted edge, not actual price change).
Micro-reward (already exists) rewards correct positioning.
Churn penalty (new) prevents rapid flips.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Temporal ops (mamba2, predictive coding, regime dropout, ISV temporal
routing, risk budget) now run inside submit_forward_ops_main() as step
1c, captured in the cuBLAS training graph. Previously these only ran in
monitoring via reduce_current_q_stats. Also adds submit_isv_signal_update
wrapper for graph-capturable ISV signal updates.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
forward_child: 4 branch streams fork after h_s2, join before loss
aux_child: IQL/IQN/attention on parallel streams
Uses CU_STREAM_CAPTURE_MODE_RELAXED with fork-join events as graph edges
Existing branch_streams[4] + events in batched_forward.rs ready to activate
Target: <40s epochs (from <80s Phase 1)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
