2 new cuBLAS GEMMs: d_h_history = W_A^T @ d_gate + W_B^T @ d_x.
Extracts last 10 dims as d_ofi_embed_mamba2, accumulated across K=8
timesteps. This enables the OFI embed MLP to learn from temporal
patterns discovered by the Mamba2 SSM.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
h_history widened from [B, K, SH2] to [B, K, SH2+10]. OFI embed (10-dim)
appended per timestep. W_A/W_B projections grow to [SH2+10, STATE_D].
W_C stays [SH2, STATE_D] — operates on scan output, not history input.
The SSM temporal scan now sees order flow momentum, book aggression, and
bar duration alongside trunk features. Enables learning sequential
patterns in microstructure dynamics.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extracts [raw_ofi(8); delta_ofi(8); book_aggression(1); log_duration(1)]
= 18-dim from state vector, compresses to 10-dim via cuBLAS SGEMM +
bias+ReLU. Xavier init. Runs in training forward path before Mamba2
and attention, making the embedding available for temporal enrichment.
190 trainable params (18×10 + 10 bias). ~0.05ms per step.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Action space 81→108: direction(4) × magnitude(3) × order(3) × urgency(3).
Short(0), Hold(1), Long(2), Flat(3). Hold keeps current position with
zero transaction cost, giving the model a "do nothing" option.
Changes: trade_physics.cuh (Hold returns current_position), env_step
(Hold skips trade execution), action.rs (ExposureLevel::Hold variant),
config (branch_0_size 3→4), all match arms updated across 11 files.
Counterfactual mirror: Short↔Long, Hold↔Hold, Flat↔Flat.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
memcpy_dtod_async is NOT captured by CUDA Graph — silently skipped
during replay. This caused:
- Attention reading stale trunk data (h_s2→scratch copy skipped)
- Attention output never written back (scratch→h_s2 copy skipped)
- Backward gradients using stale logits (d_logits staging skipped)
- Vaccine comparison using stale reference (prev_grad snapshot skipped)
Attention has been effectively a NO-OP since CUDA Graph was introduced.
Fix: new copy_f32 kernel in graph_utility_kernels.cu (same pattern as
existing mamba2_copy_enriched). 11 call sites replaced across
gpu_dqn_trainer.rs and fused_training.rs. graph_safe_copy_f32() and
graph_safe_copy_f32_on() helpers for single-stream and multi-stream.
Non-graph-captured DtoD copies (checkpoint, init, eval) left as-is.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
100 bars = ~1 hour of imbalance bars. ES futures regimes persist 4-8
hours. Val_Sharpe was artificially inflated by regime autocorrelation
bleeding through the too-short purge gap. 2000 bars = ~1 full trading
day, ensuring validation data is from a different regime.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
3 pre-existing bugs fixed:
- ofi_gpu uploaded but never passed to env_step — now wired via
full_batch_states param (OFI at state[66..74) accessible in kernel)
- concat_ofi_features read indices 42/45 (portfolio features) instead
of 66/69 (actual OFI location in 96-dim state vector)
- PORTFOLIO_STRIDE expanded 30→38 for prev-OFI delta storage (ps[30..37])
Added PREV_CLOSE_SLOT=6 and PREV_MID_SLOT=22 for MFT mark-to-market.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
target_dim expansion: adds raw_open (OHLCV) and mid_price_open
(MBP-10 midpoint at bar formation) to fxcache targets. FXCACHE_VERSION
2→3 for auto-rebuild. Legacy v2 files handled with close-price fallback.
Spec v5 adds 3 pearls:
- Bar duration encoding in Mamba2 (continuous-time SSM awareness)
- Order book center of mass from all 10 MBP-10 levels (aggression signal)
- Retrospective hold quality bonus (teaches exit timing)
Plus: Hold action (4th direction), DSR Sharpe EMA fix, counterfactual
magnitude/order sign fix, MFT mid-price mark-to-market.
OFI embed MLP now 18→10 (was 16→8). Mamba2 width SH2+10 (was SH2+8).
Attention width D+10 (was D+8).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PopArt: normalize_rewards_popart_inplace was implemented but never
called from run_full_step. Without it, rank-normalized rewards [-1,+1]
spread across C51 atom range [-50,+50] — only 2% of atoms used,
near-zero C51 gradient. Now called before forward pass (Phase 0a).
Counterfactual: when do_flip=true AND cf_cycle==0 (directional mirror),
cf_reward = -reward double-negated (reward already flipped at line 1895).
Fix: cf_reward = do_flip ? reward : -reward. Affects ~1/6 of
counterfactual experiences that were teaching wrong directional signal.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The plan head's conviction output (sigmoid, [0,1]) scales rewards.
At init, conviction ≈ 0.1 (Xavier random weights through sigmoid),
which multiplied ALL rewards by 0.1 — killing the gradient signal.
The model couldn't learn meaningful Q-values, defaulted to uniform
random action selection → 1.7M trades on 4M bars → val_Sharpe -1000.
Fix: only apply conviction scaling when readiness ≥ 0.5 (plan head
mature). Before that, conviction defaults to 1.0 (identity). This
matches the existing readiness gate on position sizing (line 1416).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
populate_q_out + q_stats_reduce launch async kernels that write to
pinned device-mapped memory. The CPU was reading immediately — before
the GPU finished writing. Added cuStreamSynchronize to ensure the
kernel results are visible before the pinned memory read.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
atomicAdd inside CUDA graph replay causes non-determinism and is
unnecessary. The only correct path: reduce_current_q_stats() calls
populate_q_out() outside the graph to compute atom_stats cleanly.
Graph-captured path stays NULL for both atom_stats and q_var.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The graph-captured forward pass never calls compute_expected_q (it
works directly on logits for C51 loss). So atom_stats_buf was never
written during training. Now reduce_current_q_stats() calls
populate_q_out() first — runs compute_expected_q outside the graph
to populate atom_stats before q_stats_reduce reads them.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The first fix only patched populate_q_out(). The graph-captured path
uses replay_forward_ungraphed() which had its own null_atom_stats=0.
This is the call site that actually runs during training.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
compute_expected_q was called with null_atom_stats=0 (NULL pointer),
so the kernel skipped atom entropy/utilization accumulation entirely.
atom_stats_buf existed but was never passed. This disabled:
- G4 adaptive gamma annealing (uses atom_utilization)
- Homeostatic regularizer atom_util observable
- ISV atom utilization signal
Now passes atom_stats_buf.raw_ptr() with a cuMemsetD32 zero before
each call (kernel uses atomicAdd). Also passes q_var_buf_trainer for
per-action Q-variance computation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All gradient buffer clears now use cuMemsetD32Async (u32-wide writes)
instead of cuMemsetD8Async (byte-wide). 4x memory bandwidth utilization
for the ~33 memset calls per training step. Size params converted from
bytes to f32 element count (.num_bytes() → .len()).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mamba2_temporal_scan (forward) and mamba2_scan_backward are no longer
called — replaced by cuBLAS projection GEMMs + lightweight scan kernels.
Removes ~200 lines of dead CUDA code and 2 unused CUfunction handles.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Default was hardcoded as 64 in train_baseline_rl.rs and evaluate_baseline.rs,
overriding the config default of 32. Added num_quantiles=32 to production
config so it's explicit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Accumulate loss + grad_norm from pinned readback on EVERY step in
FusedTrainingCtx. Fixes smoke test failure where guard only ran
every 5th step (missing data when epoch has <5 steps).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Changed per_prefix_scan, per_sample, and is_weights_f32 kernels to accept
size as a const int* pointer (pinned device-mapped) instead of a baked int.
Graph replay now uses the CURRENT buffer size, not the capture-time value.
Host updates the pinned size on every insert_batch and clear.
per_sample also reads rng_step from pinned pointer (GPU-side increment).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Guard's check_host uses host-side loss/grad_norm accumulation instead of
GPU kernel. Reads from pinned device-mapped readback (one-step lag).
Fixes 8 smoke test failures from stale zero values.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Vaccine now uses prev_grad_buf inside maintenance_child (graphed).
The external sample() call was launching ~10 ungraphed PER kernels
on the training stream every 5th step — potential CUmodule corruption
and CPU-GPU serialization point.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: rng_step was passed as 0u64 (null) to increment_step_counters
kernel, causing atomicAdd on address 0 → CUDA_ERROR_ILLEGAL_ADDRESS →
cascading CUBLAS_STATUS_EXECUTION_FAILED on all subsequent operations.
Fixes:
- Replace all atomicAdd with plain writes (single-thread kernel)
- Add null guards for optional pointers (iqn_t, attn_t, rng_step)
- Allocate pinned device-mapped rng_step in GpuReplayBuffer
- Wire rng_step_dev_ptr from replay buffer to FusedTrainingCtx
- Remove stale atomicAdd comment
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move sample_proportional (prefix scan, binary search, gather x6, IS weights)
from training_loop.rs into run_full_step, captured as per_sample child graph
node. Steps 1+ replay all 13 children via single cuGraphLaunch with no
ungraphed PER kernel dispatches.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Composes 11 child graphs into single parent via cuGraphAddChildGraphNode.
Counter increments (GPU-side), IQL modulate, PER priority update all
captured as child graph nodes. PhaseEvents removed — single parent launch
makes per-child event profiling irrelevant.
Step 0 runs ungraphed + captures all 11 children + composes parent.
Steps 1+ replay via single cuGraphLaunch on the training stream.
After graph launch: only pinned scalar reads (nanoseconds, no GPU ops).
Child graph order:
counters -> spectral -> forward -> ddqn -> aux -> post_aux ->
adam_grad -> adam -> maintenance -> iql_modulate -> per_priority
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PER gather now writes directly to GpuDqnTrainer's padded buffers via
gather_f32_rows_padded, gather_f32_scalar, and gather_i32_scalar kernels
compiled into the replay buffer cubin. set_trainer_buffers() wires stable
device pointers at init; sample_proportional uses them when available,
falling back to intermediate buffers otherwise. memset_zeros calls in
the sampling hot path converted to raw cuMemsetD8Async.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds 4 GPU-native kernels replacing host-side operations on the training
hot path: gather_f32_rows_padded (row gather + 128-byte zero-pad),
gather_f32_scalar, gather_i32_scalar, and increment_step_counters (atomic
counter bumps + cosine-annealed tau — zero CPU sync per step).
Wired into build.rs (nvcc cubin compile) and gpu_dqn_trainer.rs
(GRAPH_UTILITY_CUBIN include_bytes!).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
shrink_and_perturb() is called at epoch boundaries after child graphs are
captured. shrink_perturb_kernel lives in the same CUmodule as scale_f32,
saxpy_f32, spectral_norm, adam_update, and other graphed CUfunctions.
On Hopper (sm_90), launching any CUfunction from a graphed CUmodule
ungraphed corrupts the child graph kernel state → 3100ms replay.
scale_adam_momentum() is called at cosine LR warm restarts (also after
graph capture). scale_f32_kernel is captured in forward_child — same
CUmodule violation.
Fix: add shrink_perturb_ungraphed and scale_f32_ungraphed loaded from
the existing ungraphed_module (separate CUmodule instance of the same
DQN_UTILITY_CUBIN). Both ungraphed callers now use isolated handles.
CUmodule count: 5 → 5 (ungraphed_module already existed, reused).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ROOT CAUSE: pad_states_kernel and stochastic_depth_rng_kernel were loaded from
the SAME CUmodule as graph-captured kernels (adam_update, grad_norm, saxpy, etc.)
but launched UNGRAPHED every step. On Hopper, launching ANY CUfunction from a
CUmodule that also has graph-captured CUfunctions corrupts the graph's kernel
state — causing 3100ms replay instead of ~30ms.
Fix: load from a separate ungraphed_module. Zero CUmodule contamination between
graphed and ungraphed execution contexts.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three changes to eliminate the 3100ms replay regression on Hopper:
1. CUfunction isolation: saxpy_f32_kernel was shared across 3 child graphs
(forward, aux, adam_grad). Added saxpy_f32_adam_grad and saxpy_f32_aux from
separate CUmodules. Also isolated grad_norm_standalone for post_aux_child
(was shared with forward_child's d_logits clipping path).
2. Raw cuMemsetD8Async: replaced all cudarc memset_zeros in graph-captured
functions (submit_forward_ops_main, apply_cql_gradient,
run_causal_intervention_unconditional) with raw cuMemsetD8Async which is
properly captured by CUDA Graph. 6 call sites fixed.
3. DtoD memcpy audit: all memcpy_dtod_async calls verified — large buffer
copies (grad snapshot 2.6MB, multi-horizon blend) are correct for DtoD;
no scalar copies found that should use pinned memory.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ROOT CAUSE of 3100ms adam_child: adam_update_kernel and grad_norm_finalize_kernel
were shared between post_aux_child and adam_update/forward children. On Hopper,
CUfunction sharing across child graphs corrupts kernel state.
Fix: load utility cubin from separate CUmodule for post_aux child, giving it
isolated adam_update_post_aux, grad_norm_finalize_post_aux, and scale_f32_post_aux
handles. Zero cross-child CUfunction sharing for these kernels.
Also: convert memset_zeros to raw cuMemsetD8Async in IQN (3 calls), IQL (1),
Attention (1) — eliminates cudarc device_ptr_mut() overhead during graph capture.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds try-fused-fallback-to-separate pattern to forward_target_raw,
forward_online_f32, and forward_value_head — matching the existing
pattern in forward_online_raw. Eliminates ~10-14 separate bias+ReLU
kernel launches by fusing them into the preceding cuBLAS GEMM epilogue.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add post_aux_child (selectivity + denoise + risk_sgd + multi-horizon) and
maintenance_child (causal intervention + gradient vaccine) to the CUDA graph
pipeline. All kernel launches now run inside child graphs; only kernel-free
ops (pinned readbacks, host counters, IQL modulate_td_errors on its own
CUfunction) remain outside the graph.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
