capture_graph_mega was missing disable_event_tracking() that
capture_training_graphs already had. With state_dim=96, cudarc's
device_ptr() calls inside cuBLAS record CudaEvents during graph
capture, which are disallowed and corrupt the graph on Hopper.
Also reverts c51_grad/c51_loss/dqn_utility kernel files to working
SHA — removes all symptom-chasing changes that were red herrings.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Temporarily reverts ALL experience_kernels.cu changes to match 11b1a1ca9
exactly. If graph_mega works, the cubin-level changes (new plan_noise_inject
kernel + modified experience_state_gather/env_step bodies) are affecting
register allocation for graph_mega kernels on Hopper.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
graph_mega on Hopper requires EXACT node structure match with working
SHA 11b1a1ca9. Two remaining differences caused the hang:
- c51_grad: 18→19 params (restored unused q_mean_ema_ptr)
- HER relabel: (1,1,1)→(32,1,1) (restored warp-parallel launch)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The warp-parallel (32,1,1) change to c51_loss_reduce caused the same
hang as isv_feature_gate: changing block_dim inside graph_mega alters
CUDA Graph node structure on Hopper's TMA scheduler. Must stay (1,1,1).
Also: batch_size 8192→16384, gpu_n_episodes 1024→4096, num_atoms 51→52
to align h100.toml with dqn-production.toml.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: the read→write split (save_h_s2 → gated_h_s2_buf) removed the
WAW dependency that cuBLAS's internal TMA async ops on Hopper (sm_90)
require for correct CUDA Graph replay ordering. In-place modification
restores the dependency chain: cuBLAS forward → isv_feature_gate → backward.
Also upgrades c51_loss_reduce from sequential to warp-parallel reduction
(32 threads, __shfl_down_sync) — ~32× faster for B=8192.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The block_dim (1,1,1) → (32,1,1) change introduced race conditions
in every single-thread kernel that writes to shared scalars.
Fixed stochastic_depth_rng and c51_loss_reduce individually, but
there may be more. Safest fix: revert ALL to (1,1,1) matching the
working 11b1a1ca9. These kernels are tiny (single-thread work) —
the 32-thread "optimization" wasted 31 threads and added races.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
SECOND race condition from block_dim (1,1,1)→(32,1,1) change:
c51_loss_reduce had 32 threads all computing sum and writing to
total_loss[0] simultaneously. On sm_90 (H100) → corrupted loss →
NaN in C51 gradient → graph_mega replay hangs.
This kernel runs INSIDE graph_mega (submit_forward_ops_main),
called twice (MSE reduce + C51 reduce). Both corrupted.
Also fixed stochastic_depth_rng in previous commit (same issue,
runs in pre-replay).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause of graph_mega hang: stochastic_depth_rng kernel had NO
threadIdx.x guard. When block_dim changed from (1,1,1) to (32,1,1),
all 32 threads wrote to scale_buf[0..2] and rng_state[0] simultaneously.
Race condition → corrupted stochastic depth scales → NaN in cuBLAS
forward → C51 Bellman projection diverges → infinite loop → hang.
Fix: if (threadIdx.x != 0) return; — single writer for 3 scale + 1 rng.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
plan_noise_inject is the ONLY new kernel inside graph_mega vs the
working 11b1a1ca9. Everything else is identical (submit_forward_ops_main,
submit_aux_ops, capture_graph_mega). If this fixes the hang,
plan_noise_inject is incompatible with graph_mega on sm_90.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reverted to the structure that WORKED on H100: ISV kernels inside
submit_forward_ops_main (captured in graph_mega). Removed duplicate
pre-replay calls. The working version had ISV in graph_mega — removing
them was wrong. Added plan_noise_inject to the same block.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The per-weight-element rewrite (2 blocks, B=8192 loop) caused
graph_mega replay to hang on H100 sm_90. The original warp+block
reduction (ceil(B/256) blocks, atomicAdd per block) worked on H100.
Reverted to working version. Remove duplicate fill_gamma_buf from
pre-replay (already inside graph_mega).
Root cause investigation pending — the per-weight-element pattern
works on sm_80 (RTX 3050) but hangs on sm_90 (H100).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ROOT CAUSE of graph_mega hang: the warp+block reduction in
recursive_confidence_backward had 256 iterations of __syncthreads
inside a loop (2 syncs × SH2=256 = 512 barriers per thread).
This overwhelmed the CUDA graph node scheduler on H100.
Fix: converted to per-weight-element pattern (same as other
deterministic backward kernels). One thread per weight (SH2+1=257),
loops over B samples. Zero __syncthreads, zero atomicAdd,
zero shared memory. Graph-mega safe.
Grid: ceil(257/256)=2 blocks (was ceil(8192/256)=32 blocks).
Massive resource reduction + deterministic + graph compatible.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
NVRTC-compiled ISV kernels conflict with cuBLAS GEMM internal stream
scheduling when captured in the same CUDA Graph as cuBLAS ops.
Moving ISV+plan to pre-replay phase: they run BEFORE graph_mega
replay on every training step. Their outputs (gamma_mod_buf,
branch_gate_buf, gated_h_s2_buf, predicted_error_buf, plan_params_buf)
are stable device buffers consumed by the captured graph.
Same result, no deadlock. ISV updates every step (not every 50).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ROOT CAUSE of graph_mega hang: isv_feature_gate modified save_h_s2
in-place. cuBLAS backward (later in graph_mega) reads save_h_s2
expecting ORIGINAL (pre-gate) values for correct gradient computation.
The gated values corrupted gradients → GPU hang.
Fix: separate buffers. save_h_s2 preserved for backward (unmodified).
gated_h_s2_buf receives ISV-gated output for branch heads.
recursive_confidence_forward, trade_plan_forward, risk_budget_forward
now read from gated_h_s2_buf.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CUDA Graph mega capture hangs with block_dim=(1,1,1) kernels.
Changed all ISV/plan single-thread launches to block_dim=(32,1,1).
Kernels already guard with if(threadIdx.x != 0) return — only
thread 0 does work, rest idle. Full warp is graph scheduler
friendly. Local smoke test passes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
plan_noise_inject kernel: ±5% Philox-seeded multiplicative noise on
plan_params [B, 6] after forward pass. Creates temporal diversity —
model sees slightly different plans each step, learns robust behavior.
Called in both reduce_current_q_stats and submit_forward_ops_main.
Full 3-run spatial ensemble deferred to follow-up.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ISV kernels back inside submit_forward_ops_main (graph capture).
Local smoke test: PASS (training_sharpe=3.773). graph_forward
capture works with ISV single-thread kernels. If H100 graph_mega
hangs, the issue is mega-specific, not ISV compatibility.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
AutoBatchSizer didn't account for experience collector VRAM, causing
OOM on H100 with state_dim=96. batch_size is configured per GPU
profile (h100.toml=8192, rtx3050.toml=64). No auto-sizing needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Max unrealized P&L tracked each bar during trade. Enables hindsight
plan learning: compare plan's profit_target vs actual max P&L.
The plan head learns optimal targets from what was achievable.
Reset on trade entry, reversal, and episode hard/soft reset.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
conviction_drift = current_conviction - entry_conviction: model sees
its thesis weakening. regime_shift = |stability_now - stability_entry|:
model detects regime change invalidating the plan. Entry regime stored
in ps[29]. Both features zero when no plan active.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When readiness≥0.7 and conviction changes significantly mid-trade:
+30% conviction → extend plan (thesis strengthening)
-60% conviction → shorten to exit in 3 bars (thesis collapsing)
Normal fluctuation (0.4x-1.3x) → plan unchanged.
The model learns to revise plans based on evolving assessment.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scaleway Kapsule GPU nodes need NVIDIA driver DKMS compile on cold
start (~15 min). Setting min_size=1 keeps the node warm between runs.
Scale to 0 manually when done training for the day.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Stop threshold: max(plan_stop, 0.5%) / (1 + grad_norm_instability).
Tighter when gradient pressure high, wider when stable.
Max hold: 50 * regime_stability → 10-50 bars (shorter in transitions).
Plan's own learned stop_loss is the base — not hardcoded 2%.
Zero hardcoded constants — everything flows through ISV + plan head.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
4 new portfolio features (portfolio_dim 8→12, state_dim 88→96):
- plan_progress: hold_time / target_bars (how far through plan)
- pnl_vs_target: unrealized P&L / profit_target (how close to goal)
- pnl_vs_stop: unrealized P&L / stop_loss (how close to risk limit)
- plan_conviction: conviction at entry (confidence level)
Model now reasons about its own plan through temporal attention.
"I'm 80% through, 83% to target → hold" vs "I'm past stop → exit."
Planning becomes metacognitive — the model sees AND reasons about
its intentions, not just market state.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Hard exit removed for normal plan targets (stop/profit/time).
Model LEARNS exit timing through temporal attention:
- ISV-gated trunk → Mamba2 temporal state → branch Q-values
- Conviction scales reward → model learns confidence calibration
- Portfolio observation includes hold_time, unrealized P&L, drawdown
- Model decides when to exit through Q(Flat) vs Q(Hold)
Safety net remains: hard exit only on catastrophic loss (>2%) or
extreme hold (>50 bars). Everything else is learned, not forced.
Planning, holding, strategic positioning are all attention-driven.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a plan is active, the plan's target_bars IS the hold duration.
Adaptive hold was blocking plan exits → model held positions for
entire 5000-bar episodes → 0 completed trades with massive unrealized
P&L. Now: plan_active → hold enforcement bypassed. Plan manages
its own duration. Hold is safety net for plan-less trading only.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Plan head randomly initialized → garbage plans → immediate stop-outs →
model learns to never trade. Fix: gate plan activation + enforcement
by iqn_readiness (pinned device-mapped). readiness < 0.5 → plan
disabled (model trades freely, plan head still learns from outcomes).
readiness ≥ 0.5 → plan activates and enforces. Dynamic, not hardcoded.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
FXCACHE_VERSION in header handles cache invalidation. First attempt
runs precompute — if cache is valid, skips (fast). If version
mismatch, deletes stale cache and regenerates. No more rm on every run.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
graph_adam_m/v/step were allocated but never used — graph_params
are intentionally fixed at near-identity initialization.
q_mean_ema_ptr removed from c51_grad_kernel (ISV replaced Q-drift
penalty, no ABI to maintain for compiled cubins).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ISV forward, feature gate, recursive confidence, and trade plan
were only in reduce_current_q_stats (monitoring). Now also in
submit_forward_ops_main (training). Without this, these heads
never affected training Q-values — gamma_mod, branch_gate,
feature gating, and plan params were stale during actual training.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
selectivity_backward, risk_budget_backward, mamba2_scan_backward,
q_denoise_backward — all converted to per-weight-element accumulation
(one thread per weight, loops over batch) with plain deterministic
writes. Zero atomicAdd in these gradient paths.
recursive_confidence_backward, temporal_consistency_penalty,
predictive_coding_loss, compute_expected_q atom_stats — converted to
warp+block tree reduction, reducing to one atomicAdd per BLOCK
(8x fewer warps → near-deterministic).
kan_gate_backward, curiosity, DT linear — documented remaining
atomicAdd as acceptable (auxiliary components, not on primary DQN
gradient path).
Rust launchers updated: grid dimensions now match per-weight-element
thread counts for selectivity, mamba2, and denoise backward kernels.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OFICalculator + MicrostructureState fed per-tick. Bar close snapshots
and broadcasts 20 features via tokio::watch channel. Intermediate
snapshots available for speculative compute between bars.
Databento live client integration deferred (requires crate dep).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
speculative_forward() pre-computes from intermediate features.
check_speculative_cache() validates cache vs actual bar-close features
(5% relative L2 threshold). invalidate_speculative_cache() at bar
boundary. Full cuBLAS trunk integration deferred — cache infra ready.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Script rewritten to use argo submit --from=wftmpl/train with
train-epochs=0 — runs ensure-binary + ensure-fxcache only.
Local test fxcache regenerated with FXCACHE_VERSION=2, OFI_DIM=20.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Version constant in fxcache.rs. validate() rejects stale versions
with clear error message. ensure-fxcache Argo template now deletes
old cache and regenerates unconditionally. No manual PVC cleanup
needed — version mismatch triggers automatic regeneration.
v2: OFI_DIM=20 (20 microstructure features).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Flat->Positioned copies plan_params to ps[23-29]. Auto-exit on
profit_target (ISV*asymmetry), stop_loss (ISV-modulated), target_bars,
regime_stability<0.3. Scale schedule via scale_aggression. Conviction
sizing. Direction locked during active plan. Wired trainer->collector.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
