Atom warm-start: bitonic sort rewards → quantile positions → write to
atom_positions_buf as initialization. Existing SGD optimizer refines.
Atoms start where reward mass actually is instead of uniform [-50,+50].
Robust PopArt: median/IQR normalization from sorted rewards replaces
Welford mean/var. More robust for bimodal distribution (many ±0.1
micro-rewards + few ±5.0 trade exits). Conditional: popart_robust=true.
Both reuse the same bitonic sort (~14ms per epoch, amortized).
gather_quantiles kernel extracts positions. extract_median_iqr reads
Q25/median/Q75 from sorted buffer.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Volatile regime → price confirmation weighted higher (momentum matters)
Stable regime → book aggression + hold quality weighted higher
Regime transition → hold quality drops (don't hold through shifts)
Uses ISV signals[11] (regime_stability) and signals[2] (volatility)
already available in env_step_batch. Zero new parameters — the temporal
attention pipeline drives reward composition through ISV routing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Counterfactual: Hold(1)/Flat(3) direction mirror maps to self — trivial.
Now falls through to magnitude CF for Hold/Flat instead of wasting a
replay buffer slot on same-action same-reward experiences.
reward_noise_scale: 0.05→0.0 (dense micro-rewards are already noisy,
adding 5% label noise destroys the per-bar signal).
Added micro-reward weight config fields (price_confirm_weight=0.5,
book_aggression_weight=0.3, hold_quality_weight=0.2, micro_reward_temp=3.0,
holding_cost_rate=0.0001) — defined in TOML, kernel plumbing next session.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removed the legacy non-branching Sequential q_network and target_network
from DQNAgent. These were never used (branching+dueling always active)
but allocated VRAM and ran noise resets every step. -190 lines.
Config tuning for dense micro-reward system:
- n_steps: 5→1 (TD(0), micro-rewards cancel over n>1)
- tau: 0.007→0.01 (faster target tracking for TD(0))
- c51_alpha_max: 0.5→1.0 (full C51, PopArt handles normalization)
- curiosity_weight: 0.1→0.0 (dense micro-reward replaces curiosity)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
FXCACHE_VERSION 3→4. Three precomputed features added to OFI region:
- ofi[8..16): temporal deltas (ofi[bar] - ofi[bar-1]) for 8 features
- ofi[16]: book aggression (MBP-10 10-level center-of-mass asymmetry)
- ofi[17]: log bar duration (imbalance bar formation time, normalized)
Previously 10 of 18 OFI embed MLP inputs were zero. Now all 18 have
real data: raw_ofi(8) + delta_ofi(8) + book_aggression(1) + log_duration(1).
Added read_state_sample() diagnostic for GPU state verification.
Legacy v3 fxcache handled by zeroing new slots (v2→v4 graceful upgrade).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three signal-killing issues fixed:
1. Rank normalization DISABLED — was double-normalizing with PopArt,
destroying magnitude difference between micro-rewards (±0.1) and
trade exits (±5.0). PopArt alone preserves relative magnitude.
2. n_steps 5→1 (TD(0)) — dense micro-rewards alternate ±0.1 each bar.
With n=5, they cancel out over 5 bars. TD(0) preserves the per-bar
signal that the temporal pipeline needs to learn from.
3. per_alpha 0.6→0.3 — lower = more uniform PER sampling. Dense micro-
rewards have tiny TD-errors (easy to predict), so high alpha ignores
them. Lower alpha ensures micro-reward experiences get sampled.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Linear scaling rule (2x batch → 2x LR) doesn't hold for DQN+PER with
major architectural changes (Hold action, OFI embed, wider attention).
The model needs stability to learn the new action space, not speed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Attention W_O was Xavier-initialized, injecting ~sqrt(D) magnitude noise
into the residual connection. With the DtoD→copy_f32 fix making attention
active (was a no-op before), random W_O corrupted all branch head inputs
→ MaxDD=100% on validation.
Fix: W_O zero-init → attention starts as identity (output ≈ h_s2).
W_Q/K/V stay Xavier (they project to SDP space, not the residual).
OFI embed weights also zero-init → Mamba2 history starts as [h_s2; 0]
and attention input starts as [h_s2; 0]. No noise injection at init.
Both learn from zero as gradients shape them toward useful patterns.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Hardcoded 12 Q-values in pinned readback buffer and per-branch Q-gap
slice caused panic with b0=4 (Hold action: 4+3+3+3=13 total actions).
Now uses dynamic total_actions from config.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Widens W_Q/W_K/W_V from [D,D] to [D,D+10]. OFI embed (10-dim) is
concatenated to attention input via attn_concat_input kernel. Cross-
feature attention now sees order flow dynamics alongside trunk features.
Backward: d_input_scratch widened to [D+10,B]. extract_attn_ofi_grad
kernel extracts the OFI gradient portion. d_ofi_embed_attn buffer
exposed via accessor for Task 8 gradient accumulation.
W_O stays [D,D], LayerNorm stays D-wide — only input projection is
affected. SDP operates on Q/K/V which remain [D,B].
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2 new cuBLAS GEMMs: d_h_history = W_A^T @ d_gate + W_B^T @ d_x.
Extracts last 10 dims as d_ofi_embed_mamba2, accumulated across K=8
timesteps. This enables the OFI embed MLP to learn from temporal
patterns discovered by the Mamba2 SSM.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
h_history widened from [B, K, SH2] to [B, K, SH2+10]. OFI embed (10-dim)
appended per timestep. W_A/W_B projections grow to [SH2+10, STATE_D].
W_C stays [SH2, STATE_D] — operates on scan output, not history input.
The SSM temporal scan now sees order flow momentum, book aggression, and
bar duration alongside trunk features. Enables learning sequential
patterns in microstructure dynamics.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extracts [raw_ofi(8); delta_ofi(8); book_aggression(1); log_duration(1)]
= 18-dim from state vector, compresses to 10-dim via cuBLAS SGEMM +
bias+ReLU. Xavier init. Runs in training forward path before Mamba2
and attention, making the embedding available for temporal enrichment.
190 trainable params (18×10 + 10 bias). ~0.05ms per step.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Action space 81→108: direction(4) × magnitude(3) × order(3) × urgency(3).
Short(0), Hold(1), Long(2), Flat(3). Hold keeps current position with
zero transaction cost, giving the model a "do nothing" option.
Changes: trade_physics.cuh (Hold returns current_position), env_step
(Hold skips trade execution), action.rs (ExposureLevel::Hold variant),
config (branch_0_size 3→4), all match arms updated across 11 files.
Counterfactual mirror: Short↔Long, Hold↔Hold, Flat↔Flat.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
memcpy_dtod_async is NOT captured by CUDA Graph — silently skipped
during replay. This caused:
- Attention reading stale trunk data (h_s2→scratch copy skipped)
- Attention output never written back (scratch→h_s2 copy skipped)
- Backward gradients using stale logits (d_logits staging skipped)
- Vaccine comparison using stale reference (prev_grad snapshot skipped)
Attention has been effectively a NO-OP since CUDA Graph was introduced.
Fix: new copy_f32 kernel in graph_utility_kernels.cu (same pattern as
existing mamba2_copy_enriched). 11 call sites replaced across
gpu_dqn_trainer.rs and fused_training.rs. graph_safe_copy_f32() and
graph_safe_copy_f32_on() helpers for single-stream and multi-stream.
Non-graph-captured DtoD copies (checkpoint, init, eval) left as-is.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
100 bars = ~1 hour of imbalance bars. ES futures regimes persist 4-8
hours. Val_Sharpe was artificially inflated by regime autocorrelation
bleeding through the too-short purge gap. 2000 bars = ~1 full trading
day, ensuring validation data is from a different regime.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
3 pre-existing bugs fixed:
- ofi_gpu uploaded but never passed to env_step — now wired via
full_batch_states param (OFI at state[66..74) accessible in kernel)
- concat_ofi_features read indices 42/45 (portfolio features) instead
of 66/69 (actual OFI location in 96-dim state vector)
- PORTFOLIO_STRIDE expanded 30→38 for prev-OFI delta storage (ps[30..37])
Added PREV_CLOSE_SLOT=6 and PREV_MID_SLOT=22 for MFT mark-to-market.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
target_dim expansion: adds raw_open (OHLCV) and mid_price_open
(MBP-10 midpoint at bar formation) to fxcache targets. FXCACHE_VERSION
2→3 for auto-rebuild. Legacy v2 files handled with close-price fallback.
Spec v5 adds 3 pearls:
- Bar duration encoding in Mamba2 (continuous-time SSM awareness)
- Order book center of mass from all 10 MBP-10 levels (aggression signal)
- Retrospective hold quality bonus (teaches exit timing)
Plus: Hold action (4th direction), DSR Sharpe EMA fix, counterfactual
magnitude/order sign fix, MFT mid-price mark-to-market.
OFI embed MLP now 18→10 (was 16→8). Mamba2 width SH2+10 (was SH2+8).
Attention width D+10 (was D+8).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
target_dim 4→6: adds raw_open (OHLCV) and mid_price_open (MBP-10
best bid/ask midpoint at bar formation) to the targets buffer.
Micro-reward now uses real intra-bar move (close - open) / atr
instead of close-to-close approximation. Also adds spread cost
awareness via |open - mid| penalty. Requires fxcache recomputation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Major changes from v1:
- Fix OFI data pipeline (ofi_gpu never wired, indices 42→66)
- Expand PORTFOLIO_STRIDE 30→38 for prev-OFI storage
- Add Mamba2 d_h_history backward (2 new cuBLAS GEMMs)
- Expose attention d_input_scratch for gradient flow
- Pre-compute OFI deltas in experience collection (state[74..82])
- Lower rank normalization threshold to 1e-5 for micro-rewards
- Use close-to-close return instead of unavailable open_price
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dense per-bar reward using order flow momentum × price confirmation.
OFI embedding MLP (16→8 via cuBLAS) feeds into Mamba2 history AND
attention input. Replaces sparse exit-only reward with continuous
temporal signal for the SSM and attention heads to learn from.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PopArt: normalize_rewards_popart_inplace was implemented but never
called from run_full_step. Without it, rank-normalized rewards [-1,+1]
spread across C51 atom range [-50,+50] — only 2% of atoms used,
near-zero C51 gradient. Now called before forward pass (Phase 0a).
Counterfactual: when do_flip=true AND cf_cycle==0 (directional mirror),
cf_reward = -reward double-negated (reward already flipped at line 1895).
Fix: cf_reward = do_flip ? reward : -reward. Affects ~1/6 of
counterfactual experiences that were teaching wrong directional signal.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The plan head's conviction output (sigmoid, [0,1]) scales rewards.
At init, conviction ≈ 0.1 (Xavier random weights through sigmoid),
which multiplied ALL rewards by 0.1 — killing the gradient signal.
The model couldn't learn meaningful Q-values, defaulted to uniform
random action selection → 1.7M trades on 4M bars → val_Sharpe -1000.
Fix: only apply conviction scaling when readiness ≥ 0.5 (plan head
mature). Before that, conviction defaults to 1.0 (identity). This
matches the existing readiness gate on position sizing (line 1416).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
populate_q_out + q_stats_reduce launch async kernels that write to
pinned device-mapped memory. The CPU was reading immediately — before
the GPU finished writing. Added cuStreamSynchronize to ensure the
kernel results are visible before the pinned memory read.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
atomicAdd inside CUDA graph replay causes non-determinism and is
unnecessary. The only correct path: reduce_current_q_stats() calls
populate_q_out() outside the graph to compute atom_stats cleanly.
Graph-captured path stays NULL for both atom_stats and q_var.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The graph-captured forward pass never calls compute_expected_q (it
works directly on logits for C51 loss). So atom_stats_buf was never
written during training. Now reduce_current_q_stats() calls
populate_q_out() first — runs compute_expected_q outside the graph
to populate atom_stats before q_stats_reduce reads them.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The first fix only patched populate_q_out(). The graph-captured path
uses replay_forward_ungraphed() which had its own null_atom_stats=0.
This is the call site that actually runs during training.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
compute_expected_q was called with null_atom_stats=0 (NULL pointer),
so the kernel skipped atom entropy/utilization accumulation entirely.
atom_stats_buf existed but was never passed. This disabled:
- G4 adaptive gamma annealing (uses atom_utilization)
- Homeostatic regularizer atom_util observable
- ISV atom utilization signal
Now passes atom_stats_buf.raw_ptr() with a cuMemsetD32 zero before
each call (kernel uses atomicAdd). Also passes q_var_buf_trainer for
per-action Q-variance computation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Linear scaling rule: 2x batch → 2x LR to maintain effective update
magnitude. Tau increased to compensate for fewer steps/epoch (target
network tracks faster). H100 VRAM: 41GB free at B=8192, B=16384 adds
~4GB — easily fits.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All gradient buffer clears now use cuMemsetD32Async (u32-wide writes)
instead of cuMemsetD8Async (byte-wide). 4x memory bandwidth utilization
for the ~33 memset calls per training step. Size params converted from
bytes to f32 element count (.num_bytes() → .len()).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mamba2_temporal_scan (forward) and mamba2_scan_backward are no longer
called — replaced by cuBLAS projection GEMMs + lightweight scan kernels.
Removes ~200 lines of dead CUDA code and 2 unused CUfunction handles.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Default was hardcoded as 64 in train_baseline_rl.rs and evaluate_baseline.rs,
overriding the config default of 32. Added num_quantiles=32 to production
config so it's explicit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
