The IQN backward kernel now computes dL/d(h_s2) = dL/d(combined) ⊙ embed
and outputs it to d_h_s2_buf [B, hidden_dim]. Previously this was
explicitly NOT computed (comment: "trunk trained by C51").
Now the shared trunk receives BOTH gradient signals:
C51: dense cross-entropy gradient (can be noisy/steep)
IQN: bounded Huber quantile gradient (always stable)
The IQN gradient stabilizes trunk training when C51's gradient is steep.
With both signals, the trunk learns from C51's distributional knowledge
AND IQN's risk-aware quantile knowledge simultaneously.
New: d_h_s2_buf allocated in GpuIqnHead, zeroed before backward,
accumulated via atomicAdd across all quantiles per sample.
Accessors added: GpuDqnTrainer::bw_d_h_s2_buf(), shared_h2()
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
With the larger state space (72 dims: market + portfolio + multi-TF)
and complex reward structure, the old 1e-4 default produces steep
loss surfaces and doubling grad_norm per epoch. 3e-5 gives the
optimizer a gentler starting point. Hyperopt still searches [1e-5, 3e-4].
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Layer 3a: Fixed τ midpoints replace random sampling in IQN.
τ_i = (2i-1)/(2N) for i=1..N, pre-computed in constructor.
Eliminates sample_taus_kernel launch → fully deterministic →
CUDA Graph compatible. IQN becomes equivalent to QR-DQN.
Layer 3b: IQN per_sample_loss replaces C51 td_errors for PER.
GPU DtoD copy — zero CPU. IQN's Huber quantile loss is bounded
by construction (max = kappa²/2 per quantile). This eliminates
the PER feedback explosion that caused C51 loss to reach 242K.
No conditional flag — IQN is always primary when active.
Three-layer defense complete:
L1: Per-sample CE clamp at 50 (breaks PER loop)
L2: Label smoothing ε=0.01 on Bellman target (prevents log(0))
L3: IQN Huber loss for PER priorities (bounded by construction)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Layer 1: Per-sample CE clamped to MAX_PER_SAMPLE_CE=50 before IS weight.
Breaks PER feedback loop: high loss → high priority → high IS weight → repeat.
Layer 2: Bellman target smoothed with ε=0.01 uniform mix after projection.
Prevents any atom from having zero probability → no log(near-zero) in CE.
Root cause: C51 cross-entropy is unbounded when target and predicted
distributions are maximally misaligned. PER amplifies pathological samples.
These two layers cap the maximum possible CE and prevent the extreme
misalignment from occurring.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
grad_norm was growing 5x per epoch (523→2673→13K→67K→NaN). The old
clip at 10.0 allowed gradients to accumulate. With clip=1.0, the
Adam optimizer receives bounded updates.
Trial 2 (TPE-guided params) trains cleanly for 4 epochs:
train_loss: 4.5→3.3 (decreasing!)
Q-value: 12-19 (stable)
grad_norm: 162-567 (bounded)
Trial 1 (random initial params) still NaN's — this is expected and
handled by the hyperopt penalty (1M objective). The optimizer learns
to avoid unstable parameter regions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Gradient explosion from unbounded multi-timeframe features (240-bar
return could be ±50%). All multi-timeframe outputs now clamped:
- Return: ±10%
- Volatility: 0-10%
- Volume ratio: 0-5x
- Momentum: [0, 1] (already bounded)
NaN guards added:
- Kelly: NaN check on continuous/discrete Kelly before blend
- target_position: final NaN→0 after all scaling
- reward: final NaN→0 before writing to replay buffer
Local test shows Q-gap=18.7 (epoch 5) — 18x better than start-of-session 0.000.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add ExpertDemoGenerator using MA crossover strategy filtered by ADX trend
strength to produce (bar_index, exposure_action_index) pairs. Fast/slow EMA
crossover with ADX > threshold triggers Long100/Short100 signals, otherwise Flat.
Includes effective_ratio() for linear decay of expert_demo_ratio over training
epochs (controlled by expert_demo_ratio and expert_demo_decay_epochs fields
in DQNHyperparameters). Comprehensive unit tests for EMA, ADX, decay, and
edge cases included.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add difficulty_score field to WalkForwardWindow computed via compute_difficulty()
which calculates mean ADX(14) over training bars. Add DifficultyPhase enum (Easy/Mixed/Full)
and filter_by_difficulty() for curriculum-based window filtering — Easy phase trains
only on trending markets (ADX>30), Mixed duplicates trending windows for 2x weight,
Full uses all windows equally. The curriculum_phase field in DQNHyperparameters
(committed in previous ensemble commit) controls the active phase.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
16 new features computed directly in the state_gather CUDA kernel
from market data already on GPU. No CPU feature engineering.
Lookback windows: 5, 15, 60, 240 bars (≈5min, 15min, 1hr, 4hr)
Features per window:
- Return over N bars (directional bias)
- Volatility (high-low range / close)
- Volume trend (current / N-bar average)
- Momentum (position within N-bar range [0=bottom, 1=top])
State dim: 66 raw (72 aligned) without OFI, 74 raw (80 aligned) with.
The model now sees price action at 4 timescales simultaneously.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wire ensemble agent configuration into DQN hyperparameters:
- ensemble_count: number of agents in ensemble (default 1 = no ensemble)
- ensemble_diversity_weight: KL-divergence diversity loss weight (default 0.01)
Both fields default to backward-compatible values (single agent, no
diversity loss) and are plumbed through the conservative() constructor.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ROOT CAUSE 1: Stop-loss 0.3%→1%, take-profit 0.5%→2% (2:1 R:R).
Old 0.3% = 8.3 ticks on ES. Normal 1-min noise is 4-6 ticks.
99.88% of trades were stopped out by NOISE, not by bad entries.
ROOT CAUSE 2: Dense shaping 0.1x→0.01x. Over 50 bars, old dense
signal = 5.0 vs completion ±2.0 — dense dominated. Now dense = 0.5
vs completion ±2.0 — trade completion is the primary signal.
ROOT CAUSE 3: Action aliasing in 5-bar hold override. When model
chose Flat but was forced to Hold, replay stored (state, Flat, Hold's
reward) — corrupting Q-values for Flat. Now overwrites out_actions
with the ACTUAL held exposure action.
ROOT CAUSE 4: Hyperopt HFT activity weight 25%→5%. Old objective
penalized selective trading. MIN_VIABLE_TRADES 100→20.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
GPU experience collection is ALWAYS active in CUDA builds. The boolean
toggle was dead code — no CPU fallback exists. Removed from config,
hyperopt adapter, constructor, smoke tests, and training loop guard.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
IQN train_iqn_step_gpu() was doing a synchronous DtoH readback of the
loss scalar EVERY training step. This serializes the GPU pipeline.
Fix: return 0.0 placeholder from train step. Actual loss available via
read_loss() method — call only at epoch boundaries for logging.
Remaining readbacks (all once-per-epoch, acceptable):
- epoch_state: 32 bytes (DSR monitoring)
- q_stats: 20 bytes (Q-value diagnostics)
- gradient accum: dead code path (fused CUDA always active)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The compute_cvar_scales() was doing GPU→CPU→sort→CPU→GPU for quantile
CVaR computation. With batch_size=307 × 32 quantiles × 15 actions,
this was ~150KB of DtoH + sort + HtoD per epoch — likely the source
of the 10s/epoch overhead vs 3s baseline.
New inline CUDA kernel (iqn_cvar_kernel):
- One thread per sample (256 threads/block)
- Insertion sort of alpha_count smallest quantiles in registers
- Fast path for alpha_count=1 (just find minimum)
- Zero CPU readback, zero CPU allocation
Compiled once via OnceLock, cached for all subsequent calls.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Full Kelly implementation in CUDA kernel, no CPU path:
Enhanced Kelly = 0.7 × continuous (μ/σ²) + 0.3 × discrete ((bp-q)/b)
× confidence scaling (1 - 1/√n_trades)
× half-Kelly (0.5)
Clamped to [0.05, 0.25], normalized to position scale.
Tracks 6 statistics in portfolio state (PORTFOLIO_STRIDE 18→20):
ps[14] win_count, ps[15] loss_count, ps[16] sum_wins,
ps[17] sum_losses, ps[18] sum_returns, ps[19] sum_sq_returns
Removed CPU kelly_scale parameter — was a shortcut that violated
the zero-CPU-in-hot-path principle. All Kelly computation now
happens per-step in the env_step kernel from accumulated trade
statistics. Activates after ≥20 trade completions.
Also noted: IQN CVaR compute_cvar_scales() still does a CPU readback
for sorting quantiles. Should be a GPU kernel in next iteration.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fixed TP/SL produced bimodal reward (all trades hit exactly -0.3% or
+0.5%), making the reward distribution too deterministic. Model learned
"all actions produce the same bounded return" → Q-values converged.
Fix: stops scale with Q-gap conviction (0.5x to 2.0x of base levels):
- High conviction (Q-gap=2.0): SL=-0.6%, TP=+1.0% (let it run)
- Low conviction (Q-gap=0.5): SL=-0.15%, TP=+0.25% (quick exit)
- Time stop also scales: 50-200 bars based on conviction
This makes the reward distribution CONTINUOUS, not bimodal. High-conviction
entries with wider stops produce different returns than low-conviction entries
with tight stops → the model can learn that conviction matters.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
GpuBacktestEvaluator, GpuDqnTrainer, and hyperopt adapter had hardcoded
branch_0_size=5. Training produced 9-action indices (0-8) but backtest
only allocated 5-action buffers → SIGSEGV (exit 139) during walk-forward
evaluation after Trial 0 training.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three critical fixes from H100 Trial 0 analysis:
1. REMOVE IDLE PENALTY — was making Flat the worst Q-value (Q=-19.4),
forcing the model to trade on 95% of bars. Now Flat = reward 0.0.
DSR already penalizes inaction through the Sharpe denominator.
2. TRADE COMPLETION SCALING — ×1000 clamped to ±1 made ALL trades look
the same (can't distinguish 1-tick from 10-tick return). Now ×200
with clamp ±2, plus sqrt(hold_time) bonus for holding longer.
3. DYNAMIC TRADE MANAGEMENT — model learns ENTRY quality, exits are
managed by the system:
- Stop loss: -0.3% × vol_mult (CUSUM-adaptive, up to -0.75%)
- Take profit: +0.5% × vol_mult × trend_mult (up to +2.5% in trends)
- Time stop: 100 bars × vol_mult (up to 250 in volatile markets)
- Min hold: 5 bars before model can exit voluntarily
- Auto-exit forces flat and computes trade_completion_reward
Regime-adaptive via ADX (trend strength) and CUSUM (volatility)
features already in the GPU memory.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CRITICAL: The DQN constructor had hardcoded state_dim=45/53 (old PORTFOLIO_DIM=3).
With PORTFOLIO_DIM=8, raw state_dim is 50 (no OFI) or 58 (with OFI).
The network was sized for 56 dimensions but OFI features (8 dims) were uploaded
and ignored because state_dim didn't include them.
Also fixed:
- constructor.rs: num_actions 5→9
- All GPU trainer/head defaults: state_dim 48→56
- metrics.rs: state_dim calculation
- mod.rs test: state_dim assertions
- Argo template reapplied for stale cache clearing
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CRITICAL: gpu_monitoring.rs had [usize; 5] for action_counts but kernel
writes 9 exposure levels → memory corruption. Fixed array, buffer (12→16),
readback size, and the CUDA monitoring_reduce kernel (s_counts[5]→[9],
summary offsets updated). Also fixed remaining 45→81 references in
metrics.rs, training_loop.rs, monitoring.rs, config.rs comments.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Complete dual distributional pipeline:
1. After training: fused_ctx.compute_cvar_device_ptr() runs IQN forward
2. CVaR at α=5% computed per sample → [0.25, 1.0] position scaling
3. Device pointer set on collector via set_cvar_scales()
4. Next epoch: env_step kernel applies target_position *= cvar_scales[i]
C51 picks direction (argmax expected Q), IQN sizes risk (CVaR tail).
High tail risk → smaller position. Safe distribution → full size.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- experience_env_step kernel: new cvar_scales parameter (NULL = no scaling)
- target_position *= cvar_scales[i] when buffer is non-NULL
- GpuExperienceCollector: cvar_scales_ptr field + set_cvar_scales() setter
- Default: NULL (0) = no scaling until IQN CVaR is wired from training loop
The GpuIqnHead.compute_cvar_scales() produces the buffer, the collector
passes it to the kernel. Full wiring through training_loop.rs is the
remaining step — needs the collector to receive the device pointer from
the fused training context after each IQN training step.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New method on GpuIqnHead: samples τ values, runs forward-only kernel,
computes CVaR at α=5% per sample, returns [0.25, 1.0] scaling factors.
CVaR ≥ 0 (safe) → full position. CVaR < 0 (risky) → reduced position.
Next: wire into env_step kernel for position scaling during experience
collection, and into fused_training.rs to call after IQN training step.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
use_qr_dqn was a hacky toggle that gated the IQN dual-head behind
a num_atoms threshold. IQN is now always enabled when iqn_lambda > 0
(default 0.25). C51 remains the main loss; IQN is the auxiliary head
for CVaR risk quantification.
Removed use_qr_dqn from: DQNParams, DQNHyperparameters, fused_training,
hyperopt adapter, all tests, all examples.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Updated all fixed-size arrays across monitoring.rs, financials.rs,
metrics.rs, training_loop.rs from [_;5]→[_;9] and [_;45]→[_;81].
DIRECTION_LUT expanded to 9 levels (-1.0 to +1.0 in 0.25 steps).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
RTH/ETH cost differential, market impact (done), book depth fill quality,
macro events, weekend gap risk, margin utilization feature.
All configurable via TOML. We trade against real markets — simulation must match.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1-lot fills at bid/ask. 4-lot moves the market.
impact_scale = 1 + (|delta|/max_position)^2, ranges [1.0, 2.0].
Teaches the model that max-size positions are 2x more expensive to enter.
Makes the 9-exposure granularity meaningful — 25% positions are cheaper
to enter/exit than 100% positions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PDT $25K rule protection: if equity drops below 75% of peak_equity,
the episode terminates (done=1) with a -1.0 penalty reward and forced
flat. The model learns that approaching the floor = game over.
Uses peak_equity (not initial_capital) so it adapts as account grows.
With $35K initial, floor triggers at ~$26.25K — above the $25K minimum.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The model thinks in TRADES, not in bars. Per-bar 1-minute noise is like
flipping a coin — the real signal is the full trade return (entry→exit).
Three reward levels:
1. TRADE EXIT (sparse, strong): full trade return entry→exit, the PRIMARY
learning signal. 10x stronger than dense shaping.
2. IN TRADE (dense, weak): 0.1x DSR + PnL shaping per bar. Just enough
gradient for direction, doesn't overwhelm the completion signal.
3. FLAT (near-zero): doing nothing is almost free.
New portfolio state fields (PORTFOLIO_STRIDE 12→14):
- ps[12] = entry_price (price at trade entry)
- ps[13] = trade_start_pnl (cumulative PnL snapshot at trade entry)
Trade lifecycle detection: entering_trade / exiting_trade / in_trade
from position transitions (was_flat→positioned / positioned→is_flat).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
C51 for action selection (expected Q), IQN for risk quantification (CVaR).
Disagreement between C51 and IQN expected Q = uncertainty signal.
Architecture already 90% built — IQN head exists but output unused.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
With num_atoms=101 (just above the old threshold of 100), QR-DQN activated
alongside C51, doubling distributional computation. H100 epochs went from
~2s to ~35s. Raised threshold to 200 so num_atoms=101 uses C51 only.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- DSR warm-up: skip first 50 steps when EMA has insufficient history
- Phase Fast num_atoms: 51 → 101 (H100 can afford finer resolution,
1.19 per atom vs 2.35 — critical for distinguishing Q-values)
- Argo template: clear stale feature cache before hyperopt (ensures
fresh computation with VPIN/trades enrichment)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bug hunt findings:
- epsilon_greedy_kernel.cu: all 3 kernels (select, routed, branching) lacked
the Q-gap conviction filter. Training learned with q_gap=0.1 but inference
paths bypassed it entirely. Now all action selection paths are consistent.
- c51_loss_kernel.cu: Bellman projection boundary fix — b_pos clamped away
from exact NUM_ATOMS-1 to prevent phantom upper==lower atom collapse.
- experience_kernels.cu: NaN/Inf guards on DSR and normalized PnL outputs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ROOT CAUSE: 5 interlocking bugs made learning impossible:
1. DSR denominator floor 1e-12 produced values in millions → drowned all signal
2. Global [-1,+1] clamp destroyed Bellman equation signal (can't distinguish
catastrophic loss from mild loss)
3. v_range=20 exactly equals V_max for gamma=0.95 → Bellman target pins at
ceiling → Q-values saturate → Q-gap collapses to 0.0000
4. num_atoms=11 over 40-unit range = 4.0 per atom (C51 paper min is 51)
5. 6/7 reward components were penalties → mean_reward=-0.311 regardless of action
FIXES:
- DSR denominator floor: 1e-12 → 0.01 (prevents million-scale spikes)
- Each component individually clamped BEFORE weighting (DSR to [-1,+1],
z-score to [-3,+3], drawdown to [0,1], time decay to [0,0.3])
- Removed global [-1,+1] clamp (no longer needed with bounded components)
- profit_take_bonus: 0.1 → 0.01 (was 100x too large, caused reward hacking)
- Removed confidence scaling (positive feedback loop destabilized learning)
- Removed regime scaling (non-stationary reward confused the model)
- Dynamic v_range from gamma: v_range = 2.5/(1-gamma)*1.2 (always covers Q range)
- num_atoms minimum: 11 → 51 (C51 paper standard)
- gamma default: 0.99 → 0.95
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Q-gap was 0.0 (disabled) meaning the model traded on every bar regardless
of conviction. With 0.1, the model must have Q(best) - Q(flat) > 0.1
before entering a position. Local test showed 21% fewer trades.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The DBN feature cache was keyed ONLY on OHLCV .dbn files. If a cache was
created without trades data (VPIN/Kyle's Lambda), subsequent runs with trades
would silently serve stale features from cache, dropping VPIN enrichment.
Now cache key hashes OHLCV + MBP-10 + trades dirs together. Adding or removing
data sources invalidates the cache correctly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
