Add 7 new fields to DQNHyperparameters (fill_ioc_fill_prob,
fill_limit_fill_min, fill_limit_fill_max, fill_spread_cost_frac,
fill_spread_capture_frac, q_clip_min, q_clip_max) and wire them
through training_profile.rs into ExperienceCollectorConfig construction
in training_loop.rs. Previously these 7 values were hardcoded at the
construction site; now they flow from TOML [experience.fill_simulation]
and [risk] sections. Default values match the prior hardcoded constants
so existing behavior is unchanged.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove use_double_dqn, use_dueling, use_per, use_branching,
use_distributional, use_noisy_nets, use_huber_loss, and use_cql
from DQNConfig, DQNHyperparameters, and DqnParams structs.
These features are always enabled (Rainbow DQN standard). The boolean
flags were dead code — every constructor set them to true, and the
only code paths that set them to false were in tests that disabled
features for simplicity. With the fields removed, the features are
unconditionally active, eliminating ~490 lines of dead configuration.
Key changes:
- Struct field declarations removed from 3 core config structs
- Conditional branches (if use_X { ... } else { ... }) simplified:
dueling/branching/PER network creation is now unconditional
- Checkpoint metadata hardcodes "true" for backward compatibility
- Hyperopt search space index 11 (use_branching) fixed at 1.0
- TOML/YAML config files cleaned of removed fields
- Tests that toggled these flags updated or rewritten
45 files changed, -487 net lines. Zero new test failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Five bugs in the GPU backtest metrics kernel produced impossible results
(17,975% return, 100% MaxDD on every trial):
1. Additive return accumulation (local_cum += r) replaced with
multiplicative compounding (local_cum *= 1+r, init 1.0)
2. Absolute drawdown (peak - current) replaced with fractional
drawdown ((peak - current) / peak)
3. Strided bar processing (thread sees every Nth bar) replaced with
consecutive chunked processing for correct drawdown tracking
4. Per-bar win counting replaced with per-trade win/loss tracking
5. Total return now computed via multiplicative equity product
reduction across threads (s_sorted[0] - 1.0)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Five reward computation fixes in experience_env_step CUDA kernel:
1. Replace CUSUM vol proxy with ATR(14): CUSUM at feature[41] is a binary
direction indicator [-1,1,0], NOT volatility. When CUSUM≈0, vol_proxy
became 0.0001 causing 10000x reward amplification. ATR(14) at feature[9]
is actual realized volatility — reverse the safe_normalize encoding
(ln(atr)+7)/16 to recover atr_pct = exp(norm*16-7) / price.
2. Move loss aversion BEFORE squash: previously applied after hard clamp,
creating asymmetric [-15, +10] range making expected reward negative
even for fair strategies. Now applied pre-squash for smooth asymmetry.
3. Replace hard clamp with tanh soft squash: fmaxf(-10, fminf(10, reward))
destroyed tail information (1% and 5% wins both → 10.0). tanh preserves
that larger wins produce proportionally larger rewards.
4. Remove turnover penalty: the 0.05*|delta|/max_position penalty double-
counted transaction costs already deducted from cash via Almgren-Chriss
impact model at line ~679, over-penalizing necessary rebalancing.
5. Clarify CUSUM spread_scale usage: CUSUM at feature[41] is correctly used
as market-stress proxy for spread widening in tx cost computation — this
is distinct from the (now-fixed) vol proxy for reward normalization.
Also: annotate min_hold_bars=5 as hyperopt candidate.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The DBN loader was producing 2-element targets [close, next_close] which got
zero-padded to 4 elements. Now produces full 4-element layout matching the
kernel's expected format: [0:1]=network input, [2:3]=raw prices for portfolio sim.
Kernel reads tgt[2:3] for raw_close/raw_next (restored to original design).
All other kernels (DT, PPO, expert demos) also read tgt[2:3] correctly.
Stale feature cache invalidated by this data format change.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ROOT CAUSE of 0% win rate: The experience kernel read raw_close from tgt[2] and
raw_next from tgt[3], but the DBN data loader produces 2-element target vectors
[current_close, next_close] which get zero-padded to 4 elements. So tgt[2:3]=0,
triggering the degenerate price guard (raw_close=1.0, raw_next=1.0), making
every trade's P&L exactly zero → classified as loss.
Fix: read tgt[0] and tgt[1] which contain the actual raw prices.
Result: wins=264/594 (44.4% win rate), PF improving 0.44→0.88→0.98 over 3 epochs.
Also: segment-based trade detection correctly counts reversals,
old_pos_pnl uses saved pre-update position for correct P&L computation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Trade detection now counts reversals (S100→L50) as completed segments
- old_pos_pnl saved before position update for correct reversal P&L
- realized_pnl writeback uses old position PnL on reversal bars
- 0% win rate persists — needs deeper investigation (likely tx cost interaction)
WIP: The trade_return formula produces correct sign for raw market moves,
but every trade still shows as a loss. Suspect tx costs on both entry AND
exit of each reversal segment exceed the 1-bar price movement.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Task 9: Document ensemble consensus limitation — ensemble heads live on
FusedTrainingCtx (training-only), not accessible at inference time.
The ensemble already provides value via diversity gradient during training.
Task 10: Add three-phase training pipeline to the epoch loop:
- Phase 1 (Behavioral Cloning): C51 warmup + expert demos (already existed)
- Phase 2 (Full-Stack RL): all features, expert decay (already existed)
- Phase 3 (Refinement, last 20%): force expert_ratio=0, shrink-and-perturb
at phase boundary for plasticity consolidation
Task 11: Expand hyperopt search space from 41D to 45D with 4 new dims:
- her_ratio [0.0, 0.5]: HER relabeling ratio (was hardcoded 0.0)
- curiosity_weight [0.0, 0.2]: intrinsic reward weight (was fixed 0.0)
- use_cvar_action_selection [0.0, 1.0]: risk-aware IQN action scoring
- cvar_alpha [0.01, 0.2]: CVaR confidence level
All wired through build_hyperparams to DQNHyperparameters.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Task 7: Wire ensemble diversity gradient through value head into shared trunk.
The KL gradient kernel was computing d_logits but gradient flow stopped there.
Now: d_logits → cuBLAS backward W_v2 → ReLU mask → backward W_v1 → d_h_s2 →
trunk backward (layers 2,1) → SAXPY(diversity_weight) into grad_buf.
Uses launch_dx_only for value head layers (skip dW/db — only d_h_s2 needed).
graph_adam sees combined C51 + IQN + ensemble diversity in single update.
Task 8: Decision Transformer Phase 1 already fully implemented. DT runs before
main training loop when dt_pretrain_epochs > 0: builds trajectories from GPU
data, runs pretrain_step per epoch/batch, logs loss. Verified and confirmed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace reward v4 (pure mark-to-market return) with reward v5, a two-component
trade-aware hybrid that separates dense per-bar signal from sparse trade-exit signal:
Dense (every bar, weight 0.1): raw_pnl / equity when in a trade, zero when flat.
Keeps gradients flowing without overwhelming the sparse trade completion signal.
Sparse (at trade exit, weight 2.0): trade_return * patience_multiplier where
patience = sqrt(hold_time / expected_hold). Regime-adaptive expected hold via
ADX: trending (ADX>30) = 20 bars, ranging (ADX<20) = 8 bars, default = 12 bars.
Dynamic trailing stop: regime-adaptive trail distance (0.5% base, widens with
volatility via CUSUM and trend via ADX). Activates when trade is profitable and
held > 2 bars. Locks in profits by forcing exit when unrealized P&L drops below
the trailing floor.
Also fixes kernel signature mismatch: removes 7 old reward v2 parameters
(w_dsr, w_pnl, w_dd, w_idle, dd_threshold, time_decay_rate, eta) that were
already removed from the Rust launcher in a prior commit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaces the 8-component dense shaping + sparse trade completion with:
reward_t = (equity_t - equity_{t-1}) / equity_{t-1}
- Losing bars get NEGATIVE reward (every bar, not just exit)
- Flat bars get ZERO reward
- Trade entry: tx_cost hits cash → immediate negative reward
- Loss aversion: losses weighted 1.5x (prospect theory)
- No more DSR, no dense shaping, no hold penalty
NOTE: v_range needs recalibration for percentage returns (~0.001/bar)
instead of the old reward scale (~0.01-2.0). Current v_range=42 is
1000x too wide for the new reward magnitude.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. Risk management (CVaR, conviction, Kelly) re-enabled as ENVIRONMENT PHYSICS:
- Agent observes scaling via portfolio state features
- Learns to account for risk limits in its policy
- No longer destroys credit assignment (scaling is physics, not action override)
- Kelly uses half-Kelly (0.5x) for safety, activates after 20 trades
2. Backtest tx_cost now uses training's transaction_cost_multiplier from hyperopt
(was hardcoded 0.1 bps — 17x lower than training). Training and eval see same costs.
3. Backtest env tx_cost formula expanded to match training:
- Square-root market impact (Almgren-Chriss)
- Order-type premiums (Market=0, IoC=+2bps, LimitMaker=-5bps)
Result: first POSITIVE Sharpe (+0.0838) in project history. 134K trades.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. Monitoring kernel: prepend common_device_functions.cuh for DQN_ORDER_ACTIONS
2. Backtest metrics kernel: prepend common_device_functions.cuh
3. Backtest gather kernel: prepend common_device_functions.cuh
4. Backtest tx_cost: use training's transaction_cost_multiplier from hyperopt
(was hardcoded 0.1 bps — 17x lower than training's ~1.7 bps multiplier)
All standalone kernels now consistently include common_device_functions.cuh
for DQN action space defines. No more hardcoded constants.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. IQN gradient sequencing: train_step_gpu replays forward ONLY, caller injects
IQN/attention/ensemble gradients into grad_buf, then calls replay_adam_and_readback().
Single Adam sees combined C51+IQN gradient. Previously IQN was a NO-OP (SAXPY
happened after both graphs completed — Adam already consumed gradients).
2. Gradient clip 1.0 → 10.0: C51 with 101 atoms × 3 branches produces 300x larger
gradients than standard DQN. Clip at 1.0 made effective LR ~7e-12. Result:
grad_norm 137K → 490 (280x reduction, network actually learns now).
3. max_abs_reward 3.0 → 1.5: tighter C51 support [-42, +42] instead of [-84, +84].
Q-values at 24 (58% of v_max) instead of 81 (96%). 2x atom resolution.
4. choppy_bonus removed: Flat reward was 0.02 on 60-70% of bars, dominating
normalized reward distribution. Now Flat gets exactly 0.0.
5. Reward normalization: Welford EMA was broken (alpha=0.01 over 150K samples →
variance converges to zero → divides by 1e-8 → Q-value explosion). Fixed with
batch-level mean/std + EMA blending + variance floor 0.01.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Split CUDA Graph (forward + adam phases with gradient injection point):
- IQN trunk gradient flows through single Adam (no dual optimizer conflict)
- Spectral norm runs BEFORE forward (not after Adam — no tug-of-war)
- σ_max in 40D hyperopt search space [1.0, 10.0]
Attention Phase B backward:
- Full gradient flow through 4-head self-attention weights
- Separate Adam optimizer for attention params
- Backward kernel recomputes forward from saved_input (memory-efficient)
Ensemble multi-head:
- Real cuBLAS value head forward per ensemble head (was copying head 0 logits)
- KL diversity gradient kernel with hierarchical reduction
- forward_value_head() on CublasForward for per-head SGEMM
Regime PER scaling:
- Kernel reads target ADX/CUSUM from states_buf directly (zero CPU readback)
- Removed 2x memcpy_dtoh per training step
Decision Transformer:
- 14 CUDA kernels (embed, causal attention, FFN, CE loss + backward + trajectory building)
- GPU-native trajectory builder (return-to-go reverse cumsum, momentum expert actions)
- Wired into training loop with dt_pretrain_epochs config
HER Future/Final:
- episode_ids flow through PER buffer (GpuBatch, GpuReplayBuffer, GpuBatchSlices)
- GPU-native donor sampling (binary search on episode boundaries)
- Strategy dispatch in fused_training.rs
Backtest SEGV fix:
- Missing q_gaps_buf argument in action_select kernel launch
- Dynamic branch_sizes from agent (not hardcoded)
Local test: objective=10.48, Sharpe=0.0419, 175K trades, zero errors
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add 5 DT fields to DQNHyperparameters (dt_pretrain_epochs, dt_context_len,
dt_embed_dim, dt_num_layers, dt_target_return) with conservative() defaults
(dt_pretrain_epochs=0 = disabled)
- Wire pre-training phase before the main DQN epoch loop: logs config when
dt_pretrain_epochs > 0, noting that pretrain_step() kernels are ready in
decision_transformer.rs and full integration awaits trajectory data pipeline
- Add dt_pretrain_epochs to DQNParams struct, Default impl, from_continuous,
and the hyperparams struct literal in the hyperopt adapter (fixed to 0 for
all hyperopt trials, not in search space)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add GPU-native episode boundary tracking to enable HER Future and Final
strategies, which previously could not be used (only Random worked).
- Create her_episode_kernel.cu with 3 kernels:
- fill_episode_ids_kernel: fills episode_ids[i] = i/L, zero CPU
- her_sample_future_donors: samples random donor LATER in same episode via LCG
- her_find_episode_end: finds last transition index of same episode
- Add episode_ids CudaSlice<i32> field to GpuExperienceBatch; filled by
fill_episode_ids_gpu() on every collect_experiences_gpu() call
- Wire fill_episode_ids_kernel compilation into GpuExperienceCollector::new()
- Add future_donors_func, episode_end_func, and rng_states fields to GpuHer
- Add relabel_batch_with_strategy() dispatching to GPU episode kernels based
on HerGpuStrategy (Future or Final); errors on Random (use existing path)
- All episode ID computation uses direct integer arithmetic (i/L), no scan
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds K independent value/advantage head weight sets sharing a common DQN
trunk. Provides uncertainty estimation (Q-value variance across heads) and
diversity regularization (KL divergence between head distributions).
Architecture:
- Head 0 stays inside CUDA Graph (zero overhead for ensemble_count=1 default)
- Heads 1..K-1 run outside CUDA Graph using post-graph save_h_s2 activations
- DtoD clone of head weights at init with stream-sync; diversity grows over training
- Pairwise KL uses symmetrized Jensen–Shannon divergence for numerical stability
New files:
- ensemble_kernels.cu: two NVRTC kernels — ensemble_aggregate_kernel (mean/var
Q-values across K heads) and ensemble_diversity_kernel (hierarchical warp→block
reduction matching dqn_grad_norm_kernel pattern, no flat atomicAdd)
- compile_ensemble_kernels() function in gpu_dqn_trainer.rs
- New GpuDqnTrainer accessors: on_v_logits_buf(), tg_h_v_scratch_ptr()
FusedTrainingCtx changes:
- ensemble_extra_heads: Vec<(DuelingWeightSet, BranchingWeightSet)>
- Pre-allocated GPU buffers (logits, mean_q, var_q, diversity_loss)
- run_ensemble_step() method runs after CUDA Graph replay (EventTrackingGuard)
- Wired in run_full_step() between IQN PER step and spectral norm step
Config: ensemble_count=1 (default, zero overhead), ensemble_diversity_weight=0.01
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add backward_residual() method to GpuAttention that implements
gradient flow through the residual connection for frozen attention
weights. This is Phase A of the attention backward implementation.
The residual connection (output = input + attention(input)) ensures
that gradients flow through unchanged: d_input = d_output. With frozen
weights, we don't compute attention weight gradients, making this a
literal no-op when the input and output buffers alias (as they do in
apply_iqn_trunk_gradient).
Phase B will add attention path gradients when weights are unfrozen.
Implements Task 1 from docs/superpowers/plans/2026-03-24-remaining-gpu-features.md.
The existing attention CUDA kernel (attention_kernel.cu) is now called as a
post-graph operation that modifies save_h_s2 for the next graph replay (1-step lag).
Changes:
- config.rs: add `use_attention: bool` to DQNHyperparameters (default: false)
- fused_training.rs: add `gpu_attention: Option<GpuAttention>` to FusedTrainingCtx;
initialize from hyperparams.use_attention in new() using shared_h2 as state_dim;
call apply_attention_forward() in run_full_step() after EMA (Step 3b), before IQL
- gpu_dqn_trainer.rs: add GpuAttention import and apply_attention_forward() method
that stream-syncs, wraps in EventTrackingGuard, calls attention.forward(), then
DtoD-copies the attended output back into save_h_s2
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The IQN backward kernel now computes dL/d(h_s2) = dL/d(combined) ⊙ embed
and outputs it to d_h_s2_buf [B, hidden_dim]. Previously this was
explicitly NOT computed (comment: "trunk trained by C51").
Now the shared trunk receives BOTH gradient signals:
C51: dense cross-entropy gradient (can be noisy/steep)
IQN: bounded Huber quantile gradient (always stable)
The IQN gradient stabilizes trunk training when C51's gradient is steep.
With both signals, the trunk learns from C51's distributional knowledge
AND IQN's risk-aware quantile knowledge simultaneously.
New: d_h_s2_buf allocated in GpuIqnHead, zeroed before backward,
accumulated via atomicAdd across all quantiles per sample.
Accessors added: GpuDqnTrainer::bw_d_h_s2_buf(), shared_h2()
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
With the larger state space (72 dims: market + portfolio + multi-TF)
and complex reward structure, the old 1e-4 default produces steep
loss surfaces and doubling grad_norm per epoch. 3e-5 gives the
optimizer a gentler starting point. Hyperopt still searches [1e-5, 3e-4].
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Layer 3a: Fixed τ midpoints replace random sampling in IQN.
τ_i = (2i-1)/(2N) for i=1..N, pre-computed in constructor.
Eliminates sample_taus_kernel launch → fully deterministic →
CUDA Graph compatible. IQN becomes equivalent to QR-DQN.
Layer 3b: IQN per_sample_loss replaces C51 td_errors for PER.
GPU DtoD copy — zero CPU. IQN's Huber quantile loss is bounded
by construction (max = kappa²/2 per quantile). This eliminates
the PER feedback explosion that caused C51 loss to reach 242K.
No conditional flag — IQN is always primary when active.
Three-layer defense complete:
L1: Per-sample CE clamp at 50 (breaks PER loop)
L2: Label smoothing ε=0.01 on Bellman target (prevents log(0))
L3: IQN Huber loss for PER priorities (bounded by construction)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Layer 1: Per-sample CE clamped to MAX_PER_SAMPLE_CE=50 before IS weight.
Breaks PER feedback loop: high loss → high priority → high IS weight → repeat.
Layer 2: Bellman target smoothed with ε=0.01 uniform mix after projection.
Prevents any atom from having zero probability → no log(near-zero) in CE.
Root cause: C51 cross-entropy is unbounded when target and predicted
distributions are maximally misaligned. PER amplifies pathological samples.
These two layers cap the maximum possible CE and prevent the extreme
misalignment from occurring.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
grad_norm was growing 5x per epoch (523→2673→13K→67K→NaN). The old
clip at 10.0 allowed gradients to accumulate. With clip=1.0, the
Adam optimizer receives bounded updates.
Trial 2 (TPE-guided params) trains cleanly for 4 epochs:
train_loss: 4.5→3.3 (decreasing!)
Q-value: 12-19 (stable)
grad_norm: 162-567 (bounded)
Trial 1 (random initial params) still NaN's — this is expected and
handled by the hyperopt penalty (1M objective). The optimizer learns
to avoid unstable parameter regions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Gradient explosion from unbounded multi-timeframe features (240-bar
return could be ±50%). All multi-timeframe outputs now clamped:
- Return: ±10%
- Volatility: 0-10%
- Volume ratio: 0-5x
- Momentum: [0, 1] (already bounded)
NaN guards added:
- Kelly: NaN check on continuous/discrete Kelly before blend
- target_position: final NaN→0 after all scaling
- reward: final NaN→0 before writing to replay buffer
Local test shows Q-gap=18.7 (epoch 5) — 18x better than start-of-session 0.000.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add ExpertDemoGenerator using MA crossover strategy filtered by ADX trend
strength to produce (bar_index, exposure_action_index) pairs. Fast/slow EMA
crossover with ADX > threshold triggers Long100/Short100 signals, otherwise Flat.
Includes effective_ratio() for linear decay of expert_demo_ratio over training
epochs (controlled by expert_demo_ratio and expert_demo_decay_epochs fields
in DQNHyperparameters). Comprehensive unit tests for EMA, ADX, decay, and
edge cases included.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add difficulty_score field to WalkForwardWindow computed via compute_difficulty()
which calculates mean ADX(14) over training bars. Add DifficultyPhase enum (Easy/Mixed/Full)
and filter_by_difficulty() for curriculum-based window filtering — Easy phase trains
only on trending markets (ADX>30), Mixed duplicates trending windows for 2x weight,
Full uses all windows equally. The curriculum_phase field in DQNHyperparameters
(committed in previous ensemble commit) controls the active phase.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
16 new features computed directly in the state_gather CUDA kernel
from market data already on GPU. No CPU feature engineering.
Lookback windows: 5, 15, 60, 240 bars (≈5min, 15min, 1hr, 4hr)
Features per window:
- Return over N bars (directional bias)
- Volatility (high-low range / close)
- Volume trend (current / N-bar average)
- Momentum (position within N-bar range [0=bottom, 1=top])
State dim: 66 raw (72 aligned) without OFI, 74 raw (80 aligned) with.
The model now sees price action at 4 timescales simultaneously.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wire ensemble agent configuration into DQN hyperparameters:
- ensemble_count: number of agents in ensemble (default 1 = no ensemble)
- ensemble_diversity_weight: KL-divergence diversity loss weight (default 0.01)
Both fields default to backward-compatible values (single agent, no
diversity loss) and are plumbed through the conservative() constructor.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
