Add min_hold_bars to DQNHyperparameters (usize, default 5), ExperienceSection
in training_profile, and both TOML configs (smoketest=3, production=5).
Wired through apply_to() so TOML overrides land correctly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fixed struct name: ExperienceCollectorConfig (not GpuExperienceCollectorConfig)
- Fixed max_pos scoping: moved declaration before action_select launch
- Fixed exiting_trade ordering: preliminary variable before trailing stop
- Documented hardcoded 5-action limitation in branching_action_select
- Zero Q-gap during hold periods (conviction meaningless when forced)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Layer 1 now targets experience_action_select (GPU-fused training path)
AND branching_action_select (backtest/fallback), not just the fallback
- Action masking covers both greedy AND random exploration paths
- Exposure index uses parameterized b0_size, not hardcoded 5 or 9
- Fixed end-of-episode variable name (total_bars, not timesteps_per_episode)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Added BARS_PER_DAY, BARS_PER_YEAR, ANNUALIZATION_FACTOR to common::thresholds::time
- Added bars_per_day field to DQNHyperparameters (default 390.0, configurable)
- compute_epoch_financials() now takes bars_per_day parameter from hyperparams
- coordinator_extended.rs + ab_testing.rs use centralized ANNUALIZATION_FACTOR
- When switching to tick data, change one constant in common/thresholds.rs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Compounding per-bar returns over 10K+ bars still produces extreme numbers
(+1.8M% return, 2687% MaxDD). Replaced with:
- Return: mean_per_bar × bars_per_year (annualized, no compounding)
- MaxDD: windowed over last 10K bars AND capped at 1.0 (100%)
These are monitoring metrics — the hyperopt objective uses the backtest
evaluator which already has correct windowed metrics.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The feature cache auto-invalidates via content hash — manual rm -f
wasted ~2 minutes of MBP-10 + OFI recomputation per trial start.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Training epoch metrics compound step_returns multiplicatively. With 2M+
bars per epoch on H100, this produces Return=+2.1e24% and MaxDD=99.5% —
meaningless numbers that corrupt risk monitoring and early stopping.
Fixed: total_return and max_drawdown now use only the last 10K bars
(~25 trading days), matching the backtest evaluator's window cap.
Sharpe/Sortino are unaffected (use arithmetic mean/std, not compounding).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
dqn-smoke: Walk-forward validation DQN uses raw price returns (~0.001),
not production reward_scale=10. Set v_min/v_max to ±10 for the small
16-dim 3-action test network (was inheriting ±240 from production default).
dqn-early-stop: Gradient collapse threshold = lr × multiplier must exceed
the actual gradient norm to trigger collapse. With v_range ±240, gradient
norms reach 100-10000 (was ~0.5-2.0 with old ±2.0 range). Updated
multiplier from 1e9 to 1e12 to guarantee threshold (10000) > grad norm.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Noisy nets are now always on (mandatory feature since config unification).
Epsilon start always uses the noisy-nets-aware default (0.05).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- dqn_action_collapse_fix_test: search space grew from 39D to 45D
(added c51_warmup, her_ratio, curiosity, cvar, dt_pretrain). Updated
assertions to use >= 39 and dynamic index lookup for cql_alpha.
- training_stability: Q-value assertion widened from 2.5 to 300.0
to match v_range ±240 (computed from reward_scale=10, gamma=0.95).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds mbp10_data_dir and trades_data_dir to CampaignConfig, passed to
DQNTrainer via with_ofi_data_dirs(). Enables OFI features (VPIN,
Kyle's Lambda, etc.) in hyperopt campaigns. dqn_full() now defaults
to 50 epochs/trial for baseline runs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: backtest windows of 300K bars produced ±billions% returns via
multiplicative compounding, and sqrt(252) annualization was wrong for
1-minute bars.
Fixes:
- Window size capped to 10K bars (~25 trading days), evenly distributed
across the full validation set (was clustered in first 6%)
- Annualization: configurable bars_per_day field in GpuBacktestConfig
(default 390.0 for 1-min), produces sqrt(98280) ≈ 313.5
- tanh normalization recalibrated: Sharpe/5, Sortino/8 (was /2, /3)
- CVaR threshold scaled to per-bar: 0.003 with slope 1400 (was 0.05/200)
- VaR/CVaR strided sampling covers full window (was first 4096 only)
- financials.rs + ab_testing.rs: sqrt(252) → sqrt(98280) for consistency
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Hybrid approach: each of 256 threads exports boundary metadata (first/last
action, prefix/suffix return, interior trade count). Thread 0 stitches
boundaries in O(256) to produce exact trade count and win rate.
Fixes: trades spanning chunk boundaries were fragmented (returns lost,
counts incorrect). Now every trade's cumulative return is tracked exactly
regardless of which thread chunks it spans.
7 boundary values stored in s_sorted[stride..8*stride] — zero extra
shared memory (reuses sort scratch between equity reduction and bitonic sort).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
s_max_dd, s_trades, s_wins parallel reductions were made redundant by
the exact sequential drawdown scan and upcoming boundary stitching.
Reduces shared memory from 25KB to 22KB and removes 3 dead ops/bar
from the per-thread loop. Trade count/win rate temporarily zeroed —
restored by boundary stitching in next commit.
The stress tester creates a SECOND DQNTrainer for validation, which
OOMs on 4GB GPUs. Changed from fatal error (?) to warning + skip.
Hyperopt now successfully trains on RTX 3050 — 2 trials × 10 epochs
completed in 19 minutes. Backtest metrics still show extreme returns
(multiplicative compounding over 895K bars) — needs separate fix.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The DQNTrainer adapter in hyperopt silently returns penalty (1000000)
because build_hyperparams() fails with the cleaned config struct.
Needs: update DQNParams→DQNHyperparameters mapping for removed use_
booleans and new reward_scale field.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add 7 new fields to DQNHyperparameters (fill_ioc_fill_prob,
fill_limit_fill_min, fill_limit_fill_max, fill_spread_cost_frac,
fill_spread_capture_frac, q_clip_min, q_clip_max) and wire them
through training_profile.rs into ExperienceCollectorConfig construction
in training_loop.rs. Previously these 7 values were hardcoded at the
construction site; now they flow from TOML [experience.fill_simulation]
and [risk] sections. Default values match the prior hardcoded constants
so existing behavior is unchanged.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove use_double_dqn, use_dueling, use_per, use_branching,
use_distributional, use_noisy_nets, use_huber_loss, and use_cql
from DQNConfig, DQNHyperparameters, and DqnParams structs.
These features are always enabled (Rainbow DQN standard). The boolean
flags were dead code — every constructor set them to true, and the
only code paths that set them to false were in tests that disabled
features for simplicity. With the fields removed, the features are
unconditionally active, eliminating ~490 lines of dead configuration.
Key changes:
- Struct field declarations removed from 3 core config structs
- Conditional branches (if use_X { ... } else { ... }) simplified:
dueling/branching/PER network creation is now unconditional
- Checkpoint metadata hardcodes "true" for backward compatibility
- Hyperopt search space index 11 (use_branching) fixed at 1.0
- TOML/YAML config files cleaned of removed fields
- Tests that toggled these flags updated or rewritten
45 files changed, -487 net lines. Zero new test failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Five bugs in the GPU backtest metrics kernel produced impossible results
(17,975% return, 100% MaxDD on every trial):
1. Additive return accumulation (local_cum += r) replaced with
multiplicative compounding (local_cum *= 1+r, init 1.0)
2. Absolute drawdown (peak - current) replaced with fractional
drawdown ((peak - current) / peak)
3. Strided bar processing (thread sees every Nth bar) replaced with
consecutive chunked processing for correct drawdown tracking
4. Per-bar win counting replaced with per-trade win/loss tracking
5. Total return now computed via multiplicative equity product
reduction across threads (s_sorted[0] - 1.0)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Five reward computation fixes in experience_env_step CUDA kernel:
1. Replace CUSUM vol proxy with ATR(14): CUSUM at feature[41] is a binary
direction indicator [-1,1,0], NOT volatility. When CUSUM≈0, vol_proxy
became 0.0001 causing 10000x reward amplification. ATR(14) at feature[9]
is actual realized volatility — reverse the safe_normalize encoding
(ln(atr)+7)/16 to recover atr_pct = exp(norm*16-7) / price.
2. Move loss aversion BEFORE squash: previously applied after hard clamp,
creating asymmetric [-15, +10] range making expected reward negative
even for fair strategies. Now applied pre-squash for smooth asymmetry.
3. Replace hard clamp with tanh soft squash: fmaxf(-10, fminf(10, reward))
destroyed tail information (1% and 5% wins both → 10.0). tanh preserves
that larger wins produce proportionally larger rewards.
4. Remove turnover penalty: the 0.05*|delta|/max_position penalty double-
counted transaction costs already deducted from cash via Almgren-Chriss
impact model at line ~679, over-penalizing necessary rebalancing.
5. Clarify CUSUM spread_scale usage: CUSUM at feature[41] is correctly used
as market-stress proxy for spread widening in tx cost computation — this
is distinct from the (now-fixed) vol proxy for reward normalization.
Also: annotate min_hold_bars=5 as hyperopt candidate.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The DBN loader was producing 2-element targets [close, next_close] which got
zero-padded to 4 elements. Now produces full 4-element layout matching the
kernel's expected format: [0:1]=network input, [2:3]=raw prices for portfolio sim.
Kernel reads tgt[2:3] for raw_close/raw_next (restored to original design).
All other kernels (DT, PPO, expert demos) also read tgt[2:3] correctly.
Stale feature cache invalidated by this data format change.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ROOT CAUSE of 0% win rate: The experience kernel read raw_close from tgt[2] and
raw_next from tgt[3], but the DBN data loader produces 2-element target vectors
[current_close, next_close] which get zero-padded to 4 elements. So tgt[2:3]=0,
triggering the degenerate price guard (raw_close=1.0, raw_next=1.0), making
every trade's P&L exactly zero → classified as loss.
Fix: read tgt[0] and tgt[1] which contain the actual raw prices.
Result: wins=264/594 (44.4% win rate), PF improving 0.44→0.88→0.98 over 3 epochs.
Also: segment-based trade detection correctly counts reversals,
old_pos_pnl uses saved pre-update position for correct P&L computation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Trade detection now counts reversals (S100→L50) as completed segments
- old_pos_pnl saved before position update for correct reversal P&L
- realized_pnl writeback uses old position PnL on reversal bars
- 0% win rate persists — needs deeper investigation (likely tx cost interaction)
WIP: The trade_return formula produces correct sign for raw market moves,
but every trade still shows as a loss. Suspect tx costs on both entry AND
exit of each reversal segment exceed the 1-bar price movement.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Task 9: Document ensemble consensus limitation — ensemble heads live on
FusedTrainingCtx (training-only), not accessible at inference time.
The ensemble already provides value via diversity gradient during training.
Task 10: Add three-phase training pipeline to the epoch loop:
- Phase 1 (Behavioral Cloning): C51 warmup + expert demos (already existed)
- Phase 2 (Full-Stack RL): all features, expert decay (already existed)
- Phase 3 (Refinement, last 20%): force expert_ratio=0, shrink-and-perturb
at phase boundary for plasticity consolidation
Task 11: Expand hyperopt search space from 41D to 45D with 4 new dims:
- her_ratio [0.0, 0.5]: HER relabeling ratio (was hardcoded 0.0)
- curiosity_weight [0.0, 0.2]: intrinsic reward weight (was fixed 0.0)
- use_cvar_action_selection [0.0, 1.0]: risk-aware IQN action scoring
- cvar_alpha [0.01, 0.2]: CVaR confidence level
All wired through build_hyperparams to DQNHyperparameters.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Task 7: Wire ensemble diversity gradient through value head into shared trunk.
The KL gradient kernel was computing d_logits but gradient flow stopped there.
Now: d_logits → cuBLAS backward W_v2 → ReLU mask → backward W_v1 → d_h_s2 →
trunk backward (layers 2,1) → SAXPY(diversity_weight) into grad_buf.
Uses launch_dx_only for value head layers (skip dW/db — only d_h_s2 needed).
graph_adam sees combined C51 + IQN + ensemble diversity in single update.
Task 8: Decision Transformer Phase 1 already fully implemented. DT runs before
main training loop when dt_pretrain_epochs > 0: builds trajectories from GPU
data, runs pretrain_step per epoch/batch, logs loss. Verified and confirmed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace reward v4 (pure mark-to-market return) with reward v5, a two-component
trade-aware hybrid that separates dense per-bar signal from sparse trade-exit signal:
Dense (every bar, weight 0.1): raw_pnl / equity when in a trade, zero when flat.
Keeps gradients flowing without overwhelming the sparse trade completion signal.
Sparse (at trade exit, weight 2.0): trade_return * patience_multiplier where
patience = sqrt(hold_time / expected_hold). Regime-adaptive expected hold via
ADX: trending (ADX>30) = 20 bars, ranging (ADX<20) = 8 bars, default = 12 bars.
Dynamic trailing stop: regime-adaptive trail distance (0.5% base, widens with
volatility via CUSUM and trend via ADX). Activates when trade is profitable and
held > 2 bars. Locks in profits by forcing exit when unrealized P&L drops below
the trailing floor.
Also fixes kernel signature mismatch: removes 7 old reward v2 parameters
(w_dsr, w_pnl, w_dd, w_idle, dd_threshold, time_decay_rate, eta) that were
already removed from the Rust launcher in a prior commit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaces the 8-component dense shaping + sparse trade completion with:
reward_t = (equity_t - equity_{t-1}) / equity_{t-1}
- Losing bars get NEGATIVE reward (every bar, not just exit)
- Flat bars get ZERO reward
- Trade entry: tx_cost hits cash → immediate negative reward
- Loss aversion: losses weighted 1.5x (prospect theory)
- No more DSR, no dense shaping, no hold penalty
NOTE: v_range needs recalibration for percentage returns (~0.001/bar)
instead of the old reward scale (~0.01-2.0). Current v_range=42 is
1000x too wide for the new reward magnitude.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. Risk management (CVaR, conviction, Kelly) re-enabled as ENVIRONMENT PHYSICS:
- Agent observes scaling via portfolio state features
- Learns to account for risk limits in its policy
- No longer destroys credit assignment (scaling is physics, not action override)
- Kelly uses half-Kelly (0.5x) for safety, activates after 20 trades
2. Backtest tx_cost now uses training's transaction_cost_multiplier from hyperopt
(was hardcoded 0.1 bps — 17x lower than training). Training and eval see same costs.
3. Backtest env tx_cost formula expanded to match training:
- Square-root market impact (Almgren-Chriss)
- Order-type premiums (Market=0, IoC=+2bps, LimitMaker=-5bps)
Result: first POSITIVE Sharpe (+0.0838) in project history. 134K trades.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. Monitoring kernel: prepend common_device_functions.cuh for DQN_ORDER_ACTIONS
2. Backtest metrics kernel: prepend common_device_functions.cuh
3. Backtest gather kernel: prepend common_device_functions.cuh
4. Backtest tx_cost: use training's transaction_cost_multiplier from hyperopt
(was hardcoded 0.1 bps — 17x lower than training's ~1.7 bps multiplier)
All standalone kernels now consistently include common_device_functions.cuh
for DQN action space defines. No more hardcoded constants.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. IQN gradient sequencing: train_step_gpu replays forward ONLY, caller injects
IQN/attention/ensemble gradients into grad_buf, then calls replay_adam_and_readback().
Single Adam sees combined C51+IQN gradient. Previously IQN was a NO-OP (SAXPY
happened after both graphs completed — Adam already consumed gradients).
2. Gradient clip 1.0 → 10.0: C51 with 101 atoms × 3 branches produces 300x larger
gradients than standard DQN. Clip at 1.0 made effective LR ~7e-12. Result:
grad_norm 137K → 490 (280x reduction, network actually learns now).
3. max_abs_reward 3.0 → 1.5: tighter C51 support [-42, +42] instead of [-84, +84].
Q-values at 24 (58% of v_max) instead of 81 (96%). 2x atom resolution.
4. choppy_bonus removed: Flat reward was 0.02 on 60-70% of bars, dominating
normalized reward distribution. Now Flat gets exactly 0.0.
5. Reward normalization: Welford EMA was broken (alpha=0.01 over 150K samples →
variance converges to zero → divides by 1e-8 → Q-value explosion). Fixed with
batch-level mean/std + EMA blending + variance floor 0.01.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Split CUDA Graph (forward + adam phases with gradient injection point):
- IQN trunk gradient flows through single Adam (no dual optimizer conflict)
- Spectral norm runs BEFORE forward (not after Adam — no tug-of-war)
- σ_max in 40D hyperopt search space [1.0, 10.0]
Attention Phase B backward:
- Full gradient flow through 4-head self-attention weights
- Separate Adam optimizer for attention params
- Backward kernel recomputes forward from saved_input (memory-efficient)
Ensemble multi-head:
- Real cuBLAS value head forward per ensemble head (was copying head 0 logits)
- KL diversity gradient kernel with hierarchical reduction
- forward_value_head() on CublasForward for per-head SGEMM
Regime PER scaling:
- Kernel reads target ADX/CUSUM from states_buf directly (zero CPU readback)
- Removed 2x memcpy_dtoh per training step
Decision Transformer:
- 14 CUDA kernels (embed, causal attention, FFN, CE loss + backward + trajectory building)
- GPU-native trajectory builder (return-to-go reverse cumsum, momentum expert actions)
- Wired into training loop with dt_pretrain_epochs config
HER Future/Final:
- episode_ids flow through PER buffer (GpuBatch, GpuReplayBuffer, GpuBatchSlices)
- GPU-native donor sampling (binary search on episode boundaries)
- Strategy dispatch in fused_training.rs
Backtest SEGV fix:
- Missing q_gaps_buf argument in action_select kernel launch
- Dynamic branch_sizes from agent (not hardcoded)
Local test: objective=10.48, Sharpe=0.0419, 175K trades, zero errors
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add 5 DT fields to DQNHyperparameters (dt_pretrain_epochs, dt_context_len,
dt_embed_dim, dt_num_layers, dt_target_return) with conservative() defaults
(dt_pretrain_epochs=0 = disabled)
- Wire pre-training phase before the main DQN epoch loop: logs config when
dt_pretrain_epochs > 0, noting that pretrain_step() kernels are ready in
decision_transformer.rs and full integration awaits trajectory data pipeline
- Add dt_pretrain_epochs to DQNParams struct, Default impl, from_continuous,
and the hyperparams struct literal in the hyperopt adapter (fixed to 0 for
all hyperopt trials, not in search space)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
