Major changes:
- Eliminate ALL runtime NVRTC: c51_loss + mse_loss kernels converted to
precompiled cubins with runtime num_atoms/v_min/v_max/branch params
- Fix evaluator SIGSEGV: rng_states/q_gaps sized for chunked batch (cn)
not n_windows; NULL pointer guard in action_select kernel
- Add trade_physics.cuh shared header for train/eval consistency
- Add equity circuit breaker (25% DD from peak) + margin-aware position cap
- Consolidate 46D→22D hyperopt search space, enable ensemble by default
- Fix trade counting: use exposure index not factored action
- Fix Calmar overflow: clamp to ±100 in kernel
- Softer CVaR penalty (cap 3.0 not 10.0) for undertrained models
- Fix win_rate display (ratio→percentage)
- Remove dead code (normalize_reward, calculate_completion_penalty)
- Add tracing subscriber to hyperopt test for visible metrics
- Per-chunk sync in evaluator for reliable error reporting
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Forward kernel has shared_concat[128] (512 bytes) but Rust allocated d*4
bytes. With d=72: 288 < 512 → shared memory overflow. Same for backward
(concat[128] + d_proj[128] = 1024 bytes vs d*8).
Fix: use d.max(128) for shared memory sizing.
Also: revert c51/mse to runtime NVRTC (NUM_ATOMS controls loop bounds).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
c51_loss and mse_loss kernels use NUM_ATOMS for loop bounds AND shared
memory. Precompiling with NUM_ATOMS=51 then running with num_atoms=101
causes wrong loop iterations. These 2 kernels MUST use runtime #define
injection to match the hyperopt config.
35 of 37 kernels remain precompiled. Only c51_loss and mse_loss use
runtime NVRTC (they're the only kernels where a #define controls logic).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Same class of bug as IQN_HIDDEN: compile-time array float local_proj[51]
overflows when hyperopt picks num_atoms=101. Crashed at ~35 min when
PSO explored larger atom counts.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause (found via Zen debug + Gemini 2.5 Pro):
IQN_DIST_MAX = (IQN_HIDDEN + 31) / 32 = 8 (from IQN_HIDDEN=256).
Register arrays h_dist[8], embed_dist[8], comb_dist[8] overflow when
hyperopt picks hidden_dim > 256. With hidden_dim=512: loops iterate
16 times on 8-element arrays → stack corruption → SIGSEGV.
Fix: IQN_HIDDEN=2048 → IQN_DIST_MAX=64. Runtime loops still use
actual hidden_dim param. Only register array sizing affected — no
perf impact (unused elements are never accessed).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
IQN_EMBED_DIM (cosine embedding dimension, default 64) converted from
compile-time #define to runtime kernel parameter `embed_dim`. This was
the last remaining hardcoded dimension in the precompiled IQN cubin
that could diverge from GpuIqnConfig.embed_dim at runtime, causing
SIGSEGV from out-of-bounds weight offset computation.
All 9 IQN kernels (forward_loss, backward, grad_norm, adam, forward,
trunk_forward, ema, decode_actions, sample_taus) now receive embed_dim
as the final parameter. The iqn_compute_offsets() device helper uses
the runtime embed_dim for W_EMBED→B_EMBED offset calculation.
The other 7 kernel files in the task spec were verified safe:
- c51/mse_loss_kernel.cu: NVRTC-compiled with config values injected
- attention/attention_backward: already parameterized (state_dim, num_heads)
- dt_kernels.cu: NVRTC-compiled with config values injected
- curiosity/ppo: architecturally constant values matching Rust constants
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Build-time #define was 256 but runtime config can be 128 (hidden_dim_base).
Caused CUDA_ERROR_ILLEGAL_ADDRESS in IQN trunk gradient. Now passed as
kernel params matching the STATE_DIM parameterization pattern.
Also: warn→error for training failures in optimizer, eprintln for debugging.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Build-time #define was 256 but runtime config can be 128 (hidden_dim_base).
Caused CUDA_ERROR_ILLEGAL_ADDRESS in IQN trunk gradient. Now passed as
kernel params — matches the pattern used for STATE_DIM parameterization.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
increment_trial() was never called — budget enforcement was broken because
TrialBudgetObserver had its own internal Arc<AtomicUsize> disconnected from
the trial_counter used by evaluate_point() and cost(). With num_trials=2,
PSO ran 20 particle evaluations instead of stopping at 2.
Now TrialBudgetObserver::new() takes the caller-owned Arc<AtomicUsize> so
all evaluation paths (LHS via evaluate_point + PSO via cost()) share one
counter. ObjectiveFunction::cost() and ParallelObjectiveFunction::cost()
increment trial_counter directly (fetch_add SeqCst) and check budget inline;
observe_iter() reads the same counter to terminate PSO between iterations.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Backtest portfolio now tracks hold_time at index [5]. min_hold_bars
param enforces hold constraint during evaluation, matching training.
Fixes train/eval mismatch that caused NaN Sharpe → 1e6 penalty.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The kernel was updated with portfolio_states, min_hold_bars, max_position
params (Task 3 of trade lifecycle fix) but the backtest evaluator's launch
site wasn't updated. It passed 10 args to a 13-param kernel, causing
SIGSEGV from reading garbage for the missing params.
Fix: pass NULL/0/0.0 for the 3 new params (hold enforcement disabled
during evaluation for now — will be enabled in eval cleanup plan).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extend build.rs to compile all 25 .cu kernel files to cubins via nvcc.
Common header prepended to 24 kernels; experience_kernels.cu is standalone.
Total cubin size ~1.1 MB embedded in binary.
- cargo:rerun-if-changed tracks all .cu and .cuh files
- CUDA_COMPUTE_CAP env var selects target arch (sm_86, sm_90)
- Graceful skip when nvcc unavailable (non-CUDA builds)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The reversal P&L block (Kelly stats, entry_price reset) ran BEFORE the
hold enforcement guard. If hold_time < min_hold_bars cancelled a reversal,
the P&L was already booked for a trade that didn't execute, corrupting
portfolio_states. This caused SIGSEGV in the hyperopt path where the
corrupted state triggered invalid memory access in subsequent kernels.
Fix: moved the reversal P&L block from line 751 to after line 862
(after hold enforcement resolves final reversing_trade value).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Trade count dropped from 77% turnover (620K/800K) to 16.5% (527/3200)
with min_hold_bars=3. The hold enforcement is working correctly in the
standard training path. SIGSEGV in hyperopt path is a separate issue
(likely buffer sizing in the hyperopt adapter's experience collector).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
With v_range ±240 (vs old ±2), C51 distributional loss is naturally
~100-400 instead of ~3-6. Updated smoke test assertion accordingly.
Trade lifecycle fix validated: 467 trades (was 620K) with min_hold_bars=3.
PF=1.87, WinRate=49.7%, MaxDD=41.7% — model learns multi-bar swing trading.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add build.rs that compiles .cu kernels to cubins at cargo build time via nvcc.
Replace OnceLock<Ptx> + compile_ptx_for_device() in gpu_action_selector.rs with
include_bytes! + Ptx::from_binary() loading from the precompiled cubin.
- build.rs gracefully skips when nvcc is not available (non-CUDA builds)
- CUDA_COMPUTE_CAP env var controls target GPU arch (86=RTX 3050, 90=H100)
- common_device_functions.cuh prepended to kernel sources at build time
- Zero runtime compiler invocations for epsilon_greedy kernel
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
7 tasks: build.rs POC → parameterize #define → update launches → build all
→ replace runtime compilation → delete NVRTC infra → integration test.
Key insight: convert #define STATE_DIM/NUM_ATOMS to kernel params so each
kernel has ONE version regardless of model config. No multi-variant compilation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Integer parameter (rounded from continuous). Lets PSO discover
optimal hold duration for ES futures regime characteristics.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
During min_hold period, forces exposure to current position index.
Covers both greedy (Q-value) and random (epsilon) paths.
Portfolio state already GPU-resident — zero-copy parameter pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add min_hold_bars to DQNHyperparameters (usize, default 5), ExperienceSection
in training_profile, and both TOML configs (smoketest=3, production=5).
Wired through apply_to() so TOML overrides land correctly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fixed struct name: ExperienceCollectorConfig (not GpuExperienceCollectorConfig)
- Fixed max_pos scoping: moved declaration before action_select launch
- Fixed exiting_trade ordering: preliminary variable before trailing stop
- Documented hardcoded 5-action limitation in branching_action_select
- Zero Q-gap during hold periods (conviction meaningless when forced)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Layer 1 now targets experience_action_select (GPU-fused training path)
AND branching_action_select (backtest/fallback), not just the fallback
- Action masking covers both greedy AND random exploration paths
- Exposure index uses parameterized b0_size, not hardcoded 5 or 9
- Fixed end-of-episode variable name (total_bars, not timesteps_per_episode)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Added BARS_PER_DAY, BARS_PER_YEAR, ANNUALIZATION_FACTOR to common::thresholds::time
- Added bars_per_day field to DQNHyperparameters (default 390.0, configurable)
- compute_epoch_financials() now takes bars_per_day parameter from hyperparams
- coordinator_extended.rs + ab_testing.rs use centralized ANNUALIZATION_FACTOR
- When switching to tick data, change one constant in common/thresholds.rs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Compounding per-bar returns over 10K+ bars still produces extreme numbers
(+1.8M% return, 2687% MaxDD). Replaced with:
- Return: mean_per_bar × bars_per_year (annualized, no compounding)
- MaxDD: windowed over last 10K bars AND capped at 1.0 (100%)
These are monitoring metrics — the hyperopt objective uses the backtest
evaluator which already has correct windowed metrics.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The feature cache auto-invalidates via content hash — manual rm -f
wasted ~2 minutes of MBP-10 + OFI recomputation per trial start.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Training epoch metrics compound step_returns multiplicatively. With 2M+
bars per epoch on H100, this produces Return=+2.1e24% and MaxDD=99.5% —
meaningless numbers that corrupt risk monitoring and early stopping.
Fixed: total_return and max_drawdown now use only the last 10K bars
(~25 trading days), matching the backtest evaluator's window cap.
Sharpe/Sortino are unaffected (use arithmetic mean/std, not compounding).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
dqn-smoke: Walk-forward validation DQN uses raw price returns (~0.001),
not production reward_scale=10. Set v_min/v_max to ±10 for the small
16-dim 3-action test network (was inheriting ±240 from production default).
dqn-early-stop: Gradient collapse threshold = lr × multiplier must exceed
the actual gradient norm to trigger collapse. With v_range ±240, gradient
norms reach 100-10000 (was ~0.5-2.0 with old ±2.0 range). Updated
multiplier from 1e9 to 1e12 to guarantee threshold (10000) > grad norm.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Noisy nets are now always on (mandatory feature since config unification).
Epsilon start always uses the noisy-nets-aware default (0.05).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- dqn_action_collapse_fix_test: search space grew from 39D to 45D
(added c51_warmup, her_ratio, curiosity, cvar, dt_pretrain). Updated
assertions to use >= 39 and dynamic index lookup for cql_alpha.
- training_stability: Q-value assertion widened from 2.5 to 300.0
to match v_range ±240 (computed from reward_scale=10, gamma=0.95).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds mbp10_data_dir and trades_data_dir to CampaignConfig, passed to
DQNTrainer via with_ofi_data_dirs(). Enables OFI features (VPIN,
Kyle's Lambda, etc.) in hyperopt campaigns. dqn_full() now defaults
to 50 epochs/trial for baseline runs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: backtest windows of 300K bars produced ±billions% returns via
multiplicative compounding, and sqrt(252) annualization was wrong for
1-minute bars.
Fixes:
- Window size capped to 10K bars (~25 trading days), evenly distributed
across the full validation set (was clustered in first 6%)
- Annualization: configurable bars_per_day field in GpuBacktestConfig
(default 390.0 for 1-min), produces sqrt(98280) ≈ 313.5
- tanh normalization recalibrated: Sharpe/5, Sortino/8 (was /2, /3)
- CVaR threshold scaled to per-bar: 0.003 with slope 1400 (was 0.05/200)
- VaR/CVaR strided sampling covers full window (was first 4096 only)
- financials.rs + ab_testing.rs: sqrt(252) → sqrt(98280) for consistency
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Hybrid approach: each of 256 threads exports boundary metadata (first/last
action, prefix/suffix return, interior trade count). Thread 0 stitches
boundaries in O(256) to produce exact trade count and win rate.
Fixes: trades spanning chunk boundaries were fragmented (returns lost,
counts incorrect). Now every trade's cumulative return is tracked exactly
regardless of which thread chunks it spans.
7 boundary values stored in s_sorted[stride..8*stride] — zero extra
shared memory (reuses sort scratch between equity reduction and bitonic sort).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
s_max_dd, s_trades, s_wins parallel reductions were made redundant by
the exact sequential drawdown scan and upcoming boundary stitching.
Reduces shared memory from 25KB to 22KB and removes 3 dead ops/bar
from the per-thread loop. Trade count/win rate temporarily zeroed —
restored by boundary stitching in next commit.
The stress tester creates a SECOND DQNTrainer for validation, which
OOMs on 4GB GPUs. Changed from fatal error (?) to warning + skip.
Hyperopt now successfully trains on RTX 3050 — 2 trials × 10 epochs
completed in 19 minutes. Backtest metrics still show extreme returns
(multiplicative compounding over 895K bars) — needs separate fix.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The DQNTrainer adapter in hyperopt silently returns penalty (1000000)
because build_hyperparams() fails with the cleaned config struct.
Needs: update DQNParams→DQNHyperparameters mapping for removed use_
booleans and new reward_scale field.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add 7 new fields to DQNHyperparameters (fill_ioc_fill_prob,
fill_limit_fill_min, fill_limit_fill_max, fill_spread_cost_frac,
fill_spread_capture_frac, q_clip_min, q_clip_max) and wire them
through training_profile.rs into ExperienceCollectorConfig construction
in training_loop.rs. Previously these 7 values were hardcoded at the
construction site; now they flow from TOML [experience.fill_simulation]
and [risk] sections. Default values match the prior hardcoded constants
so existing behavior is unchanged.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
