gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

refactor(dqn): f64 → f32 for kernel-facing hyperparams

Browse Source 
Eliminates the f64→f32 cudarc ABI trap (feedback_cudarc_f64_f32_abi.md,
task #82) at the type level: hyperparameters consumed by CUDA kernels
now live as f32 in Rust, cast once at the TOML/PSO ingest boundary
instead of at every kernel call site.

Structs changed:
  - DQNHyperparameters (crates/ml/src/trainers/dqn/config.rs) —
    ~85 scalar fields migrated from f64 → f32. Covers all
    kernel-facing scalars: reward weights (w_pnl/w_dd/w_idle,
    dd_threshold, cea_weight, micro_reward_*, price_confirm_weight,
    book_aggression_weight, hold_quality_weight), exploration
    (epsilon_* and the 4 branch mults, noisy_sigma_*, count_bonus,
    noise_sigma, q_gap_threshold), distributional RL (v_min, v_max,
    reward_scale, iqn_lambda, qr_kappa, spectral_*,
    gradient_collapse_multiplier), fill simulation (5 fill_*
    fields), risk/Kelly (kelly_fractional, kelly_max_fraction,
    max_leverage, max_position_absolute, minimum_profit_factor),
    ensemble/curiosity (curiosity_weight,
    curiosity_q_penalty_lambda, ensemble_*, beta_*, variance_cap),
    anti-LR (anti_lr_*, adversarial_dd_threshold,
    beta_penalty_strength), walk-forward (wf_*),
    experience (avg_spread, transaction_cost_multiplier,
    holding_cost_rate, churn_penalty_scale, contract_multiplier,
    margin_pct, tick_size, bars_per_day, cash_reserve_percent),
    misc kernel scalars (gamma, tau, huber_delta, q_clip_*,
    shrink_perturb_*, regime_replay_decay, per_alpha,
    per_beta_start, dt_target_return, etc.). Also
    `noisy_epsilon_floor: Option<f32>` and
    `count_bonus_coefficient: Option<f32>`.
  - `computed_v_min` / `computed_v_max` now return f32.
  - `compute_max_position` returns f32 (f64 internally for the
    notional division).

Fields preserved as f64 (precision-sensitive, NOT kernel-facing
scalars — per task spec and feedback_cudarc_f64_f32_abi.md):
  - `learning_rate` — tested at 1e-10 tolerance; f32 rounds to
    2e-12 for a 1e-5 LR.
  - `entropy_coefficient` — tested at 1e-9 tolerance.
  - `weight_decay` — tiny 1e-5..1e-3 range.
  - `adam_epsilon` — 1e-8 default; f32 preserves denorms here but
    paired with weight_decay/learning_rate for symmetry.
  - `gradient_clip_norm: Option<f64>` — grad norms are f64
    accumulators by project convention.
  - `min_loss_improvement_pct`, `q_value_floor` — early-stopping
    long-horizon stats.
  - `cql_alpha` — flows into DQNConfig (ml-dqn) still f64.
  - `min_learning_rate`, `lr_min` — paired with learning_rate.
  - Family intensity scalars (6 `*_intensity` fields) — f64 PSO
    search space; the intensity applies via `as f32` at each call
    site in `apply_family_scaling`.

No checkpoint format change: DQNHyperparameters has
`#[derive(Debug, Clone)]` only (not Serialize/Deserialize), so the
TOML-ingest path is the only serde boundary and already casts
explicitly via `hp.field = v as f32;` in
`DqnTrainingProfile::apply_to`. PSO hyperopt bounds stay f64 in
`SearchSpaceSection` and cast at the adapter boundary.

Call-site impact:
  - ~30 `as f32` casts removed from hot paths (fused_training
    FusedConfig builder, training_loop kernel launches,
    constructor DQNConfig builder, action.rs GPU action selector,
    trainer/mod.rs WF config). Kernels now receive the hp field
    directly via `&hp.x`.
  - ~75 `as f32` casts added at the `apply_to` / hyperopt-adapter
    ingest boundary — the single conversion point.
  - Cross-crate contracts (DQNConfig in ml-dqn, PortfolioTracker
    in ml-core, GAECalculator, DropoutScheduler, NoisySigmaScheduler,
    KellyOptimizerConfig, RewardConfig) retain their f64 signatures;
    ml calls cast at the boundary with `f64::from(hp.x)` so the
    contract is explicit and greppable.

Two test-side adjustments:
  - `test_kelly_fields_are_public` now asserts `f32` for
    kelly_fractional / kelly_max_fraction (these migrated).
  - `test_early_stopping_termination` uses `f64::from(...)` to
    preserve the f64 threshold computation against the now-f32
    `gradient_collapse_multiplier`.

Verified:
  - `SQLX_OFFLINE=true CARGO_INCREMENTAL=0 RUSTC_WRAPPER=sccache
    cargo check --workspace` — clean, zero new warnings.
  - `cargo check --workspace --tests` — clean.
  - `cargo test -p ml --lib training_profile::tests` — all 18
    migrated tests pass. The one pre-existing failure
    (`test_production_profile_applies_all_sections` n_steps
    mismatch, 1 vs 5) reproduces on stashed baseline, so it is
    unrelated to this change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-04-24 01:55:14 +02:00
parent b12483d91b
commit d64adc14f5
13 changed files with 334 additions and 307 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
crates/ml/src/hyperopt/adapters/dqn.rs
View File

@@ -2142,7 +2142,7 @@ impl HyperparameterOptimizable for DQNTrainer {
hyperparams.warmup_steps = 0; // CRITICAL: batch training path doesn't increment total_steps hyperparams.warmup_steps = 0; // CRITICAL: batch training path doesn't increment total_steps
// Preprocessing is always active // Preprocessing is always active
hyperparams.preprocessing_window = self.preprocessing_window; hyperparams.preprocessing_window = self.preprocessing_window;
hyperparams.preprocessing_clip_sigma = self.preprocessing_clip_sigma; hyperparams.preprocessing_clip_sigma = self.preprocessing_clip_sigma as f32;
hyperparams.target_update_mode = self.target_update_mode.clone(); hyperparams.target_update_mode = self.target_update_mode.clone();
hyperparams.target_update_frequency = self.target_update_frequency; hyperparams.target_update_frequency = self.target_update_frequency;
#[allow(clippy::cast_possible_truncation)] #[allow(clippy::cast_possible_truncation)]
@@ -2160,18 +2160,19 @@ impl HyperparameterOptimizable for DQNTrainer {
/ (49152.0 - 8192.0)) / (49152.0 - 8192.0))
.clamp(0.0, 1.0); .clamp(0.0, 1.0);
let max_fraction = if budget.gpu_memory_mb >= 81920 { 0.80 } else { 0.70 }; let max_fraction = if budget.gpu_memory_mb >= 81920 { 0.80 } else { 0.70 };
raw * max_fraction (raw * max_fraction) as f32
}; };
// Clamp buffer to [1024, max] — below 1024 the GPU PER segment tree fails. // Clamp buffer to [1024, max] — below 1024 the GPU PER segment tree fails.
hyperparams.buffer_size = hyperparams.buffer_size.max(1024).min(self.buffer_size_max); hyperparams.buffer_size = hyperparams.buffer_size.max(1024).min(self.buffer_size_max);
// === 4 breakout parameters from PSO === // === 4 breakout parameters from PSO ===
hyperparams.gamma = params.gamma; // PSO search space is f64; cast to f32 at the hp boundary.
hyperparams.gamma = params.gamma as f32;
// Recompute v_min/v_max (reward_scale-derived, gamma-independent) // Recompute v_min/v_max (reward_scale-derived, gamma-independent)
hyperparams.v_min = hyperparams.computed_v_min(); hyperparams.v_min = hyperparams.computed_v_min();
hyperparams.v_max = hyperparams.computed_v_max(); hyperparams.v_max = hyperparams.computed_v_max();
hyperparams.iqn_lambda = params.iqn_lambda; hyperparams.iqn_lambda = params.iqn_lambda as f32;
hyperparams.c51_warmup_epochs = params.c51_warmup_epochs.round() as usize; hyperparams.c51_warmup_epochs = params.c51_warmup_epochs.round() as usize;
hyperparams.c51_alpha_max = params.c51_alpha_max as f32; hyperparams.c51_alpha_max = params.c51_alpha_max as f32;
@@ -2566,7 +2567,7 @@ impl HyperparameterOptimizable for DQNTrainer {
window_size, window_size,
stride, stride,
&device, &device,
hyperparams.max_position_absolute, f64::from(hyperparams.max_position_absolute),
) { ) {
Ok(m) => { Ok(m) => {
if m.is_some() { if m.is_some() {

300
crates/ml/src/trainers/dqn/config.rs
View File

@@ -404,32 +404,35 @@ impl DQNAgentType {
/// DQN training hyperparameters from gRPC request /// DQN training hyperparameters from gRPC request
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct DQNHyperparameters { pub struct DQNHyperparameters {
/// Learning rate (typically 1e-4 to 1e-3) /// Learning rate (typically 1e-4 to 1e-3).
///
/// Kept as f64 for numerical precision of very small LRs (1e-5..1e-8) and for
/// hyperopt bound arithmetic. Cast to f32 at the Adam/optimizer boundary only.
pub learning_rate: f64, pub learning_rate: f64,
/// Batch size (dynamically capped by AutoBatchSizer based on GPU VRAM) /// Batch size (dynamically capped by AutoBatchSizer based on GPU VRAM)
pub batch_size: usize, pub batch_size: usize,
/// Discount factor (typically 0.95-0.99) /// Discount factor (typically 0.95-0.99)
pub gamma: f64, pub gamma: f32,
/// Initial exploration rate /// Initial exploration rate
pub epsilon_start: f64, pub epsilon_start: f32,
/// Final exploration rate /// Final exploration rate
pub epsilon_end: f64, pub epsilon_end: f32,
/// Exploration decay rate /// Exploration decay rate
pub epsilon_decay: f64, pub epsilon_decay: f32,
/// Per-branch epsilon multiplier: direction (relative to base epsilon). Default 1.0. /// Per-branch epsilon multiplier: direction (relative to base epsilon). Default 1.0.
pub epsilon_direction_mult: f64, pub epsilon_direction_mult: f32,
/// Per-branch epsilon multiplier: magnitude (relative to base epsilon). Default 2.0 — needs more exploration. /// Per-branch epsilon multiplier: magnitude (relative to base epsilon). Default 2.0 — needs more exploration.
pub epsilon_magnitude_mult: f64, pub epsilon_magnitude_mult: f32,
/// Per-branch epsilon multiplier: order type (relative to base epsilon). Default 1.5. /// Per-branch epsilon multiplier: order type (relative to base epsilon). Default 1.5.
pub epsilon_order_mult: f64, pub epsilon_order_mult: f32,
/// Per-branch epsilon multiplier: urgency (relative to base epsilon). Default 1.5. /// Per-branch epsilon multiplier: urgency (relative to base epsilon). Default 1.5.
pub epsilon_urgency_mult: f64, pub epsilon_urgency_mult: f32,
/// Replay buffer capacity /// Replay buffer capacity
pub buffer_size: usize, pub buffer_size: usize,
/// Fraction of free VRAM to allocate for replay buffer (0.0 = use static buffer_size). /// Fraction of free VRAM to allocate for replay buffer (0.0 = use static buffer_size).
/// When > 0.0, overrides buffer_size with VRAM-optimal capacity. /// When > 0.0, overrides buffer_size with VRAM-optimal capacity.
/// Typical: 0.70 (70% of free VRAM after model load). /// Typical: 0.70 (70% of free VRAM after model load).
pub replay_buffer_vram_fraction: f64, pub replay_buffer_vram_fraction: f32,
/// Minimum replay buffer size before training starts /// Minimum replay buffer size before training starts
pub min_replay_size: usize, pub min_replay_size: usize,
/// Number of training epochs /// Number of training epochs
@@ -438,41 +441,43 @@ pub struct DQNHyperparameters {
pub checkpoint_frequency: usize, pub checkpoint_frequency: usize,
/// Enable early stopping based on convergence criteria /// Enable early stopping based on convergence criteria
pub early_stopping_enabled: bool, pub early_stopping_enabled: bool,
/// Minimum Q-value threshold before stopping (default: 0.5) /// Minimum Q-value threshold before stopping (default: 0.5).
/// f64: early-stopping plateau stat, not a kernel-facing scalar.
pub q_value_floor: f64, pub q_value_floor: f64,
/// Minimum loss improvement percentage over window (default: 2.0%) /// Minimum loss improvement percentage over window (default: 2.0%).
/// f64: ratio of long-horizon loss accumulators, keep precision.
pub min_loss_improvement_pct: f64, pub min_loss_improvement_pct: f64,
/// Window size for plateau detection (default: 30 epochs) /// Window size for plateau detection (default: 30 epochs)
pub plateau_window: usize, pub plateau_window: usize,
/// Minimum epochs before early stopping can trigger (default: 50) /// Minimum epochs before early stopping can trigger (default: 50)
pub min_epochs_before_stopping: usize, pub min_epochs_before_stopping: usize,
/// Deprecated: replaced by `w_idle` in composite reward. Kept for hyperopt adapter compat. /// Deprecated: replaced by `w_idle` in composite reward. Kept for hyperopt adapter compat.
pub hold_penalty: f64, pub hold_penalty: f32,
// GPU composite reward weights (6-component composite in CUDA kernel) // GPU composite reward weights (6-component composite in CUDA kernel)
/// Normalized PnL weight for composite reward /// Normalized PnL weight for composite reward
pub w_pnl: f64, pub w_pnl: f32,
/// Drawdown penalty weight for composite reward /// Drawdown penalty weight for composite reward
pub w_dd: f64, pub w_dd: f32,
/// Idle penalty weight for composite reward /// Idle penalty weight for composite reward
pub w_idle: f64, pub w_idle: f32,
/// Drawdown tolerance before penalty kicks in (fraction, e.g. 0.02 = 2%) /// Drawdown tolerance before penalty kicks in (fraction, e.g. 0.02 = 2%)
pub dd_threshold: f64, pub dd_threshold: f32,
/// Asymmetric loss scaling factor (prospect theory, default 1.5) /// Asymmetric loss scaling factor (prospect theory, default 1.5)
pub loss_aversion: f64, pub loss_aversion: f32,
/// Reward v7: Counterfactual Exposure Advantage weight. /// Reward v7: Counterfactual Exposure Advantage weight.
/// Blends per-branch advantage signal into the scalar reward. /// Blends per-branch advantage signal into the scalar reward.
/// 0.0 = disabled, 1.0 = full advantage signal. /// 0.0 = disabled, 1.0 = full advantage signal.
pub cea_weight: f64, pub cea_weight: f32,
// exposure_aux_weight and exposure_aux_warmup_epochs removed (4-branch refactor) // exposure_aux_weight and exposure_aux_warmup_epochs removed (4-branch refactor)
/// Position staleness rent per step held /// Position staleness rent per step held
pub time_decay_rate: f64, pub time_decay_rate: f32,
/// Q-value gap threshold for trade conviction filter (FRACTION of Q-range). /// Q-value gap threshold for trade conviction filter (FRACTION of Q-range).
/// When greedy Q(best_dir) - Q(flat) < threshold × Q_range(direction), default to flat. /// When greedy Q(best_dir) - Q(flat) < threshold × Q_range(direction), default to flat.
/// Auto-scales with v_min/v_max: 0.05 = "need ≥5% of Q-spread conviction to trade." /// Auto-scales with v_min/v_max: 0.05 = "need ≥5% of Q-spread conviction to trade."
/// 0.0 = disabled. Typical range [0.0, 0.5]. /// 0.0 = disabled. Typical range [0.0, 0.5].
pub q_gap_threshold: f64, pub q_gap_threshold: f32,
/// v8: Dense directional micro-reward scale (R5 reward term). /// v8: Dense directional micro-reward scale (R5 reward term).
/// Parameter, NOT a feature flag. Controls the tanh-bounded per-bar /// Parameter, NOT a feature flag. Controls the tanh-bounded per-bar
/// shaping reward built from OFI momentum, price confirmation, and /// shaping reward built from OFI momentum, price confirmation, and
@@ -482,17 +487,17 @@ pub struct DQNHyperparameters {
/// Phase 2 Task 2.3 FIX-documented-disable — per /// Phase 2 Task 2.3 FIX-documented-disable — per
/// feedback_no_functionality_removal.md the kernel branch and buffer /// feedback_no_functionality_removal.md the kernel branch and buffer
/// infrastructure are preserved even when the scale is 0 in a given profile. /// infrastructure are preserved even when the scale is 0 in a given profile.
pub micro_reward_scale: f64, pub micro_reward_scale: f32,
/// Dense micro-reward: price confirmation weight (default 0.5). /// Dense micro-reward: price confirmation weight (default 0.5).
pub price_confirm_weight: f64, pub price_confirm_weight: f32,
/// Dense micro-reward: book aggression weight (default 0.3). /// Dense micro-reward: book aggression weight (default 0.3).
pub book_aggression_weight: f64, pub book_aggression_weight: f32,
/// Dense micro-reward: retrospective hold quality weight (default 0.2). /// Dense micro-reward: retrospective hold quality weight (default 0.2).
pub hold_quality_weight: f64, pub hold_quality_weight: f32,
/// Dense micro-reward: tanh temperature (default 3.0). /// Dense micro-reward: tanh temperature (default 3.0).
pub micro_reward_temp: f64, pub micro_reward_temp: f32,
/// v8: TD(λ) trace parameter. 0=1-step, 1=Monte Carlo. /// v8: TD(λ) trace parameter. 0=1-step, 1=Monte Carlo.
pub td_lambda: f64, pub td_lambda: f32,
/// v8: Maximum trace length for truncated TD(λ). /// v8: Maximum trace length for truncated TD(λ).
pub max_trace_length: usize, pub max_trace_length: usize,
/// v8: PopArt reward normalization enabled. /// v8: PopArt reward normalization enabled.
@@ -503,29 +508,31 @@ pub struct DQNHyperparameters {
/// v8: Curriculum learning enabled. /// v8: Curriculum learning enabled.
pub curriculum_enabled: bool, pub curriculum_enabled: bool,
/// v8: Fraction of training before full dataset (curriculum). /// v8: Fraction of training before full dataset (curriculum).
pub curriculum_warmup_fraction: f64, pub curriculum_warmup_fraction: f32,
/// v8: Fraction of replay batch relabeled with hindsight optimal exit. /// v8: Fraction of replay batch relabeled with hindsight optimal exit.
pub hindsight_fraction: f64, pub hindsight_fraction: f32,
/// v8: Lookahead bars for hindsight optimal exit. /// v8: Lookahead bars for hindsight optimal exit.
pub hindsight_lookahead: usize, pub hindsight_lookahead: usize,
/// Huber loss delta threshold (default: 1.0) /// Huber loss delta threshold (default: 1.0)
pub huber_delta: f64, pub huber_delta: f32,
/// Gradient clipping max norm (None = disabled) /// Gradient clipping max norm (None = disabled).
/// f64: aggregated norm is an f64 accumulator by convention; keep precision at the
/// norm-computation boundary.
pub gradient_clip_norm: Option<f64>, pub gradient_clip_norm: Option<f64>,
/// Deprecated: was HOLD action penalty weight, replaced by `w_idle` in GPU composite reward. /// Deprecated: was HOLD action penalty weight, replaced by `w_idle` in GPU composite reward.
/// Kept for backward compatibility with hyperopt adapters. Not used in reward computation. /// Kept for backward compatibility with hyperopt adapters. Not used in reward computation.
pub hold_penalty_weight: f64, pub hold_penalty_weight: f32,
/// Price movement threshold for HOLD penalty (as fraction, e.g., 0.02 = 2%) /// Price movement threshold for HOLD penalty (as fraction, e.g., 0.02 = 2%)
pub movement_threshold: f64, pub movement_threshold: f32,
/// Preprocessing window size (default: 50) /// Preprocessing window size (default: 50)
pub preprocessing_window: i64, pub preprocessing_window: i64,
/// Preprocessing clip sigma (default: 5.0) /// Preprocessing clip sigma (default: 5.0)
pub preprocessing_clip_sigma: f64, pub preprocessing_clip_sigma: f32,
// WAVE 16 (Agent 36): Target update configuration // WAVE 16 (Agent 36): Target update configuration
/// Polyak averaging coefficient for soft target updates (default: 0.001) /// Polyak averaging coefficient for soft target updates (default: 0.001)
/// Rainbow DQN standard: τ=0.001 gives 693-step convergence half-life /// Rainbow DQN standard: τ=0.001 gives 693-step convergence half-life
pub tau: f64, pub tau: f32,
/// Target update mode: Soft (Polyak averaging) or Hard (periodic full copy) /// Target update mode: Soft (Polyak averaging) or Hard (periodic full copy)
pub target_update_mode: crate::trainers::TargetUpdateMode, pub target_update_mode: crate::trainers::TargetUpdateMode,
/// Target network hard update frequency in training steps (default: 10000) /// Target network hard update frequency in training steps (default: 10000)
@@ -547,15 +554,15 @@ pub struct DQNHyperparameters {
/// 390 = 1-minute (6.5h × 60min), 78 = 5-minute, 1 = daily. /// 390 = 1-minute (6.5h × 60min), 78 = 5-minute, 1 = daily.
/// Used for Sharpe/Sortino annualization and return scaling. /// Used for Sharpe/Sortino annualization and return scaling.
/// Derived from data pipeline's BarSize when available. /// Derived from data pipeline's BarSize when available.
pub bars_per_day: f64, pub bars_per_day: f32,
/// Holding cost rate per bar per unit position. Continuous inventory penalty. /// Holding cost rate per bar per unit position. Continuous inventory penalty.
/// Default: 0.00000052 (~0.52 bps / 100 bars) /// Default: 0.00000052 (~0.52 bps / 100 bars)
pub holding_cost_rate: f64, pub holding_cost_rate: f32,
/// Bars threshold for churn penalty. Trades within this window get graduated penalty. /// Bars threshold for churn penalty. Trades within this window get graduated penalty.
pub churn_threshold_bars: usize, pub churn_threshold_bars: usize,
/// Scale factor for churn penalty. /// Scale factor for churn penalty.
pub churn_penalty_scale: f64, pub churn_penalty_scale: f32,
/// Auxiliary ops (IQL, IQN, attention, CQL) run every Nth training step. /// Auxiliary ops (IQL, IQN, attention, CQL) run every Nth training step.
/// Default: 4 (aux ops every 4th step, ~4× faster epochs). /// Default: 4 (aux ops every 4th step, ~4× faster epochs).
@@ -565,13 +572,13 @@ pub struct DQNHyperparameters {
// P2-B Enhancement: Cash reserve requirement // P2-B Enhancement: Cash reserve requirement
/// Cash reserve requirement as percentage of portfolio value (0-100) /// Cash reserve requirement as percentage of portfolio value (0-100)
/// Default: 0.0 (no reserve, backward compatible) /// Default: 0.0 (no reserve, backward compatible)
pub cash_reserve_percent: f64, pub cash_reserve_percent: f32,
// WAVE 16S: Adaptive Risk Management Features // WAVE 16S: Adaptive Risk Management Features
/// Kelly fractional multiplier (0.5 = half-Kelly, conservative) /// Kelly fractional multiplier (0.5 = half-Kelly, conservative)
pub kelly_fractional: f64, pub kelly_fractional: f32,
/// Maximum Kelly fraction (cap at 0.25 = 25% of portfolio) /// Maximum Kelly fraction (cap at 0.25 = 25% of portfolio)
pub kelly_max_fraction: f64, pub kelly_max_fraction: f32,
/// Minimum trades required for Kelly calculation /// Minimum trades required for Kelly calculation
pub kelly_min_trades: usize, pub kelly_min_trades: usize,
/// Volatility rolling window size /// Volatility rolling window size
@@ -584,14 +591,14 @@ pub struct DQNHyperparameters {
/// Periodic shrink-and-perturb interval (0=disabled, 20=every 20 epochs) /// Periodic shrink-and-perturb interval (0=disabled, 20=every 20 epochs)
pub shrink_perturb_interval: usize, pub shrink_perturb_interval: usize,
/// Shrink-and-perturb alpha (0.85 = keep 85%, reinit 15%) /// Shrink-and-perturb alpha (0.85 = keep 85%, reinit 15%)
pub shrink_perturb_alpha: f64, pub shrink_perturb_alpha: f32,
/// Shrink-and-perturb noise scale (Xavier) /// Shrink-and-perturb noise scale (Xavier)
pub shrink_perturb_sigma: f64, pub shrink_perturb_sigma: f32,
/// Adversarial regime injection: MaxDD threshold below which we suspect overfitting. /// Adversarial regime injection: MaxDD threshold below which we suspect overfitting.
/// If MaxDD stays below this for `adversarial_epochs_trigger` consecutive epochs, /// If MaxDD stays below this for `adversarial_epochs_trigger` consecutive epochs,
/// inject harsh market conditions to stress-test the policy. /// inject harsh market conditions to stress-test the policy.
pub adversarial_dd_threshold: f64, pub adversarial_dd_threshold: f32,
/// Consecutive low-drawdown epochs before adversarial injection activates /// Consecutive low-drawdown epochs before adversarial injection activates
pub adversarial_epochs_trigger: usize, pub adversarial_epochs_trigger: usize,
@@ -599,7 +606,7 @@ pub struct DQNHyperparameters {
/// At epoch boundary, Pearson correlation between model returns and market returns /// At epoch boundary, Pearson correlation between model returns and market returns
/// is computed. If |corr| > 0.3, next epoch reward_scale *= (1 - penalty * |corr|). /// is computed. If |corr| > 0.3, next epoch reward_scale *= (1 - penalty * |corr|).
/// Forces alpha generation over beta riding. 0.0 = disabled. Typical: 0.3-0.5. /// Forces alpha generation over beta riding. 0.0 = disabled. Typical: 0.3-0.5.
pub beta_penalty_strength: f64, pub beta_penalty_strength: f32,
// ── Gems & Pearls: generalization techniques ──────────────────── // ── Gems & Pearls: generalization techniques ────────────────────
@@ -607,21 +614,21 @@ pub struct DQNHyperparameters {
/// kick the model out of overfit minima. When Sharpe < -threshold, decrease /// kick the model out of overfit minima. When Sharpe < -threshold, decrease
/// to stabilize. Opposite of standard practice. Always active. /// to stabilize. Opposite of standard practice. Always active.
/// LR multiplier when epoch Sharpe is "good" (> threshold) /// LR multiplier when epoch Sharpe is "good" (> threshold)
pub anti_lr_good_mult: f64, pub anti_lr_good_mult: f32,
/// LR multiplier when epoch Sharpe is "bad" (< -threshold) /// LR multiplier when epoch Sharpe is "bad" (< -threshold)
pub anti_lr_bad_mult: f64, pub anti_lr_bad_mult: f32,
/// Sharpe threshold for anti-intuitive LR switching /// Sharpe threshold for anti-intuitive LR switching
pub anti_lr_sharpe_threshold: f64, pub anti_lr_sharpe_threshold: f32,
/// #27 Ensemble disagreement Q-penalty weight. When ensemble heads disagree /// #27 Ensemble disagreement Q-penalty weight. When ensemble heads disagree
/// (high std across Q-values), subtract weighted std from Q-target. /// (high std across Q-values), subtract weighted std from Q-target.
/// Makes model conservative on novel/uncertain states. 0.0 = disabled. /// Makes model conservative on novel/uncertain states. 0.0 = disabled.
pub ensemble_disagreement_penalty: f64, pub ensemble_disagreement_penalty: f32,
/// #34 Causal intervention: compute per-feature causal sensitivity by running /// #34 Causal intervention: compute per-feature causal sensitivity by running
/// intervened forward passes. Always active. /// intervened forward passes. Always active.
/// #34 Weight for causal regularization loss. 0.0 = disabled. /// #34 Weight for causal regularization loss. 0.0 = disabled.
pub causal_weight: f64, pub causal_weight: f32,
/// #34 How often to run causal intervention (every N training steps). /// #34 How often to run causal intervention (every N training steps).
/// Higher = less overhead. 10 = every 10th step. /// Higher = less overhead. 10 = every 10th step.
pub causal_intervention_interval: usize, pub causal_intervention_interval: usize,
@@ -635,12 +642,12 @@ pub struct DQNHyperparameters {
/// #23 Causal feature masking: each epoch, randomly zero out this fraction /// #23 Causal feature masking: each epoch, randomly zero out this fraction
/// of features. Forces redundant representations (random forest principle). /// of features. Forces redundant representations (random forest principle).
/// 0.0 = disabled, 0.4 = mask 40% of features each epoch. /// 0.0 = disabled, 0.4 = mask 40% of features each epoch.
pub feature_mask_fraction: f64, pub feature_mask_fraction: f32,
/// #22 Feature-wise noise injection: add Gaussian noise with sigma = /// #22 Feature-wise noise injection: add Gaussian noise with sigma =
/// noise_scale * std(feature). Strictly better than dropout for continuous /// noise_scale * std(feature). Strictly better than dropout for continuous
/// features. 0.0 = disabled. /// features. 0.0 = disabled.
pub feature_noise_scale: f64, pub feature_noise_scale: f32,
// #13 Volatility regime normalization: always active. // #13 Volatility regime normalization: always active.
// #10 Mirror universe: always active. // #10 Mirror universe: always active.
@@ -651,17 +658,17 @@ pub struct DQNHyperparameters {
/// #17 Counterfactual regret blend: 0.0 = pure PnL reward, 1.0 = pure regret. /// #17 Counterfactual regret blend: 0.0 = pure PnL reward, 1.0 = pure regret.
/// Regret = reward(taken) - max(reward(all_exposures)). Normalizes across regimes. /// Regret = reward(taken) - max(reward(all_exposures)). Normalizes across regimes.
pub regret_blend: f64, pub regret_blend: f32,
/// #25 Trade clustering penalty: penalize temporally clustered trades. /// #25 Trade clustering penalty: penalize temporally clustered trades.
/// CV(inter-trade intervals) * this weight is subtracted from reward. /// CV(inter-trade intervals) * this weight is subtracted from reward.
/// 0.0 = disabled. /// 0.0 = disabled.
pub trade_clustering_penalty: f64, pub trade_clustering_penalty: f32,
/// #20 Epoch at which to compute lottery ticket pruning mask. /// #20 Epoch at which to compute lottery ticket pruning mask.
pub pruning_epoch: usize, pub pruning_epoch: usize,
/// #20 Fraction of weights to prune (0.7 = remove 70% smallest). /// #20 Fraction of weights to prune (0.7 = remove 70% smallest).
pub pruning_fraction: f64, pub pruning_fraction: f32,
// #32 Gradient vaccine: always active. // #32 Gradient vaccine: always active.
@@ -673,12 +680,12 @@ pub struct DQNHyperparameters {
/// #21 Stochastic depth: drop probability per hidden layer during training. /// #21 Stochastic depth: drop probability per hidden layer during training.
/// Each layer is independently dropped with this probability. Surviving layers /// Each layer is independently dropped with this probability. Surviving layers
/// are scaled by 1/(1-p) for expected-value correction. 0.0 = disabled. /// are scaled by 1/(1-p) for expected-value correction. 0.0 = disabled.
pub stochastic_depth_prob: f64, pub stochastic_depth_prob: f32,
/// Manifold Mixup alpha for Beta distribution. /// Manifold Mixup alpha for Beta distribution.
/// Mixes pairs of C51 target distributions during training. /// Mixes pairs of C51 target distributions during training.
/// 0.0 = disabled. Range: [0.0, 2.0]. Default: 0.2. /// 0.0 = disabled. Range: [0.0, 2.0]. Default: 0.2.
pub mixup_alpha: f64, pub mixup_alpha: f32,
// ── Core hyperopt families (1.0 = defaults, range [0.0, 2.0]) ── // ── Core hyperopt families (1.0 = defaults, range [0.0, 2.0]) ──
@@ -711,28 +718,29 @@ pub struct DQNHyperparameters {
/// #18 Asymmetric drawdown loss: extra penalty on Q-overestimation when /// #18 Asymmetric drawdown loss: extra penalty on Q-overestimation when
/// portfolio is in drawdown. Weight for L_dd = max(0, Q_pred-Q_target) * dd^2. /// portfolio is in drawdown. Weight for L_dd = max(0, Q_pred-Q_target) * dd^2.
/// 0.0 = disabled. /// 0.0 = disabled.
pub asymmetric_dd_weight: f64, pub asymmetric_dd_weight: f32,
// Wave 16 Portfolio Features (action masking, entropy regularization, stress testing always active) // Wave 16 Portfolio Features (action masking, entropy regularization, stress testing always active)
/// Maximum absolute position size for action masking (contracts). /// Maximum absolute position size for action masking (contracts).
/// DERIVED: computed from `max_leverage`, `initial_capital`, `contract_multiplier`, /// DERIVED: computed from `max_leverage`, `initial_capital`, `contract_multiplier`,
/// and a reference price. Do not set directly in config — use `max_leverage` instead. /// and a reference price. Do not set directly in config — use `max_leverage` instead.
/// Fallback default: 2.0 (matches legacy production behavior). /// Fallback default: 2.0 (matches legacy production behavior).
pub max_position_absolute: f64, pub max_position_absolute: f32,
/// Maximum leverage ratio relative to initial capital. /// Maximum leverage ratio relative to initial capital.
/// Position cap = floor(initial_capital * max_leverage / (reference_price * contract_multiplier)). /// Position cap = floor(initial_capital * max_leverage / (reference_price * contract_multiplier)).
/// Default: 5.0 (5x leverage). Scaled by `risk_intensity` during hyperopt. /// Default: 5.0 (5x leverage). Scaled by `risk_intensity` during hyperopt.
pub max_leverage: f64, pub max_leverage: f32,
// WAVE 17: Hyperopt-tuned parameters // WAVE 17: Hyperopt-tuned parameters
/// SAC-style entropy coefficient for Q-value softmax regularization (default: 0.001) /// SAC-style entropy coefficient for Q-value softmax regularization (default: 0.001).
/// f64: tested at 1e-9 tolerance; reward-scale ratio sensitive to sub-1e-6 offsets.
pub entropy_coefficient: f64, pub entropy_coefficient: f64,
/// Minimum epsilon when noisy nets are enabled (default: 0.05) /// Minimum epsilon when noisy nets are enabled (default: 0.05)
pub noisy_epsilon_floor: Option<f64>, pub noisy_epsilon_floor: Option<f32>,
/// Count-based exploration bonus coefficient (default: 0.1, range: [0.01, 0.5]) /// Count-based exploration bonus coefficient (default: 0.1, range: [0.01, 0.5])
pub count_bonus_coefficient: Option<f64>, pub count_bonus_coefficient: Option<f32>,
/// Transaction cost multiplier for reward calculation /// Transaction cost multiplier for reward calculation
pub transaction_cost_multiplier: f64, pub transaction_cost_multiplier: f32,
// Wave 3 (Phase 2): Triple Barrier Method (always active) // Wave 3 (Phase 2): Triple Barrier Method (always active)
pub triple_barrier_profit_target_bps: u32, pub triple_barrier_profit_target_bps: u32,
@@ -740,13 +748,13 @@ pub struct DQNHyperparameters {
pub triple_barrier_max_holding_seconds: u64, pub triple_barrier_max_holding_seconds: u64,
// P0: Prioritized Experience Replay (always enabled) // P0: Prioritized Experience Replay (always enabled)
pub per_alpha: f64, pub per_alpha: f32,
pub per_beta_start: f64, pub per_beta_start: f32,
/// Regime-aware replay decay factor. PER priorities for cross-regime /// Regime-aware replay decay factor. PER priorities for cross-regime
/// transitions are multiplied by decay^|regime_distance|. /// transitions are multiplied by decay^|regime_distance|.
/// 1.0 = no bias. 0.0 = only current regime. Default: 0.3. /// 1.0 = no bias. 0.0 = only current regime. Default: 0.3.
pub regime_replay_decay: f64, pub regime_replay_decay: f32,
// Wave 2.1: Dueling Networks (always enabled) // Wave 2.1: Dueling Networks (always enabled)
/// Hidden dimension for dueling value/advantage streams /// Hidden dimension for dueling value/advantage streams
@@ -761,19 +769,19 @@ pub struct DQNHyperparameters {
/// Number of atoms for value distribution (Rainbow DQN standard: 51) /// Number of atoms for value distribution (Rainbow DQN standard: 51)
pub num_atoms: usize, pub num_atoms: usize,
/// Minimum value for distribution support (must be -v_max for symmetric support) /// Minimum value for distribution support (must be -v_max for symmetric support)
pub v_min: f64, pub v_min: f32,
/// Maximum value for distribution support (v_min = -v_max enforced by hyperopt) /// Maximum value for distribution support (v_min = -v_max enforced by hyperopt)
pub v_max: f64, pub v_max: f32,
/// Reward scale factor for v_range computation. /// Reward scale factor for v_range computation.
/// Q* ~ reward_scale / (1 - gamma). With 20% headroom and [20, 300] clamp: /// Q* ~ reward_scale / (1 - gamma). With 20% headroom and [20, 300] clamp:
/// v_range = (reward_scale / (1 - gamma) * 1.2).clamp(20.0, 300.0) /// v_range = (reward_scale / (1 - gamma) * 1.2).clamp(20.0, 300.0)
/// Default: 10.0 (reward v6 REWARD_SCALE). /// Default: 10.0 (reward v6 REWARD_SCALE).
pub reward_scale: f64, pub reward_scale: f32,
// Wave 2.4: Noisy Networks for Exploration (always enabled) // Wave 2.4: Noisy Networks for Exploration (always enabled)
/// Initial noise std dev (Rainbow DQN standard: 0.5, scaled by 1/√in_features) /// Initial noise std dev (Rainbow DQN standard: 0.5, scaled by 1/√in_features)
pub noisy_sigma_init: f64, pub noisy_sigma_init: f32,
// Two-Phase Feature Normalization Configuration // Two-Phase Feature Normalization Configuration
/// Ratio of total epochs to use for feature statistics collection (default: 0.3 = 30%) /// Ratio of total epochs to use for feature statistics collection (default: 0.3 = 30%)
@@ -790,7 +798,7 @@ pub struct DQNHyperparameters {
/// Adaptive gradient collapse threshold multiplier (default: 100.0) /// Adaptive gradient collapse threshold multiplier (default: 100.0)
/// Threshold = learning_rate × gradient_collapse_multiplier /// Threshold = learning_rate × gradient_collapse_multiplier
/// WAVE 23: Replaces hardcoded 0.1 threshold with learning-rate aware detection /// WAVE 23: Replaces hardcoded 0.1 threshold with learning-rate aware detection
pub gradient_collapse_multiplier: f64, pub gradient_collapse_multiplier: f32,
/// Consecutive epoch patience before early stopping (default: 5) /// Consecutive epoch patience before early stopping (default: 5)
/// Prevents false positives from single-epoch anomalies /// Prevents false positives from single-epoch anomalies
pub gradient_collapse_patience: usize, pub gradient_collapse_patience: usize,
@@ -799,14 +807,16 @@ pub struct DQNHyperparameters {
// NOTE: warmup_steps is defined above (line 331) - using same field for both purposes // NOTE: warmup_steps is defined above (line 331) - using same field for both purposes
/// Learning rate decay type after warmup /// Learning rate decay type after warmup
pub lr_decay_type: super::lr_scheduler::LRDecayType, pub lr_decay_type: super::lr_scheduler::LRDecayType,
/// Minimum learning rate for Cosine decay /// Minimum learning rate for Cosine decay.
/// f64: paired with `learning_rate` (f64); precision matters for the 1e-6..1e-5 floor.
pub min_learning_rate: f64, pub min_learning_rate: f64,
/// Minimum learning rate floor (absolute minimum, default: 1e-6) /// Minimum learning rate floor (absolute minimum, default: 1e-6).
/// f64: precision matters for the 1e-6 floor.
pub lr_min: f64, pub lr_min: f64,
/// Steps for learning rate decay (default: 1000) /// Steps for learning rate decay (default: 1000)
pub lr_decay_steps: usize, pub lr_decay_steps: usize,
/// Learning rate decay rate (default: 0.99) /// Learning rate decay rate (default: 0.99)
pub lr_decay_rate: f64, pub lr_decay_rate: f32,
// WAVE 26 P1.4: Ensemble Uncertainty for Exploration (6D hyperopt expansion) // WAVE 26 P1.4: Ensemble Uncertainty for Exploration (6D hyperopt expansion)
/// Enable ensemble uncertainty-based exploration (replaces noisy networks) /// Enable ensemble uncertainty-based exploration (replaces noisy networks)
@@ -814,23 +824,23 @@ pub struct DQNHyperparameters {
/// Number of Q-network heads in ensemble (3-10, default: 5) /// Number of Q-network heads in ensemble (3-10, default: 5)
pub ensemble_size: usize, pub ensemble_size: usize,
/// Variance penalty weight for uncertainty estimation (0.1-1.0) /// Variance penalty weight for uncertainty estimation (0.1-1.0)
pub beta_variance: f64, pub beta_variance: f32,
/// Disagreement penalty weight for epistemic uncertainty (0.1-1.0) /// Disagreement penalty weight for epistemic uncertainty (0.1-1.0)
pub beta_disagreement: f64, pub beta_disagreement: f32,
/// Entropy bonus weight for exploration diversity (0.05-0.5) /// Entropy bonus weight for exploration diversity (0.05-0.5)
pub beta_entropy: f64, pub beta_entropy: f32,
/// Variance cap to prevent over-penalization (0.1-2.0) /// Variance cap to prevent over-penalization (0.1-2.0)
pub variance_cap: f64, pub variance_cap: f32,
// WAVE 26 P1.8: Curiosity-Driven Exploration // WAVE 26 P1.8: Curiosity-Driven Exploration
/// Curiosity weight for intrinsic reward (0.0 = disabled, typical range: 0.0-0.5) /// Curiosity weight for intrinsic reward (0.0 = disabled, typical range: 0.0-0.5)
/// Balances extrinsic reward from trading vs intrinsic reward from novelty /// Balances extrinsic reward from trading vs intrinsic reward from novelty
pub curiosity_weight: f64, pub curiosity_weight: f32,
/// Curiosity Q-penalty lambda: scales Bellman gamma by 1/(1 + lambda * prediction_error). /// Curiosity Q-penalty lambda: scales Bellman gamma by 1/(1 + lambda * prediction_error).
/// When the curiosity forward model can't predict next state well (high error), /// When the curiosity forward model can't predict next state well (high error),
/// Q-values are discounted toward immediate reward — preventing overconfident /// Q-values are discounted toward immediate reward — preventing overconfident
/// extrapolation into novel/OOS states. 0.0 = disabled. Typical: 1.0-5.0. /// extrapolation into novel/OOS states. 0.0 = disabled. Typical: 1.0-5.0.
pub curiosity_q_penalty_lambda: f64, pub curiosity_q_penalty_lambda: f32,
// WAVE 26 P2.2: Gradient Accumulation // WAVE 26 P2.2: Gradient Accumulation
/// Number of mini-batches to accumulate gradients over before optimizer step /// Number of mini-batches to accumulate gradients over before optimizer step
@@ -846,21 +856,21 @@ pub struct DQNHyperparameters {
// P1.3: Sharpe Ratio Reward Component // P1.3: Sharpe Ratio Reward Component
/// Enable Sharpe ratio as reward component (0.0 = disabled, typical: 0.2-0.5) /// Enable Sharpe ratio as reward component (0.0 = disabled, typical: 0.2-0.5)
pub sharpe_weight: f64, pub sharpe_weight: f32,
/// Rolling window size for Sharpe calculation (default: 20) /// Rolling window size for Sharpe calculation (default: 20)
pub sharpe_window: usize, pub sharpe_window: usize,
// P1.6: Adaptive Dropout Scheduling (always active) // P1.6: Adaptive Dropout Scheduling (always active)
/// Initial dropout rate (default: 0.5) /// Initial dropout rate (default: 0.5)
pub dropout_initial: f64, pub dropout_initial: f32,
/// Final dropout rate (default: 0.1) /// Final dropout rate (default: 0.1)
pub dropout_final: f64, pub dropout_final: f32,
/// Steps for dropout annealing (default: 10000) /// Steps for dropout annealing (default: 10000)
pub dropout_anneal_steps: usize, pub dropout_anneal_steps: usize,
// P1.7: Hindsight Experience Replay (HER) // P1.7: Hindsight Experience Replay (HER)
/// Enable HER for improved data efficiency (0.0 = disabled, typical: 0.3-0.8) /// Enable HER for improved data efficiency (0.0 = disabled, typical: 0.3-0.8)
pub her_ratio: f64, pub her_ratio: f32,
/// HER strategy: "final" or "future" /// HER strategy: "final" or "future"
pub her_strategy: String, pub her_strategy: String,
@@ -869,28 +879,30 @@ pub struct DQNHyperparameters {
// P1.9: Generalized Advantage Estimation (GAE) — always active // P1.9: Generalized Advantage Estimation (GAE) — always active
/// GAE lambda parameter (default: 0.95) /// GAE lambda parameter (default: 0.95)
pub gae_lambda: f64, pub gae_lambda: f32,
// P1.11: Noisy Network Sigma Scheduling — always active // P1.11: Noisy Network Sigma Scheduling — always active
/// Initial sigma for noisy networks (default: 0.6) /// Initial sigma for noisy networks (default: 0.6)
pub noisy_sigma_initial: f64, pub noisy_sigma_initial: f32,
/// Final sigma for noisy networks (default: 0.4) /// Final sigma for noisy networks (default: 0.4)
pub noisy_sigma_final: f64, pub noisy_sigma_final: f32,
/// Steps for sigma annealing (default: 10000) /// Steps for sigma annealing (default: 10000)
pub noisy_sigma_anneal_steps: usize, pub noisy_sigma_anneal_steps: usize,
// WAVE 30: L2 Weight Decay for Overfitting Prevention // WAVE 30: L2 Weight Decay for Overfitting Prevention
/// L2 weight decay for AdamW optimizer (default: 1e-4, range: [1e-5, 1e-3]) /// L2 weight decay for AdamW optimizer (default: 1e-4, range: [1e-5, 1e-3]).
/// Prevents overfitting by penalizing large weights via L2 regularization. /// Prevents overfitting by penalizing large weights via L2 regularization.
/// Recommended: 1e-4 for standard training, 1e-3 for aggressive regularization. /// Recommended: 1e-4 for standard training, 1e-3 for aggressive regularization.
/// f64: tiny values (1e-5..1e-3) benefit from f64 precision during decay math.
pub weight_decay: f64, pub weight_decay: f64,
/// Adam epsilon — BF16 requires ≥1e-4 (1e-8 rounds to 0 in f32 → div-by-zero). /// Adam epsilon — BF16 requires ≥1e-4 (1e-8 rounds to 0 in f32 → div-by-zero).
/// f64: tiny values (1e-8..1e-4) preserve denorm behaviour.
pub adam_epsilon: f64, pub adam_epsilon: f64,
// Conservative Q-Learning (CQL) — always enabled, alpha controls strength // Conservative Q-Learning (CQL) — always enabled, alpha controls strength
/// CQL regularization strength (default: 0.1, range 0.0-1.0) /// CQL regularization strength (default: 0.1, range 0.0-1.0).
/// 0.0 = disabled, 0.1 = mild conservatism, 1.0 = full offline-RL strength /// f64: flows into DQNConfig.cql_alpha (ml-dqn crate, f64); convert in ml-dqn or at boundary.
pub cql_alpha: f64, pub cql_alpha: f64,
// QR-DQN (Quantile Regression DQN) -- replaces disabled C51 // QR-DQN (Quantile Regression DQN) -- replaces disabled C51
@@ -898,18 +910,18 @@ pub struct DQNHyperparameters {
/// Number of quantiles for QR-DQN (32-200) /// Number of quantiles for QR-DQN (32-200)
pub num_quantiles: usize, pub num_quantiles: usize,
/// Kappa for quantile Huber loss (0.5-2.0) /// Kappa for quantile Huber loss (0.5-2.0)
pub qr_kappa: f64, pub qr_kappa: f32,
/// Lambda weight for IQN quantile loss relative to C51 loss in the dual-head ensemble. /// Lambda weight for IQN quantile loss relative to C51 loss in the dual-head ensemble.
/// L_total = L_c51 + iqn_lambda * L_iqn /// L_total = L_c51 + iqn_lambda * L_iqn
/// Range [0.0, 2.0]: 0.0 = C51 only, 0.5 = balanced, 1.0 = equal weight /// Range [0.0, 2.0]: 0.0 = C51 only, 0.5 = balanced, 1.0 = equal weight
pub iqn_lambda: f64, pub iqn_lambda: f32,
/// Spectral norm σ_max — constrains ||W||_σ ≤ σ_max. /// Spectral norm σ_max — constrains ||W||_σ ≤ σ_max.
/// Range [1.0, 10.0]. Default 3.0 (permits Xavier scaling, prevents Q-explosion). /// Range [1.0, 10.0]. Default 3.0 (permits Xavier scaling, prevents Q-explosion).
pub spectral_norm_sigma_max: f64, pub spectral_norm_sigma_max: f32,
/// Spectral decoupling: L2 penalty on Q-value logit magnitudes. /// Spectral decoupling: L2 penalty on Q-value logit magnitudes.
/// Prevents overconfident Q-values. Pezeshki et al. 2021. /// Prevents overconfident Q-values. Pezeshki et al. 2021.
/// Range: [0.0, 0.1]. Default: 0.01. /// Range: [0.0, 0.1]. Default: 0.01.
pub spectral_decoupling_lambda: f64, pub spectral_decoupling_lambda: f32,
/// When true, IQN's bounded Huber loss is used for PER priorities instead of /// When true, IQN's bounded Huber loss is used for PER priorities instead of
/// C51's unbounded cross-entropy. Prevents the PER feedback explosion. /// C51's unbounded cross-entropy. Prevents the PER feedback explosion.
@@ -924,7 +936,7 @@ pub struct DQNHyperparameters {
/// Controls VRAM usage: total = gpu_n_episodes × gpu_timesteps_per_episode × 2 (counterfactual) × state_dim × 4 bytes. /// Controls VRAM usage: total = gpu_n_episodes × gpu_timesteps_per_episode × 2 (counterfactual) × state_dim × 4 bytes.
pub gpu_n_episodes: usize, pub gpu_n_episodes: usize,
/// Average bid-ask spread for GPU portfolio simulation (default: 0.0001 = 1bp) /// Average bid-ask spread for GPU portfolio simulation (default: 0.0001 = 1bp)
pub avg_spread: f64, pub avg_spread: f32,
/// Maximum training steps per epoch (0 = unlimited, uses full dataset). /// Maximum training steps per epoch (0 = unlimited, uses full dataset).
// max_training_steps_per_epoch removed — always train on full dataset. // max_training_steps_per_epoch removed — always train on full dataset.
@@ -937,7 +949,7 @@ pub struct DQNHyperparameters {
/// Minimum profit factor for trade execution (BUG #7 fix) /// Minimum profit factor for trade execution (BUG #7 fix)
/// Trades must have profit > cost × this factor to be executed. /// Trades must have profit > cost × this factor to be executed.
/// Range [1.1, 2.0]: 1.1 = 10% margin above costs, 2.0 = 100% margin. /// Range [1.1, 2.0]: 1.1 = 10% margin above costs, 2.0 = 100% margin.
pub minimum_profit_factor: f64, pub minimum_profit_factor: f32,
// Differential Sharpe Ratio (DSR) — always enabled in GPU composite reward kernel. // Differential Sharpe Ratio (DSR) — always enabled in GPU composite reward kernel.
/// Deprecated: DSR is always enabled. Kept for backward compatibility with hyperopt adapters. /// Deprecated: DSR is always enabled. Kept for backward compatibility with hyperopt adapters.
@@ -970,11 +982,11 @@ pub struct DQNHyperparameters {
/// Imbalance bar threshold for MBP-10 to bar conversion. /// Imbalance bar threshold for MBP-10 to bar conversion.
/// Higher = fewer bars (less noise), lower = more bars (more signal). /// Higher = fewer bars (less noise), lower = more bars (more signal).
/// Only used when `data_source = "mbp10"`. Default: 100.0. /// Only used when `data_source = "mbp10"`. Default: 100.0.
pub imbalance_bar_threshold: f64, pub imbalance_bar_threshold: f32,
/// EWMA alpha for adaptive imbalance threshold. Default: 0.1. /// EWMA alpha for adaptive imbalance threshold. Default: 0.1.
/// Only used when `data_source = "mbp10"`. /// Only used when `data_source = "mbp10"`.
pub imbalance_bar_ewma_alpha: f64, pub imbalance_bar_ewma_alpha: f32,
// Offline RL mode (CQL/IQL on fixed datasets) // Offline RL mode (CQL/IQL on fixed datasets)
/// Offline RL mode: skip online experience collection, train from a fixed dataset. /// Offline RL mode: skip online experience collection, train from a fixed dataset.
@@ -997,21 +1009,21 @@ pub struct DQNHyperparameters {
/// Enable IQL mode (Implicit Q-Learning for offline RL). /// Enable IQL mode (Implicit Q-Learning for offline RL).
// Fill simulation parameters for GPU experience collector // Fill simulation parameters for GPU experience collector
/// IoC base fill probability (default 0.85, range [0.5, 1.0]) /// IoC base fill probability (default 0.85, range [0.5, 1.0])
pub fill_ioc_fill_prob: f64, pub fill_ioc_fill_prob: f32,
/// Minimum limit fill probability at wide spread (default 0.30, range [0.1, 0.5]) /// Minimum limit fill probability at wide spread (default 0.30, range [0.1, 0.5])
pub fill_limit_fill_min: f64, pub fill_limit_fill_min: f32,
/// Maximum limit fill probability at tight spread (default 0.80, range [0.5, 0.95]) /// Maximum limit fill probability at tight spread (default 0.80, range [0.5, 0.95])
pub fill_limit_fill_max: f64, pub fill_limit_fill_max: f32,
/// Fraction of spread paid by market orders (default 0.50, range [0.2, 0.8]) /// Fraction of spread paid by market orders (default 0.50, range [0.2, 0.8])
pub fill_spread_cost_frac: f64, pub fill_spread_cost_frac: f32,
/// Fraction of spread captured by limit orders (default 0.50, range [0.2, 0.8]) /// Fraction of spread captured by limit orders (default 0.50, range [0.2, 0.8])
pub fill_spread_capture_frac: f64, pub fill_spread_capture_frac: f32,
// Q-value clipping for GPU experience collector // Q-value clipping for GPU experience collector
/// Minimum Q-value clamp in GPU kernel (default -200.0) /// Minimum Q-value clamp in GPU kernel (default -200.0)
pub q_clip_min: f64, pub q_clip_min: f32,
/// Maximum Q-value clamp in GPU kernel (default 200.0) /// Maximum Q-value clamp in GPU kernel (default 200.0)
pub q_clip_max: f64, pub q_clip_max: f32,
// Phase C+: Branching DQN (Tavakoli et al., 2018) — always enabled // Phase C+: Branching DQN (Tavakoli et al., 2018) — always enabled
/// Hidden dimension for each branching advantage head (64-256, step=64). /// Hidden dimension for each branching advantage head (64-256, step=64).
@@ -1021,13 +1033,13 @@ pub struct DQNHyperparameters {
// GPU Walk-Forward: expanding-window cross-validation entirely on GPU (always active) // GPU Walk-Forward: expanding-window cross-validation entirely on GPU (always active)
/// Initial training window as a fraction of total data (default: 0.5). /// Initial training window as a fraction of total data (default: 0.5).
pub wf_initial_train_fraction: f64, pub wf_initial_train_fraction: f32,
/// Validation window as a fraction of total data (default: 0.125). /// Validation window as a fraction of total data (default: 0.125).
pub wf_val_fraction: f64, pub wf_val_fraction: f32,
/// Test window as a fraction of total data (default: 0.125). /// Test window as a fraction of total data (default: 0.125).
pub wf_test_fraction: f64, pub wf_test_fraction: f32,
/// Step size between folds as a fraction of total data (default: 0.125). /// Step size between folds as a fraction of total data (default: 0.125).
pub wf_step_fraction: f64, pub wf_step_fraction: f32,
// Ensemble Agent Configuration // Ensemble Agent Configuration
/// Number of agents in ensemble (default: 1 = no ensemble). /// Number of agents in ensemble (default: 1 = no ensemble).
@@ -1037,7 +1049,7 @@ pub struct DQNHyperparameters {
/// KL-divergence diversity loss weight for ensemble training (default: 0.01). /// KL-divergence diversity loss weight for ensemble training (default: 0.01).
/// Encourages each agent in the ensemble to learn different strategies /// Encourages each agent in the ensemble to learn different strategies
/// by penalizing Q-value distribution similarity between ensemble members. /// by penalizing Q-value distribution similarity between ensemble members.
pub ensemble_diversity_weight: f64, pub ensemble_diversity_weight: f32,
// Curriculum learning: difficulty-based walk-forward window filtering // Curriculum learning: difficulty-based walk-forward window filtering
/// Curriculum phase for difficulty-based window filtering. /// Curriculum phase for difficulty-based window filtering.
@@ -1049,7 +1061,7 @@ pub struct DQNHyperparameters {
/// Fraction of each training batch sampled from expert demonstrations (0.0-1.0). /// Fraction of each training batch sampled from expert demonstrations (0.0-1.0).
/// 0.0 = no demonstrations (pure RL), 0.5 = half batch from expert. /// 0.0 = no demonstrations (pure RL), 0.5 = half batch from expert.
/// Default: 0.0 (disabled, backward compatible). /// Default: 0.0 (disabled, backward compatible).
pub expert_demo_ratio: f64, pub expert_demo_ratio: f32,
/// Number of epochs over which expert_demo_ratio decays linearly to 0. /// Number of epochs over which expert_demo_ratio decays linearly to 0.
/// After this many epochs, training is pure RL. /// After this many epochs, training is pure RL.
/// Default: 20. /// Default: 20.
@@ -1087,7 +1099,7 @@ pub struct DQNHyperparameters {
/// DT number of transformer layers. /// DT number of transformer layers.
pub dt_num_layers: usize, pub dt_num_layers: usize,
/// DT target return-to-go for conditioning (in units of cumulative reward). /// DT target return-to-go for conditioning (in units of cumulative reward).
pub dt_target_return: f64, pub dt_target_return: f32,
// CVaR (Conditional Value at Risk) action selection for IQN // CVaR (Conditional Value at Risk) action selection for IQN
/// CVaR confidence level alpha (0.0-1.0). Lower alpha = more risk-averse. /// CVaR confidence level alpha (0.0-1.0). Lower alpha = more risk-averse.
@@ -1101,7 +1113,7 @@ pub struct DQNHyperparameters {
/// outcomes, typically large-magnitude positions) get an exploration boost, /// outcomes, typically large-magnitude positions) get an exploration boost,
/// countering C51's distributional bias toward low-variance (Small) actions. /// countering C51's distributional bias toward low-variance (Small) actions.
/// 0.0 = disabled. Typical range [0.1, 0.5]. Default: 0.3. /// 0.0 = disabled. Typical range [0.1, 0.5]. Default: 0.3.
pub iqr_exploration_alpha: f64, pub iqr_exploration_alpha: f32,
/// Ensemble variance exploration beta: weight for adding ensemble Q-value /// Ensemble variance exploration beta: weight for adding ensemble Q-value
/// variance (disagreement between heads) as an exploration bonus. /// variance (disagreement between heads) as an exploration bonus.
@@ -1109,23 +1121,23 @@ pub struct DQNHyperparameters {
/// Applied uniformly across all actions: Q_select += beta * sqrt(mean_var). /// Applied uniformly across all actions: Q_select += beta * sqrt(mean_var).
/// Only active when ensemble_count > 1. No-op when ensemble_count <= 1. /// Only active when ensemble_count > 1. No-op when ensemble_count <= 1.
/// 0.0 = disabled. Typical range [0.05, 0.3]. Default: 0.2. /// 0.0 = disabled. Typical range [0.05, 0.3]. Default: 0.2.
pub ensemble_exploration_beta: f64, pub ensemble_exploration_beta: f32,
// Contract specification — symbol-dependent constants // Contract specification — symbol-dependent constants
/// Futures tick size (minimum price increment). /// Futures tick size (minimum price increment).
/// ES = 0.25, NQ = 0.25, 6E = 0.00005, ZN = 1/64 ≈ 0.015625. /// ES = 0.25, NQ = 0.25, 6E = 0.00005, ZN = 1/64 ≈ 0.015625.
/// Used with `contract_multiplier` to compute spread cost. /// Used with `contract_multiplier` to compute spread cost.
/// Default: 0.25 (ES/NQ). /// Default: 0.25 (ES/NQ).
pub tick_size: f64, pub tick_size: f32,
/// Contract multiplier (dollar value per point). /// Contract multiplier (dollar value per point).
/// ES = 50, NQ = 20, 6E = 125_000, ZN = 1000. /// ES = 50, NQ = 20, 6E = 125_000, ZN = 1000.
/// Used for margin computation and spread cost. /// Used for margin computation and spread cost.
/// Default: 50.0 (ES). /// Default: 50.0 (ES).
pub contract_multiplier: f64, pub contract_multiplier: f32,
/// Initial margin as fraction of notional value. /// Initial margin as fraction of notional value.
/// CME initial margin ≈ 6% of notional for equity index futures. /// CME initial margin ≈ 6% of notional for equity index futures.
/// Default: 0.06. /// Default: 0.06.
pub margin_pct: f64, pub margin_pct: f32,
/// Trading days per year for annualization. /// Trading days per year for annualization.
/// CME equity/futures: 252. Crypto: 365. Forex: 252. /// CME equity/futures: 252. Crypto: 365. Forex: 252.
pub trading_days_per_year: usize, pub trading_days_per_year: usize,
@@ -1134,7 +1146,7 @@ pub struct DQNHyperparameters {
/// Per-action Gaussian noise replaces dead NoisyLinear (cuBLAS bypasses Candle). /// Per-action Gaussian noise replaces dead NoisyLinear (cuBLAS bypasses Candle).
/// Breaks the dueling symmetry trap where advantage heads produce zero Q-gap. /// Breaks the dueling symmetry trap where advantage heads produce zero Q-gap.
/// 0.0 = disabled. Default: 0.1. /// 0.0 = disabled. Default: 0.1.
pub noise_sigma: f64, pub noise_sigma: f32,
} }
impl Default for DQNHyperparameters { impl Default for DQNHyperparameters {
@@ -1148,13 +1160,13 @@ impl Default for DQNHyperparameters {
impl DQNHyperparameters { impl DQNHyperparameters {
/// Initial v_min for C51 atom support (used at construction only — /// Initial v_min for C51 atom support (used at construction only —
/// per-sample IQL support replaces batch-level v_range during training). /// per-sample IQL support replaces batch-level v_range during training).
pub fn computed_v_min(&self) -> f64 { pub fn computed_v_min(&self) -> f32 {
let r = (self.reward_scale * 1.5).clamp(10.0, 50.0); let r = (self.reward_scale * 1.5).clamp(10.0, 50.0);
-r -r
} }
/// Initial v_max for C51 atom support. /// Initial v_max for C51 atom support.
pub fn computed_v_max(&self) -> f64 { pub fn computed_v_max(&self) -> f32 {
(self.reward_scale * 1.5).clamp(10.0, 50.0) (self.reward_scale * 1.5).clamp(10.0, 50.0)
} }
@@ -1164,39 +1176,45 @@ impl DQNHyperparameters {
/// ///
/// Always returns at least 1.0 so the agent can take at least one contract. /// Always returns at least 1.0 so the agent can take at least one contract.
/// `reference_price` is typically the median close price from training data. /// `reference_price` is typically the median close price from training data.
pub fn compute_max_position(&self, reference_price: f64) -> f64 { ///
/// Computed in f64 internally for the leverage × capital division, then returned as f32
/// to match the f32-native position/max_leverage/contract_multiplier kernel contract.
pub fn compute_max_position(&self, reference_price: f64) -> f32 {
if reference_price <= 0.0 || self.contract_multiplier <= 0.0 { if reference_price <= 0.0 || self.contract_multiplier <= 0.0 {
return 1.0; return 1.0;
} }
let notional_per_contract = reference_price * self.contract_multiplier; let notional_per_contract = reference_price * f64::from(self.contract_multiplier);
(self.initial_capital as f64 * self.max_leverage / notional_per_contract) (f64::from(self.initial_capital) * f64::from(self.max_leverage) / notional_per_contract)
.floor() .floor()
.max(1.0) .max(1.0) as f32
} }
/// Apply family intensity scalars to their respective parameters. /// Apply family intensity scalars to their respective parameters.
/// Call after constructing from hyperopt PSO vector. /// Call after constructing from hyperopt PSO vector.
/// Intensity 1.0 = no change. 0.5 = halved. 2.0 = doubled. /// Intensity 1.0 = no change. 0.5 = halved. 2.0 = doubled.
/// For interval/epoch params: higher intensity = more frequent (divide by intensity). /// For interval/epoch params: higher intensity = more frequent (divide by intensity).
///
/// Intensity scalars stay f64 (PSO search space is f64); each multiplicand is cast to
/// match its target field's storage type at the call site.
pub fn apply_family_scaling(&mut self) { pub fn apply_family_scaling(&mut self) {
// ── Core families ── // ── Core families ──
// Learning family: scales learning_rate, tau, weight_decay // Learning family: scales learning_rate, tau, weight_decay
let li = self.learning_intensity; let li = self.learning_intensity;
self.learning_rate *= li; self.learning_rate *= li;
self.tau *= li; self.tau *= li as f32;
self.weight_decay *= li; self.weight_decay *= li;
// Exploration family: scales entropy_coefficient, noisy_sigma_init // Exploration family: scales entropy_coefficient, noisy_sigma_init
let exi = self.exploration_intensity; let exi = self.exploration_intensity;
self.entropy_coefficient = (self.entropy_coefficient * exi).clamp(0.001, 1.0); self.entropy_coefficient = (self.entropy_coefficient * exi).clamp(0.001, 1.0);
self.noisy_sigma_init = (self.noisy_sigma_init * exi).clamp(0.01, 2.0); self.noisy_sigma_init = (self.noisy_sigma_init * exi as f32).clamp(0.01, 2.0);
// Replay family: scales buffer_size, per_alpha, per_beta_start, n_steps // Replay family: scales buffer_size, per_alpha, per_beta_start, n_steps
let rpi = self.replay_intensity; let rpi = self.replay_intensity;
self.buffer_size = (self.buffer_size as f64 * rpi).max(1000.0) as usize; self.buffer_size = (self.buffer_size as f64 * rpi).max(1000.0) as usize;
self.per_alpha = (self.per_alpha * rpi).clamp(0.0, 1.0); self.per_alpha = (self.per_alpha * rpi as f32).clamp(0.0, 1.0);
self.per_beta_start = (self.per_beta_start * rpi).clamp(0.0, 1.0); self.per_beta_start = (self.per_beta_start * rpi as f32).clamp(0.0, 1.0);
self.n_steps = (self.n_steps as f64 * rpi).clamp(1.0, 20.0) as usize; self.n_steps = (self.n_steps as f64 * rpi).clamp(1.0, 20.0) as usize;
// Architecture family: scales hidden_dim_base, dueling_hidden_dim, num_atoms // Architecture family: scales hidden_dim_base, dueling_hidden_dim, num_atoms
@@ -1211,7 +1229,7 @@ impl DQNHyperparameters {
// Risk family: scales max_leverage (not max_position_absolute directly), // Risk family: scales max_leverage (not max_position_absolute directly),
// huber_delta, minimum_profit_factor, w_dd, dd_threshold. // huber_delta, minimum_profit_factor, w_dd, dd_threshold.
// max_position_absolute is derived later via compute_max_position(reference_price). // max_position_absolute is derived later via compute_max_position(reference_price).
let rki = self.risk_intensity; let rki = self.risk_intensity as f32;
self.max_leverage *= rki; self.max_leverage *= rki;
self.huber_delta *= rki; self.huber_delta *= rki;
self.minimum_profit_factor = 1.0 + (self.minimum_profit_factor - 1.0) * rki; self.minimum_profit_factor = 1.0 + (self.minimum_profit_factor - 1.0) * rki;
@@ -1222,7 +1240,7 @@ impl DQNHyperparameters {
// Adversarial family // Adversarial family
let ai = self.adversarial_intensity; let ai = self.adversarial_intensity;
self.adversarial_dd_threshold *= ai; self.adversarial_dd_threshold *= ai as f32;
if ai > 0.01 { if ai > 0.01 {
self.adversarial_warmup_epochs = (self.adversarial_warmup_epochs as f64 / ai) as usize; self.adversarial_warmup_epochs = (self.adversarial_warmup_epochs as f64 / ai) as usize;
self.adversarial_checkpoint_interval = self.adversarial_checkpoint_interval =
@@ -1230,31 +1248,31 @@ impl DQNHyperparameters {
} }
// Regularization family // Regularization family
let ri = self.regularization_intensity; let ri = self.regularization_intensity as f32;
self.dropout_initial = (self.dropout_initial * ri).clamp(0.0, 0.5); self.dropout_initial = (self.dropout_initial * ri).clamp(0.0, 0.5);
self.spectral_norm_sigma_max *= ri; self.spectral_norm_sigma_max *= ri;
self.stochastic_depth_prob = (self.stochastic_depth_prob * ri).clamp(0.0, 0.5); self.stochastic_depth_prob = (self.stochastic_depth_prob * ri).clamp(0.0, 0.5);
self.spectral_decoupling_lambda *= ri; self.spectral_decoupling_lambda *= ri;
// Data augmentation family // Data augmentation family
let di = self.augmentation_intensity; let di = self.augmentation_intensity as f32;
self.feature_mask_fraction = (self.feature_mask_fraction * di).clamp(0.0, 0.8); self.feature_mask_fraction = (self.feature_mask_fraction * di).clamp(0.0, 0.8);
self.feature_noise_scale *= di; self.feature_noise_scale *= di;
// Loss shaping family // Loss shaping family
let li = self.loss_shaping_intensity; let ls = self.loss_shaping_intensity as f32;
self.cea_weight = (self.cea_weight * li).clamp(0.0, 2.0); self.cea_weight = (self.cea_weight * ls).clamp(0.0, 2.0);
self.micro_reward_scale = (self.micro_reward_scale * li).clamp(0.0, 0.01); self.micro_reward_scale = (self.micro_reward_scale * ls).clamp(0.0, 0.01);
// v6 legacy (zeroed defaults, no-op) // v6 legacy (zeroed defaults, no-op)
self.asymmetric_dd_weight *= li; self.asymmetric_dd_weight *= ls;
// Ensemble family // Ensemble family
let ei = self.ensemble_intensity; let ei = self.ensemble_intensity as f32;
self.ensemble_disagreement_penalty *= ei; self.ensemble_disagreement_penalty *= ei;
// Causal family // Causal family
let ci = self.causal_intensity; let ci = self.causal_intensity;
self.causal_weight *= ci; self.causal_weight *= ci as f32;
if ci > 0.01 { if ci > 0.01 {
self.causal_intervention_interval = self.causal_intervention_interval =
(self.causal_intervention_interval as f64 / ci) as usize; (self.causal_intervention_interval as f64 / ci) as usize;
@@ -1432,8 +1450,8 @@ impl DQNHyperparameters {
// Tight support: delta_z = 30/50 = 0.6 (was 9.6 at ±240). // Tight support: delta_z = 30/50 = 0.6 (was 9.6 at ±240).
// 16× better atom resolution in the active Q-value range. // 16× better atom resolution in the active Q-value range.
reward_scale: 10.0, reward_scale: 10.0,
v_min: -(10.0_f64 * 1.5).clamp(10.0, 50.0), // = -15.0 v_min: -(10.0_f32 * 1.5).clamp(10.0, 50.0), // = -15.0
v_max: (10.0_f64 * 1.5).clamp(10.0, 50.0), // = 15.0 v_max: (10.0_f32 * 1.5).clamp(10.0, 50.0), // = 15.0
// Wave 2.4: Noisy Networks (WAVE 6.4: ENABLED BY DEFAULT) // Wave 2.4: Noisy Networks (WAVE 6.4: ENABLED BY DEFAULT)
noisy_sigma_init: 0.5, // Rainbow DQN standard: 0.5 noisy_sigma_init: 0.5, // Rainbow DQN standard: 0.5

4
crates/ml/src/trainers/dqn/data_loading.rs
View File

@@ -220,8 +220,8 @@ impl DQNTrainer {
let bars = crate::features::mbp10_loader::mbp10_to_imbalance_bars( let bars = crate::features::mbp10_loader::mbp10_to_imbalance_bars(
&mbp10_path, &mbp10_path,
symbol, symbol,
self.hyperparams.imbalance_bar_threshold, f64::from(self.hyperparams.imbalance_bar_threshold),
self.hyperparams.imbalance_bar_ewma_alpha, f64::from(self.hyperparams.imbalance_bar_ewma_alpha),
).map_err(|e| anyhow::anyhow!("MBP-10 imbalance bar loading failed: {}", e))?; ).map_err(|e| anyhow::anyhow!("MBP-10 imbalance bar loading failed: {}", e))?;
info!("Loaded {} imbalance bars from MBP-10 data", bars.len()); info!("Loaded {} imbalance bars from MBP-10 data", bars.len());
bars bars

28
crates/ml/src/trainers/dqn/fused_training.rs
View File

@@ -371,7 +371,9 @@ impl FusedTrainingCtx {
branch_2_size: dqn.config.num_urgency_levels, branch_2_size: dqn.config.num_urgency_levels,
branch_3_size: 3, // urgency — fixed at 3 (4-branch refactor) branch_3_size: 3, // urgency — fixed at 3 (4-branch refactor)
batch_size, batch_size,
gamma: hyperparams.gamma as f32, // Hyperparams are now f32 native (post f64→f32 migration); pass-through without cast.
// Adam lr/eps/decay still derive from f64 hyperparams → f32 at optimizer boundary.
gamma: hyperparams.gamma,
n_steps: hyperparams.n_steps, n_steps: hyperparams.n_steps,
lr: hyperparams.learning_rate as f32, lr: hyperparams.learning_rate as f32,
beta1: 0.9, beta1: 0.9,
@@ -379,22 +381,22 @@ impl FusedTrainingCtx {
epsilon: hyperparams.adam_epsilon as f32, epsilon: hyperparams.adam_epsilon as f32,
weight_decay: hyperparams.weight_decay as f32, weight_decay: hyperparams.weight_decay as f32,
max_grad_norm: resolved_grad_norm as f32, max_grad_norm: resolved_grad_norm as f32,
spectral_norm_sigma_max: hyperparams.spectral_norm_sigma_max as f32, spectral_norm_sigma_max: hyperparams.spectral_norm_sigma_max,
spectral_decoupling_lambda: hyperparams.spectral_decoupling_lambda as f32, spectral_decoupling_lambda: hyperparams.spectral_decoupling_lambda,
iqn_lambda: hyperparams.iqn_lambda as f32, iqn_lambda: hyperparams.iqn_lambda,
iqn_num_quantiles: if hyperparams.iqn_lambda > 0.0 { hyperparams.num_quantiles } else { 0 }, iqn_num_quantiles: if hyperparams.iqn_lambda > 0.0 { hyperparams.num_quantiles } else { 0 },
iqn_embedding_dim: dqn.config.iqn_embedding_dim, iqn_embedding_dim: dqn.config.iqn_embedding_dim,
iqn_kappa: dqn.config.iqn_kappa, iqn_kappa: dqn.config.iqn_kappa,
entropy_coefficient: dqn.config.entropy_coefficient as f32, entropy_coefficient: dqn.config.entropy_coefficient as f32,
c51_warmup_epochs: hyperparams.c51_warmup_epochs, c51_warmup_epochs: hyperparams.c51_warmup_epochs,
cql_alpha: hyperparams.cql_alpha as f32, cql_alpha: hyperparams.cql_alpha as f32,
curiosity_q_penalty_lambda: hyperparams.curiosity_q_penalty_lambda as f32, curiosity_q_penalty_lambda: hyperparams.curiosity_q_penalty_lambda,
asymmetric_dd_weight: hyperparams.asymmetric_dd_weight as f32, asymmetric_dd_weight: hyperparams.asymmetric_dd_weight,
ensemble_disagreement_penalty: hyperparams.ensemble_disagreement_penalty as f32, ensemble_disagreement_penalty: hyperparams.ensemble_disagreement_penalty,
stochastic_depth_prob: hyperparams.stochastic_depth_prob as f32, stochastic_depth_prob: hyperparams.stochastic_depth_prob,
pruning_epoch: hyperparams.pruning_epoch, pruning_epoch: hyperparams.pruning_epoch,
pruning_fraction: hyperparams.pruning_fraction as f32, pruning_fraction: hyperparams.pruning_fraction,
causal_weight: hyperparams.causal_weight, causal_weight: f64::from(hyperparams.causal_weight),
causal_intervention_interval: hyperparams.causal_intervention_interval, causal_intervention_interval: hyperparams.causal_intervention_interval,
bottleneck_dim: hyperparams.bottleneck_dim, bottleneck_dim: hyperparams.bottleneck_dim,
market_dim: 42, // Always 42 base market features — OFI features bypass bottleneck via portfolio_dim market_dim: 42, // Always 42 base market features — OFI features bypass bottleneck via portfolio_dim
@@ -441,7 +443,7 @@ impl FusedTrainingCtx {
let gpu_her = if hyperparams.her_ratio > 0.0 { let gpu_her = if hyperparams.her_ratio > 0.0 {
let her_config = GpuHerConfig { let her_config = GpuHerConfig {
goal_dim: 1, // Single scalar goal (target return) goal_dim: 1, // Single scalar goal (target return)
her_ratio: hyperparams.her_ratio, her_ratio: f64::from(hyperparams.her_ratio),
strategy: parse_her_strategy(&hyperparams.her_strategy), strategy: parse_her_strategy(&hyperparams.her_strategy),
goal_threshold: 0.01, goal_threshold: 0.01,
batch_size, batch_size,
@@ -467,7 +469,7 @@ impl FusedTrainingCtx {
// Pre-compute HER source indices and reward ones on GPU (avoid per-step Vec + HtoD) // Pre-compute HER source indices and reward ones on GPU (avoid per-step Vec + HtoD)
let (her_source_indices_gpu, her_reward_ones_gpu) = if gpu_her.is_some() { let (her_source_indices_gpu, her_reward_ones_gpu) = if gpu_her.is_some() {
let her_batch_size = (batch_size as f64 * hyperparams.her_ratio) as usize; let her_batch_size = (batch_size as f64 * f64::from(hyperparams.her_ratio)) as usize;
let normal_count = batch_size.saturating_sub(her_batch_size); let normal_count = batch_size.saturating_sub(her_batch_size);
// Source indices: [normal_count, normal_count+1, ..., batch_size-1] // Source indices: [normal_count, normal_count+1, ..., batch_size-1]
let source_idx: Vec<i32> = (normal_count..batch_size).map(|i| i as i32).collect(); let source_idx: Vec<i32> = (normal_count..batch_size).map(|i| i as i32).collect();
@@ -778,7 +780,7 @@ impl FusedTrainingCtx {
cached_median: 0.0, cached_median: 0.0,
cached_iqr: 0.0, cached_iqr: 0.0,
prev_popart_var: 0.0, prev_popart_var: 0.0,
hindsight_fraction: hyperparams.hindsight_fraction, hindsight_fraction: f64::from(hyperparams.hindsight_fraction),
curriculum_enabled: hyperparams.curriculum_enabled, curriculum_enabled: hyperparams.curriculum_enabled,
}) })
} }

4
crates/ml/src/trainers/dqn/risk.rs
View File

@@ -119,10 +119,10 @@ impl DQNTrainer {
let kelly_fraction = kelly_result.unwrap_or(0.1); let kelly_fraction = kelly_result.unwrap_or(0.1);
// Apply fractional Kelly (conservative) // Apply fractional Kelly (conservative)
let fractional_kelly = kelly_fraction * self.hyperparams.kelly_fractional; let fractional_kelly = kelly_fraction * f64::from(self.hyperparams.kelly_fractional);
// Clamp to safety bounds // Clamp to safety bounds
fractional_kelly.clamp(0.01, self.hyperparams.kelly_max_fraction) fractional_kelly.clamp(0.01, f64::from(self.hyperparams.kelly_max_fraction))
} }
/// Update adaptive risk trackers with new market data /// Update adaptive risk trackers with new market data

15
crates/ml/src/trainers/dqn/trainer/action.rs
View File

@@ -26,16 +26,17 @@ impl DQNTrainer {
let action_idx = self.epsilon_greedy_action(&state_tensor).await?; let action_idx = self.epsilon_greedy_action(&state_tensor).await?;
let exposure = ExposureLevel::from_index(action_idx) let exposure = ExposureLevel::from_index(action_idx)
.map_err(|e| anyhow::anyhow!("Invalid action index {}: {}", action_idx, e))?; .map_err(|e| anyhow::anyhow!("Invalid action index {}: {}", action_idx, e))?;
Ok(self.route_action(exposure, self.hyperparams.avg_spread as f32)) Ok(self.route_action(exposure, self.hyperparams.avg_spread))
} }
pub(crate) fn route_action(&self, exposure: ExposureLevel, current_spread: f32) -> FactoredAction { pub(crate) fn route_action(&self, exposure: ExposureLevel, current_spread: f32) -> FactoredAction {
OrderRouter::route(exposure, current_spread, self.hyperparams.avg_spread as f32, self.vol_ema as f32, self.median_vol as f32) OrderRouter::route(exposure, current_spread, self.hyperparams.avg_spread, self.vol_ema as f32, self.median_vol as f32)
} }
pub(crate) fn simulate_fill(&self, action: FactoredAction, step: usize) -> (FactoredAction, FillResult) { pub(crate) fn simulate_fill(&self, action: FactoredAction, step: usize) -> (FactoredAction, FillResult) {
let normalized_vol = if self.median_vol > 0.0 { (self.vol_ema / self.median_vol) as f32 } else { 1.0 }; let normalized_vol = if self.median_vol > 0.0 { (self.vol_ema / self.median_vol) as f32 } else { 1.0 };
let spread_bps = self.hyperparams.avg_spread * 10000.0; // fill_simulator expects bps in f64 for downstream f64 math; cast once.
let spread_bps = f64::from(self.hyperparams.avg_spread) * 10000.0;
let fill_result = self.fill_simulator.simulate_fill(action.order, action.urgency, normalized_vol, spread_bps, step, action.exposure as usize); let fill_result = self.fill_simulator.simulate_fill(action.order, action.urgency, normalized_vol, spread_bps, step, action.exposure as usize);
if fill_result.filled { (action, fill_result) } else { (OrderRouter::route_default(ExposureLevel::Flat), fill_result) } if fill_result.filled { (action, fill_result) } else { (OrderRouter::route_default(ExposureLevel::Flat), fill_result) }
} }
@@ -93,7 +94,7 @@ impl DQNTrainer {
let stream = std::sync::Arc::clone(self.cuda_stream.as_ref().ok_or_else(|| anyhow::anyhow!("CUDA stream required"))?); let stream = std::sync::Arc::clone(self.cuda_stream.as_ref().ok_or_else(|| anyhow::anyhow!("CUDA stream required"))?);
let mut selector = crate::cuda_pipeline::gpu_action_selector::GpuActionSelector::new(stream, self.hyperparams.batch_size.max(batch_size).max(8192), 0xDEAD_BEEF_CAFE_u64) let mut selector = crate::cuda_pipeline::gpu_action_selector::GpuActionSelector::new(stream, self.hyperparams.batch_size.max(batch_size).max(8192), 0xDEAD_BEEF_CAFE_u64)
.map_err(|e| anyhow::anyhow!("GPU fused action selector init failed: {e}"))?; .map_err(|e| anyhow::anyhow!("GPU fused action selector init failed: {e}"))?;
selector.set_q_gap_threshold(self.hyperparams.q_gap_threshold as f32); selector.set_q_gap_threshold(self.hyperparams.q_gap_threshold);
info!("GPU action selector initialized for select_actions_batch (q_gap={:.3})", self.hyperparams.q_gap_threshold); info!("GPU action selector initialized for select_actions_batch (q_gap={:.3})", self.hyperparams.q_gap_threshold);
self.gpu_action_selector = Some(selector); self.gpu_action_selector = Some(selector);
} }
@@ -105,7 +106,7 @@ impl DQNTrainer {
if true { if true {
let agent_r = self.agent.read().await; let agent_r = self.agent.read().await;
if let Some(bonuses) = agent_r.get_count_bonuses_branched() { if let Some(bonuses) = agent_r.get_count_bonuses_branched() {
let coeff = self.hyperparams.count_bonus_coefficient.unwrap_or(0.0) as f32; let coeff = self.hyperparams.count_bonus_coefficient.unwrap_or(0.0);
if coeff > 0.0 { if coeff > 0.0 {
let (be, bo, bu) = bonuses; let (be, bo, bu) = bonuses;
let be_f32: [f32; 4] = std::array::from_fn(|i| be.get(i).copied().unwrap_or(0.0) as f32 * coeff); let be_f32: [f32; 4] = std::array::from_fn(|i| be.get(i).copied().unwrap_or(0.0) as f32 * coeff);
@@ -119,11 +120,11 @@ impl DQNTrainer {
let factored_slice = if let Some((ref q_exp, ref q_ord, ref q_urg)) = branching_q_slices { let factored_slice = if let Some((ref q_exp, ref q_ord, ref q_urg)) = branching_q_slices {
// Kernel uses single per-sample epsilon for all 3 heads; apply exposure multiplier // Kernel uses single per-sample epsilon for all 3 heads; apply exposure multiplier
let eps = epsilon * self.hyperparams.epsilon_magnitude_mult as f32; let eps = epsilon * self.hyperparams.epsilon_magnitude_mult;
selector.select_actions_branching(q_exp, q_ord, q_urg, eps, batch_size).map_err(|e| anyhow::anyhow!("GPU branching action selection failed: {e}"))? selector.select_actions_branching(q_exp, q_ord, q_urg, eps, batch_size).map_err(|e| anyhow::anyhow!("GPU branching action selection failed: {e}"))?
} else { } else {
let exposure_slice = selector.select_actions(&batch_q_values, epsilon, batch_size, 4).map_err(|e| anyhow::anyhow!("GPU fused action selection failed: {e}"))?; let exposure_slice = selector.select_actions(&batch_q_values, epsilon, batch_size, 4).map_err(|e| anyhow::anyhow!("GPU fused action selection failed: {e}"))?;
selector.route_exposure_to_factored(&exposure_slice, batch_size, self.hyperparams.avg_spread as f32, self.hyperparams.avg_spread as f32, self.vol_ema as f32, self.median_vol as f32) selector.route_exposure_to_factored(&exposure_slice, batch_size, self.hyperparams.avg_spread, self.hyperparams.avg_spread, self.vol_ema as f32, self.median_vol as f32)
.map_err(|e| anyhow::anyhow!("GPU route exposure->factored: {e}"))? .map_err(|e| anyhow::anyhow!("GPU route exposure->factored: {e}"))?
}; };
let host_actions = crate::cuda_pipeline::gpu_action_selector::GpuActionSelector::readback_actions(selector.stream(), &factored_slice, batch_size) let host_actions = crate::cuda_pipeline::gpu_action_selector::GpuActionSelector::readback_actions(selector.stream(), &factored_slice, batch_size)

78
crates/ml/src/trainers/dqn/trainer/constructor.rs
View File

@@ -152,9 +152,9 @@ impl DQNTrainer {
match detect_gpu_hardware() { match detect_gpu_hardware() {
Ok(hw) => { Ok(hw) => {
let frac = if hyperparams.replay_buffer_vram_fraction > 0.0 { let frac = if hyperparams.replay_buffer_vram_fraction > 0.0 {
hyperparams.replay_buffer_vram_fraction f64::from(hyperparams.replay_buffer_vram_fraction)
} else { } else {
0.70 // default fraction 0.70_f64 // default fraction
}; };
(hw.free_memory_mb * frac / 3.0 * 1024.0 * 1024.0) as usize (hw.free_memory_mb * frac / 3.0 * 1024.0 * 1024.0) as usize
} }
@@ -179,7 +179,7 @@ impl DQNTrainer {
let aligned_sd = (raw_sd + 7) & !7; let aligned_sd = (raw_sd + 7) & !7;
// Divide VRAM budget by 3 regime heads (each gets its own PER buffer) // Divide VRAM budget by 3 regime heads (each gets its own PER buffer)
let num_regime_heads = 3; let num_regime_heads = 3;
let per_head_fraction = hyperparams.replay_buffer_vram_fraction / num_regime_heads as f64; let per_head_fraction = f64::from(hyperparams.replay_buffer_vram_fraction) / num_regime_heads as f64;
let replay_cfg = hw.optimal_replay_config( let replay_cfg = hw.optimal_replay_config(
aligned_sd, aligned_sd,
per_head_fraction, per_head_fraction,
@@ -232,32 +232,34 @@ impl DQNTrainer {
num_actions: 4, // 4 direction levels (Short/Hold/Long/Flat) num_actions: 4, // 4 direction levels (Short/Hold/Long/Flat)
hidden_dims, hidden_dims,
learning_rate: hyperparams.learning_rate, learning_rate: hyperparams.learning_rate,
gamma: hyperparams.gamma as f32, // hyperparams fields below are f32 native (kernel-facing); DQNConfig stores some as f32
epsilon_start: hyperparams.epsilon_start as f32, // and some as f64 (ml-dqn crate not migrated). Cast once at the boundary.
epsilon_end: hyperparams.epsilon_end as f32, gamma: hyperparams.gamma,
epsilon_decay: hyperparams.epsilon_decay as f32, epsilon_start: hyperparams.epsilon_start,
epsilon_end: hyperparams.epsilon_end,
epsilon_decay: hyperparams.epsilon_decay,
replay_buffer_capacity: hyperparams.buffer_size, replay_buffer_capacity: hyperparams.buffer_size,
collapse_warmup_capacity: original_buffer_size, collapse_warmup_capacity: original_buffer_size,
batch_size: hyperparams.batch_size, batch_size: hyperparams.batch_size,
min_replay_size: hyperparams.min_replay_size.min(hyperparams.buffer_size), // Cap at buffer_size to prevent deadlock min_replay_size: hyperparams.min_replay_size.min(hyperparams.buffer_size), // Cap at buffer_size to prevent deadlock
target_update_freq: hyperparams.target_update_frequency, // Use hyperparameter instead of hardcoded 1000 target_update_freq: hyperparams.target_update_frequency, // Use hyperparameter instead of hardcoded 1000
huber_delta: hyperparams.huber_delta as f32, huber_delta: hyperparams.huber_delta,
leaky_relu_alpha: 0.01, // Standard LeakyReLU alpha (prevents dead neurons) leaky_relu_alpha: 0.01, // Standard LeakyReLU alpha (prevents dead neurons)
gradient_clip_norm: grad_norm, // Bug #4 fix: resolved once above gradient_clip_norm: grad_norm, // Bug #4 fix: resolved once above
// WAVE 16 (Agent 36): Target update configuration // WAVE 16 (Agent 36): Target update configuration
tau: hyperparams.tau, tau: f64::from(hyperparams.tau),
tau_final: hyperparams.tau * 0.5, // Anneal to 50% of base (was 10% — too frozen) tau_final: f64::from(hyperparams.tau) * 0.5, // Anneal to 50% of base (was 10% — too frozen)
tau_anneal_steps: 500_000, // was 100K tau_anneal_steps: 500_000, // was 100K
// Rainbow DQN warmup period // Rainbow DQN warmup period
warmup_steps: hyperparams.warmup_steps, warmup_steps: hyperparams.warmup_steps,
// PER configuration // PER configuration
initial_capital: hyperparams.initial_capital as f64, initial_capital: f64::from(hyperparams.initial_capital),
per_alpha: hyperparams.per_alpha, per_alpha: f64::from(hyperparams.per_alpha),
per_beta_start: hyperparams.per_beta_start, per_beta_start: f64::from(hyperparams.per_beta_start),
per_beta_max: 1.0, per_beta_max: 1.0,
per_beta_annealing_steps: hyperparams.epochs * 500, // reach beta=1.0 by ~25% of training (was 2000 -> too slow) per_beta_annealing_steps: hyperparams.epochs * 500, // reach beta=1.0 by ~25% of training (was 2000 -> too slow)
per_max_memory_bytes, per_max_memory_bytes,
@@ -270,35 +272,35 @@ impl DQNTrainer {
// Wave 2.3: Distributional RL (C51) (always enabled) // Wave 2.3: Distributional RL (C51) (always enabled)
num_atoms: hyperparams.num_atoms, // Rainbow DQN standard: 51 atoms num_atoms: hyperparams.num_atoms, // Rainbow DQN standard: 51 atoms
v_min: hyperparams.v_min as f32, // Minimum value for distribution support v_min: hyperparams.v_min, // Minimum value for distribution support
v_max: hyperparams.v_max as f32, // Maximum value for distribution support v_max: hyperparams.v_max, // Maximum value for distribution support
// Wave 2.4: Noisy Networks for Exploration (always enabled) // Wave 2.4: Noisy Networks for Exploration (always enabled)
noisy_sigma_init: hyperparams.noisy_sigma_init, // Rainbow DQN standard: 0.5 noisy_sigma_init: f64::from(hyperparams.noisy_sigma_init), // Rainbow DQN standard: 0.5
// BUG #37 FIX: Q-value clipping (prevents step-level explosions) // BUG #37 FIX: Q-value clipping (prevents step-level explosions)
q_value_clip_min: hyperparams.q_clip_min as f32, q_value_clip_min: hyperparams.q_clip_min,
q_value_clip_max: hyperparams.q_clip_max as f32, q_value_clip_max: hyperparams.q_clip_max,
// WAVE 23 P0 Fix #1: Adaptive gradient collapse threshold (from hyperparams) // WAVE 23 P0 Fix #1: Adaptive gradient collapse threshold (from hyperparams)
gradient_collapse_multiplier: hyperparams.gradient_collapse_multiplier, gradient_collapse_multiplier: f64::from(hyperparams.gradient_collapse_multiplier),
gradient_collapse_patience: hyperparams.gradient_collapse_patience, gradient_collapse_patience: hyperparams.gradient_collapse_patience,
cql_alpha: hyperparams.cql_alpha, cql_alpha: hyperparams.cql_alpha,
iqn_num_quantiles: hyperparams.num_quantiles, // Controlled by hyperopt iqn_num_quantiles: hyperparams.num_quantiles, // Controlled by hyperopt
iqn_kappa: hyperparams.qr_kappa as f32, // Controlled by hyperopt (f64->f32) iqn_kappa: hyperparams.qr_kappa, // Controlled by hyperopt
iqn_embedding_dim: 64, // Fixed (not in search space) iqn_embedding_dim: 64, // Fixed (not in search space)
iqn_lambda: hyperparams.iqn_lambda as f32, // IQN dual-head loss weight (f64->f32) iqn_lambda: hyperparams.iqn_lambda, // IQN dual-head loss weight
branch_hidden_dim: hyperparams.branch_hidden_dim, branch_hidden_dim: hyperparams.branch_hidden_dim,
cvar_alpha: hyperparams.cvar_alpha, cvar_alpha: hyperparams.cvar_alpha,
#[allow(clippy::cast_possible_truncation)] #[allow(clippy::cast_possible_truncation)]
minimum_profit_factor: hyperparams.minimum_profit_factor as f32, minimum_profit_factor: hyperparams.minimum_profit_factor,
weight_decay: hyperparams.weight_decay, weight_decay: hyperparams.weight_decay,
dropout_rate: hyperparams.dropout_initial, dropout_rate: f64::from(hyperparams.dropout_initial),
entropy_coefficient: hyperparams.entropy_coefficient, entropy_coefficient: hyperparams.entropy_coefficient,
noisy_epsilon_floor: hyperparams.noisy_epsilon_floor.unwrap_or(0.0) as f32, // C2: NoisyNet handles exploration noisy_epsilon_floor: hyperparams.noisy_epsilon_floor.unwrap_or(0.0), // C2: NoisyNet handles exploration
count_bonus_coefficient: hyperparams.count_bonus_coefficient.unwrap_or(0.0), count_bonus_coefficient: f64::from(hyperparams.count_bonus_coefficient.unwrap_or(0.0)),
curiosity_market_dim: 42, // Always 42 raw market features -- OFI features are separate curiosity_market_dim: 42, // Always 42 raw market features -- OFI features are separate
..DQNConfig::default() ..DQNConfig::default()
}; };
@@ -332,7 +334,7 @@ impl DQNTrainer {
let portfolio_tracker = PortfolioTracker::new( let portfolio_tracker = PortfolioTracker::new(
hyperparams.initial_capital, // P2-A: Configurable capital hyperparams.initial_capital, // P2-A: Configurable capital
0.0001, // 1 basis point spread (0.01%) 0.0001, // 1 basis point spread (0.01%)
hyperparams.cash_reserve_percent, // Cash reserve requirement f64::from(hyperparams.cash_reserve_percent), // Cash reserve requirement
); );
// Initialize reward function with hyperparameter-driven configuration // Initialize reward function with hyperparameter-driven configuration
@@ -341,7 +343,7 @@ impl DQNTrainer {
pnl_weight: Decimal::ONE, pnl_weight: Decimal::ONE,
risk_weight: Decimal::try_from(0.1).unwrap_or(Decimal::ZERO), risk_weight: Decimal::try_from(0.1).unwrap_or(Decimal::ZERO),
cost_weight: Decimal::ONE, // Bug #2 fix: 100% transaction cost weight (was 0.05, 20x too low) cost_weight: Decimal::ONE, // Bug #2 fix: 100% transaction cost weight (was 0.05, 20x too low)
movement_threshold: Decimal::try_from(hyperparams.movement_threshold) movement_threshold: Decimal::try_from(f64::from(hyperparams.movement_threshold))
.unwrap_or(Decimal::ZERO), .unwrap_or(Decimal::ZERO),
diversity_weight: Decimal::try_from(-0.1).unwrap_or(Decimal::ZERO), diversity_weight: Decimal::try_from(-0.1).unwrap_or(Decimal::ZERO),
circuit_breaker_config: CircuitBreakerConfig::default(), circuit_breaker_config: CircuitBreakerConfig::default(),
@@ -355,7 +357,7 @@ impl DQNTrainer {
// value here avoids a cross-crate change. Safe to drop when // value here avoids a cross-crate change. Safe to drop when
// ml-dqn's RewardConfig loses the field. // ml-dqn's RewardConfig loses the field.
dsr_eta: 0.01, dsr_eta: 0.01,
initial_capital: hyperparams.initial_capital as f64, initial_capital: f64::from(hyperparams.initial_capital),
}; };
let reward_fn = RewardFunction::new_with_debug(reward_config, debug_logging)?; let reward_fn = RewardFunction::new_with_debug(reward_config, debug_logging)?;
@@ -367,7 +369,7 @@ impl DQNTrainer {
let kelly_optimizer = { let kelly_optimizer = {
use crate::risk::kelly_optimizer::{KellyCriterionOptimizer, KellyOptimizerConfig}; use crate::risk::kelly_optimizer::{KellyCriterionOptimizer, KellyOptimizerConfig};
let kelly_config = KellyOptimizerConfig { let kelly_config = KellyOptimizerConfig {
max_fraction: hyperparams.kelly_max_fraction, max_fraction: f64::from(hyperparams.kelly_max_fraction),
min_fraction: 0.01, min_fraction: 0.01,
lookback_period: 252, lookback_period: 252,
confidence_threshold: 0.6, confidence_threshold: 0.6,
@@ -382,7 +384,7 @@ impl DQNTrainer {
}; };
// Wave 16 Portfolio Features: action masking and entropy regularization always active // Wave 16 Portfolio Features: action masking and entropy regularization always active
let max_position = hyperparams.max_position_absolute; // BLOCKER #2: Use hyperopt-tunable position limit let max_position: f64 = f64::from(hyperparams.max_position_absolute); // BLOCKER #2: Use hyperopt-tunable position limit
// Entropy regularization: SAC-style computed directly on Q-values in DQN::compute_loss_internal // Entropy regularization: SAC-style computed directly on Q-values in DQN::compute_loss_internal
{ {
@@ -454,7 +456,7 @@ impl DQNTrainer {
// WAVE 26 P1: Initialize advanced DQN features // WAVE 26 P1: Initialize advanced DQN features
// P1.3: Sharpe ratio reward component // P1.3: Sharpe ratio reward component
let sharpe_weight = hyperparams.sharpe_weight; let sharpe_weight: f64 = f64::from(hyperparams.sharpe_weight);
let sharpe_window = hyperparams.sharpe_window; let sharpe_window = hyperparams.sharpe_window;
// P1.6: Adaptive dropout scheduler (always active) // P1.6: Adaptive dropout scheduler (always active)
@@ -463,8 +465,8 @@ impl DQNTrainer {
info!("Dropout scheduler enabled (initial={}, final={}, steps={})", info!("Dropout scheduler enabled (initial={}, final={}, steps={})",
hyperparams.dropout_initial, hyperparams.dropout_final, hyperparams.dropout_anneal_steps); hyperparams.dropout_initial, hyperparams.dropout_final, hyperparams.dropout_anneal_steps);
Some(DropoutScheduler::new( Some(DropoutScheduler::new(
hyperparams.dropout_initial, f64::from(hyperparams.dropout_initial),
hyperparams.dropout_final, f64::from(hyperparams.dropout_final),
hyperparams.dropout_anneal_steps, hyperparams.dropout_anneal_steps,
)) ))
}; };
@@ -474,7 +476,7 @@ impl DQNTrainer {
use crate::dqn::gae::GAECalculator; use crate::dqn::gae::GAECalculator;
info!("GAE calculator enabled (lambda={}, gamma={})", info!("GAE calculator enabled (lambda={}, gamma={})",
hyperparams.gae_lambda, hyperparams.gamma); hyperparams.gae_lambda, hyperparams.gamma);
Some(GAECalculator::new(hyperparams.gae_lambda, hyperparams.gamma)) Some(GAECalculator::new(f64::from(hyperparams.gae_lambda), f64::from(hyperparams.gamma)))
}; };
// P1.11: Noisy network sigma scheduling -- always active // P1.11: Noisy network sigma scheduling -- always active
@@ -483,8 +485,8 @@ impl DQNTrainer {
info!("Noisy sigma scheduler enabled (initial={}, final={}, steps={})", info!("Noisy sigma scheduler enabled (initial={}, final={}, steps={})",
hyperparams.noisy_sigma_initial, hyperparams.noisy_sigma_final, hyperparams.noisy_sigma_anneal_steps); hyperparams.noisy_sigma_initial, hyperparams.noisy_sigma_final, hyperparams.noisy_sigma_anneal_steps);
Some(NoisySigmaScheduler::new( Some(NoisySigmaScheduler::new(
hyperparams.noisy_sigma_initial, f64::from(hyperparams.noisy_sigma_initial),
hyperparams.noisy_sigma_final, f64::from(hyperparams.noisy_sigma_final),
hyperparams.noisy_sigma_anneal_steps, hyperparams.noisy_sigma_anneal_steps,
)) ))
}; };
@@ -521,8 +523,8 @@ impl DQNTrainer {
// Capture values before hyperparams is moved into Self // Capture values before hyperparams is moved into Self
let initial_batch_size = hyperparams.batch_size; let initial_batch_size = hyperparams.batch_size;
let base_tau = hyperparams.tau; let base_tau: f64 = f64::from(hyperparams.tau);
let initial_epsilon = hyperparams.epsilon_start as f32; let initial_epsilon = hyperparams.epsilon_start;
// Multi-GPU: auto-detect if multiple CUDA devices are available // Multi-GPU: auto-detect if multiple CUDA devices are available
let multi_gpu = crate::cuda_pipeline::multi_gpu::MultiGpuConfig::detect() let multi_gpu = crate::cuda_pipeline::multi_gpu::MultiGpuConfig::detect()

2
crates/ml/src/trainers/dqn/trainer/metrics.rs
View File

@@ -292,7 +292,7 @@ impl DQNTrainer {
last_ts, last_ts,
&total_action_counts, &total_action_counts,
100_000.0, 100_000.0,
self.hyperparams.bars_per_day, f64::from(self.hyperparams.bars_per_day),
); );
metrics.add_metric("sharpe_ratio", financials.sharpe); metrics.add_metric("sharpe_ratio", financials.sharpe);
metrics.add_metric("max_drawdown", financials.max_drawdown); metrics.add_metric("max_drawdown", financials.max_drawdown);

12
crates/ml/src/trainers/dqn/trainer/mod.rs
View File

@@ -870,10 +870,10 @@ impl DQNTrainer {
}).collect(); }).collect();
let wf_config = GpuWalkForwardConfig { let wf_config = GpuWalkForwardConfig {
initial_train_fraction: self.hyperparams.wf_initial_train_fraction, initial_train_fraction: f64::from(self.hyperparams.wf_initial_train_fraction),
val_fraction: self.hyperparams.wf_val_fraction, val_fraction: f64::from(self.hyperparams.wf_val_fraction),
test_fraction: self.hyperparams.wf_test_fraction, test_fraction: f64::from(self.hyperparams.wf_test_fraction),
step_fraction: self.hyperparams.wf_step_fraction, step_fraction: f64::from(self.hyperparams.wf_step_fraction),
}; };
// Upload FULL dataset to GPU FIRST so we can use GPU prefix sums for fold generation // Upload FULL dataset to GPU FIRST so we can use GPU prefix sums for fold generation
@@ -970,7 +970,7 @@ impl DQNTrainer {
// Task 15: Per-fold adaptive regime replay decay // Task 15: Per-fold adaptive regime replay decay
let max_regime_pct = trending_pct.max(ranging_pct).max(volatile_pct); let max_regime_pct = trending_pct.max(ranging_pct).max(volatile_pct);
let adaptive_decay = self.hyperparams.regime_replay_decay let adaptive_decay = self.hyperparams.regime_replay_decay
* (1.0 - max_regime_pct / 100.0); * (1.0 - max_regime_pct as f32 / 100.0);
let adaptive_decay = adaptive_decay.clamp(0.05, 0.95); let adaptive_decay = adaptive_decay.clamp(0.05, 0.95);
let dominant_regime: u8 = if trending_pct >= ranging_pct && trending_pct >= volatile_pct { let dominant_regime: u8 = if trending_pct >= ranging_pct && trending_pct >= volatile_pct {
@@ -981,7 +981,7 @@ impl DQNTrainer {
1 1
}; };
self.regime_replay_decay_override = adaptive_decay as f32; self.regime_replay_decay_override = adaptive_decay;
self.fold_dominant_regime = dominant_regime; self.fold_dominant_regime = dominant_regime;
info!( info!(

24
crates/ml/src/trainers/dqn/trainer/training_loop.rs
View File

@@ -139,7 +139,7 @@ impl DQNTrainer {
features_gpu, features_gpu,
targets_gpu, targets_gpu,
num_bars, num_bars,
self.hyperparams.gamma, f64::from(self.hyperparams.gamma),
) )
.map_err(|e| anyhow::anyhow!("DT trajectory build: {e}"))?; .map_err(|e| anyhow::anyhow!("DT trajectory build: {e}"))?;
@@ -778,7 +778,7 @@ impl DQNTrainer {
); );
// Apply E2: adaptive epsilon // Apply E2: adaptive epsilon
self.hyperparams.epsilon_end = result.epsilon as f64; self.hyperparams.epsilon_end = result.epsilon;
// Apply E3: dynamic gamma (EMA blend) // Apply E3: dynamic gamma (EMA blend)
if let Some(eval_gamma) = result.gamma { if let Some(eval_gamma) = result.gamma {
@@ -863,7 +863,7 @@ impl DQNTrainer {
pub(crate) async fn log_training_config(&mut self) { pub(crate) async fn log_training_config(&mut self) {
match self.hyperparams.target_update_mode { match self.hyperparams.target_update_mode {
TargetUpdateMode::Soft => { TargetUpdateMode::Soft => {
let half_life = convergence_half_life(self.hyperparams.tau); let half_life = convergence_half_life(f64::from(self.hyperparams.tau));
info!("WAVE 16: Using soft target updates (Polyak averaging)"); info!("WAVE 16: Using soft target updates (Polyak averaging)");
info!(" Tau: {}", self.hyperparams.tau); info!(" Tau: {}", self.hyperparams.tau);
info!(" Convergence half-life: {} steps", half_life as usize); info!(" Convergence half-life: {} steps", half_life as usize);
@@ -1180,7 +1180,7 @@ impl DQNTrainer {
if self.hyperparams.expert_demo_ratio > 0.0 { if self.hyperparams.expert_demo_ratio > 0.0 {
use crate::trainers::dqn::expert_demos::ExpertDemoGenerator; use crate::trainers::dqn::expert_demos::ExpertDemoGenerator;
let effective = ExpertDemoGenerator::effective_ratio( let effective = ExpertDemoGenerator::effective_ratio(
self.hyperparams.expert_demo_ratio, f64::from(self.hyperparams.expert_demo_ratio),
self.hyperparams.expert_demo_decay_epochs, self.hyperparams.expert_demo_decay_epochs,
self.current_epoch, self.current_epoch,
) as f32; ) as f32;
@@ -1867,8 +1867,8 @@ impl DQNTrainer {
} }
// Q-value explosion guard: halt if Q-values exceed configured clip range // Q-value explosion guard: halt if Q-values exceed configured clip range
let q_clip_min = self.hyperparams.q_clip_min; let q_clip_min = f64::from(self.hyperparams.q_clip_min);
let q_clip_max = self.hyperparams.q_clip_max; let q_clip_max = f64::from(self.hyperparams.q_clip_max);
if avg_q < q_clip_min || avg_q > q_clip_max { if avg_q < q_clip_min || avg_q > q_clip_max {
return Err(anyhow::anyhow!( return Err(anyhow::anyhow!(
"Q-value explosion: avg_q={:.4} outside clip range [{}, {}]", "Q-value explosion: avg_q={:.4} outside clip range [{}, {}]",
@@ -3151,14 +3151,14 @@ impl DQNTrainer {
.take(long_window) .take(long_window)
.sum::<f64>() / long_window as f64; .sum::<f64>() / long_window as f64;
let thresh = self.hyperparams.anti_lr_sharpe_threshold; let thresh = f64::from(self.hyperparams.anti_lr_sharpe_threshold);
let base_lr = self.lr_scheduler.get_initial_lr(); let base_lr = self.lr_scheduler.get_initial_lr();
// "Good" wins ties — we prefer exploration over dampening when // "Good" wins ties — we prefer exploration over dampening when
// both triggers fire (rare but possible in a noisy window). // both triggers fire (rare but possible in a noisy window).
let anti_mult = if recent_max > thresh { let anti_mult: f64 = if recent_max > thresh {
self.hyperparams.anti_lr_good_mult f64::from(self.hyperparams.anti_lr_good_mult)
} else if sustained_mean < -thresh { } else if sustained_mean < -thresh {
self.hyperparams.anti_lr_bad_mult f64::from(self.hyperparams.anti_lr_bad_mult)
} else { } else {
1.0 1.0
}; };
@@ -3267,7 +3267,7 @@ impl DQNTrainer {
"Epoch {}/{}: val_Sharpe={:.2} (deterministic backtest on fixed validation window)", "Epoch {}/{}: val_Sharpe={:.2} (deterministic backtest on fixed validation window)",
epoch + 1, self.hyperparams.epochs, val_sharpe, epoch + 1, self.hyperparams.epochs, val_sharpe,
); );
let val_sharpe_raw = val_sharpe / (self.hyperparams.bars_per_day * 252.0).sqrt(); let val_sharpe_raw = val_sharpe / f64::from(self.hyperparams.bars_per_day * 252.0).sqrt();
info!(" val_Sharpe_raw={:.6} (un-annualized per-bar)", val_sharpe_raw); info!(" val_Sharpe_raw={:.6} (un-annualized per-bar)", val_sharpe_raw);
// Prometheus metrics // Prometheus metrics
@@ -3442,7 +3442,7 @@ impl DQNTrainer {
&trade_stats, &trade_stats,
&monitor.action_counts, &monitor.action_counts,
100_000.0, 100_000.0,
self.hyperparams.bars_per_day, f64::from(self.hyperparams.bars_per_day),
); );
training_metrics::set_epoch_financial_metrics( training_metrics::set_epoch_financial_metrics(
"dqn", "current", "dqn", "current",

151
crates/ml/src/training_profile.rs
View File

@@ -727,13 +727,13 @@ impl DqnTrainingProfile {
hp.learning_rate = v; hp.learning_rate = v;
} }
if let Some(v) = t.gamma { if let Some(v) = t.gamma {
hp.gamma = v; hp.gamma = v as f32;
// Recompute v_min/v_max (reward_scale-derived, gamma-independent) // Recompute v_min/v_max (reward_scale-derived, gamma-independent)
hp.v_min = hp.computed_v_min(); hp.v_min = hp.computed_v_min();
hp.v_max = hp.computed_v_max(); hp.v_max = hp.computed_v_max();
} }
if let Some(v) = t.tau { if let Some(v) = t.tau {
hp.tau = v; hp.tau = v as f32;
} }
if let Some(v) = t.warmup_steps { if let Some(v) = t.warmup_steps {
hp.warmup_steps = v; hp.warmup_steps = v;
@@ -748,13 +748,13 @@ impl DqnTrainingProfile {
hp.hidden_dim_base = Some(v); hp.hidden_dim_base = Some(v);
} }
if let Some(v) = t.reward_scale { if let Some(v) = t.reward_scale {
hp.reward_scale = v; hp.reward_scale = v as f32;
// Recompute v_min/v_max from the new reward_scale // Recompute v_min/v_max from the new reward_scale
hp.v_min = hp.computed_v_min(); hp.v_min = hp.computed_v_min();
hp.v_max = hp.computed_v_max(); hp.v_max = hp.computed_v_max();
} }
if let Some(v) = t.huber_delta { if let Some(v) = t.huber_delta {
hp.huber_delta = v; hp.huber_delta = v as f32;
} }
if let Some(v) = t.adam_epsilon { if let Some(v) = t.adam_epsilon {
hp.adam_epsilon = v; hp.adam_epsilon = v;
@@ -792,38 +792,38 @@ impl DqnTrainingProfile {
hp.max_bars = v; hp.max_bars = v;
} }
if let Some(v) = t.imbalance_bar_threshold { if let Some(v) = t.imbalance_bar_threshold {
hp.imbalance_bar_threshold = v; hp.imbalance_bar_threshold = v as f32;
} }
if let Some(v) = t.imbalance_bar_ewma_alpha { if let Some(v) = t.imbalance_bar_ewma_alpha {
hp.imbalance_bar_ewma_alpha = v; hp.imbalance_bar_ewma_alpha = v as f32;
} }
} }
// [exploration] // [exploration]
if let Some(ref e) = self.exploration { if let Some(ref e) = self.exploration {
if let Some(v) = e.epsilon_start { if let Some(v) = e.epsilon_start {
hp.epsilon_start = v; hp.epsilon_start = v as f32;
} }
if let Some(v) = e.epsilon_end { if let Some(v) = e.epsilon_end {
hp.epsilon_end = v; hp.epsilon_end = v as f32;
} }
if let Some(v) = e.epsilon_decay { if let Some(v) = e.epsilon_decay {
hp.epsilon_decay = v; hp.epsilon_decay = v as f32;
} }
if let Some(v) = e.noisy_sigma_init { if let Some(v) = e.noisy_sigma_init {
hp.noisy_sigma_init = v; hp.noisy_sigma_init = v as f32;
} }
if let Some(v) = e.entropy_coefficient { if let Some(v) = e.entropy_coefficient {
hp.entropy_coefficient = v; hp.entropy_coefficient = v;
} }
if let Some(v) = e.count_bonus_coefficient { if let Some(v) = e.count_bonus_coefficient {
hp.count_bonus_coefficient = Some(v); hp.count_bonus_coefficient = Some(v as f32);
} }
if let Some(v) = e.q_gap_threshold { if let Some(v) = e.q_gap_threshold {
hp.q_gap_threshold = v; hp.q_gap_threshold = v as f32;
} }
if let Some(v) = e.noise_sigma { if let Some(v) = e.noise_sigma {
hp.noise_sigma = v; hp.noise_sigma = v as f32;
} }
} }
@@ -836,13 +836,13 @@ impl DqnTrainingProfile {
hp.min_replay_size = v; hp.min_replay_size = v;
} }
if let Some(v) = rb.per_alpha { if let Some(v) = rb.per_alpha {
hp.per_alpha = v; hp.per_alpha = v as f32;
} }
if let Some(v) = rb.per_beta_start { if let Some(v) = rb.per_beta_start {
hp.per_beta_start = v; hp.per_beta_start = v as f32;
} }
if let Some(v) = rb.regime_replay_decay { if let Some(v) = rb.regime_replay_decay {
hp.regime_replay_decay = v; hp.regime_replay_decay = v as f32;
} }
} }
@@ -852,10 +852,10 @@ impl DqnTrainingProfile {
hp.num_atoms = v; hp.num_atoms = v;
} }
if let Some(v) = d.v_min { if let Some(v) = d.v_min {
hp.v_min = v; hp.v_min = v as f32;
} }
if let Some(v) = d.v_max { if let Some(v) = d.v_max {
hp.v_max = v; hp.v_max = v as f32;
} }
} }
@@ -866,7 +866,7 @@ impl DqnTrainingProfile {
// [advanced] // [advanced]
if let Some(ref a) = self.advanced { if let Some(ref a) = self.advanced {
if let Some(v) = a.noisy_sigma_init { if let Some(v) = a.noisy_sigma_init {
hp.noisy_sigma_init = v; hp.noisy_sigma_init = v as f32;
} }
if let Some(v) = a.cql_alpha { if let Some(v) = a.cql_alpha {
hp.cql_alpha = v; hp.cql_alpha = v;
@@ -878,7 +878,7 @@ impl DqnTrainingProfile {
hp.n_steps = v; hp.n_steps = v;
} }
if let Some(v) = a.tau { if let Some(v) = a.tau {
hp.tau = v; hp.tau = v as f32;
} }
if let Some(v) = a.c51_warmup_epochs { if let Some(v) = a.c51_warmup_epochs {
hp.c51_warmup_epochs = v; hp.c51_warmup_epochs = v;
@@ -887,48 +887,48 @@ impl DqnTrainingProfile {
hp.c51_alpha_max = v; hp.c51_alpha_max = v;
} }
if let Some(v) = a.her_ratio { if let Some(v) = a.her_ratio {
hp.her_ratio = v; hp.her_ratio = v as f32;
} }
if let Some(v) = a.iqn_lambda { if let Some(v) = a.iqn_lambda {
hp.iqn_lambda = v; hp.iqn_lambda = v as f32;
} }
if let Some(v) = a.spectral_norm_sigma_max { if let Some(v) = a.spectral_norm_sigma_max {
hp.spectral_norm_sigma_max = v; hp.spectral_norm_sigma_max = v as f32;
} }
if let Some(v) = a.gradient_clip_norm { if let Some(v) = a.gradient_clip_norm {
hp.gradient_clip_norm = Some(v); hp.gradient_clip_norm = Some(v);
} }
if let Some(v) = a.curiosity_weight { if let Some(v) = a.curiosity_weight {
hp.curiosity_weight = v; hp.curiosity_weight = v as f32;
} }
} }
// [generalization] // [generalization]
if let Some(ref g) = self.generalization { if let Some(ref g) = self.generalization {
if let Some(v) = g.shrink_perturb_interval { hp.shrink_perturb_interval = v; } if let Some(v) = g.shrink_perturb_interval { hp.shrink_perturb_interval = v; }
if let Some(v) = g.shrink_perturb_alpha { hp.shrink_perturb_alpha = v; } if let Some(v) = g.shrink_perturb_alpha { hp.shrink_perturb_alpha = v as f32; }
if let Some(v) = g.shrink_perturb_sigma { hp.shrink_perturb_sigma = v; } if let Some(v) = g.shrink_perturb_sigma { hp.shrink_perturb_sigma = v as f32; }
if let Some(v) = g.adversarial_dd_threshold { hp.adversarial_dd_threshold = v; } if let Some(v) = g.adversarial_dd_threshold { hp.adversarial_dd_threshold = v as f32; }
if let Some(v) = g.adversarial_epochs_trigger { hp.adversarial_epochs_trigger = v; } if let Some(v) = g.adversarial_epochs_trigger { hp.adversarial_epochs_trigger = v; }
if let Some(v) = g.anti_lr_good_mult { hp.anti_lr_good_mult = v; } if let Some(v) = g.anti_lr_good_mult { hp.anti_lr_good_mult = v as f32; }
if let Some(v) = g.anti_lr_bad_mult { hp.anti_lr_bad_mult = v; } if let Some(v) = g.anti_lr_bad_mult { hp.anti_lr_bad_mult = v as f32; }
if let Some(v) = g.anti_lr_sharpe_threshold { hp.anti_lr_sharpe_threshold = v; } if let Some(v) = g.anti_lr_sharpe_threshold { hp.anti_lr_sharpe_threshold = v as f32; }
if let Some(v) = g.feature_mask_fraction { hp.feature_mask_fraction = v; } if let Some(v) = g.feature_mask_fraction { hp.feature_mask_fraction = v as f32; }
if let Some(v) = g.feature_noise_scale { hp.feature_noise_scale = v; } if let Some(v) = g.feature_noise_scale { hp.feature_noise_scale = v as f32; }
if let Some(v) = g.pruning_epoch { hp.pruning_epoch = v; } if let Some(v) = g.pruning_epoch { hp.pruning_epoch = v; }
if let Some(v) = g.pruning_fraction { hp.pruning_fraction = v; } if let Some(v) = g.pruning_fraction { hp.pruning_fraction = v as f32; }
if let Some(v) = g.causal_weight { hp.causal_weight = v; } if let Some(v) = g.causal_weight { hp.causal_weight = v as f32; }
if let Some(v) = g.causal_intervention_interval { hp.causal_intervention_interval = v; } if let Some(v) = g.causal_intervention_interval { hp.causal_intervention_interval = v; }
if let Some(v) = g.adversarial_warmup_epochs { hp.adversarial_warmup_epochs = v; } if let Some(v) = g.adversarial_warmup_epochs { hp.adversarial_warmup_epochs = v; }
if let Some(v) = g.adversarial_checkpoint_interval { hp.adversarial_checkpoint_interval = v; } if let Some(v) = g.adversarial_checkpoint_interval { hp.adversarial_checkpoint_interval = v; }
if let Some(v) = g.bottleneck_dim { hp.bottleneck_dim = v; } if let Some(v) = g.bottleneck_dim { hp.bottleneck_dim = v; }
if let Some(v) = g.stochastic_depth_prob { hp.stochastic_depth_prob = v; } if let Some(v) = g.stochastic_depth_prob { hp.stochastic_depth_prob = v as f32; }
if let Some(v) = g.asymmetric_dd_weight { hp.asymmetric_dd_weight = v; } if let Some(v) = g.asymmetric_dd_weight { hp.asymmetric_dd_weight = v as f32; }
if let Some(v) = g.time_reversal_mod { hp.time_reversal_mod = v; } if let Some(v) = g.time_reversal_mod { hp.time_reversal_mod = v; }
if let Some(v) = g.regret_blend { hp.regret_blend = v; } if let Some(v) = g.regret_blend { hp.regret_blend = v as f32; }
if let Some(v) = g.trade_clustering_penalty { hp.trade_clustering_penalty = v; } if let Some(v) = g.trade_clustering_penalty { hp.trade_clustering_penalty = v as f32; }
if let Some(v) = g.ensemble_disagreement_penalty { hp.ensemble_disagreement_penalty = v; } if let Some(v) = g.ensemble_disagreement_penalty { hp.ensemble_disagreement_penalty = v as f32; }
if let Some(v) = g.mixup_alpha { hp.mixup_alpha = v; } if let Some(v) = g.mixup_alpha { hp.mixup_alpha = v as f32; }
// Core family intensity scalars // Core family intensity scalars
if let Some(v) = g.learning_intensity { hp.learning_intensity = v; } if let Some(v) = g.learning_intensity { hp.learning_intensity = v; }
if let Some(v) = g.exploration_intensity { hp.exploration_intensity = v; } if let Some(v) = g.exploration_intensity { hp.exploration_intensity = v; }
@@ -947,29 +947,29 @@ impl DqnTrainingProfile {
// [risk] // [risk]
if let Some(ref r) = self.risk { if let Some(ref r) = self.risk {
if let Some(v) = r.kelly_fractional { if let Some(v) = r.kelly_fractional {
hp.kelly_fractional = v; hp.kelly_fractional = v as f32;
} }
if let Some(v) = r.kelly_max_fraction { if let Some(v) = r.kelly_max_fraction {
hp.kelly_max_fraction = v; hp.kelly_max_fraction = v as f32;
} }
if let Some(v) = r.max_position_absolute { if let Some(v) = r.max_position_absolute {
hp.max_position_absolute = v; hp.max_position_absolute = v as f32;
} }
// max_position is an alias for max_position_absolute // max_position is an alias for max_position_absolute
if let Some(v) = r.max_position { if let Some(v) = r.max_position {
hp.max_position_absolute = v; hp.max_position_absolute = v as f32;
} }
if let Some(v) = r.max_leverage { if let Some(v) = r.max_leverage {
hp.max_leverage = v; hp.max_leverage = v as f32;
} }
if let Some(v) = r.q_clip_min { if let Some(v) = r.q_clip_min {
hp.q_clip_min = v; hp.q_clip_min = v as f32;
} }
if let Some(v) = r.q_clip_max { if let Some(v) = r.q_clip_max {
hp.q_clip_max = v; hp.q_clip_max = v as f32;
} }
if let Some(v) = r.loss_aversion { if let Some(v) = r.loss_aversion {
hp.loss_aversion = v; hp.loss_aversion = v as f32;
} }
} }
@@ -1005,33 +1005,33 @@ impl DqnTrainingProfile {
hp.initial_capital = v as f32; hp.initial_capital = v as f32;
} }
if let Some(v) = ex.tx_cost_multiplier { if let Some(v) = ex.tx_cost_multiplier {
hp.transaction_cost_multiplier = v; hp.transaction_cost_multiplier = v as f32;
} }
if let Some(v) = ex.holding_cost_rate { if let Some(v) = ex.holding_cost_rate {
hp.holding_cost_rate = v; hp.holding_cost_rate = v as f32;
} }
if let Some(v) = ex.churn_threshold_bars { if let Some(v) = ex.churn_threshold_bars {
hp.churn_threshold_bars = v; hp.churn_threshold_bars = v;
} }
if let Some(v) = ex.churn_penalty_scale { if let Some(v) = ex.churn_penalty_scale {
hp.churn_penalty_scale = v; hp.churn_penalty_scale = v as f32;
} }
// [experience.fill_simulation] // [experience.fill_simulation]
if let Some(ref fs) = ex.fill_simulation { if let Some(ref fs) = ex.fill_simulation {
if let Some(v) = fs.ioc_fill_prob { if let Some(v) = fs.ioc_fill_prob {
hp.fill_ioc_fill_prob = v; hp.fill_ioc_fill_prob = v as f32;
} }
if let Some(v) = fs.limit_fill_min { if let Some(v) = fs.limit_fill_min {
hp.fill_limit_fill_min = v; hp.fill_limit_fill_min = v as f32;
} }
if let Some(v) = fs.limit_fill_max { if let Some(v) = fs.limit_fill_max {
hp.fill_limit_fill_max = v; hp.fill_limit_fill_max = v as f32;
} }
if let Some(v) = fs.spread_cost_frac { if let Some(v) = fs.spread_cost_frac {
hp.fill_spread_cost_frac = v; hp.fill_spread_cost_frac = v as f32;
} }
if let Some(v) = fs.spread_capture_frac { if let Some(v) = fs.spread_capture_frac {
hp.fill_spread_capture_frac = v; hp.fill_spread_capture_frac = v as f32;
} }
} }
} }
@@ -1040,28 +1040,28 @@ impl DqnTrainingProfile {
// GPU composite reward weights — map directly to DQNHyperparameters // GPU composite reward weights — map directly to DQNHyperparameters
if let Some(ref rw) = self.reward { if let Some(ref rw) = self.reward {
if let Some(v) = rw.w_pnl { if let Some(v) = rw.w_pnl {
hp.w_pnl = v; hp.w_pnl = v as f32;
} }
if let Some(v) = rw.w_dd { if let Some(v) = rw.w_dd {
hp.w_dd = v; hp.w_dd = v as f32;
} }
if let Some(v) = rw.w_idle { if let Some(v) = rw.w_idle {
hp.w_idle = v; hp.w_idle = v as f32;
} }
if let Some(v) = rw.dd_threshold { if let Some(v) = rw.dd_threshold {
hp.dd_threshold = v; hp.dd_threshold = v as f32;
} }
if let Some(v) = rw.loss_aversion { if let Some(v) = rw.loss_aversion {
hp.loss_aversion = v; hp.loss_aversion = v as f32;
} }
if let Some(v) = rw.time_decay_rate { if let Some(v) = rw.time_decay_rate {
hp.time_decay_rate = v; hp.time_decay_rate = v as f32;
} }
if let Some(v) = rw.q_gap_threshold { if let Some(v) = rw.q_gap_threshold {
hp.q_gap_threshold = v; hp.q_gap_threshold = v as f32;
} }
if let Some(v) = rw.reward_scale { if let Some(v) = rw.reward_scale {
hp.reward_scale = v; hp.reward_scale = v as f32;
hp.v_min = hp.computed_v_min(); hp.v_min = hp.computed_v_min();
hp.v_max = hp.computed_v_max(); hp.v_max = hp.computed_v_max();
} }
@@ -1072,34 +1072,34 @@ impl DqnTrainingProfile {
hp.popart_robust = v; hp.popart_robust = v;
} }
if let Some(v) = rw.micro_reward_scale { if let Some(v) = rw.micro_reward_scale {
hp.micro_reward_scale = v; hp.micro_reward_scale = v as f32;
} }
if let Some(v) = rw.td_lambda { if let Some(v) = rw.td_lambda {
hp.td_lambda = v; hp.td_lambda = v as f32;
} }
if let Some(v) = rw.max_trace_length { if let Some(v) = rw.max_trace_length {
hp.max_trace_length = v; hp.max_trace_length = v;
} }
if let Some(v) = rw.hindsight_fraction { if let Some(v) = rw.hindsight_fraction {
hp.hindsight_fraction = v; hp.hindsight_fraction = v as f32;
} }
if let Some(v) = rw.hindsight_lookahead { if let Some(v) = rw.hindsight_lookahead {
hp.hindsight_lookahead = v; hp.hindsight_lookahead = v;
} }
if let Some(v) = rw.price_confirm_weight { if let Some(v) = rw.price_confirm_weight {
hp.price_confirm_weight = v; hp.price_confirm_weight = v as f32;
} }
if let Some(v) = rw.book_aggression_weight { if let Some(v) = rw.book_aggression_weight {
hp.book_aggression_weight = v; hp.book_aggression_weight = v as f32;
} }
if let Some(v) = rw.hold_quality_weight { if let Some(v) = rw.hold_quality_weight {
hp.hold_quality_weight = v; hp.hold_quality_weight = v as f32;
} }
if let Some(v) = rw.micro_reward_temp { if let Some(v) = rw.micro_reward_temp {
hp.micro_reward_temp = v; hp.micro_reward_temp = v as f32;
} }
if let Some(v) = rw.holding_cost_rate { if let Some(v) = rw.holding_cost_rate {
hp.holding_cost_rate = v; hp.holding_cost_rate = v as f32;
} }
} }
@@ -1134,7 +1134,8 @@ mod tests {
#[test] #[test]
fn test_sparse_profile_preserves_defaults() { fn test_sparse_profile_preserves_defaults() {
// dqn-smoketest has [distributional] with num_atoms but no v_min/v_max // dqn-smoketest has [distributional] with num_atoms but no v_min/v_max.
// Section-side TOML values stay f64 (ingest boundary); conversion to f32 happens in apply_to.
let p = DqnTrainingProfile::load("dqn-smoketest"); let p = DqnTrainingProfile::load("dqn-smoketest");
let d = p.distributional.expect("smoketest profile should have [distributional]"); let d = p.distributional.expect("smoketest profile should have [distributional]");
assert_eq!(d.num_atoms, Some(52)); assert_eq!(d.num_atoms, Some(52));
@@ -1286,7 +1287,7 @@ mod tests {
let hp = crate::trainers::dqn::DQNHyperparameters::conservative(); let hp = crate::trainers::dqn::DQNHyperparameters::conservative();
// Verify v_min/v_max are computed from reward_scale // Verify v_min/v_max are computed from reward_scale
let expected_v_range = (hp.reward_scale * 1.5).clamp(10.0, 50.0); let expected_v_range: f32 = (hp.reward_scale * 1.5).clamp(10.0, 50.0);
assert!( assert!(
(hp.v_min - (-expected_v_range)).abs() < 0.01, (hp.v_min - (-expected_v_range)).abs() < 0.01,
"v_min should be computed from reward_scale, got {} expected {}", "v_min should be computed from reward_scale, got {} expected {}",
@@ -1502,7 +1503,7 @@ mod tests {
hp.v_max = hp.computed_v_max(); hp.v_max = hp.computed_v_max();
// reward_scale=20 × 1.5 = 30, clamped to [10, 50] → ±30 // reward_scale=20 × 1.5 = 30, clamped to [10, 50] → ±30
let expected_v_range = (20.0_f64 * 1.5).clamp(10.0, 50.0); let expected_v_range: f32 = (20.0_f32 * 1.5).clamp(10.0, 50.0);
assert!( assert!(
(hp.v_max - expected_v_range).abs() < 0.01, (hp.v_max - expected_v_range).abs() < 0.01,
"v_max should be recomputed from reward_scale: got {} expected {}", "v_max should be recomputed from reward_scale: got {} expected {}",

5
crates/ml/tests/dqn_early_stopping_termination_test.rs
View File

@@ -346,8 +346,9 @@ fn test_early_stopping_config_fields_exist() {
"gradient_collapse_patience should default to 5" "gradient_collapse_patience should default to 5"
); );
// Verify adaptive threshold calculation // Verify adaptive threshold calculation.
let threshold = hyperparams.learning_rate * hyperparams.gradient_collapse_multiplier; // gradient_collapse_multiplier is f32 (kernel-facing); learning_rate stays f64 for precision.
let threshold = hyperparams.learning_rate * f64::from(hyperparams.gradient_collapse_multiplier);
assert!( assert!(
threshold > 0.0, threshold > 0.0,
"Adaptive threshold should be positive: {}", "Adaptive threshold should be positive: {}",

7
crates/ml/tests/dqn_hyperparams_kelly_fields_test.rs
View File

@@ -165,9 +165,10 @@ mod dqn_hyperparams_kelly_tests {
// This test verifies that all Kelly fields are pub (accessible to hyperopt adapter) // This test verifies that all Kelly fields are pub (accessible to hyperopt adapter)
let hyperparams = DQNHyperparameters::conservative(); let hyperparams = DQNHyperparameters::conservative();
// If these compile, the fields are public // If these compile, the fields are public.
let _: f64 = hyperparams.kelly_fractional; // Post f64→f32 migration: risk/kelly scalars now f32 (kernel-facing).
let _: f64 = hyperparams.kelly_max_fraction; let _: f32 = hyperparams.kelly_fractional;
let _: f32 = hyperparams.kelly_max_fraction;
let _: usize = hyperparams.kelly_min_trades; let _: usize = hyperparams.kelly_min_trades;
let _: usize = hyperparams.volatility_window; let _: usize = hyperparams.volatility_window;
} }