H100 has the compute budget — 200 epochs with cosine decay gives the
model full room to learn through C51 transition and converge.
Early stopping can't fire before epoch 100.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Added lr_decay_type (0=constant, 1=linear, 2=cosine) and lr_min to
the TOML training profile system. Wired through apply_to() with
total_steps derived from epochs.
- dqn-localdev.toml: lr_decay_type=2 (cosine 1e-4→1e-5 over 200 epochs),
min_epochs_before_stopping=80 (was 50, gives model more room to recover
after C51 transition).
- Smoketest config unchanged (lr_decay_type not set → default Constant).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. Localdev config: gpu_n_episodes 32→64, gpu_timesteps 100→200
(12,800 exp/epoch, 4x more → stable Sharpe estimation)
2. Localdev config: buffer_size 10K→50K, min_replay_size 100→500
(fills over ~4 epochs instead of <1, reduces overfitting)
3. Hyperopt: added CampaignConfig::dqn_localdev() (10 trials × 20 epochs),
updated test_local_hyperopt to use it with env var overrides
(FOXHUNT_HYPEROPT_TRIALS, FOXHUNT_HYPEROPT_EPOCHS)
4. Q-value range: was [X,X] every epoch because:
- train_step.rs Q-stats code was DEAD (training loop uses
fused.run_full_step() directly, not self.train_step())
- Only sampled 10 experiences for Q-stats (tiny batch → tight range)
Fix: reduce_current_q_stats() reads the training batch's q_out_buf
directly (64 samples, no extra forward pass), wired into the training
loop's inner step. q_stats_kernel.cu converted to f32 arithmetic.
Now shows real variation: [-5.25, 5.03] instead of [2.80, 2.80]
Removed dead Q-estimation code from train_step.rs (was never reached
in the fused GPU training path).
11/11 smoke tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Hyperopt adapter now sets max_training_steps_per_epoch:
RTX 3050 (≤40GB) = 200 steps, H100 (≥40GB) = 2000 steps.
Without this, each trial trained the full dataset (2917 steps/epoch)
making hyperopt 11x slower than necessary on local GPU.
- adam_epsilon default 1e-3→1e-8 everywhere (conservative(), DQNConfig).
The old 1e-3 was a BF16 workaround (bf16(1e-8)=0 → div-by-zero).
Adam is now f32, so standard 1e-8 is correct.
- Early stopping enabled in dqn-localdev.toml (patience=20).
Hyperopt: 2 trials × 5 epochs in 132s (was ~20min). Zero NaN.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- acc_buf bf16→f32: epoch gradient/loss accumulators were bf16, causing
avg_grad to be quantized to 0.895 every epoch. Now f32, shows real
variation (0.776→0.783→0.788).
- EPOCH_DIAG warn!→info!: diagnostic logging, not a warning condition.
- Removed legacy training_guard_check + training_guard_accumulate kernels
(dead code, replaced by fused training_guard_check_and_accumulate).
- Added dqn-localdev.toml: 200-epoch RTX 3050 profile for extended runs.
200-epoch run completed successfully:
Best Sharpe: +6.43 at epoch 107
Final: Sharpe=+2.05, PF=1.35, Return=+1052%, 0 NaN
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three root causes of sporadic NaN during training:
1. --use_fast_math (nvcc) breaks IEEE 754 NaN semantics: fmaxf(NaN,x)
returns NaN instead of x, isnan()/isinf() compile to false.
Replaced with --ftz=true --fmad=true --prec-div=true --prec-sqrt=true
across all 4 build.rs (ml, ml-dqn, ml-ppo, ml-core).
2. Cross-stream race: replay buffer wrote batch data on the device's
original stream while the trainer read it on a forked stream.
Fixed by passing the forked stream to the DQN agent via agent_device,
so all GPU components share a single CUDA stream (zero sync overhead).
3. Rewards/dones stored as bf16 in replay buffer caused done=0xFFFF NaN.
Converted entire rewards/dones pipeline to f32: experience collector,
replay buffer storage, nstep kernel, loss/grad kernels.
Also:
- Removed fast_isnan/fast_isinf/fast_isfinite wrappers — standard
isnan/isinf/isfinite work correctly without --use_fast_math
- Updated dqn-smoketest.toml: lr=1e-4, epsilon=1e-8 (f32 Adam values)
- Removed debug printfs from gather kernels
- Added curiosity_weight to training profile system
- Cleaned up smoke_params() inline overrides
11/11 smoke tests pass, 5/5 stress runs of 50-epoch test pass,
359/359 ml-dqn + 895/895 ml unit tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause #1: Adam epsilon=1e-8 rounds to 0 in BF16 → sqrt(v_hat)+0 = sqrt(v_hat) → div-by-zero when v_hat≈0. Fix: adam_epsilon configurable, default 1e-3.
Root cause #2 (partial): compute_q_values produces NaN even with sync. NOT a race — the graph_adam's Adam kernel writes NaN to params in async mode but works in CUDA_LAUNCH_BLOCKING=1 mode. The Adam kernel has no shared mem / atomics / warps — it's per-element. The ONLY shared read is grad_norm_sq[0]. Investigation continues.
Added: adam_epsilon to DQNConfig, DQNHyperparameters, GpuDqnTrainConfig, dqn-smoketest.toml, training_profile.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add min_hold_bars to DQNHyperparameters (usize, default 5), ExperienceSection
in training_profile, and both TOML configs (smoketest=3, production=5).
Wired through apply_to() so TOML overrides land correctly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add 7 new fields to DQNHyperparameters (fill_ioc_fill_prob,
fill_limit_fill_min, fill_limit_fill_max, fill_spread_cost_frac,
fill_spread_capture_frac, q_clip_min, q_clip_max) and wire them
through training_profile.rs into ExperienceCollectorConfig construction
in training_loop.rs. Previously these 7 values were hardcoded at the
construction site; now they flow from TOML [experience.fill_simulation]
and [risk] sections. Default values match the prior hardcoded constants
so existing behavior is unchanged.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove use_double_dqn, use_dueling, use_per, use_branching,
use_distributional, use_noisy_nets, use_huber_loss, and use_cql
from DQNConfig, DQNHyperparameters, and DqnParams structs.
These features are always enabled (Rainbow DQN standard). The boolean
flags were dead code — every constructor set them to true, and the
only code paths that set them to false were in tests that disabled
features for simplicity. With the fields removed, the features are
unconditionally active, eliminating ~490 lines of dead configuration.
Key changes:
- Struct field declarations removed from 3 core config structs
- Conditional branches (if use_X { ... } else { ... }) simplified:
dueling/branching/PER network creation is now unconditional
- Checkpoint metadata hardcodes "true" for backward compatibility
- Hyperopt search space index 11 (use_branching) fixed at 1.0
- TOML/YAML config files cleaned of removed fields
- Tests that toggled these flags updated or rewritten
45 files changed, -487 net lines. Zero new test failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- DSR warm-up: skip first 50 steps when EMA has insufficient history
- Phase Fast num_atoms: 51 → 101 (H100 can afford finer resolution,
1.19 per atom vs 2.35 — critical for distinguishing Q-values)
- Argo template: clear stale feature cache before hyperopt (ensures
fresh computation with VPIN/trades enrichment)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ROOT CAUSE: 5 interlocking bugs made learning impossible:
1. DSR denominator floor 1e-12 produced values in millions → drowned all signal
2. Global [-1,+1] clamp destroyed Bellman equation signal (can't distinguish
catastrophic loss from mild loss)
3. v_range=20 exactly equals V_max for gamma=0.95 → Bellman target pins at
ceiling → Q-values saturate → Q-gap collapses to 0.0000
4. num_atoms=11 over 40-unit range = 4.0 per atom (C51 paper min is 51)
5. 6/7 reward components were penalties → mean_reward=-0.311 regardless of action
FIXES:
- DSR denominator floor: 1e-12 → 0.01 (prevents million-scale spikes)
- Each component individually clamped BEFORE weighting (DSR to [-1,+1],
z-score to [-3,+3], drawdown to [0,1], time decay to [0,0.3])
- Removed global [-1,+1] clamp (no longer needed with bounded components)
- profit_take_bonus: 0.1 → 0.01 (was 100x too large, caused reward hacking)
- Removed confidence scaling (positive feedback loop destabilized learning)
- Removed regime scaling (non-stationary reward confused the model)
- Dynamic v_range from gamma: v_range = 2.5/(1-gamma)*1.2 (always covers Q range)
- num_atoms minimum: 11 → 51 (C51 paper standard)
- gamma default: 0.99 → 0.95
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Q-gap was 0.0 (disabled) meaning the model traded on every bar regardless
of conviction. With 0.1, the model must have Q(best) - Q(flat) > 0.1
before entering a position. Local test showed 21% fewer trades.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add q_gap_threshold to action selection kernel: when greedy Q(best) - Q(flat)
< threshold, default to flat. Teaches model to trade only with conviction.
39D search space (was 38D). Default 0.0 (disabled), hyperopt range [0.0, 0.5].
Remove use_branching parameter from experience_action_select — GPU pipeline
always uses branching DQN. Flat mode was dead code.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
6% drawdown tolerance too generous for HFT. Tightened search range
to 0.5%-3%, default 1%. Forces aggressive loss cutting.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add 7 composite reward fields to DQNHyperparameters: w_dsr, w_pnl,
w_dd, w_idle, dd_threshold, loss_aversion, time_decay_rate.
Add RewardSection to training_profile.rs with Option<f64> fields and
apply_to() mapping. Add [reward] section to all 3 DQN TOML profiles
(production, smoketest, hyperopt) with identical defaults.
Remove hold_reward from ExperienceSection (replaced by w_idle).
Add 7 reward search bounds to SearchSpaceSection and bound() match.
Add 7 reward phase_fast defaults to PhaseFastSection.
hold_penalty kept as deprecated field for hyperopt adapter compat
(Task 4 will clean it up).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Smoke test loads all hyperparams from TOML profile instead of hardcoding.
TOML: hidden_dim=64, batch=64, lr=0.0003 (stable on RTX 3050 + H100).
Restored check_err() drain in device.rs — required to clear stale CUDA
errors from primary context reuse between tests.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Production defaults are sufficient for hyperopt exploration. Larger networks
can be tested in a separate phase with the best hyperparams found.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add SearchSpaceSection, PsoSection, HyperoptProfile structs to
training_profile.rs. All 31 PSO search bounds now configurable in
config/training/dqn-hyperopt.toml — no code changes needed to
adjust search ranges.
HyperoptProfile::bound("field", default) returns the TOML value
or falls back to the hardcoded default. Adapter wiring is next step.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
