dqn-production.toml hard-coded three H100-tuned values that shadow
GpuProfile auto-detection: batch_size=16384, buffer_size=500K,
gpu_n_episodes=4096. After the L40S-default flip lands (prior commit
8b8bb1af7), workflow `train-ft8ph` deterministically OOMed at fold 0/1/2
with `build_next_states_f32` 4 GiB alloc — because the H100-sized
hyperparams.batch_size + buffer_size + gpu_n_episodes ate ~38 GB of the
L40S's 46 GB usable VRAM before the rollout step.
DqnTrainingProfile.apply_to() runs AFTER train_baseline_rl.rs populates
hyperparams from GpuProfile, so the production TOML always wins. All
three fields are `Option<...>` in the TOML schema — removing the lines
turns apply_to into a no-op for them, and the GpuProfile-detected values
flow through:
field | L40S | H100 | (was forced)
batch_size | 4096 | 8192 | 16384
buffer_size | 300K | 500K | 500K
gpu_n_episodes | 2048 | 4096 | 4096
Two pinned assertions in training_profile.rs::tests checked the old
contract `hp.batch_size == 16384`. Rewritten to assert
`hp.batch_size == baseline_batch_size` — locks the new contract that
VRAM-tuned values stay GpuProfile-sourced.
Lib suite 1016/0 green. Audit doc updated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
At threshold=0.5 the sampler produced 209M bars from 209M trade ticks (1:1
ratio, essentially per-tick) on workflow f5wnd today. Feature extraction hit
54Gi/56Gi memory limit — near OOM. The default was wrong: with EWMA bypassed
(per 1aaf94306), threshold=0.5 contracts of imbalance is below the natural
ES.FUT trade size, so every trade fires a bar.
threshold=20 produces ~5-6M bars matching the volume-bar density baseline
(5.74M bars at 100 contracts). Math: 209M / 5.74M = 36×, so threshold needs
0.5 × 36 ≈ 18 → round to 20.
Three files updated atomically per feedback_no_partial_refactor:
- config/training/dqn-production.toml: 2.5 → 20.0 (wgdc8 left it at 2.5)
- infra/k8s/argo/train-template.yaml: workflow default 0.5 → 20.0
- infra/k8s/argo/train-multi-seed-template.yaml: workflow default 0.5 → 20.0
The 1:1 bar:tick observation alongside the price-continuity proof (22 jumps
out of 209M = ~1e-7 — front-month filter works) confirms the front-month
fix on this branch (6c1ab8850 + 3b5f17913) is correct. Threshold was the
remaining mistuning.
Cache-key implication: changing the threshold invalidates all existing
fxcache files (per the cache-key architectural fix). Next dispatch on this
branch will regenerate fxcache from scratch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hypothesis test branch off sp19-20-wr-first HEAD 235e83842 (Phase 1 producers
landed, reward semantics unchanged via alpha=0 / per_bar_hold_reward=0
placeholders).
Goal: decide whether bar-resolution drives the ~46% WR plateau pinned across
16+ SP runs. wgdc7 produced ~2880 bars/day (~8 sec/bar) on ES.FUT volume
data. 5× threshold should give ~600 bars/day (~30-60 sec/bar). If WR moves
materially → resolution IS the bottleneck → SP21 hybrid justified. If WR
stays at 46% → resolution is NOT the bottleneck → SP21 needs different
framing (multi-instrument, additional features, or accept data ceiling).
Single-config change. No code changes. Reverts cleanly by deleting branch.
See project_bar_resolution_is_actual_architecture.md for context.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per Class A audit-fix Batch 4-A (deferred from P1-wiring/P1-producer due to
audit-doc errors). Fixes 2 of 4 deferred items; Batch 4-B handles plan_threshold
floor + MIN_HOLD_TEMPERATURE in a separate commit.
Item 1: DD saturation floor (the upper end of the DD ramp at trade_physics.cuh:154
in apply_margin_cap, NOT line 548 as the audit doc claimed — that line is a
magnitude action constant; the actual saturation floor lives in apply_margin_cap)
- NEW slot DD_SATURATION_FLOOR_ADAPTIVE_INDEX=458
- Producer dd_saturation_floor_update_kernel.cu — p75(per-env DD_MAX) × 1.5
via Welford `mean + Z_75 × sigma` estimator with `max(p75, mean)` robustness
guard, mirrors P0-A REWARD_POS_CAP producer pattern (Pearl-A bootstrap +
Welford α=0.01)
- Cold-start fallback: 0.25f (DD_SATURATION_FLOOR_DEFAULT in state_layout.cuh)
- Bounds: [0.10, 0.50] (Category-1 dimensional safety)
- Distinct from SP15_DD_THRESHOLD_INDEX=421 (the SP15 quadratic DD-penalty
*trigger* threshold, a *lower* bound; this slot is the *upper* end of the
linear position-size scaling ramp dd_scale = max(0.05, 1.0 − dd_frac/floor))
- Threaded `isv_signals_ptr` into `apply_margin_cap` with NULL-tolerant
cold-start fallback to DD_SATURATION_FLOOR_DEFAULT
- 4 oracle tests (Pearl-A bootstrap, no-DD guard, bounds clamp, Welford EMA)
Item 2: Legacy compute_drawdown_penalty path → Case A (DELETED)
- Decision rationale: SP15's quadratic asymmetric DD penalty
(compute_sp15_final_reward_kernel.cu:154 via sp15_dd_penalty helper) runs
unconditionally as a post-modifier on the SP11-composed reward with
ISV-driven λ_dd (slot 420) and DD threshold (slot 421). Layering the legacy
linear-ramp penalty inside the SP11 composer on top of the SP15 quadratic
creates double-counting of DD shaping — exactly the code-smell the Class A
audit was designed to eliminate. Per `feedback_no_legacy_aliases.md` and
`feedback_no_partial_refactor.md`.
- Atomic deletion across:
- `compute_drawdown_penalty` device function (trade_physics.cuh)
- Single call site at experience_kernels.cu:3822
- `dd_threshold` and `w_dd` kernel arguments
- `w_dd` Rust config field (gpu_experience_collector.rs +
trainers/dqn/config.rs DQNHyperparameters)
- `w_dd` profile section + dispatch (training_profile.rs RewardSection,
OptimizableParameterRanges, FixedRewardParameters, ParamLookup
dispatch, profile→hyperparam mapping, test assertion)
- `w_dd *= rki` risk-intensity multiplier (config.rs)
- `w_dd` TOML keys (dqn-hyperopt.toml × 2, dqn-localdev.toml,
dqn-production.toml, dqn-smoketest.toml)
- Stale doc comments on hyperopt/adapters/dqn.rs + config.rs
risk_intensity field
- `config.dd_threshold` SURVIVES (still consumed by `launch_sp15_dd_state`
as the dd_budget for DD_PCT scaling). Documented in field comment.
ISV_TOTAL_DIM: 458 → 459 (Item 1 adds 1 slot; Item 2 is pure deletion)
Cumulative WR-plateau fix series (this is commit 7):
- Class C bug 1 + P0-B (8f218cab2)
- P0-C (316db416b)
- P0-A (394de7d43)
- P1 wiring (c4b6d6ef2) — 1 of 4 wireable
- P0-A downstream (657972a4b)
- P1 producer (87d597d5d)
- audit-fix 4-A (this commit)
Verification: 16 sp14_oracle_tests pass (incl. 4 new), 36 sp15_phase1_oracle_tests
pass, 12 sp14_isv_slots layout tests pass, 4 state_reset_registry tests pass
(every-FoldReset-arm-has-dispatch contract holds), workspace cargo check clean.
Per feedback_isv_for_adaptive_bounds + feedback_no_partial_refactor +
feedback_no_legacy_aliases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Smoke and localdev profiles previously used data_source = "ohlcv", divergent
from production (mbp10). Mismatch silently produced cache-key collisions in
the SHA256 hash path: smoke fxcache could not be loaded by production-shape
training without an explicit override. Local-dev fxcache regen also required
remembering to pass --data-source ohlcv to match.
Globalize mbp10 as the default everywhere it isn't deliberately overridden:
- config/training/dqn-smoketest.toml: data_source = "mbp10"
- config/training/dqn-localdev.toml: data_source = "mbp10"
- training_profile.rs: doc Default → "mbp10"
- trainers/dqn/config.rs: Default impl → "mbp10"
- hyperopt/adapters/dqn.rs: default → "mbp10"
- examples/precompute_features.rs: doc updated
- fxcache.rs / feature_cache.rs: discover_and_load + cache-key tests
use "mbp10" arguments
- docs/dqn-wire-up-audit.md: new entry per Invariant 7
Documentation strings retained "ohlcv" only where they document the two
available choices (config.rs:946, training_profile.rs:83).
Local fxcache regenerated to v6 mbp10:
test_data/feature-cache/13c0b086a975cc7e2384377a2cd0e97738c9410292fcfecb5807c29bf885cb48.fxcache
(175874 bars, 55 MB, OFI_DIM=32). Stale v5 ohlcv fxcache untracked
from git index (already gitignored post-79578bbaf).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces random τ ∈ U(0,1) sampling with `FIXED_TAUS = [0.05, 0.25, 0.50,
0.75, 0.95]`. Kernel-side `IQN_NUM_QUANTILES` macro 32 → 5; `GpuIqnConfig::
default().num_quantiles` 32 → 5. Construction-time τ broadcast (option B2)
populates `online_taus` / `target_taus` / `cos_features` once via
`clone_htod`; both online and target IQN forwards plus the CVaR cold path
read this static buffer. The Philox-driven `iqn_sample_taus_kernel` deleted
along with its only Rust consumer (in `compute_cvar_scales`); the
`rng_step` Philox seed counter also gone. Action ranking in the IQN
inference kernel switched from mean-over-quantiles to MEDIAN
(`q_acc[a] = q_val` only when `t == IQN_MEDIAN_INDEX = 2`); the off-median
positions feed four new ISV diagnostic slots.
Four new ISV slots tail-appended:
IQN_Q_P05_EMA_INDEX = 99 (mean |Q| at τ=0.05, EMA)
IQN_Q_P25_EMA_INDEX = 100 (τ=0.25)
IQN_Q_P75_EMA_INDEX = 101 (τ=0.75)
IQN_Q_P95_EMA_INDEX = 102 (τ=0.95)
Median (τ=0.50) intentionally skipped — already in greedy-Q diagnostic.
Fingerprint pair shifted 97→103, 98→104; ISV_TOTAL_DIM 99→105.
Layout fingerprint: 0x3e21acecd922e540 → 0x5789155b683ab59c.
New kernel `iqn_quantile_ema_kernel.cu` (4-block × 256-thread shmem-reduce,
no atomicAdd) reads `save_q_online [TBA, B*Q]` and EMA-updates the four
slots. Launched from `training_loop.rs` per-step alongside
`launch_h_s2_rms_ema`. StateResetRegistry extended with 4 FoldReset
entries (cold-start 0.0).
Hyperparam plumbing: `hyperparams.num_quantiles` and
`DQNConfig::iqn_num_quantiles` pinned to `FIXED_TAUS.len()` at the
`GpuIqnConfig` construction site in `fused_training.rs::new` and
`trainer/constructor.rs`. Legacy fields stay for compat; production /
hyperopt configs (dqn-production.toml, DQNHyperparameters defaults)
aligned to 5.
Adam state for IQN params auto-resizes via `m_buf`/`v_buf` sizing
through `total_params + cublas_pad`. **Checkpoint break** — IQN head
parameter shapes change with `num_quantiles`; new fingerprint hash
fails-fast at constructor load on pre-Task-3 checkpoints.
Smoke tests:
- New `iqn_multi_quantile_heads_produce_monotonic_estimates` (1.23s on
RTX 3050 Ti): asserts ISV[99..103) finite + non-zero + spread > 1e-6
after 1 epoch — PASS (Q_p05=0.0187 Q_p25=0.0200 Q_p75=0.0193
Q_p95=0.0190).
- `multi_fold_convergence` (606.50s, 3 folds × 5 epochs): all 3 fold
checkpoints written; per-fold best train Sharpe -8.17 / 74.24 / 63.44
at epochs 2 / 4 / 2 (mean 43.17 vs 2c.3c.6 baseline mean 23.43 — folds
1+2 substantially up, fold 0 down -16 points; absolute-mean comfortably
above the plan's 3.8 floor). No NaN/Inf, no panic.
cargo check clean at 11 warnings (baseline preserved); cargo build
compiles 61 cubins (was 60; +iqn_quantile_ema, -nothing — the old
sample_taus kernel was inside iqn_dual_head_kernel.cu, not a separate
cubin file).
Files touched: 14 modified (`iqn_dual_head_kernel.cu`, `iqn_cvar_kernel.cu`,
`gpu_iqn_head.rs`, `gpu_dqn_trainer.rs`, `build.rs`, `state_reset_registry
.rs`, `training_loop.rs`, `constructor.rs`, `fused_training.rs`,
`config.rs`, `dqn-production.toml`, `smoke_tests/mod.rs`,
`docs/dqn-wire-up-audit.md`, `dqn-production.toml`) + 2 new (`iqn_quantile
_ema_kernel.cu`, `smoke_tests/iqn_quantile_monotonicity.rs`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mass deletion of the "v7 gem" reward terms identified in the Phase 1
inventory as category errors — each one rewarded an outcome-adjacent
behavior instead of encoding the underlying physics, and each
empirically hurt validation metrics more than it helped training
stability.
Deleted from experience_kernels.cu and all plumbing (Rust configs,
launch args, hyperopt logs, TOML entries):
* order_credit_weight - reward redundant with compute_tx_cost
(order_type_idx already differentiates
fills by order type)
* risk_efficiency_weight - reward double-counted drawdown penalty
asymmetrically (only on winners)
* urgency_credit_weight - reward was vol-normalized unrealized P&L,
pure rename of core return
* commitment_lambda - triple-counted churn + tx_cost
* w_dsr - kernel wrote DSR EMA but no longer added
to reward (dead); removed the EMA
bookkeeping too
* dsr_eta - kernel arg for the deleted DSR EMA
* position_entropy_weight - rewarded action-bucket diversity
regardless of outcome; histogram buffer
+ zero-init removed too
* exit_timing_weight - already inactive (used raw_next future
price, comment-deleted earlier)
* ofi_reward_weight - dead plumbing; OFI already passed as
feature through state[OFI_START..]
* opportunity_cost_scale - penalized flat when Q-gap wide;
redundant with Q-values themselves
Kernel arg count: experience_env_step_batch shrank from ~55 to ~45 args.
Rust-side config surface reduced correspondingly.
Results on E1 smoke test (20-epoch):
BEFORE any Phase 2 work:
Val Sharpe -120 to -150, MaxDD 10-15%, Sharpe_raw -0.39
AFTER reward_noise + Kelly (both envs) + urgency + this sweep:
Val Sharpe -17 to -22 (7× better)
Val MaxDD 0.27% (40× better)
Val Sharpe_raw ~-0.09 (4× better)
Training Sharpe_raw ~0 (stabilized from ±20 swings)
Final q_gap 0.1712 (highest yet, collapse mechanism fine)
The extreme train-Sharpe swings (+17 one epoch, -13 next) were not
learning dynamics — they were shaping-term noise. Core reward (P&L +
drawdown + churn + holding + tx_cost + Kelly physics cap) gives training
metrics that actually reflect what the model does.
Inventory doc (docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md)
extended with a "better-form taxonomy" section: every deleted gem has
a correct layer it belongs to (physics, feature, diagnostic, gradient-
level regularization — not reward). Kelly cap and Q-target smoothing
are already relocated; others are scheduled per the taxonomy's P1/P2/P3
priority list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 2 second relocation: the kelly_sizing_weight reward penalty
(experience_kernels.cu:1727-1746 — penalized deviation from Kelly-optimal
sizing) is deleted. Kelly is now a physics constraint in trade_physics.cuh:
the environment refuses to let the agent over-lever, not the reward
scoring the agent for matching a formula.
New helper in trade_physics.cuh (shared device function, reusable by
the forthcoming unified env kernel):
kelly_position_cap(win_count, loss_count, sum_wins, sum_losses,
max_position, safety_multiplier)
Applied in experience_kernels.cu between margin cap and execute_trade,
with health-coupled safety multiplier:
safety = 0.5 + 0.5 × health
- health=1 (healthy): full Kelly — trust the learned policy
- health=0 (collapsing): half Kelly — constrain when decisions less
reliable
Cold-start warmup (critical — otherwise balanced priors yield kelly_f=0
until real trades accumulate, starving Q-learning):
maturity = min(1.0, total_trades / 10)
effective_kelly = maturity × kelly_f + (1 - maturity) × 0.5
Early on (0 trades): cap dominated by 50% floor.
As real trades accumulate (10+): pure data-driven Kelly.
Validation env (backtest_env_kernel.cu) does NOT yet get the Kelly cap —
that requires extending its portfolio state or adding a separate
kelly_stats buffer, which naturally belongs in the Phase 3 unified env
kernel refactor. The current asymmetry is a KNOWN temporary — training
is constrained, validation is not — and will be resolved when both
kernels share the same env_step() device function.
Also completes removal of kelly_sizing_weight from all plumbing:
- experience_kernels.cu: kernel arg deleted
- gpu_experience_collector.rs: launch arg, config field, default
- training_loop.rs: hyperparam propagation
- config.rs: field, default, intensity clamp (with tombstone)
- hyperopt/adapters/dqn.rs: log reference
- config/training/*.toml (6 entries across 4 files): orphan configs
(none were wired to a profile parser field)
Verification:
- cargo check -p ml --lib clean
- E1 smoke test passes: final epoch q_gap=0.1109, health=0.51 (warmup
floor of 0.5 gives early exploration enough room; floor of 0.25
was too tight and failed at q_gap=0.0496)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 2 kick-off from the env-unification design. First relocation: the
reward_noise_scale field that perturbed training rewards is deleted, and
its regularization effect moves to the correct layer — Q-target label
smoothing in c51_loss_kernel.cu — now health-coupled rather than fixed.
Before:
- reward += pseudo_noise × max(|reward| × 0.05, 0.01) (in env reward path)
- Q-target label smoothing = fixed LABEL_SMOOTHING_EPS = 0.01
After:
- Reward untouched by noise. Core reward = actual outcome + aligned penalties.
- Q-target label smoothing eps_eff = 0.02 × (1 − health) read from ISV[12]
- health=1 (healthy): eps_eff=0, sharp targets preserved
- health=0.5: eps_eff=0.01, matches old fixed behavior at mid-health
- health=0 (collapsing): eps_eff=0.02, maximum regularization prevents
overcommitment to the collapsed distribution
Why health-coupled:
Same insight as the distillation SAXPY fix — every fixed kernel scalar is
a temporal-coupling candidate when we have the ISV pinned buffer available.
Regularization strength should scale INVERSELY with network health: it's
most needed exactly when things are falling apart.
Files touched:
- c51_loss_kernel.cu: LABEL_SMOOTHING_EPS const replaced with
LABEL_SMOOTHING_BASE + in-kernel health read from isv_signals[12]
- experience_kernels.cu: deleted reward noise block + kernel arg
- gpu_experience_collector.rs: dropped launch .arg + config field + default
- training_loop.rs: dropped hyperparam propagation
- config.rs: deleted field + intensity clamp + default (with tombstone)
- hyperopt/adapters/dqn.rs: dropped log reference
- config/training/*.toml (4 files): dropped orphan reward_noise_scale
entries (none were being parsed — the profile parser had no field)
Verification:
- `cargo check -p ml --lib` clean
- E1 smoke test passes: final q_gap=0.1190, health=0.52 (health-coupled
smoothing at ~mid-health matches old fixed behavior, collapse-prevention
mechanism intact)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Atom warm-start: bitonic sort rewards → quantile positions → write to
atom_positions_buf as initialization. Existing SGD optimizer refines.
Atoms start where reward mass actually is instead of uniform [-50,+50].
Robust PopArt: median/IQR normalization from sorted rewards replaces
Welford mean/var. More robust for bimodal distribution (many ±0.1
micro-rewards + few ±5.0 trade exits). Conditional: popart_robust=true.
Both reuse the same bitonic sort (~14ms per epoch, amortized).
gather_quantiles kernel extracts positions. extract_median_iqr reads
Q25/median/Q75 from sorted buffer.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Counterfactual: Hold(1)/Flat(3) direction mirror maps to self — trivial.
Now falls through to magnitude CF for Hold/Flat instead of wasting a
replay buffer slot on same-action same-reward experiences.
reward_noise_scale: 0.05→0.0 (dense micro-rewards are already noisy,
adding 5% label noise destroys the per-bar signal).
Added micro-reward weight config fields (price_confirm_weight=0.5,
book_aggression_weight=0.3, hold_quality_weight=0.2, micro_reward_temp=3.0,
holding_cost_rate=0.0001) — defined in TOML, kernel plumbing next session.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removed the legacy non-branching Sequential q_network and target_network
from DQNAgent. These were never used (branching+dueling always active)
but allocated VRAM and ran noise resets every step. -190 lines.
Config tuning for dense micro-reward system:
- n_steps: 5→1 (TD(0), micro-rewards cancel over n>1)
- tau: 0.007→0.01 (faster target tracking for TD(0))
- c51_alpha_max: 0.5→1.0 (full C51, PopArt handles normalization)
- curiosity_weight: 0.1→0.0 (dense micro-reward replaces curiosity)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three signal-killing issues fixed:
1. Rank normalization DISABLED — was double-normalizing with PopArt,
destroying magnitude difference between micro-rewards (±0.1) and
trade exits (±5.0). PopArt alone preserves relative magnitude.
2. n_steps 5→1 (TD(0)) — dense micro-rewards alternate ±0.1 each bar.
With n=5, they cancel out over 5 bars. TD(0) preserves the per-bar
signal that the temporal pipeline needs to learn from.
3. per_alpha 0.6→0.3 — lower = more uniform PER sampling. Dense micro-
rewards have tiny TD-errors (easy to predict), so high alpha ignores
them. Lower alpha ensures micro-reward experiences get sampled.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Linear scaling rule (2x batch → 2x LR) doesn't hold for DQN+PER with
major architectural changes (Hold action, OFI embed, wider attention).
The model needs stability to learn the new action space, not speed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Linear scaling rule: 2x batch → 2x LR to maintain effective update
magnitude. Tau increased to compensate for fewer steps/epoch (target
network tracks faster). H100 VRAM: 41GB free at B=8192, B=16384 adds
~4GB — easily fits.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Default was hardcoded as 64 in train_baseline_rl.rs and evaluate_baseline.rs,
overriding the config default of 32. Added num_quantiles=32 to production
config so it's explicit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removed: enforce_hold(), min_hold_bars from config/kernels/backtest.
Added: holding_cost_rate (inventory penalty), churn_threshold_bars +
churn_penalty_scale (graduated flip penalty) to reward in
experience_env_step and backtest kernels.
The model learns optimal hold timing from cost signals:
- Per-trade tx cost prevents churning (existing)
- Inventory penalty makes large positions expensive to hold
- Churn penalty graduates cost for rapid flips
- Temporal attention learns when holding cost > expected profit
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
min_hold=10 allowed ~1200 trades/day — costs destroyed the edge.
min_hold=50 (~6.5 min) limits to ~230 trades/day max.
Adaptive extension now scales 0-2× base (was hardcoded 0-8 bars).
With base=50: range is 50-150 bars depending on ISV stability.
Losing positions (negative reward_ema) get base only (50 bars).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The warp-parallel (32,1,1) change to c51_loss_reduce caused the same
hang as isv_feature_gate: changing block_dim inside graph_mega alters
CUDA Graph node structure on Hopper's TMA scheduler. Must stay (1,1,1).
Also: batch_size 8192→16384, gpu_n_episodes 1024→4096, num_atoms 51→52
to align h100.toml with dqn-production.toml.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mbp10_data_dir and trades_data_dir uncommented in localdev config.
Doc comments updated: 20 microstructure features always appended.
OFI is no longer optional — it's core to the model's feature set.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
NoisyLinear was completely dead after cuBLAS migration — neither training
nor experience collection applied ANY exploration noise. With identical
Q-values across actions, Boltzmann selection was uniform, making the
dueling mean-subtraction cancel ALL advantage gradients. The advantage
heads could NEVER learn (Q-gap permanently 0.0000).
Fix: add_advantage_noise CUDA kernel injects per-action Gaussian noise
into Q-values AFTER compute_expected_q, BEFORE action selection.
- Philox PRNG + Box-Muller for GPU-native Gaussian sampling
- Per-sample, per-action noise (breaks within-batch symmetry)
- Only during experience collection (not training targets)
- noise_sigma=0.1 (configurable, TOML + hyperopt tunable)
The noise creates non-uniform Boltzmann selection → different actions
have different frequencies → advantage gradient survives the dueling
mean-subtraction → Q-gap can grow.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Q-gap = max_a Q(s,a) - mean_a Q(s,a): measures action preference per state.
Q-var = Var_s[Q(s,a)]: measures state differentiation across the batch.
Both are 0.0000 through all 10 smoketest epochs despite Sharpe 8+ and
Q-values growing to 0.012. This proves the good metrics are from
reward-induced mechanical bias, NOT learned Q-values.
Also: c51_warmup_epochs=0 (C51 from step 1, bypass MSE dead zone),
smoketest lr=1e-4 (50× higher, makes 40 steps representative).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
With per-component clipping removed, raw gradient norms are ~4000.
The old gradient_clip_norm=1.0 (calibrated for pre-clipped norms ~7)
discarded 99.97% of gradient magnitude every step, making the effective
learning rate 570× too small. Q-values grew 9× slower than old run.
10000 lets normal gradients (~4000) pass through unclipped. Only fires
on genuine divergence spikes (>2.5× normal). Adam's internal adaptive
step sizing handles the rest.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fundamental fix: C51 atom spacing (dz=0.6) was larger than the Q-value
range (0.02), making the distributional loss unable to resolve rewards.
The network was blind to its own learning signal.
Adaptive v_range via device buffer (same pattern as tau_buf):
- v_range_buf[2] on GPU, all C51/MSE/CQL/expected_q kernels read via
pointer (graph captures stable address, reads value at replay time)
- Cold start: v_range=±1.0 → dz=2/51=0.039 → 25× atom resolution
- Monotonic expansion based on Q-stats every 50 training steps
- Never shrinks (avoids invalidating Bellman targets in replay buffer)
Kernel changes (5 files):
- c51_loss_kernel.cu, mse_loss_kernel.cu, mse_grad_kernel.cu,
cql_grad_kernel.cu, compute_expected_q: scalar v_min/v_max → pointer
Also fixed:
- PopArt warmup 100→1 (start normalizing from step 1)
- PopArt variance floor 0.0001→1e-8 (allow sigma < 0.01)
- Backtest evaluator: allocate own v_range_buf for compute_expected_q
- num_atoms 51→52 in all TOMLs (TF32 4-element alignment)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
v_min/v_max ±15 with gamma=0.99 only covers 15% of theoretical Q range
(Q_max = 1/(1-0.99) = 100 for unit-variance PopArt rewards). C51 top atom
saturates on sustained winners, degrading distributional learning. ±50
covers 95th percentile.
PopArt GPU buffers (popart_mean, popart_var, popart_count) were never
zeroed between walk-forward folds — fold 2's reward normalization was
contaminated by fold 1's statistics. Now reset alongside Adam state in
reset_adam_state().
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Added OwnedGpuLinear to ml-core (self-contained linear layer with owned
weight/bias CudaSlice, forward_with_slices bypasses GpuVarStore lookup)
- Removed GpuVarStore from all 12 ml-dqn modules: Sequential, branching,
distributional_dueling, dueling, network, rmsnorm, residual, curiosity,
attention, iql, quantile_regression, regime_conditional
- Removed get_q_network_vars/vars/store accessors from DQN, branching,
distributional_dueling, dueling, network, quantile_regression
- Replaced GpuLinear+GpuVarStore with OwnedGpuLinear in all cold-path modules
- Made load_from_safetensors a no-op (flat buffer path handles checkpoints)
- Made sync_to_varmap a no-op (flat buffer is source of truth)
- Moved branching->VarStore conversion to gpu_weights::branching_to_varstore
- Re-enabled bottleneck_dim=16 in config defaults + all 3 TOML configs
- Removed flatten_online_weights bottleneck skip workaround
- Zero GpuVarStore references in crates/ml-dqn/src + crates/ml/src/trainers/dqn
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
bottleneck_dim=16 causes w_s1 size mismatch between VarStore (full state_dim)
and flat params_buf (reduced bottleneck_concat_dim). EMA target sync crashes
with CUDA_ERROR_ILLEGAL_ADDRESS.
Root cause: GpuVarStore allocates weights at full state_dim but GpuDqnTrainer's
flat buffer uses bottleneck-reduced dimensions. These two weight sources are
fundamentally incompatible when bottleneck is active.
Fix: eliminate GpuVarStore entirely (next task), then re-enable bottleneck.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Task 3: Enable PopArt reward normalization (default true, all 3 TOML configs,
wiring confirmed in fused_training.rs submit_forward_ops_main()).
Task 4: Couple PopArt variance with tau reset — add prev_popart_var to
FusedTrainingCtx, read_popart_var() to GpuDqnTrainer, read_popart_variance()
+ should_reset_tau() to FusedTrainingCtx, tau reset injected at epoch boundary
in training_loop.rs after log_phase_timing().
Task 5: Add best_sharpe/best_epoch/best_val_loss reset to reset_for_fold() so
each walk-forward fold competes independently. Also wire v8 reward fields
(popart_enabled, micro_reward_scale, td_lambda, max_trace_length,
hindsight_fraction, hindsight_lookahead) through training_profile.rs apply_to().
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- iqn_lambda restored to 0.25 (was 0.0 for NaN isolation)
- STEP_DIAG per-step logging removed (was temporary)
- FOXHUNT_GRAD_DIAG env var removed from Argo template
IQN NaN root cause fixed in 8cbabcef5 (f32-as-bf16 type mismatch).
Ready for H100 baseline deployment.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Graph recapture ruled out as NaN cause (NaN persists without graph
invalidation). Now testing: does NaN disappear when IQN is disabled?
If yes → IQN backward/Adam produces NaN on H100.
If no → NaN is in the main C51/MSE pipeline.
Also reverts graph invalidation hack from previous diagnostic commit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The f32→bf16 cast in cast_d_logits_to_bf16() reads
b*(b0+b1+b2+b3)*na + 32*4 elements, but d_adv_logits_buf and
d_adv_logits_bf16 were allocated with only +32*3 padding (96 vs 128).
The 32 extra f32 reads went past the allocation into adjacent GPU
memory. On H100 with 80GB and different memory layout, this grabbed
NaN/garbage values. The NaN propagated through the cuBLAS backward
pass into grad_buf, killing training at step ~720 (non-deterministic
depending on what lives in adjacent memory).
Fix: allocate +32*4 padding in d_adv_logits_buf, d_adv_logits_bf16,
and d_adv_logits_mse to match the cast kernel's access pattern.
Also reverts c51_alpha_max default to 1.0 — the gradient collapse
was from this buffer bug, not C51 premature convergence. The
c51_alpha_max parameter is kept for hyperopt exploration.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
H100 baseline (20 epochs) showed gradient collapse at epoch 10: C51
cross-entropy converges its distributional fit before the policy converges,
leaving zero gradient signal. The collapse happened 5 epochs after C51
reached alpha=1.0 (pure C51, zero MSE).
Fix: cap the C51 alpha ramp at c51_alpha_max (default 0.5) so MSE always
contributes (1 - alpha_max) of the primary gradient. MSE loss measures
Q-error directly and only goes to zero when Q-values are correct, not
just when the distribution shape is right.
- c51_alpha_max added to DQNHyperparameters (default 0.5)
- Added to PSO search space as 15th dimension (range [0.3, 0.9])
- Added to TOML profiles: smoketest, production, hyperopt
- Training loop caps alpha ramp at alpha_max instead of 1.0
- All 907 ml tests + 300 ml-core tests + 6 smoke tests pass
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three root causes found and fixed:
1. SUM-reduced gradients without 1/N: all CUDA loss gradient kernels
(C51, MSE, IQN backward, CQL) accumulated per-sample gradients as
raw SUM. At batch=16384 (H100) the raw norm was 282x larger than
batch=58, causing gradient clipping to destroy signal-to-noise ratio
and collapse training at epoch 2-3. Now all kernels multiply by
1/batch_size, making gradient scale batch-invariant.
2. ExposureLevel::target_exposure() used a flat 9-level scale that did
not match the 4-branch dir*mag encoding. The Rust backtest evaluator
computed wrong position sizes (e.g. 4x oversize for Short+Small).
Now uses dir x mag formula. Also fixed is_buy/is_sell/is_hold and
from_trading_action for 4-branch semantics.
3. Rust epsilon-greedy only explored 5/9 exposure combos (0..5 instead
of dir*3+mag), ignored the magnitude branch on greedy, and used wrong
indices for order/urgency (get(1)/get(2) instead of get(2)/get(3)).
LR recalibrated: old gradient_clip_norm was accidentally a batch-size-
dependent LR reducer (~60x at batch=58, ~16000x at batch=16384). With
mean-reduced gradients the clip rarely fires, so LR is now the sole
training speed control. Smoketest 1e-4 -> 2e-6, production 1e-4 -> 1e-5,
hyperopt range [1e-5,3e-4] -> [1e-7,1e-4].
Diagnostics: FOXHUNT_GRAD_DIAG=1 enables per-stage gradient norm logging.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three fixes:
1. spread_cost removed contract_multiplier ($50) — was computing in dollars
but PnL is in points, making spread 12.5× actual. The model saw every
trade as guaranteed loss → preferred Flat.
2. Mirror universe now negates ALL 17 directional features (returns, MACD,
Bollinger, SMA ratios, regression slope, CUSUM) + flips RSI. Was only
negating 4, teaching wrong direction on 50% of epochs.
3. tx_cost_multiplier updated to 0.18 bps (IBKR ES RT=$4.50/contract).
min_hold_bars=10 for volume bars (~26s at 23 bars/min).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
With separate optimizer, no gradient competition — safe to increase.
Smoke test shows within-group differentiation starting (S100≠S25≠L50).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The exposure aux backward GEMM amplifies gradients through the
hidden activation matrix (d_logits × h_b0), producing grad_norm=5000+.
With max_grad_norm=10, this throttles ALL gradients by 500x (lr/500).
Fix: reduce aux_weight 0.5→0.01 (50x) + clip_grad_buf_inplace after
aux GEMM to cap combined gradient at max_grad_norm.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
