Per project_ml_alpha_starting_capital greenfield posture: there is no V1
to differentiate from (the V1 trunk forward was dead code, no V1
checkpoint files exist in the wild). The 'v2' prefix on every identifier
was historical baggage from the migration period.
Renames:
- CheckpointV2 -> Checkpoint (also drops the version: u32 field —
bincode either deserialises a current envelope or errors; no migration
path needed)
- CheckpointVersionProbe removed (was only for V1 rejection)
- LAYER_NORM_CUBIN_V2 / VARIABLE_SELECTION_CUBIN_V2 / ATTENTION_POOL_CUBIN_V2
-> LAYER_NORM_CUBIN / VARIABLE_SELECTION_CUBIN / ATTENTION_POOL_CUBIN
- _ln_module_v2 / _vsn_module_v2 / _attn_module_v2 -> drop _v2 suffix
- smoke_load_v2_checkpoint test -> smoke_load_checkpoint
- config/ml/sweep_v2_*.yaml -> config/ml/sweep_*.yaml
- migration-era 'V2 weight skeleton' / 'V2 fields' / etc. comments
cleaned to remove the v2 prefix
Pre-existing 'v2' references in ml-backtesting CUDA files
(decision_policy.cu, pnl_track.cu) are NOT touched — those refer to
future planned 'v2' refinements (Portfolio mode, multi-fill averaging)
from the C1-C19 commits and reflect aspirational features unrelated to
this session's trunk-grows work.
Verification: ml-alpha + ml-backtesting + fxt-backtest all build clean.
perception_forward_golden bit-exact (max_diff = 0.000000).
New `fxt-backtest sweep --grid <yaml> --out <dir>` subcommand: iterates
over a grid of Run configs, writes each cell's artifacts to
<out>/<cell_name>/, then automatically invokes the existing aggregate
path to produce aggregate.parquet + pareto_frontier.json at the root.
All cells run sequentially on the same GPU (single-machine). For
cluster fan-out the underlying mechanism is the same — Argo can wrap
this binary in a workflow that runs each cell as a separate pod
(left as infra-side work for a follow-up commit; the binary's
contract is the same).
Sweep grid YAML schema:
base: # defaults applied to every cell unless overridden
data: ...
n_parallel: ...
decision_stride: ...
latency_ns: ...
target_annual_vol_units: ...
annualisation_factor: ...
max_lots: ...
max_events: ...
seed: ...
checkpoint: ... # optional — load real trained weights
cells:
- name: cell_a
decision_stride: 1 # override base
- name: cell_b
latency_ns: 250000000
...
Each SweepCell may override any subset of fields; unset fields fall
back to base defaults. --max-cells gates the run for smoke-testing
large grids.
Adds an example grid at config/ml/sweep_decision_stride_example.yaml
that re-runs the decision_stride ∈ {1, 2, 4, 8} sweep deferred from
the original plan (task #202).
Closes#201 (sweep tool). #202 (first decision_stride sweep) is now
executable end-to-end via the example grid.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
C51 distributional Q-network with GPU Thompson selection borrowed
minimally from production (alpha_c51.cu: forward, project, grad,
expected_q, thompson_select kernels; ~260 lines). Uses Huber
negative-tail compression in projection per production
block_bellman_project_f. Action selection 100% GPU via mapped-pinned
i32 output + __threadfence_system + host volatile read (matches
gpu_training_guard MappedBuffer pattern).
Backtest result (2D sweep, 500 episodes per cell, 30 cells):
cost=0 C51 +10.41 vs linear-Q -15.72 (+26pt, BEATS Phase 1d.4
no-RL baseline +4.4 by 6pt)
cost=0.125 C51 -13.81 vs -29.17 (+15pt closes half-tick gap)
Win rate at cost=0 best τ: linear-Q 0.008 → C51 0.552.
Calibration hypothesis vindicated; documented in
memory/pearl_c51_thompson_closed_phase_e3_gap.md.
Also in this commit (Phase E.3 follow-up cleanup):
- --pruned-actions falsified (2.4× worse Sharpe). Documented in
memory/pearl_action_pruning_falsified.md.
- --real-spread falsified for ES futures (76% of bars at 1-tick floor).
- SnapshotRow bid_l/ask_l extended from [f32; 3] to [f32; 10].
L4-L10 synthesized in this commit; real MBP-10 peek lands in E.4.A T5.
- docs/isv-slots.md updated per kernel-audit-doc hook requirement.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase E.3 follow-up. Adds --train-threshold to alpha_compose_backtest so
the Q-network can be trained against a FIXED gate (instead of just
applying the gate at eval). Default 0.39 = the equilibrium the smoke's
controller stabilized to at ep 200+ (alpha_dqn_h600_smoke gated run).
Smoke result (gated training, controller running):
ep 100: thresh=0.32 obs=0.226 R_mean=-5.5 atten=0.75
ep 200: thresh=0.38 obs=0.082 R_mean=-3.0 atten=0.50
ep 300: thresh=0.39 obs=0.081 R_mean=-3.2 atten=0.25
ep 1000: thresh=0.39 obs=0.039 R_mean=-4.7 atten=0.10
The controller CONVERGES cleanly to threshold ≈ 0.39 with observed
trade rate at/below the 0.08 target. rollout_R_mean drops from -19
(no-gate training) to -4.7 (gated training): 4× less loss per episode.
rvr stays at +1.045σ (unchanged). The closed-loop architecture works
end to end.
(Note: smoke verdict FAILs on ACTION_ENTROPY (0.68 < threshold 1.10).
This is the policy correctly Waiting 95%+ of the time — the kill
criterion was designed to catch "collapse to one bad action," but
collapse-to-Wait under a strong gate is the RIGHT behavior. Verdict
threshold is misaligned with the gated paradigm; not a regression.)
Backtest result with --train-threshold 0.39:
cost eval-gate only train+eval gated Δ
------ -------------- ---------------- ----
0.0000 -15.72 -17.06 -1.3
0.0625 -21.30 -22.91 -1.6
0.1250 -29.17 -31.26 -2.1
0.2500 -42.12 -36.68 +5.4
0.5000 -54.86 -53.83 +1.0
Training with the gate did NOT meaningfully improve absolute Sharpe.
The eval-best threshold remains 0.20-0.25 in BOTH runs (not 0.39).
The Q-network's primary contribution is the binary trade/don't-trade
decision; the action-choice (Buy direction + placement) is largely
determined by alpha sign — linear Q can't time entry better than the
threshold filter does on its own.
Honest analysis: the gap to Phase 1d.4 baseline (+4.4 at cost=0,
-4.0 at half-tick) is NOT architectural but ECONOMIC:
Env spread: bid/ask synthesized at ±0.125-tick around mid
→ round-trip spread cost = 0.25 per trade
At τ=0.20 with 168 trades/ep: 168 × 0.25 = 42 in spread costs
Mean reward = -5 → alpha extracts ~37 of value
All eaten by spread
Phase 1d.4 baseline likely trades much less (~20-50 trades/ep at best
operating point — pure threshold-only policy, no RL). Our policy
trades 3-8× more because the DQN's action choices add fine-grained
trade attempts beyond the threshold filter's wait/trade gate.
The control loop architecture (Phase E.1 + E.2 + E.3 gate consumption)
is VALIDATED — gate produces monotone Sharpe lift, +1.045σ rvr held,
trade-rate-self-correction converges cleanly. But beating Phase 1d.4's
absolute Sharpe requires:
1. MLP for the Q-network (more representation capacity for
entry-timing decisions within the alpha confidence band)
2. OR action-space constraints (collapse the 9-action space — drop
fine-grained L1/L2 placement, keep just {Wait, BuyMarket,
SellMarket, FlatMarket})
3. OR better fill economics (real LOB instead of fixed ±0.125-tick
synthesis)
These are Milestone E.3 follow-up work (Tasks 24-28 sweeps + future
architectural changes). The composition backtest validated what it
was designed to: the cost-edge frontier of the linear Q + Phase 1d.3
alpha + controller setup, and surfaced the next architectural
question (representation capacity vs action-space size vs fill
realism).
Branch: sp20-aux-h-fixed, pushed.
Phase E.3 Task 23 follow-up. Adds the confidence-threshold gate that
consumes the controller's ISV[543] output. Both binaries:
fn epsilon_greedy_gated(q, alpha_confidence, threshold, eps, rng) -> u8 {
if alpha_confidence < threshold { return 0; /* Wait */ }
epsilon_greedy(q, eps, rng)
}
State[1] is the env's alpha_confidence = |sigmoid(alpha_logit) - 0.5|
which is in [0, 0.5]; threshold is also clamped [0, 0.5], so direct
comparison is valid.
alpha_dqn_h600_smoke (closed-loop with controller):
Adds current_threshold: f32 cache, initialised to 0.0 (no gate),
refreshed via stream.clone_dtoh(&isv_dev) after each per-episode
controller invocation. Action selector reads current_threshold for
the NEXT episode's step decisions.
alpha_compose_backtest (2D sweep):
Adds --threshold-grid CLI flag (default [0.0, 0.05, 0.10, 0.15, 0.20,
0.25] — Phase 1d.4 pattern). Eval loop becomes 2D (threshold × cost).
Per-bin includes avg_n_trades for trade-rate visibility. End-of-run
prints BEST per-cost = max Sharpe_ann across τ.
Results (1000 train ep, 300 eval ep × 5 τ × 5 costs):
cost τ=0.00 best τ Sharpe lift trades/ep saved
------- ---------- --------- ----------- ---------------
0.0000 -41.78 -15.72 (τ=0.20) +26.1 477 → 168 (-65%)
0.0625 -71.46 -21.30 (τ=0.25) +50.2 476 → 138 (-71%)
0.1250 -86.78 -29.17 (τ=0.20) +57.6 482 → 167 (-65%)
0.2500 -108.57 -42.12 (τ=0.25) +66.5 480 → 132 (-73%)
0.5000 -146.76 -54.86 (τ=0.25) +91.9 478 → 136 (-72%)
Win rate at cost=0: 7.7% (no gate) → 20.3% (τ=0.20).
The gate architecture is VALIDATED: monotone improvement in win rate +
Sharpe + trade-rate reduction across all costs. The control loop
(controller → slot 543 → policy gate → observed rate feedback) is
sound. But the policy is STILL negative-Sharpe at every cost.
Phase 1d.4 baseline at half-tick: -4.0 (ours: -29.17). 25-pt gap.
Root cause of the remaining gap: the Q-network was TRAINED without
gate awareness. It learned Q-values for the over-trading regime. The
eval-only gate filters those decisions but can't fix miscalibrated
Q-values. Phase 1d.4 baseline beats us because its policy
(always-market-when-confident) is INHERENTLY gated by design — no
mismatched Q-values to fix.
Next iteration to close the 25-pt gap: train WITH gate on, so the
Q-network learns weights for the gated policy class. This means:
either (a) controller runs during training (smoke pattern) and the
threshold develops endogenously, or (b) fixed --train-threshold CLI
during training. Either way, the Q-network sees Wait-at-low-confidence
during the learning phase and adapts.
Files touched:
crates/ml/examples/alpha_dqn_h600_smoke.rs (gate + threshold cache)
crates/ml/examples/alpha_compose_backtest.rs (gate + 2D sweep)
config/ml/alpha_compose_backtest.json (2D verdict)
Phase E.3 Task 23. Trains the Phase E execution-policy DQN on the first
80% of fxcache snapshots, then evaluates the frozen policy (ε=0) on the
held-out 20% across a transaction-cost sweep. Compares absolute Sharpe
vs the Phase 1d.4 always-market-when-confident baseline.
Pipeline pieces:
- Shared loaders extracted into crates/ml/src/env/loaders.rs (used by
both alpha_dqn_h600_smoke and alpha_compose_backtest)
- alpha_compose_backtest.rs: train DQN on first n_train bars, then
frozen-eval n_eval episodes per cost level
- cost grid: [0.0, 0.0625, 0.125, 0.25, 0.5] (price units per
contract round-turn)
- Annualised Sharpe via per-episode Sharpe × sqrt(episodes/year)
where episodes/year ≈ 252 · 6.5h · 3600s / (horizon · 12s)
Run (horizon=600, 1000 train ep, 500 eval ep/cost, 1.5M snapshots):
cost n_ep mean_R std_R Sharpe/ep Sharpe_ann win_rate
0.0000 500 -11.09 8.03 -1.380 -39.50 0.090
0.0625 500 -20.68 9.88 -2.093 -59.89 0.012
0.1250 500 -29.38 9.49 -3.095 -88.58 0.000
0.2500 500 -48.23 11.90 -4.052 -115.98 0.000
0.5000 500 -84.59 17.43 -4.854 -138.92 0.000
Phase 1d.4 baseline for comparison: +4.4 ann. at cost=0, -4.0 at half-tick.
The Phase E policy LOSES MONEY across the whole cost grid — even at
frictionless cost=0. This is not a contradiction with the H=600 PASS
verdict (rvr=+1.04σ): the smoke's rvr is RELATIVE TO RANDOM, while
backtest Sharpe is ABSOLUTE. "Better than random by 1 std" is still
losing if random loses big.
The diagnostic that the E.2 controller already surfaced:
ISV[543] STACKER_THRESHOLD saturated at upper clamp (0.5) — policy
trades 85% of the time vs the 8% target. Over-trading pays spread on
every bar regardless of alpha confidence. Even with perfect alpha
(Phase 1d.3 AUC=0.673), trading 85% × spread cost > alpha edge.
The Phase 1d.4 baseline beats us at cost=0 because it WAITS unless
|stacker_logit| > threshold — the threshold gate filters bars with
weak alpha signal. The Phase E controller PRODUCES slot 543 but the
DQN's action selection doesn't CONSUME it.
This is exactly what the E.3 backtest is FOR: revealing that the
Phase E.1/E.2 producer-side architecture without consumer-side gating
is incomplete. The composition backtest validates the architecture's
weak link.
NEXT (E.3 task 24-28 or a side fix): wire slot 543 consumption into
the action selection. At each step:
if |ISV[543] − 0.5| > |stacker_logit − 0.5|:
action = Wait // confidence below threshold, sit out
else:
action = argmax(Q)
Or equivalently: action = if confidence_high(alpha_logit, ISV[543])
{ argmax(Q) over Buy/Sell actions } else { Wait }.
Once slot 543 is consumed, re-run alpha_compose_backtest and expect
Sharpe to move toward / past the Phase 1d.4 baseline.
Loader refactor: extracted load_fill_model_from_json, load_alpha_cache,
load_snapshots_from_fxcache from alpha_dqn_h600_smoke.rs into
crates/ml/src/env/loaders.rs. The smoke now calls the shared module
via ml::env::loaders::*. ~150 lines of duplicated code removed.
Build + run verified: smoke still builds clean. Backtest runs in ~30s
(train 8s + eval 20s + setup).
Branch: sp20-aux-h-fixed, pushed.
Phase E.2 Task 17. Loads stacker_threshold_controller.cubin at smoke
startup, initialises ISV[544] (TRADE_RATE_TARGET) to 0.08 (CLI flag
--trade-rate-target, never reset), allocates a 3-float Wiener state
buffer for slot 545's Pearl A+D state.
Invokes the controller at every episode end with:
rollout_trade_count = count of non-Wait actions in the episode
rollout_total_decisions = actions_host.len() (= ep_len)
rollout_realized_sharpe = ep_terminal_R / RANDOM_BASELINE_STD
(per-rollout analog of the rvr metric;
lets the Kelly-atten controller respond
to in-policy performance vs the baseline
noise floor)
CLI args added:
--trade-rate-target default 0.08 (8% per-step trade rate target)
--k-threshold default 0.01
--k-atten default 0.005
--target-sharpe default 0.5
--wiener-alpha-floor default 0.4
--ctl-alpha-meta default 0.1
Periodic log line extended:
ep ... | KC q/H/rvr/ΔQ ... | CTL thresh=... obs=... atten=...
Final JSON adds:
final_stacker_threshold
final_trade_rate_observed_ema
final_stacker_kelly_attenuation
trade_rate_target
Smoke run (H=600, 1000 episodes) verifies the controller is alive:
ISV[543] STACKER_THRESHOLD: 0.000 → 0.5000 (saturated at ceiling)
ISV[545] TRADE_RATE_OBSERVED_EMA: 0.000 → 0.712
ISV[546] STACKER_KELLY_ATTENUATION:0.000 → 0.100 (hit floor)
Verdict: PASS — rvr=+1.043σ (unchanged from Task 12b PASS, expected
since smoke doesn't yet CONSUME slots 543/546).
Tuning notes (calibration for production, not bugs):
• Threshold saturating at 0.5 → policy trades ~85% (target 8%, off by
10×). Either re-calibrate target_trade_rate from realistic backtest
behaviour, or raise the clamp ceiling. Current ε-greedy with low
threshold-consumption gate produces high trade rate.
• Kelly atten hit floor (0.1) because rollout_sharpe (~-0.004) is far
below target_sharpe=0.5. The target needs to match the rollout
metric's scale, OR the metric should be time-normalised. The
current ep_terminal_R / baseline_std proxy is meaningful but its
scale doesn't match a typical annualised Sharpe target.
These tuning items don't gate Milestone E.2 — the producer-side
controller is correctly driving the ISV slots; *consuming* those slots
(threshold gate on alpha signal, Kelly-cap multiplier) is Phase E.3
work (alpha + execution composition).
Phase E.2 Tasks 16 + 17 close-out: kernel + launcher + GPU smoke test
+ wired into smoke binary + initialisation + verified end-to-end. Tasks
19-22 (NoisyNet) are gated on Task 12 FAIL, which we passed — skipped.
Task 18 (alpha-trust ablation, ~9-18 hours compute) deferred to a
dedicated session if needed.
Two carried-over limitations from Phase E.0 / E.1 fixed and verified.
1. MBP-10 parser bug fix (`parse_mbp10_streaming` + `parse_mbp10_file`)
The DBN crate's `Mbp10Msg` carries the FULL post-update top-10 book
in `levels: [BidAskPair; 10]` per message — not just the single
update event's price/size. Previously the parser only called
`update_level(0, ...)` with the update event's fields, leaving
`current_snapshot.levels[1..10]` at default-empty. Downstream:
- OFI calculator reading L2-L5 got zeros → produced wrong OFI
features (the canonical Phase 1c/1d 81-dim feature stack has
multi-level OFI as features 0..5; with the bug these were
constant zero).
- microprice (`snapshot.levels[1]`) got zeros.
- FillModel L2/L3 fit observations got zeros, so L2/L3
coefficients were undefined (we worked around by replicating
L1 with attenuated intercept).
Fix: after `update_level(0, ...)`, copy fields from
`mbp10.levels[lvl]` into `current_snapshot.levels[lvl]` for `lvl
in 1..max_lvl`. Field-by-field copy preserves the existing scale
convention (raw 1e9 fixed-point i64). Applied to both streaming
and async file-parse code paths.
Comment "For simplicity, store all updates in level 0 / A full
implementation would maintain proper level ordering" removed.
2. fit_poisson L2 regularization
New `fit_poisson_l2(features, observed, max_iters, lr, l2_lambda)`
API (the old `fit_poisson` delegates with l2_lambda=0). L2 penalty
applies to slope coefficients β[1..5] but NOT to intercept β[0]
(penalizing the intercept biases toward p≈0.5 for all-zero-feature
samples, breaking the recovery test). Per-iteration update:
β[0] -= lr · grad[0] / n (intercept)
β[k] -= lr · (grad[k] / n + λ · β[k]) (slope, k ∈ 1..5)
Canonical motivation: on real 5.2M-trade ES.FUT data the
unregularized fitter converged to β_spread ≈ -40 (Task 5c commit
12151ccf6), producing near-zero limit fill probability at typical
spreads despite empirical fill rate ~70%. With l2_lambda=0.01 the
slope shrinks modestly while intercept tracks the empirical rate.
Default in the calibration binary bumped to 0.01.
New unit test `fit_poisson_l2_shrinks_slope_on_pathological_outlier`
constructs 990 typical samples + 10 wide-spread outliers and
verifies `|β_spread|` with L2 < `|β_spread|` without L2. Passes.
3. Cascade re-run verifies the fix is verdict-robust:
New fit (with L2 + parser fix, 500K snapshots):
BID L1: β_0=-0.24 β_spread=-1.87 β_imbal=-0.10 β_ofi=-0.006 β_logτ=-0.30
ASK L1: β_0=+0.21 β_spread=-36.41 β_imbal=+0.19 β_ofi=+0.81 β_logτ=+0.22
(β_spread on ask still large but β_0 sane; cloglog model
fundamentally mis-fits the binary tight-spread / wide-spread regime.)
New baseline (with new fill model):
mean = -5191.53 (vs old -5185.13)
std = 4963.62 (vs old 4952.85)
Negligible drift, env dynamics essentially unchanged.
H=6000 smoke re-run (same alpha cache, new fill model + parser):
Q_SPREAD_EMA = 29.59 (was 35.44)
ACTION_ENTROPY_EMA = 2.00 (was 2.00)
RETURN_VS_RANDOM_EMA = +1.001σ (was +1.003σ)
EARLY_Q_MOVEMENT_EMA = 0.130 (was 0.130)
Overall: PASS (was PASS)
Verdict is ROBUST to the fixes — the fxcache-based smoke is
insulated from the MBP-10 parser bug (uses synthesized bid/ask
from mid), and the FillModel quality improvement is minor enough
that the policy's behaviour is essentially unchanged. The fixes
matter MORE for production training paths that read MBP-10
directly (those see the full L2-L10 book now).
Files touched:
crates/data/src/providers/databento/dbn_parser.rs (parser fix in
both parse_mbp10_streaming and parse_mbp10_file)
crates/ml/src/env/fill_model.rs (new fit_poisson_l2 + test)
crates/ml/examples/alpha_fit_fill_model.rs (--l2-lambda flag)
crates/ml/examples/alpha_dqn_h600_smoke.rs (updated hardcoded
baseline values to match the new random baseline run)
config/ml/alpha_fill_coeffs.json (re-fitted with both fixes)
config/ml/alpha_random_baseline.json (re-run with new fill model)
config/ml/alpha_dqn_h6000_smoke.json (verified PASS)
All 8 fill_model tests pass. Build clean across data, ml-alpha, ml.
Phase E.1 Task 13. Same pipeline as the H=600 PASS run (cd5aa3402),
just `--horizon 6000` — the plan's production horizon. Result:
Q_SPREAD_EMA = 35.44 ≥ 0.05 PASS
ACTION_ENTROPY_EMA = 2.00 ≥ 1.099 PASS
RETURN_VS_RANDOM_EMA = +1.003σ ≥ 0.0 PASS
EARLY_Q_MOVEMENT_EMA = 0.130 ≥ 0.01 PASS
Overall: PASS (H=6000 scale-up VIABLE)
H=600 vs H=6000 side-by-side (same DQN, only horizon changed):
H=600 H=6000
rollout_R_mean -18 -236 (13× for 10× horizon — sublinear)
RETURN_VS_RANDOM_EMA +1.043σ +1.003σ (alpha signal generalizes)
Q_SPREAD_EMA 10.92 35.44 (sharper action discrimination)
EARLY_Q_MOVEMENT_EMA 0.099 0.130 (more weight movement per episode)
Overall PASS PASS
The Mamba2 K=6000 alpha-cache (config/ml/alpha_logits_cache.bin) was
directly trained for this horizon, so generalization at H=6000 is the
expected result. Confirmed empirically.
Runtime: 80s for 1000 episodes × 6000 steps = 6M transitions
(~75K transitions/sec, same throughput as H=600).
Milestone E.1 is now CLOSED with all kill criteria PASSING at the
production horizon. Per the plan, this unlocks Milestone E.2 — the
stacker-threshold ISV controller (engagement-rate self-correction).
Reproduction:
cargo run -p ml --release --example alpha_dqn_h600_smoke -- \
--fxcache-path .../9297....fxcache \
--alpha-cache config/ml/alpha_logits_cache.bin \
--horizon 6000 --n-episodes 1000 \
--max-snapshots 1500000
Verdict + per-checkpoint KC trajectory in config/ml/alpha_dqn_h6000_smoke.json.
Phase E.1 Task 12b complete. The H=600 DQN smoke now consumes real
alpha_logit from the Phase 1d.3 stacker (Mamba2 + 7-input MLP stacker
trained for AUC=0.673 on test), and PASSES all four kill criteria:
Q_SPREAD_EMA = 10.92 ≥ 0.05 PASS
ACTION_ENTROPY_EMA = 1.97 ≥ 1.099 PASS
RETURN_VS_RANDOM_EMA = +1.043 ≥ 0.0 PASS ← jumped +3.62σ
EARLY_Q_MOVEMENT_EMA = 0.099 ≥ 0.01 PASS
Overall: PASS (H=6000 scale-up VIABLE)
Before/after comparison (same env, same DQN, only alpha_logit changed):
alpha_logit=0 alpha_logit=Phase1d.3
rollout_R_mean (final) -18,272 -18
RETURN_VS_RANDOM_EMA -2.58σ +1.04σ
Overall verdict FAIL PASS
The 1000× reduction in episode loss + the +3.62σ rvr swing definitively
proves the "first-best-action lock-in" hypothesis from the previous FAIL
analysis was a SYMPTOM, not the cause. The cause was alpha_logit=0
placeholder starving the policy of directional signal. With real Phase
1d.3 alpha, the linear Q-network learns to use it cleanly — no
NoisyNet, no MLP, no architectural change needed.
Integration pieces in this commit:
1. Cargo workspace registration: ml-alpha added as a workspace dep,
ml's manifest now depends on ml-alpha for FxCacheReader access.
(ml-alpha already depends only on ml-core, so no circular risk.)
2. alpha_dqn_h600_smoke.rs: two new CLI args
--fxcache-path <PATH> load snapshots from precomputed fxcache
(mid from raw_close, bid/ask synthesized
at fixed half-tick, 81-dim features extracted
for spread_bps / l1_imbalance / ofi / mid_drift)
--alpha-cache <PATH> load Phase 1d.3 stacker logit cache produced
by `alpha_train_stacker --alpha-cache-out`.
Each cache entry aligns to the corresponding
fxcache bar, populates SnapshotRow.alpha_logit
(and derives alpha_confidence = |sigmoid(z)-0.5|).
3. Snapshot source selection: in main(), --fxcache-path takes priority
when both paths are set; --alpha-cache requires --fxcache-path
(alignment guarantee). Original --mbp10-dir path unchanged for
non-cached runs.
4. Two new helper fns: load_alpha_cache (binary [u32 n] + [f32; n]
reader), load_snapshots_from_fxcache (FxCacheReader → Vec<SnapshotRow>
with synthesized bid/ask and alpha_logit/alpha_confidence from cache).
alpha_logits_cache.bin (7.6 MB, 1.97M f32 entries) is .gitignore'd —
regenerable from `cargo run -p ml-alpha --release --example
alpha_train_stacker -- --fxcache-path <FXC> --alpha-cache-out
config/ml/alpha_logits_cache.bin` (~2 min on RTX 3050 Ti).
Reproduction of this PASS verdict:
cargo run -p ml --release --example alpha_dqn_h600_smoke -- \
--fxcache-path /home/jgrusewski/Work/foxhunt/test_data/feature-cache/9297....fxcache \
--alpha-cache config/ml/alpha_logits_cache.bin \
--horizon 600 --n-episodes 1000
Total run time ~10s after fxcache load. Verdict + per-checkpoint KC
trajectory in config/ml/alpha_dqn_h600_smoke.json.
NEXT: Task 13 — scale to H=6000 (the production horizon). Per the plan,
PASS at H=600 unlocks H=6000.
Phase E.1 Task 12. Stabilized H=600 DQN smoke ran end-to-end on full
500K-snapshot data. All three preconditions PASS but the rvr gate FAILS:
Q_SPREAD_EMA = 35.54 ≥ 0.05 PASS
ACTION_ENTROPY_EMA = 1.91 ≥ 1.099 PASS
RETURN_VS_RANDOM_EMA = -2.58 ≥ 0.0 FAIL ← policy WORSE than random
EARLY_Q_MOVEMENT_EMA = 0.096 ≥ 0.01 PASS
rvr trajectory across 1000 episodes:
ep 50 | rollout_R= -11049 | rvr = -1.18 (near random)
ep 200 | rollout_R= -9620 | rvr = -0.97 (briefly improving)
ep 600 | rollout_R= -18140 | rvr = -2.03 (degrading)
ep 1000 | rollout_R= -18272 | rvr = -2.58 (deterministic-bad)
Random baseline at H=600 = -5185 mean, std=4953. Trained policy loses
3.5× worse than random.
Diagnostics performed:
reward_scale=10000 → rvr=-2.37 (no help)
alpha_m=0 (vanilla DQN, no Munchausen) → rvr=-2.44 (no help)
Root cause: "first-best-action lock-in" of linear Q + ε-greedy. DQN's
TD update only modifies Q[a] for the TAKEN action; with ε-decay, the
argmax action self-reinforces while other actions' Q stays frozen at
random Xavier init. Random policy samples all 9 uniformly → 11% chance
of "lucky" close-position at any step → exits bad trades. Trained
policy converges deterministic on one bad action → never exits.
Per plan: pivot to NoisyNet (Task 19) — parameter-space noise breaks
the lock-in. Alternative: wire alpha_logit from Phase 1d.3 stacker
(currently hardcoded to 0.0 placeholder) so the policy has actual
directional signal to work with.
Kill-criteria gate worked as designed — correctly flagged that linear
Q + ε-greedy on this env is insufficient without further intervention.
Memory note: project_phase_e1_h600_smoke_verdict.md (full analysis +
hypothesis tree + recommended next steps).
The kill-criteria producer, Munchausen target kernel, Rust launchers,
fit/baseline binaries, and their output JSON artifacts are *durable
infrastructure* of the alpha trading system (live across Phase E/F/G/...),
not milestone-scoped to Phase E specifically. Aligns with the earlier
`phase_e_isv_slots.rs` → `alpha_isv_slots.rs` rename rationale.
What was renamed:
Code files:
crates/ml/src/cuda_pipeline/phase_e_kill_criteria.cu → alpha_kill_criteria.cu
crates/ml/src/cuda_pipeline/phase_e_munchausen_target.cu → alpha_munchausen_target.cu
crates/ml/src/cuda_pipeline/phase_e_kernels.rs → alpha_kernels.rs
crates/ml/examples/phase_e_fit_fill_model.rs → alpha_fit_fill_model.rs
crates/ml/examples/phase_e_random_baseline.rs → alpha_random_baseline.rs
Artifacts:
config/ml/phase_e_fill_coeffs.json → alpha_fill_coeffs.json
config/ml/phase_e_random_baseline.json → alpha_random_baseline.json
Kernel function names:
phase_e_kill_criteria_compute_kernel → alpha_kill_criteria_compute_kernel
phase_e_munchausen_target_kernel → alpha_munchausen_target_kernel
Rust launcher names:
launch_phase_e_kill_criteria → launch_alpha_kill_criteria
launch_phase_e_munchausen_target → launch_alpha_munchausen_target
Static cubin names:
PHASE_E_MUNCHAUSEN_TARGET_CUBIN → ALPHA_MUNCHAUSEN_TARGET_CUBIN
Historical milestone tags in doc-comments ("Phase E.1 Task N (2026-05-15)")
are RETAINED — they record WHEN the work landed and what plan it
implemented, which doesn't change with the system-scoped rename.
Plus: ADDS the alpha_munchausen_target GPU smoke test in alpha_kernels.rs.
End-to-end validates the launcher + kernel against hand-computed expected
values: batch=2 with one terminal sample; expected targets [29.8, 1.1];
got match within 0.05 tolerance on RTX 3050 Ti. PROVES the Task 9/10
kernels actually run on GPU.
All affected references updated in:
- build.rs (kernel compile list)
- mod.rs (module registration)
- state_reset_registry.rs (4 RegistryEntry descriptions for slots 539-542)
- alpha_isv_slots.rs (slot table comment)
- docs/isv-slots.md (audit-doc cross-references)
Verified:
cargo test -p ml --lib alpha_kernels: 2/2 pass (including GPU smoke)
cargo test -p ml --lib state_reset_registry: 10/10 pass
cargo build -p ml --release --example alpha_fit_fill_model --example alpha_random_baseline: clean
Phase E.0 Task 7c. Ran phase_e_random_baseline against the fitted L1
FillModel on 500K MBP-10 snapshots from ES.FUT 2024-Q1. Completed in
~2 minutes (snapshot load dominated; episode loop ~150ms total).
Results:
mean reward = -5185.13
std reward = 4952.85
p05 = -13972.31
p25 = -7251.85
p50 (median) = -2804.56
p75 = -1787.90
p95 = -954.75 (best 5% of random episodes still lose)
kill threshold = +4720.57 (= mean + 2σ; E.1 DQN must exceed)
avg fills/ep = 139.22 (~1 fill every 4.3 steps)
These numbers feed ISV slots:
547 (RANDOM_BASELINE_MEAN_INDEX) = -5185.13
548 (RANDOM_BASELINE_STD_INDEX) = 4952.85
Interpretation: the broken fitter (β_spread = -40 → near-zero limit fill
probability at typical spreads) causes the random policy to over-rely on
market orders, paying full spread + fee on every flip. With 139 fills
per episode this compounds into the strongly-negative baseline. The
baseline is *still meaningful* — the DQN will face the same env and the
same fill model, so a DQN that beats this learns something real.
Open follow-up for Phase E.1: regularise fit_poisson (add L2 penalty on
β to prevent runaway β_spread on wide-spread tail samples), then re-run
both Task 5 and Task 7. Until then, the current baseline is the
operational reference point.
Phase E.0 Task 5c. Ran phase_e_fit_fill_model on the ES.FUT 2024-Q1 MBP-10
+ trade tape (5.2M trades, 3.9M MBP-10 events, 500K snapshots accumulated
at snapshot_interval=50 over a ~24-minute window). Total runtime ~80s.
Empirical fill rates within 60s window:
- bid_l1: 4.97% (matches L1 maker-side activity in trending market)
- ask_l1: 71.34% (high — most 60s windows see an aggressive buy)
Fitted L1 cloglog coefficients (all 5 features):
BID L1: β_0=-0.213 β_spread=-2.064 β_imbal=-0.099 β_ofi=-0.006 β_logτ=-0.286
ASK L1: β_0=+0.016 β_spread=-40.336 β_imbal=+0.041 β_ofi=+0.652 β_logτ=-0.055
Sanity (sign checks all pass):
- β_spread < 0 both sides (wider spread → fewer fills) ✓
- bid β_imbal < 0 (more bid stack → harder to get hit by sell) ✓
- ask β_ofi > 0 (buying pressure correlates with ask fills) ✓
- β_logτ < 0 both sides (quieter markets → slower execution) ✓
L1-only limitation: as documented in the binary header, the parser only
populates levels[0]; L2/L3 in the JSON are L1 with β_0 -= ln(L+1) attenuation.
Default --out-path bumped to config/ml/phase_e_fill_coeffs.json so future
re-runs land in the same committed location.
dqn-production.toml hard-coded three H100-tuned values that shadow
GpuProfile auto-detection: batch_size=16384, buffer_size=500K,
gpu_n_episodes=4096. After the L40S-default flip lands (prior commit
8b8bb1af7), workflow `train-ft8ph` deterministically OOMed at fold 0/1/2
with `build_next_states_f32` 4 GiB alloc — because the H100-sized
hyperparams.batch_size + buffer_size + gpu_n_episodes ate ~38 GB of the
L40S's 46 GB usable VRAM before the rollout step.
DqnTrainingProfile.apply_to() runs AFTER train_baseline_rl.rs populates
hyperparams from GpuProfile, so the production TOML always wins. All
three fields are `Option<...>` in the TOML schema — removing the lines
turns apply_to into a no-op for them, and the GpuProfile-detected values
flow through:
field | L40S | H100 | (was forced)
batch_size | 4096 | 8192 | 16384
buffer_size | 300K | 500K | 500K
gpu_n_episodes | 2048 | 4096 | 4096
Two pinned assertions in training_profile.rs::tests checked the old
contract `hp.batch_size == 16384`. Rewritten to assert
`hp.batch_size == baseline_batch_size` — locks the new contract that
VRAM-tuned values stay GpuProfile-sourced.
Lib suite 1016/0 green. Audit doc updated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
At threshold=0.5 the sampler produced 209M bars from 209M trade ticks (1:1
ratio, essentially per-tick) on workflow f5wnd today. Feature extraction hit
54Gi/56Gi memory limit — near OOM. The default was wrong: with EWMA bypassed
(per 1aaf94306), threshold=0.5 contracts of imbalance is below the natural
ES.FUT trade size, so every trade fires a bar.
threshold=20 produces ~5-6M bars matching the volume-bar density baseline
(5.74M bars at 100 contracts). Math: 209M / 5.74M = 36×, so threshold needs
0.5 × 36 ≈ 18 → round to 20.
Three files updated atomically per feedback_no_partial_refactor:
- config/training/dqn-production.toml: 2.5 → 20.0 (wgdc8 left it at 2.5)
- infra/k8s/argo/train-template.yaml: workflow default 0.5 → 20.0
- infra/k8s/argo/train-multi-seed-template.yaml: workflow default 0.5 → 20.0
The 1:1 bar:tick observation alongside the price-continuity proof (22 jumps
out of 209M = ~1e-7 — front-month filter works) confirms the front-month
fix on this branch (6c1ab8850 + 3b5f17913) is correct. Threshold was the
remaining mistuning.
Cache-key implication: changing the threshold invalidates all existing
fxcache files (per the cache-key architectural fix). Next dispatch on this
branch will regenerate fxcache from scratch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hypothesis test branch off sp19-20-wr-first HEAD 235e83842 (Phase 1 producers
landed, reward semantics unchanged via alpha=0 / per_bar_hold_reward=0
placeholders).
Goal: decide whether bar-resolution drives the ~46% WR plateau pinned across
16+ SP runs. wgdc7 produced ~2880 bars/day (~8 sec/bar) on ES.FUT volume
data. 5× threshold should give ~600 bars/day (~30-60 sec/bar). If WR moves
materially → resolution IS the bottleneck → SP21 hybrid justified. If WR
stays at 46% → resolution is NOT the bottleneck → SP21 needs different
framing (multi-instrument, additional features, or accept data ceiling).
Single-config change. No code changes. Reverts cleanly by deleting branch.
See project_bar_resolution_is_actual_architecture.md for context.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per Class A audit-fix Batch 4-A (deferred from P1-wiring/P1-producer due to
audit-doc errors). Fixes 2 of 4 deferred items; Batch 4-B handles plan_threshold
floor + MIN_HOLD_TEMPERATURE in a separate commit.
Item 1: DD saturation floor (the upper end of the DD ramp at trade_physics.cuh:154
in apply_margin_cap, NOT line 548 as the audit doc claimed — that line is a
magnitude action constant; the actual saturation floor lives in apply_margin_cap)
- NEW slot DD_SATURATION_FLOOR_ADAPTIVE_INDEX=458
- Producer dd_saturation_floor_update_kernel.cu — p75(per-env DD_MAX) × 1.5
via Welford `mean + Z_75 × sigma` estimator with `max(p75, mean)` robustness
guard, mirrors P0-A REWARD_POS_CAP producer pattern (Pearl-A bootstrap +
Welford α=0.01)
- Cold-start fallback: 0.25f (DD_SATURATION_FLOOR_DEFAULT in state_layout.cuh)
- Bounds: [0.10, 0.50] (Category-1 dimensional safety)
- Distinct from SP15_DD_THRESHOLD_INDEX=421 (the SP15 quadratic DD-penalty
*trigger* threshold, a *lower* bound; this slot is the *upper* end of the
linear position-size scaling ramp dd_scale = max(0.05, 1.0 − dd_frac/floor))
- Threaded `isv_signals_ptr` into `apply_margin_cap` with NULL-tolerant
cold-start fallback to DD_SATURATION_FLOOR_DEFAULT
- 4 oracle tests (Pearl-A bootstrap, no-DD guard, bounds clamp, Welford EMA)
Item 2: Legacy compute_drawdown_penalty path → Case A (DELETED)
- Decision rationale: SP15's quadratic asymmetric DD penalty
(compute_sp15_final_reward_kernel.cu:154 via sp15_dd_penalty helper) runs
unconditionally as a post-modifier on the SP11-composed reward with
ISV-driven λ_dd (slot 420) and DD threshold (slot 421). Layering the legacy
linear-ramp penalty inside the SP11 composer on top of the SP15 quadratic
creates double-counting of DD shaping — exactly the code-smell the Class A
audit was designed to eliminate. Per `feedback_no_legacy_aliases.md` and
`feedback_no_partial_refactor.md`.
- Atomic deletion across:
- `compute_drawdown_penalty` device function (trade_physics.cuh)
- Single call site at experience_kernels.cu:3822
- `dd_threshold` and `w_dd` kernel arguments
- `w_dd` Rust config field (gpu_experience_collector.rs +
trainers/dqn/config.rs DQNHyperparameters)
- `w_dd` profile section + dispatch (training_profile.rs RewardSection,
OptimizableParameterRanges, FixedRewardParameters, ParamLookup
dispatch, profile→hyperparam mapping, test assertion)
- `w_dd *= rki` risk-intensity multiplier (config.rs)
- `w_dd` TOML keys (dqn-hyperopt.toml × 2, dqn-localdev.toml,
dqn-production.toml, dqn-smoketest.toml)
- Stale doc comments on hyperopt/adapters/dqn.rs + config.rs
risk_intensity field
- `config.dd_threshold` SURVIVES (still consumed by `launch_sp15_dd_state`
as the dd_budget for DD_PCT scaling). Documented in field comment.
ISV_TOTAL_DIM: 458 → 459 (Item 1 adds 1 slot; Item 2 is pure deletion)
Cumulative WR-plateau fix series (this is commit 7):
- Class C bug 1 + P0-B (8f218cab2)
- P0-C (316db416b)
- P0-A (394de7d43)
- P1 wiring (c4b6d6ef2) — 1 of 4 wireable
- P0-A downstream (657972a4b)
- P1 producer (87d597d5d)
- audit-fix 4-A (this commit)
Verification: 16 sp14_oracle_tests pass (incl. 4 new), 36 sp15_phase1_oracle_tests
pass, 12 sp14_isv_slots layout tests pass, 4 state_reset_registry tests pass
(every-FoldReset-arm-has-dispatch contract holds), workspace cargo check clean.
Per feedback_isv_for_adaptive_bounds + feedback_no_partial_refactor +
feedback_no_legacy_aliases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Smoke and localdev profiles previously used data_source = "ohlcv", divergent
from production (mbp10). Mismatch silently produced cache-key collisions in
the SHA256 hash path: smoke fxcache could not be loaded by production-shape
training without an explicit override. Local-dev fxcache regen also required
remembering to pass --data-source ohlcv to match.
Globalize mbp10 as the default everywhere it isn't deliberately overridden:
- config/training/dqn-smoketest.toml: data_source = "mbp10"
- config/training/dqn-localdev.toml: data_source = "mbp10"
- training_profile.rs: doc Default → "mbp10"
- trainers/dqn/config.rs: Default impl → "mbp10"
- hyperopt/adapters/dqn.rs: default → "mbp10"
- examples/precompute_features.rs: doc updated
- fxcache.rs / feature_cache.rs: discover_and_load + cache-key tests
use "mbp10" arguments
- docs/dqn-wire-up-audit.md: new entry per Invariant 7
Documentation strings retained "ohlcv" only where they document the two
available choices (config.rs:946, training_profile.rs:83).
Local fxcache regenerated to v6 mbp10:
test_data/feature-cache/13c0b086a975cc7e2384377a2cd0e97738c9410292fcfecb5807c29bf885cb48.fxcache
(175874 bars, 55 MB, OFI_DIM=32). Stale v5 ohlcv fxcache untracked
from git index (already gitignored post-79578bbaf).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the convergence guardrail: every per-epoch HEALTH_DIAG metric is
checked against the bands in config/metric-bands.toml; N consecutive
warn-band epochs emit a tracing::warn; 2N consecutive error-band epochs
return Err(CommonError::RegressionDetected{...}) cleanly from the
training loop, which propagates to the train_baseline_rl subprocess
exit code (no libc::raise — clean Rust error path).
Wire-points:
- New module: crates/ml/src/trainers/dqn/trainer/monitoring.rs
- MetricBands {warn_low, warn_high, error_low, error_high}
- BandSettings {consecutive_epochs_for_warn, consecutive_epochs_for_error}
- MetricBandsRegistry: load_from_toml + update_and_check
- TerminationReason {RegressionWarn, RegressionError}
- NaN treated as out-of-band (consecutive++; never resets streak)
- Unknown metrics return None (silent OK per Invariant 7 audit)
- crates/common/src/error.rs: new CommonError::RegressionDetected variant
carrying {metric, value, band, consecutive}
- crates/ml/src/trainers/dqn/trainer/constructor.rs: load
config/metric-bands.toml at trainer init; warn-only on missing file
(backward compat for environments without the config)
- crates/ml/src/trainers/dqn/trainer/training_loop.rs: harvest per-epoch
metrics (parallel emit alongside HEALTH_DIAG), feed each through
registry.update_and_check; on Some(TerminationReason::RegressionError)
emit final HEALTH_DIAG[N]: TERMINATED_BY_REGRESSION line and return Err
- services/trading_service/src/error.rs: minimal handler for the new
CommonError variant (existing pattern)
Validation:
- 8 unit tests in monitoring::tests pass (band logic, NaN, warn-only
behaviour, error-streak threshold, unknown-metric, invalid TOML)
- regression_detection GPU smoke (3.19s): trainer with intentionally
narrow train_loss error band [0, 1e-9] self-terminates at epoch 5
after 6 consecutive error-band epochs; final HEALTH_DIAG line emits
TERMINATED_BY_REGRESSION with metric/value/consecutive/band fields
- multi_fold_convergence smoke (650s, --release): all 3 folds train
to completion, all 3 checkpoints saved, no false-positive
termination on the populated metric bands. Per-fold best train
Sharpe: F0=-9.7831 (bit-baseline), F1=25.8272, F2=39.2687. F1/F2
on the lower end of observed noise distribution
({74.56, 61.10, 71.53, 25.83} for F1; {88.20, 61.57, 65.96, 39.27}
for F2) but training healthy throughout: aux clauses fire every
epoch, sharpe_ema recovers from F0 collapse (-9.78 → +14.8 by start
of F2), no regression detection trips.
config/metric-bands.toml populated for the metrics emitted by
HEALTH_DIAG today (avg_q_value, train_loss, val_sharpe, train_sharpe,
aux_next_bar_mse, aux_regime_ce, isv_* slot EMAs, sharpe_ema, etc.).
Bands derived from current cleanroom smoke + permissive defaults
where only one sample exists; populate-metric-bands-from-runs.py will
tighten them after Plan 5 Task 5's multi-seed pass produces real
distributions.
Constraints honoured: GPU-only in hot path (band check is CPU-side
post-HEALTH_DIAG, off the captured graph); no atomicAdd; no stubs;
no // ok: band-aids; no tuned constants beyond the toml-loaded bands;
no .unwrap() introduced; cargo check clean at 11 warnings (workspace
baseline preserved, plus ml-dqn pre-existing 1 warning).
Audit doc: new row added documenting monitoring.rs module, the
CommonError variant, the training_loop wire-point, and the design
choice that band-checks run AFTER HEALTH_DIAG emit (not before) so
the diag log already reflects the metric values that triggered any
termination.
Plan 5 T1 (multi-seed harness) landed at c6634254e+47c8b783c; T2
(this) gives the regression hard-stop that the multi-seed final
pass (T5) consumes to bail out early on bad seeds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces random τ ∈ U(0,1) sampling with `FIXED_TAUS = [0.05, 0.25, 0.50,
0.75, 0.95]`. Kernel-side `IQN_NUM_QUANTILES` macro 32 → 5; `GpuIqnConfig::
default().num_quantiles` 32 → 5. Construction-time τ broadcast (option B2)
populates `online_taus` / `target_taus` / `cos_features` once via
`clone_htod`; both online and target IQN forwards plus the CVaR cold path
read this static buffer. The Philox-driven `iqn_sample_taus_kernel` deleted
along with its only Rust consumer (in `compute_cvar_scales`); the
`rng_step` Philox seed counter also gone. Action ranking in the IQN
inference kernel switched from mean-over-quantiles to MEDIAN
(`q_acc[a] = q_val` only when `t == IQN_MEDIAN_INDEX = 2`); the off-median
positions feed four new ISV diagnostic slots.
Four new ISV slots tail-appended:
IQN_Q_P05_EMA_INDEX = 99 (mean |Q| at τ=0.05, EMA)
IQN_Q_P25_EMA_INDEX = 100 (τ=0.25)
IQN_Q_P75_EMA_INDEX = 101 (τ=0.75)
IQN_Q_P95_EMA_INDEX = 102 (τ=0.95)
Median (τ=0.50) intentionally skipped — already in greedy-Q diagnostic.
Fingerprint pair shifted 97→103, 98→104; ISV_TOTAL_DIM 99→105.
Layout fingerprint: 0x3e21acecd922e540 → 0x5789155b683ab59c.
New kernel `iqn_quantile_ema_kernel.cu` (4-block × 256-thread shmem-reduce,
no atomicAdd) reads `save_q_online [TBA, B*Q]` and EMA-updates the four
slots. Launched from `training_loop.rs` per-step alongside
`launch_h_s2_rms_ema`. StateResetRegistry extended with 4 FoldReset
entries (cold-start 0.0).
Hyperparam plumbing: `hyperparams.num_quantiles` and
`DQNConfig::iqn_num_quantiles` pinned to `FIXED_TAUS.len()` at the
`GpuIqnConfig` construction site in `fused_training.rs::new` and
`trainer/constructor.rs`. Legacy fields stay for compat; production /
hyperopt configs (dqn-production.toml, DQNHyperparameters defaults)
aligned to 5.
Adam state for IQN params auto-resizes via `m_buf`/`v_buf` sizing
through `total_params + cublas_pad`. **Checkpoint break** — IQN head
parameter shapes change with `num_quantiles`; new fingerprint hash
fails-fast at constructor load on pre-Task-3 checkpoints.
Smoke tests:
- New `iqn_multi_quantile_heads_produce_monotonic_estimates` (1.23s on
RTX 3050 Ti): asserts ISV[99..103) finite + non-zero + spread > 1e-6
after 1 epoch — PASS (Q_p05=0.0187 Q_p25=0.0200 Q_p75=0.0193
Q_p95=0.0190).
- `multi_fold_convergence` (606.50s, 3 folds × 5 epochs): all 3 fold
checkpoints written; per-fold best train Sharpe -8.17 / 74.24 / 63.44
at epochs 2 / 4 / 2 (mean 43.17 vs 2c.3c.6 baseline mean 23.43 — folds
1+2 substantially up, fold 0 down -16 points; absolute-mean comfortably
above the plan's 3.8 floor). No NaN/Inf, no panic.
cargo check clean at 11 warnings (baseline preserved); cargo build
compiles 61 cubins (was 60; +iqn_quantile_ema, -nothing — the old
sample_taus kernel was inside iqn_dual_head_kernel.cu, not a separate
cubin file).
Files touched: 14 modified (`iqn_dual_head_kernel.cu`, `iqn_cvar_kernel.cu`,
`gpu_iqn_head.rs`, `gpu_dqn_trainer.rs`, `build.rs`, `state_reset_registry
.rs`, `training_loop.rs`, `constructor.rs`, `fused_training.rs`,
`config.rs`, `dqn-production.toml`, `smoke_tests/mod.rs`,
`docs/dqn-wire-up-audit.md`, `dqn-production.toml`) + 2 new (`iqn_quantile
_ema_kernel.cu`, `smoke_tests/iqn_quantile_monotonicity.rs`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plan 1 Task 1. Creates the five audit docs plus config/metric-bands.toml
that track Invariants 2, 7, 8 per the DQN v2 spec, and extends the
pre-commit hook with two checks:
- component-adding commits must touch an audit doc (Invariant 7)
- added code may not contain TODO/FIXME/XXX/HACK/TBD/unimplemented!/
todo! markers (Invariant 9)
Tests: manually verified by staging a TODO-marked file; commit
rejected with the correct error message.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mass deletion of the "v7 gem" reward terms identified in the Phase 1
inventory as category errors — each one rewarded an outcome-adjacent
behavior instead of encoding the underlying physics, and each
empirically hurt validation metrics more than it helped training
stability.
Deleted from experience_kernels.cu and all plumbing (Rust configs,
launch args, hyperopt logs, TOML entries):
* order_credit_weight - reward redundant with compute_tx_cost
(order_type_idx already differentiates
fills by order type)
* risk_efficiency_weight - reward double-counted drawdown penalty
asymmetrically (only on winners)
* urgency_credit_weight - reward was vol-normalized unrealized P&L,
pure rename of core return
* commitment_lambda - triple-counted churn + tx_cost
* w_dsr - kernel wrote DSR EMA but no longer added
to reward (dead); removed the EMA
bookkeeping too
* dsr_eta - kernel arg for the deleted DSR EMA
* position_entropy_weight - rewarded action-bucket diversity
regardless of outcome; histogram buffer
+ zero-init removed too
* exit_timing_weight - already inactive (used raw_next future
price, comment-deleted earlier)
* ofi_reward_weight - dead plumbing; OFI already passed as
feature through state[OFI_START..]
* opportunity_cost_scale - penalized flat when Q-gap wide;
redundant with Q-values themselves
Kernel arg count: experience_env_step_batch shrank from ~55 to ~45 args.
Rust-side config surface reduced correspondingly.
Results on E1 smoke test (20-epoch):
BEFORE any Phase 2 work:
Val Sharpe -120 to -150, MaxDD 10-15%, Sharpe_raw -0.39
AFTER reward_noise + Kelly (both envs) + urgency + this sweep:
Val Sharpe -17 to -22 (7× better)
Val MaxDD 0.27% (40× better)
Val Sharpe_raw ~-0.09 (4× better)
Training Sharpe_raw ~0 (stabilized from ±20 swings)
Final q_gap 0.1712 (highest yet, collapse mechanism fine)
The extreme train-Sharpe swings (+17 one epoch, -13 next) were not
learning dynamics — they were shaping-term noise. Core reward (P&L +
drawdown + churn + holding + tx_cost + Kelly physics cap) gives training
metrics that actually reflect what the model does.
Inventory doc (docs/superpowers/specs/2026-04-21-phase1-reward-inventory.md)
extended with a "better-form taxonomy" section: every deleted gem has
a correct layer it belongs to (physics, feature, diagnostic, gradient-
level regularization — not reward). Kelly cap and Q-target smoothing
are already relocated; others are scheduled per the taxonomy's P1/P2/P3
priority list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 2 second relocation: the kelly_sizing_weight reward penalty
(experience_kernels.cu:1727-1746 — penalized deviation from Kelly-optimal
sizing) is deleted. Kelly is now a physics constraint in trade_physics.cuh:
the environment refuses to let the agent over-lever, not the reward
scoring the agent for matching a formula.
New helper in trade_physics.cuh (shared device function, reusable by
the forthcoming unified env kernel):
kelly_position_cap(win_count, loss_count, sum_wins, sum_losses,
max_position, safety_multiplier)
Applied in experience_kernels.cu between margin cap and execute_trade,
with health-coupled safety multiplier:
safety = 0.5 + 0.5 × health
- health=1 (healthy): full Kelly — trust the learned policy
- health=0 (collapsing): half Kelly — constrain when decisions less
reliable
Cold-start warmup (critical — otherwise balanced priors yield kelly_f=0
until real trades accumulate, starving Q-learning):
maturity = min(1.0, total_trades / 10)
effective_kelly = maturity × kelly_f + (1 - maturity) × 0.5
Early on (0 trades): cap dominated by 50% floor.
As real trades accumulate (10+): pure data-driven Kelly.
Validation env (backtest_env_kernel.cu) does NOT yet get the Kelly cap —
that requires extending its portfolio state or adding a separate
kelly_stats buffer, which naturally belongs in the Phase 3 unified env
kernel refactor. The current asymmetry is a KNOWN temporary — training
is constrained, validation is not — and will be resolved when both
kernels share the same env_step() device function.
Also completes removal of kelly_sizing_weight from all plumbing:
- experience_kernels.cu: kernel arg deleted
- gpu_experience_collector.rs: launch arg, config field, default
- training_loop.rs: hyperparam propagation
- config.rs: field, default, intensity clamp (with tombstone)
- hyperopt/adapters/dqn.rs: log reference
- config/training/*.toml (6 entries across 4 files): orphan configs
(none were wired to a profile parser field)
Verification:
- cargo check -p ml --lib clean
- E1 smoke test passes: final epoch q_gap=0.1109, health=0.51 (warmup
floor of 0.5 gives early exploration enough room; floor of 0.25
was too tight and failed at q_gap=0.0496)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 2 kick-off from the env-unification design. First relocation: the
reward_noise_scale field that perturbed training rewards is deleted, and
its regularization effect moves to the correct layer — Q-target label
smoothing in c51_loss_kernel.cu — now health-coupled rather than fixed.
Before:
- reward += pseudo_noise × max(|reward| × 0.05, 0.01) (in env reward path)
- Q-target label smoothing = fixed LABEL_SMOOTHING_EPS = 0.01
After:
- Reward untouched by noise. Core reward = actual outcome + aligned penalties.
- Q-target label smoothing eps_eff = 0.02 × (1 − health) read from ISV[12]
- health=1 (healthy): eps_eff=0, sharp targets preserved
- health=0.5: eps_eff=0.01, matches old fixed behavior at mid-health
- health=0 (collapsing): eps_eff=0.02, maximum regularization prevents
overcommitment to the collapsed distribution
Why health-coupled:
Same insight as the distillation SAXPY fix — every fixed kernel scalar is
a temporal-coupling candidate when we have the ISV pinned buffer available.
Regularization strength should scale INVERSELY with network health: it's
most needed exactly when things are falling apart.
Files touched:
- c51_loss_kernel.cu: LABEL_SMOOTHING_EPS const replaced with
LABEL_SMOOTHING_BASE + in-kernel health read from isv_signals[12]
- experience_kernels.cu: deleted reward noise block + kernel arg
- gpu_experience_collector.rs: dropped launch .arg + config field + default
- training_loop.rs: dropped hyperparam propagation
- config.rs: deleted field + intensity clamp + default (with tombstone)
- hyperopt/adapters/dqn.rs: dropped log reference
- config/training/*.toml (4 files): dropped orphan reward_noise_scale
entries (none were being parsed — the profile parser had no field)
Verification:
- `cargo check -p ml --lib` clean
- E1 smoke test passes: final q_gap=0.1190, health=0.52 (health-coupled
smoothing at ~mid-health matches old fixed behavior, collapse-prevention
mechanism intact)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Atom warm-start: bitonic sort rewards → quantile positions → write to
atom_positions_buf as initialization. Existing SGD optimizer refines.
Atoms start where reward mass actually is instead of uniform [-50,+50].
Robust PopArt: median/IQR normalization from sorted rewards replaces
Welford mean/var. More robust for bimodal distribution (many ±0.1
micro-rewards + few ±5.0 trade exits). Conditional: popart_robust=true.
Both reuse the same bitonic sort (~14ms per epoch, amortized).
gather_quantiles kernel extracts positions. extract_median_iqr reads
Q25/median/Q75 from sorted buffer.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Counterfactual: Hold(1)/Flat(3) direction mirror maps to self — trivial.
Now falls through to magnitude CF for Hold/Flat instead of wasting a
replay buffer slot on same-action same-reward experiences.
reward_noise_scale: 0.05→0.0 (dense micro-rewards are already noisy,
adding 5% label noise destroys the per-bar signal).
Added micro-reward weight config fields (price_confirm_weight=0.5,
book_aggression_weight=0.3, hold_quality_weight=0.2, micro_reward_temp=3.0,
holding_cost_rate=0.0001) — defined in TOML, kernel plumbing next session.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removed the legacy non-branching Sequential q_network and target_network
from DQNAgent. These were never used (branching+dueling always active)
but allocated VRAM and ran noise resets every step. -190 lines.
Config tuning for dense micro-reward system:
- n_steps: 5→1 (TD(0), micro-rewards cancel over n>1)
- tau: 0.007→0.01 (faster target tracking for TD(0))
- c51_alpha_max: 0.5→1.0 (full C51, PopArt handles normalization)
- curiosity_weight: 0.1→0.0 (dense micro-reward replaces curiosity)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three signal-killing issues fixed:
1. Rank normalization DISABLED — was double-normalizing with PopArt,
destroying magnitude difference between micro-rewards (±0.1) and
trade exits (±5.0). PopArt alone preserves relative magnitude.
2. n_steps 5→1 (TD(0)) — dense micro-rewards alternate ±0.1 each bar.
With n=5, they cancel out over 5 bars. TD(0) preserves the per-bar
signal that the temporal pipeline needs to learn from.
3. per_alpha 0.6→0.3 — lower = more uniform PER sampling. Dense micro-
rewards have tiny TD-errors (easy to predict), so high alpha ignores
them. Lower alpha ensures micro-reward experiences get sampled.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Linear scaling rule (2x batch → 2x LR) doesn't hold for DQN+PER with
major architectural changes (Hold action, OFI embed, wider attention).
The model needs stability to learn the new action space, not speed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Linear scaling rule: 2x batch → 2x LR to maintain effective update
magnitude. Tau increased to compensate for fewer steps/epoch (target
network tracks faster). H100 VRAM: 41GB free at B=8192, B=16384 adds
~4GB — easily fits.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Default was hardcoded as 64 in train_baseline_rl.rs and evaluate_baseline.rs,
overriding the config default of 32. Added num_quantiles=32 to production
config so it's explicit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Auto-sizer inflated replay buffer to 15.8M entries on H100 (45% of 80GB VRAM).
The PER prefix scan inside the CUDA graph processes ALL 15.8M entries every step,
dominating step time. With buffer_size=500K, the scan processes 500K entries instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
At 16384, each GEMM saturates 97% of 132 SMs — no room for multi-stream
parallelism. At 8192, GEMMs use ~50% SMs, allowing branch streams and
aux parallelism to fill idle SMs. 2x more steps/epoch but each ~2x faster.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removed: enforce_hold(), min_hold_bars from config/kernels/backtest.
Added: holding_cost_rate (inventory penalty), churn_threshold_bars +
churn_penalty_scale (graduated flip penalty) to reward in
experience_env_step and backtest kernels.
The model learns optimal hold timing from cost signals:
- Per-trade tx cost prevents churning (existing)
- Inventory penalty makes large positions expensive to hold
- Churn penalty graduates cost for rapid flips
- Temporal attention learns when holding cost > expected profit
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
min_hold=10 allowed ~1200 trades/day — costs destroyed the edge.
min_hold=50 (~6.5 min) limits to ~230 trades/day max.
Adaptive extension now scales 0-2× base (was hardcoded 0-8 bars).
With base=50: range is 50-150 bars depending on ISV stability.
Losing positions (negative reward_ema) get base only (50 bars).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The warp-parallel (32,1,1) change to c51_loss_reduce caused the same
hang as isv_feature_gate: changing block_dim inside graph_mega alters
CUDA Graph node structure on Hopper's TMA scheduler. Must stay (1,1,1).
Also: batch_size 8192→16384, gpu_n_episodes 1024→4096, num_atoms 51→52
to align h100.toml with dqn-production.toml.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mbp10_data_dir and trades_data_dir uncommented in localdev config.
Doc comments updated: 20 microstructure features always appended.
OFI is no longer optional — it's core to the model's feature set.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
