From 1ce99efc5353de20c9fe7acc9dfa33b616db5413 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Wed, 22 Apr 2026 09:23:52 +0200 Subject: [PATCH] =?UTF-8?q?plan(policy-quality):=20Phase=202=20implementat?= =?UTF-8?q?ion=20=E2=80=94=20synthesis=20of=204=20tracks?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Consolidates Phase 1 triage findings from Tracks 1-4 into an 11-task Phase 2 plan at docs/superpowers/plans/2026-04-21-policy-quality-phase2.md. Task inventory: - 2.0 Per-component gradient decomposition (H4 keystone diagnostic) - 2.1 H4 fix — magnitude-head gradient starvation (decision tree from 2.0) - 2.2 H10 fix — stable argmax tie-break at eval - 2.3 DELETE R5 micro-reward - 2.4 DELETE R6 loss-aversion; relocate neg-tail to C51 target smoothing - 2.5 Wiring-bug sweep (7 bugs: C1 fire, epsilon gen_range, if !true, sigma_mean scale, fold-boundary reset, stale docstring, C5 ISV null) - 2.6 E4 entropy-reg DELETE-or-KEEP (data-driven, post-2.0) - 2.7 C4 adaptive grad-clip ablation + DELETE-or-KEEP - 2.8 L40S validation run — all 4 tracks re-measured - 2.9 Mandatory-gate verification + phase3-results.md - 2.10 Tag policy-quality-phase2-complete (and policy-quality-v1 if soft pass) Matches Phase 0/1 plan formatting (checkbox steps, concrete file paths, code snippets, bash commands, per-task commit templates). References project standing rules (no quickfixes, no stubs, no atomic-adds on hot paths, no feature flags, no hiding errors) and the pinned-readback pattern from Task 0.4 (commit bb42c9963). --- .../plans/2026-04-21-policy-quality-phase2.md | 1502 +++++++++++++++++ 1 file changed, 1502 insertions(+) create mode 100644 docs/superpowers/plans/2026-04-21-policy-quality-phase2.md diff --git a/docs/superpowers/plans/2026-04-21-policy-quality-phase2.md b/docs/superpowers/plans/2026-04-21-policy-quality-phase2.md new file mode 100644 index 000000000..fe91dbc02 --- /dev/null +++ b/docs/superpowers/plans/2026-04-21-policy-quality-phase2.md @@ -0,0 +1,1502 @@ +# Policy Quality Phase 2 — Implementation Plan (Synthesis of 4 Tracks) + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** land every fix / deletion / instrumentation gap surfaced by Phase 1 Tracks 1–4, then re-validate on L40S against the design-spec §2.1 mandatory gates. Close sub-project A with `policy-quality-phase2-complete` if all gates pass; otherwise iterate per design spec §2.3. + +**Architecture:** main-branch only, per design spec §3. Commits incrementally — one per task unless related — each commit leaves smokes green. Each task has its own validation step before commit; the task-level commit message references the triage finding it closes. + +**Scope:** synthesised from +- `docs/superpowers/specs/2026-04-21-policy-quality-track1-triage.md` (magnitude: H4 + H10 CONFIRMED) +- `docs/superpowers/specs/2026-04-21-policy-quality-track2-triage.md` (reward: R5 / R6 DELETE, R7 already-removed docstring cleanup) +- `docs/superpowers/specs/2026-04-21-policy-quality-track3-triage.md` (controllers: 0 load-bearing, C4 candidate-for-delete, C1 wiring bug) +- `docs/superpowers/specs/2026-04-21-policy-quality-track4-triage.md` (exploration: E2 instrumentation TUNE, E4 delete-candidate) +- `docs/superpowers/specs/2026-04-21-policy-quality-design.md` §6 (Phase 2 structure), §2.1 (mandatory gates), §7 (validation gate) +- `docs/superpowers/specs/2026-04-21-policy-quality-baseline-metrics.md` (reference values to beat) + +**Baseline:** HEAD `0611d32b0`, tag `policy-quality-baseline` (Phase 0 exit). Any task that breaks main → `git reset --hard policy-quality-baseline` and replan. + +**Tech Stack:** Rust 1.85 (workspace crates `ml`, `ml-dqn`, `ml-core`), CUDA 12.4 kernels via nvcc (cubin cached), Argo Workflows on Scaleway Kapsule, L40S (sm_89) GPU pool, fxcache binary feature cache, `#[ignore]`-gated smoke tests under `crates/ml/src/trainers/dqn/smoke_tests/`. + +**Budget:** ~1.5 days coder time + ~3 hrs L40S validation (one full 6-fold × 50-epoch run after Task 2.8, up to 2 more iterations per design spec §2.3). + +--- + +## Project rules this plan follows (standing feedback memory) + +Before writing any code, reread the corresponding feedback file: + +- **No quickfixes.** Every issue gets the proper fix. Don't relax thresholds to make a test pass. → `~/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/feedback_no_quickfixes.md` +- **Fix everything — zero tolerance.** If a task surfaces additional broken wiring, fix it or open a follow-up task; do not paper over. → `~/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/feedback_fix_everything.md` +- **No hiding errors.** `Result` + `tracing::warn!` or propagate. No silent `.unwrap_or(default)` on a load-bearing path. → `~/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/feedback_no_hiding.md` +- **No feature flags / no `enable_*` booleans.** Wire for real or delete the slot; conditional compilation and runtime `if enable_…` guards are banned. → `~/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/feedback_no_feature_flags.md` +- **No stubs.** Dead slots must be deleted, not parked behind `// TODO: wire this`. → `~/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/feedback_fix_everything.md` +- **No atomicAdd on training hot paths.** Use per-sample arrays + host-reduce — the pattern established by Task 0.5 (trail counts) and Task 0.8 (reward-contrib). AtomicAdd is acceptable only for epoch-boundary debug counters already outside the CUDA graph capture region. When in doubt, per-sample + host-reduce. → `~/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/feedback_gpu_cpu_roundtrip.md` (adjacent rule: no GPU→CPU roundtrips on the step hot path — these per-sample reductions are epoch-boundary only). +- **Commit after each task; push only when stable.** Don't push per-task unless the task closes a user-visible gap. Push at clean multi-task boundaries (e.g. after Task 2.5 bundle, after Task 2.8 validation run kick-off). → `~/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/feedback_push_before_deploy.md` +- **Pinned-readback pattern for per-branch grad norms.** Reference implementation at commit `bb42c9963` (`diag(policy-quality): Task 0.4 — per-branch grad norm via pinned readback`). +- **Trust code, not comments.** Stale docstrings referencing deleted terms are foot-guns (Track 2 R7 finding). Delete them. + +--- + +## Cross-cutting concerns surfaced during synthesis + +Three concerns span multiple tracks and shape task ordering: + +1. **Per-component gradient decomposition is the keystone measurement.** Track 1 H4 fix depends on it (which loss component starves magnitude?). Track 4 E4 delete-or-keep verdict depends on it (does C51 entropy actually reach magnitude?). Track 2 R5 TD-propagation diagnostic is similarly downstream. **Task 2.0 lands first.** Every downstream decision that previously said "blocked on per-component grad split" unblocks after Task 2.0. + +2. **H4 (gradient starvation) and H10 (argmax tie-break) are cause-and-effect, not independent.** The Track 1 triage explicitly flags H10 as "the observable consequence of H4." Fix H4 first (Task 2.1); H10 is then a safety net (Task 2.2). If H4's fix makes magnitude Q-values spread genuinely, H10 may self-resolve in smoke data — but we still land the tie-break patch so future regressions can't re-produce the symptom. + +3. **Several L40S-dependent verdicts are still PENDING** (R2 PopArt warmup zero, C2/C3/C5 regime-stability gating, E3 fold-3 entropy anomaly, E2 anneal-effective σ). Task 2.8's L40S run resolves all four in one shot. Do NOT guess at them in earlier tasks — wait for the real-health-signal run. + +--- + +## Task inventory (11 tasks) + +| # | Task | Category | Estimated LOC | Depends on | +|---|---|---|---|---| +| 2.0 | Per-component gradient decomposition (IQN/CQL/C51/Ens → magnitude) | instrumentation | ~60 | — | +| 2.1 | H4 fix — rebalance gradient flow to magnitude head | behavioral | ~40 decision-tree-driven | 2.0 | +| 2.2 | H10 fix — stable argmax tie-breaking at eval | behavioral | ~25 | — | +| 2.3 | DELETE R5 micro-reward path | code removal | ~-150 | TD-propagation diagnostic (this plan extends Task 2.0) | +| 2.4 | DELETE R6 loss-aversion, relocate neg-tail compression to Q-target smoothing | behavioral + code removal | ~-30 +20 | — | +| 2.5 | Wiring-bug sweep (7 bugs from Tracks 1/3/4) | correctness | ~50 | — | +| 2.6 | E4 entropy-reg DELETE-or-KEEP decision | data-driven | 0 or ~-30 | 2.0 + 2.1 | +| 2.7 | C4 adaptive grad-clip ablation + DELETE-or-KEEP | data-driven | 0 or ~-80 | — | +| 2.8 | L40S validation run — all Phase 1 triages re-measured with fixes applied | validation | 0 | 2.0–2.7 landed | +| 2.9 | Mandatory-gate verification + phase3-results.md | validation | 0 | 2.8 | +| 2.10 | Tag `policy-quality-phase2-complete` + optional `policy-quality-v1` | release | 0 | 2.9 pass | + +--- + +## Task 2.0: Per-component gradient decomposition for magnitude head (H4 precursor) + +**Why first:** Track 1 H4 is CONFIRMED but the fix choice depends on *which* loss component (IQN / CQL / C51 / Ens) sends zero gradient to the magnitude branch. The H4 fix candidates diverge sharply: +- All four zero → architecture issue (direction-conditioning on `w_b1fc` or init scale) +- One component dominates direction → rebalance per-component weights (IQN=60%, CQL=25%, C51=10%, Ens=5% budgets are tunable) +- Gradient arrives but magnitudes still converge → H9 territory (data favors Quarter) + +Task 2.0 produces the data to pick the right branch of Task 2.1's decision tree. + +**Files:** +- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` — clone the Task 0.4 pinned-readback pattern for per-component grad snapshots +- Modify: `crates/ml/src/cuda_pipeline/fused_training.rs` — snapshot grad buffer *between* each loss component's Adam accumulation +- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs` — extend HEALTH_DIAG with `grad_mag_iqn`, `grad_mag_cql`, `grad_mag_c51`, `grad_mag_ens` + +- [ ] **Step 1: Locate the per-loss-component gradient accumulation path** + +Run: +```bash +grep -n "iqn_grad\|cql_grad\|c51_grad\|ens_grad\|component_grad\|iqn_backward\|cql_backward" \ + crates/ml/src/cuda_pipeline/fused_training.rs \ + crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs | head -30 +``` +Expected: each component adds into the shared `adam_grad` buffer via its own kernel. The current Task 0.4 accessor sums the final `adam_grad` post-all-components; this task captures the per-component deltas. + +- [ ] **Step 2: Add a `grad_snapshot_pinned_ptr` + snapshot helper** + +In `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs`, alongside the existing Task 0.4 `grad_readback_pinned_ptr`: + +```rust +/// Task 2.0 — per-component magnitude-branch grad decomposition. +/// Clone the adam_grad buffer into a pinned host slot BETWEEN each +/// component's backward kernel, then compute magnitude-head L2 norm +/// on the delta (post − pre). Produces [iqn, cql, c51, ens] norms. +grad_snapshot_pinned_ptr: *mut f32, +grad_component_norms_mag: [f32; 4], // [iqn, cql, c51, ens] +grad_component_norms_dir: [f32; 4], // [iqn, cql, c51, ens] for ratio denominator +``` + +Allocate the pinned slot at construction (`stream.alloc_pinned::(TOTAL_PARAMS)`); free in the trainer's `Drop` alongside other pinned slots. + +Add the snapshot helper (non-graph, epoch-boundary or pre-/post-component-kernel — **not** inside the captured CUDA graph; this is diagnostic-only): + +```rust +/// Copy current adam_grad to pinned host. Synchronous (sync stream first). +/// Not inside graph capture. +fn snapshot_adam_grad(&mut self) -> Result<(), MLError> { + self.stream.synchronize()?; + unsafe { + let slice = std::slice::from_raw_parts_mut( + self.grad_snapshot_pinned_ptr, + TOTAL_PARAMS, + ); + self.stream.memcpy_dtoh(&self.adam_grad, slice)?; + } + Ok(()) +} + +/// Compute magnitude-branch and direction-branch L2 norms on +/// (adam_grad_now − snapshot). Uses Task 0.4's compute_param_sizes +/// + padded_byte_offset to slice branch tensors 8–11 (dir) and +/// 12–15 (mag) out of the 42-tensor layout. +fn component_grad_norms(&mut self, prev_snapshot: &[f32]) -> Result<(f32, f32), MLError> { + /* sync, dtoh current adam_grad, subtract prev_snapshot, L2 on mag/dir slices */ +} +``` + +- [ ] **Step 3: Wire per-component snapshots in the training step** + +In `crates/ml/src/cuda_pipeline/fused_training.rs`, find the loss-component backward sequence (grep for `iqn_backward`, `cql_backward`, `c51_grad_kernel`, `ensemble_backward` or equivalent). Insert snapshot + post-compute-norm calls: + +```rust +// Before IQN backward +trainer.snapshot_adam_grad()?; +iqn_backward(/* args */)?; +let (mag_iqn, dir_iqn) = trainer.component_grad_norms(pinned_snapshot_prev)?; +trainer.grad_component_norms_mag[0] = mag_iqn; +trainer.grad_component_norms_dir[0] = dir_iqn; + +// Before CQL backward +trainer.snapshot_adam_grad()?; +cql_backward(/* args */)?; +let (mag_cql, dir_cql) = trainer.component_grad_norms(pinned_snapshot_prev)?; +trainer.grad_component_norms_mag[1] = mag_cql; +trainer.grad_component_norms_dir[1] = dir_cql; + +// ... repeat for c51 (index 2) and ens (index 3) ... +``` + +**Important:** this breaks graph capture for the diagnostic steps. Guard the snapshots inside the non-captured epoch-boundary path — per-step capture is preserved. If the loss backwards are all inside one captured graph, fall back to running the diagnostic every Nth *epoch* (not step) by replaying without capture. Document the chosen approach in the commit message. + +- [ ] **Step 4: Accessors on `FusedTrainingCtx`** + +```rust +impl FusedTrainingCtx { + pub fn grad_mag_iqn_ratio(&mut self) -> f32 { + let m = self.trainer().grad_component_norms_mag[0]; + let d = self.trainer().grad_component_norms_dir[0]; + if d > 1e-9 { m / d } else { 0.0 } + } + pub fn grad_mag_cql_ratio(&mut self) -> f32 { /* index 1 */ } + pub fn grad_mag_c51_ratio(&mut self) -> f32 { /* index 2 */ } + pub fn grad_mag_ens_ratio(&mut self) -> f32 { /* index 3 */ } +} +``` + +- [ ] **Step 5: Extend HEALTH_DIAG** + +In `crates/ml/src/trainers/dqn/trainer/training_loop.rs` at the existing HEALTH_DIAG `info!(…)` block (~line 2160), append a new field group: + +```rust +" grad_split [iqn={:.4} cql={:.4} c51={:.4} ens={:.4}]" +``` + +Populate with four `f.grad_mag_*_ratio()` calls. Use `unwrap_or(0.0)` only when `fused_ctx` is unavailable; any other default is a quickfix. + +- [ ] **Step 6: Compile + smoke** + +```bash +SQLX_OFFLINE=true cargo check -p ml +SQLX_OFFLINE=true CUBLAS_WORKSPACE_CONFIG=:4096:8 \ + FOXHUNT_TEST_DATA=test_data/futures-baseline \ + cargo test -p ml --release --lib -- \ + trainers::dqn::smoke_tests::td_propagation::test_td_propagation_sparse_rewards_multi_trial \ + --ignored --nocapture 2>&1 | tail -10 +``` +Expected: test passes; HEALTH_DIAG now prints `grad_split [iqn=… cql=… c51=… ens=…]` with real values. + +- [ ] **Step 7: Capture the decision-tree data (20-epoch smoke)** + +```bash +SQLX_OFFLINE=true CUBLAS_WORKSPACE_CONFIG=:4096:8 \ + FOXHUNT_TEST_DATA=test_data/futures-baseline \ + cargo test -p ml --release --lib -- \ + trainers::dqn::smoke_tests::magnitude_distribution::test_magnitude_distribution \ + --ignored --nocapture 2>&1 | tee /tmp/foxhunt_smoke/magnitude_distribution_grad_split.log +grep "grad_split" /tmp/foxhunt_smoke/magnitude_distribution_grad_split.log | tail -20 +``` + +Record the 20-epoch trajectory of `iqn / cql / c51 / ens` ratios in the Task 2.1 commit message. This is the input to the Task 2.1 decision tree. + +- [ ] **Step 8: Commit** + +```bash +git add crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs \ + crates/ml/src/cuda_pipeline/fused_training.rs \ + crates/ml/src/trainers/dqn/trainer/training_loop.rs +git commit -m "$(cat <<'EOF' +diag(policy-quality): Task 2.0 — per-component grad decomposition for H4 fix + +Adds grad_mag_iqn / grad_mag_cql / grad_mag_c51 / grad_mag_ens HEALTH_DIAG +fields via snapshot-around-backward per loss component. Follows the +Task 0.4 pinned-readback pattern (commit bb42c9963). + +Decomposes the CONFIRMED H4 gradient starvation (Track 1 triage): we now +know WHICH loss component sends zero gradient to the magnitude head, not +just that the total mag/dir ratio is <1e-4. Data feeds Task 2.1 decision +tree. + +Per plan Task 2.0. No stubs, no hiding — unwrap_or(0.0) only on the +pinned-buffer-absent path (construction failure, not a silent fallback). +EOF +)" +``` + +**Do not push yet** — Task 2.1 will consume this data and commit the fix back-to-back. + +--- + +## Task 2.1: H4 fix — rebalance gradient flow to magnitude head + +**Why:** Track 1 H4 is CONFIRMED (20/20 epochs with `grad_ratio_mag_dir ≈ 0.0000`). The magnitude head stays at initialization and cannot learn to distinguish Quarter / Half / Full. Task 2.0's per-component data dictates *which* of three fix families applies. + +**Files (depends on decision tree outcome):** +- Branch A: `crates/ml-dqn/src/network/heads.rs` (or equivalent) — initial weight scale for magnitude head +- Branch B: `crates/ml/src/cuda_pipeline/fused_training.rs` — per-component gradient budget allocation to magnitude vs direction +- Branch C: `crates/ml-dqn/src/network/b1fc.rs` (or wherever `w_b1fc [AH, SH2+3]` lives) — direction-conditioning layout + +- [ ] **Step 1: Read the Task 2.0 trajectory and pick the branch** + +Decision tree (populate from `/tmp/foxhunt_smoke/magnitude_distribution_grad_split.log`): + +| Task 2.0 observation (mean across 20 epochs) | Root cause | Fix family | +|---|---|---| +| ALL FOUR `grad_mag_*_ratio < 1e-3` | No component reaches magnitude — architecture issue | **Branch A or C**: init scale OR direction-conditioning mask | +| One component ≈ 1.0, others < 1e-3 (e.g. `grad_mag_iqn=0.8`, rest ≈ 0) | Budget allocation skips magnitude on 3/4 components | **Branch B**: rebalance per-component weights | +| All four `grad_mag_*_ratio ≥ 0.1` but `q_full ≈ q_half ≈ q_quarter` | Gradient flows, Q-values don't spread → data-driven collapse | **H9 territory** — revisit in Phase 3, DO NOT fix architecturally | + +Record the chosen branch in `/tmp/policy-quality-task21-branch.txt` for reference in the commit message. + +- [ ] **Step 2a: Branch A — Xavier init scaling for magnitude head** + +If all four components are near-zero, the head weights may be initialized at a scale so small the first backward pass produces sub-representable gradients. Fix: scale the magnitude-head weight initializer. + +Locate the construction: +```bash +grep -rn "w_b1fc\|init_b1fc\|magnitude_head.*init\|xavier.*b1fc" \ + crates/ml-dqn/ crates/ml/ | head -10 +``` + +Adjust the Xavier fan-in / fan-out for the magnitude FC. Current layout per MEMORY.md: `w_b1fc: [AH, SH2+3]`. If the direction-conditioned slice contributes `+3` degenerate columns (zero when direction is one-hot-dominant), fan-in is effectively `SH2` not `SH2+3`, and Xavier's `sqrt(2 / (fan_in + fan_out))` is too tight. + +Fix: either +- (A1) compute fan-in using the active portion (`SH2` only) — keeps Xavier meaning and doesn't inflate noise for direction-active samples; OR +- (A2) multiply the magnitude-head init by a fixed 2× to compensate (blunter, revert-friendly). + +Prefer (A1). Write a unit test asserting the post-init weight RMS for the magnitude head is within 1.5× of the direction head's. + +- [ ] **Step 2b: Branch B — per-component gradient budget rebalance** + +If Task 2.0 shows one component dominates direction while magnitude is starved (e.g. `grad_mag_iqn=0.8` but `grad_mag_cql=grad_mag_c51=grad_mag_ens≈0`), the per-component gradient budget is skewing. + +Current budgets (MEMORY.md, 2026-04-08): IQN=60%, CQL=25%, C51=10%, Ens=5%. These are applied as *global* scalars to each component's gradient contribution to `adam_grad`. The fix: apply the budget per-branch, not globally, so the magnitude branch sees its share of each component independent of the direction branch's consumption. + +Implementation sketch (in `fused_training.rs`, inside the backward orchestration): + +```rust +// Before: each component's Adam contribution was scaled by a global budget fraction. +// After: compute per-branch budget so magnitude gets its guaranteed share. +let per_branch_budget = [budget_dir, budget_mag, budget_ord, budget_urg]; +// In the loss backward, multiply each branch's grad slice by its budget, +// not the full param-gradient by a single scalar. +``` + +- [ ] **Step 2c: Branch C — direction-conditioning layout fix** + +If architecture is the cause and init rescaling (Branch A) doesn't close the gap, the `w_b1fc: [AH, SH2+3]` layout inadvertently zeros the magnitude gradient when the direction branch is one-hot-dominant. The `+3` slice (direction features broadcast to magnitude FC input) goes to zero for non-active directions, which means gradient on those columns is also zero — but those are the columns that *would* carry the magnitude-specific signal. + +Fix: change `w_b1fc` to `[AH, SH2]` (pure magnitude-input FC) and implement direction-conditioning as an additive bias `b_b1fc_per_dir: [3, AH]` selected by direction one-hot. This preserves direction-conditioning without blanking magnitude columns. + +This is the heaviest branch — it changes weight-tensor count (42 → 42 or 43 depending on how the bias is tracked) and touches Task 0.4's padded-byte-offset slicing. Only proceed if Branches A and B don't produce a `grad_ratio_mag_dir > 0.1`. + +- [ ] **Step 3: Add a regression smoke test** + +Create or extend `crates/ml/src/trainers/dqn/smoke_tests/magnitude_distribution.rs` (already exists per Phase 0). Add an assertion that forces the H4 fix to hold: + +```rust +// After the 20-epoch train, read final-epoch HEALTH_DIAG. +// Assert: grad_ratio_mag_dir > 0.1 in at least 15/20 epochs. +let violating_epochs = health_diag_history.iter() + .filter(|row| row.grad_ratio_mag_dir < 0.1) + .count(); +assert!( + violating_epochs <= 5, + "H4 regression: grad_ratio_mag_dir < 0.1 in {} / 20 epochs (cap: 5)", + violating_epochs, +); +``` + +**Do not relax this threshold** to make the test pass. If the fix produces `grad_ratio_mag_dir` of 0.05, the fix is wrong — revisit the decision tree. + +- [ ] **Step 4: Compile + smoke** + +```bash +SQLX_OFFLINE=true cargo check -p ml -p ml-dqn +SQLX_OFFLINE=true CUBLAS_WORKSPACE_CONFIG=:4096:8 \ + FOXHUNT_TEST_DATA=test_data/futures-baseline \ + cargo test --profile release-test -p ml --lib -- \ + trainers::dqn::smoke_tests::magnitude_distribution \ + --ignored --nocapture 2>&1 | tail -30 +``` +Expected: passes with grad_ratio ≥ 0.1 on ≥ 15 epochs; `F_Half ≥ 0.05` and `F_Full ≥ 0.05` still hold (or improve). + +- [ ] **Step 5: Re-run the full 5-smoke suite** + +```bash +for t in magnitude_distribution reward_component_audit controller_activity \ + exploration_coverage multi_fold_convergence; do + SQLX_OFFLINE=true CUBLAS_WORKSPACE_CONFIG=:4096:8 \ + FOXHUNT_TEST_DATA=test_data/futures-baseline \ + cargo test --profile release-test -p ml --lib -- \ + "trainers::dqn::smoke_tests::$t" --ignored --nocapture 2>&1 | tail -5 +done +``` +All five must pass. + +- [ ] **Step 6: Commit + push** + +```bash +git add -p # stage the chosen-branch files + regression test +git commit -m "$(cat <<'EOF' +fix(dqn): H4 — per track1 triage + +Track 2.0 per-component gradient decomposition showed : + grad_mag_iqn=, grad_mag_cql=, grad_mag_c51=, grad_mag_ens= + +Decision: Branch . . + +Regression test asserts grad_ratio_mag_dir > 0.1 on ≥ 15 / 20 epochs. + +Closes Track 1 H4 CONFIRMED verdict. H10 safety-net patch lands +separately in Task 2.2. +EOF +)" +git push +``` + +--- + +## Task 2.2: H10 fix — stable argmax tie-breaking at eval + +**Why:** Track 1 H10 CONFIRMED. Eval-mode action distribution is `eq=1.000, eh=0.000, ef=0.000` across all 20 epochs — 100% Quarter — despite training-mode `ent_mag ≈ 0.99`. The cause is argmax over near-identical Q-values picking bin 0 (Quarter) deterministically on ties. + +Even after Task 2.1 spreads magnitude Q-values, there is no guarantee they spread *enough* at every state for argmax to be unambiguous. A Q-margin tie-break prevents future regressions. + +**Files:** +- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` — eval-path argmax (or the factored-branch version) +- Modify: `crates/ml/src/trainers/dqn/trainer/action.rs` — cold-path single-action greedy +- Modify: `crates/ml/src/trainers/dqn/smoke_tests/magnitude_distribution.rs` — regression assertion + +- [ ] **Step 1: Locate the eval-path argmax** + +Run: +```bash +grep -n "argmax\|max_by\|best_action\|select_actions_branch" \ + crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs \ + crates/ml/src/trainers/dqn/trainer/action.rs \ + crates/ml/src/cuda_pipeline/experience_kernels.cu \ + crates/ml-dqn/src/ | head -20 +``` +Expected locations: factored-action eval selection (CUDA kernel side), Rust-side `epsilon_greedy_action` fallback (`action.rs:170`), and any eval-mode branch in `select_actions_branching`. + +- [ ] **Step 2: Define the tie-break rule** + +Preferred rule (no bin-index bias): on argmax ties within `1e-6`, sample **uniformly** among tied indices. Not a Boltzmann softmax — strictly greedy behaviour when Q-values differ; deterministic randomness (seeded per sample) only when they are indistinguishable. + +In the CUDA kernel (pseudo): + +```cuda +// Per branch (direction / magnitude / order / urgency) +float q_max = -INFINITY; +int max_count = 0; +for (int a = 0; a < BRANCH_SIZE; ++a) { + float q = q_branch[a]; + if (q > q_max + 1e-6f) { q_max = q; max_count = 1; } + else if (fabsf(q - q_max) <= 1e-6f) { max_count += 1; } +} +// Second pass: random pick among tied indices. +int tie_idx = (int)(philox_uniform(rng_state) * max_count); +int chosen = -1, seen = 0; +for (int a = 0; a < BRANCH_SIZE; ++a) { + if (fabsf(q_branch[a] - q_max) <= 1e-6f) { + if (seen == tie_idx) { chosen = a; break; } + seen += 1; + } +} +out_action[b] = chosen; +``` + +**Must use the existing Philox RNG state** (the same state the experience kernel uses for CF-flip sampling — already plumbed; no new RNG allocation). Eval mode seeds this deterministically per-sample per-epoch for reproducibility. + +- [ ] **Step 3: Apply the same rule to the Rust cold-path** + +At `crates/ml/src/trainers/dqn/trainer/action.rs:170` the greedy fallback reads `indices = agent.batch_greedy_actions(state)`. Replace the internal `.iter().max_by(...)` with the tie-break rule. Use the same ε-threshold (`1e-6`). Seed from the existing `rng: StdRng` so the path is deterministic in tests. + +Note: this path is also part of the Task 2.5 wiring-bug sweep (bug #2: stale `gen_range(0..5)` for 5-level exposure). Leave the stale line for Task 2.5 to handle to keep the commits tight; only touch the greedy branch here. + +- [ ] **Step 4: Add the regression assertion to the smoke test** + +In `crates/ml/src/trainers/dqn/smoke_tests/magnitude_distribution.rs`: + +```rust +// After 20-epoch train + eval rollout: +let eval_dist = final_epoch_health_diag.eval_dist; // [eq, eh, ef] +assert!( + eval_dist.eh + eval_dist.ef >= 0.3, + "H10 regression: eval Half+Full share {:.3} < 0.30 — argmax tie-break \ + did not prevent collapse (eq={:.3})", + eval_dist.eh + eval_dist.ef, + eval_dist.eq, +); +``` + +Threshold `0.3` comes from the user spec ("Eval dist `eh + ef ≥ 0.3` must fix H10 + upstream H4"). Do not relax — if it fails, the fix is insufficient. + +- [ ] **Step 5: Compile + smoke** + +```bash +# Rebuild kernel (cubin invalidation) +touch crates/ml/src/cuda_pipeline/experience_kernels.cu \ + crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs +SQLX_OFFLINE=true cargo build -p ml --release 2>&1 | tail -5 +SQLX_OFFLINE=true CUBLAS_WORKSPACE_CONFIG=:4096:8 \ + FOXHUNT_TEST_DATA=test_data/futures-baseline \ + cargo test --profile release-test -p ml --lib -- \ + trainers::dqn::smoke_tests::magnitude_distribution \ + --ignored --nocapture 2>&1 | tail -20 +``` +Expected: passes; `eval_dist` shows `eh + ef ≥ 0.3`. + +- [ ] **Step 6: Commit** + +```bash +git add -p +git commit -m "$(cat <<'EOF' +fix(dqn): H10 — stable argmax tie-break at eval per track1 triage + +Replaces first-index-wins argmax with uniform-sample-among-tied-indices +(|q_a − q_b| < 1e-6). Uses the existing Philox state; eval mode is +deterministic per (sample, epoch). + +Closes Track 1 H10 CONFIRMED verdict (eval_dist [eq=1.000 eh=0.000 ef=0.000] +across all 20 baseline epochs). Regression asserts eh + ef ≥ 0.3. + +Upstream root cause H4 fixed in Task 2.1; H10 here is the safety-net patch +so future Q-value near-ties cannot re-produce the Quarter collapse. +EOF +)" +git push +``` + +--- + +## Task 2.3: DELETE R5 micro-reward + +**Why:** Track 2 R5 DELETE. `micro_reward_scale = 0` in smoke AND `reward_contrib[3] = 0.000` in 60 / 60 epochs. The dense-OFI signal it provides is redundant with the 42-dim state vector already including 8 OFI features. Reward-inventory §"Updated disposition table" tagged it DELETE in Phase 1. + +**Gating note:** Task 2 triage flagged deletion "gated on TD-propagation diagnostic (Phase 2 Task #8)". Task 2.0 effectively delivers that diagnostic — per-component gradient decomposition shows whether the Q function learns from sparse rewards. If Task 2.0 shows the sparse-reward path carries gradient to the magnitude head (`grad_mag_iqn ≥ 0.2` on magnitude-relevant batches), TD-propagation works and R5 deletion is safe. Record that cross-reference in the Task 2.3 commit message. + +**Files:** +- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` — delete the dense-OFI branch at lines 1818–1902 +- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` — remove `micro_reward_scale` kernel arg +- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` — remove `micro_reward_per_sample` buffer + reducer +- Modify: `crates/ml/src/trainers/dqn/config.rs` — delete `micro_reward_scale` field +- Modify: `config/training/dqn-smoketest.toml`, `config/training/dqn-baseline.toml`, `config/training/dqn-localdev.toml`, `config/training/dqn-hyperopt.toml` — drop the config key +- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` — shrink `reward_contrib_fractions()` from 5 to 4 slots +- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs` — drop `micro={:.3}` from HEALTH_DIAG `reward_contrib [...]` + +- [ ] **Step 1: Pre-verify the Task 2.0 cross-reference** + +```bash +grep "grad_split" /tmp/foxhunt_smoke/magnitude_distribution_grad_split.log | \ + awk -F'iqn=' '{print $2}' | awk '{print $1}' | sort -n | tail -5 +``` +Expected: `grad_mag_iqn` ≥ 0.1 on at least one epoch (evidence the sparse-reward IQN path reaches magnitude). If zero across all 20 epochs, **pause this task** — deletion risks removing a load-bearing dense signal. Return to Task 2.1 decision tree. + +- [ ] **Step 2: Remove the kernel branch** + +In `crates/ml/src/cuda_pipeline/experience_kernels.cu`, delete lines 1818–1902 (the `if (!segment_complete && fabsf(position) > 0.001f && micro_reward_scale > 0.0f …)` block plus the per-sample write at the end of that branch). Delete the fallback line at 1904 that guards `micro_reward_scale=0` — replace with the unguarded holding-cost fallback. + +Also delete the `micro_reward_scale` parameter from the kernel signature (line 1133) and every call site's kernel-launch args. Grep to confirm: +```bash +grep -rn "micro_reward_scale" crates/ml/ +``` +Expected after edit: zero matches. + +- [ ] **Step 3: Remove the config field** + +In `crates/ml/src/trainers/dqn/config.rs`, delete the `micro_reward_scale` field from `DQNHyperparameters`, `Default`, `conservative()` and any `serde` rename aliases. Check `serde(default)` is not needed (no stub default left behind). + +Delete the TOML key from all four config files. + +Per project rules (no feature flags / no stubs): do **not** leave the field as `#[deprecated]` or behind `#[cfg(feature = "…")]`. Delete outright. + +- [ ] **Step 4: Shrink the reward-contrib reducer** + +In `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs`, shrink the slot-3 accumulation. Move `loss_aversion` from slot 4 → slot 3, then drop slot 4 entirely. The reducer returns a 4-element array after this task (R6 deletion in Task 2.4 will drop it to 3; that's a deliberate sequencing, not a redundancy). + +Update the HEALTH_DIAG `reward_contrib [popart={:.3} cf={:.3} trail_r={:.3} micro={:.3} la={:.3}]` format string at `training_loop.rs` to `reward_contrib [popart={:.3} cf={:.3} trail_r={:.3} la={:.3}]`. Adjust the value tuple accordingly. + +- [ ] **Step 5: Update the reward-component-audit smoke test** + +Smoke test `crates/ml/src/trainers/dqn/smoke_tests/reward_component_audit.rs` currently reads slot 3 for micro. Delete those assertions. The slot-4 → slot-3 shift for loss_aversion must be reflected in any index-based assertion. + +- [ ] **Step 6: Compile + full smoke suite** + +```bash +touch crates/ml/src/cuda_pipeline/experience_kernels.cu +SQLX_OFFLINE=true cargo check --workspace +SQLX_OFFLINE=true cargo build -p ml --release 2>&1 | tail -5 + +for t in magnitude_distribution reward_component_audit controller_activity \ + exploration_coverage multi_fold_convergence; do + SQLX_OFFLINE=true CUBLAS_WORKSPACE_CONFIG=:4096:8 \ + FOXHUNT_TEST_DATA=test_data/futures-baseline \ + cargo test --profile release-test -p ml --lib -- \ + "trainers::dqn::smoke_tests::$t" --ignored --nocapture 2>&1 | tail -5 +done +``` +All five must pass. `reward_component_audit` in particular must pass after the slot-count change. + +- [ ] **Step 7: Commit** + +```bash +git add -p +git commit -m "$(cat <<'EOF' +cleanup(reward): DELETE R5 micro-reward per track2 triage + +reward_contrib[3] = 0.000 in 60 / 60 baseline epochs (micro_reward_scale=0 +in every profile). Dense per-bar directional intent is already covered by +the 8-dim OFI block in the 42-dim state vector. + +Task 2.0 gradient decomposition confirmed sparse-reward TD propagation +reaches the magnitude head (grad_mag_iqn > 0 on magnitude-relevant +batches) — deletion is safe. + +Removed: + - kernel branch experience_kernels.cu:1818-1902 + micro_reward_scale arg + - micro_reward_per_sample buffer + reducer slot + - micro_reward_scale config field (all four TOMLs) + - reward_contrib[3] slot (reducer + HEALTH_DIAG format) + +~150 LOC net reduction. Closes Track 2 R5 DELETE verdict. +EOF +)" +git push +``` + +--- + +## Task 2.4: DELETE R6 loss-aversion + relocate negative-tail compression to Q-target smoothing + +**Why:** Track 2 R6 DELETE. `reward_contrib[4]` (loss-aversion) fires in 6 / 60 epochs (10%), mean 0.26%, max 3.1%. The asymmetric soft-clamp `-10 × (1 − exp(x/10))` is sub-noise at the reward level. + +However, the clamp's `+10` upper cap IS structurally load-bearing on rare-large-win inputs (numerical safety against reward explosions). And the *intent* of the negative-tail compression — prevent catastrophic-loss gradients from dominating — is worth preserving at the *gradient* level, not the reward level (per Track 2 recommendation referencing reward-inventory §"Better-form taxonomy"). That means relocating to C51-Bellman / Q-target smoothing. + +**Files:** +- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu:1774-1783` — delete the clamp's negative-tail branch, keep upper cap as a simple `fminf(reward, 10.0f)` +- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu:78-81` — delete `asymmetric_soft_clamp` function (no longer called anywhere) +- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu:2872` — second call site at `opt_reward`; apply same relocation (keep +10 cap) +- Modify: `crates/ml/src/cuda_pipeline/c51_grad_kernel.cu` — add Q-target smoothing parameter (Huber-style negative-tail compression on Bellman targets) +- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` — drop `loss_aversion_per_sample` buffer + reducer slot +- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs` — drop `la={:.3}` from HEALTH_DIAG `reward_contrib [...]` + +- [ ] **Step 1: Relocate the negative-tail compression to C51 Bellman** + +In `crates/ml/src/cuda_pipeline/c51_grad_kernel.cu`, where the Bellman target distribution is projected onto the C51 atoms, add a soft-clamp on the target scalar **before** projection: + +```cuda +// Negative-tail compression on Bellman target (was R6 reward-layer clamp). +// Compresses catastrophic-loss targets proportionally; keeps gradient +// well-conditioned. Upper side unchanged — high positives already capped +// by v_range clamp upstream. +float target = /* existing target computation */; +if (target < 0.0f) { + target = -10.0f * (1.0f - expf(target / 10.0f)); +} +// ... existing atom projection continues ... +``` + +This is the proper home for the invariant: Huber/clamp logic on Q-targets, not on the raw reward. Document the move in the kernel comment. + +- [ ] **Step 2: Delete the reward-kernel clamp (except upper cap)** + +In `experience_kernels.cu`, replace lines 1774–1775: + +```c +// Before: +// float base_reward = 2.0f * vol_normalized_return; +// reward = asymmetric_soft_clamp(base_reward); + +// After: +float base_reward = 2.0f * vol_normalized_return; +reward = fminf(base_reward, 10.0f); // upper cap preserved (numerical safety) +``` + +Delete the diagnostic write at 1783 (`loss_aversion_per_sample[out_off] = base_reward - reward;`) and the associated buffer in the collector. + +Apply the same change at the second call site (line 2872 `opt_reward`): replace `asymmetric_soft_clamp(...)` with `fminf(..., 10.0f)`. + +Finally, delete the `asymmetric_soft_clamp` definition at lines 78–81. `grep -rn "asymmetric_soft_clamp"` must return zero matches after the edit. + +- [ ] **Step 3: Shrink the reward-contrib reducer (again)** + +`reward_contrib_fractions()` is now 3 slots: `[popart, cf, trail_r]` after Tasks 2.3 and 2.4. Update HEALTH_DIAG format string to `reward_contrib [popart={:.3} cf={:.3} trail_r={:.3}]` and adjust the tuple. + +- [ ] **Step 4: Update reward_component_audit smoke** + +Delete loss-aversion assertions from `crates/ml/src/trainers/dqn/smoke_tests/reward_component_audit.rs`. The smoke now asserts only on popart / cf / trail_r. + +- [ ] **Step 5: Compile + full smoke suite** + +```bash +touch crates/ml/src/cuda_pipeline/experience_kernels.cu \ + crates/ml/src/cuda_pipeline/c51_grad_kernel.cu +SQLX_OFFLINE=true cargo build -p ml --release 2>&1 | tail -5 + +for t in magnitude_distribution reward_component_audit controller_activity \ + exploration_coverage multi_fold_convergence; do + SQLX_OFFLINE=true CUBLAS_WORKSPACE_CONFIG=:4096:8 \ + FOXHUNT_TEST_DATA=test_data/futures-baseline \ + cargo test --profile release-test -p ml --lib -- \ + "trainers::dqn::smoke_tests::$t" --ignored --nocapture 2>&1 | tail -5 +done +``` +All five must pass. multi_fold_convergence in particular must still show ≥ 2 / 3 folds with Best Sharpe > 0 — if it regresses after Task 2.4, the relocation to C51 target smoothing is mis-calibrated; revisit Step 1. + +- [ ] **Step 6: Commit** + +```bash +git add -p +git commit -m "$(cat <<'EOF' +cleanup(reward): DELETE R6 loss-aversion, relocate to C51 target smoothing + +Per track2 triage: asymmetric_soft_clamp at experience_kernels.cu:1774-1783 +fires in 6/60 baseline epochs (mean 0.26% of reward magnitude — sub-noise). +The invariant "don't let catastrophic-loss gradients dominate" is better +expressed at the gradient level. + +Changes: + - Delete asymmetric_soft_clamp() from experience_kernels.cu:78-81. + - Replace reward-layer clamp with fminf(reward, 10.0f) — upper cap + preserved for rare-large-win numerical safety. + - Add negative-tail compression on C51 Bellman target in c51_grad_kernel: + target = -10 * (1 - exp(target / 10)) when target < 0. Same shape as + the old reward clamp, applied where it belongs (Q-target, not reward). + - Drop loss_aversion_per_sample buffer + reward_contrib slot 4. + +reward_contrib reducer is now 3 slots: [popart, cf, trail_r]. + +Closes Track 2 R6 DELETE verdict. +EOF +)" +git push +``` + +--- + +## Task 2.5: Wiring-bug sweep (7 bugs from Tracks 1, 3, 4) + +**Why:** the triage docs surfaced 7 wiring bugs that are independent of the H4/H10 headline issues. These are correctness fixes — not behavioural changes — but each one makes a L40S triage result trustworthy. Bundle into one task since they share review context; each step is a separate sub-commit so git log is readable. + +The bugs are ordered by dependency on other tasks: bug 1 must land before the Task 2.8 L40S run (otherwise C1 verdict on L40S is garbage). + +**Files:** see per-step. + +- [ ] **Step 1: Bug #1 — `fire_lr` detects scheduler, not anti-LR intervention** (Track 3 C1) + +**File:** `crates/ml/src/trainers/dqn/trainer/training_loop.rs:2460, 2468` + +**Bug:** `cur_lr = self.lr_scheduler.get_lr()` is captured before the anti-LR multiplier is applied. Under any non-Constant scheduler (Cosine / Linear / Exponential), `fire_lr` ticks every epoch from scheduler drift, false-positive-firing the anti-LR controller. + +**Fix:** detect anti-LR via the multiplier itself, already computed at `:2936-2942`. Store `self.last_anti_mult` after the anti-LR block and read it in the fire-detection block. + +```rust +// At the top of the anti-LR block (around line 2935), persist the multiplier: +let anti_mult = if recent_max > thresh { + self.hyperparams.anti_lr_good_mult +} else if recent_mean < -thresh { + self.hyperparams.anti_lr_bad_mult +} else { + 1.0 +}; +self.last_anti_mult = anti_mult; // NEW + +// At fire detection (around line 2468), replace: +// let fire_lr = prev.lr.is_finite() && (cur_lr - prev.lr).abs() > 1e-10; +// With: +let fire_lr = (self.last_anti_mult - 1.0).abs() > 0.01; +``` + +Add `last_anti_mult: f32` to the trainer struct, initialize to 1.0 in constructor + `reset_epoch_state`. + +**Test:** no smoke asserts `fire_lr` directly, but `controller_activity` must still show no controller > 50% firing. Run it after the fix. + +Commit: +```bash +git commit -am "fix(dqn): C1 fire detection — observe anti-LR multiplier, not scheduler LR (track3) + +Was capturing cur_lr = lr_scheduler.get_lr() BEFORE anti-LR multiplier. +Under Cosine/Linear decay this reports every-epoch-fire from scheduler +drift. Fix: detect via (anti_mult != 1.0) directly, per track3 triage §C1 +wiring surprise. + +Prerequisite for Task 2.8 L40S run — without this, L40S C1 verdict is +uninterpretable under any non-Constant scheduler." +``` + +- [ ] **Step 2: Bug #2 — `epsilon_greedy_action` uses stale `gen_range(0..5)`** (Track 4 E1 tech-debt) + +**File:** `crates/ml/src/trainers/dqn/trainer/action.rs:167` + +**Bug:** pre-4-branch legacy 5-level exposure range. Only called by tests, but misleading. + +**Fix:** sample the factored action space correctly (direction × magnitude × order × urgency = 3×3×3×3 = 81 actions, encoded as `dir*27 + mag*9 + ord*3 + urg` per MEMORY.md). + +```rust +if rng.gen::() < epsilon { + let dir = rng.gen_range(0..3usize); + let mag = rng.gen_range(0..3usize); + let ord = rng.gen_range(0..3usize); + let urg = rng.gen_range(0..3usize); + Ok(dir * 27 + mag * 9 + ord * 3 + urg) +} else { + let agent = self.agent.read().await; + let indices = agent.batch_greedy_actions(state)?; + let best_action = *indices.first() + .ok_or_else(|| anyhow::anyhow!("argmax on empty Q-values"))? as usize; + Ok(best_action) +} +``` + +Return type is still `Result`; encoded action index in `0..81`. Downstream consumers must decode with the same formula. + +Commit: +```bash +git commit -am "fix(dqn): epsilon_greedy_action samples 4-branch factored space (track4) + +Stale 0..5 range from pre-2026-04-08 9-level code. Only tests call this +path, but the stale value was a future foot-gun per the track4 triage +tech-debt flag." +``` + +- [ ] **Step 3: Bug #3 — `agent.update_epsilon()` gated by `if !true`** (Track 4 E1) + +**File:** `crates/ml/src/trainers/dqn/trainer/training_loop.rs:2861` + +**Bug:** dead code — `if !true { … }` is a no-op. Either the decay was intentional (remove the guard) or it was abandoned (delete the whole block). + +**Decision:** delete. Per MEMORY.md + Track 4 triage, epsilon is driven by the explicit `epsilon_start / epsilon_end / epsilon_decay` schedule applied at the agent level via `get_effective_epsilon()`. `update_epsilon()` is a legacy step-count-based decay that is no longer the source of truth. Deleting removes a foot-gun. + +```rust +// Delete lines 2860-2864 entirely: +// // Epsilon decay (skip when noisy nets) +// if !true { +// let mut agent = self.agent.write().await; +// agent.update_epsilon(); +// } +``` + +Commit: +```bash +git commit -am "cleanup(dqn): delete dead \`if !true\` update_epsilon block (track4) + +Per feedback_no_feature_flags: if !true is a disabled toggle pattern. +Epsilon is driven by the explicit start/end/decay schedule; update_epsilon +is legacy step-count decay. Zero behavior change; removes foot-gun." +``` + +- [ ] **Step 4: Bug #4 — `sigma_mean` returns raw σ, ignores `sigma_scale`** (Track 4 E2) + +**File:** `crates/ml-dqn/src/noisy_layers.rs:345` + +**Bug:** HEALTH_DIAG reports `sigma_mean=0.0320` constant across 20 epochs because the accessor reads the σ tensor directly, not the scheduler-scaled effective σ. Track 4 cannot verdict E2's annealing behaviour against the broken accessor. + +**Fix:** multiply by `self.current_sigma_scale` (already stored by `reset_noise_with_sigma`): + +```rust +pub fn sigma_mean(&self) -> Result { + let n_w = self.weight_sigma.len(); + let n_b = self.bias_sigma.len(); + if n_w + n_b == 0 { return Ok(0.0); } + let mut h_w = vec![0.0_f32; n_w]; + let mut h_b = vec![0.0_f32; n_b]; + self.stream.memcpy_dtoh(&self.weight_sigma, &mut h_w) + .map_err(|e| MLError::ModelError(format!("sigma_mean weight dtoh: {e}")))?; + self.stream.memcpy_dtoh(&self.bias_sigma, &mut h_b) + .map_err(|e| MLError::ModelError(format!("sigma_mean bias dtoh: {e}")))?; + let sum_w: f64 = h_w.iter().map(|x| x.abs() as f64).sum(); + let sum_b: f64 = h_b.iter().map(|x| x.abs() as f64).sum(); + let raw = ((sum_w + sum_b) / ((n_w + n_b) as f64)) as f32; + Ok(raw * self.current_sigma_scale) // scheduler-adjusted effective σ +} +``` + +Add `current_sigma_scale: f32` field to the layer (initialize to 1.0; `reset_noise_with_sigma` already has the value as its input arg — store it). + +Commit: +```bash +git commit -am "fix(noisy): sigma_mean returns scheduler-scaled effective σ (track4 E2) + +Before, sigma_mean reads raw σ tensor → HEALTH_DIAG shows constant 0.0320 +while the scheduler drives 0.8 → 0.4 anneal. Track 4 could not triage E2 +contribution against the broken accessor. + +Fix: multiply by current_sigma_scale (stored at reset_noise_with_sigma). +This makes Track 1 H7 retrospective verdict sound and unblocks Track 4 E2 +re-measurement on L40S (Task 2.8)." +``` + +- [ ] **Step 5: Bug #5 — `controller_fire_counts` does not reset per fold** (Track 3 C2/C5 fold-boundary artefact) + +**File:** `crates/ml/src/trainers/dqn/trainer/training_loop.rs` — find the fold-boundary hook + +Run: +```bash +grep -n "fold\|walk_forward" crates/ml/src/trainers/dqn/trainer/training_loop.rs \ + | grep -iE "boundary|start|begin|reset|train_fold" | head +``` +Expected: there is some fold-entry hook (possibly in the walk-forward orchestrator). If not found inline, look in `crates/ml/src/trainers/dqn/trainer/mod.rs` for the fold iteration. If the fold loop lives above `training_loop.rs` (e.g., in the walk-forward harness that constructs the trainer per fold), the counters are already per-fold — confirm. + +**Fix:** reset `controller_fire_counts`, `controller_total_epochs`, `prev_controller_values` at the fold-entry point. If the trainer is reconstructed per fold, these already reset via the constructor; verify with a log. + +Add an explicit reset: +```rust +fn on_fold_start(&mut self) { + self.controller_fire_counts = ControllerFireCounts::default(); + self.controller_total_epochs = 0; + self.prev_controller_values = ControllerPrevValues::default(); + // NOT last_anti_mult — that's within-epoch state not counter state. +} +``` + +Wire into the fold-loop entry. If already correct, document in commit "no-op fix, already per-fold by trainer reconstruction". + +Commit: +```bash +git commit -am "fix(dqn): reset controller_fire_counts at fold boundary (track3) + +Track 3 triage §C2/C5: the 2/60 fire rates for tau and cql_alpha were +fold-boundary artifacts (cosine-annealed tau jumps when train_step resets). +Reset counters per fold to decouple fold-boundary bookkeeping from +intra-fold controller interventions. Smoke rates at fold 0 should then +equal full-run rates." +``` + +- [ ] **Step 6: Bug #6 — stale R7 `patience_mult` docstring** (Track 2 R7) + +**File:** `crates/ml/src/cuda_pipeline/experience_kernels.cu:1049` (inside the function-level docstring) + +**Bug:** `patience_mult` is referenced in a comment block describing a v8 reward shape that was removed in commit `83d524f86`. Track 2 R7 flags this as a foot-gun per "trust code, not comments". + +**Fix:** rewrite the comment block to describe the *current* reward shape (from Task 2.4: step_return with fminf upper cap, plus CF-flip and trailing-stop physics). Or delete the stale paragraph entirely. + +Example replacement: + +```c +/* ... existing comment up to line 1045 ... + * 6. Computes reward (current shape): + * reward = fminf(2.0f * vol_normalized_return, 10.0f) (trade exit) + * reward = -shaping_scale * holding_cost * |position| (flat-holding fallback) + * Counterfactual direction flip mirrors reward sign for 50% of eligible + * episodes (symmetry augmentation; see experience_kernels.cu:1252-1261). + * Trailing-stop physics live in trade_physics.cuh, not here — they gate + * exits, not rewards. + * 7. Writes (batch_states, action, reward, done) ... + */ +``` + +Commit: +```bash +git commit -am "cleanup(kernel): delete stale patience_mult docstring (track2 R7) + +patience_mult was removed in 83d524f86; the comment at line 1049 described +a v8 reward shape that no longer exists. Per feedback 'trust code, not +comments', stale docs are foot-guns. + +Replaced with the current (post-Task-2.4) reward shape. Zero behavior +change." +``` + +- [ ] **Step 7: Bug #7 — C5 `cql_alpha` regime gate reads null ISV pointer** (Track 3 C5) + +**File:** `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs:4905-4921` + +**Bug:** `cql_alpha_eff = config.cql_alpha × (1 − regime_stability) × health`. At smoke scale, `isv_signals_ptr[11]` (regime_stability) is null/unpopulated; the `read_isv_regime` helper at 8180 falls back to `(0.5, 0.5)`. Silent fallback violates feedback_no_hiding ("Result + tracing::warn! or propagate"). + +**Fix:** either + +- (a) wire ISV[11] at smoke scale (populate regime_stability via an upstream computation that runs every epoch regardless of profile — ISV warmup policy change); OR +- (b) change the fallback to `tracing::warn!("isv_signals null — cql_alpha formula uses fallback 0.5")` AND gate the formula to not use ISV when the pointer is null: + +```rust +let (health, regime_stability) = if self.isv_signals_pinned.is_null() { + tracing::warn!("isv_signals_pinned is null; cql_alpha formula disabled this epoch"); + (1.0, 0.0) // health=1.0, regime=0 → cql_alpha = config.cql_alpha × 1.0 × 1.0 +} else { + // ... existing read ... +}; +``` + +**Decision:** prefer (a) if the ISV warmup path is a 1-commit fix. If (a) requires a multi-task ISV pipeline rework, do (b) and flag ISV warmup as a follow-up for the next phase. Document the choice in the commit message. + +Commit: +```bash +git commit -am "fix(dqn): C5 cql_alpha regime gate handles null ISV without silent fallback (track3) + +Before: when isv_signals_pinned is null (smoke scale), read_isv_regime +silently returned (0.5, 0.5) → cql_alpha_eff near zero by degenerate math, +not by design. + +Per feedback_no_hiding: either wire the signal or warn + gate. This commit +does the gate: null ISV → tracing::warn! + formula disabled; real ISV → +formula as before. + +Track 3 triage §C5 identified this as a hiding-errors case. ISV warmup +pipeline work is out of scope — follow-up task." +``` + +- [ ] **Step 8: Compile + full smoke suite after all 7 sub-commits** + +```bash +SQLX_OFFLINE=true cargo check --workspace +touch crates/ml/src/cuda_pipeline/experience_kernels.cu +SQLX_OFFLINE=true cargo build -p ml --release 2>&1 | tail -5 + +for t in magnitude_distribution reward_component_audit controller_activity \ + exploration_coverage multi_fold_convergence; do + SQLX_OFFLINE=true CUBLAS_WORKSPACE_CONFIG=:4096:8 \ + FOXHUNT_TEST_DATA=test_data/futures-baseline \ + cargo test --profile release-test -p ml --lib -- \ + "trainers::dqn::smoke_tests::$t" --ignored --nocapture 2>&1 | tail -3 +done +``` +All five must pass. `git push` after the full suite is green (one push covers all 7 sub-commits). + +--- + +## Task 2.6: E4 entropy-regularization DELETE-or-KEEP decision (data-driven, post-Task 2.0) + +**Why:** Track 4 E4 DELETE-CANDIDATE. `entropy_coefficient = 0.001` on C51 logits; Track 4 flagged it as provisionally-redundant with NoisyNets (E2) + count bonus (E3) + naturally-uniform Q-values from H4. Final verdict is blocked on Task 2.0 per-component gradient decomposition: *does C51 entropy actually send gradient to the magnitude head?* + +**Files (if DELETE):** +- Modify: `crates/ml/src/cuda_pipeline/c51_grad_kernel.cu` — remove entropy-reg term and the `entropy_coefficient` arg +- Modify: `crates/ml/src/cuda_pipeline/fused_training.rs:388` — stop passing `entropy_coefficient` to the kernel +- Modify: `crates/ml/src/trainers/dqn/config.rs` — delete `entropy_coefficient` field +- Modify: `config/training/dqn-*.toml` — drop the `entropy_coefficient` key + +- [ ] **Step 1: Re-read Task 2.0's grad_split data** + +From `/tmp/foxhunt_smoke/magnitude_distribution_grad_split.log`: +```bash +grep "grad_split" /tmp/foxhunt_smoke/magnitude_distribution_grad_split.log \ + | awk -F'c51=' '{print $2}' | awk '{print $1}' | sort -n | head -10 +``` + +Decision: +- If `grad_mag_c51` mean > 0.1 across epochs → C51 (including its entropy term) IS sending gradient to magnitude; entropy regularization is plausibly contributing. **Re-measure by disabling only the entropy term**, not the whole C51 path — see Step 2. +- If `grad_mag_c51` mean < 0.01 → C51 path is nearly dead to magnitude; entropy term within it cannot be doing useful work. **DELETE.** + +- [ ] **Step 2: (Conditional) ablate entropy with a single-variable comparison** + +Only if Step 1 is inconclusive: run the 5-smoke suite with `entropy_coefficient = 0.0` in `dqn-smoketest.toml`, compare `ent_mag` trajectory across 20 epochs against baseline. If `ent_mag` drops > 0.05 on any fold, KEEP (entropy is load-bearing). Otherwise DELETE. + +This ablation is ~12 min local (3 folds × 20 epochs on RTX 3050). Do NOT commit the `entropy_coefficient = 0.0` TOML change — revert after the comparison. + +- [ ] **Step 3: (If DELETE) remove the term** + +In `crates/ml/src/cuda_pipeline/c51_grad_kernel.cu:25, 64-68`, delete the `if (entropy_coeff > 0.0f)` branch that adds `inv_batch * entropy_coeff * (1.0 + lp_clamped)`. Delete the `entropy_coefficient` kernel arg. + +In `fused_training.rs:388`, remove the arg from the kernel launch. + +In `config.rs`, delete the field + default. In each TOML, drop the `entropy_coefficient = …` line. + +Per feedback_no_feature_flags: no flag, no `enable_entropy_reg`. Just delete. + +- [ ] **Step 4: (If KEEP) document the data** + +Add a comment in `c51_grad_kernel.cu` block explaining why it stays (with a reference to the Task 2.0 grad_split evidence). Commit message says "KEEP — evidence `grad_mag_c51` ≥ 0.1; removing drops `ent_mag` by X on fold Y". + +- [ ] **Step 5: Compile + smoke suite** + +```bash +SQLX_OFFLINE=true cargo check --workspace +touch crates/ml/src/cuda_pipeline/c51_grad_kernel.cu +SQLX_OFFLINE=true cargo build -p ml --release 2>&1 | tail -5 + +for t in magnitude_distribution reward_component_audit controller_activity \ + exploration_coverage multi_fold_convergence; do + SQLX_OFFLINE=true CUBLAS_WORKSPACE_CONFIG=:4096:8 \ + FOXHUNT_TEST_DATA=test_data/futures-baseline \ + cargo test --profile release-test -p ml --lib -- \ + "trainers::dqn::smoke_tests::$t" --ignored --nocapture 2>&1 | tail -3 +done +``` +All five must pass regardless of DELETE/KEEP outcome. + +- [ ] **Step 6: Commit + push** + +If DELETE: +```bash +git commit -am "cleanup(dqn): DELETE E4 entropy regularization per track4 + Task 2.0 data + +Task 2.0 grad_mag_c51 = < 0.01 across all 20 epochs → C51 entropy term +contributes no gradient to the magnitude head. Track 4 provisional DELETE +confirmed. + +Removed: + - c51_grad_kernel entropy term (branch lines 64-68) + entropy_coefficient arg + - config field DQNHyperparameters.entropy_coefficient + TOML keys + - fused_training.rs call-site arg + +~30 LOC net reduction. One mechanism fewer covering the action-uniformity +invariant (E1 ε-floor + E2 NoisyNets + E3 count-bonus remain)." +git push +``` + +If KEEP: +```bash +git commit -am "docs(dqn): E4 entropy regularization — KEEP per track4 + Task 2.0 data + +grad_mag_c51 = > 0.1 and disabling entropy_coefficient drops ent_mag +by on fold . Entropy reg IS contributing measurable coverage +pressure; Track 4's provisional DELETE does not hold up under direct data. + +Annotated c51_grad_kernel.cu with the evidence. Flag for re-evaluation at +Phase 3 L40S scale if production grad_mag_c51 diverges from smoke." +git push +``` + +--- + +## Task 2.7: C4 adaptive grad-clip ablation + DELETE-or-KEEP decision + +**Why:** Track 3 C4 CANDIDATE-FOR-DELETE. Fires in 12 / 60 (20%) of baseline epochs — above the 10% DIAGNOSTIC ceiling, below the 50% LOAD-BEARING floor. Per design spec §5.3 decision matrix, this is the ablation band. + +**Files (if DELETE):** +- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` — remove `adaptive_clip_value` tracking (the EMA controller) +- Modify: `crates/ml/src/cuda_pipeline/fused_training.rs` — replace adaptive-clip step with fixed clip (re-use `gradient_clip_norm = 1.0` config, currently the outer-ring fallback) +- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs:2472` — drop `fire_clip` tracking + `clip={:.3}` HEALTH_DIAG field + +- [ ] **Step 1: Run the disable-ablation locally** + +Temporarily set the adaptive clip to a fixed `1e6` threshold so it never triggers. Concretely, at the `adaptive_clip_value` setter in `gpu_dqn_trainer.rs`, guard an override: + +```rust +// TEMPORARY — revert after Step 2 comparison: +self.adaptive_clip_value = 1e6; +``` + +Run the smoke suite: + +```bash +touch crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs +SQLX_OFFLINE=true cargo build -p ml --release 2>&1 | tail -5 + +for t in controller_activity multi_fold_convergence magnitude_distribution; do + SQLX_OFFLINE=true CUBLAS_WORKSPACE_CONFIG=:4096:8 \ + FOXHUNT_TEST_DATA=test_data/futures-baseline \ + cargo test --profile release-test -p ml --lib -- \ + "trainers::dqn::smoke_tests::$t" --ignored --nocapture 2>&1 | tee /tmp/c4-ablation-$t.log +done +``` + +Expected runtime: ~25 min total (3 smokes × ~8 min each). Record: +- `controller_activity` final `[CTRL_FIRE]` line — the ablated grad_clip column should be `0.000`. +- `multi_fold_convergence` Best Sharpe per fold — must pass the ≥ 2 / 3 folds > 0 gate. +- `magnitude_distribution` — must pass `F_Half ≥ 0.05` AND `F_Full ≥ 0.05`. + +**`git checkout -- crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs`** to revert the override. + +- [ ] **Step 2: Interpret the data** + +Decision matrix: + +| Ablation outcome | Verdict | Action | +|---|---|---| +| All 3 smokes pass within baseline noise (Sharpe within ±1 std) | DELETE C4 | Step 3 | +| multi_fold_convergence regresses (Best Sharpe drops > 1σ, or any fold produces NaN) | KEEP C4, reclassify LOAD-BEARING | Step 4 | +| magnitude_distribution regresses (F_Half or F_Full drops below 0.05) | KEEP C4, reclassify LOAD-BEARING | Step 4 | +| Mixed — 1 smoke regresses by noise-margin | Re-run 3× to reject sample noise; if persistent, KEEP; if not, DELETE | Step 3 or 4 | + +- [ ] **Step 3: (If DELETE) remove the adaptive clip** + +Remove `adaptive_clip_value` field + EMA update code from `GpuDqnTrainer`. In `fused_training.rs`, replace the adaptive clip call with a fixed `gradient_clip_norm` read from config (1.0 per MEMORY.md default). In `training_loop.rs`, drop `fire_clip`, the `clip={:.3}` HEALTH_DIAG field, and the `grad_clip` slot from `controller_fire_counts`. + +Note: per Track 3 triage, the per-component gradient budgets (IQN=60%, CQL=25%, C51=10%, Ens=5%) are NOT adaptive controllers; they remain. We only delete the EMA-adaptive clip. + +Update `controller_activity.rs` smoke test's `[CTRL_FIRE]` assertion to drop the `clip` column. + +- [ ] **Step 4: (If KEEP) annotate reclassification** + +Add a comment at the `adaptive_clip_value` declaration in `gpu_dqn_trainer.rs`: + +```rust +/// ## LOAD-BEARING — do not delete +/// +/// Track 3 Phase 2 ablation (Task 2.7) showed disabling the adaptive clip +/// causes . Keep. +/// +/// Reclassified from CANDIDATE-FOR-DELETE to LOAD-BEARING at commit +/// . Flag in design spec §5.3 production-readiness notes. +``` + +Flag in the phase3 results doc (Task 2.9) that this controller became load-bearing. + +- [ ] **Step 5: Compile + full smoke suite** + +```bash +SQLX_OFFLINE=true cargo check --workspace +touch crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs +SQLX_OFFLINE=true cargo build -p ml --release 2>&1 | tail -5 + +for t in magnitude_distribution reward_component_audit controller_activity \ + exploration_coverage multi_fold_convergence; do + SQLX_OFFLINE=true CUBLAS_WORKSPACE_CONFIG=:4096:8 \ + FOXHUNT_TEST_DATA=test_data/futures-baseline \ + cargo test --profile release-test -p ml --lib -- \ + "trainers::dqn::smoke_tests::$t" --ignored --nocapture 2>&1 | tail -3 +done +``` +All five must pass. + +- [ ] **Step 6: Commit + push** + +DELETE: +```bash +git commit -am "cleanup(dqn): DELETE C4 adaptive grad-clip per track3 ablation + +Task 2.7 ablation: fixing adaptive clip to 1e6 (never triggers) leaves +controller_activity + multi_fold_convergence + magnitude_distribution +within baseline noise. Per track3 §C4 decision matrix, DELETE. + +Removed: + - adaptive_clip_value EMA controller in GpuDqnTrainer + - fused_training adaptive-clip call site (replaced with fixed + gradient_clip_norm = 1.0 read from config) + - HEALTH_DIAG fire_clip + clip={...} field + - controller_fire_counts.grad_clip slot + CTRL_FIRE assertion + +~80 LOC net reduction. Per-component gradient budgets (IQN/CQL/C51/Ens) +are unchanged — those are fixed allocation, not adaptive controllers." +git push +``` + +KEEP: +```bash +git commit -am "docs(dqn): C4 adaptive grad-clip reclassified LOAD-BEARING per track3 ablation + +Task 2.7 ablation showed . C4 stays; annotated +gpu_dqn_trainer.rs with the evidence. Flagged in phase3-results.md for +sub-project B production-readiness follow-up." +git push +``` + +--- + +## Task 2.8: L40S validation run — re-run all triages with fixes applied + +**Why:** Phase 1 triages were preliminary-smoke-scale. Tasks 2.0–2.7 are the behavioural changes. A single L40S run is the design spec §7.1 gate AND the re-measurement needed to promote preliminary verdicts. Matches the plan's Phase 3 task structure. + +**Files:** +- No code changes. Orchestration only. + +- [ ] **Step 1: Confirm main HEAD has all fixes + is pushed** + +```bash +git status +git log --oneline -15 +git log origin/main..HEAD --oneline # must be empty — local is at origin +``` +Expected: clean tree, Tasks 2.0–2.7 commits present, `git push` has landed everything. + +- [ ] **Step 2: Launch the L40S validation workflow** + +```bash +./scripts/argo-train.sh dqn --gpu-pool ci-training-l40s --epochs 50 --baseline +``` + +Capture the workflow name (e.g. `train-policy-quality-phase2-abc123`) for the rest of Task 2.8: + +```bash +WF=$(argo list -n foxhunt --status Running | grep train | head -1 | awk '{print $1}') +echo "$WF" > /tmp/policy-quality-phase2-wf.txt +``` + +- [ ] **Step 3: Stream training logs until completion** + +```bash +argo watch -n foxhunt $(cat /tmp/policy-quality-phase2-wf.txt) +``` +Expected: ~1 hour. + +Do NOT run concurrent tasks that change config or code during this run — the checkpoints produced here feed Task 2.9. + +- [ ] **Step 4: Archive the full log** + +```bash +argo logs -n foxhunt $(cat /tmp/policy-quality-phase2-wf.txt) \ + > /tmp/foxhunt_l40s/phase2-validation.log +``` + +- [ ] **Step 5: Extract all 4 track re-measurements** + +Track 1 — magnitude: +```bash +grep "HEALTH_DIAG" /tmp/foxhunt_l40s/phase2-validation.log | \ + awk -F'mag \\[' '{print $2}' | awk -F'\\]' '{print $1}' > /tmp/phase2-track1.txt +grep "eval_dist" /tmp/foxhunt_l40s/phase2-validation.log | \ + awk -F'eval_dist \\[' '{print $2}' | awk -F'\\]' '{print $1}' >> /tmp/phase2-track1.txt +``` + +Track 2 — reward contrib: +```bash +grep "reward_contrib" /tmp/foxhunt_l40s/phase2-validation.log | \ + awk -F'reward_contrib \\[' '{print $2}' | awk -F'\\]' '{print $1}' > /tmp/phase2-track2.txt +``` + +Track 3 — controllers: +```bash +grep "CTRL_FIRE" /tmp/foxhunt_l40s/phase2-validation.log | tail -10 \ + > /tmp/phase2-track3.txt +``` + +Track 4 — exploration: +```bash +grep "explore" /tmp/foxhunt_l40s/phase2-validation.log | \ + awk -F'explore \\[' '{print $2}' | awk -F'\\]' '{print $1}' > /tmp/phase2-track4.txt +grep "grad_split" /tmp/foxhunt_l40s/phase2-validation.log | \ + awk -F'grad_split \\[' '{print $2}' | awk -F'\\]' '{print $1}' >> /tmp/phase2-track4.txt +``` + +- [ ] **Step 6: Write the re-measurement annexe** + +Create `docs/superpowers/specs/2026-04-21-policy-quality-phase2-validation.md` with: +- Workflow name + commit SHA + tag reference (ideally `policy-quality-phase2-complete` once tagged at Task 2.10) +- Track 1 → H4 verdict (was CONFIRMED, now ???), H10 verdict (was CONFIRMED, now ???), preliminaries promoted +- Track 2 → R2 PopArt drift promoted (was PENDING — verdict now?), R4 trail rate at production volume +- Track 3 → C2/C3/C5/C6/C7 promoted from preliminary; C4 ablation verdict; C1 wiring-fix-validated +- Track 4 → E2 (post sigma_mean fix) promoted; E3 fold-3 entropy replication; E4 decision post-fix +- Cross-references between tracks (which L40S observations change which track's smoke-scale verdicts) + +Commit the doc: + +```bash +git add docs/superpowers/specs/2026-04-21-policy-quality-phase2-validation.md +git commit -m "docs(policy-quality): Phase 2 L40S re-measurement per all 4 triages + +Workflow: $(cat /tmp/policy-quality-phase2-wf.txt) +Log: /tmp/foxhunt_l40s/phase2-validation.log (local — not committed)" +git push +``` + +--- + +## Task 2.9: Mandatory-gate verification + `phase3-results.md` + +**Why:** design spec §2.1 mandatory gates must pass for the spec to close. Task 2.8 produced the raw numbers; Task 2.9 translates them into gate pass/fail + commits the verdicts. + +**Files:** +- New: `docs/superpowers/specs/2026-04-21-policy-quality-phase3-results.md` + +- [ ] **Step 1: Extract Best Sharpe per fold** + +```bash +grep "Best Sharpe" /tmp/foxhunt_l40s/phase2-validation.log \ + | tee /tmp/phase2-sharpe.txt +``` +Expected: 6 lines (one per fold). + +- [ ] **Step 2: Extract WinRate per fold** + +```bash +grep "Validation backtest:" /tmp/foxhunt_l40s/phase2-validation.log \ + | tee /tmp/phase2-winrate.txt +``` + +- [ ] **Step 3: Extract action-distribution per fold (F_Half, F_Full)** + +```bash +grep -E "action_dist|F_Half|F_Full|LOW EXPOSURE" /tmp/foxhunt_l40s/phase2-validation.log \ + | tee /tmp/phase2-actions.txt +``` + +- [ ] **Step 4: Extract trade count per val window** + +```bash +grep "trades" /tmp/foxhunt_l40s/phase2-validation.log \ + | grep -iE "val|validation" | tee /tmp/phase2-trades.txt +``` + +- [ ] **Step 5: Run surrogate-noise check against the fresh checkpoint** + +Download `dqn_fold0_best.safetensors` etc. from the workflow artifact path (see plan §Task 3.3). Then: + +```bash +SQLX_OFFLINE=true CUBLAS_WORKSPACE_CONFIG=:4096:8 \ + FOXHUNT_TEST_DATA=test_data/futures-baseline \ + cargo test --profile release-test -p ml --lib -- \ + surrogate_noise_check --ignored --nocapture 2>&1 | tee /tmp/phase2-surrogate.log +``` +Expected: pooled trained Sharpe > 95th-percentile of N=30 surrogate random-action Sharpes (design spec §4.1 definition). + +- [ ] **Step 6: Gate evaluation** + +Populate `docs/superpowers/specs/2026-04-21-policy-quality-phase3-results.md`: + +```markdown +# Policy Quality — Phase 3 Validation Results + +**Workflow:** +**Commit under validation:** +**Tag:** `policy-quality-phase2-complete` (Task 2.10 pending this file) +**L40S run:** 6 folds × 50 epochs, `ci-training-l40s` pool +**Log:** /tmp/foxhunt_l40s/phase2-validation.log (local; size prohibits committing) + +## Mandatory gates (design spec §2.1) + +| Gate | Pass condition | Observed | Pass/Fail | +|---|---|---|---| +| B-1 Multi-fold Sharpe | Best val Sharpe > 10 on ≥5/6 folds | | | +| B-2 Multi-fold WinRate | Val WinRate > 55% on ≥5/6 folds | | | +| SN Surrogate-noise | Pooled Sharpe > 95th-%ile of N=30 | | | + +## Soft gates (design spec §2.2) + +| Gate | Pass condition | Observed | Pass/Fail | +|---|---|---|---| +| A-1 Action-dist | F_Half ≥ 15% AND F_Full ≥ 15% on ≥4/6 folds | | | +| A-2 WinRate range | ≥ 55% across all folds | | | +| A-3 Trade count | ≥ 100 trades per val window on ≥5/6 folds | | | +| CA Controller-activity | No controller > 50% epochs | | | + +## Additional user-specified gates + +| Gate | Pass condition | Observed | Pass/Fail | +|---|---|---|---| +| F_Half ≥ 15% at eval (production threshold) | | | | +| F_Full ≥ 15% at eval (production threshold) | | | | +| grad_ratio_mag_dir ≥ 0.2 on most epochs | | | | +| eval_dist eh + ef ≥ 0.3 | | | | +| 5/5 Phase 0 smokes still pass | | | | + +## Outcome + + documented for follow-up" / "Mandatory miss on ; Phase 2 iteration N of 3 per design spec §2.3"> +``` + +Fill every `<...>` with actual values before committing. No placeholders. + +- [ ] **Step 7: Commit the results doc** + +```bash +git add docs/superpowers/specs/2026-04-21-policy-quality-phase3-results.md +git commit -m "$(cat <<'EOF' +docs(policy-quality): Phase 3 validation results — Phase 2 L40S run + +Mandatory gates (B-Sharpe, B-WinRate, SN surrogate-noise): +Soft gates (A-1/A-2/A-3, CA): +Additional user gates (F_Half ≥ 15%, F_Full ≥ 15%, grad_ratio ≥ 0.2, +eh+ef ≥ 0.3, 5/5 smokes): + +Next action: +EOF +)" +git push +``` + +- [ ] **Step 8: Branch decision per design spec §2.3** + +- All mandatory pass → **Task 2.10** tags `policy-quality-phase2-complete`. If all soft pass too, also tag `policy-quality-v1`. If soft misses, open a follow-up spec. +- Any mandatory miss → this is iteration 1 of 3 allowed per §2.3. Diagnose from log, add a fix commit, **RE-RUN TASK 2.8**. Track iterations in phase3-results.md. If 3 iterations fail, escalate per §2.3 (open `docs/superpowers/specs/YYYY-MM-DD-policy-quality-triage.md` and **do not tag**). + +--- + +## Task 2.10: Tag `policy-quality-phase2-complete` (and optionally `policy-quality-v1`) + +**Why:** permanent artefact + rollback anchor for sub-project B dependency. + +**Prerequisites:** Task 2.9 shows all MANDATORY gates pass. + +- [ ] **Step 1: Confirm HEAD has results doc** + +```bash +git log -1 --oneline +git status # must be clean +``` + +- [ ] **Step 2: Tag Phase 2 complete** + +```bash +git tag -a policy-quality-phase2-complete -m "$(cat <<'EOF' +Policy quality V7 audit — Phase 2 complete. + +Synthesis of 4 Phase 1 tracks landed: + - Track 1 H4 CONFIRMED → gradient-flow fix (Task 2.1) + - Track 1 H10 CONFIRMED → argmax tie-break (Task 2.2) + - Track 2 R5 DELETE → micro-reward removed (Task 2.3) + - Track 2 R6 DELETE → loss-aversion relocated to C51 smoothing (Task 2.4) + - Track 2 R7 → stale docstring deleted (Task 2.5 sub-commit) + - Track 3 C1 wiring bug fixed (Task 2.5 sub-commit) + - Track 3 C4 ablation → -LOAD-BEARING (Task 2.7) + - Track 4 E2 instrumentation TUNE (Task 2.5 sigma_mean fix) + - Track 4 E4 → post-Task-2.0 data (Task 2.6) + +Phase 3 L40S validation: all MANDATORY gates passed. +See docs/superpowers/specs/2026-04-21-policy-quality-phase3-results.md +EOF +)" +git push origin policy-quality-phase2-complete +``` + +- [ ] **Step 3: (If all soft also passed) tag `policy-quality-v1`** + +```bash +git tag -a policy-quality-v1 -m "$(cat <<'EOF' +Policy quality V7 audit — sub-project A closed. + +All MANDATORY + all soft gates passed in Phase 3 validation. +See docs/superpowers/specs/2026-04-21-policy-quality-phase3-results.md +EOF +)" +git push origin policy-quality-v1 +``` + +- [ ] **Step 4: Update spec status header** + +At the top of `docs/superpowers/specs/2026-04-21-policy-quality-design.md`, insert: + +```markdown +**Status (2026-MM-DD): CLOSED — tagged `policy-quality-v1`.** +(Or: **Status (2026-MM-DD): Phase 2 complete — tagged `policy-quality-phase2-complete`. Soft-gate follow-up: docs/superpowers/specs/YYYY-MM-DD-policy-quality-followup.md.**) +**Phase 3 results:** `docs/superpowers/specs/2026-04-21-policy-quality-phase3-results.md` +``` + +Commit + push: + +```bash +git add docs/superpowers/specs/2026-04-21-policy-quality-design.md +git commit -m "docs(policy-quality): mark Phase 2 complete, " +git push +``` + +- [ ] **Step 5: If iterations exceeded, open triage spec (fallback path)** + +If Task 2.9 exited with any mandatory gate failing after 3 L40S runs: + +```bash +NEW_SPEC="docs/superpowers/specs/$(date +%Y-%m-%d)-policy-quality-triage.md" +# Populate per design spec §2.3 template: +# - Summary of 3 failed runs + correlating hypotheses +# - Diagnosis: policy / reward / state / architecture / data bottleneck +# - Decision: continue with new hypotheses OR scope reward/architecture/algorithm change +git add "$NEW_SPEC" +git commit -m "spec(policy-quality): Phase 2 triage — 3-iteration failure per design spec §2.3" +git push +``` + +Do NOT tag `policy-quality-v1` on a failing run. Spec is paused, not closed. + +--- + +## Summary — what this plan delivers if executed end-to-end + +- **Task 2.0** locates which loss component starves the magnitude head (keystone diagnostic) +- **Tasks 2.1, 2.2** fix the two Track-1 CONFIRMED findings (H4 gradient starvation, H10 argmax tie-break) +- **Tasks 2.3, 2.4** delete two dead reward terms (R5 micro, R6 loss-aversion) and relocate the one genuine invariant (neg-tail compression → C51 target smoothing) +- **Task 2.5** sweeps 7 wiring bugs in one pass: C1 fire detection, stale 5-level epsilon-greedy, dead `if !true` block, broken `sigma_mean` accessor, fold-boundary counter reset, stale kernel docstring, C5 ISV null-fallback +- **Tasks 2.6, 2.7** execute the two ablation-pending decisions with explicit decision trees (E4 entropy, C4 grad-clip) +- **Tasks 2.8, 2.9** run the L40S validation + grade against all 3 mandatory gates + 4 soft gates + 5 additional user-specified gates +- **Task 2.10** tags the successful commit (or opens a triage spec on failure) + +**Net code delta estimate:** ~250 LOC added (Task 2.0 instrumentation + Task 2.4 relocation + Task 2.5 fixes + regression assertions) minus ~260 LOC removed (R5 + R6 + possibly E4 + possibly C4) = net ~-10 LOC. The plan is simplifying the policy training stack while closing the magnitude collapse.