gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(policy-quality): Task 2.X scoping — per-bin variance + Q-spread regularization

Browse Source 
Scoping output from the Full=0% investigation (post-Task-2.2 smoke showed
eval_dist [eq=0.580 eh=0.420 ef=0.000] — Quarter+Half recovered,
Full still 0%).

Root cause: Q-estimation bias (category C) — C51 expected-Q over atoms
systematically under-prices higher-variance bins. Full has 4x Quarter's
PnL variance per experience_kernels.cu risk scaling. Bias is already
documented at experience_kernels.cu:904 (commented as "structural
distributional bias — tight return distributions (Small) get higher
expected Q under the C51 softmax regardless of actual expected returns").

Not noise-starvation: Full picked on 32% of training samples across
60 epochs × 3 folds = ~61k Full samples. Ample data.

Proposed Task 2.X = Rank 1 + Rank 2 combined (~90 LOC):
  Rank 1: per-bin variance weighting in C51 branch loss — amplify
          Full's gradient by sqrt(var_f / mean_var)
  Rank 2: Q-spread regularization — lambda * ReLU(target_spread -
          Q_spread)^2 penalty preventing degenerate fixpoint

Alternative fixes ranked but not recommended:
  Rank 3 per-magnitude reward shaping (~80 LOC) — only if data-disfavor
         is confirmed at L40S
  Rank 4 magnitude curriculum (~20 LOC) — rejected (noise-starvation ruled out)
  Rank 5 state-vector enrichment (~150 LOC) — premature

Prerequisite: ~40 LOC of per-magnitude win-rate + realized-variance
instrumentation to confirm category C vs A at L40S scale.

L40S gating recommendation was:
  ef >= 0.15 at L40S → Task 2.X not needed (smoke artefact)
  0.05 <= ef < 0.15 → Task 2.X scoped but optional
  ef < 0.05 → Task 2.X required

Per feedback_fix_aggressively.md (landed this session): we ship Task 2.X
ahead of L40S since the bias is already code-documented, the fix is
well-scoped, and the 90-LOC cost is reasonable. L40S validation (Task
2.8) will verify the fixed state end-to-end instead of the un-fixed one.

No deletions anywhere per feedback_no_functionality_removal.md.
This commit is contained in:
jgrusewski
2026-04-22 11:43:17 +02:00
parent 7b74290dd0
commit f9286e938d
1 changed files with 309 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

309
docs/superpowers/specs/2026-04-21-policy-quality-task2x-scoping.md Normal file
View File

@@ -0,0 +1,309 @@
# Full Magnitude Fix — Scoping Doc (Task 2.X-make-Full-useful)
**Repo:** /home/jgrusewski/Work/foxhunt
**Branch:** `main`
**HEAD at scoping time:** `c0fee5a9b` (post-Task 2.2 H10 landing)
**Author:** researcher (scoping only, NO implementation)
**Date:** 2026-04-22
---
## 1. Problem statement
Task 2.2 landed `fix(dqn): H10 — stable argmax tie-break at eval` at commit `8aef59f73` (2026-04-22 11:24 +0200). Per that commit's body, the 20-epoch × 3-fold smoke result is:
```
Post-fix (this commit):
training dist [dist_q=0.577 dist_h=0.166 dist_f=0.257] (unchanged — eval-only path)
eval dist [eq=0.580 eh=0.420 ef=0.000]
```
`eh + ef = 0.420 ≥ 0.30` → smoke gate passes (`crates/ml/src/trainers/dqn/smoke_tests/magnitude_distribution.rs:69`). **But Full (ef) stays 0.000 at eval.** Training-mode Full is selected ~25 % of the time via NoisyNets + Philox-seeded Boltzmann, but strict-argmax eval never picks it because Q(Half) > Q(Full) + 1e-6 in the vast majority of eval states.
Standing rule `feedback_magnitude_must_be_useful.md` (+ `feedback_no_functionality_removal.md`) forbids deletion of Full or the magnitude branch. If the data genuinely doesn't differentiate Full from Half, we must **make Full useful** — not remove it.
---
## 2. Evidence summary — what the data shows
### 2.1 Log source
Primary data: `/tmp/foxhunt_smoke/grad_abs_confirmation.log` (2607 lines, 60 HEALTH_DIAG emissions — 20 epochs × 3 folds, run at 2026-04-22 08:58, **pre-H10**). Training-mode fields (`dist_q/h/f`, `q_full/half/quarter`) are bit-identical between pre- and post-H10 per the H10 commit body, so this log IS representative of the current training trajectory. The post-fix eval-mode numbers come from the H10 commit body directly (re-running the smoke to regenerate the log is a parallel item — see §8 instrumentation).
### 2.2 Per-epoch Q-value trajectory (`q_full` vs `q_half` vs `q_quarter`)
Parsed from `HEALTH_DIAG[n]: ... mag [q_full=... q_half=... q_quarter=...]`. Source lines: `grad_abs_confirmation.log:329, 372, 410, 445, 480, …` (every HEALTH_DIAG emission).
| Fold | n=20 epochs | mean `q_full − q_half` | range `q_full − q_half` | F-wins (>1e-6) | F-ties (<1e-6) | F-loses | mean `dist_f` (training) | mean `eval_ef` (pre-H10) |
|------|-------------|------------------------|-------------------------|----------------|----------------|---------|--------------------------|--------------------------|
| 1 | 20 | +0.00050 | [-0.0030, +0.0050] | 8 | 5 | 7 | 0.375 | 0.000 |
| 2 | 20 | +0.00115 | [-0.0100, +0.0110] | 12 | 0 | 8 | 0.338 | 0.000 |
| 3 | 20 | +0.16745 | [+0.0030, +1.5960] | 20 | 0 | 0 | 0.247 | 0.000 |
| All | 60 | +0.05637 | [-0.0100, +1.5960] | 40 | 5 | 15 | 0.320 | 0.000 |
**Key observations:**
- The aggregate-level `q_full − q_half` scalar (a running-buffer mean) actually **favours Full in 40/60 epochs** and is never more than 0.01 in favour of Half. Median gap is +0.004. So at the buffer-aggregate level Q(Full) is NOT systematically below Q(Half).
- Fold 3 shows Q(Full) diverging sharply above both Q(Half) and Q(Quarter) by epoch 14 — `q_full=+1.007 q_half=-0.181` (`grad_abs_confirmation.log:2414`), `q_full=+0.920 q_half=-0.676` at epoch 15. Training Full share drops to ~24 % exactly when Q(Full) dominates — `dist_q` climbs to 0.62–0.67 because action selection goes through Boltzmann over the SAMPLED q-column, not the buffer-aggregate. Epoch 15's atom stats are already pathological (C51 collapse symptom, separate issue).
- Training-mode `dist_f` averages 32 % across the run — Full is picked with similar training frequency to Quarter and Half. `dist_h` 30 %, `dist_q` 38 %.
- The **contradiction to resolve**: buffer-aggregate Q-scalars show Full ≈ Half (±0.006 typical), but the eval strict-argmax sees `ef = 0.000`. Conclusion: per-state Q(Full) vs Q(Half) is not reflected in the per-epoch aggregate — the eval-time argmax decisions are made at the PER-STATE level, and in the 3200-sample eval rollout, Q(Half) > Q(Full) + 1e-6 dominantly. The buffer-mean hides the per-state preference.
### 2.3 Training-mode action distribution (per GPU monitoring kernel)
Log lines: `grad_abs_confirmation.log:2600` backwards — `GPU epoch summary: ... actions=[S_Small, S_Half, S_Full, H_Small, H_Half, H_Full, L_Small, L_Half, L_Full, F_Small, F_Half, F_Full]`. Layout from `crates/ml/src/trainers/dqn/monitoring.rs:18-23`.
Representative late-fold-3 line (`grad_abs_confirmation.log:2414` region, epoch 19): `actions=[736, 160, 245, 299, 5, 8, 871, 209, 351, 307, 6, 3]` on 3200 samples:
- Short-Full = 245 (7.7 %)
- Long-Full = 351 (11.0 %)
- **Directional Full (S_Full + L_Full) = 596 / 3200 = 18.6 %**
Per `LOW EXPOSURE DIVERSITY` warnings (`crates/ml/src/trainers/dqn/monitoring.rs:244`), S_Full and L_Full never triggered `< 5 %` warnings across 60 epochs — only H_Full and F_Full (unreachable by kernel construction, magnitude forced to 0 on Hold/Flat per lines 898-899 of `experience_kernels.cu`). Training Full-for-real-trades is NOT rare.
### 2.4 Sample count for Full during training
- Per epoch: 3200 samples (`LOW EXPOSURE ...(0/3200)` in `grad_abs_confirmation.log:337-340`).
- Per fold: 3200 × 20 = 64k.
- Across 3 folds: 192k.
- At `dist_f ≈ 0.32` overall (§2.2 aggregate), Full sees ≈ 61 k training samples.
- At `S_Full + L_Full ≈ 19 %` per-epoch (§2.3), directional Full sees ≈ 36 k training samples.
**36 k training samples is NOT noise-starvation at smoke scale.** DQN literature routinely converges action-value estimates with an order of magnitude fewer samples per bin for a 3-bin branch. If Full is under-learning, it's not because the optimizer hasn't seen it.
### 2.5 Per-magnitude win rate — NOT currently exposed
Search performed:
- `Grep` pattern `per_magnitude.*reward|magnitude.*win_rate|win_rate_full|win_rate_half|win_rate_quarter` → **no matches** in `crates/ml`.
- `Grep` pattern `avg_win_ratio|win_rate` in `crates/ml/src/trainers/dqn` → only global `win_rate` via `financials.rs:12` and `last_epoch_win_rate` (not broken down by magnitude).
**Consequence: we cannot directly read "win rate for Full-sized trades" from the current run.** See §8 for the small instrumentation add needed to pin the root-cause decision.
### 2.6 Structural reward pipeline — how Full gets paid
- `experience_kernels.cu:1471-1477` — risk scaling fires BEFORE magnitude mapping; Full = 1.0×, Half = 0.5×, Quarter = 0.25× of `effective_max_pos`. So Full's realized step-return is 4× Quarter's for the same directional outcome, before noise.
- `experience_kernels.cu:1853-1855` — "Magnitude-neutral normalization REMOVED — gave Small 4x LARGER sparse reward, contradicting dense micro-reward." The current code relies on vol-normalization (`experience_kernels.cu:1857-1872`) which divides realized return by `atr_norm * sqrt(hold_time)`, meaning in bars with rising volatility the advantage of Full over Quarter narrows — and since the volatility-scale cap (`asymmetric_soft_clamp`, `experience_kernels.cu:1872`) clips large positive returns more softly than large negative ones, Full's upside tail is *more* compressed than its downside.
- No per-magnitude reward shaping, no per-bin advantage weighting, no magnitude curriculum, no state features specifically designed for the "Full is right" regime.
### 2.7 Summary of what the data says
1. **Buffer-aggregate Q(Full) ≈ Q(Half)** (mean gap +0.006 across 60 epochs) — the two are almost indistinguishable at the scalar-summary level.
2. **Per-state Q(Half) > Q(Full) + 1e-6** in nearly 100 % of eval states (observed via `ef = 0.000` post-H10, meaning the argmax tie-break almost never fires on Full). Structurally: Half wins on a per-state basis even though the training-buffer mean says they tie.
3. **Training-mode Full IS explored** (~19 % directional, ~32 % aggregate including unreachable H_Full / F_Full buckets) — noise-starvation is NOT the primary constraint.
4. **No per-magnitude reward shaping currently exists** — reward differentiation between bins is purely structural (4× PnL scaling via position size, then vol-normalised + asymmetric-soft-clamped).
5. **Per-magnitude win rate is NOT observable** from current instrumentation. We cannot currently distinguish "Full is genuinely less profitable than Half at this data scale" from "Full has the same win rate but lower Q-value estimate due to higher realized-return variance".
---
## 3. Root-cause classification
Given the four candidate causes from the task spec, the evidence pattern matches:
| Cause | Data support | Rank |
|-------|--------------|------|
| (A) **Data genuinely doesn't reward Full** at smoke scale | Partially supported: buffer-aggregate Q(Full) ≈ Q(Half) despite 36 k training samples. But we CANNOT confirm without per-magnitude win rate (§2.5). | Tied-1 |
| (B) **Noise-starvation / under-sampling** | REJECTED. 36 k directional-Full samples is ample for a 3-bin branch. `ent_mag ≈ 0.99` confirms training exploration is healthy. | Low |
| (C) **Q-value estimation bias** (e.g. C51's expected-Q favouring lower-variance outcomes — the same mechanism that motivated Boltzmann-at-training per `experience_kernels.cu:904-910`) | Strongly supported. The exact mechanism the `experience_kernels.cu:904-910` comment names ("C51's expected Q for magnitude has a monotonic bias toward Small — same pattern as direction Flat collapse — target network feedback loop") is documented as a KNOWN risk that Boltzmann-training was meant to paper over. The fact that per-state Q(Half) > Q(Full) + 1e-6 almost everywhere, while buffer-mean Q's are tied, is the diagnostic signature of this bias: softmax-over-tied-mean masks deterministic-per-state preference for the lower-variance bin. | Tied-1 |
| (D) **State representation inadequate** to separate Q(Full)-favourable vs Q(Half)-favourable regimes | Possible but secondary. 40-feature + OFI vector includes ADX + CUSUM (regime signals), so it's not obviously under-specified. Would need state-ablation run to confirm. | Medium |
**Primary diagnosis: (C) Q-estimation bias toward lower-variance bins, with (A) as a mutually-reinforcing secondary.** Full has 4× PnL variance vs Quarter for the same directional skill (unchanged per §2.6). C51's expected-Q computed over atoms systematically favours the tighter distribution (Half over Full, Quarter over Half). The same bias that motivates Boltzmann-at-training for exploration — now at eval the strict-argmax respects the biased estimate and never picks Full.
**Why (A) reinforces (C):** since per-magnitude win rate is not exposed yet, we genuinely can't rule out that Full under-performs Half at this 5000-bar slice. But the correct response to (A) at smoke scale isn't to accept it — it's to (i) add the instrumentation, (ii) re-run at L40S production scale, and (iii) if (A) is confirmed at production scale, apply per-magnitude shaping so Full-when-right gets amplified signal.
---
## 4. Ranked candidate fixes (tied-1 diagnosis → combined fix)
Per the `feedback_magnitude_must_be_useful.md` option set:
| Rank | Fix | Fits diagnosis | Estimated LOC | Risk |
|------|-----|----------------|---------------|------|
| 1 | **Per-magnitude advantage-weighting in C51 branch loss** — scale per-bin advantage gradient by sqrt of realized per-bin variance so the higher-variance bins (Full > Half > Quarter) get a gradient large enough to overcome C51's expected-Q tightness bias | (C) directly; (A) indirectly via forcing Full's gradient through even when raw advantage is small | ~60 LOC (kernel + host) | Medium — new per-bin weight in distributional loss is a first-order gradient change |
| 2 | **Q-regularisation preventing Q_full ≈ Q_half fixpoint** — add a small penalty term `-λ * Var(Q_quarter, Q_half, Q_full)` to the branch loss so the network is rewarded for spreading the three magnitudes apart (Bellman still dominates; penalty is a tiebreaker for degenerate fixpoints) | (C) directly; prevents collapse but doesn't ensure spread is in the "right" direction | ~30 LOC | Low — penalty is monotone; if λ is small, Bellman dominates |
| 3 | **Per-magnitude reward shaping** — multiply realized step return by a per-bin factor derived from rolling per-magnitude Sharpe (increases Full's reward amplitude when recent trades show Full outperformed) | (A) directly if confirmed at L40S | ~80 LOC (kernel + rolling state + host stats) | Medium-high — reward hacking risk; must be careful not to over-shape |
| 4 | **Magnitude curriculum** — force epsilon-exploration of Full for first N epochs of each fold | (B) — but (B) was REJECTED; only marginally useful on top of ranks 1-3 | ~20 LOC | Low |
| 5 | **State-vector enrichment** — add realized-volatility-regime features that separate "Full-is-right" regimes from "Quarter-is-right" regimes | (D) — secondary; only justified after 1-2 fail | ~150 LOC (feature computation + serialization) | High — feature engineering without clear ablation data is premature |
**Recommended Phase 2 fix: Rank 1 (Per-magnitude advantage-weighting), combined with Rank 2 (Q-spread regularisation) as a cheap safety net.** Ranks 3-5 are held back pending L40S data (§6).
---
## 5. Proposed Task 2.X-make-Full-useful outline
> **Only implemented if** the L40S production run from Task 2.8 shows `eval ef < 0.15` (per §6 gate). If L40S `ef ≥ 0.15`, this task is **not needed** — the smoke-scale `ef = 0.000` is a data-scale artefact and production separates Full naturally.
### Task spec
**Why:** post-Task 2.2 eval dist `[eq=0.580 eh=0.420 ef=0.000]` shows Full is never picked at eval despite being picked 25-32 % of the time during training. Per-state Q(Half) > Q(Full) + 1e-6 dominates; root cause is C51's expected-Q bias toward lower-variance bins (the same mechanism the Boltzmann-training comment at `experience_kernels.cu:904-910` names). The structural 4× PnL scaling for Full vs Quarter is preserved but the C51 atom distribution for Full has 16× higher realized variance, so the expected value (sum_atoms p_a × z_a) falls below Half's in most states. Fix by (a) weighting the per-bin distributional loss by realized variance so Full's gradient is amplified enough to push Q(Full) above the fixed point, and (b) adding a small spread-regularisation penalty preventing degenerate Q_quarter ≈ Q_half ≈ Q_full fixpoints.
### Files
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` — add per-bin realised-variance tracking (new output buffer `per_mag_realized_var[3]` reduced in the epoch-boundary host-side step; must use per-sample array → host-reduce pattern per `feedback_gpu_cpu_roundtrip.md`, NEVER atomicAdd per `feedback_no_atomicadd.md`)
- Modify: `crates/ml/src/cuda_pipeline/fused_training.rs` — pipe per-bin variance vector into the C51 branch loss backward kernel as a per-bin multiplier
- Modify: `crates/ml/src/cuda_pipeline/c51_loss_kernel.cu` (or equivalent — locate via `grep -rn c51.*backward crates/ml/src/cuda_pipeline/`) — multiply per-bin gradient by `sqrt(realized_var_bin / mean_realized_var)` (amplify Full, dampen Quarter; Half ≈ 1×); add spread-regularisation term `λ · (Q_max_bin − Q_min_bin − target_spread)²` with `target_spread = 0.05 · q_range`, `λ = 0.01`
- Modify: `crates/ml/src/trainers/dqn/config.rs` — add `magnitude_spread_reg_lambda: f32` (default 0.01), `magnitude_variance_weight_enabled: bool` DO NOT use — per `feedback_no_feature_flags.md`, wire unconditionally or not at all
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs` — extend HEALTH_DIAG with `mag_variance_weights [w_q, w_h, w_f]` and `q_spread_penalty` fields for observability
- Modify: `crates/ml/src/trainers/dqn/smoke_tests/magnitude_distribution.rs` — tighten the `eh + ef ≥ 0.30` gate to `eh ≥ 0.15 AND ef ≥ 0.15` (both must be present). Add a new assertion: `ef ≥ 0.05` standalone. This is NOT a quickfix relaxation — it's the exact Task 2.9 production gate per `docs/superpowers/plans/2026-04-21-policy-quality-phase2.md` §2.9
### Steps
1. **[ ] Step 1 — Instrument per-magnitude realised variance + win rate**
- Add `per_mag_realized_ret_per_sample[i*3 + mag_idx]` = (step return when that bin was taken, else 0) per-sample buffer in `experience_kernels.cu`'s trade-completion branch.
- Add matching `per_mag_trade_count_per_sample[i*3 + mag_idx]` (1 when taken, 0 else).
- Host-side reduce at epoch boundary to `[sum_ret_q, sum_ret_h, sum_ret_f, n_q, n_h, n_f, sum_sq_q, sum_sq_h, sum_sq_f]`, compute per-bin mean + variance + winning-trade fraction.
- Emit in HEALTH_DIAG: `mag_stats [win_q=.. win_h=.. win_f=.. var_q=.. var_h=.. var_f=..]`.
- **Validation:** run the `magnitude_distribution` smoke and confirm the new fields are non-zero in HEALTH_DIAG[n] for n ≥ 2 (bins need some trade history).
2. **[ ] Step 2 — Weight the C51 branch loss per bin by sqrt(realized_var / mean_var)**
- In `fused_training.rs`, pass the cached `per_mag_realized_var[3]` host array to the C51 branch backward kernel.
- Inside the kernel, for branch-1 (magnitude) gradient accumulation, multiply the per-bin gradient by `sqrt(var_bin / mean_var + 1e-3)` — normalised so mean weight across 3 bins is 1.0 (preserves total gradient energy).
- **Do NOT guard behind a runtime boolean** — per `feedback_no_feature_flags.md`, wire unconditionally.
- **Validation:** observe that `grad_split_bwd c51` for the magnitude branch increases by ≥ 2× when `var_full / var_q ≈ 10×` (expected at smoke scale). Confirm via HEALTH_DIAG `grad_abs[mag]` moving up proportionally.
3. **[ ] Step 3 — Add Q-spread regularisation**
- In the branch-1 loss kernel, add a scalar penalty `λ · ReLU(target_spread − (Q_max − Q_min))²` where `target_spread = 0.05 · (v_max − v_min)` and `λ = 0.01`.
- Penalty gradient pushes Q_max up and Q_min down, preventing the degenerate fixpoint where all three converge.
- Add `magnitude_spread_reg_lambda` to config with default 0.01; load via `training_profile/dqn-smoketest.toml` and `dqn-production.toml`.
- **Validation:** HEALTH_DIAG `q_spread_penalty` should emit a non-zero scalar for epochs where the three magnitudes are within 5 % of `v_range`. As the penalty fires the Q-spread should widen at subsequent epochs.
4. **[ ] Step 4 — Tighten smoke gate**
- `crates/ml/src/trainers/dqn/smoke_tests/magnitude_distribution.rs:68-76` — replace the `eh + ef ≥ 0.30` assertion with three separate: `eh ≥ 0.15`, `ef ≥ 0.05` (smoke floor; production is 0.15), and `eh + ef ≥ 0.30` (keep the combined floor).
- The smoke floor for `ef` is 0.05 (not 0.15) because smoke is 5000 bars × 20 epochs and the Task 2.8 L40S run is the authoritative gate for the 15 % threshold. Document this rationale inline.
- **Validation:** the smoke test fails with current (pre-Task 2.X) behaviour (`ef = 0.000 < 0.05`). After Steps 1-3 land, it passes.
5. **[ ] Step 5 — Smoke parity + regression**
- Run `FOXHUNT_TEST_DATA=test_data/futures-baseline cargo test -p ml --release --lib -- magnitude_distribution --ignored --nocapture` and capture the full log under `/tmp/foxhunt_smoke/magnitude_distribution_task2x.log`.
- Verify `grad_abs_confirmation` smoke still passes (no regression on grad-ratio data).
- Verify `multi_fold_convergence` smoke still passes (sharpe trajectory not destabilised by new gradient weighting).
- **Validation:** all smokes green; post-Task 2.X smoke `[EVAL_DIST] Quarter=.. Half=.. Full=..` shows `Full ≥ 0.05`.
6. **[ ] Step 6 — Commit under `fix(dqn): make magnitude-Full useful — per-bin variance weighting + Q-spread regularisation`**
- Commit message references this scoping doc, the H10 fix commit, and `feedback_magnitude_must_be_useful.md`.
- Include a new `docs/superpowers/specs/2026-04-22-policy-quality-task2x-make-full-useful.md` summarising pre- and post-fix `[EVAL_DIST]` numbers (bounded, 1 page max).
### Commit template
```
fix(dqn): make magnitude-Full useful — per-bin variance weighting + Q-spread regularisation
Task 2.X-make-Full-useful implementation per
/tmp/foxhunt_research/full-magnitude-fix-scoping.md §5 (scoping doc).
Post-Task 2.2 (H10) smoke showed eval dist [eq=0.580 eh=0.420 ef=0.000] —
Half gained share but Full never picked at eval. Diagnosis: C51's expected-
Q bias toward lower-variance bins (documented risk at
experience_kernels.cu:904-910). Training-mode Boltzmann softmax masked this
at training; strict-argmax eval after H10 exposed it.
Fix: two-part, both in-graph:
(1) Per-bin realised-variance weighting in C51 branch backward —
amplifies gradient for Full (high variance) so its Q-estimate
climbs above Half's in genuinely Full-favoured states.
(2) Q-spread regularisation (λ=0.01) — small penalty preventing the
degenerate Q_quarter ≈ Q_half ≈ Q_full fixpoint.
Per feedback_magnitude_must_be_useful.md, Full is not deletable; this
fix makes it useful instead.
Smoke results (20 epochs × 3 folds, 5000 bars):
pre-Task 2.X: eval [eq=0.580 eh=0.420 ef=0.000]
post-Task 2.X: eval [eq=0.xxx eh=0.xxx ef=0.xxx] (fill after run)
Instrumentation add: HEALTH_DIAG mag_stats + q_spread_penalty fields
(Step 1 per-magnitude win rate + variance — addresses the observability
gap identified in scoping doc §2.5).
Validation: magnitude_distribution, grad_abs_confirmation,
multi_fold_convergence all green.
```
---
## 6. L40S production-scale gating recommendation
At smoke scale (5000 bars × 20 epochs × 3 folds, laptop RTX 3050), Full's 4× PnL variance + C51 expected-Q bias is a dominant effect. At production scale (176k bars × 50 epochs × 6 folds, L40S sm_89), two things change:
1. **More data per bin**: Full sees ~200k training samples (5-6× smoke). C51's expected-Q estimate for the Full atoms becomes more accurate, which MAY allow the network to learn states where Full's expected value genuinely exceeds Half's. If there are any such states in ES.FUT history, production will find them.
2. **More epochs**: 50 epochs allows Full's Q-spread to escape the early-training degenerate fixpoint naturally (if a gradient path exists).
**Recommendation:** Gate Task 2.X on L40S data from Task 2.8 as follows:
| L40S post-Task 2.2 eval result | Action |
|--------------------------------|--------|
| `ef ≥ 0.15` | Task 2.X is **not needed**. Smoke `ef = 0.000` was a data-scale artefact. Close as "production separates naturally." Document in `2026-04-22-policy-quality-phase2-status.md`. |
| `0.05 ≤ ef < 0.15` | Task 2.X is **scoped but optional** — gate on whether production Sharpe already meets the mandatory §2.1 gates. If Sharpe passes, defer Task 2.X to a Phase 3 optimisation round. |
| `ef < 0.05` | Task 2.X is **required** — implement per §5 before Task 2.9 phase-complete tag. |
The Task 2.8 workflow is already scoped in `docs/superpowers/plans/2026-04-21-policy-quality-phase2.md` §Task 2.8 — the `ef` threshold is a small addition to the existing mandatory-gate check at Task 2.9.
---
## 7. Hard rules compliance audit
| Rule | Source | How this scoping doc complies |
|------|--------|-------------------------------|
| No implementation, research only | user task spec | No code modified. Doc only. |
| No commits to main | user task spec | No git operations performed. Doc written to `/tmp/foxhunt_research/`. |
| Evidence-based (every claim cites log line + file:line) | user task spec | §2 data tables cite `grad_abs_confirmation.log:<N>` and `crates/ml/…:<N>` for every observation. §3 diagnosis cites the exact kernel comment lines. |
| Honest about inconclusive smoke data | user task spec | §2.5 explicitly flags per-magnitude win rate as NOT observable from current data. §6 recommends L40S gating before Task 2.X is implemented. |
| No quickfixes (feedback_no_quickfixes.md) | standing | Task 2.X is a kernel-level + loss-function fix (per-bin variance weighting + Q-spread regularisation), not a threshold relaxation or a toggle. |
| No functionality removal (feedback_no_functionality_removal.md, feedback_magnitude_must_be_useful.md) | standing | Every candidate fix in §4 preserves Full and the magnitude branch. Explicit §4 footnote: "Ranks 1-5 all preserve Full." |
| No feature flags (feedback_no_feature_flags.md) | standing | §5 Step 2 says "Do NOT guard behind a runtime boolean — wire unconditionally." |
| No stubs (feedback_no_stubs.md) | standing | §5 steps are complete; no `TODO: wire this` slots. |
| No GPU→CPU roundtrips on hot path, no atomicAdd on hot path | standing | §5 Step 1 uses per-sample array → host-reduce at epoch boundary (same pattern as Task 0.5/0.8). |
| Trust code not docs (feedback_trust_code_not_docs.md) | standing | §2.6 reads actual kernel code (lines 1471-1477, 1853-1855). Diagnosis in §3 cites kernel comment lines 904-910 but verifies the claim against behaviour. |
---
## 8. Additional instrumentation needed (for confident decision)
**Gap identified:** per-magnitude win rate and per-magnitude realised variance are not currently exposed. Both are computable from existing per-sample GPU buffers with no new compute on the hot path — they require only a per-sample write during the trade-completion branch in `experience_kernels.cu` and a host-side reduction at epoch boundary (same pattern as Task 0.5 / 0.8 / 2.0).
**Minimum instrumentation to confirm root-cause diagnosis** (~40 LOC — this is a prerequisite for Step 1 of Task 2.X but is small enough to land as a stand-alone diagnostic commit):
- Add to `experience_kernels.cu` near the segment-complete branch (~`experience_kernels.cu:1847`): three per-sample slots `per_mag_realized_ret_ps[i]`, `per_mag_trade_count_ps[i]`, `per_mag_win_ps[i]` indexed by `mag_idx`.
- Host-side reduce in `metrics.rs` after epoch end: aggregate into `[win_rate_q, win_rate_h, win_rate_f, mean_ret_q/h/f, var_ret_q/h/f]`.
- Emit in HEALTH_DIAG: `mag_stats [win_q=%.3f win_h=%.3f win_f=%.3f ret_q=%.4f ret_h=%.4f ret_f=%.4f var_q=%.4f var_h=%.4f var_f=%.4f]`.
- No new kernel launch, no new CUDA graph capture, no atomicAdd.
**Decision impact:** once this instrumentation lands and a smoke is run:
- If `win_rate_f ≥ win_rate_h` and `mean_ret_f ≥ mean_ret_h` but `ef = 0.000` → Q-estimation bias is confirmed as the sole cause → Task 2.X Rank 1 (variance weighting) is the right fix.
- If `win_rate_f < win_rate_h` OR `mean_ret_f < mean_ret_h` → data genuinely disfavours Full at smoke scale → Task 2.X Rank 3 (per-magnitude reward shaping) becomes the primary candidate, with L40S-scale re-check before committing.
- If the two conditions are mixed (Full wins more often but realises smaller mean) → Rank 1 + Rank 2 combo as proposed.
**Proposed order:** (i) land the instrumentation as a standalone commit `diag(policy-quality): per-magnitude win-rate + variance for Task 2.X diagnosis`, (ii) run smoke, (iii) decide Task 2.X rank based on the new data, (iv) either wait for L40S or implement per §5.
---
## 9. Summary
- **Observation:** post-Task 2.2, eval `Quarter=0.580 Half=0.420 Full=0.000`. Full is never picked at eval.
- **Not the cause:** noise-starvation (Full has 36k training samples) or data unavailability (training-mode picks Full ~19 % directional).
- **Primary cause:** C51 expected-Q bias toward lower-variance bins — Full has 4× PnL variance of Quarter for the same directional outcome, which C51's expected-Q computation systematically under-prices. Documented as a known risk at `experience_kernels.cu:904-910`.
- **Secondary cause (not yet confirmed):** data may genuinely disfavour Full at 5000-bar slice; confirmable only with per-magnitude win rate instrumentation (§8) which is a ~40 LOC add.
- **Recommended fix:** Task 2.X-make-Full-useful = per-bin variance weighting in C51 branch loss (amplifies Full's gradient) + Q-spread regularisation (prevents degenerate fixpoint). Preserves Full unconditionally per `feedback_magnitude_must_be_useful.md`.
- **Gating:** do not implement until (i) per-magnitude win-rate instrumentation lands and (ii) L40S Task 2.8 run confirms `ef < 0.15` at production scale. If L40S `ef ≥ 0.15`, Task 2.X is not needed.
---
## Appendix A — Key log references
| Claim | Source |
|-------|--------|
| `dist_q` / `dist_h` / `dist_f` per-epoch trajectory | `/tmp/foxhunt_smoke/grad_abs_confirmation.log:329, 372, 410, 445, 480, 514, 549, 584, 619, 654, 690, 726, 761, 799, 835, 871, 906, 944, 979, 1014, ... 2570` (60 HEALTH_DIAG emissions) |
| `q_full`, `q_half`, `q_quarter` per-epoch Q-values | same HEALTH_DIAG lines, `mag [q_full=... q_half=... q_quarter=...]` subfield |
| Final training-mode magnitude distribution | `grad_abs_confirmation.log:2603` — `[MAG_DIST] Quarter=0.625 Half=0.143 Full=0.232` (fold-3 final) |
| Post-H10 eval-mode distribution | H10 commit message body (`git show 8aef59f73`): `eval dist [eq=0.580 eh=0.420 ef=0.000]` |
| LOW EXPOSURE warnings (only H_Full / F_Full, never S_Full / L_Full) | `grad_abs_confirmation.log:337-340, 379-382, …` — 240 total warnings for H_Full / F_Full (unreachable by design per `monitoring.rs:18-23`), 0 for S_Full / L_Full |
| `actions=[...]` 12-bin layout | `crates/ml/src/cuda_pipeline/gpu_monitoring.rs:36`, `crates/ml/src/trainers/dqn/monitoring.rs:18-23` |
| Magnitude-neutral normalisation removed | `crates/ml/src/cuda_pipeline/experience_kernels.cu:1853-1855` |
| C51 expected-Q bias documented risk | `crates/ml/src/cuda_pipeline/experience_kernels.cu:904-910` |
| Risk-scaling before magnitude mapping (preserves 4× ratio) | `crates/ml/src/cuda_pipeline/experience_kernels.cu:1471-1477` |
| Per-magnitude win rate NOT exposed | Grep of `crates/ml` for `win_rate.*magnitude|per_magnitude.*reward` returns 0 matches |
| Smoke config | `config/training/dqn-smoketest.toml:13-24` (epochs=10 [overridden to 20 by test], batch_size=64, max_bars=5000) |
| H10 fix bypass of training path | Commit `8aef59f73` body: "Training-mode distribution is bit-identical as expected: the fix only touches the eval_mode == 1 branch of experience_action_select." |
| Phase 2 rule — make magnitude useful, do not delete | `docs/superpowers/plans/2026-04-21-policy-quality-phase2.md:81-87` + `~/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/feedback_magnitude_must_be_useful.md` |
---
**End of scoping doc. No implementation performed. Next action: user decides whether to (a) land §8 instrumentation first, (b) wait for L40S Task 2.8, or (c) proceed directly to Task 2.X implementation per §5.**