plan(policy-quality): Phase 2 pivot — H4 REJECTED by Task 2.0 data

Task 2.0 instrumentation (commits d60e5375a / 980f3b07f / 41b0c559c)
revealed two silent bugs in Task 0.4's grad_ratio_mag_dir accessor:

  Bug A — readback size mismatch: pinned buffer allocated at
  total_params, but grad_buf length is total_params+cutlass_tile_pad,
  so size check always failed → Err silently coerced to 0.0 by proxy.

  Bug B — readback timing wipe: estimate_avg_q_value_with_early_stopping
  in process_epoch_boundary replays the forward graph, which zeros
  grad_buf. Any subsequent readback sees all zeros.

Both fixed in 41b0c559c; snapshot hoisted to top of process_epoch_boundary.

With those bugs fixed, the measured gradient ratio is NOT 0.0000 — it is
50–400× mag/dir across 60 epochs. Magnitude branch is over-fed, not
starved. Direction gradient is small but non-zero (~2e-2 to 7e0).
Direction policy is observably healthy (Short/Hold/Long/Flat 38/12/42/14%).

Track 1 triage's H4 CONFIRMED verdict was a measurement artefact
produced by Bugs A+B. H4 as originally defined is now REJECTED.

Plan changes:
  - Add new "Task 2.0 findings" section after cross-cutting concerns,
    documenting bug A, bug B, observed data, and revised strategy.
  - Task 2.1 marked DEFERRED (not deleted — kept for reference).
  - Task 2.2 promoted to PRIMARY fix.
  - New fallback: H9 delete-magnitude-branch as replacement for Task 2.1
    if Task 2.2 alone is insufficient.
  - Task 2.0 inventory row updated with LANDED status and commit chain.

Net effect: Phase 2 simplifies. The hardest task (2.1 three-branch
architectural fix) is skipped; primary path is Task 2.2 (~25 LOC).
This commit is contained in:
jgrusewski
2026-04-22 11:13:48 +02:00
parent 41b0c559c9
commit 7e78bf4f85

View File

@@ -50,13 +50,48 @@ Three concerns span multiple tracks and shape task ordering:
---
## Task 2.0 findings — PIVOT from H4-architectural-fix to H10-only-fix
> **Added 2026-04-22 after Task 2.0 landed (commits `d60e5375a` → `980f3b07f` → `41b0c559c`).**
>
> The keystone concern #2 above is **superseded** by Task 2.0's actual measurement data. What Task 0.4's `grad_ratio_mag_dir = 0.0000` reading meant in the Track 1 triage is not what it actually meant.
### Two silent bugs discovered during Task 2.0's confirmation diagnostic (commit `41b0c559c`)
Both bugs were silently zeroing Task 0.4's `grad_ratio_mag_dir` for the entire baseline capture. Both root-caused and fixed in the same commit:
- **Bug A — readback size mismatch** (`gpu_dqn_trainer.rs:2195` `per_branch_grad_norms`). `grad_buf.len() = total_params + cutlass_tile_pad` (~4096 padding slots) but `grad_readback_pinned_capacity` was allocated at exactly `total_params`. Size check at 2202 was always true → accessor always returned `Err(MLError::ModelError(...))` → the fused proxy's `.unwrap_or(0.0)` silently coerced Err to 0.0. Fix: read only the first `total_params` prefix.
- **Bug B — readback timing wipe** (`training_loop.rs:1764` `process_epoch_boundary`). `estimate_avg_q_value_with_early_stopping` near the top replays `eval_forward_exec`, whose first op is `cuMemsetD32Async(grad_buf, 0, total_params)` (`submit_forward_ops_main:11207`). Any grad_buf readback after that call sees all zeros. Fix: snapshot `(grad_dir_abs, grad_mag_abs, grad_ratio_mag_dir)` at the top of `process_epoch_boundary` before avg_q; HEALTH_DIAG reads the cached snapshot.
### What the data actually shows (once Task 2.0 + both bug fixes landed)
- Magnitude branch receives **100400× more gradient than direction** from CQL and C51 (ratios stable across 60 epochs × 3 folds). IQN / Ens / recursive_confidence / predictive_coding all contribute 0 to branch tensors **by design** — they flow through trunk (tensors 0..4) or value head (4..8), never touching branches 8..24. Confirmed via code read, not guessed.
- Epoch-boundary absolute L2 norms: `dir ∈ [2e-2, 7e0]`, `mag ∈ [3.6, 400]`. Direction gradient is **small but non-zero** — a factor 50400× below magnitude, not floating-point zero.
- Direction policy stays observably diverse (F_Short = 38% / F_Long = 42% / F_Hold = 12% / F_Flat = 14%, `ent_dir = 0.90`). That's Xavier-init plus the small gradient the direction branch does receive, plus Philox noise in action selection — enough for a diverse policy.
- Magnitude is NOT starved. It's **over-fed** at the gradient level; the three magnitude Q-values converge to near-identical values (`q_full = 0.050, q_half = -0.046, q_quarter = -0.044` at epoch 19) because the data truly does not differentiate the three bins at smoke scale.
### Track 1 triage's H4 CONFIRMED reading was a measurement artefact
The Track 1 triage scored H4 CONFIRMED based on `grad_ratio_mag_dir = 0.0000 across 20/20 epochs`. That reading was produced by Bugs A+B, not by gradient starvation. The true ratio is 50400×, not zero. **H4 — as originally defined — is REJECTED** (in favour of **H9** — data favours Quarter).
### Revised Phase 2 strategy
- **Task 2.1 is DEFERRED / likely unnecessary.** Magnitude doesn't need architectural unblocking. Branch A (Xavier rescale for magnitude head) would make over-feeding worse; Branch B (per-component weight rebalance) targets a non-existent starvation; Branch C (`w_b1fc` direction-conditioning rewrite) is a structural change the data doesn't justify.
- **Task 2.2 (H10 argmax tie-break) is PROMOTED TO PRIMARY FIX.** The observable symptom (eval `eq=1.000 eh=0.000 ef=0.000` despite training `ent_mag ≈ 0.99`) IS argmax tie-breaking. Fix the tie-break and the smoke test likely passes.
- **New fallback if Task 2.2 alone is insufficient**: promote H9's spec §5.1 proposed fix to a new Task — **delete the magnitude branch entirely** (108 → 36 action space: direction × order × urgency only). The data says magnitude doesn't differentiate trades; removing it is a strict simplification with no expected loss.
- **Task 2.3 / 2.4 / 2.5 / 2.6 / 2.7 are unchanged.** The Task 2.0 data also confirms R5 TD-propagation is fine (C51+CQL reach magnitude robustly), so R5 DELETE (Task 2.3) is unblocked.
Net effect: Phase 2 gets a big simplification. The hardest task (2.1 with three architectural branches) is dropped. Primary fix becomes Task 2.2 (~25 LOC), with H9-delete-magnitude-branch as fallback.
---
## Task inventory (11 tasks)
| # | Task | Category | Estimated LOC | Depends on |
|---|---|---|---|---|
| 2.0 | Per-component gradient decomposition (IQN/CQL/C51/Ens → magnitude) | instrumentation | ~60 | — |
| 2.1 | H4 fix — rebalance gradient flow to magnitude head | behavioral | ~40 decision-tree-driven | 2.0 |
| 2.2 | H10 fix — stable argmax tie-breaking at eval | behavioral | ~25 | — |
| 2.0 | Per-component gradient decomposition (IQN/CQL/C51/Ens → magnitude) | instrumentation | ~90 LANDED (`d60e5375a``980f3b07f``41b0c559c`; includes 5-writer extension + 2 silent-bug fixes) | — |
| 2.1 | ~~H4 fix — rebalance gradient flow to magnitude head~~ **DEFERRED** — Task 2.0 data REJECTS H4 (ratio 100-400× mag/dir, not starvation). Task 2.2 becomes primary; if insufficient, promote the new H9-delete-magnitude-branch task as replacement. | — | — | — |
| 2.2 | H10 fix — stable argmax tie-breaking at eval (**PROMOTED TO PRIMARY FIX**) | behavioral | ~25 | — |
| 2.3 | DELETE R5 micro-reward path | code removal | ~-150 | TD-propagation diagnostic (this plan extends Task 2.0) |
| 2.4 | DELETE R6 loss-aversion, relocate neg-tail compression to Q-target smoothing | behavioral + code removal | ~-30 +20 | — |
| 2.5 | Wiring-bug sweep (7 bugs from Tracks 1/3/4) | correctness | ~50 | — |