diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index e3d9f83cb..3895a2817 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -16396,3 +16396,130 @@ After smoke (3-epoch baseline on `ci-training-l40s`): H3 (reward density mismatch per `pearl_event_driven_reward_density_alignment`) or audit the V/A unidentifiability fix (`project_dueling_va_unidentifiable`). + +### Smoke result (2026-05-12) — H6 FALSIFIED + +Workflow `train-cr9hl` dispatched on `sp20-aux-h-fixed` @ `7fc979934`: + +``` +argo submit dqn --baseline --branch sp20-aux-h-fixed \ + --epochs 3 --gpu-pool ci-training-l40s +``` + +Terminated at `epoch=1` end after 36m58s wall-clock (saved ~1h35m of L40S +time per `feedback_kill_runs_on_anomaly_quickly`). + +**Verification chain (pre-smoke gates, all clean)** + +- `CAPTURE_PHASE_AUX_DONE` + `CAPTURE_PHASE_POST_AUX_DONE` — the new + `aux_softmax_to_per_env_kernel` launched inside the captured forward + graph without breaking graph recording. This was the highest-risk + failure mode (per `pearl_no_host_branches_in_captured_graph` host + branches would have broken capture); the kernel is pure per-thread + gather with no host branches and capture worked end-to-end. +- 13 child graphs captured + parent composed; step 0 = 540ms (capture), + steps 1-50 replaying at 0-486ms (graph replay nominal). +- No CUDA errors, no panics, no OOM through epoch 1. +- `fill_f32` cold-start + (one-shot, since we only ran 1 epoch) seemed + to leave the buffer correctly initialized — no init-related failures. + +**Epoch 1 outcome (cycle 1 verdict)** + +``` +GPU trades=489959 wins=245998 losses=243961 PF=0.947 +``` + +Win Rate = 245998 / 489959 = **50.21%** + +This sits **squarely in the runbook's pre-declared falsification band** +(50.1–50.2%) — same plateau observed across all v5–v11 baselines pre-H6. + +**Aux head telemetry at HEALTH_DIAG[0]** + +``` +aux_dir_acc target=0.5500 short=0.2798 long=0.2801 pred_tanh=0.6626 +aux [next_bar_mse=5.686e-1 regime_ce=3.044e0 w=1.175] +aux_b1_diag dir_acc_short=0.2798 dir_acc_long=0.2801 n_down=0 n_up=0 n_skip=0 mask_frac=0.0000 +sp20_isv [wr_ema=0.3911 hold_pct_ema=0.2004 hold_reward_ema=-0.2044] +``` + +The `pred_tanh=0.6626` (batch mean of `softmax[1] - softmax[0]`) shows +the aux head was producing strongly directional predictions (biased +"up") at cycle 0. So the **signal content was there** — the bridge +was conducting non-trivial directional data. The policy just couldn't +turn that into a directional Q-gap. + +`dir_acc_short/long = 0.28` with `n_up = n_down = n_skip = 0` at +HEALTH_DIAG[0] is the cold-start sentinel-decay path (labels hadn't +flowed yet at the moment HEALTH_DIAG fired); the eventual epoch-1 +trade-stats `WR = 50.21%` is the post-warmup ground truth. + +**Other corroborating signals** + +`LOW EXPOSURE DIVERSITY at epoch 1` warnings: +``` +S_Small 3.8% (309619/8192000) +H_Half 0.4% (33713/8192000) +H_Full 0.8% (63340/8192000) +L_Small 4.2% (341432/8192000) +F_Half 1.7% (141220/8192000) +F_Full 3.5% (283700/8192000) +``` + +The policy is concentrating on a few branches. This is the same pattern +documented in `project_dueling_va_unidentifiable` — Q(Flat) attractor, +V parking at Q(Flat), A reducing to deltas. When the action distribution +is degenerate, state slot 121 has no leverage to differentiate Q-values +*regardless* of its content. Implies the bottleneck is upstream of the +state-bridge wire. + +**What the smoke ruled in vs. out** + +| Hypothesis | Status | +|---|---| +| H6 wiring crashes capture / breaks training | Ruled out (clean capture, clean run) | +| Aux head produces zero-information predictions | Ruled out (`pred_tanh = 0.66`, strongly directional) | +| Aux→state bridge moves policy WR off 50% in ≤3 epochs | Ruled out (50.21% at epoch 1) | +| Policy can't gradient-couple to a new directional state dim under current rewards | **Supported** (consistent with reward-density + V/A pathology hypotheses) | + +**Phase 1 close-out** + +- Phase 1 H6 wiring **stays merged** per `feedback_no_functionality_removal`. + The state slot is allocated, the buffer is initialized to a neutral + sentinel via pure-GPU fill, and the copy kernel runs inside captured + graphs. None of this is harmful; if H3 or a V/A fix lands and the + policy starts gradient-coupling to directional features, slot 121 is + already wired. +- A2 (eval-side aux integration) is **deferred indefinitely** — A3 NULL + fallback works correctly for the eval and there is no production + justification to add an eval-side aux trunk forward until training-side + evidence shows the bridge is doing useful work. + +**Next-investigation framing** + +The smoke supports the "policy can't gradient-couple under current +rewards" branch. Two upstream candidates: + +1. **H3 — reward density alignment** (per + `pearl_event_driven_reward_density_alignment`): event-driven + objective (trade close) with per-step shaping creates exposure-positive + bias. Per-step micro/opp_cost rewards dominate sparse directional + alpha. The cycle-1 telemetry has `hold_reward_ema = -0.2044`, + `hold_pct_ema = 0.2004` against `target_hold_pct = 0.1151` — the + policy is holding ~74% above target and the hold-reward is strongly + negative; cost dominance pattern. + +2. **V/A unidentifiability** (per `project_dueling_va_unidentifiable`): + structural; the LOW EXPOSURE DIVERSITY pattern at cycle 1 is a direct + tell. The dueling architecture lets the optimizer park V at Q(Flat), + degenerating the action distribution regardless of state content. + +**Sequencing**: H3 first, V/A second. Reasoning: if rewards aren't +producing directional gradient pressure, then "fixing" V/A is optimizing +the same degenerate objective with a different head shape. Heal the +gradient signal upstream, then re-test whether V/A still pathologizes +with healthy rewards. + +If H3 lands and the WR moves but action-space stays narrow, V/A is +binding and gets fixed next. If H3 doesn't move WR either, V/A goes +first and we re-test in combination.