From ce1a81552b3dbca8734f5bfd8f6db3cc801ece73 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Tue, 12 May 2026 22:10:31 +0200 Subject: [PATCH] =?UTF-8?q?docs(sp22):=20H6=20smoke=20verdict=20=E2=80=94?= =?UTF-8?q?=20falsified=20at=20cycle=201=20(50.21%=20WR)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Workflow train-cr9hl on sp20-aux-h-fixed @ 7fc979934, terminated at epoch=1 end after 36m58s wall-clock per `feedback_kill_runs_on_anomaly_quickly`. Pre-smoke gates (all clean): CAPTURE_PHASE_AUX_DONE + POST_AUX_DONE, 13 child graphs captured, no CUDA errors, no panics, no OOM. The new `aux_softmax_to_per_env_kernel` launched inside the captured forward graph without breaking recording (pure per-thread gather, no host branches — passes `pearl_no_host_branches_in_captured_graph`). Epoch 1 trade stats: 489959 trades, 245998 wins, 243961 losses, PF=0.947 — WR = 50.21%, squarely in the runbook's pre-declared falsification band (50.1–50.2%). Aux head was producing non-trivial directional content (HEALTH_DIAG[0] pred_tanh = 0.6626, batch mean of softmax[1]-softmax[0]) — the bridge was conducting signal; the policy just couldn't gradient-couple to it. LOW EXPOSURE DIVERSITY warnings at epoch 1 (S_Small=3.8%, H_Half=0.4%, H_Full=0.8%, L_Small=4.2%, F_Half=1.7%, F_Full=3.5%) corroborate the V/A unidentifiability hypothesis (`project_dueling_va_unidentifiable`). Combined with `hold_pct_ema=0.2004` against `target_hold_pct=0.1151` and `hold_reward_ema=-0.2044`, the cost-dominance pattern from `pearl_event_driven_reward_density_alignment` is the more upstream candidate. Phase 1 H6 wiring STAYS merged per `feedback_no_functionality_removal`: slot 121 is allocated, the buffer is initialized via pure-GPU fill, the copy kernel runs inside captured graphs. None is harmful; if a future fix produces policy gradient-coupling to directional features, the bridge is already in place. A2 (eval-side aux integration) is deferred indefinitely — A3 NULL fallback is sufficient for eval and there is no production case for A2 until training-side evidence shows the bridge is doing useful work. Next direction (framing only, not implementation): - H3 (reward density mismatch per `pearl_event_driven_reward_density_alignment`) is the upstream candidate; V/A unidentifiability fix is downstream. - Sequencing: heal gradient signal first, then test if V/A still pathologizes with healthy rewards. Audit doc append: `## 2026-05-12 — SP22 H6 implementation` section gets a new "Smoke result (2026-05-12) — H6 FALSIFIED" subsection capturing the verdict, ruling-out table, and next-investigation framing. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/dqn-wire-up-audit.md | 127 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 127 insertions(+) diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index e3d9f83cb..3895a2817 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -16396,3 +16396,130 @@ After smoke (3-epoch baseline on `ci-training-l40s`): H3 (reward density mismatch per `pearl_event_driven_reward_density_alignment`) or audit the V/A unidentifiability fix (`project_dueling_va_unidentifiable`). + +### Smoke result (2026-05-12) — H6 FALSIFIED + +Workflow `train-cr9hl` dispatched on `sp20-aux-h-fixed` @ `7fc979934`: + +``` +argo submit dqn --baseline --branch sp20-aux-h-fixed \ + --epochs 3 --gpu-pool ci-training-l40s +``` + +Terminated at `epoch=1` end after 36m58s wall-clock (saved ~1h35m of L40S +time per `feedback_kill_runs_on_anomaly_quickly`). + +**Verification chain (pre-smoke gates, all clean)** + +- `CAPTURE_PHASE_AUX_DONE` + `CAPTURE_PHASE_POST_AUX_DONE` — the new + `aux_softmax_to_per_env_kernel` launched inside the captured forward + graph without breaking graph recording. This was the highest-risk + failure mode (per `pearl_no_host_branches_in_captured_graph` host + branches would have broken capture); the kernel is pure per-thread + gather with no host branches and capture worked end-to-end. +- 13 child graphs captured + parent composed; step 0 = 540ms (capture), + steps 1-50 replaying at 0-486ms (graph replay nominal). +- No CUDA errors, no panics, no OOM through epoch 1. +- `fill_f32` cold-start + (one-shot, since we only ran 1 epoch) seemed + to leave the buffer correctly initialized — no init-related failures. + +**Epoch 1 outcome (cycle 1 verdict)** + +``` +GPU trades=489959 wins=245998 losses=243961 PF=0.947 +``` + +Win Rate = 245998 / 489959 = **50.21%** + +This sits **squarely in the runbook's pre-declared falsification band** +(50.1–50.2%) — same plateau observed across all v5–v11 baselines pre-H6. + +**Aux head telemetry at HEALTH_DIAG[0]** + +``` +aux_dir_acc target=0.5500 short=0.2798 long=0.2801 pred_tanh=0.6626 +aux [next_bar_mse=5.686e-1 regime_ce=3.044e0 w=1.175] +aux_b1_diag dir_acc_short=0.2798 dir_acc_long=0.2801 n_down=0 n_up=0 n_skip=0 mask_frac=0.0000 +sp20_isv [wr_ema=0.3911 hold_pct_ema=0.2004 hold_reward_ema=-0.2044] +``` + +The `pred_tanh=0.6626` (batch mean of `softmax[1] - softmax[0]`) shows +the aux head was producing strongly directional predictions (biased +"up") at cycle 0. So the **signal content was there** — the bridge +was conducting non-trivial directional data. The policy just couldn't +turn that into a directional Q-gap. + +`dir_acc_short/long = 0.28` with `n_up = n_down = n_skip = 0` at +HEALTH_DIAG[0] is the cold-start sentinel-decay path (labels hadn't +flowed yet at the moment HEALTH_DIAG fired); the eventual epoch-1 +trade-stats `WR = 50.21%` is the post-warmup ground truth. + +**Other corroborating signals** + +`LOW EXPOSURE DIVERSITY at epoch 1` warnings: +``` +S_Small 3.8% (309619/8192000) +H_Half 0.4% (33713/8192000) +H_Full 0.8% (63340/8192000) +L_Small 4.2% (341432/8192000) +F_Half 1.7% (141220/8192000) +F_Full 3.5% (283700/8192000) +``` + +The policy is concentrating on a few branches. This is the same pattern +documented in `project_dueling_va_unidentifiable` — Q(Flat) attractor, +V parking at Q(Flat), A reducing to deltas. When the action distribution +is degenerate, state slot 121 has no leverage to differentiate Q-values +*regardless* of its content. Implies the bottleneck is upstream of the +state-bridge wire. + +**What the smoke ruled in vs. out** + +| Hypothesis | Status | +|---|---| +| H6 wiring crashes capture / breaks training | Ruled out (clean capture, clean run) | +| Aux head produces zero-information predictions | Ruled out (`pred_tanh = 0.66`, strongly directional) | +| Aux→state bridge moves policy WR off 50% in ≤3 epochs | Ruled out (50.21% at epoch 1) | +| Policy can't gradient-couple to a new directional state dim under current rewards | **Supported** (consistent with reward-density + V/A pathology hypotheses) | + +**Phase 1 close-out** + +- Phase 1 H6 wiring **stays merged** per `feedback_no_functionality_removal`. + The state slot is allocated, the buffer is initialized to a neutral + sentinel via pure-GPU fill, and the copy kernel runs inside captured + graphs. None of this is harmful; if H3 or a V/A fix lands and the + policy starts gradient-coupling to directional features, slot 121 is + already wired. +- A2 (eval-side aux integration) is **deferred indefinitely** — A3 NULL + fallback works correctly for the eval and there is no production + justification to add an eval-side aux trunk forward until training-side + evidence shows the bridge is doing useful work. + +**Next-investigation framing** + +The smoke supports the "policy can't gradient-couple under current +rewards" branch. Two upstream candidates: + +1. **H3 — reward density alignment** (per + `pearl_event_driven_reward_density_alignment`): event-driven + objective (trade close) with per-step shaping creates exposure-positive + bias. Per-step micro/opp_cost rewards dominate sparse directional + alpha. The cycle-1 telemetry has `hold_reward_ema = -0.2044`, + `hold_pct_ema = 0.2004` against `target_hold_pct = 0.1151` — the + policy is holding ~74% above target and the hold-reward is strongly + negative; cost dominance pattern. + +2. **V/A unidentifiability** (per `project_dueling_va_unidentifiable`): + structural; the LOW EXPOSURE DIVERSITY pattern at cycle 1 is a direct + tell. The dueling architecture lets the optimizer park V at Q(Flat), + degenerating the action distribution regardless of state content. + +**Sequencing**: H3 first, V/A second. Reasoning: if rewards aren't +producing directional gradient pressure, then "fixing" V/A is optimizing +the same degenerate objective with a different head shape. Heal the +gradient signal upstream, then re-test whether V/A still pathologizes +with healthy rewards. + +If H3 lands and the WR moves but action-space stays narrow, V/A is +binding and gets fixed next. If H3 doesn't move WR either, V/A goes +first and we re-test in combination.