docs(sp22): H6 smoke verdict — falsified at cycle 1 (50.21% WR)

Workflow train-cr9hl on sp20-aux-h-fixed @ 7fc979934, terminated at
epoch=1 end after 36m58s wall-clock per
`feedback_kill_runs_on_anomaly_quickly`.

Pre-smoke gates (all clean): CAPTURE_PHASE_AUX_DONE + POST_AUX_DONE,
13 child graphs captured, no CUDA errors, no panics, no OOM. The new
`aux_softmax_to_per_env_kernel` launched inside the captured forward
graph without breaking recording (pure per-thread gather, no host
branches — passes `pearl_no_host_branches_in_captured_graph`).

Epoch 1 trade stats: 489959 trades, 245998 wins, 243961 losses,
PF=0.947 — WR = 50.21%, squarely in the runbook's pre-declared
falsification band (50.1–50.2%).

Aux head was producing non-trivial directional content (HEALTH_DIAG[0]
pred_tanh = 0.6626, batch mean of softmax[1]-softmax[0]) — the bridge
was conducting signal; the policy just couldn't gradient-couple to it.

LOW EXPOSURE DIVERSITY warnings at epoch 1 (S_Small=3.8%, H_Half=0.4%,
H_Full=0.8%, L_Small=4.2%, F_Half=1.7%, F_Full=3.5%) corroborate the
V/A unidentifiability hypothesis (`project_dueling_va_unidentifiable`).
Combined with `hold_pct_ema=0.2004` against `target_hold_pct=0.1151`
and `hold_reward_ema=-0.2044`, the cost-dominance pattern from
`pearl_event_driven_reward_density_alignment` is the more upstream
candidate.

Phase 1 H6 wiring STAYS merged per `feedback_no_functionality_removal`:
slot 121 is allocated, the buffer is initialized via pure-GPU fill,
the copy kernel runs inside captured graphs. None is harmful; if a
future fix produces policy gradient-coupling to directional features,
the bridge is already in place.

A2 (eval-side aux integration) is deferred indefinitely — A3 NULL
fallback is sufficient for eval and there is no production case for
A2 until training-side evidence shows the bridge is doing useful work.

Next direction (framing only, not implementation):
- H3 (reward density mismatch per
  `pearl_event_driven_reward_density_alignment`) is the upstream
  candidate; V/A unidentifiability fix is downstream.
- Sequencing: heal gradient signal first, then test if V/A still
  pathologizes with healthy rewards.

Audit doc append: `## 2026-05-12 — SP22 H6 implementation` section gets
a new "Smoke result (2026-05-12) — H6 FALSIFIED" subsection capturing
the verdict, ruling-out table, and next-investigation framing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-12 22:10:31 +02:00
parent 7fc9799343
commit ce1a81552b

View File

@@ -16396,3 +16396,130 @@ After smoke (3-epoch baseline on `ci-training-l40s`):
H3 (reward density mismatch per
`pearl_event_driven_reward_density_alignment`) or audit the V/A
unidentifiability fix (`project_dueling_va_unidentifiable`).
### Smoke result (2026-05-12) — H6 FALSIFIED
Workflow `train-cr9hl` dispatched on `sp20-aux-h-fixed` @ `7fc979934`:
```
argo submit dqn --baseline --branch sp20-aux-h-fixed \
--epochs 3 --gpu-pool ci-training-l40s
```
Terminated at `epoch=1` end after 36m58s wall-clock (saved ~1h35m of L40S
time per `feedback_kill_runs_on_anomaly_quickly`).
**Verification chain (pre-smoke gates, all clean)**
- `CAPTURE_PHASE_AUX_DONE` + `CAPTURE_PHASE_POST_AUX_DONE` — the new
`aux_softmax_to_per_env_kernel` launched inside the captured forward
graph without breaking graph recording. This was the highest-risk
failure mode (per `pearl_no_host_branches_in_captured_graph` host
branches would have broken capture); the kernel is pure per-thread
gather with no host branches and capture worked end-to-end.
- 13 child graphs captured + parent composed; step 0 = 540ms (capture),
steps 1-50 replaying at 0-486ms (graph replay nominal).
- No CUDA errors, no panics, no OOM through epoch 1.
- `fill_f32` cold-start + (one-shot, since we only ran 1 epoch) seemed
to leave the buffer correctly initialized — no init-related failures.
**Epoch 1 outcome (cycle 1 verdict)**
```
GPU trades=489959 wins=245998 losses=243961 PF=0.947
```
Win Rate = 245998 / 489959 = **50.21%**
This sits **squarely in the runbook's pre-declared falsification band**
(50.150.2%) — same plateau observed across all v5v11 baselines pre-H6.
**Aux head telemetry at HEALTH_DIAG[0]**
```
aux_dir_acc target=0.5500 short=0.2798 long=0.2801 pred_tanh=0.6626
aux [next_bar_mse=5.686e-1 regime_ce=3.044e0 w=1.175]
aux_b1_diag dir_acc_short=0.2798 dir_acc_long=0.2801 n_down=0 n_up=0 n_skip=0 mask_frac=0.0000
sp20_isv [wr_ema=0.3911 hold_pct_ema=0.2004 hold_reward_ema=-0.2044]
```
The `pred_tanh=0.6626` (batch mean of `softmax[1] - softmax[0]`) shows
the aux head was producing strongly directional predictions (biased
"up") at cycle 0. So the **signal content was there** — the bridge
was conducting non-trivial directional data. The policy just couldn't
turn that into a directional Q-gap.
`dir_acc_short/long = 0.28` with `n_up = n_down = n_skip = 0` at
HEALTH_DIAG[0] is the cold-start sentinel-decay path (labels hadn't
flowed yet at the moment HEALTH_DIAG fired); the eventual epoch-1
trade-stats `WR = 50.21%` is the post-warmup ground truth.
**Other corroborating signals**
`LOW EXPOSURE DIVERSITY at epoch 1` warnings:
```
S_Small 3.8% (309619/8192000)
H_Half 0.4% (33713/8192000)
H_Full 0.8% (63340/8192000)
L_Small 4.2% (341432/8192000)
F_Half 1.7% (141220/8192000)
F_Full 3.5% (283700/8192000)
```
The policy is concentrating on a few branches. This is the same pattern
documented in `project_dueling_va_unidentifiable` — Q(Flat) attractor,
V parking at Q(Flat), A reducing to deltas. When the action distribution
is degenerate, state slot 121 has no leverage to differentiate Q-values
*regardless* of its content. Implies the bottleneck is upstream of the
state-bridge wire.
**What the smoke ruled in vs. out**
| Hypothesis | Status |
|---|---|
| H6 wiring crashes capture / breaks training | Ruled out (clean capture, clean run) |
| Aux head produces zero-information predictions | Ruled out (`pred_tanh = 0.66`, strongly directional) |
| Aux→state bridge moves policy WR off 50% in ≤3 epochs | Ruled out (50.21% at epoch 1) |
| Policy can't gradient-couple to a new directional state dim under current rewards | **Supported** (consistent with reward-density + V/A pathology hypotheses) |
**Phase 1 close-out**
- Phase 1 H6 wiring **stays merged** per `feedback_no_functionality_removal`.
The state slot is allocated, the buffer is initialized to a neutral
sentinel via pure-GPU fill, and the copy kernel runs inside captured
graphs. None of this is harmful; if H3 or a V/A fix lands and the
policy starts gradient-coupling to directional features, slot 121 is
already wired.
- A2 (eval-side aux integration) is **deferred indefinitely** — A3 NULL
fallback works correctly for the eval and there is no production
justification to add an eval-side aux trunk forward until training-side
evidence shows the bridge is doing useful work.
**Next-investigation framing**
The smoke supports the "policy can't gradient-couple under current
rewards" branch. Two upstream candidates:
1. **H3 — reward density alignment** (per
`pearl_event_driven_reward_density_alignment`): event-driven
objective (trade close) with per-step shaping creates exposure-positive
bias. Per-step micro/opp_cost rewards dominate sparse directional
alpha. The cycle-1 telemetry has `hold_reward_ema = -0.2044`,
`hold_pct_ema = 0.2004` against `target_hold_pct = 0.1151` — the
policy is holding ~74% above target and the hold-reward is strongly
negative; cost dominance pattern.
2. **V/A unidentifiability** (per `project_dueling_va_unidentifiable`):
structural; the LOW EXPOSURE DIVERSITY pattern at cycle 1 is a direct
tell. The dueling architecture lets the optimizer park V at Q(Flat),
degenerating the action distribution regardless of state content.
**Sequencing**: H3 first, V/A second. Reasoning: if rewards aren't
producing directional gradient pressure, then "fixing" V/A is optimizing
the same degenerate objective with a different head shape. Heal the
gradient signal upstream, then re-test whether V/A still pathologizes
with healthy rewards.
If H3 lands and the WR moves but action-space stays narrow, V/A is
binding and gets fixed next. If H3 doesn't move WR either, V/A goes
first and we re-test in combination.