docs(sp22): H6 smoke verdict — falsified at cycle 1 (50.21% WR)
Workflow train-cr9hl on sp20-aux-h-fixed @ 7fc979934, terminated at
epoch=1 end after 36m58s wall-clock per
`feedback_kill_runs_on_anomaly_quickly`.
Pre-smoke gates (all clean): CAPTURE_PHASE_AUX_DONE + POST_AUX_DONE,
13 child graphs captured, no CUDA errors, no panics, no OOM. The new
`aux_softmax_to_per_env_kernel` launched inside the captured forward
graph without breaking recording (pure per-thread gather, no host
branches — passes `pearl_no_host_branches_in_captured_graph`).
Epoch 1 trade stats: 489959 trades, 245998 wins, 243961 losses,
PF=0.947 — WR = 50.21%, squarely in the runbook's pre-declared
falsification band (50.1–50.2%).
Aux head was producing non-trivial directional content (HEALTH_DIAG[0]
pred_tanh = 0.6626, batch mean of softmax[1]-softmax[0]) — the bridge
was conducting signal; the policy just couldn't gradient-couple to it.
LOW EXPOSURE DIVERSITY warnings at epoch 1 (S_Small=3.8%, H_Half=0.4%,
H_Full=0.8%, L_Small=4.2%, F_Half=1.7%, F_Full=3.5%) corroborate the
V/A unidentifiability hypothesis (`project_dueling_va_unidentifiable`).
Combined with `hold_pct_ema=0.2004` against `target_hold_pct=0.1151`
and `hold_reward_ema=-0.2044`, the cost-dominance pattern from
`pearl_event_driven_reward_density_alignment` is the more upstream
candidate.
Phase 1 H6 wiring STAYS merged per `feedback_no_functionality_removal`:
slot 121 is allocated, the buffer is initialized via pure-GPU fill,
the copy kernel runs inside captured graphs. None is harmful; if a
future fix produces policy gradient-coupling to directional features,
the bridge is already in place.
A2 (eval-side aux integration) is deferred indefinitely — A3 NULL
fallback is sufficient for eval and there is no production case for
A2 until training-side evidence shows the bridge is doing useful work.
Next direction (framing only, not implementation):
- H3 (reward density mismatch per
`pearl_event_driven_reward_density_alignment`) is the upstream
candidate; V/A unidentifiability fix is downstream.
- Sequencing: heal gradient signal first, then test if V/A still
pathologizes with healthy rewards.
Audit doc append: `## 2026-05-12 — SP22 H6 implementation` section gets
a new "Smoke result (2026-05-12) — H6 FALSIFIED" subsection capturing
the verdict, ruling-out table, and next-investigation framing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -16396,3 +16396,130 @@ After smoke (3-epoch baseline on `ci-training-l40s`):
|
||||
H3 (reward density mismatch per
|
||||
`pearl_event_driven_reward_density_alignment`) or audit the V/A
|
||||
unidentifiability fix (`project_dueling_va_unidentifiable`).
|
||||
|
||||
### Smoke result (2026-05-12) — H6 FALSIFIED
|
||||
|
||||
Workflow `train-cr9hl` dispatched on `sp20-aux-h-fixed` @ `7fc979934`:
|
||||
|
||||
```
|
||||
argo submit dqn --baseline --branch sp20-aux-h-fixed \
|
||||
--epochs 3 --gpu-pool ci-training-l40s
|
||||
```
|
||||
|
||||
Terminated at `epoch=1` end after 36m58s wall-clock (saved ~1h35m of L40S
|
||||
time per `feedback_kill_runs_on_anomaly_quickly`).
|
||||
|
||||
**Verification chain (pre-smoke gates, all clean)**
|
||||
|
||||
- `CAPTURE_PHASE_AUX_DONE` + `CAPTURE_PHASE_POST_AUX_DONE` — the new
|
||||
`aux_softmax_to_per_env_kernel` launched inside the captured forward
|
||||
graph without breaking graph recording. This was the highest-risk
|
||||
failure mode (per `pearl_no_host_branches_in_captured_graph` host
|
||||
branches would have broken capture); the kernel is pure per-thread
|
||||
gather with no host branches and capture worked end-to-end.
|
||||
- 13 child graphs captured + parent composed; step 0 = 540ms (capture),
|
||||
steps 1-50 replaying at 0-486ms (graph replay nominal).
|
||||
- No CUDA errors, no panics, no OOM through epoch 1.
|
||||
- `fill_f32` cold-start + (one-shot, since we only ran 1 epoch) seemed
|
||||
to leave the buffer correctly initialized — no init-related failures.
|
||||
|
||||
**Epoch 1 outcome (cycle 1 verdict)**
|
||||
|
||||
```
|
||||
GPU trades=489959 wins=245998 losses=243961 PF=0.947
|
||||
```
|
||||
|
||||
Win Rate = 245998 / 489959 = **50.21%**
|
||||
|
||||
This sits **squarely in the runbook's pre-declared falsification band**
|
||||
(50.1–50.2%) — same plateau observed across all v5–v11 baselines pre-H6.
|
||||
|
||||
**Aux head telemetry at HEALTH_DIAG[0]**
|
||||
|
||||
```
|
||||
aux_dir_acc target=0.5500 short=0.2798 long=0.2801 pred_tanh=0.6626
|
||||
aux [next_bar_mse=5.686e-1 regime_ce=3.044e0 w=1.175]
|
||||
aux_b1_diag dir_acc_short=0.2798 dir_acc_long=0.2801 n_down=0 n_up=0 n_skip=0 mask_frac=0.0000
|
||||
sp20_isv [wr_ema=0.3911 hold_pct_ema=0.2004 hold_reward_ema=-0.2044]
|
||||
```
|
||||
|
||||
The `pred_tanh=0.6626` (batch mean of `softmax[1] - softmax[0]`) shows
|
||||
the aux head was producing strongly directional predictions (biased
|
||||
"up") at cycle 0. So the **signal content was there** — the bridge
|
||||
was conducting non-trivial directional data. The policy just couldn't
|
||||
turn that into a directional Q-gap.
|
||||
|
||||
`dir_acc_short/long = 0.28` with `n_up = n_down = n_skip = 0` at
|
||||
HEALTH_DIAG[0] is the cold-start sentinel-decay path (labels hadn't
|
||||
flowed yet at the moment HEALTH_DIAG fired); the eventual epoch-1
|
||||
trade-stats `WR = 50.21%` is the post-warmup ground truth.
|
||||
|
||||
**Other corroborating signals**
|
||||
|
||||
`LOW EXPOSURE DIVERSITY at epoch 1` warnings:
|
||||
```
|
||||
S_Small 3.8% (309619/8192000)
|
||||
H_Half 0.4% (33713/8192000)
|
||||
H_Full 0.8% (63340/8192000)
|
||||
L_Small 4.2% (341432/8192000)
|
||||
F_Half 1.7% (141220/8192000)
|
||||
F_Full 3.5% (283700/8192000)
|
||||
```
|
||||
|
||||
The policy is concentrating on a few branches. This is the same pattern
|
||||
documented in `project_dueling_va_unidentifiable` — Q(Flat) attractor,
|
||||
V parking at Q(Flat), A reducing to deltas. When the action distribution
|
||||
is degenerate, state slot 121 has no leverage to differentiate Q-values
|
||||
*regardless* of its content. Implies the bottleneck is upstream of the
|
||||
state-bridge wire.
|
||||
|
||||
**What the smoke ruled in vs. out**
|
||||
|
||||
| Hypothesis | Status |
|
||||
|---|---|
|
||||
| H6 wiring crashes capture / breaks training | Ruled out (clean capture, clean run) |
|
||||
| Aux head produces zero-information predictions | Ruled out (`pred_tanh = 0.66`, strongly directional) |
|
||||
| Aux→state bridge moves policy WR off 50% in ≤3 epochs | Ruled out (50.21% at epoch 1) |
|
||||
| Policy can't gradient-couple to a new directional state dim under current rewards | **Supported** (consistent with reward-density + V/A pathology hypotheses) |
|
||||
|
||||
**Phase 1 close-out**
|
||||
|
||||
- Phase 1 H6 wiring **stays merged** per `feedback_no_functionality_removal`.
|
||||
The state slot is allocated, the buffer is initialized to a neutral
|
||||
sentinel via pure-GPU fill, and the copy kernel runs inside captured
|
||||
graphs. None of this is harmful; if H3 or a V/A fix lands and the
|
||||
policy starts gradient-coupling to directional features, slot 121 is
|
||||
already wired.
|
||||
- A2 (eval-side aux integration) is **deferred indefinitely** — A3 NULL
|
||||
fallback works correctly for the eval and there is no production
|
||||
justification to add an eval-side aux trunk forward until training-side
|
||||
evidence shows the bridge is doing useful work.
|
||||
|
||||
**Next-investigation framing**
|
||||
|
||||
The smoke supports the "policy can't gradient-couple under current
|
||||
rewards" branch. Two upstream candidates:
|
||||
|
||||
1. **H3 — reward density alignment** (per
|
||||
`pearl_event_driven_reward_density_alignment`): event-driven
|
||||
objective (trade close) with per-step shaping creates exposure-positive
|
||||
bias. Per-step micro/opp_cost rewards dominate sparse directional
|
||||
alpha. The cycle-1 telemetry has `hold_reward_ema = -0.2044`,
|
||||
`hold_pct_ema = 0.2004` against `target_hold_pct = 0.1151` — the
|
||||
policy is holding ~74% above target and the hold-reward is strongly
|
||||
negative; cost dominance pattern.
|
||||
|
||||
2. **V/A unidentifiability** (per `project_dueling_va_unidentifiable`):
|
||||
structural; the LOW EXPOSURE DIVERSITY pattern at cycle 1 is a direct
|
||||
tell. The dueling architecture lets the optimizer park V at Q(Flat),
|
||||
degenerating the action distribution regardless of state content.
|
||||
|
||||
**Sequencing**: H3 first, V/A second. Reasoning: if rewards aren't
|
||||
producing directional gradient pressure, then "fixing" V/A is optimizing
|
||||
the same degenerate objective with a different head shape. Heal the
|
||||
gradient signal upstream, then re-test whether V/A still pathologizes
|
||||
with healthy rewards.
|
||||
|
||||
If H3 lands and the WR moves but action-space stays narrow, V/A is
|
||||
binding and gets fixed next. If H3 doesn't move WR either, V/A goes
|
||||
first and we re-test in combination.
|
||||
|
||||
Reference in New Issue
Block a user