docs(sp22): smoke verdict template for H6 Phase 3 α

Defines verdict criteria for the train-qsltr smoke at commit 5106e3b1:
- Mechanism activity (eval_dist, intent_dist, dist_q/h/f)
- WR signal (last_epoch_win_rate vs 46% baseline → 50%+ target)
- Stability (health, action_entropy, no regression)

Possible outcome table maps WR + dist pattern combinations to next-action
recommendations (Phase D vs tune vs revert vs investigate).

Deliberately deferred from this smoke: W[0..4] direct readback via mapped
pinned + HEALTH_DIAG log line (avoids bug risk during smoke window).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-13 08:25:15 +02:00
parent 3a004256a8
commit a57ec3da8b

View File

@@ -17251,3 +17251,43 @@ If the trainer-side mechanism MOVES WR (training-time, not eval-time), then Phas
- D4 SUPERSEDED: eval-side `compute_expected_q` already inherits atom-shift via `QValueProvider::compute_q_and_b_logits_to``replay_forward_for_q_values`. No separate eval-side launcher needed.
This refines the runbook's original Phase D estimate from "~25-35 hr" to "~3-4 hr" engineering, contingent on the trainer-side smoke validating the mechanism first.
#### Smoke verdict template — SP22 H6 Phase 3 α (2026-05-13)
Workflow: `train-qsltr` @ commit `5106e3b117e89ae5a7e6f0de9f09ed0fe600ad87`, 3 epochs, L40S pool, baseline mode.
**Verdict criteria** (read HEALTH_DIAG output across 3 epochs):
**A. Mechanism activity (does atom-shift engage?)**
1. `eval_dist [eq, eh, ef]` — eval action distribution. If atom-shift biases Q toward aux-aligned dir actions, the distribution should TIGHTEN around the dominant direction.
2. `intent_dist [iq, ih, if]` — rollout action distribution. Atom-shift biases per-env action selection, so intent_dist should diverge from baseline when state_121 ≠ 0.
3. `dist_q / dist_h / dist_f` per HEALTH_DIAG mag line — magnitude dimension action distribution. Atom-shift only affects direction; magnitude should stay relatively stable.
**B. Win rate signal (does mechanism translate to performance?)**
1. `last_epoch_win_rate` (logged per epoch in financials path)
- Baseline: ~46% per `pearl_per_bar_vs_segment_pnl_signal_mismatch` (segment-level, post-bug-fix)
- Mechanism-positive: ≥ 50% in any of epochs 1-3
- Strong signal: ≥ 53% by epoch 3
2. `mag_stats wr_q / wr_h / wr_f` — per-magnitude-bin win rates. If atom-shift's direction bias is correct, the dominant magnitude bin's WR should lift.
**C. Stability (no regression)**
1. `health` value — overall training health. Should stay ≥ 0.5; collapse < 0.3 triggers plasticity injection.
2. `action_entropy` — policy entropy. Should NOT collapse below 0.3 (action-collapse trigger).
3. No `TERMINATED_BY_REGRESSION` log lines.
4. `c51_loss` shouldn't diverge — atom-shift changes the loss landscape, so SOME shift in c51_loss magnitude is expected, but >10× jump signals instability.
**Possible outcomes**:
| Pattern | Interpretation | Next action |
|---------|----------------|-------------|
| WR ≥ 53% epoch 3, no regression | Mechanism works end-to-end on trainer/rollout | Phase D for eval validation |
| WR 50-53% epoch 3, dist changes visible | Mechanism partially working; dir bias too weak | Tune W init magnitudes (0.5 → 1.0?) |
| WR ~46% but action dist shifts | Mechanism running but no WR translation | Investigate β reward signal disconnect |
| WR ~46% no dist shift | Atom-shift not active (state_121 always 0?) | Check aux head producing recentered state[121] in rollout |
| Training collapse / regression | Loss landscape change destabilized training | Reduce α magnitude or revert Phase 3 |
**Diagnostic to add for next smoke** (deliberately deferred this run):
- W[0..4] direct readback per epoch via mapped pinned [4] f32 + DtoD copy outside captured graph + HEALTH_DIAG log line. Would directly show Adam updates on W.