docs(sp22): H6 Phase 3 spec — combined α (bypass-head) + β (event-driven aux bonus)
Phase 2 verdict gave mechanistic evidence that the encoder's weights
for state[121] are essentially unused (action distribution
bit-identical across Phase 1 and Phase 2 encodings; drift < 0.5% on
every action bin). The aux signal reaches state[121] (`pred_tanh =
0.66`) but the policy ignores it because slot 121's encoder weights
need many training steps to grow from zero.
Phase 3 routes aux around the cold-encoder bottleneck via two
non-overlapping mechanisms (combined per
`pearl_no_deferrals_for_complementary_fixes`):
α (post-encoder bypass-head)
────────────────────────────
- New learnable param `W_aux_to_Q_dir: [4]` f32 (one weight per
direction action). Adam states `m`, `v` ([4] each). 12 floats total.
- Forward (rollout + training graphs, post-Q_dir):
Q_dir[b, a] += W_aux_to_Q_dir[a] * state_121[b]
- Backward (training only):
dW[a] = Σ_b state_121[b] * dQ_dir[b, a] (per-action block reduce)
dstate_121[b] += Σ_a W[a] * dQ_dir[b, a] (per-sample sum)
- New cubins:
aux_to_q_dir_bias_kernel.cu (forward)
aux_to_q_dir_bias_backward_kernel.cu (backward)
- Zero-init; Phase-3 step-0 output identical to Phase-2 baseline.
Adam-driven growth tracks aux-Q correlation.
- Reads state_121 from the encoder INPUT (batch_states[121]), bypassing
the encoder entirely.
β (event-driven aux-aligned trade-close bonus)
──────────────────────────────────────────────
At trade-close (existing segment_complete branch in
experience_env_step):
position_sign = sign(prev_position)
aux_at_close = state_121[env]
alignment = max(0, aux_at_close * position_sign) ∈ [0, +1]
profit_term = max(0, realized_pnl) ∈ [0, +∞)
r_aux_align = scale_beta * alignment * profit_term
Added to existing r_trail (no new reward component for first test —
YAGNI). scale_beta = 0.5 fixed. Non-negative-only (no anti-alignment
penalty; let r_trail's loss carry the negative signal). Event-driven
per `pearl_event_driven_reward_density_alignment`. Bounded by
`pearl_one_unbounded_signal_per_reward` (realized_pnl is the single
unbounded multiplicand, which IS the natural per-trade scale).
Why combined
────────────
α: direct forward-pass channel (aux shifts Q in real-time).
β: indirect training-signal channel (aux-aligned profitable trades
get bigger gradient → encoder learns to use state[121] faster, AND
the policy learns to take aux-aligned trades).
Decision points captured (with rationale)
─────────────────────────────────────────
| Question | Decision | Rationale |
| α scope | 4-weight vector | aux is directionally signed |
| β formula | non-negative only | avoid confusing policy when aux wrong |
| β scale | fixed 0.5 | minimum scope for verdict experiment |
| state[121] grad routing | both paths open | simplest; α dominates initially |
| β placement | added to r_trail | YAGNI; 7th component deferred |
| W sharing | trainer holds, collector reads | mirrors GEMM pattern |
Verification (all required clean before commit)
───────────────────────────────────────────────
- cargo check -p ml --features cuda (0 errors, 21 pre-existing
warnings expected — parity with Phase 2 baseline)
- gpu_backtest_validation (6/6 pass; post-Phase-2-test-fix baseline)
- compute-sanitizer --tool=memcheck (0 errors)
- Captured-graph integrity check (CAPTURE_PHASE_FORWARD_DONE +
parent compose visible in smoke logs)
- α weight movement check (W_aux_to_Q_dir non-zero after epoch 1)
Verdict criteria
────────────────
- WR > 50.5% within 3 epochs → confirmed → justify A2.
- W_aux_to_Q_dir non-zero magnitudes → α gradient flowed.
- Action distribution diverges from Phase 1/2 baselines → policy is
exercising different actions.
- a_var [m, o, u] > 1e-3 → sub-branches gradient-coupled.
- WR pinned + W non-zero + action distribution moved → bypass routing
is working but aux signal isn't actionable on current fixtures →
pivot to deeper hypothesis (per-branch IQN tau audit, reward
density rewrite, longer horizon).
- WR pinned + W near-zero → gradient flow broken; kernel debug.
Estimated effort: ~10–16 hr engineering + ~37 min smoke wall-clock.
Refs
────
- docs/plans/2026-05-12-sp22-h6-aux-policy-state-bridge.md (Phase 1)
- docs/plans/2026-05-12-sp22-h6-phase2-recenter.md (Phase 2)
- pearl_no_deferrals_for_complementary_fixes (combined plan)
- pearl_event_driven_reward_density_alignment (β rationale)
- pearl_one_unbounded_signal_per_reward (β bound)
- pearl_first_observation_bootstrap (Phase 2 sentinel = 0)
- pearl_no_host_branches_in_captured_graph (kernel discipline)
- pearl_separate_aux_trunk_when_shared_starves (aux trunk untouched)
- pearl_no_atomicadd (backward reduce discipline)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
754aea6995