feat(sp22): H6 Phase 1 — aux→policy state bridge (atomic)

Wires the aux head's per-env directional probability into policy STATE
slot 121 (AUX_DIR_PROB_INDEX = PADDING_START + 0), preserving the trunk-
separation invariant from `pearl_separate_aux_trunk_when_shared_starves`.
H1 (label horizon) confirmed aux learns 78% dir-acc within-fold at H=200
but the policy was walled off; this commit conducts that signal through
the state input with a one-step lag.

Mechanism (rollout-time, collector-only)
────────────────────────────────────────
- State[env, t] reads `prev_aux_dir_prob[env]` (= p_up from step t-1).
- After aux forward at step t, the new copy kernel writes
  `aux_softmax[env, 1]` → `prev_aux_dir_prob[env]` for step t+1.
- Cold-start + FoldReset seed the buffer to 0.5 (neutral; p_up = 50%)
  via the pure-GPU `fill_f32` kernel — no HtoD per
  `feedback_no_htod_htoh_only_mapped_pinned.md`.
- Launch sits in the same `isv_signals && trainer_params != 0` gate as
  the aux forward, so when aux is skipped the cache keeps its previous
  (sentinel or last-good) value instead of copying stale `alloc_zeros`.

Three state-gather kernels updated atomically (per
`feedback_no_partial_refactor.md`):
- `experience_state_gather` — training, reads `aux_dir_prob_per_env[i]`
- `backtest_state_gather` — eval (single-step), NULL → 0.5 (A3 fallback)
- `backtest_state_gather_chunk` — eval (chunked), NULL → 0.5 (A3 fallback)

`assemble_state` gained a 7th param `float aux_dir_prob` written to
`out[SL_PADDING_START + 0]`; the remaining 6 padding slots stay zero
for 8-alignment.

Phase 1 scope = training-side + eval A3 NULL fallback. A2 (aux trunk
forward in eval) is deferred per the runbook — gates on whether the
smoke moves WR off the 50.1–50.2% plateau.

New files
─────────
- crates/ml/src/cuda_pipeline/aux_softmax_to_per_env_kernel.cu

Modified
────────
- crates/ml-core/src/state_layout.rs           (+AUX_DIR_PROB_INDEX)
- crates/ml/src/cuda_pipeline/state_layout.cuh (+assemble_state param)
- crates/ml/src/cuda_pipeline/experience_kernels.cu
    (3 state-gather kernels + NULL-defensive sentinel)
- crates/ml/src/cuda_pipeline/gpu_experience_collector.rs
    (per-env buffer + 2 kernel handles + cold-start fill + copy launch
     + FoldReset re-fill + state-gather arg)
- crates/ml/src/cuda_pipeline/gpu_backtest_evaluator.rs
    (NULL aux_dir_prob_per_env at all 3 launchers for A3)
- crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs
    (SP22_AUX_SOFTMAX_TO_PER_ENV_CUBIN static)
- crates/ml/src/cuda_pipeline/gpu_action_selector.rs
    (EPSILON_GREEDY_CUBIN → pub(crate) so collector reuses fill_f32)
- crates/ml/build.rs                            (register new kernel)
- docs/dqn-wire-up-audit.md
    (## 2026-05-12 — SP22 H6 implementation: Phase 1 entry)

Verification (all three gates clean, no smoke yet)
──────────────────────────────────────────────────
- cargo check -p ml --features cuda: 0 errors
- gpu_backtest_validation: 4/4 expected-passing tests still pass
  (the 2 PnL-assertion failures are pre-existing per the runbook)
- compute-sanitizer --tool=memcheck: 0 CUDA errors

Refs
────
- docs/plans/2026-05-12-sp22-h6-aux-policy-state-bridge.md
- docs/plans/2026-05-12-sp22-h6-next-session-prompt.md
- pearl_separate_aux_trunk_when_shared_starves
- pearl_first_observation_bootstrap (sentinel = 0.5 cold-start)
- feedback_no_htod_htoh_only_mapped_pinned (fill_f32 not HtoD)
- feedback_no_partial_refactor (3 state-gather kernels atomic)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-12 21:31:32 +02:00
parent ff9ec76f18
commit 7fc9799343
10 changed files with 453 additions and 13 deletions

View File

@@ -672,6 +672,18 @@ pub(crate) static SP13_AUX_PRED_TO_ISV_TANH_CUBIN: &[u8] =
pub(crate) static SP14_Q_DISAGREEMENT_CUBIN: &[u8] =
include_bytes!(concat!(env!("OUT_DIR"), "/q_disagreement_update_kernel.cubin"));
/// SP22 H6 (2026-05-12): per-env p_up extractor. Copies
/// `aux_softmax[env, 1]` → `prev_aux_dir_prob[env]` after each aux forward
/// during rollout. The cache buffer is read at the next step's state
/// assembly to populate `state[AUX_DIR_PROB_INDEX = SL_PADDING_START + 0]`,
/// wiring the aux head's directional probability into policy STATE.
/// Loaded from `aux_softmax_to_per_env_kernel.cubin`. Consumed by the
/// collector (`GpuExperienceCollector::collect_experiences_gpu`); no
/// trainer-side consumer (this is a rollout-time bridge, not a training-
/// step kernel).
pub(crate) static SP22_AUX_SOFTMAX_TO_PER_ENV_CUBIN: &[u8] =
include_bytes!(concat!(env!("OUT_DIR"), "/aux_softmax_to_per_env_kernel.cubin"));
/// SP17 Task PP.3 (2026-05-08): pre-centering V/A mean diagnostic.
/// Single-block tree-reduce kernel reading the existing `on_v_logits_buf
/// [B, NA]` and `on_b_logits_buf [B, (B0+B1+B2+B3)*NA]` (BRANCH-MAJOR) and