From 3a004256a8ddbd3a6c239d74c9cd76993d9ce216 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Wed, 13 May 2026 08:18:56 +0200 Subject: [PATCH] =?UTF-8?q?docs(sp22):=20H6=20Phase=203=20=CE=B1=20Phase?= =?UTF-8?q?=20D=20scoping=20refinement?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Architectural finding: eval evaluator's Q-value computation routes through QValueProvider::compute_q_and_b_logits_to (fused_training.rs:4892) which delegates to trainer.replay_forward_for_q_values — already atom-shift-wired since commit 195c051e7. Implication: eval-side Q-eval INHERITS atom-shift for free. Original runbook's D4 (eval alpha launcher) is SUPERSEDED — no separate eval-side compute_expected_q launch needed. Actual eval-side gap: state[121] is the 0.5 sentinel (aux_dir_prob_null passed to state_gather kernels) → recentered state_121 = 0 → atom_shift = W[a] × 0 = 0. Atom-shift is a runtime no-op on eval side until D2/D3/D5 land the aux trunk forward + state[121] population. Phase D revised estimate: ~25-35 hr → ~3-4 hr if pursued, contingent on trainer-side smoke validating the mechanism first. Recommendation: defer Phase D until smoke results show W movement and training-time WR shift on the trainer/rollout side. If trainer-side moves WR, Phase D's eval-side activation is justified investment. If not, re-evaluate H6 hypothesis itself. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/dqn-wire-up-audit.md | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index dbf1ae8bb..626fa749b 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -17214,3 +17214,40 @@ With the trainer's W updated each step by Step 8+11 Adam, the rollout policy's d - **Phase F** (5 tasks): audit doc + atomic commit + push + smoke + verdict. **Trainer + collector are now smoke-ready end-to-end for the trainer/rollout side**. Eval-side activation (Phase D) is needed for the validation backtests to also benefit from atom-shift. + +#### Phase D scoping refinement (2026-05-13) — Q-eval side inherits via QValueProvider + +While investigating Phase D's eval-side aux infrastructure, a key architectural finding: + +**The eval evaluator's Q-value computation does NOT call `compute_expected_q` directly.** It routes through the `QValueProvider` trait abstraction (`crates/ml/src/cuda_pipeline/q_value_provider.rs:69`), whose canonical implementation lives on `FusedTrainingCtx` (`crates/ml/src/trainers/dqn/fused_training.rs:4892`). + +`FusedTrainingCtx::compute_q_and_b_logits_to`: +1. `launch_pad_states(trainer.states_buf, eval_states_ptr, sub_b)` — copies eval's chunked states into the trainer's `states_buf`. +2. `trainer.replay_forward_for_q_values(sub_b)` — runs the trainer's forward graph + `compute_expected_q` (already atom-shift-wired since commit `195c051e7`). +3. DtoD-copies `q_out_buf` / `on_b_logits_buf` / `on_v_logits_buf` back into eval's chunked outputs. + +**Implication**: The eval side INHERITS atom-shift on the Q-eval side for free. No D4-equivalent launch is needed for the eval's `compute_expected_q` — that's already happening inside `replay_forward_for_q_values`. + +**The actual gap on the eval side is the state buffer's `state[121]` value**: + +The eval state-gather launchers (`launch_state_gather` lines ~1572, ~3173; `launch_gather_chunk` line ~3243) currently pass `aux_dir_prob_null = 0u64` per the H6 Phase 1 fallback. This makes `state[121] = 0.5` (sentinel) → recentered `state_121 = 1 - 2*0.5 = 0` → `atom_shift = W[a] × 0 = 0` → atom-shift is a runtime no-op on the eval side. + +For eval-side atom-shift to ACTUALLY be active, the missing pieces are: +1. **D2-equivalent**: Allocate `prev_aux_dir_prob_per_env_buf` on the evaluator (`[n_windows]`). +2. **D3-equivalent**: Per-step (or per-chunk) launch the aux trunk forward on the eval's gathered states, compute softmax, and write `p_up` (or recentered `1 - 2*p_up`) into the per-env buffer. +3. **D5-equivalent**: Replace `aux_dir_prob_null` with the populated buffer in the 3 state-gather launches. +4. **D6/D7-equivalent**: Either share trainer's aux trunk weights via pointer (cheap — mirror the W ptr setter pattern) OR replicate them onto the evaluator (expensive, mirror-state risk). + +**Recommendation**: Defer Phase D until trainer/rollout-side smoke results validate the mechanism. The trainer side W training + rollout-side action shift can be measured via: +- W[a] values drifting off structural prior (Adam updates) +- Training rollout's direction-action distribution per state_121 sign +- HEALTH_DIAG `r_aux_align` EMA component (β reward shaping) + +If the trainer-side mechanism MOVES WR (training-time, not eval-time), then Phase D's eval-side activation is justified investment. If it doesn't, Phase D is wasted effort and we re-evaluate the H6 hypothesis itself. + +**Phase D revised scope** (if pursued): +- D7 simplification: share trainer's aux trunk weight ptr via setter (instead of replicating to eval). The aux trunk weights are part of `trainer.params_buf` already — the eval can borrow `trainer.aux_trunk_param_ptr_range()` and call a per-step kernel that performs the aux forward on eval's chunked states. +- D2-D5: ~3-4 hours of focused CUDA work, mirroring the trainer's `submit_aux_ops` chain (aux trunk forward → softmax → recentering to state_121 → write into the state buffer slot before state_gather). +- D4 SUPERSEDED: eval-side `compute_expected_q` already inherits atom-shift via `QValueProvider::compute_q_and_b_logits_to` → `replay_forward_for_q_values`. No separate eval-side launcher needed. + +This refines the runbook's original Phase D estimate from "~25-35 hr" to "~3-4 hr" engineering, contingent on the trainer-side smoke validating the mechanism first.