docs(sp22): H6 Phase 3 α Phase D scoping refinement
Architectural finding: eval evaluator's Q-value computation routes through
QValueProvider::compute_q_and_b_logits_to (fused_training.rs:4892) which
delegates to trainer.replay_forward_for_q_values — already atom-shift-wired
since commit 195c051e7.
Implication: eval-side Q-eval INHERITS atom-shift for free. Original
runbook's D4 (eval alpha launcher) is SUPERSEDED — no separate eval-side
compute_expected_q launch needed.
Actual eval-side gap: state[121] is the 0.5 sentinel (aux_dir_prob_null
passed to state_gather kernels) → recentered state_121 = 0 → atom_shift =
W[a] × 0 = 0. Atom-shift is a runtime no-op on eval side until D2/D3/D5
land the aux trunk forward + state[121] population.
Phase D revised estimate: ~25-35 hr → ~3-4 hr if pursued, contingent on
trainer-side smoke validating the mechanism first.
Recommendation: defer Phase D until smoke results show W movement and
training-time WR shift on the trainer/rollout side. If trainer-side moves
WR, Phase D's eval-side activation is justified investment. If not,
re-evaluate H6 hypothesis itself.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -17214,3 +17214,40 @@ With the trainer's W updated each step by Step 8+11 Adam, the rollout policy's d
|
||||
- **Phase F** (5 tasks): audit doc + atomic commit + push + smoke + verdict.
|
||||
|
||||
**Trainer + collector are now smoke-ready end-to-end for the trainer/rollout side**. Eval-side activation (Phase D) is needed for the validation backtests to also benefit from atom-shift.
|
||||
|
||||
#### Phase D scoping refinement (2026-05-13) — Q-eval side inherits via QValueProvider
|
||||
|
||||
While investigating Phase D's eval-side aux infrastructure, a key architectural finding:
|
||||
|
||||
**The eval evaluator's Q-value computation does NOT call `compute_expected_q` directly.** It routes through the `QValueProvider` trait abstraction (`crates/ml/src/cuda_pipeline/q_value_provider.rs:69`), whose canonical implementation lives on `FusedTrainingCtx` (`crates/ml/src/trainers/dqn/fused_training.rs:4892`).
|
||||
|
||||
`FusedTrainingCtx::compute_q_and_b_logits_to`:
|
||||
1. `launch_pad_states(trainer.states_buf, eval_states_ptr, sub_b)` — copies eval's chunked states into the trainer's `states_buf`.
|
||||
2. `trainer.replay_forward_for_q_values(sub_b)` — runs the trainer's forward graph + `compute_expected_q` (already atom-shift-wired since commit `195c051e7`).
|
||||
3. DtoD-copies `q_out_buf` / `on_b_logits_buf` / `on_v_logits_buf` back into eval's chunked outputs.
|
||||
|
||||
**Implication**: The eval side INHERITS atom-shift on the Q-eval side for free. No D4-equivalent launch is needed for the eval's `compute_expected_q` — that's already happening inside `replay_forward_for_q_values`.
|
||||
|
||||
**The actual gap on the eval side is the state buffer's `state[121]` value**:
|
||||
|
||||
The eval state-gather launchers (`launch_state_gather` lines ~1572, ~3173; `launch_gather_chunk` line ~3243) currently pass `aux_dir_prob_null = 0u64` per the H6 Phase 1 fallback. This makes `state[121] = 0.5` (sentinel) → recentered `state_121 = 1 - 2*0.5 = 0` → `atom_shift = W[a] × 0 = 0` → atom-shift is a runtime no-op on the eval side.
|
||||
|
||||
For eval-side atom-shift to ACTUALLY be active, the missing pieces are:
|
||||
1. **D2-equivalent**: Allocate `prev_aux_dir_prob_per_env_buf` on the evaluator (`[n_windows]`).
|
||||
2. **D3-equivalent**: Per-step (or per-chunk) launch the aux trunk forward on the eval's gathered states, compute softmax, and write `p_up` (or recentered `1 - 2*p_up`) into the per-env buffer.
|
||||
3. **D5-equivalent**: Replace `aux_dir_prob_null` with the populated buffer in the 3 state-gather launches.
|
||||
4. **D6/D7-equivalent**: Either share trainer's aux trunk weights via pointer (cheap — mirror the W ptr setter pattern) OR replicate them onto the evaluator (expensive, mirror-state risk).
|
||||
|
||||
**Recommendation**: Defer Phase D until trainer/rollout-side smoke results validate the mechanism. The trainer side W training + rollout-side action shift can be measured via:
|
||||
- W[a] values drifting off structural prior (Adam updates)
|
||||
- Training rollout's direction-action distribution per state_121 sign
|
||||
- HEALTH_DIAG `r_aux_align` EMA component (β reward shaping)
|
||||
|
||||
If the trainer-side mechanism MOVES WR (training-time, not eval-time), then Phase D's eval-side activation is justified investment. If it doesn't, Phase D is wasted effort and we re-evaluate the H6 hypothesis itself.
|
||||
|
||||
**Phase D revised scope** (if pursued):
|
||||
- D7 simplification: share trainer's aux trunk weight ptr via setter (instead of replicating to eval). The aux trunk weights are part of `trainer.params_buf` already — the eval can borrow `trainer.aux_trunk_param_ptr_range()` and call a per-step kernel that performs the aux forward on eval's chunked states.
|
||||
- D2-D5: ~3-4 hours of focused CUDA work, mirroring the trainer's `submit_aux_ops` chain (aux trunk forward → softmax → recentering to state_121 → write into the state buffer slot before state_gather).
|
||||
- D4 SUPERSEDED: eval-side `compute_expected_q` already inherits atom-shift via `QValueProvider::compute_q_and_b_logits_to` → `replay_forward_for_q_values`. No separate eval-side launcher needed.
|
||||
|
||||
This refines the runbook's original Phase D estimate from "~25-35 hr" to "~3-4 hr" engineering, contingent on the trainer-side smoke validating the mechanism first.
|
||||
|
||||
Reference in New Issue
Block a user