diff --git a/crates/ml/src/trainers/dqn/fused_training.rs b/crates/ml/src/trainers/dqn/fused_training.rs index c7f5258b4..74d6f45a1 100644 --- a/crates/ml/src/trainers/dqn/fused_training.rs +++ b/crates/ml/src/trainers/dqn/fused_training.rs @@ -1971,73 +1971,42 @@ impl FusedTrainingCtx { self.trainer.populate_q_out(self.batch_size) .map_err(|e| anyhow::anyhow!("IQL populate_q_out: {e}"))?; - // SP14 Layer B Task B.11 (2026-05-07 cadence fix): per-step EGF - // producer chain. The original B.11 wire-up (commit 857722e77) - // placed these in `process_epoch_boundary` — that ran them ONCE - // per epoch instead of per training step, leaving every consumer - // step inside the epoch reading ISV[393]=ALPHA_GRAD_SMOOTHED - // values that were updated up to (steps_per_epoch - 1) steps ago. - // The L40S `train-v8ztm` 10-epoch run showed α_smoothed=0.0002 - // (vs. expected ~0.5 in steady state) — exactly the symptom of - // a producer chain firing < 1% as often as the consumer. - // - // Moving them here puts the launches inside `aux_child` graph - // capture, so each parent-graph replay (i.e. each training step) - // re-fires the EGF chain. The kernels are graph-safe — pure - // device-side state machines reading only ISV slots and the - // captured-graph-resident `aux_nb_softmax_buf` + `q_out_buf`, - // no host branches per `pearl_no_host_branches_in_captured_graph`. - // - // Order is load-bearing (consumer chain): - // 1. populate_q_out (existing, line above) writes q_out_buf - // from this step's online forward logits. - // 2. launch_sp13_aux_dir_metrics reduces aux_nb_softmax_buf - // → ISV[AUX_DIR_ACC_SHORT_EMA=373] / ISV[*_LONG_EMA=374] - // / ISV[AUX_DIR_PREDICTION=375]. Moved per-step from - // process_epoch_boundary because slot 373/374 are - // precondition reads for launch_sp14_alpha_grad_compute - // below (alpha_grad's kernel header explicitly requires - // sp13 to have fired earlier in the SAME step). Leaving - // sp13 per-epoch while making alpha_grad per-step would - // have alpha_grad reading aux_dir_acc EMAs from - // (steps_per_epoch - 1) steps ago — same staleness - // defeat as the original wire-up bug. Atomic dependency - // migration per `feedback_no_partial_refactor`. - // 3. launch_sp14_q_disagreement_update reads aux_nb_softmax - // (K=2) + q_out_buf direction-Q slice (K=4) and writes - // ISV[Q_DISAGREEMENT_SHORT/LONG/VAR_EMA = 383/384/389]. - // 4. launch_sp14_alpha_grad_compute consumes #2's slots - // 373/374 + #3's slot 383/389 + the rest of the EGF - // driver block, runs the Schmitt-trigger Gate 1 + - // adaptive k + adaptive β state machine, writes - // ISV[ALPHA_GRAD_RAW=392] / ISV[ALPHA_GRAD_SMOOTHED=393] - // and the supporting bookkeeping slots [385..395]. - // The captured backward consumer - // (launch_sp14_scale_wire_col inside - // launch_cublas_backward_to) reads slot 393 within the - // SAME captured replay — fresh per step, no one-step - // lag like the prior pattern. - // - // α_short = 0.3, α_long = 0.05, α_var = 0.05 per spec - // §2581-2585. The variance-driven k_aux/k_q in - // alpha_grad_compute_kernel handles warmup adaptively (no - // explicit warmup_gate per `feedback_isv_for_adaptive_bounds`, - // c0fc28e45). var_aux producer is inline in - // alpha_grad_compute_kernel (Welford on aux_short vs aux_long). + // SP14 β-migration step 5 (2026-05-07): EGF producer chain + // MOVED to `gpu_experience_collector::collect_experiences_gpu` + // per-rollout-step body. The training-time chain that used + // to live here (`launch_sp13_aux_dir_metrics` + + // `launch_sp14_q_disagreement_update` + + // `launch_sp14_alpha_grad_compute`, commits 857722e77 → + // 200f05fce per-step migration → 9d0c124ce empty-batch gate) + // is RETIRED from production: even with the per-step cadence + // and the empty-batch gate, replay-batch Q-direction picks + // are dominated by Hold/Flat (the natural argmax distribution + // of the converged 4-way head) so the EGF producers see + // total_cnt=0 every step and the gate keeps EMAs at zero. + // The collector-native β path (commit c691bd381) reads + // rollout-time q_values where Thompson sampling forces + // Short/Long picks every step — that's the genuine source + // of aux↔Q disagreement signal. // // The per-epoch `launch_sp14_gradient_hack_detect` circuit - // breaker stays at process_epoch_boundary — its lockout - // counter decrement is one-per-epoch by design. - self.trainer.launch_sp13_aux_dir_metrics() - .map_err(|e| anyhow::anyhow!("SP14 B.11 sp13_aux_dir_metrics: {e}"))?; - let alpha_short_qd: f32 = 0.3; - let alpha_long_qd: f32 = 0.05; - let alpha_var: f32 = 0.05; - self.trainer.launch_sp14_q_disagreement_update( - alpha_short_qd, alpha_long_qd, alpha_var, - ).map_err(|e| anyhow::anyhow!("SP14 B.11 q_disagreement_update: {e}"))?; - self.trainer.launch_sp14_alpha_grad_compute(alpha_var) - .map_err(|e| anyhow::anyhow!("SP14 B.11 alpha_grad_compute: {e}"))?; + // breaker stays at process_epoch_boundary unchanged — its + // lockout counter decrement is one-per-epoch by design. + // + // Trainer launcher methods at gpu_dqn_trainer.rs:8671 / 8733 + // / 15270 stay defined as `pub(crate)` (no compiler warnings + // — Rust doesn't warn unused crate-public methods). Oracle + // tests in `crates/ml/tests/sp14_oracle_tests.rs` exercise + // the SAME kernels directly via `load_cubin` / + // `load_function`, not through these launchers — so deletion + // of the launchers would not affect oracle coverage. They + // are retained for atomic-rollback potential and for the + // possibility that the trainer-time path becomes useful + // again if a future curriculum stage reverses the rollout- + // only signal-quality assumption. Future cleanup: delete + // them along with their associated CudaFunction fields if + // the rollout-only path is validated stable across a full + // 50-epoch L40S run (tracked but not blocking the β + // migration close-out). // 2026-05-07 cadence-fix continuation (after SP14 B.11 commit // 200f05fce): four MORE producers whose docstrings explicitly diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index 60418f7a3..fb2586c4e 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -2,6 +2,8 @@ **Status:** Populated during Plan 1 Task 6 (A.5 orphan audit). Updated on every commit per Invariant 7. +SP14 β-migration step 5 — remove training-time EGF launches from `submit_aux_ops` (2026-05-07): Final atomic step of the 5-commit β migration. Deletes the 3 trainer-time launches at `crates/ml/src/trainers/dqn/fused_training.rs:2031-2040` (`launch_sp13_aux_dir_metrics` + `launch_sp14_q_disagreement_update` + `launch_sp14_alpha_grad_compute`) — these were the producers introduced by SP14 B.11 commit `200f05fce` (per-step migration) and stabilised by commit `9d0c124ce` (empty-batch decay gate). Even with both fixes the trainer-time path could only fire against replay-batch Q-direction picks dominated by Hold/Flat (the natural argmax distribution of the converged 4-way head), so `total_cnt=0` every step and the gate kept EMAs at zero. The collector-native β path (commit c691bd381 / step 4) reads rollout-time q_values where Thompson sampling forces Short/Long picks every step — that's the genuine source of aux↔Q disagreement signal. The collector-native chain is now the SINGLE production caller. **Trainer launcher methods preserved**: `launch_sp14_q_disagreement_update` (line 8671), `launch_sp14_alpha_grad_compute` (line 8733), `launch_sp13_aux_dir_metrics` (line 15270) and the 3 sub-launchers (`launch_aux_dir_acc_reduce`, `launch_apply_fixed_alpha_ema`, `launch_aux_pred_to_isv_tanh`) stay defined as `pub(crate)` (no Rust compiler warnings — `pub(crate)` methods don't trigger dead-code warnings). Oracle tests in `crates/ml/tests/sp14_oracle_tests.rs` exercise the SAME kernels directly via `load_cubin` / `load_function`, not through these launchers — verified by `grep -n "launch_sp14|launch_sp13_aux" tests/sp14_oracle_tests.rs` returning 0 hits — so deletion of the launchers would not affect oracle coverage. They are retained for atomic-rollback potential and the possibility that the trainer-time path becomes useful again if a future curriculum stage reverses the rollout-only signal-quality assumption. **`launch_sp14_gradient_hack_detect` (per-epoch circuit breaker) unchanged** — the lockout-counter decrement is one-per-epoch by design, stays at `process_epoch_boundary`. The replacement comment at the deletion site in `fused_training.rs` documents the migration trail (β-migration step 5 → c691bd381 → 9d0c124ce) so a future archeologist can reconstruct the chain. **Verification**: `SQLX_OFFLINE=true cargo check -p ml --lib` clean (dev profile, only pre-existing unused-var warnings unrelated to this change); `cargo test -p ml --test sp14_oracle_tests` 2/2 non-GPU pass (7 GPU tests ignored on RTX 3050 Ti host — verify kernel correctness, unchanged); `grep -rn "trainer.launch_sp14|trainer.launch_sp13_aux" crates/ml/src/` returns 0 production callers (the 3 calls in fused_training.rs are deleted). L40S smoke validation pending — expect EGF chain to drive non-zero ISV[383/384/389/393] across the full curriculum, unblocking Gate 1 from its closed-forever state. Files modified: `crates/ml/src/trainers/dqn/fused_training.rs` (−10 / +14 lines: deleted 3 launches, replaced 60-line audit-comment block with 12-line redirect comment). + SP14 β-migration step 4 — wire collector-native SP14 EGF producer chain after aux forward (2026-05-07): Step 4 of 5. Inserts the 3 SP14/SP13-EGF producer launches into the collector's per-rollout-step body, immediately after the `expected_q_kernel` (which produces `q_values`) and BEFORE the IQR/ensemble/noise SAXPY blocks (so the SP14 q_disagreement consumer sees noise-free `q_values` matching the trainer's `q_out_buf` semantic). Producer order matches the trainer's `submit_aux_ops` chain (preserved per `feedback_no_partial_refactor.md`): (1) SP13 dir-acc reduce + 2 fixed-α EMAs (α=0.3 short → ISV[373], α=0.05 long → ISV[374]) + aux_pred → ISV[375] tanh — same 4 launches the trainer's `launch_sp13_aux_dir_metrics` orchestrates inline (gpu_dqn_trainer.rs:15270), reading the collector's `exp_aux_nb_softmax_buf [n_episodes, 2]` (from step 3) + `exp_aux_nb_label_buf [n_episodes]` (from step 3's per-step label producer) + writing to the shared ISV via `isv_signals_dev_ptr`. Sentinel 0.5 = random-guessing baseline per `pearl_first_observation_bootstrap`. (2) SP14 `q_disagreement_update_kernel` — reads `exp_aux_nb_softmax_buf [n_episodes, K=2]` + collector's `q_values [n_episodes, q_stride=13]` (post-`expected_q_kernel` output); writes ISV[383/384/389] (q_dis_short, q_dis_long, var_q EMAs). q_stride=13 matches the trainer's q_out_buf stride; the kernel's argmax-over-K_DIR=4 finds the direction-head pick from the first 4 columns of each row. Single block, 256 threads, smem = 2 × 256 × f32. (3) SP14 `alpha_grad_compute_kernel` — pure ISV-state state machine reading the q_dis EMAs just written above (same-stream serial ordering enforces the dependency) plus ISV[372..395) drivers; writes ISV[392/393] (α_grad raw + smoothed) plus the Schmitt-trigger / k_q / k_aux state machine bookkeeping at ISV[385..391]. Single thread (we launch with 32 to mirror the B.4 oracle test launch shape). The kernel's empty-batch decay gate (commit 9d0c124ce) preserves prior EMA values when `total_cnt == 0` for a given step — same protection extends to the rollout path. **Gating**: `if self.isv_signals_dev_ptr != 0 && self.trainer_params_ptr != 0` — without ISV the EMAs have nowhere to write; without trainer params the aux forward in step 3 would have been skipped and the softmax tile would still be `alloc_zeros`. **No `seed_phase_active_cache` gate** here (unlike SP13 hold-rate which prices Hold-cost only post-seed): the EGF Schmitt-trigger Gate 1 needs to observe BOTH seed-phase scripted-policy actions and post-seed Q-policy actions to discriminate aux signal quality across the curriculum. **q_logits placement note**: the SP14 chain runs AFTER `expected_q_kernel` (line 4581) and BEFORE the IQR (3b) / ensemble (3c) / noise (3d) SAXPY bonuses — so `q_values` here is bit-equivalent to the trainer's `q_out_buf` semantic at the trainer's `launch_sp14_q_disagreement_update` call site. The IQR/ensemble bonuses below are exploration noise added for action selection, not gradient flow. Per `pearl_no_host_branches_in_captured_graph` the launches are OUTSIDE any captured graph (the captured `forward` graph has already ended). The trainer's existing private `launch_sp14_q_disagreement_update` / `launch_sp14_alpha_grad_compute` / `launch_sp13_aux_dir_metrics` methods at gpu_dqn_trainer.rs:8671 / 8733 / 15270 are still defined and still pass kernel correctness via the 7 sp14_oracle GPU tests — production has migrated to the collector path; step 5 will delete the now-unused `submit_aux_ops` invocations (commit `200f05fce` lines 2031-2040). Files modified: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` (+200 lines: 3 producer launch blocks at line ~4583 with comprehensive doc-comments). Verification: `SQLX_OFFLINE=true cargo check -p ml --lib` clean (dev profile); `cargo test -p ml --test sp14_oracle_tests` 2/2 non-GPU pass (7 GPU tests ignored on RTX 3050 Ti host — they verify kernel correctness, which is unchanged by this commit). SP14 β-migration step 3 — wire collector-side aux-head forward + per-step label producer (2026-05-07): Step 3 of 5. Inserts the aux next-bar forward + per-step sign-label producer into the collector's per-rollout-step body, immediately after the captured forward graph completes (`cuStreamEndCapture` / `cuGraphLaunch` block at line 4444) and before the expected_q kernel + the SP14 EGF chain (step 4). Two launches per rollout step, both on `self.stream`: (1) `aux_sign_label_per_step_kernel` — new thin variant of `aux_sign_label_kernel`. The trajectory kernel writes the entire `[total]` ring at the end of `collect_experiences_gpu`; the per-step kernel writes only the current step's `[n_episodes]` slice using `bar = episode_starts[ep] + t` (host-scalar `t` passed as kernel arg). Same per-thread O(1) map: read `targets[bar*6+2]` (raw_close column) at `bar` and `bar+lookahead`, classify by strict greater-than. Skip sentinel -1 fires when the lookahead window runs past `total_bars`; the CE / dir-acc / q_disagreement consumers all mask -1 rows out of denominators. Lookahead matches the trajectory kernel's `config.hindsight_lookahead.max(1)` so both encode the same K=2 "up vs not-up" boundary. Pure GPU map; no atomicAdd; per `feedback_no_atomicadd.md`. (2) `AuxHeadsForwardOps::forward_next_bar` — same orchestrator method the trainer's `aux_heads_forward` (gpu_dqn_trainer.rs:17208) uses, but bound to `self.stream` and operating on `n_episodes` rows of `exp_h_s2_f32` (the rollout-time h_s2). Param tensor pointers resolved via the existing `f32_weight_ptrs_from_base(self.trainer_params_ptr, &self.param_sizes)` path used elsewhere in the collector — same indices [119..123) the trainer uses (`nb_w1`, `nb_b1`, `nb_w2`, `nb_b2`). Writes hidden + logits + softmax tiles; the `exp_aux_nb_softmax_buf [n_episodes, 2]` is the production output for step 4's SP14 EGF chain. **Cold-start gate**: the entire block is gated on `self.trainer_params_ptr != 0`. The collector's constructor sets `trainer_params_ptr: 0` initially, then the trainer's `set_trainer_params_ptr()` (called after `FusedTrainingCtx::new`) wires the real pointer. Without that wire `f32_weight_ptrs_from_base` would return offsets from a NULL base and the aux forward would dereference garbage. Same defensive-NULL pattern the existing `forward_online_f32` call at line 4160 uses (the captured graph itself fails to capture if `trainer_params_ptr == 0`, so the test scaffold path that doesn't allocate params doesn't reach this block in production code paths either). **Placement**: AFTER captured graph end (`cuStreamEndCapture` for capture path, `cuGraphLaunch` for replay path) and BEFORE expected_q. The aux forward reads `exp_h_s2_f32` populated by `forward_online_f32` inside the captured graph; same-stream serial ordering within the per-step body suffices (no event sync needed). Per `pearl_no_host_branches_in_captured_graph` the launches are OUTSIDE the captured graph — the `f32_weight_ptrs_from_base` call returns a fixed `[u64; NUM_WEIGHT_TENSORS]` array post-capture, the host-scalar `t` is passed as kernel arg outside capture, and the cudarc `launch_builder` calls don't allocate inside any active capture (the captured `forward` graph already ended). Step 4 will wire the SP14 EGF producers (3 launches) AFTER this block but in the same per-step body. Files added: `crates/ml/src/cuda_pipeline/aux_sign_label_per_step_kernel.cu` (per-thread O(1) map kernel, +66 lines). Files modified: `crates/ml/build.rs` (+8 lines: register new cubin); `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` (+1 struct field for the new kernel handle, +1 cubin static include, +14 lines for the load in `new()`, +1 wire in struct literal, +90 lines for the per-step launch block at line ~4445). Verification: `SQLX_OFFLINE=true cargo check -p ml --lib` clean (dev profile); `cargo test -p ml --test sp14_oracle_tests` 2/2 non-GPU pass.