gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(sp14-β): remove training-time EGF launches from submit_aux_ops — collector path is canonical

Browse Source 
Atomic step 5 of β migration. SP14 EGF producer chain now fires
ONLY from the experience collector (commit c691bd381).
Training-time launches removed; the collector-native chain is the
single production caller.

Trainer's pub(crate) launcher methods retained (no Rust dead-code
warnings on pub(crate) — atomic-rollback potential preserved).
Oracle tests in sp14_oracle_tests.rs exercise the SAME kernels
directly via load_cubin / load_function, not through these launchers
— deletion of the launchers would not affect oracle coverage. They
stay for the possibility that a future curriculum stage reverses
the rollout-only signal-quality assumption.

gradient_hack_detect (per-epoch circuit breaker) unchanged —
lockout-counter decrement is one-per-epoch by design.

Verification: cargo check clean; sp14_oracle_tests 2/2 non-GPU pass
(7 GPU tests ignored on RTX 3050 Ti host); L40S smoke validation
pending.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-07 21:58:06 +02:00
parent c691bd381a
commit c260dca8bd
2 changed files with 36 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files

99
crates/ml/src/trainers/dqn/fused_training.rs
View File

@@ -1971,73 +1971,42 @@ impl FusedTrainingCtx {
self.trainer.populate_q_out(self.batch_size)
.map_err(|e| anyhow::anyhow!("IQL populate_q_out: {e}"))?;
// SP14 Layer B Task B.11 (2026-05-07 cadence fix): per-step EGF
// producer chain. The original B.11 wire-up (commit 857722e77)
// placed these in `process_epoch_boundary` — that ran them ONCE
// per epoch instead of per training step, leaving every consumer
// step inside the epoch reading ISV[393]=ALPHA_GRAD_SMOOTHED
// values that were updated up to (steps_per_epoch - 1) steps ago.
// The L40S `train-v8ztm` 10-epoch run showed α_smoothed=0.0002
// (vs. expected ~0.5 in steady state) — exactly the symptom of
// a producer chain firing < 1% as often as the consumer.
//
// Moving them here puts the launches inside `aux_child` graph
// capture, so each parent-graph replay (i.e. each training step)
// re-fires the EGF chain. The kernels are graph-safe — pure
// device-side state machines reading only ISV slots and the
// captured-graph-resident `aux_nb_softmax_buf` + `q_out_buf`,
// no host branches per `pearl_no_host_branches_in_captured_graph`.
//
// Order is load-bearing (consumer chain):
// 1. populate_q_out (existing, line above) writes q_out_buf
// from this step's online forward logits.
// 2. launch_sp13_aux_dir_metrics reduces aux_nb_softmax_buf
// → ISV[AUX_DIR_ACC_SHORT_EMA=373] / ISV[*_LONG_EMA=374]
// / ISV[AUX_DIR_PREDICTION=375]. Moved per-step from
// process_epoch_boundary because slot 373/374 are
// precondition reads for launch_sp14_alpha_grad_compute
// below (alpha_grad's kernel header explicitly requires
// sp13 to have fired earlier in the SAME step). Leaving
// sp13 per-epoch while making alpha_grad per-step would
// have alpha_grad reading aux_dir_acc EMAs from
// (steps_per_epoch - 1) steps ago — same staleness
// defeat as the original wire-up bug. Atomic dependency
// migration per `feedback_no_partial_refactor`.
// 3. launch_sp14_q_disagreement_update reads aux_nb_softmax
// (K=2) + q_out_buf direction-Q slice (K=4) and writes
// ISV[Q_DISAGREEMENT_SHORT/LONG/VAR_EMA = 383/384/389].
// 4. launch_sp14_alpha_grad_compute consumes #2's slots
// 373/374 + #3's slot 383/389 + the rest of the EGF
// driver block, runs the Schmitt-trigger Gate 1 +
// adaptive k + adaptive β state machine, writes
// ISV[ALPHA_GRAD_RAW=392] / ISV[ALPHA_GRAD_SMOOTHED=393]
// and the supporting bookkeeping slots [385..395].
// The captured backward consumer
// (launch_sp14_scale_wire_col inside
// launch_cublas_backward_to) reads slot 393 within the
// SAME captured replay — fresh per step, no one-step
// lag like the prior pattern.
//
// α_short = 0.3, α_long = 0.05, α_var = 0.05 per spec
// §2581-2585. The variance-driven k_aux/k_q in
// alpha_grad_compute_kernel handles warmup adaptively (no
// explicit warmup_gate per `feedback_isv_for_adaptive_bounds`,
// c0fc28e45). var_aux producer is inline in
// alpha_grad_compute_kernel (Welford on aux_short vs aux_long).
// SP14 β-migration step 5 (2026-05-07): EGF producer chain
// MOVED to `gpu_experience_collector::collect_experiences_gpu`
// per-rollout-step body. The training-time chain that used
// to live here (`launch_sp13_aux_dir_metrics` +
// `launch_sp14_q_disagreement_update` +
// `launch_sp14_alpha_grad_compute`, commits 857722e77 →
// 200f05fce per-step migration → 9d0c124ce empty-batch gate)
// is RETIRED from production: even with the per-step cadence
// and the empty-batch gate, replay-batch Q-direction picks
// are dominated by Hold/Flat (the natural argmax distribution
// of the converged 4-way head) so the EGF producers see
// total_cnt=0 every step and the gate keeps EMAs at zero.
// The collector-native β path (commit c691bd381) reads
// rollout-time q_values where Thompson sampling forces
// Short/Long picks every step — that's the genuine source
// of aux↔Q disagreement signal.
//
// The per-epoch `launch_sp14_gradient_hack_detect` circuit
// breaker stays at process_epoch_boundary — its lockout
// counter decrement is one-per-epoch by design.
self.trainer.launch_sp13_aux_dir_metrics()
.map_err(|e| anyhow::anyhow!("SP14 B.11 sp13_aux_dir_metrics: {e}"))?;
let alpha_short_qd: f32 = 0.3;
let alpha_long_qd: f32 = 0.05;
let alpha_var: f32 = 0.05;
self.trainer.launch_sp14_q_disagreement_update(
alpha_short_qd, alpha_long_qd, alpha_var,
).map_err(|e| anyhow::anyhow!("SP14 B.11 q_disagreement_update: {e}"))?;
self.trainer.launch_sp14_alpha_grad_compute(alpha_var)
.map_err(|e| anyhow::anyhow!("SP14 B.11 alpha_grad_compute: {e}"))?;
// breaker stays at process_epoch_boundary unchanged — its
// lockout counter decrement is one-per-epoch by design.
//
// Trainer launcher methods at gpu_dqn_trainer.rs:8671 / 8733
// / 15270 stay defined as `pub(crate)` (no compiler warnings
// — Rust doesn't warn unused crate-public methods). Oracle
// tests in `crates/ml/tests/sp14_oracle_tests.rs` exercise
// the SAME kernels directly via `load_cubin` /
// `load_function`, not through these launchers — so deletion
// of the launchers would not affect oracle coverage. They
// are retained for atomic-rollback potential and for the
// possibility that the trainer-time path becomes useful
// again if a future curriculum stage reverses the rollout-
// only signal-quality assumption. Future cleanup: delete
// them along with their associated CudaFunction fields if
// the rollout-only path is validated stable across a full
// 50-epoch L40S run (tracked but not blocking the β
// migration close-out).
// 2026-05-07 cadence-fix continuation (after SP14 B.11 commit
// 200f05fce): four MORE producers whose docstrings explicitly

2
docs/dqn-wire-up-audit.md
View File

@@ -2,6 +2,8 @@
**Status:** Populated during Plan 1 Task 6 (A.5 orphan audit). Updated on every commit per Invariant 7.
SP14 β-migration step 5 — remove training-time EGF launches from `submit_aux_ops` (2026-05-07): Final atomic step of the 5-commit β migration. Deletes the 3 trainer-time launches at `crates/ml/src/trainers/dqn/fused_training.rs:2031-2040` (`launch_sp13_aux_dir_metrics` + `launch_sp14_q_disagreement_update` + `launch_sp14_alpha_grad_compute`) — these were the producers introduced by SP14 B.11 commit `200f05fce` (per-step migration) and stabilised by commit `9d0c124ce` (empty-batch decay gate). Even with both fixes the trainer-time path could only fire against replay-batch Q-direction picks dominated by Hold/Flat (the natural argmax distribution of the converged 4-way head), so `total_cnt=0` every step and the gate kept EMAs at zero. The collector-native β path (commit c691bd381 / step 4) reads rollout-time q_values where Thompson sampling forces Short/Long picks every step — that's the genuine source of aux↔Q disagreement signal. The collector-native chain is now the SINGLE production caller. **Trainer launcher methods preserved**: `launch_sp14_q_disagreement_update` (line 8671), `launch_sp14_alpha_grad_compute` (line 8733), `launch_sp13_aux_dir_metrics` (line 15270) and the 3 sub-launchers (`launch_aux_dir_acc_reduce`, `launch_apply_fixed_alpha_ema`, `launch_aux_pred_to_isv_tanh`) stay defined as `pub(crate)` (no Rust compiler warnings — `pub(crate)` methods don't trigger dead-code warnings). Oracle tests in `crates/ml/tests/sp14_oracle_tests.rs` exercise the SAME kernels directly via `load_cubin` / `load_function`, not through these launchers — verified by `grep -n "launch_sp14|launch_sp13_aux" tests/sp14_oracle_tests.rs` returning 0 hits — so deletion of the launchers would not affect oracle coverage. They are retained for atomic-rollback potential and the possibility that the trainer-time path becomes useful again if a future curriculum stage reverses the rollout-only signal-quality assumption. **`launch_sp14_gradient_hack_detect` (per-epoch circuit breaker) unchanged** — the lockout-counter decrement is one-per-epoch by design, stays at `process_epoch_boundary`. The replacement comment at the deletion site in `fused_training.rs` documents the migration trail (β-migration step 5 → c691bd381 → 9d0c124ce) so a future archeologist can reconstruct the chain. **Verification**: `SQLX_OFFLINE=true cargo check -p ml --lib` clean (dev profile, only pre-existing unused-var warnings unrelated to this change); `cargo test -p ml --test sp14_oracle_tests` 2/2 non-GPU pass (7 GPU tests ignored on RTX 3050 Ti host — verify kernel correctness, unchanged); `grep -rn "trainer.launch_sp14|trainer.launch_sp13_aux" crates/ml/src/` returns 0 production callers (the 3 calls in fused_training.rs are deleted). L40S smoke validation pending — expect EGF chain to drive non-zero ISV[383/384/389/393] across the full curriculum, unblocking Gate 1 from its closed-forever state. Files modified: `crates/ml/src/trainers/dqn/fused_training.rs` (−10 / +14 lines: deleted 3 launches, replaced 60-line audit-comment block with 12-line redirect comment).
SP14 β-migration step 4 — wire collector-native SP14 EGF producer chain after aux forward (2026-05-07): Step 4 of 5. Inserts the 3 SP14/SP13-EGF producer launches into the collector's per-rollout-step body, immediately after the `expected_q_kernel` (which produces `q_values`) and BEFORE the IQR/ensemble/noise SAXPY blocks (so the SP14 q_disagreement consumer sees noise-free `q_values` matching the trainer's `q_out_buf` semantic). Producer order matches the trainer's `submit_aux_ops` chain (preserved per `feedback_no_partial_refactor.md`): (1) SP13 dir-acc reduce + 2 fixed-α EMAs (α=0.3 short → ISV[373], α=0.05 long → ISV[374]) + aux_pred → ISV[375] tanh — same 4 launches the trainer's `launch_sp13_aux_dir_metrics` orchestrates inline (gpu_dqn_trainer.rs:15270), reading the collector's `exp_aux_nb_softmax_buf [n_episodes, 2]` (from step 3) + `exp_aux_nb_label_buf [n_episodes]` (from step 3's per-step label producer) + writing to the shared ISV via `isv_signals_dev_ptr`. Sentinel 0.5 = random-guessing baseline per `pearl_first_observation_bootstrap`. (2) SP14 `q_disagreement_update_kernel` — reads `exp_aux_nb_softmax_buf [n_episodes, K=2]` + collector's `q_values [n_episodes, q_stride=13]` (post-`expected_q_kernel` output); writes ISV[383/384/389] (q_dis_short, q_dis_long, var_q EMAs). q_stride=13 matches the trainer's q_out_buf stride; the kernel's argmax-over-K_DIR=4 finds the direction-head pick from the first 4 columns of each row. Single block, 256 threads, smem = 2 × 256 × f32. (3) SP14 `alpha_grad_compute_kernel` — pure ISV-state state machine reading the q_dis EMAs just written above (same-stream serial ordering enforces the dependency) plus ISV[372..395) drivers; writes ISV[392/393] (α_grad raw + smoothed) plus the Schmitt-trigger / k_q / k_aux state machine bookkeeping at ISV[385..391]. Single thread (we launch with 32 to mirror the B.4 oracle test launch shape). The kernel's empty-batch decay gate (commit 9d0c124ce) preserves prior EMA values when `total_cnt == 0` for a given step — same protection extends to the rollout path. **Gating**: `if self.isv_signals_dev_ptr != 0 && self.trainer_params_ptr != 0` — without ISV the EMAs have nowhere to write; without trainer params the aux forward in step 3 would have been skipped and the softmax tile would still be `alloc_zeros`. **No `seed_phase_active_cache` gate** here (unlike SP13 hold-rate which prices Hold-cost only post-seed): the EGF Schmitt-trigger Gate 1 needs to observe BOTH seed-phase scripted-policy actions and post-seed Q-policy actions to discriminate aux signal quality across the curriculum. **q_logits placement note**: the SP14 chain runs AFTER `expected_q_kernel` (line 4581) and BEFORE the IQR (3b) / ensemble (3c) / noise (3d) SAXPY bonuses — so `q_values` here is bit-equivalent to the trainer's `q_out_buf` semantic at the trainer's `launch_sp14_q_disagreement_update` call site. The IQR/ensemble bonuses below are exploration noise added for action selection, not gradient flow. Per `pearl_no_host_branches_in_captured_graph` the launches are OUTSIDE any captured graph (the captured `forward` graph has already ended). The trainer's existing private `launch_sp14_q_disagreement_update` / `launch_sp14_alpha_grad_compute` / `launch_sp13_aux_dir_metrics` methods at gpu_dqn_trainer.rs:8671 / 8733 / 15270 are still defined and still pass kernel correctness via the 7 sp14_oracle GPU tests — production has migrated to the collector path; step 5 will delete the now-unused `submit_aux_ops` invocations (commit `200f05fce` lines 2031-2040). Files modified: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` (+200 lines: 3 producer launch blocks at line ~4583 with comprehensive doc-comments). Verification: `SQLX_OFFLINE=true cargo check -p ml --lib` clean (dev profile); `cargo test -p ml --test sp14_oracle_tests` 2/2 non-GPU pass (7 GPU tests ignored on RTX 3050 Ti host — they verify kernel correctness, which is unchanged by this commit).
SP14 β-migration step 3 — wire collector-side aux-head forward + per-step label producer (2026-05-07): Step 3 of 5. Inserts the aux next-bar forward + per-step sign-label producer into the collector's per-rollout-step body, immediately after the captured forward graph completes (`cuStreamEndCapture` / `cuGraphLaunch` block at line 4444) and before the expected_q kernel + the SP14 EGF chain (step 4). Two launches per rollout step, both on `self.stream`: (1) `aux_sign_label_per_step_kernel` — new thin variant of `aux_sign_label_kernel`. The trajectory kernel writes the entire `[total]` ring at the end of `collect_experiences_gpu`; the per-step kernel writes only the current step's `[n_episodes]` slice using `bar = episode_starts[ep] + t` (host-scalar `t` passed as kernel arg). Same per-thread O(1) map: read `targets[bar*6+2]` (raw_close column) at `bar` and `bar+lookahead`, classify by strict greater-than. Skip sentinel -1 fires when the lookahead window runs past `total_bars`; the CE / dir-acc / q_disagreement consumers all mask -1 rows out of denominators. Lookahead matches the trajectory kernel's `config.hindsight_lookahead.max(1)` so both encode the same K=2 "up vs not-up" boundary. Pure GPU map; no atomicAdd; per `feedback_no_atomicadd.md`. (2) `AuxHeadsForwardOps::forward_next_bar` — same orchestrator method the trainer's `aux_heads_forward` (gpu_dqn_trainer.rs:17208) uses, but bound to `self.stream` and operating on `n_episodes` rows of `exp_h_s2_f32` (the rollout-time h_s2). Param tensor pointers resolved via the existing `f32_weight_ptrs_from_base(self.trainer_params_ptr, &self.param_sizes)` path used elsewhere in the collector — same indices [119..123) the trainer uses (`nb_w1`, `nb_b1`, `nb_w2`, `nb_b2`). Writes hidden + logits + softmax tiles; the `exp_aux_nb_softmax_buf [n_episodes, 2]` is the production output for step 4's SP14 EGF chain. **Cold-start gate**: the entire block is gated on `self.trainer_params_ptr != 0`. The collector's constructor sets `trainer_params_ptr: 0` initially, then the trainer's `set_trainer_params_ptr()` (called after `FusedTrainingCtx::new`) wires the real pointer. Without that wire `f32_weight_ptrs_from_base` would return offsets from a NULL base and the aux forward would dereference garbage. Same defensive-NULL pattern the existing `forward_online_f32` call at line 4160 uses (the captured graph itself fails to capture if `trainer_params_ptr == 0`, so the test scaffold path that doesn't allocate params doesn't reach this block in production code paths either). **Placement**: AFTER captured graph end (`cuStreamEndCapture` for capture path, `cuGraphLaunch` for replay path) and BEFORE expected_q. The aux forward reads `exp_h_s2_f32` populated by `forward_online_f32` inside the captured graph; same-stream serial ordering within the per-step body suffices (no event sync needed). Per `pearl_no_host_branches_in_captured_graph` the launches are OUTSIDE the captured graph — the `f32_weight_ptrs_from_base` call returns a fixed `[u64; NUM_WEIGHT_TENSORS]` array post-capture, the host-scalar `t` is passed as kernel arg outside capture, and the cudarc `launch_builder` calls don't allocate inside any active capture (the captured `forward` graph already ended). Step 4 will wire the SP14 EGF producers (3 launches) AFTER this block but in the same per-step body. Files added: `crates/ml/src/cuda_pipeline/aux_sign_label_per_step_kernel.cu` (per-thread O(1) map kernel, +66 lines). Files modified: `crates/ml/build.rs` (+8 lines: register new cubin); `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` (+1 struct field for the new kernel handle, +1 cubin static include, +14 lines for the load in `new()`, +1 wire in struct literal, +90 lines for the per-step launch block at line ~4445). Verification: `SQLX_OFFLINE=true cargo check -p ml --lib` clean (dev profile); `cargo test -p ml --test sp14_oracle_tests` 2/2 non-GPU pass.