gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(sp14-B.11): move EGF producer chain into per-step training loop — fixes per-epoch staleness

Browse Source 
Root cause from train-v8ztm 10-ep validation (commit 1396b62ec): HEALTH_DIAG
showed alpha_smoothed=0.0002 (vs ~0.5 expected steady-state), gate1=closed,
var_aux:var_q ratio 290:1 — symptoms of an EGF producer chain firing < 1%
as often as the consumer.

The original B.11 wire-up (commit 857722e77) placed `launch_sp14_q_disagreement_update`,
`launch_sp14_alpha_grad_compute`, and the prerequisite `launch_sp13_aux_dir_metrics`
in `process_epoch_boundary` — which runs ONCE per epoch (single call site at
training_loop.rs:780, called from the per-epoch loop, not from the per-step
loop in `run_training_steps_slices`). The captured backward consumer
`launch_sp14_scale_wire_col` (inside launch_cublas_backward_to, replays every
training step via parent graph) reads ISV[ALPHA_GRAD_SMOOTHED=393] per step,
but the producer was firing only at epoch boundary — every step inside the
epoch observed (steps_per_epoch − 1)-step-stale alpha values, with the EMA
chain barely accumulating past sentinel between rare per-epoch updates. The
plan §2550 explicitly specifies per-step cadence; the existing wire violated
the plan.

Fix (atomic, graph-capture-safe):
- MOVED launch_sp13_aux_dir_metrics, launch_sp14_q_disagreement_update,
  launch_sp14_alpha_grad_compute from process_epoch_boundary into
  fused_training.rs:submit_aux_ops, immediately after populate_q_out.
  submit_aux_ops captures into the aux_child sub-graph, so each parent-graph
  replay re-fires the full producer chain — restoring per-step cadence.
- launch_sp13_aux_dir_metrics had to migrate alongside the SP14 launches:
  alpha_grad_compute_kernel consumes its outputs (ISV[373/374]); leaving
  sp13 per-epoch while moving SP14 per-step would re-introduce the same
  staleness bug for aux_dir_acc reads (atomic dependency migration per
  feedback_no_partial_refactor).
- Per-epoch launch_sp14_gradient_hack_detect circuit breaker stays in
  process_epoch_boundary — its lockout decrement IS one-per-epoch by design.
- Forward consumers (6 launch_sp14_dir_concat_qaux sites) and backward
  consumers (2 launch_sp14_scale_wire_col sites) unchanged — they read the
  same ISV[393], but now see live per-step values instead of per-epoch
  staleness.

Verification:
- cargo check -p ml --tests --all-targets clean (no errors, no new warnings).
- All 6 SP14 oracle GPU tests pass (alpha_grad_adaptive_beta,
  alpha_grad_schmitt_hysteresis, dir_concat_qaux_correct,
  gradient_hack_circuit_breaker_fires, q_disagreement_all_hold_no_contribution,
  q_disagreement_k4_k2_mapping).
- HEALTH_DIAG validation pending L40S re-dispatch — expect alpha_smoothed
  to track real EGF-driven values (~0.5 in steady state).

Invariants:
- pearl_no_host_branches_in_captured_graph (kernels are pure GPU state
  machines using launch_builder + pre-loaded CudaFunction; no per-call
  load_cubin)
- feedback_no_partial_refactor (sp13 + 2 SP14 launches migrated atomically)
- feedback_wire_everything_up (all 3 producers now production hot-path,
  re-fire on every parent-graph replay)
- feedback_isv_for_adaptive_bounds (no warmup_gate parameter — variance-
  driven k_aux/k_q in alpha_grad_compute_kernel handles cold-start
  adaptively, per c0fc28e45)

Refs: train-v8ztm trajectory analysis 2026-05-07T15:59:49 HEALTH_DIAG[10]
showed dir_entropy=0.6545 kill-fast breach with model converging to 64%
Hold + 84% Quarter magnitude — exactly the pathology B.11 was designed
to prevent by routing aux's directional signal into Q.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-07 18:35:03 +02:00
parent 1396b62ec6
commit 200f05fcef
3 changed files with 96 additions and 105 deletions
Download Patch File Download Diff File Expand all files Collapse all files

68
crates/ml/src/trainers/dqn/fused_training.rs
View File

@@ -1971,6 +1971,74 @@ impl FusedTrainingCtx {
self.trainer.populate_q_out(self.batch_size) self.trainer.populate_q_out(self.batch_size)
.map_err(|e| anyhow::anyhow!("IQL populate_q_out: {e}"))?; .map_err(|e| anyhow::anyhow!("IQL populate_q_out: {e}"))?;
// SP14 Layer B Task B.11 (2026-05-07 cadence fix): per-step EGF
// producer chain. The original B.11 wire-up (commit 857722e77)
// placed these in `process_epoch_boundary` — that ran them ONCE
// per epoch instead of per training step, leaving every consumer
// step inside the epoch reading ISV[393]=ALPHA_GRAD_SMOOTHED
// values that were updated up to (steps_per_epoch - 1) steps ago.
// The L40S `train-v8ztm` 10-epoch run showed α_smoothed=0.0002
// (vs. expected ~0.5 in steady state) — exactly the symptom of
// a producer chain firing < 1% as often as the consumer.
//
// Moving them here puts the launches inside `aux_child` graph
// capture, so each parent-graph replay (i.e. each training step)
// re-fires the EGF chain. The kernels are graph-safe — pure
// device-side state machines reading only ISV slots and the
// captured-graph-resident `aux_nb_softmax_buf` + `q_out_buf`,
// no host branches per `pearl_no_host_branches_in_captured_graph`.
//
// Order is load-bearing (consumer chain):
// 1. populate_q_out (existing, line above) writes q_out_buf
// from this step's online forward logits.
// 2. launch_sp13_aux_dir_metrics reduces aux_nb_softmax_buf
// → ISV[AUX_DIR_ACC_SHORT_EMA=373] / ISV[*_LONG_EMA=374]
// / ISV[AUX_DIR_PREDICTION=375]. Moved per-step from
// process_epoch_boundary because slot 373/374 are
// precondition reads for launch_sp14_alpha_grad_compute
// below (alpha_grad's kernel header explicitly requires
// sp13 to have fired earlier in the SAME step). Leaving
// sp13 per-epoch while making alpha_grad per-step would
// have alpha_grad reading aux_dir_acc EMAs from
// (steps_per_epoch - 1) steps ago — same staleness
// defeat as the original wire-up bug. Atomic dependency
// migration per `feedback_no_partial_refactor`.
// 3. launch_sp14_q_disagreement_update reads aux_nb_softmax
// (K=2) + q_out_buf direction-Q slice (K=4) and writes
// ISV[Q_DISAGREEMENT_SHORT/LONG/VAR_EMA = 383/384/389].
// 4. launch_sp14_alpha_grad_compute consumes #2's slots
// 373/374 + #3's slot 383/389 + the rest of the EGF
// driver block, runs the Schmitt-trigger Gate 1 +
// adaptive k + adaptive β state machine, writes
// ISV[ALPHA_GRAD_RAW=392] / ISV[ALPHA_GRAD_SMOOTHED=393]
// and the supporting bookkeeping slots [385..395].
// The captured backward consumer
// (launch_sp14_scale_wire_col inside
// launch_cublas_backward_to) reads slot 393 within the
// SAME captured replay — fresh per step, no one-step
// lag like the prior pattern.
//
// α_short = 0.3, α_long = 0.05, α_var = 0.05 per spec
// §2581-2585. The variance-driven k_aux/k_q in
// alpha_grad_compute_kernel handles warmup adaptively (no
// explicit warmup_gate per `feedback_isv_for_adaptive_bounds`,
// c0fc28e45). var_aux producer is inline in
// alpha_grad_compute_kernel (Welford on aux_short vs aux_long).
//
// The per-epoch `launch_sp14_gradient_hack_detect` circuit
// breaker stays at process_epoch_boundary — its lockout
// counter decrement is one-per-epoch by design.
self.trainer.launch_sp13_aux_dir_metrics()
.map_err(|e| anyhow::anyhow!("SP14 B.11 sp13_aux_dir_metrics: {e}"))?;
let alpha_short_qd: f32 = 0.3;
let alpha_long_qd: f32 = 0.05;
let alpha_var: f32 = 0.05;
self.trainer.launch_sp14_q_disagreement_update(
alpha_short_qd, alpha_long_qd, alpha_var,
).map_err(|e| anyhow::anyhow!("SP14 B.11 q_disagreement_update: {e}"))?;
self.trainer.launch_sp14_alpha_grad_compute(alpha_var)
.map_err(|e| anyhow::anyhow!("SP14 B.11 alpha_grad_compute: {e}"))?;
// 2. Gather Q(s, a_taken) for both IQL trainers. // 2. Gather Q(s, a_taken) for both IQL trainers.
let q_out = self.trainer.q_out_buf(); let q_out = self.trainer.q_out_buf();
let actions = self.trainer.actions_buf(); let actions = self.trainer.actions_buf();

131
crates/ml/src/trainers/dqn/trainer/training_loop.rs
View File

@@ -4209,114 +4209,35 @@ impl DQNTrainer {
} }
} }
// SP13 Phase 0a P0a.T4 (2026-05-04): per-step aux // SP13 P0a.T4 + SP14 B.11 (2026-05-07 cadence fix):
// directional-accuracy producer chain. SP13 B1.1a // `launch_sp13_aux_dir_metrics`, `launch_sp14_q_disagreement_update`,
// (2026-05-05) flipped the upstream head from K=1 MSE // and `launch_sp14_alpha_grad_compute` MOVED from this
// regression to K=2 softmax CE classification — this // per-epoch block into `submit_aux_ops` (per-step, captured
// orchestrator now reads the captured-forward-graph // into `aux_child` graph), right after `populate_q_out`.
// `aux_nb_softmax_buf [B, K]` + `aux_nb_label_buf [B] i32`
// (just populated above by the same per-step block that
// fires `launch_aux_heads_loss_ema`), reduces them into
// the trainer's mapped-pinned `aux_dir_acc_buf [6]`
// (was [3]; B1.1a added n_down/n_up/n_skip for
// HEALTH_DIAG observability), then chains:
// - fixed-α EMA (α=0.3, sentinel 0.5) → ISV[373]
// - fixed-α EMA (α=0.05, sentinel 0.5) → ISV[374]
// - mean(softmax[1]-softmax[0]) → ISV[375]
// (structural [-1, +1] bound; no runtime tanh)
// All on the same stream as the producer, so no host
// sync is required between captured graph and consumer.
// Producer-only — no consumer kernel reads slots 373/374/
// 375 in this commit; Phase 0b wires the controller and
// the Q-head input layer that reads slot 375. Same
// post-cascade-fix invariant as `launch_aux_heads_loss_ema`:
// ISV producer launches MUST run BEFORE the HEALTH_DIAG
// line emit further down.
// //
// B1.1a known degraded state: labels stay zero-init // Why: `process_epoch_boundary` runs ONCE per epoch (call
// (`aux_nb_label_buf` is `alloc_zeros`-i32 every step; // site is the per-epoch loop body at line ~780, not the
// every sample receives label 0 = "down") until B1.1b // per-step loop in `run_training_steps_slices`). Placing
// lands the producer kernel that fills real -1/0/1 from // these producer launches here meant ISV[383/384/389/393]
// the price trajectory. Until then, the model converges // updated < 1% as often as the captured backward consumer
// on "predict class 0 (down) everywhere" — the cascade // reads them — every training step inside the epoch
// is internally consistent (every consumer migrated // observed (steps_per_epoch - 1)-step-stale α_grad_smoothed.
// atomically per `feedback_no_partial_refactor`); the // L40S `train-v8ztm` 10-epoch run showed the symptom
// labels are placeholder. // exactly: α_smoothed=0.0002 (vs ~0.5 expected steady-state)
if let Some(ref fused) = self.fused_ctx { // with var_aux:var_q ratio 290:1 — the EGF chain was barely
if let Err(e) = fused.trainer().launch_sp13_aux_dir_metrics() { // firing.
tracing::warn!("SP13 P0a.T4 aux_dir_metrics launch failed: {e}");
}
}
// SP14 Layer B Task B.11 (2026-05-05): EGF producer chain.
// //
// Three launches; the first two run per training step here, the // The kernels are pure GPU state machines reading ISV +
// third (gradient_hack_detect) fires once per epoch in // captured-graph buffers (`aux_nb_softmax_buf`, `q_out_buf`)
// `process_epoch_boundary`. Order is load-bearing: // — graph-capture safe per `pearl_no_host_branches_in_captured_graph`,
// and `submit_aux_ops` is captured into `aux_child` (see
// `capture_training_graph`). All three launches re-fire on
// every parent-graph replay, restoring the per-step cadence
// the SP14 plan §2550 specifies.
// //
// 1. q_disagreement_update_kernel — reads the captured-graph // The per-epoch `launch_sp14_gradient_hack_detect` circuit
// `aux_nb_softmax_buf [B, 2]` (aux head forward this step) // breaker stays in `process_epoch_boundary` below (its
// + `q_out_buf [B, 13]` (populated by the captured // lockout decrement IS one-per-epoch by design).
// `populate_q_out` in `submit_aux_ops`) and updates
// ISV[Q_DISAGREEMENT_SHORT/LONG_EMA = 383/384] +
// ISV[Q_DISAGREEMENT_VARIANCE_EMA = 389]. Same launch
// cadence as `launch_sp13_aux_dir_metrics` above —
// producer-only.
//
// 2. alpha_grad_compute_kernel — consumes #1's ISV[383] /
// ISV[389] outputs AND `launch_sp13_aux_dir_metrics`'s
// ISV[373] / ISV[374] outputs to compute the EGF gate
// ISV[ALPHA_GRAD_SMOOTHED = 393]. This is the kernel that
// activates the EGF wire — pre-B.11 the slot stayed at
// sentinel 0.0 so the backward wire-column scaling
// (B.10) effectively zeroed gradient flow. Post-B.11 the
// slot tracks the live gate value ∈ [0, 1] each step.
//
// The backward consumer (`launch_sp14_scale_wire_col` inside
// `launch_cublas_backward_to`) reads ISV[393] inside the
// CAPTURED graph. Because these two producer launches run
// OUTSIDE the captured graph (after `run_full_step` returns,
// identical position to `launch_sp13_aux_dir_metrics`), the
// backward consumer at training step N reads the value
// written by step N-1's producer chain (one-step lag). This
// is the same one-step-lag pattern as the forward wire
// (`launch_sp14_dir_concat_qaux` consumes the prior step's
// `aux_nb_softmax_buf`). Bootstrap effect: at step 0, ISV[393]
// is still 0.0 → wire column zeroed; from step 1 onward the
// gate value accrues.
//
// SP14 L1+L2 follow-up (2026-05-05): warmup_gate parameter
// and `fold_step_counter` ramp deleted per
// `feedback_isv_for_adaptive_bounds`. The variance-driven
// k_aux/k_q already provide warmup behavior — when var_aux
// and var_q EMAs are high (cold-start, EMAs still moving),
// k → K_MIN → flat sigmoid → gate ≈ 0.5; when EMAs settle,
// k → K_BASE → sharp sigmoid → gates respond correctly.
//
// Constants per plan §2581-2585:
// * α_short = 0.3 (fast EMA, ~3-step half-life)
// * α_long = 0.05 (slow EMA, ~14-step half-life)
// * α_var = 0.05 (Welford EMA blend; shared by var_q
// in this kernel and var_alpha + var_aux
// in alpha_grad_compute)
//
// var_aux producer gap (from B.4 status report) is now closed:
// alpha_grad_compute_kernel writes ISV[VAR_AUX = 388] inline
// (Welford on aux_short vs aux_long), so the adaptive k_aux
// branch is no longer degenerate at K_BASE_AUX = 20.
if let Some(ref fused) = self.fused_ctx {
let alpha_short_qd: f32 = 0.3;
let alpha_long_qd: f32 = 0.05;
let alpha_var: f32 = 0.05;
if let Err(e) = fused.trainer().launch_sp14_q_disagreement_update(
alpha_short_qd, alpha_long_qd, alpha_var,
) {
tracing::warn!("SP14 B.11 q_disagreement_update launch failed: {e}");
}
if let Err(e) = fused.trainer().launch_sp14_alpha_grad_compute(alpha_var) {
tracing::warn!("SP14 B.11 alpha_grad_compute launch failed: {e}");
}
}
// Phase 3 T3.5: MoE expert-utilisation EMA + gate entropy EMA. // Phase 3 T3.5: MoE expert-utilisation EMA + gate entropy EMA.
// Reads `moe_gate_softmax_buf [B, K]` (valid after the captured // Reads `moe_gate_softmax_buf [B, K]` (valid after the captured

2
docs/dqn-wire-up-audit.md
View File

@@ -2,6 +2,8 @@
**Status:** Populated during Plan 1 Task 6 (A.5 orphan audit). Updated on every commit per Invariant 7. **Status:** Populated during Plan 1 Task 6 (A.5 orphan audit). Updated on every commit per Invariant 7.
SP14 B.11 cadence fix — EGF producer chain MOVED to per-step in `submit_aux_ops` (2026-05-07): root-caused the L40S `train-v8ztm` 10-epoch run's `α_smoothed=0.0002` + `gate1=closed` + `var_aux:var_q = 290:1` symptoms (HEALTH_DIAG `pearl_egf_diag` at epoch 10) to the original B.11 wire-up (commit `857722e77`) placing the per-step launches in `process_epoch_boundary` instead of the per-step training body. `process_epoch_boundary` runs ONCE per epoch (single call site at `training_loop.rs:780`, called from the per-epoch `for epoch in 0..n_epochs` loop, NOT from the per-step `for _step in 0..num_training_steps` loop in `run_training_steps_slices`). The captured backward consumer `launch_sp14_scale_wire_col` (inside `launch_cublas_backward_to`, called via the parent graph's replay every training step) reads `ISV[ALPHA_GRAD_SMOOTHED=393]` per step but the PRODUCER chain was firing only at epoch boundary — every step inside the epoch observed (steps_per_epoch − 1)-step-stale `α_grad_smoothed`, with the EMA chain barely accumulating past sentinel between rare per-epoch updates. Fix (atomic): MOVED `launch_sp13_aux_dir_metrics`, `launch_sp14_q_disagreement_update`, `launch_sp14_alpha_grad_compute` from `training_loop.rs:process_epoch_boundary` (lines 4245-4319) into `fused_training.rs:submit_aux_ops` immediately after `populate_q_out` (line ~1972). `submit_aux_ops` is captured into the `aux_child` sub-graph (see `capture_training_graph`), so each parent-graph replay re-fires the full producer chain — restoring the per-step cadence the SP14 plan §2550 specifies. `launch_sp13_aux_dir_metrics` had to migrate alongside the SP14 launches because `alpha_grad_compute_kernel` consumes its outputs (`ISV[373/374]`); leaving sp13 per-epoch while SP14 went per-step would have re-introduced the same staleness bug for the aux-dir-acc reads (atomic dependency migration per `feedback_no_partial_refactor`). The per-epoch `launch_sp14_gradient_hack_detect` circuit breaker stays in `process_epoch_boundary` — its lockout decrement is one-per-epoch by design. Forward consumers (the 6 `launch_sp14_dir_concat_qaux` sites at lines 12120, 25666, 25735, 27788, 27978, 29076 in `gpu_dqn_trainer.rs`) and backward consumers (the 2 `launch_sp14_scale_wire_col` sites at lines 12171, 29126) are unchanged — they continue reading the same ISV slot 393, but now see a value that updates every step instead of every epoch. Build clean (`cargo check -p ml --tests --all-targets`); all 6 SP14 oracle GPU tests pass (`alpha_grad_adaptive_beta`, `alpha_grad_schmitt_hysteresis`, `dir_concat_qaux_correct`, `gradient_hack_circuit_breaker_fires`, `q_disagreement_all_hold_no_contribution`, `q_disagreement_k4_k2_mapping`). HEALTH_DIAG validation pending L40S re-dispatch — expect `α_smoothed` to track real EGF-driven values (~0.5 in steady state) instead of the bootstrap-near-zero observed in `train-v8ztm`. Invariants honored: `pearl_no_host_branches_in_captured_graph` (kernels are pure GPU state machines, capture-safe — verified all use `launch_builder` with primitive args, no per-call `load_cubin`); `feedback_no_partial_refactor` (sp13 + 2 SP14 launches migrated atomically); `feedback_wire_everything_up` (all 3 producers fire on every parent graph replay; the previously-stale-cadence EGF wire is now production hot-path); `feedback_isv_for_adaptive_bounds` (no `warmup_gate` parameter — variance-driven k_aux/k_q in alpha_grad_compute_kernel handles cold-start adaptively, per c0fc28e45). Files modified: `crates/ml/src/trainers/dqn/fused_training.rs` (+74 lines: 3 launches + comment block in `submit_aux_ops`); `crates/ml/src/trainers/dqn/trainer/training_loop.rs` (−109 / +24 lines: deletion of stale per-epoch launches + replacement audit comment).
SP15 Wave 5 follow-up — pre-load sp15_baseline + cost_net cubins (2026-05-07): root-caused L40S workflow `train-xggfc` (commit `5d63762ab`) hyperopt-trial CUDA failure to the same per-call `load_cubin` + `load_function` anti-pattern that Wave 4.1b's bn_tanh_concat_dd-fix (commit `5d63762ab`, see entry below) addressed for the trainer's bottleneck-concat path. The 5 SP15 evaluation launchers in `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` (`launch_sp15_cost_net_sharpe` at ~787, `launch_sp15_baseline_buyhold` at ~1171, `launch_sp15_baseline_hold_only` at ~1218, `launch_sp15_baseline_naive_momentum` at ~1263, `launch_sp15_baseline_naive_reversion` at ~1308) were resolving `cost_net_sharpe_kernel` / `baseline_*_kernel` symbols via `stream.context().load_cubin(...)` + `module.load_function(...)` ON EVERY CALL inside `GpuBacktestEvaluator`'s per-window eval hot loop (`gpu_backtest_evaluator.rs:2877..2922`, `for w in 0..self.n_windows`). Pattern works in single-pass train-best context (smoke `train-9bcwm` at `5d63762ab` ran the SAME launcher chain successfully through 2 folds × 5 epochs of train-best evaluation), but fails when re-entered from a hyperopt trial's child stream context: `train-xggfc` ran 8 hyperopt search epochs successfully, then started 20 hyperopt trial trainings; trial 1 failed at `load sp15_baseline_kernels cubin: DriverError(CUDA_ERROR_ILLEGAL_ADDRESS, "an illegal memory access was encountered")`; after trial 1's CUDA context was poisoned, trials 2-20 cascade-failed at `Fork CUDA stream for trial: ILLEGAL_ADDRESS`. Fix (atomic, mirrors `5d63762ab` precedent): added 5 `CudaFunction` fields on `GpuBacktestEvaluator` (`sp15_cost_net_sharpe_kernel`, `sp15_baseline_buyhold_kernel`, `sp15_baseline_hold_only_kernel`, `sp15_baseline_naive_momentum_kernel`, `sp15_baseline_naive_reversion_kernel`); pre-loaded both `SP15_COST_NET_SHARPE_CUBIN` (1 function) and `SP15_BASELINE_KERNELS_CUBIN` (4 functions) once in `GpuBacktestEvaluator::new()` alongside the existing `env_module` / `metrics_module` / `gather_module` loads; changed all 5 launcher signatures in `gpu_dqn_trainer.rs` to take `&CudaFunction` as a trailing parameter (no more per-call resolution); updated all 5 call sites in `gpu_backtest_evaluator.rs:2877..2922` to pass `&self.sp15_*_kernel`; updated 4 oracle test call sites in `crates/ml/tests/sp15_phase1_oracle_tests.rs` (cost_net + 3 baseline tests covering all 4 baseline kernels) to pre-load the cubin and pass the kernel handle. Both `SP15_BASELINE_KERNELS_CUBIN` and `SP15_COST_NET_SHARPE_CUBIN` were already `pub static` so no visibility promotion needed. Verification: `SQLX_OFFLINE=true CUDA_COMPUTE_CAP=86 cargo check -p ml --tests --all-targets` clean (dev profile, warnings only — no errors). Per `pearl_no_host_branches_in_captured_graph` (this is the eval-context analogue: even outside graph capture, host-side `cuModuleLoadData` is fragile across CUDA context lifetimes when the trial-level context fork hasn't fully promoted the parent's loaded modules), `feedback_no_partial_refactor`, `feedback_wire_everything_up`. SP15 Wave 5 follow-up — pre-load sp15_baseline + cost_net cubins (2026-05-07): root-caused L40S workflow `train-xggfc` (commit `5d63762ab`) hyperopt-trial CUDA failure to the same per-call `load_cubin` + `load_function` anti-pattern that Wave 4.1b's bn_tanh_concat_dd-fix (commit `5d63762ab`, see entry below) addressed for the trainer's bottleneck-concat path. The 5 SP15 evaluation launchers in `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` (`launch_sp15_cost_net_sharpe` at ~787, `launch_sp15_baseline_buyhold` at ~1171, `launch_sp15_baseline_hold_only` at ~1218, `launch_sp15_baseline_naive_momentum` at ~1263, `launch_sp15_baseline_naive_reversion` at ~1308) were resolving `cost_net_sharpe_kernel` / `baseline_*_kernel` symbols via `stream.context().load_cubin(...)` + `module.load_function(...)` ON EVERY CALL inside `GpuBacktestEvaluator`'s per-window eval hot loop (`gpu_backtest_evaluator.rs:2877..2922`, `for w in 0..self.n_windows`). Pattern works in single-pass train-best context (smoke `train-9bcwm` at `5d63762ab` ran the SAME launcher chain successfully through 2 folds × 5 epochs of train-best evaluation), but fails when re-entered from a hyperopt trial's child stream context: `train-xggfc` ran 8 hyperopt search epochs successfully, then started 20 hyperopt trial trainings; trial 1 failed at `load sp15_baseline_kernels cubin: DriverError(CUDA_ERROR_ILLEGAL_ADDRESS, "an illegal memory access was encountered")`; after trial 1's CUDA context was poisoned, trials 2-20 cascade-failed at `Fork CUDA stream for trial: ILLEGAL_ADDRESS`. Fix (atomic, mirrors `5d63762ab` precedent): added 5 `CudaFunction` fields on `GpuBacktestEvaluator` (`sp15_cost_net_sharpe_kernel`, `sp15_baseline_buyhold_kernel`, `sp15_baseline_hold_only_kernel`, `sp15_baseline_naive_momentum_kernel`, `sp15_baseline_naive_reversion_kernel`); pre-loaded both `SP15_COST_NET_SHARPE_CUBIN` (1 function) and `SP15_BASELINE_KERNELS_CUBIN` (4 functions) once in `GpuBacktestEvaluator::new()` alongside the existing `env_module` / `metrics_module` / `gather_module` loads; changed all 5 launcher signatures in `gpu_dqn_trainer.rs` to take `&CudaFunction` as a trailing parameter (no more per-call resolution); updated all 5 call sites in `gpu_backtest_evaluator.rs:2877..2922` to pass `&self.sp15_*_kernel`; updated 4 oracle test call sites in `crates/ml/tests/sp15_phase1_oracle_tests.rs` (cost_net + 3 baseline tests covering all 4 baseline kernels) to pre-load the cubin and pass the kernel handle. Both `SP15_BASELINE_KERNELS_CUBIN` and `SP15_COST_NET_SHARPE_CUBIN` were already `pub static` so no visibility promotion needed. Verification: `SQLX_OFFLINE=true CUDA_COMPUTE_CAP=86 cargo check -p ml --tests --all-targets` clean (dev profile, warnings only — no errors). Per `pearl_no_host_branches_in_captured_graph` (this is the eval-context analogue: even outside graph capture, host-side `cuModuleLoadData` is fragile across CUDA context lifetimes when the trial-level context fork hasn't fully promoted the parent's loaded modules), `feedback_no_partial_refactor`, `feedback_wire_everything_up`.
SP15 Wave 4.1b OOB-followup — pre-load bn_tanh_concat_dd_kernel (2026-05-07): root-caused the L40S smoke `train-vg5f7` (commit `bfc3ffa9d`) capture-mode SEGV to a hot-path `cuModuleLoadData` + `cuModuleGetFunction` inside `submit_forward_ops_main`'s captured `forward` child. Wave 4.1b (eb9515e41) deleted the legacy `bn_tanh_concat_kernel: CudaFunction` field from `GpuDqnTrainer` and migrated the 3 production trainer call sites (online forward at `launch_cublas_forward` ~27755, target forward at ~27949, DDQN argmax at `submit_forward_ops_ddqn` ~27418) to a `launch_sp15_bn_concat_dd` free function that resolved `bn_tanh_concat_dd_kernel` via `stream.context().load_cubin(...)` + `module.load_function(...)` ON EVERY CALL. The collector's matching call site (gpu_experience_collector.rs:4127) already pre-loaded into `bn_tanh_concat_fn` field at construction — Wave 4.1b's mistake was asymmetric: it kept the collector's pre-load pattern but introduced an on-demand loader for the trainer's three captured-graph call sites. Inside a `cuStreamBeginCapture` region, those CUDA driver API calls are NOT capturable (they allocate memory and mutate driver state), and the resulting host-side state corruption surfaced as a SEGV at exit 139 before `CAPTURE_PHASE_FORWARD_DONE` could print. Per `pearl_no_host_branches_in_captured_graph`. Fix: re-add `bn_tanh_concat_dd_kernel: CudaFunction` field on `GpuDqnTrainer`, pre-load it in `compile_training_kernels` from the same `module` as `bn_tanh_bw` / `bn_bias_grad` (forward-child captured replay group, same `DQN_UTILITY_CUBIN`), wire through ctor + the 43->44 utility-kernel return tuple, and change `launch_sp15_bn_concat_dd` to take `&CudaFunction` as a parameter (no more per-call resolution). All 3 trainer call sites now pass `&self.bn_tanh_concat_dd_kernel`. Promoted `DQN_UTILITY_CUBIN` from `pub(crate)` -> `pub` so the SP15 oracle tests + Wave 4.1c behavioral KL test (which used the launcher's old free-function pattern) can pre-load the symbol once in their setup blocks. Verification: `cargo check -p ml --tests` clean (dev + release); `cargo test -p ml --test sp15_phase1_oracle_tests -- --ignored` 36/36 pass including the dd_pct trunk-input KL test that exercises two end-to-end forwards through the modified launcher. SP15 Wave 4.1b OOB-followup — pre-load bn_tanh_concat_dd_kernel (2026-05-07): root-caused the L40S smoke `train-vg5f7` (commit `bfc3ffa9d`) capture-mode SEGV to a hot-path `cuModuleLoadData` + `cuModuleGetFunction` inside `submit_forward_ops_main`'s captured `forward` child. Wave 4.1b (eb9515e41) deleted the legacy `bn_tanh_concat_kernel: CudaFunction` field from `GpuDqnTrainer` and migrated the 3 production trainer call sites (online forward at `launch_cublas_forward` ~27755, target forward at ~27949, DDQN argmax at `submit_forward_ops_ddqn` ~27418) to a `launch_sp15_bn_concat_dd` free function that resolved `bn_tanh_concat_dd_kernel` via `stream.context().load_cubin(...)` + `module.load_function(...)` ON EVERY CALL. The collector's matching call site (gpu_experience_collector.rs:4127) already pre-loaded into `bn_tanh_concat_fn` field at construction — Wave 4.1b's mistake was asymmetric: it kept the collector's pre-load pattern but introduced an on-demand loader for the trainer's three captured-graph call sites. Inside a `cuStreamBeginCapture` region, those CUDA driver API calls are NOT capturable (they allocate memory and mutate driver state), and the resulting host-side state corruption surfaced as a SEGV at exit 139 before `CAPTURE_PHASE_FORWARD_DONE` could print. Per `pearl_no_host_branches_in_captured_graph`. Fix: re-add `bn_tanh_concat_dd_kernel: CudaFunction` field on `GpuDqnTrainer`, pre-load it in `compile_training_kernels` from the same `module` as `bn_tanh_bw` / `bn_bias_grad` (forward-child captured replay group, same `DQN_UTILITY_CUBIN`), wire through ctor + the 43->44 utility-kernel return tuple, and change `launch_sp15_bn_concat_dd` to take `&CudaFunction` as a parameter (no more per-call resolution). All 3 trainer call sites now pass `&self.bn_tanh_concat_dd_kernel`. Promoted `DQN_UTILITY_CUBIN` from `pub(crate)` -> `pub` so the SP15 oracle tests + Wave 4.1c behavioral KL test (which used the launcher's old free-function pattern) can pre-load the symbol once in their setup blocks. Verification: `cargo check -p ml --tests` clean (dev + release); `cargo test -p ml --test sp15_phase1_oracle_tests -- --ignored` 36/36 pass including the dd_pct trunk-input KL test that exercises two end-to-end forwards through the modified launcher.