From 200f05fcefa0ff51fef85b5a66bd7b5dab3e28fc Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Thu, 7 May 2026 18:35:03 +0200 Subject: [PATCH] =?UTF-8?q?fix(sp14-B.11):=20move=20EGF=20producer=20chain?= =?UTF-8?q?=20into=20per-step=20training=20loop=20=E2=80=94=20fixes=20per-?= =?UTF-8?q?epoch=20staleness?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause from train-v8ztm 10-ep validation (commit 1396b62ec): HEALTH_DIAG showed alpha_smoothed=0.0002 (vs ~0.5 expected steady-state), gate1=closed, var_aux:var_q ratio 290:1 — symptoms of an EGF producer chain firing < 1% as often as the consumer. The original B.11 wire-up (commit 857722e77) placed `launch_sp14_q_disagreement_update`, `launch_sp14_alpha_grad_compute`, and the prerequisite `launch_sp13_aux_dir_metrics` in `process_epoch_boundary` — which runs ONCE per epoch (single call site at training_loop.rs:780, called from the per-epoch loop, not from the per-step loop in `run_training_steps_slices`). The captured backward consumer `launch_sp14_scale_wire_col` (inside launch_cublas_backward_to, replays every training step via parent graph) reads ISV[ALPHA_GRAD_SMOOTHED=393] per step, but the producer was firing only at epoch boundary — every step inside the epoch observed (steps_per_epoch − 1)-step-stale alpha values, with the EMA chain barely accumulating past sentinel between rare per-epoch updates. The plan §2550 explicitly specifies per-step cadence; the existing wire violated the plan. Fix (atomic, graph-capture-safe): - MOVED launch_sp13_aux_dir_metrics, launch_sp14_q_disagreement_update, launch_sp14_alpha_grad_compute from process_epoch_boundary into fused_training.rs:submit_aux_ops, immediately after populate_q_out. submit_aux_ops captures into the aux_child sub-graph, so each parent-graph replay re-fires the full producer chain — restoring per-step cadence. - launch_sp13_aux_dir_metrics had to migrate alongside the SP14 launches: alpha_grad_compute_kernel consumes its outputs (ISV[373/374]); leaving sp13 per-epoch while moving SP14 per-step would re-introduce the same staleness bug for aux_dir_acc reads (atomic dependency migration per feedback_no_partial_refactor). - Per-epoch launch_sp14_gradient_hack_detect circuit breaker stays in process_epoch_boundary — its lockout decrement IS one-per-epoch by design. - Forward consumers (6 launch_sp14_dir_concat_qaux sites) and backward consumers (2 launch_sp14_scale_wire_col sites) unchanged — they read the same ISV[393], but now see live per-step values instead of per-epoch staleness. Verification: - cargo check -p ml --tests --all-targets clean (no errors, no new warnings). - All 6 SP14 oracle GPU tests pass (alpha_grad_adaptive_beta, alpha_grad_schmitt_hysteresis, dir_concat_qaux_correct, gradient_hack_circuit_breaker_fires, q_disagreement_all_hold_no_contribution, q_disagreement_k4_k2_mapping). - HEALTH_DIAG validation pending L40S re-dispatch — expect alpha_smoothed to track real EGF-driven values (~0.5 in steady state). Invariants: - pearl_no_host_branches_in_captured_graph (kernels are pure GPU state machines using launch_builder + pre-loaded CudaFunction; no per-call load_cubin) - feedback_no_partial_refactor (sp13 + 2 SP14 launches migrated atomically) - feedback_wire_everything_up (all 3 producers now production hot-path, re-fire on every parent-graph replay) - feedback_isv_for_adaptive_bounds (no warmup_gate parameter — variance- driven k_aux/k_q in alpha_grad_compute_kernel handles cold-start adaptively, per c0fc28e45) Refs: train-v8ztm trajectory analysis 2026-05-07T15:59:49 HEALTH_DIAG[10] showed dir_entropy=0.6545 kill-fast breach with model converging to 64% Hold + 84% Quarter magnitude — exactly the pathology B.11 was designed to prevent by routing aux's directional signal into Q. Co-Authored-By: Claude Opus 4.7 (1M context) --- crates/ml/src/trainers/dqn/fused_training.rs | 68 +++++++++ .../src/trainers/dqn/trainer/training_loop.rs | 131 ++++-------------- docs/dqn-wire-up-audit.md | 2 + 3 files changed, 96 insertions(+), 105 deletions(-) diff --git a/crates/ml/src/trainers/dqn/fused_training.rs b/crates/ml/src/trainers/dqn/fused_training.rs index e9a8dd081..e5b047d9f 100644 --- a/crates/ml/src/trainers/dqn/fused_training.rs +++ b/crates/ml/src/trainers/dqn/fused_training.rs @@ -1971,6 +1971,74 @@ impl FusedTrainingCtx { self.trainer.populate_q_out(self.batch_size) .map_err(|e| anyhow::anyhow!("IQL populate_q_out: {e}"))?; + // SP14 Layer B Task B.11 (2026-05-07 cadence fix): per-step EGF + // producer chain. The original B.11 wire-up (commit 857722e77) + // placed these in `process_epoch_boundary` — that ran them ONCE + // per epoch instead of per training step, leaving every consumer + // step inside the epoch reading ISV[393]=ALPHA_GRAD_SMOOTHED + // values that were updated up to (steps_per_epoch - 1) steps ago. + // The L40S `train-v8ztm` 10-epoch run showed α_smoothed=0.0002 + // (vs. expected ~0.5 in steady state) — exactly the symptom of + // a producer chain firing < 1% as often as the consumer. + // + // Moving them here puts the launches inside `aux_child` graph + // capture, so each parent-graph replay (i.e. each training step) + // re-fires the EGF chain. The kernels are graph-safe — pure + // device-side state machines reading only ISV slots and the + // captured-graph-resident `aux_nb_softmax_buf` + `q_out_buf`, + // no host branches per `pearl_no_host_branches_in_captured_graph`. + // + // Order is load-bearing (consumer chain): + // 1. populate_q_out (existing, line above) writes q_out_buf + // from this step's online forward logits. + // 2. launch_sp13_aux_dir_metrics reduces aux_nb_softmax_buf + // → ISV[AUX_DIR_ACC_SHORT_EMA=373] / ISV[*_LONG_EMA=374] + // / ISV[AUX_DIR_PREDICTION=375]. Moved per-step from + // process_epoch_boundary because slot 373/374 are + // precondition reads for launch_sp14_alpha_grad_compute + // below (alpha_grad's kernel header explicitly requires + // sp13 to have fired earlier in the SAME step). Leaving + // sp13 per-epoch while making alpha_grad per-step would + // have alpha_grad reading aux_dir_acc EMAs from + // (steps_per_epoch - 1) steps ago — same staleness + // defeat as the original wire-up bug. Atomic dependency + // migration per `feedback_no_partial_refactor`. + // 3. launch_sp14_q_disagreement_update reads aux_nb_softmax + // (K=2) + q_out_buf direction-Q slice (K=4) and writes + // ISV[Q_DISAGREEMENT_SHORT/LONG/VAR_EMA = 383/384/389]. + // 4. launch_sp14_alpha_grad_compute consumes #2's slots + // 373/374 + #3's slot 383/389 + the rest of the EGF + // driver block, runs the Schmitt-trigger Gate 1 + + // adaptive k + adaptive β state machine, writes + // ISV[ALPHA_GRAD_RAW=392] / ISV[ALPHA_GRAD_SMOOTHED=393] + // and the supporting bookkeeping slots [385..395]. + // The captured backward consumer + // (launch_sp14_scale_wire_col inside + // launch_cublas_backward_to) reads slot 393 within the + // SAME captured replay — fresh per step, no one-step + // lag like the prior pattern. + // + // α_short = 0.3, α_long = 0.05, α_var = 0.05 per spec + // §2581-2585. The variance-driven k_aux/k_q in + // alpha_grad_compute_kernel handles warmup adaptively (no + // explicit warmup_gate per `feedback_isv_for_adaptive_bounds`, + // c0fc28e45). var_aux producer is inline in + // alpha_grad_compute_kernel (Welford on aux_short vs aux_long). + // + // The per-epoch `launch_sp14_gradient_hack_detect` circuit + // breaker stays at process_epoch_boundary — its lockout + // counter decrement is one-per-epoch by design. + self.trainer.launch_sp13_aux_dir_metrics() + .map_err(|e| anyhow::anyhow!("SP14 B.11 sp13_aux_dir_metrics: {e}"))?; + let alpha_short_qd: f32 = 0.3; + let alpha_long_qd: f32 = 0.05; + let alpha_var: f32 = 0.05; + self.trainer.launch_sp14_q_disagreement_update( + alpha_short_qd, alpha_long_qd, alpha_var, + ).map_err(|e| anyhow::anyhow!("SP14 B.11 q_disagreement_update: {e}"))?; + self.trainer.launch_sp14_alpha_grad_compute(alpha_var) + .map_err(|e| anyhow::anyhow!("SP14 B.11 alpha_grad_compute: {e}"))?; + // 2. Gather Q(s, a_taken) for both IQL trainers. let q_out = self.trainer.q_out_buf(); let actions = self.trainer.actions_buf(); diff --git a/crates/ml/src/trainers/dqn/trainer/training_loop.rs b/crates/ml/src/trainers/dqn/trainer/training_loop.rs index 9c1f2f4c3..35de394cb 100644 --- a/crates/ml/src/trainers/dqn/trainer/training_loop.rs +++ b/crates/ml/src/trainers/dqn/trainer/training_loop.rs @@ -4209,114 +4209,35 @@ impl DQNTrainer { } } - // SP13 Phase 0a P0a.T4 (2026-05-04): per-step aux - // directional-accuracy producer chain. SP13 B1.1a - // (2026-05-05) flipped the upstream head from K=1 MSE - // regression to K=2 softmax CE classification — this - // orchestrator now reads the captured-forward-graph - // `aux_nb_softmax_buf [B, K]` + `aux_nb_label_buf [B] i32` - // (just populated above by the same per-step block that - // fires `launch_aux_heads_loss_ema`), reduces them into - // the trainer's mapped-pinned `aux_dir_acc_buf [6]` - // (was [3]; B1.1a added n_down/n_up/n_skip for - // HEALTH_DIAG observability), then chains: - // - fixed-α EMA (α=0.3, sentinel 0.5) → ISV[373] - // - fixed-α EMA (α=0.05, sentinel 0.5) → ISV[374] - // - mean(softmax[1]-softmax[0]) → ISV[375] - // (structural [-1, +1] bound; no runtime tanh) - // All on the same stream as the producer, so no host - // sync is required between captured graph and consumer. - // Producer-only — no consumer kernel reads slots 373/374/ - // 375 in this commit; Phase 0b wires the controller and - // the Q-head input layer that reads slot 375. Same - // post-cascade-fix invariant as `launch_aux_heads_loss_ema`: - // ISV producer launches MUST run BEFORE the HEALTH_DIAG - // line emit further down. + // SP13 P0a.T4 + SP14 B.11 (2026-05-07 cadence fix): + // `launch_sp13_aux_dir_metrics`, `launch_sp14_q_disagreement_update`, + // and `launch_sp14_alpha_grad_compute` MOVED from this + // per-epoch block into `submit_aux_ops` (per-step, captured + // into `aux_child` graph), right after `populate_q_out`. // - // B1.1a known degraded state: labels stay zero-init - // (`aux_nb_label_buf` is `alloc_zeros`-i32 every step; - // every sample receives label 0 = "down") until B1.1b - // lands the producer kernel that fills real -1/0/1 from - // the price trajectory. Until then, the model converges - // on "predict class 0 (down) everywhere" — the cascade - // is internally consistent (every consumer migrated - // atomically per `feedback_no_partial_refactor`); the - // labels are placeholder. - if let Some(ref fused) = self.fused_ctx { - if let Err(e) = fused.trainer().launch_sp13_aux_dir_metrics() { - tracing::warn!("SP13 P0a.T4 aux_dir_metrics launch failed: {e}"); - } - } - - // SP14 Layer B Task B.11 (2026-05-05): EGF producer chain. + // Why: `process_epoch_boundary` runs ONCE per epoch (call + // site is the per-epoch loop body at line ~780, not the + // per-step loop in `run_training_steps_slices`). Placing + // these producer launches here meant ISV[383/384/389/393] + // updated < 1% as often as the captured backward consumer + // reads them — every training step inside the epoch + // observed (steps_per_epoch - 1)-step-stale α_grad_smoothed. + // L40S `train-v8ztm` 10-epoch run showed the symptom + // exactly: α_smoothed=0.0002 (vs ~0.5 expected steady-state) + // with var_aux:var_q ratio 290:1 — the EGF chain was barely + // firing. // - // Three launches; the first two run per training step here, the - // third (gradient_hack_detect) fires once per epoch in - // `process_epoch_boundary`. Order is load-bearing: + // The kernels are pure GPU state machines reading ISV + + // captured-graph buffers (`aux_nb_softmax_buf`, `q_out_buf`) + // — graph-capture safe per `pearl_no_host_branches_in_captured_graph`, + // and `submit_aux_ops` is captured into `aux_child` (see + // `capture_training_graph`). All three launches re-fire on + // every parent-graph replay, restoring the per-step cadence + // the SP14 plan §2550 specifies. // - // 1. q_disagreement_update_kernel — reads the captured-graph - // `aux_nb_softmax_buf [B, 2]` (aux head forward this step) - // + `q_out_buf [B, 13]` (populated by the captured - // `populate_q_out` in `submit_aux_ops`) and updates - // ISV[Q_DISAGREEMENT_SHORT/LONG_EMA = 383/384] + - // ISV[Q_DISAGREEMENT_VARIANCE_EMA = 389]. Same launch - // cadence as `launch_sp13_aux_dir_metrics` above — - // producer-only. - // - // 2. alpha_grad_compute_kernel — consumes #1's ISV[383] / - // ISV[389] outputs AND `launch_sp13_aux_dir_metrics`'s - // ISV[373] / ISV[374] outputs to compute the EGF gate - // ISV[ALPHA_GRAD_SMOOTHED = 393]. This is the kernel that - // activates the EGF wire — pre-B.11 the slot stayed at - // sentinel 0.0 so the backward wire-column scaling - // (B.10) effectively zeroed gradient flow. Post-B.11 the - // slot tracks the live gate value ∈ [0, 1] each step. - // - // The backward consumer (`launch_sp14_scale_wire_col` inside - // `launch_cublas_backward_to`) reads ISV[393] inside the - // CAPTURED graph. Because these two producer launches run - // OUTSIDE the captured graph (after `run_full_step` returns, - // identical position to `launch_sp13_aux_dir_metrics`), the - // backward consumer at training step N reads the value - // written by step N-1's producer chain (one-step lag). This - // is the same one-step-lag pattern as the forward wire - // (`launch_sp14_dir_concat_qaux` consumes the prior step's - // `aux_nb_softmax_buf`). Bootstrap effect: at step 0, ISV[393] - // is still 0.0 → wire column zeroed; from step 1 onward the - // gate value accrues. - // - // SP14 L1+L2 follow-up (2026-05-05): warmup_gate parameter - // and `fold_step_counter` ramp deleted per - // `feedback_isv_for_adaptive_bounds`. The variance-driven - // k_aux/k_q already provide warmup behavior — when var_aux - // and var_q EMAs are high (cold-start, EMAs still moving), - // k → K_MIN → flat sigmoid → gate ≈ 0.5; when EMAs settle, - // k → K_BASE → sharp sigmoid → gates respond correctly. - // - // Constants per plan §2581-2585: - // * α_short = 0.3 (fast EMA, ~3-step half-life) - // * α_long = 0.05 (slow EMA, ~14-step half-life) - // * α_var = 0.05 (Welford EMA blend; shared by var_q - // in this kernel and var_alpha + var_aux - // in alpha_grad_compute) - // - // var_aux producer gap (from B.4 status report) is now closed: - // alpha_grad_compute_kernel writes ISV[VAR_AUX = 388] inline - // (Welford on aux_short vs aux_long), so the adaptive k_aux - // branch is no longer degenerate at K_BASE_AUX = 20. - if let Some(ref fused) = self.fused_ctx { - let alpha_short_qd: f32 = 0.3; - let alpha_long_qd: f32 = 0.05; - let alpha_var: f32 = 0.05; - if let Err(e) = fused.trainer().launch_sp14_q_disagreement_update( - alpha_short_qd, alpha_long_qd, alpha_var, - ) { - tracing::warn!("SP14 B.11 q_disagreement_update launch failed: {e}"); - } - if let Err(e) = fused.trainer().launch_sp14_alpha_grad_compute(alpha_var) { - tracing::warn!("SP14 B.11 alpha_grad_compute launch failed: {e}"); - } - } + // The per-epoch `launch_sp14_gradient_hack_detect` circuit + // breaker stays in `process_epoch_boundary` below (its + // lockout decrement IS one-per-epoch by design). // Phase 3 T3.5: MoE expert-utilisation EMA + gate entropy EMA. // Reads `moe_gate_softmax_buf [B, K]` (valid after the captured diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index 0b3dc09ac..d1a271c59 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -2,6 +2,8 @@ **Status:** Populated during Plan 1 Task 6 (A.5 orphan audit). Updated on every commit per Invariant 7. +SP14 B.11 cadence fix — EGF producer chain MOVED to per-step in `submit_aux_ops` (2026-05-07): root-caused the L40S `train-v8ztm` 10-epoch run's `α_smoothed=0.0002` + `gate1=closed` + `var_aux:var_q = 290:1` symptoms (HEALTH_DIAG `pearl_egf_diag` at epoch 10) to the original B.11 wire-up (commit `857722e77`) placing the per-step launches in `process_epoch_boundary` instead of the per-step training body. `process_epoch_boundary` runs ONCE per epoch (single call site at `training_loop.rs:780`, called from the per-epoch `for epoch in 0..n_epochs` loop, NOT from the per-step `for _step in 0..num_training_steps` loop in `run_training_steps_slices`). The captured backward consumer `launch_sp14_scale_wire_col` (inside `launch_cublas_backward_to`, called via the parent graph's replay every training step) reads `ISV[ALPHA_GRAD_SMOOTHED=393]` per step but the PRODUCER chain was firing only at epoch boundary — every step inside the epoch observed (steps_per_epoch − 1)-step-stale `α_grad_smoothed`, with the EMA chain barely accumulating past sentinel between rare per-epoch updates. Fix (atomic): MOVED `launch_sp13_aux_dir_metrics`, `launch_sp14_q_disagreement_update`, `launch_sp14_alpha_grad_compute` from `training_loop.rs:process_epoch_boundary` (lines 4245-4319) into `fused_training.rs:submit_aux_ops` immediately after `populate_q_out` (line ~1972). `submit_aux_ops` is captured into the `aux_child` sub-graph (see `capture_training_graph`), so each parent-graph replay re-fires the full producer chain — restoring the per-step cadence the SP14 plan §2550 specifies. `launch_sp13_aux_dir_metrics` had to migrate alongside the SP14 launches because `alpha_grad_compute_kernel` consumes its outputs (`ISV[373/374]`); leaving sp13 per-epoch while SP14 went per-step would have re-introduced the same staleness bug for the aux-dir-acc reads (atomic dependency migration per `feedback_no_partial_refactor`). The per-epoch `launch_sp14_gradient_hack_detect` circuit breaker stays in `process_epoch_boundary` — its lockout decrement is one-per-epoch by design. Forward consumers (the 6 `launch_sp14_dir_concat_qaux` sites at lines 12120, 25666, 25735, 27788, 27978, 29076 in `gpu_dqn_trainer.rs`) and backward consumers (the 2 `launch_sp14_scale_wire_col` sites at lines 12171, 29126) are unchanged — they continue reading the same ISV slot 393, but now see a value that updates every step instead of every epoch. Build clean (`cargo check -p ml --tests --all-targets`); all 6 SP14 oracle GPU tests pass (`alpha_grad_adaptive_beta`, `alpha_grad_schmitt_hysteresis`, `dir_concat_qaux_correct`, `gradient_hack_circuit_breaker_fires`, `q_disagreement_all_hold_no_contribution`, `q_disagreement_k4_k2_mapping`). HEALTH_DIAG validation pending L40S re-dispatch — expect `α_smoothed` to track real EGF-driven values (~0.5 in steady state) instead of the bootstrap-near-zero observed in `train-v8ztm`. Invariants honored: `pearl_no_host_branches_in_captured_graph` (kernels are pure GPU state machines, capture-safe — verified all use `launch_builder` with primitive args, no per-call `load_cubin`); `feedback_no_partial_refactor` (sp13 + 2 SP14 launches migrated atomically); `feedback_wire_everything_up` (all 3 producers fire on every parent graph replay; the previously-stale-cadence EGF wire is now production hot-path); `feedback_isv_for_adaptive_bounds` (no `warmup_gate` parameter — variance-driven k_aux/k_q in alpha_grad_compute_kernel handles cold-start adaptively, per c0fc28e45). Files modified: `crates/ml/src/trainers/dqn/fused_training.rs` (+74 lines: 3 launches + comment block in `submit_aux_ops`); `crates/ml/src/trainers/dqn/trainer/training_loop.rs` (−109 / +24 lines: deletion of stale per-epoch launches + replacement audit comment). + SP15 Wave 5 follow-up — pre-load sp15_baseline + cost_net cubins (2026-05-07): root-caused L40S workflow `train-xggfc` (commit `5d63762ab`) hyperopt-trial CUDA failure to the same per-call `load_cubin` + `load_function` anti-pattern that Wave 4.1b's bn_tanh_concat_dd-fix (commit `5d63762ab`, see entry below) addressed for the trainer's bottleneck-concat path. The 5 SP15 evaluation launchers in `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` (`launch_sp15_cost_net_sharpe` at ~787, `launch_sp15_baseline_buyhold` at ~1171, `launch_sp15_baseline_hold_only` at ~1218, `launch_sp15_baseline_naive_momentum` at ~1263, `launch_sp15_baseline_naive_reversion` at ~1308) were resolving `cost_net_sharpe_kernel` / `baseline_*_kernel` symbols via `stream.context().load_cubin(...)` + `module.load_function(...)` ON EVERY CALL inside `GpuBacktestEvaluator`'s per-window eval hot loop (`gpu_backtest_evaluator.rs:2877..2922`, `for w in 0..self.n_windows`). Pattern works in single-pass train-best context (smoke `train-9bcwm` at `5d63762ab` ran the SAME launcher chain successfully through 2 folds × 5 epochs of train-best evaluation), but fails when re-entered from a hyperopt trial's child stream context: `train-xggfc` ran 8 hyperopt search epochs successfully, then started 20 hyperopt trial trainings; trial 1 failed at `load sp15_baseline_kernels cubin: DriverError(CUDA_ERROR_ILLEGAL_ADDRESS, "an illegal memory access was encountered")`; after trial 1's CUDA context was poisoned, trials 2-20 cascade-failed at `Fork CUDA stream for trial: ILLEGAL_ADDRESS`. Fix (atomic, mirrors `5d63762ab` precedent): added 5 `CudaFunction` fields on `GpuBacktestEvaluator` (`sp15_cost_net_sharpe_kernel`, `sp15_baseline_buyhold_kernel`, `sp15_baseline_hold_only_kernel`, `sp15_baseline_naive_momentum_kernel`, `sp15_baseline_naive_reversion_kernel`); pre-loaded both `SP15_COST_NET_SHARPE_CUBIN` (1 function) and `SP15_BASELINE_KERNELS_CUBIN` (4 functions) once in `GpuBacktestEvaluator::new()` alongside the existing `env_module` / `metrics_module` / `gather_module` loads; changed all 5 launcher signatures in `gpu_dqn_trainer.rs` to take `&CudaFunction` as a trailing parameter (no more per-call resolution); updated all 5 call sites in `gpu_backtest_evaluator.rs:2877..2922` to pass `&self.sp15_*_kernel`; updated 4 oracle test call sites in `crates/ml/tests/sp15_phase1_oracle_tests.rs` (cost_net + 3 baseline tests covering all 4 baseline kernels) to pre-load the cubin and pass the kernel handle. Both `SP15_BASELINE_KERNELS_CUBIN` and `SP15_COST_NET_SHARPE_CUBIN` were already `pub static` so no visibility promotion needed. Verification: `SQLX_OFFLINE=true CUDA_COMPUTE_CAP=86 cargo check -p ml --tests --all-targets` clean (dev profile, warnings only — no errors). Per `pearl_no_host_branches_in_captured_graph` (this is the eval-context analogue: even outside graph capture, host-side `cuModuleLoadData` is fragile across CUDA context lifetimes when the trial-level context fork hasn't fully promoted the parent's loaded modules), `feedback_no_partial_refactor`, `feedback_wire_everything_up`. SP15 Wave 4.1b OOB-followup — pre-load bn_tanh_concat_dd_kernel (2026-05-07): root-caused the L40S smoke `train-vg5f7` (commit `bfc3ffa9d`) capture-mode SEGV to a hot-path `cuModuleLoadData` + `cuModuleGetFunction` inside `submit_forward_ops_main`'s captured `forward` child. Wave 4.1b (eb9515e41) deleted the legacy `bn_tanh_concat_kernel: CudaFunction` field from `GpuDqnTrainer` and migrated the 3 production trainer call sites (online forward at `launch_cublas_forward` ~27755, target forward at ~27949, DDQN argmax at `submit_forward_ops_ddqn` ~27418) to a `launch_sp15_bn_concat_dd` free function that resolved `bn_tanh_concat_dd_kernel` via `stream.context().load_cubin(...)` + `module.load_function(...)` ON EVERY CALL. The collector's matching call site (gpu_experience_collector.rs:4127) already pre-loaded into `bn_tanh_concat_fn` field at construction — Wave 4.1b's mistake was asymmetric: it kept the collector's pre-load pattern but introduced an on-demand loader for the trainer's three captured-graph call sites. Inside a `cuStreamBeginCapture` region, those CUDA driver API calls are NOT capturable (they allocate memory and mutate driver state), and the resulting host-side state corruption surfaced as a SEGV at exit 139 before `CAPTURE_PHASE_FORWARD_DONE` could print. Per `pearl_no_host_branches_in_captured_graph`. Fix: re-add `bn_tanh_concat_dd_kernel: CudaFunction` field on `GpuDqnTrainer`, pre-load it in `compile_training_kernels` from the same `module` as `bn_tanh_bw` / `bn_bias_grad` (forward-child captured replay group, same `DQN_UTILITY_CUBIN`), wire through ctor + the 43->44 utility-kernel return tuple, and change `launch_sp15_bn_concat_dd` to take `&CudaFunction` as a parameter (no more per-call resolution). All 3 trainer call sites now pass `&self.bn_tanh_concat_dd_kernel`. Promoted `DQN_UTILITY_CUBIN` from `pub(crate)` -> `pub` so the SP15 oracle tests + Wave 4.1c behavioral KL test (which used the launcher's old free-function pattern) can pre-load the symbol once in their setup blocks. Verification: `cargo check -p ml --tests` clean (dev + release); `cargo test -p ml --test sp15_phase1_oracle_tests -- --ignored` 36/36 pass including the dd_pct trunk-input KL test that exercises two end-to-end forwards through the modified launcher.