Files
foxhunt/crates/ml/tests
jgrusewski 1396b62ec6 fix(sp15-wave5-followup): pre-load sp15_baseline + cost_net cubins — fixes hyperopt-trial CUDA_ERROR_ILLEGAL_ADDRESS
Root cause: 5 SP15 evaluation launchers (cost_net_sharpe + 4 baseline_*)
were doing `load_cubin` + `load_function` PER-CALL inside
`GpuBacktestEvaluator`'s eval hot loop. Pattern is fragile across CUDA
context lifetimes — works in single-pass train-best context (smoke
train-9bcwm verified), fails in hyperopt-trial child stream context
(workflow train-xggfc trial 1 failed at "load sp15_baseline_kernels
cubin: ILLEGAL_ADDRESS"; after the host-side load corrupted the trial's
context, trials 2-20 all cascade-failed at "Fork CUDA stream for trial").

Fix (atomic, matches 5d63762ab precedent for bn_tanh_concat_dd):
- Add 5 `CudaFunction` fields on `GpuBacktestEvaluator`.
- Pre-load both `SP15_BASELINE_KERNELS_CUBIN` (4 functions) and
  `SP15_COST_NET_SHARPE_CUBIN` (1 function) once in
  `GpuBacktestEvaluator::new()`, alongside the existing `env_module` /
  `metrics_module` loads.
- Change all 5 launcher signatures in `gpu_dqn_trainer.rs` to take
  `&CudaFunction` instead of doing per-call cubin load.
- Update all 5 call sites in `gpu_backtest_evaluator.rs:2877..2922` to
  pass `&self.sp15_*_kernel`.
- Update 4 oracle test call sites in
  `crates/ml/tests/sp15_phase1_oracle_tests.rs` (cost_net + 4 baselines)
  to pre-load and pass the kernel handle directly, matching 5d63762ab's
  pattern for `DQN_UTILITY_CUBIN`.
- Update Invariant 7 audit doc (`docs/dqn-wire-up-audit.md`).

Verification: `SQLX_OFFLINE=true CUDA_COMPUTE_CAP=86 cargo check -p ml
--tests --all-targets` clean (warnings only, no errors).

Refs: train-xggfc failure 2026-05-07T13:11:43, 5d63762ab precedent,
pearl_no_host_branches_in_captured_graph (this is the eval analogue),
feedback_no_partial_refactor, feedback_wire_everything_up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 16:47:41 +02:00
..