gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(build): ml-backtesting honors CUDA_COMPUTE_CAP for sm_90 on H100

Browse Source 
alpha-rl-mbg2n on H100 failed at LobSimCuda::new with
`CUDA_ERROR_NO_BINARY_FOR_GPU` loading book_update.cubin. Root cause:
ml-backtesting/build.rs read only `FOXHUNT_CUDA_ARCH` (set by
lob-backtest-sweep-template) but NOT `CUDA_COMPUTE_CAP` (set by
alpha-rl-template from `nvidia-smi --query-gpu=compute_cap` inside the
compile pod). On H100 it silently fell through to default sm_86; the
sm_86 cubin contains no PTX → no JIT path on sm_90 device.

The docstring claimed "Mirrors crates/ml-alpha/build.rs" — it didn't
(ml-alpha reads CUDA_COMPUTE_CAP). Completing the mirror now: detect_arch
returns sm_<NN> honoring (in order):
  1. CUDA_COMPUTE_CAP numeric env (alpha-rl-template)
  2. FOXHUNT_CUDA_ARCH sm_-prefixed env (lob-backtest-sweep-template)
  3. nvidia-smi --query-gpu=compute_cap at build time
  4. Default sm_86 (RTX 3050 Ti local dev)

Both production argo templates now produce sm_90 cubins on H100 and
sm_89 on L40S without further changes.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-06-01 11:00:53 +02:00
parent b7bbb700f8
commit 251b871294
1 changed files with 50 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

53
crates/ml-backtesting/build.rs
View File

@@ -11,9 +11,21 @@ fn main() -> Result<(), String> {
// Per pearl_build_rs_rerun_if_env_changed.md: pair every env::var
// with rerun-if-env-changed.
// Default sm_86 covers RTX 3050 (local dev). Production Argo workflows
// override via FOXHUNT_CUDA_ARCH=sm_89 (L40S Ada) or sm_90 (H100).
let arch = std::env::var("FOXHUNT_CUDA_ARCH").unwrap_or_else(|_| "sm_86".into());
//
// Arch detection (mirrors crates/ml-alpha/build.rs:detect_arch — the
// original "mirror" docstring above was aspirational, B-9 cluster
// alpha-rl-mbg2n on H100 caught the gap: ml-alpha picked sm_90 via
// CUDA_COMPUTE_CAP but ml-backtesting fell through to default sm_86
// → no kernel image error at LobSimCuda::new). Priority:
// 1. CUDA_COMPUTE_CAP (numeric, e.g. "90") — set by alpha-rl-template
// from nvidia-smi inside the compile pod
// 2. FOXHUNT_CUDA_ARCH (sm_-prefixed, e.g. "sm_90") — set by
// lob-backtest-sweep-template
// 3. nvidia-smi --query-gpu=compute_cap at build time
// 4. Default sm_86 (RTX 3050 Ti local dev)
let arch = detect_arch();
eprintln!(" ml-backtesting: compiling kernels for {arch}");
println!("cargo:rerun-if-env-changed=CUDA_COMPUTE_CAP");
println!("cargo:rerun-if-env-changed=FOXHUNT_CUDA_ARCH");
println!("cargo:rerun-if-env-changed=CUDA_PATH");
println!("cargo:rerun-if-env-changed=NVCC");
@@ -60,3 +72,38 @@ fn main() -> Result<(), String> {
}
Ok(())
}
/// Returns the `sm_<NN>` arch string, honoring (in order):
/// `CUDA_COMPUTE_CAP` numeric env → `FOXHUNT_CUDA_ARCH` sm_-prefixed env
/// → `nvidia-smi --query-gpu=compute_cap` → default `sm_86`.
fn detect_arch() -> String {
// 1. Numeric CUDA_COMPUTE_CAP (e.g. "90") from alpha-rl-template.
if let Ok(cap) = std::env::var("CUDA_COMPUTE_CAP") {
let trimmed = cap.trim();
if !trimmed.is_empty() {
return format!("sm_{trimmed}");
}
}
// 2. sm_-prefixed FOXHUNT_CUDA_ARCH (e.g. "sm_90") from lob-backtest-sweep.
if let Ok(arch) = std::env::var("FOXHUNT_CUDA_ARCH") {
let trimmed = arch.trim();
if !trimmed.is_empty() {
return trimmed.to_string();
}
}
// 3. Query the GPU on this machine (e.g. "9.0" → "90" → "sm_90").
if let Ok(output) = Command::new("nvidia-smi")
.args(["--query-gpu=compute_cap", "--format=csv,noheader"])
.output()
{
if output.status.success() {
let s = String::from_utf8_lossy(&output.stdout);
let cap = s.trim().replace('.', "");
if !cap.is_empty() {
return format!("sm_{cap}");
}
}
}
// 4. Default — RTX 3050 Ti local dev.
"sm_86".to_string()
}