gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(infra): auto-detect CUDA_COMPUTE_CAP from nvidia-smi + show build stderr

Browse Source 
Sets CUDA_COMPUTE_CAP dynamically from the GPU on the node. Clears
both ml-alpha and ml-core build caches. Captures build.rs stderr in
pod logs for debugging.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-26 14:03:12 +02:00
parent c899fe9f1b
commit 2bcfadf5f4
1 changed files with 7 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
infra/k8s/argo/alpha-rl-template.yaml
View File

@@ -156,12 +156,16 @@ spec:
SHA=$(git rev-parse --short=9 HEAD)
echo "=== Branch: $BRANCH SHA: $SHA ==="
# Clear stale build artifacts (cubin→fatbin migration)
rm -rf ${CARGO_TARGET_DIR}/build/ml-alpha-*
# Auto-detect GPU compute capability for all crates' build.rs
export CUDA_COMPUTE_CAP=$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader | head -1 | tr -d '.')
echo "=== GPU arch: sm_${CUDA_COMPUTE_CAP} ==="
# Clear stale build artifacts when arch changes
rm -rf ${CARGO_TARGET_DIR}/build/ml-alpha-* ${CARGO_TARGET_DIR}/build/ml-core-*
# Compile (~90s cold, ~3s warm after first run)
echo "=== Compile ==="
time cargo build --release -p ml-alpha --example alpha_rl_train
time cargo build --release -p ml-alpha --example alpha_rl_train 2>&1
# Train
OUT="/feature-cache/alpha-rl-runs/$SHA/fold{{workflow.parameters.fold-idx}}"