gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

perf(ci): serialize dqn-pipeline tests to avoid GPU contention

Browse Source 
4 concurrent DQN trainers on a shared H100 causes severe GPU contention
(28-39% utilization, 42 GB VRAM). Serializing with --test-threads=1
gives each test full GPU access, reducing total wallclock time despite
sequential execution.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-03-13 19:52:04 +01:00
parent b87b7b4bea
commit 3112e07a5a
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
infra/k8s/argo/gpu-test-pipeline-template.yaml
View File

@@ -263,7 +263,7 @@ spec:
fi
if [ "$SCOPE" = "integration" ] || [ "$SCOPE" = "all" ]; then
run_tests "dqn-smoke" cargo test -p ml --features cuda --test smoke_test_real_data
run_tests "dqn-pipeline" cargo test -p ml --features cuda --test dqn_training_pipeline_test
run_tests "dqn-pipeline" cargo test -p ml --features cuda --test dqn_training_pipeline_test -- --test-threads=1
run_tests "dqn-smoke-train" cargo test -p ml --features cuda --test dqn_training_smoke_test
run_tests "dqn-early-stop" cargo test -p ml --features cuda --test dqn_early_stopping_termination_test
run_tests "dqn-collapse" cargo test -p ml --features cuda --test dqn_action_collapse_fix_test