fix(infra): make gpu-warmup tolerant of missing nvidia-smi

nvidia-smi is driver-mounted by the GPU operator, which may not be
ready when the warmup pod starts on a fresh autoscaled node. The
warmup's purpose is just triggering autoscale, not GPU validation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-03-07 00:56:35 +01:00
parent f356ce157c
commit fbb674f9e2

View File

@@ -83,8 +83,8 @@ spec:
template: upload-results
dependencies: [evaluate]
# GPU warmup: triggers H100 autoscale during compilation so the node
# is ready when hyperopt starts. Exits immediately after nvidia-smi.
# GPU warmup: triggers GPU node autoscale during fetch-binary so the node
# and NVIDIA device plugin are ready when hyperopt starts.
- name: gpu-warmup
nodeSelector:
k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
@@ -102,7 +102,7 @@ spec:
args:
- |
echo "GPU warmup: triggering node autoscale..."
nvidia-smi --query-gpu=name,memory.total --format=csv,noheader
nvidia-smi --query-gpu=name,memory.total --format=csv,noheader 2>/dev/null || echo "nvidia-smi not yet available (driver still mounting)"
echo "GPU node ready, exiting to free resources"
resources:
requests: