fix(infra): make gpu-warmup tolerant of missing nvidia-smi
nvidia-smi is driver-mounted by the GPU operator, which may not be ready when the warmup pod starts on a fresh autoscaled node. The warmup's purpose is just triggering autoscale, not GPU validation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -83,8 +83,8 @@ spec:
|
||||
template: upload-results
|
||||
dependencies: [evaluate]
|
||||
|
||||
# GPU warmup: triggers H100 autoscale during compilation so the node
|
||||
# is ready when hyperopt starts. Exits immediately after nvidia-smi.
|
||||
# GPU warmup: triggers GPU node autoscale during fetch-binary so the node
|
||||
# and NVIDIA device plugin are ready when hyperopt starts.
|
||||
- name: gpu-warmup
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
|
||||
@@ -102,7 +102,7 @@ spec:
|
||||
args:
|
||||
- |
|
||||
echo "GPU warmup: triggering node autoscale..."
|
||||
nvidia-smi --query-gpu=name,memory.total --format=csv,noheader
|
||||
nvidia-smi --query-gpu=name,memory.total --format=csv,noheader 2>/dev/null || echo "nvidia-smi not yet available (driver still mounting)"
|
||||
echo "GPU node ready, exiting to free resources"
|
||||
resources:
|
||||
requests:
|
||||
|
||||
Reference in New Issue
Block a user