fix: CI OOMKilled on submit-gpu-test + YAML indentation fixes

submit-gpu-test wait sidecar had 64Mi limit — OOMKilled when tracking
large child workflow status JSON. Bumped to 256Mi.

Fixed YAML indentation errors in compile-and-train and training-pipeline
templates (misaligned labels, duplicate component keys).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-03-22 10:45:43 +01:00
parent ca94c0e851
commit a7c22ca5aa
3 changed files with 5 additions and 4 deletions

View File

@@ -595,6 +595,9 @@ spec:
# ── submit-gpu-test: launch GPU test workflow as a child workflow ──
- name: submit-gpu-test
# Argo wait sidecar needs 256Mi to track large child workflow status JSON.
# Default 64Mi causes OOMKilled when child has many parameters.
podSpecPatch: '{"containers":[{"name":"wait","resources":{"requests":{"memory":"128Mi"},"limits":{"memory":"256Mi"}}}]}'
resource:
action: create
setOwnerReference: true

View File

@@ -17,7 +17,7 @@ metadata:
labels:
app.kubernetes.io/name: compile-and-train
app.kubernetes.io/part-of: foxhunt
app.kubernetes.io/component: gpu-test
app.kubernetes.io/component: gpu-test
spec:
entrypoint: pipeline
onExit: notify-result
@@ -25,7 +25,6 @@ spec:
podMetadata:
labels:
app.kubernetes.io/part-of: foxhunt
app.kubernetes.io/component: gpu-test
app.kubernetes.io/component: compile-and-train
# No runAsUser — compile (ci-builder) needs root, training runtime defaults
# to its image user. fsGroup ensures workspace PVC is writable by all pods.

View File

@@ -6,7 +6,7 @@ metadata:
labels:
app.kubernetes.io/name: training-pipeline
app.kubernetes.io/part-of: foxhunt
app.kubernetes.io/component: gpu-test
app.kubernetes.io/component: gpu-test
spec:
entrypoint: train-model
serviceAccountName: argo-workflow
@@ -14,7 +14,6 @@ spec:
podMetadata:
labels:
app.kubernetes.io/part-of: foxhunt
app.kubernetes.io/component: gpu-test
app.kubernetes.io/component: training-workflow
securityContext:
runAsUser: 1000