From 14bafb5b58d46884455f341789e9ea503b17a3c8 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sun, 10 May 2026 17:47:44 +0200 Subject: [PATCH] fix(argo-train): apply train-template.yaml before single-job submission MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The multi-seed path already does `kubectl apply` before `argo submit`, so cluster template stays in sync with source. The single-job path used `argo submit --from=wftmpl/train` directly, expecting the cluster's template to already match — which silently drifts when defaults change. Caused workflow train-jpxvn (2026-05-10) to dispatch with stale imbalance-bar-threshold=0.5 default (the cluster's old value) when the source had been bumped to 20.0. Triggered near-OOM in feature extraction. One-line fix: apply the template before submission. Mirrors what multi-seed already does. No behavior change for users who pass explicit flags; just makes implicit defaults track source. Co-Authored-By: Claude Opus 4.7 (1M context) --- scripts/argo-train.sh | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/scripts/argo-train.sh b/scripts/argo-train.sh index e257bb456..7ab881dd4 100755 --- a/scripts/argo-train.sh +++ b/scripts/argo-train.sh @@ -152,6 +152,14 @@ if [[ "$MULTI_SEED" -gt 1 || "$FOLDS" -gt 1 || "$PROFILE" == "true" ]]; then fi if [[ "$USE_MULTI_SEED" == "false" ]]; then + # Apply the local train-template.yaml to the cluster BEFORE submission. + # Without this, `argo submit --from=wftmpl/train` runs against whatever + # was last applied — defaults can drift from source. Caused workflow + # train-jpxvn (2026-05-10) to dispatch with a stale threshold=0.5 + # default and trigger near-OOM. Multi-seed path already does this; the + # single-job path was missing it. + kubectl apply -n foxhunt -f infra/k8s/argo/train-template.yaml >/dev/null + CMD="argo submit -n foxhunt --from=wftmpl/train" CMD="$CMD -p commit-sha=$SHA" CMD="$CMD -p git-branch=$BRANCH"