fix(argo-train): apply train-template.yaml before single-job submission

The multi-seed path already does `kubectl apply` before `argo submit`,
so cluster template stays in sync with source. The single-job path used
`argo submit --from=wftmpl/train` directly, expecting the cluster's
template to already match — which silently drifts when defaults change.

Caused workflow train-jpxvn (2026-05-10) to dispatch with stale
imbalance-bar-threshold=0.5 default (the cluster's old value) when the
source had been bumped to 20.0. Triggered near-OOM in feature extraction.

One-line fix: apply the template before submission. Mirrors what
multi-seed already does. No behavior change for users who pass explicit
flags; just makes implicit defaults track source.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-10 17:47:44 +02:00
parent c78c4766ca
commit 14bafb5b58

View File

@@ -152,6 +152,14 @@ if [[ "$MULTI_SEED" -gt 1 || "$FOLDS" -gt 1 || "$PROFILE" == "true" ]]; then
fi
if [[ "$USE_MULTI_SEED" == "false" ]]; then
# Apply the local train-template.yaml to the cluster BEFORE submission.
# Without this, `argo submit --from=wftmpl/train` runs against whatever
# was last applied — defaults can drift from source. Caused workflow
# train-jpxvn (2026-05-10) to dispatch with a stale threshold=0.5
# default and trigger near-OOM. Multi-seed path already does this; the
# single-job path was missing it.
kubectl apply -n foxhunt -f infra/k8s/argo/train-template.yaml >/dev/null
CMD="argo submit -n foxhunt --from=wftmpl/train"
CMD="$CMD -p commit-sha=$SHA"
CMD="$CMD -p git-branch=$BRANCH"