fix(argo-train): apply train-template.yaml before single-job submission
The multi-seed path already does `kubectl apply` before `argo submit`, so cluster template stays in sync with source. The single-job path used `argo submit --from=wftmpl/train` directly, expecting the cluster's template to already match — which silently drifts when defaults change. Caused workflow train-jpxvn (2026-05-10) to dispatch with stale imbalance-bar-threshold=0.5 default (the cluster's old value) when the source had been bumped to 20.0. Triggered near-OOM in feature extraction. One-line fix: apply the template before submission. Mirrors what multi-seed already does. No behavior change for users who pass explicit flags; just makes implicit defaults track source. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -152,6 +152,14 @@ if [[ "$MULTI_SEED" -gt 1 || "$FOLDS" -gt 1 || "$PROFILE" == "true" ]]; then
|
||||
fi
|
||||
|
||||
if [[ "$USE_MULTI_SEED" == "false" ]]; then
|
||||
# Apply the local train-template.yaml to the cluster BEFORE submission.
|
||||
# Without this, `argo submit --from=wftmpl/train` runs against whatever
|
||||
# was last applied — defaults can drift from source. Caused workflow
|
||||
# train-jpxvn (2026-05-10) to dispatch with a stale threshold=0.5
|
||||
# default and trigger near-OOM. Multi-seed path already does this; the
|
||||
# single-job path was missing it.
|
||||
kubectl apply -n foxhunt -f infra/k8s/argo/train-template.yaml >/dev/null
|
||||
|
||||
CMD="argo submit -n foxhunt --from=wftmpl/train"
|
||||
CMD="$CMD -p commit-sha=$SHA"
|
||||
CMD="$CMD -p git-branch=$BRANCH"
|
||||
|
||||
Reference in New Issue
Block a user