jgrusewski 14bafb5b58 fix(argo-train): apply train-template.yaml before single-job submission ...

The multi-seed path already does `kubectl apply` before `argo submit`,
so cluster template stays in sync with source. The single-job path used
`argo submit --from=wftmpl/train` directly, expecting the cluster's
template to already match — which silently drifts when defaults change.

Caused workflow train-jpxvn (2026-05-10) to dispatch with stale
imbalance-bar-threshold=0.5 default (the cluster's old value) when the
source had been bumped to 20.0. Triggered near-OOM in feature extraction.

One-line fix: apply the template before submission. Mirrors what
multi-seed already does. No behavior change for users who pass explicit
flags; just makes implicit defaults track source.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-10 17:47:44 +02:00

13 KiB

Executable File

Raw Blame History

View Raw