c6634254e4
Adds the orchestration surface for Plan 5 Task 1 — Multi-Seed × Multi-Fold Validation Harness: - scripts/argo-train.sh: new --multi-seed N, --folds K, --tag T, --dry-run flags. Default --multi-seed 1 --folds 1 routes to the existing train-template.yaml (backward compat — existing callers unchanged). When N>1 or K>1 the script renders train-multi-seed-template.yaml with an inline-generated (seed, fold) matrix and either prints the YAML (--dry-run) or applies + submits it. - infra/k8s/argo/train-multi-seed-template.yaml: new WorkflowTemplate with entrypoint multi-seed-matrix → ensure-binary, gpu-warmup, ensure-fxcache, then N*K parallel train-single instances. Each train-single receives seed/fold via inputs.parameters and forwards them to the training binary via --seed/--fold CLI args + SEED/FOLD env vars. The dag.tasks placeholder `# __MATRIX_TASKS__` is substituted programmatically by argo-train.sh (awk) — no hand-written 30-task matrix. - scripts/tests/test_multi_seed_harness.sh: dry-run regression test. Asserts --multi-seed 3 --folds 2 emits 6 WorkflowTask markers AND --multi-seed 1 --folds 1 emits zero (single-template path preserved). Validation: - argo lint --offline passes on both the source template and the rendered 3x2 / 5x6 outputs. - test_multi_seed_harness.sh passes locally. - Single-job dry-run still produces the unchanged train-template YAML. Note: plan Step 0.1 pre-plan check expects ISV_TOTAL_DIM=72 and seven ATTN_*_FOCUS_EMA_INDEX slots — both stale (Plan 4 landed ISV_TOTAL_DIM=117 and the VSN_MAG_EMA / VSN_DIR_EMA / MAMBA2_RETENTION_EMA slots instead). Plan 4 validation doc never landed (T8 deferred → Plan 5 T5). T1 is pure infrastructure that builds the harness consumed by T5, so the stale pre-plan expectations do not block this commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
36 lines
1.5 KiB
Bash
Executable File
36 lines
1.5 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Test: argo-train.sh --multi-seed N --folds K --dry-run emits one
|
|
# WorkflowTask per (seed, fold) pair (N * K total).
|
|
#
|
|
# This is the validation surface for the Plan 5 Task 1A multi-seed Argo DAG —
|
|
# verifies the matrix is generated correctly without submitting to a live
|
|
# cluster.
|
|
set -euo pipefail
|
|
|
|
# Resolve repo root from this script's location so the test runs from any cwd.
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
|
|
cd "${REPO_ROOT}"
|
|
|
|
# Dry-run: --multi-seed 3 --folds 2 should emit a workflow with 6 training jobs.
|
|
output=$(./scripts/argo-train.sh dqn --multi-seed 3 --folds 2 --dry-run 2>&1)
|
|
actual_count=$(echo "$output" | grep -c "kind: WorkflowTask" || true)
|
|
expected=6
|
|
if [[ "$actual_count" -ne "$expected" ]]; then
|
|
echo "FAIL: expected $expected jobs (3 seeds x 2 folds), got $actual_count"
|
|
echo "--- output ---"
|
|
echo "$output"
|
|
exit 1
|
|
fi
|
|
echo "PASS: multi-seed harness produces $expected jobs for 3x2 matrix"
|
|
|
|
# Backward-compat: --multi-seed 1 --folds 1 must NOT emit WorkflowTask lines
|
|
# (single-job path uses the existing template, no DAG matrix).
|
|
output_single=$(./scripts/argo-train.sh dqn --multi-seed 1 --folds 1 --dry-run 2>&1)
|
|
single_tasks=$(echo "$output_single" | grep -c "kind: WorkflowTask" || true)
|
|
if [[ "$single_tasks" -ne 0 ]]; then
|
|
echo "FAIL: single-job dry-run should not emit multi-seed DAG tasks (got $single_tasks)"
|
|
exit 1
|
|
fi
|
|
echo "PASS: single-job dry-run preserves existing single-template path"
|