Closes the operational gap from P6: when a sweep grid YAML carries
sim_variants, argo-lob-sweep.sh now emits ONE run-sweep-batched task
instead of N fan-out run-cell tasks. The new run-sweep template
invokes `fxt-backtest sweep` against the full grid (base64-encoded
inline as Argo parameter to avoid YAML special-char encoding traps).
The binary's P6 sweep() function handles cell × variant fan-out
internally via BatchedSimConfig::from_grid, so one pod processes all
4 windows × 140 variants sequentially. Trade-off: no inter-window
parallelism in this rev (4 quarters sequential in one pod ≈ firm-bound
2h wall per spec §9). 3-pod scale-out is P7 future work — needs the
script to split the YAML into per-window sub-grids and emit one
run-sweep task per sub-grid.
Backward-compat: legacy (no sim_variants) flow unchanged. The dry-run
of sweep_smoke.yaml continues to emit run-cell-* fan-out tasks; the
dry-run of sweep_deployability.yaml emits one run-sweep-batched task.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
argo-lob-sweep.sh's awk substitution for # __SWEEP_CELLS__ matched BOTH
the actual marker (line 110, whitespace-indented) AND the docstring
reference at the top of the template that mentions `# __SWEEP_CELLS__`
inside a sentence (line 8). Result: cell task was injected twice,
once before `apiVersion:` (breaking kubectl apply with 'invalid object'
validation error) and once at the correct DAG location.
Fix: anchor the match to lines that START with whitespace + the marker
(^[[:space:]]+# __SWEEP_CELLS__), so the docstring sentence (which has
'# The' first) no longer matches.
sweep_smoke.yaml: rewrite from the spec's idealised (axes:/windows:)
schema to the actual base:/cells: schema that fxt-backtest sweep +
argo-lob-sweep.sh parse. Hardcodes SHA 58b5ebbd3 (the production
training run); bounds smoke runtime via max_events=100000.
Verified: smoke sweep workflow lob-backtest-sweep-hxwmq submits cleanly
and starts running.
Cluster fan-out for the `fxt-backtest sweep` single-machine path.
Reads the same grid YAML format as the binary; runs each cell on a
dedicated GPU pod in parallel; aggregates at the end on a CPU pod.
infra/k8s/argo/lob-backtest-sweep-template.yaml:
WorkflowTemplate `lob-backtest-sweep` with three job templates:
ensure-binary — cache-or-compile fxt-backtest by short-SHA into
/mnt/training-data/bin/<sha>/. Mirrors the
train-multi-seed-template.yaml ensure-binary
shape but for a single binary.
run-cell — single GPU pod (ci-training-l40s default per
feedback_default_to_l40s_pool.md). Receives
cell-name + every Run arg via inputs.parameters.
Writes artifacts to <sweep-root>/<sweep-tag>/<cell>/.
aggregate — CPU pod runs `fxt-backtest aggregate <sweep-dir>`
producing aggregate.parquet + pareto_frontier.json
at the sweep root.
DAG marker `# __SWEEP_CELLS__` replaced at submission time with N
WorkflowTask stanzas (one per cell), and the aggregate's
`dependencies: [ensure-binary]` is rewritten to include every
run-cell-* dep — so aggregate waits for ALL cells.
scripts/argo-lob-sweep.sh:
Companion submission script following the argo-train.sh pattern.
Parses the grid YAML via python3 + PyYAML (no `yq` dependency —
yq isn't used elsewhere in foxhunt scripts; python3+PyYAML is
universal in our CI images). Emits per-cell WorkflowTask stanzas
+ aggregate dependency list, awks them into the template, then
`kubectl apply` + `argo submit`. Supports --dry-run for offline
rendering and --watch for live log following.
Defaults match the spec / pearl set:
- sm_89 / ci-training-l40s default (override via --gpu-pool
ci-training-h100 for sm_90 + 80 GB)
- data root /mnt/training-data/futures-baseline/ES.FUT
- sweep results under /mnt/training-data/sweeps/lob-backtest/<tag>/
- sweep-tag defaults to <basename of grid>-<short-sha>
Verified locally:
- bash -n syntax-check passes
- --help renders
- --dry-run against the existing
config/ml/sweep_decision_stride_example.yaml renders a valid
workflow with 4 cells + correct aggregate dependency list
Live submission is operational work that needs cluster access to
verify; the rendered YAML follows the same conventions as the
existing argo-train.sh workflows that ship in this repo.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
