From 8c92db09f3c15e1073ea6fbec55fc0a2adc46d92 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Fri, 10 Apr 2026 20:26:55 +0200 Subject: [PATCH] =?UTF-8?q?spec:=20Argo=20training=20pipeline=20redesign?= =?UTF-8?q?=20=E2=80=94=20one=20workflow,=20smart=20caching?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Single train template replaces 7 overlapping workflows. Binary cache on PVC by commit SHA, fxcache skip-if-exists. Delete 5 dead templates. Co-Authored-By: Claude Opus 4.6 (1M context) --- ...6-04-10-argo-training-pipeline-redesign.md | 202 ++++++++++++++++++ 1 file changed, 202 insertions(+) create mode 100644 docs/superpowers/specs/2026-04-10-argo-training-pipeline-redesign.md diff --git a/docs/superpowers/specs/2026-04-10-argo-training-pipeline-redesign.md b/docs/superpowers/specs/2026-04-10-argo-training-pipeline-redesign.md new file mode 100644 index 000000000..cb46c0704 --- /dev/null +++ b/docs/superpowers/specs/2026-04-10-argo-training-pipeline-redesign.md @@ -0,0 +1,202 @@ +# Argo Training Pipeline Redesign + +## Problem + +The current Argo workflow setup has 7 overlapping templates for training, causing: + +1. **Duplication**: `train-dqn`, `train-baseline-rl`, `train-supervised`, `training-workflow` are near-identical copies of `compile-and-train` +2. **Wasted time**: Every training run recompiles from source (~10 min) even when the code hasn't changed +3. **Stale binaries**: `precompute-features` reads `/data/bin/precompute_features` which may be from a different commit — caused corrupt fxcache (f64 binary producing f64 cache when f32 was expected) +4. **No caching**: Binary and fxcache are rebuilt unconditionally on every invocation +5. **Monolithic**: Can't run precompute without also compiling, can't train without also precomputing + +Today's incident: 25 min wasted — `compile-and-deploy` built Docker images (wrong workflow), `precompute-features` used stale binary (wrong format), had to run full `compile-and-train` from scratch. + +## Design + +### One workflow template: `train` + +``` +argo-train.sh dqn [--sha HEAD] [--gpu-pool ci-training-h100] [--epochs 50] + │ + ├─ ensure-binary (CPU: check PVC for /data/bin/{sha}/ → compile if missing) + ├─ ensure-fxcache (CPU: check /feature-cache/{hash}.fxcache → precompute if missing) + ├─ gpu-warmup (parallel — autoscale GPU node pool) + │ + └─ train (GPU: depends on all three above) + ├─ hyperopt (if trials > 0) + ├─ train-best + ├─ evaluate + └─ upload-results +``` + +### Cache strategy + +**Binary cache** — PVC at `/data/bin/{sha}/`: +- Key: git commit SHA (short, 9 chars) +- Contains: `hyperopt_baseline_rl`, `train_baseline_rl`, `evaluate_baseline`, `precompute_features` +- Check: `test -x /data/bin/{sha}/precompute_features` +- Invalidation: new commit SHA = new directory. Old SHAs cleaned up by TTL (keep last 5). +- Compilation: same as current `compile-training` step but outputs to `/data/bin/{sha}/` instead of workspace PVC + +**fxcache** — PVC at `/feature-cache/{hash}.fxcache`: +- Key: SHA256 of (raw data file list + sizes + feature extraction code hash) +- Already works this way — the precompute binary generates the hash +- Check: `test -f /feature-cache/{hash}.fxcache` +- If missing: run precompute using `/data/bin/{sha}/precompute_features` + +### PVC layout + +``` +training-data-pvc (/data): + bin/ + 904a158df/ ← binaries for commit 904a158df + precompute_features + train_baseline_rl + hyperopt_baseline_rl + evaluate_baseline + 8919eccbb/ ← binaries for commit 8919eccbb + ... + futures-baseline/ ← raw OHLCV DBN data + futures-baseline-mbp10/ ← MBP-10 DBN data + futures-baseline-trades/ ← trades DBN data + +feature-cache-pvc (/feature-cache): + {hash}.fxcache ← precomputed f32 features + {hash}.norm_stats.json ← normalization statistics +``` + +### Templates to DELETE + +| Template | Replacement | +|----------|------------| +| `compile-and-train-template.yaml` | Replaced by new `train-template.yaml` | +| `train-dqn-template.yaml` | DELETE — unused duplicate | +| `train-baseline-rl-template.yaml` | DELETE — unused duplicate | +| `train-supervised-template.yaml` | DELETE — unused duplicate | +| `training-workflow-template.yaml` | DELETE — unused duplicate | +| `precompute-features-template.yaml` | Absorbed into `ensure-fxcache` step in train | + +### Templates to KEEP (unchanged) + +| Template | Purpose | +|----------|---------| +| `compile-and-deploy-template.yaml` | Service deployment (API, trading, etc.) — separate concern | +| `databento-download-template.yaml` | Data acquisition — separate concern | +| `gpu-test-pipeline-template.yaml` | CI GPU tests — separate concern | +| `build-ci-image-template.yaml` | Docker base image builds — separate concern | + +### Script: `scripts/argo-train.sh` + +Simplified CLI: + +```bash +./scripts/argo-train.sh dqn # defaults: HEAD, H100, 50 epochs +./scripts/argo-train.sh dqn --sha abc1234 # specific commit +./scripts/argo-train.sh dqn --epochs 100 --trials 40 # override training params +./scripts/argo-train.sh ppo --gpu-pool ci-training # L40S instead of H100 +./scripts/argo-train.sh dqn --watch # follow logs +``` + +One script. One workflow. Model is a parameter, not a separate template. + +### DAG detail + +```yaml +templates: + - name: pipeline + dag: + tasks: + - name: ensure-binary + template: ensure-binary + - name: ensure-fxcache + template: ensure-fxcache + dependencies: [ensure-binary] # needs the precompute binary + - name: gpu-warmup + template: gpu-warmup + - name: hyperopt + template: hyperopt + dependencies: [ensure-binary, ensure-fxcache, gpu-warmup] + when: "{{workflow.parameters.hyperopt-trials}} != 0" + - name: train-best + template: train-best + dependencies: [ensure-binary, ensure-fxcache, gpu-warmup, hyperopt] + - name: evaluate + template: evaluate + dependencies: [train-best] + - name: upload-results + template: upload-results + dependencies: [evaluate] +``` + +### ensure-binary step (pseudo-bash) + +```bash +SHA="{{workflow.parameters.commit-sha}}" +BIN_DIR="/data/bin/$SHA" + +if [ -x "$BIN_DIR/precompute_features" ] && [ -x "$BIN_DIR/train_baseline_rl" ]; then + echo "Binary cache HIT for $SHA" + exit 0 +fi + +echo "Binary cache MISS for $SHA — compiling..." +# Clone, build, copy to $BIN_DIR (same as current compile-training step) +git clone ... && cargo build --release ... +mkdir -p "$BIN_DIR" +cp target/release/{precompute_features,train_baseline_rl,...} "$BIN_DIR/" + +# Cleanup: keep only last 5 SHAs +ls -dt /data/bin/*/ | tail -n +6 | xargs rm -rf +``` + +### ensure-fxcache step (pseudo-bash) + +```bash +SHA="{{workflow.parameters.commit-sha}}" +BINARY="/data/bin/$SHA/precompute_features" + +# The binary computes its own cache key (hash of data files + code) +# and skips writing if the output file already exists +$BINARY \ + --data-dir /data/futures-baseline \ + --mbp10-data-dir /data/futures-baseline-mbp10 \ + --trades-data-dir /data/futures-baseline-trades \ + --output-dir /feature-cache \ + --symbol ES.FUT \ + --yes + +# precompute_features already checks if cache exists and skips if valid +``` + +Note: The precompute binary should gain a `--skip-if-exists` behavior (or already has it via the cache key check). If the fxcache file for the computed hash already exists, it exits immediately. + +### Parameters + +The unified `train` template accepts: + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `commit-sha` | HEAD | Git commit to build/use | +| `git-branch` | main | Branch to clone | +| `model` | dqn | Model type (dqn, ppo, tft, mamba2, ...) | +| `gpu-pool` | ci-training-h100 | GPU node pool | +| `hyperopt-trials` | 20 | PSO trials (0 = skip hyperopt) | +| `hyperopt-epochs` | 8 | Epochs per hyperopt trial | +| `train-epochs` | 50 | Final training epochs | +| `symbol` | ES.FUT | Trading symbol | +| `initial-capital` | 35000 | Backtest starting capital | + +Data paths are NOT parameters — they're hardcoded to the PVC layout. No reason to make them configurable. + +### What changes for the binary + +The `precompute_features` binary needs one enhancement: **skip if output exists**. Currently it always overwrites. Add logic: if `{output_dir}/{hash}.fxcache` already exists and has the correct size, print "Cache HIT" and exit 0. + +### Migration + +1. Create new `train-template.yaml` +2. Update `scripts/argo-train.sh` to submit from new template +3. `kubectl apply` the new template +4. Delete the 5 dead templates from the repo +5. `kubectl delete wftmpl` for each dead template in the cluster