spec: Argo training pipeline redesign — one workflow, smart caching
Single train template replaces 7 overlapping workflows. Binary cache on PVC by commit SHA, fxcache skip-if-exists. Delete 5 dead templates. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -0,0 +1,202 @@
|
||||
# Argo Training Pipeline Redesign
|
||||
|
||||
## Problem
|
||||
|
||||
The current Argo workflow setup has 7 overlapping templates for training, causing:
|
||||
|
||||
1. **Duplication**: `train-dqn`, `train-baseline-rl`, `train-supervised`, `training-workflow` are near-identical copies of `compile-and-train`
|
||||
2. **Wasted time**: Every training run recompiles from source (~10 min) even when the code hasn't changed
|
||||
3. **Stale binaries**: `precompute-features` reads `/data/bin/precompute_features` which may be from a different commit — caused corrupt fxcache (f64 binary producing f64 cache when f32 was expected)
|
||||
4. **No caching**: Binary and fxcache are rebuilt unconditionally on every invocation
|
||||
5. **Monolithic**: Can't run precompute without also compiling, can't train without also precomputing
|
||||
|
||||
Today's incident: 25 min wasted — `compile-and-deploy` built Docker images (wrong workflow), `precompute-features` used stale binary (wrong format), had to run full `compile-and-train` from scratch.
|
||||
|
||||
## Design
|
||||
|
||||
### One workflow template: `train`
|
||||
|
||||
```
|
||||
argo-train.sh dqn [--sha HEAD] [--gpu-pool ci-training-h100] [--epochs 50]
|
||||
│
|
||||
├─ ensure-binary (CPU: check PVC for /data/bin/{sha}/ → compile if missing)
|
||||
├─ ensure-fxcache (CPU: check /feature-cache/{hash}.fxcache → precompute if missing)
|
||||
├─ gpu-warmup (parallel — autoscale GPU node pool)
|
||||
│
|
||||
└─ train (GPU: depends on all three above)
|
||||
├─ hyperopt (if trials > 0)
|
||||
├─ train-best
|
||||
├─ evaluate
|
||||
└─ upload-results
|
||||
```
|
||||
|
||||
### Cache strategy
|
||||
|
||||
**Binary cache** — PVC at `/data/bin/{sha}/`:
|
||||
- Key: git commit SHA (short, 9 chars)
|
||||
- Contains: `hyperopt_baseline_rl`, `train_baseline_rl`, `evaluate_baseline`, `precompute_features`
|
||||
- Check: `test -x /data/bin/{sha}/precompute_features`
|
||||
- Invalidation: new commit SHA = new directory. Old SHAs cleaned up by TTL (keep last 5).
|
||||
- Compilation: same as current `compile-training` step but outputs to `/data/bin/{sha}/` instead of workspace PVC
|
||||
|
||||
**fxcache** — PVC at `/feature-cache/{hash}.fxcache`:
|
||||
- Key: SHA256 of (raw data file list + sizes + feature extraction code hash)
|
||||
- Already works this way — the precompute binary generates the hash
|
||||
- Check: `test -f /feature-cache/{hash}.fxcache`
|
||||
- If missing: run precompute using `/data/bin/{sha}/precompute_features`
|
||||
|
||||
### PVC layout
|
||||
|
||||
```
|
||||
training-data-pvc (/data):
|
||||
bin/
|
||||
904a158df/ ← binaries for commit 904a158df
|
||||
precompute_features
|
||||
train_baseline_rl
|
||||
hyperopt_baseline_rl
|
||||
evaluate_baseline
|
||||
8919eccbb/ ← binaries for commit 8919eccbb
|
||||
...
|
||||
futures-baseline/ ← raw OHLCV DBN data
|
||||
futures-baseline-mbp10/ ← MBP-10 DBN data
|
||||
futures-baseline-trades/ ← trades DBN data
|
||||
|
||||
feature-cache-pvc (/feature-cache):
|
||||
{hash}.fxcache ← precomputed f32 features
|
||||
{hash}.norm_stats.json ← normalization statistics
|
||||
```
|
||||
|
||||
### Templates to DELETE
|
||||
|
||||
| Template | Replacement |
|
||||
|----------|------------|
|
||||
| `compile-and-train-template.yaml` | Replaced by new `train-template.yaml` |
|
||||
| `train-dqn-template.yaml` | DELETE — unused duplicate |
|
||||
| `train-baseline-rl-template.yaml` | DELETE — unused duplicate |
|
||||
| `train-supervised-template.yaml` | DELETE — unused duplicate |
|
||||
| `training-workflow-template.yaml` | DELETE — unused duplicate |
|
||||
| `precompute-features-template.yaml` | Absorbed into `ensure-fxcache` step in train |
|
||||
|
||||
### Templates to KEEP (unchanged)
|
||||
|
||||
| Template | Purpose |
|
||||
|----------|---------|
|
||||
| `compile-and-deploy-template.yaml` | Service deployment (API, trading, etc.) — separate concern |
|
||||
| `databento-download-template.yaml` | Data acquisition — separate concern |
|
||||
| `gpu-test-pipeline-template.yaml` | CI GPU tests — separate concern |
|
||||
| `build-ci-image-template.yaml` | Docker base image builds — separate concern |
|
||||
|
||||
### Script: `scripts/argo-train.sh`
|
||||
|
||||
Simplified CLI:
|
||||
|
||||
```bash
|
||||
./scripts/argo-train.sh dqn # defaults: HEAD, H100, 50 epochs
|
||||
./scripts/argo-train.sh dqn --sha abc1234 # specific commit
|
||||
./scripts/argo-train.sh dqn --epochs 100 --trials 40 # override training params
|
||||
./scripts/argo-train.sh ppo --gpu-pool ci-training # L40S instead of H100
|
||||
./scripts/argo-train.sh dqn --watch # follow logs
|
||||
```
|
||||
|
||||
One script. One workflow. Model is a parameter, not a separate template.
|
||||
|
||||
### DAG detail
|
||||
|
||||
```yaml
|
||||
templates:
|
||||
- name: pipeline
|
||||
dag:
|
||||
tasks:
|
||||
- name: ensure-binary
|
||||
template: ensure-binary
|
||||
- name: ensure-fxcache
|
||||
template: ensure-fxcache
|
||||
dependencies: [ensure-binary] # needs the precompute binary
|
||||
- name: gpu-warmup
|
||||
template: gpu-warmup
|
||||
- name: hyperopt
|
||||
template: hyperopt
|
||||
dependencies: [ensure-binary, ensure-fxcache, gpu-warmup]
|
||||
when: "{{workflow.parameters.hyperopt-trials}} != 0"
|
||||
- name: train-best
|
||||
template: train-best
|
||||
dependencies: [ensure-binary, ensure-fxcache, gpu-warmup, hyperopt]
|
||||
- name: evaluate
|
||||
template: evaluate
|
||||
dependencies: [train-best]
|
||||
- name: upload-results
|
||||
template: upload-results
|
||||
dependencies: [evaluate]
|
||||
```
|
||||
|
||||
### ensure-binary step (pseudo-bash)
|
||||
|
||||
```bash
|
||||
SHA="{{workflow.parameters.commit-sha}}"
|
||||
BIN_DIR="/data/bin/$SHA"
|
||||
|
||||
if [ -x "$BIN_DIR/precompute_features" ] && [ -x "$BIN_DIR/train_baseline_rl" ]; then
|
||||
echo "Binary cache HIT for $SHA"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "Binary cache MISS for $SHA — compiling..."
|
||||
# Clone, build, copy to $BIN_DIR (same as current compile-training step)
|
||||
git clone ... && cargo build --release ...
|
||||
mkdir -p "$BIN_DIR"
|
||||
cp target/release/{precompute_features,train_baseline_rl,...} "$BIN_DIR/"
|
||||
|
||||
# Cleanup: keep only last 5 SHAs
|
||||
ls -dt /data/bin/*/ | tail -n +6 | xargs rm -rf
|
||||
```
|
||||
|
||||
### ensure-fxcache step (pseudo-bash)
|
||||
|
||||
```bash
|
||||
SHA="{{workflow.parameters.commit-sha}}"
|
||||
BINARY="/data/bin/$SHA/precompute_features"
|
||||
|
||||
# The binary computes its own cache key (hash of data files + code)
|
||||
# and skips writing if the output file already exists
|
||||
$BINARY \
|
||||
--data-dir /data/futures-baseline \
|
||||
--mbp10-data-dir /data/futures-baseline-mbp10 \
|
||||
--trades-data-dir /data/futures-baseline-trades \
|
||||
--output-dir /feature-cache \
|
||||
--symbol ES.FUT \
|
||||
--yes
|
||||
|
||||
# precompute_features already checks if cache exists and skips if valid
|
||||
```
|
||||
|
||||
Note: The precompute binary should gain a `--skip-if-exists` behavior (or already has it via the cache key check). If the fxcache file for the computed hash already exists, it exits immediately.
|
||||
|
||||
### Parameters
|
||||
|
||||
The unified `train` template accepts:
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `commit-sha` | HEAD | Git commit to build/use |
|
||||
| `git-branch` | main | Branch to clone |
|
||||
| `model` | dqn | Model type (dqn, ppo, tft, mamba2, ...) |
|
||||
| `gpu-pool` | ci-training-h100 | GPU node pool |
|
||||
| `hyperopt-trials` | 20 | PSO trials (0 = skip hyperopt) |
|
||||
| `hyperopt-epochs` | 8 | Epochs per hyperopt trial |
|
||||
| `train-epochs` | 50 | Final training epochs |
|
||||
| `symbol` | ES.FUT | Trading symbol |
|
||||
| `initial-capital` | 35000 | Backtest starting capital |
|
||||
|
||||
Data paths are NOT parameters — they're hardcoded to the PVC layout. No reason to make them configurable.
|
||||
|
||||
### What changes for the binary
|
||||
|
||||
The `precompute_features` binary needs one enhancement: **skip if output exists**. Currently it always overwrites. Add logic: if `{output_dir}/{hash}.fxcache` already exists and has the correct size, print "Cache HIT" and exit 0.
|
||||
|
||||
### Migration
|
||||
|
||||
1. Create new `train-template.yaml`
|
||||
2. Update `scripts/argo-train.sh` to submit from new template
|
||||
3. `kubectl apply` the new template
|
||||
4. Delete the 5 dead templates from the repo
|
||||
5. `kubectl delete wftmpl` for each dead template in the cluster
|
||||
Reference in New Issue
Block a user