# Argo Training Pipeline Redesign

## Problem

The current Argo workflow setup has 7 overlapping templates for training, causing:

1. **Duplication**: `train-dqn`, `train-baseline-rl`, `train-supervised`, `training-workflow` are near-identical copies of `compile-and-train`
2. **Wasted time**: Every training run recompiles from source (~10 min) even when the code hasn't changed
3. **Stale binaries**: `precompute-features` reads `/data/bin/precompute_features` which may be from a different commit — caused corrupt fxcache (f64 binary producing f64 cache when f32 was expected)
4. **No caching**: Binary and fxcache are rebuilt unconditionally on every invocation
5. **Monolithic**: Can't run precompute without also compiling, can't train without also precomputing

Today's incident: 25 min wasted — `compile-and-deploy` built Docker images (wrong workflow), `precompute-features` used stale binary (wrong format), had to run full `compile-and-train` from scratch.

## Design

### One workflow template: `train`

```
argo-train.sh dqn [--sha HEAD] [--gpu-pool ci-training-h100] [--epochs 50]
    │
    ├─ ensure-binary    (CPU: check PVC for /data/bin/{sha}/ → compile if missing)
    ├─ ensure-fxcache   (CPU: check /feature-cache/{hash}.fxcache → precompute if missing)
    ├─ gpu-warmup       (parallel — autoscale GPU node pool)
    │
    └─ train            (GPU: depends on all three above)
         ├─ hyperopt    (if trials > 0)
         ├─ train-best
         ├─ evaluate
         └─ upload-results
```

### Cache strategy

**Binary cache** — PVC at `/data/bin/{sha}/`:
- Key: git commit SHA (short, 9 chars)
- Contains: `hyperopt_baseline_rl`, `train_baseline_rl`, `evaluate_baseline`, `precompute_features`
- Check: `test -x /data/bin/{sha}/precompute_features`
- Invalidation: new commit SHA = new directory. Old SHAs cleaned up by TTL (keep last 5).
- Compilation: same as current `compile-training` step but outputs to `/data/bin/{sha}/` instead of workspace PVC

**fxcache** — PVC at `/feature-cache/{hash}.fxcache`:
- Key: SHA256 of (raw data file list + sizes + feature extraction code hash)
- Already works this way — the precompute binary generates the hash
- Check: `test -f /feature-cache/{hash}.fxcache`
- If missing: run precompute using `/data/bin/{sha}/precompute_features`

### PVC layout

```
training-data-pvc (/data):
  bin/
    904a158df/           ← binaries for commit 904a158df
      precompute_features
      train_baseline_rl
      hyperopt_baseline_rl
      evaluate_baseline
    8919eccbb/           ← binaries for commit 8919eccbb
      ...
  futures-baseline/      ← raw OHLCV DBN data
  futures-baseline-mbp10/ ← MBP-10 DBN data
  futures-baseline-trades/ ← trades DBN data

feature-cache-pvc (/feature-cache):
  {hash}.fxcache         ← precomputed f32 features
  {hash}.norm_stats.json ← normalization statistics
```

### Templates to DELETE

| Template | Replacement |
|----------|------------|
| `compile-and-train-template.yaml` | Replaced by new `train-template.yaml` |
| `train-dqn-template.yaml` | DELETE — unused duplicate |
| `train-baseline-rl-template.yaml` | DELETE — unused duplicate |
| `train-supervised-template.yaml` | DELETE — unused duplicate |
| `training-workflow-template.yaml` | DELETE — unused duplicate |
| `precompute-features-template.yaml` | Absorbed into `ensure-fxcache` step in train |

### Templates to KEEP (unchanged)

| Template | Purpose |
|----------|---------|
| `compile-and-deploy-template.yaml` | Service deployment (API, trading, etc.) — separate concern |
| `databento-download-template.yaml` | Data acquisition — separate concern |
| `gpu-test-pipeline-template.yaml` | CI GPU tests — separate concern |
| `build-ci-image-template.yaml` | Docker base image builds — separate concern |

### Script: `scripts/argo-train.sh`

Simplified CLI:

```bash
./scripts/argo-train.sh dqn                          # defaults: HEAD, H100, 50 epochs
./scripts/argo-train.sh dqn --sha abc1234            # specific commit
./scripts/argo-train.sh dqn --epochs 100 --trials 40 # override training params
./scripts/argo-train.sh ppo --gpu-pool ci-training    # L40S instead of H100
./scripts/argo-train.sh dqn --watch                  # follow logs
```

One script. One workflow. Model is a parameter, not a separate template.

### DAG detail

```yaml
templates:
  - name: pipeline
    dag:
      tasks:
        - name: ensure-binary
          template: ensure-binary
        - name: ensure-fxcache
          template: ensure-fxcache
          dependencies: [ensure-binary]  # needs the precompute binary
        - name: gpu-warmup
          template: gpu-warmup
        - name: hyperopt
          template: hyperopt
          dependencies: [ensure-binary, ensure-fxcache, gpu-warmup]
          when: "{{workflow.parameters.hyperopt-trials}} != 0"
        - name: train-best
          template: train-best
          dependencies: [ensure-binary, ensure-fxcache, gpu-warmup, hyperopt]
        - name: evaluate
          template: evaluate
          dependencies: [train-best]
        - name: upload-results
          template: upload-results
          dependencies: [evaluate]
```

### ensure-binary step (pseudo-bash)

```bash
SHA="{{workflow.parameters.commit-sha}}"
BIN_DIR="/data/bin/$SHA"

if [ -x "$BIN_DIR/precompute_features" ] && [ -x "$BIN_DIR/train_baseline_rl" ]; then
    echo "Binary cache HIT for $SHA"
    exit 0
fi

echo "Binary cache MISS for $SHA — compiling..."
# Clone, build, copy to $BIN_DIR (same as current compile-training step)
git clone ... && cargo build --release ...
mkdir -p "$BIN_DIR"
cp target/release/{precompute_features,train_baseline_rl,...} "$BIN_DIR/"

# Cleanup: keep only last 5 SHAs
ls -dt /data/bin/*/ | tail -n +6 | xargs rm -rf
```

### ensure-fxcache step (pseudo-bash)

```bash
SHA="{{workflow.parameters.commit-sha}}"
BINARY="/data/bin/$SHA/precompute_features"

# The binary computes its own cache key (hash of data files + code)
# and skips writing if the output file already exists
$BINARY \
  --data-dir /data/futures-baseline \
  --mbp10-data-dir /data/futures-baseline-mbp10 \
  --trades-data-dir /data/futures-baseline-trades \
  --output-dir /feature-cache \
  --symbol ES.FUT \
  --yes

# precompute_features already checks if cache exists and skips if valid
```

Note: The precompute binary should gain a `--skip-if-exists` behavior (or already has it via the cache key check). If the fxcache file for the computed hash already exists, it exits immediately.

### Parameters

The unified `train` template accepts:

| Parameter | Default | Description |
|-----------|---------|-------------|
| `commit-sha` | HEAD | Git commit to build/use |
| `git-branch` | main | Branch to clone |
| `model` | dqn | Model type (dqn, ppo, tft, mamba2, ...) |
| `gpu-pool` | ci-training-h100 | GPU node pool |
| `hyperopt-trials` | 20 | PSO trials (0 = skip hyperopt) |
| `hyperopt-epochs` | 8 | Epochs per hyperopt trial |
| `train-epochs` | 50 | Final training epochs |
| `symbol` | ES.FUT | Trading symbol |
| `initial-capital` | 35000 | Backtest starting capital |

Data paths are NOT parameters — they're hardcoded to the PVC layout. No reason to make them configurable.

### What changes for the binary

The `precompute_features` binary needs one enhancement: **skip if output exists**. Currently it always overwrites. Add logic: if `{output_dir}/{hash}.fxcache` already exists and has the correct size, print "Cache HIT" and exit 0.

### Migration

1. Create new `train-template.yaml`
2. Update `scripts/argo-train.sh` to submit from new template
3. `kubectl apply` the new template
4. Delete the 5 dead templates from the repo
5. `kubectl delete wftmpl` for each dead template in the cluster
