# Argo fxcache Infrastructure Fix

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Fix the Argo training pipeline so precomputed fxcache is reliably generated once, persisted across runs, and found by ALL training binaries (hyperopt + baseline_rl).

**Architecture:** Create a dedicated `feature-cache-pvc` (RWO, 10Gi, retain) that the precompute step writes to and all training steps read from. Set `FOXHUNT_FEATURE_CACHE_DIR` consistently in every step. Add `--feature-cache-dir` CLI arg to both training binaries. Add `baseline` mode to `argo-train.sh`.

**Tech Stack:** Argo Workflows, Kubernetes PVCs, Rust CLI (clap), Bash

---

## Root Causes

1. **Hyperopt step has no `FOXHUNT_FEATURE_CACHE_DIR`** → falls back to walk-up heuristic → finds stale `/data/feature-cache/` on training-data PVC
2. **Precompute writes to `/workspace/` (ephemeral)** → cache lost when workflow completes → every run regenerates from scratch
3. **Training-data PVC is ReadOnlyMany** → can't write fxcache there
4. **`argo-train.sh` can't skip hyperopt** → forced full pipeline when we just want baseline_rl
5. **Neither `hyperopt_baseline_rl` nor `train_baseline_rl` accept `--feature-cache-dir`** → can't override cache path from CLI

## File Map

| File | Changes |
|------|---------|
| `infra/k8s/argo/feature-cache-pvc.yaml` | **Create**: dedicated PVC for fxcache |
| `infra/k8s/argo/compile-and-train-template.yaml` | Add feature-cache-pvc volume, set env var in ALL steps |
| `infra/k8s/argo/precompute-features-template.yaml` | Use feature-cache-pvc instead of training-data-pvc for writes |
| `scripts/argo-train.sh` | Add `--baseline` flag to skip hyperopt |
| `crates/ml/examples/train_baseline_rl.rs` | Add `--feature-cache-dir` CLI arg |
| `crates/ml/examples/hyperopt_baseline_rl.rs` | Add `--feature-cache-dir` CLI arg |

---

### Task 1: Create feature-cache PVC

**Files:**
- Create: `infra/k8s/argo/feature-cache-pvc.yaml`

- [ ] **Step 1: Create PVC manifest**

```yaml
# Dedicated PVC for precomputed .fxcache files.
# Survives workflow completions (unlike workspace volumeClaimTemplate).
# Precompute writes here, all training steps read from here.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: feature-cache-pvc
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: feature-cache
    app.kubernetes.io/part-of: foxhunt
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: scw-bssd-retain
  resources:
    requests:
      storage: 10Gi
```

- [ ] **Step 2: Apply to cluster**

```bash
kubectl apply -f infra/k8s/argo/feature-cache-pvc.yaml -n foxhunt
kubectl get pvc feature-cache-pvc -n foxhunt
```

Expected: `Bound` status, 10Gi

- [ ] **Step 3: Commit**

```bash
git add infra/k8s/argo/feature-cache-pvc.yaml
git commit -m "infra: add feature-cache-pvc (persistent fxcache storage)"
```

---

### Task 2: Wire feature-cache-pvc into compile-and-train

**Files:**
- Modify: `infra/k8s/argo/compile-and-train-template.yaml`

- [ ] **Step 1: Add volume definition**

In the `volumes:` section (after `cargo-target-cuda`), add:

```yaml
    - name: feature-cache
      persistentVolumeClaim:
        claimName: feature-cache-pvc
```

- [ ] **Step 2: Update `feature-cache-dir` parameter default**

Change line 65-66 from:
```yaml
      - name: feature-cache-dir
        value: /workspace/feature-cache
```
to:
```yaml
      - name: feature-cache-dir
        value: /feature-cache
```

- [ ] **Step 3: Mount feature-cache-pvc in precompute step**

In the `precompute` template, replace the volumeMounts:
```yaml
        volumeMounts:
          - name: training-data
            mountPath: /data
            readOnly: true
          - name: feature-cache
            mountPath: /feature-cache
          - name: workspace
            mountPath: /workspace
```

Update the precompute args to write to `/feature-cache`:
```yaml
            $BINARY \
              --data-dir /data/futures-baseline \
              --mbp10-data-dir /data/futures-baseline-mbp10 \
              --trades-data-dir /data/futures-baseline-trades \
              --output-dir /feature-cache \
              --symbol ES.FUT \
              --yes
```

Note: training-data is now `readOnly: true` in precompute (it only reads DBN files). feature-cache PVC is writable.

- [ ] **Step 4: Mount feature-cache-pvc in hyperopt step + add env var**

In the `hyperopt` template volumeMounts, add:
```yaml
          - name: feature-cache
            mountPath: /feature-cache
            readOnly: true
```

In the `hyperopt` template env, add:
```yaml
          - name: FOXHUNT_FEATURE_CACHE_DIR
            value: "{{workflow.parameters.feature-cache-dir}}"
```

- [ ] **Step 5: Mount feature-cache-pvc in train-best step**

In the `train-best` template volumeMounts, add:
```yaml
          - name: feature-cache
            mountPath: /feature-cache
            readOnly: true
```

The `FOXHUNT_FEATURE_CACHE_DIR` env var is already set in train-best (line 518-519). Verify it uses `{{workflow.parameters.feature-cache-dir}}` which now resolves to `/feature-cache`.

- [ ] **Step 6: Mount feature-cache-pvc in evaluate step**

In the `evaluate` template volumeMounts, add:
```yaml
          - name: feature-cache
            mountPath: /feature-cache
            readOnly: true
```

- [ ] **Step 7: Apply template to cluster and verify**

```bash
kubectl apply -f infra/k8s/argo/compile-and-train-template.yaml -n foxhunt
```

- [ ] **Step 8: Commit**

```bash
git add infra/k8s/argo/compile-and-train-template.yaml
git commit -m "infra: wire feature-cache-pvc into compile-and-train pipeline"
```

---

### Task 3: Update standalone precompute template

**Files:**
- Modify: `infra/k8s/argo/precompute-features-template.yaml`

- [ ] **Step 1: Add feature-cache-pvc volume**

In the `volumes:` section, add:
```yaml
    - name: feature-cache
      persistentVolumeClaim:
        claimName: feature-cache-pvc
```

- [ ] **Step 2: Update default output-dir parameter**

Change:
```yaml
      - name: output-dir
        value: /data/feature-cache
```
to:
```yaml
      - name: output-dir
        value: /feature-cache
```

- [ ] **Step 3: Mount feature-cache-pvc + make training-data readOnly**

Update volumeMounts:
```yaml
        volumeMounts:
          - name: training-data
            mountPath: /data
            readOnly: true
          - name: feature-cache
            mountPath: /feature-cache
```

- [ ] **Step 4: Update binary path to check both PVC and workspace**

Update the args script:
```yaml
          - |
            set -e
            # Binary on PVC (deployed by compile-and-train) or fallback
            BINARY=/data/bin/precompute_features
            if [ ! -x "$BINARY" ]; then
              echo "ERROR: $BINARY not found. Run compile-and-train first to deploy binaries."
              exit 1
            fi
            ...
```

The `OUTPUT_DIR` variable already uses the parameter which now defaults to `/feature-cache`.

- [ ] **Step 5: Apply and commit**

```bash
kubectl apply -f infra/k8s/argo/precompute-features-template.yaml -n foxhunt
git add infra/k8s/argo/precompute-features-template.yaml
git commit -m "infra: precompute writes to feature-cache-pvc (not training-data PVC)"
```

---

### Task 4: Add `--feature-cache-dir` to training binaries

**Files:**
- Modify: `crates/ml/examples/train_baseline_rl.rs`
- Modify: `crates/ml/examples/hyperopt_baseline_rl.rs`

- [ ] **Step 1: Add CLI arg to train_baseline_rl**

In the `Args` struct (find with `struct Args`), add:

```rust
    /// Feature cache directory (overrides FOXHUNT_FEATURE_CACHE_DIR and auto-discovery)
    #[arg(long)]
    feature_cache_dir: Option<String>,
```

In the fxcache discovery code (around line 541-555), prepend:

```rust
    let fxcache_dir: Option<PathBuf> = args.feature_cache_dir.as_ref()
        .map(PathBuf::from)
        .filter(|p| p.exists())
        .or_else(|| {
            std::env::var("FOXHUNT_FEATURE_CACHE_DIR").ok()
                .map(PathBuf::from)
                .filter(|p| p.exists())
        })
        .or_else(|| {
            // walk-up sibling from data_dir
            ...existing walk-up code...
        });
```

- [ ] **Step 2: Add CLI arg to hyperopt_baseline_rl**

Same pattern in the `Args` struct. Then in the adapter setup, call:

```rust
    if let Some(ref dir) = args.feature_cache_dir {
        adapter = adapter.with_feature_cache(PathBuf::from(dir));
    }
```

- [ ] **Step 3: Build and verify**

```bash
SQLX_OFFLINE=true cargo check -p ml --examples
```

- [ ] **Step 4: Commit**

```bash
git add crates/ml/examples/train_baseline_rl.rs crates/ml/examples/hyperopt_baseline_rl.rs
git commit -m "feat: --feature-cache-dir CLI arg for training binaries"
```

---

### Task 5: Add `--baseline` mode to argo-train.sh

**Files:**
- Modify: `scripts/argo-train.sh`

- [ ] **Step 1: Add --baseline flag**

In the argument parsing section, add:

```bash
BASELINE=false
```

In the `while` loop:
```bash
    --baseline)   BASELINE=true; shift ;;
```

- [ ] **Step 2: Use different template when --baseline**

After building the CMD:

```bash
if $BASELINE; then
  # Skip hyperopt — run baseline training directly
  CMD="$CMD -p hyperopt-trials=0"
  echo "  mode:   baseline (hyperopt skipped)"
fi
```

- [ ] **Step 3: Add --feature-cache-dir passthrough**

Add to argument parsing:
```bash
    --cache-dir)  CACHE_DIR="$2"; shift 2 ;;
```

And in CMD building:
```bash
[[ -n "$CACHE_DIR" ]] && CMD="$CMD -p feature-cache-dir=$CACHE_DIR"
```

- [ ] **Step 4: Commit**

```bash
git add scripts/argo-train.sh
git commit -m "feat: argo-train.sh --baseline flag (skips hyperopt) + --cache-dir"
```

---

### Task 6: Clean up stale fxcache on PVC + deploy binaries

**Files:** No code changes — cluster operations only

- [ ] **Step 1: Deploy updated binaries to PVC via compile-and-train**

The compile step in compile-and-train already copies binaries to `$BUILD/bin/training/` on the cargo-target PVC. The precompute template reads from `/data/bin/precompute_features`.

We need to copy binaries from cargo-target PVC to training-data PVC. Create a one-time cleanup job:

```bash
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: pvc-cleanup
  namespace: foxhunt
spec:
  restartPolicy: Never
  nodeSelector:
    k8s.scaleway.com/pool-name: platform
  containers:
    - name: cleanup
      image: ubuntu:24.04
      command: ["/bin/bash", "-c"]
      args:
        - |
          echo "=== Cleaning stale fxcache ==="
          rm -f /data/feature-cache/*.fxcache /data/feature-cache/*.json
          ls -la /data/feature-cache/ 2>/dev/null || echo "dir clean"
          
          echo "=== Deploying binary from cargo-target ==="
          mkdir -p /data/bin
          cp /cargo-target/src/bin/training/precompute_features /data/bin/ 2>/dev/null && echo "Binary deployed" || echo "No binary on cargo-target"
          chmod +x /data/bin/* 2>/dev/null
          ls -la /data/bin/
          
          echo "=== Done ==="
      volumeMounts:
        - name: training-data
          mountPath: /data
        - name: cargo-target
          mountPath: /cargo-target
          readOnly: true
  volumes:
    - name: training-data
      persistentVolumeClaim:
        claimName: training-data-pvc
    - name: cargo-target
      persistentVolumeClaim:
        claimName: cargo-target-cuda
        readOnly: true
EOF
```

Wait for completion:
```bash
kubectl wait --for=condition=Ready pod/pvc-cleanup -n foxhunt --timeout=60s || true
kubectl logs pvc-cleanup -n foxhunt
kubectl delete pod pvc-cleanup -n foxhunt
```

- [ ] **Step 2: Push all changes and rebuild**

```bash
git push origin main
```

Wait for CI to apply templates, then run compile to build new binaries:

```bash
./scripts/argo-compile-deploy.sh --sha $(git rev-parse HEAD)
```

- [ ] **Step 3: Run precompute (standalone) to populate feature-cache-pvc**

```bash
./scripts/argo-precompute.sh ES.FUT --watch
```

Verify: cache written to `/feature-cache/` on the new PVC.

- [ ] **Step 4: Run baseline training**

```bash
./scripts/argo-train.sh dqn --baseline --watch
```

Expected: precompute step says "Cache already exists — skipping", hyperopt skipped (trials=0), train-best uses fxcache from feature-cache-pvc.

---

### Task 7: Verify end-to-end

- [ ] **Step 1: Check precompute skips on second run**

```bash
./scripts/argo-train.sh dqn --baseline
```

The precompute step should output: `Cache already exists: /feature-cache/xxx.fxcache (470 MB) — skipping extraction`

- [ ] **Step 2: Check training uses fxcache**

In the train-best logs, verify:
```
fxcache hit: 1119777 bars from "/feature-cache/xxx.fxcache"
init_from_fxcache: 1119777 bars uploaded to GPU
```

NOT:
```
No fxcache found, falling back to DBN loading
```

- [ ] **Step 3: Check no NaN in training**

Verify epoch logs show finite loss, no `NaN/Inf at step`.
