foxhunt/docs/superpowers/plans/2026-04-03-argo-fxcache-infrastructure-fix.md

Argo fxcache Infrastructure Fix

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Fix the Argo training pipeline so precomputed fxcache is reliably generated once, persisted across runs, and found by ALL training binaries (hyperopt + baseline_rl).

Architecture: Create a dedicated feature-cache-pvc (RWO, 10Gi, retain) that the precompute step writes to and all training steps read from. Set FOXHUNT_FEATURE_CACHE_DIR consistently in every step. Add --feature-cache-dir CLI arg to both training binaries. Add baseline mode to argo-train.sh.

Tech Stack: Argo Workflows, Kubernetes PVCs, Rust CLI (clap), Bash

---

Root Causes

1. Hyperopt step has no FOXHUNT_FEATURE_CACHE_DIR → falls back to walk-up heuristic → finds stale /data/feature-cache/ on training-data PVC
2. Precompute writes to /workspace/ (ephemeral) → cache lost when workflow completes → every run regenerates from scratch
3. Training-data PVC is ReadOnlyMany → can't write fxcache there
4. argo-train.sh can't skip hyperopt → forced full pipeline when we just want baseline_rl
5. Neither hyperopt_baseline_rl nor train_baseline_rl accept --feature-cache-dir → can't override cache path from CLI

File Map

File	Changes
infra/k8s/argo/feature-cache-pvc.yaml	Create: dedicated PVC for fxcache
infra/k8s/argo/compile-and-train-template.yaml	Add feature-cache-pvc volume, set env var in ALL steps
infra/k8s/argo/precompute-features-template.yaml	Use feature-cache-pvc instead of training-data-pvc for writes
scripts/argo-train.sh	Add --baseline flag to skip hyperopt
crates/ml/examples/train_baseline_rl.rs	Add --feature-cache-dir CLI arg
crates/ml/examples/hyperopt_baseline_rl.rs	Add --feature-cache-dir CLI arg

---

Task 1: Create feature-cache PVC

Files:

• Create: infra/k8s/argo/feature-cache-pvc.yaml

• Step 1: Create PVC manifest

# Dedicated PVC for precomputed .fxcache files.
# Survives workflow completions (unlike workspace volumeClaimTemplate).
# Precompute writes here, all training steps read from here.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: feature-cache-pvc
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: feature-cache
    app.kubernetes.io/part-of: foxhunt
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: scw-bssd-retain
  resources:
    requests:
      storage: 10Gi

• Step 2: Apply to cluster

kubectl apply -f infra/k8s/argo/feature-cache-pvc.yaml -n foxhunt
kubectl get pvc feature-cache-pvc -n foxhunt

Expected: Bound status, 10Gi

• Step 3: Commit

git add infra/k8s/argo/feature-cache-pvc.yaml
git commit -m "infra: add feature-cache-pvc (persistent fxcache storage)"

---

Task 2: Wire feature-cache-pvc into compile-and-train

Files:

• Modify: infra/k8s/argo/compile-and-train-template.yaml

• Step 1: Add volume definition

In the volumes: section (after cargo-target-cuda), add:

    - name: feature-cache
      persistentVolumeClaim:
        claimName: feature-cache-pvc

• Step 2: Update feature-cache-dir parameter default

Change line 65-66 from:

      - name: feature-cache-dir
        value: /workspace/feature-cache

to:

      - name: feature-cache-dir
        value: /feature-cache

• Step 3: Mount feature-cache-pvc in precompute step

In the precompute template, replace the volumeMounts:

        volumeMounts:
          - name: training-data
            mountPath: /data
            readOnly: true
          - name: feature-cache
            mountPath: /feature-cache
          - name: workspace
            mountPath: /workspace

Update the precompute args to write to /feature-cache:

            $BINARY \
              --data-dir /data/futures-baseline \
              --mbp10-data-dir /data/futures-baseline-mbp10 \
              --trades-data-dir /data/futures-baseline-trades \
              --output-dir /feature-cache \
              --symbol ES.FUT \
              --yes

Note: training-data is now readOnly: true in precompute (it only reads DBN files). feature-cache PVC is writable.

• Step 4: Mount feature-cache-pvc in hyperopt step + add env var

In the hyperopt template volumeMounts, add:

          - name: feature-cache
            mountPath: /feature-cache
            readOnly: true

In the hyperopt template env, add:

          - name: FOXHUNT_FEATURE_CACHE_DIR
            value: "{{workflow.parameters.feature-cache-dir}}"

• Step 5: Mount feature-cache-pvc in train-best step

In the train-best template volumeMounts, add:

          - name: feature-cache
            mountPath: /feature-cache
            readOnly: true

The FOXHUNT_FEATURE_CACHE_DIR env var is already set in train-best (line 518-519). Verify it uses {{workflow.parameters.feature-cache-dir}} which now resolves to /feature-cache.

• Step 6: Mount feature-cache-pvc in evaluate step

In the evaluate template volumeMounts, add:

          - name: feature-cache
            mountPath: /feature-cache
            readOnly: true

• Step 7: Apply template to cluster and verify

kubectl apply -f infra/k8s/argo/compile-and-train-template.yaml -n foxhunt

• Step 8: Commit

git add infra/k8s/argo/compile-and-train-template.yaml
git commit -m "infra: wire feature-cache-pvc into compile-and-train pipeline"

---

Task 3: Update standalone precompute template

Files:

• Modify: infra/k8s/argo/precompute-features-template.yaml

• Step 1: Add feature-cache-pvc volume

In the volumes: section, add:

    - name: feature-cache
      persistentVolumeClaim:
        claimName: feature-cache-pvc

• Step 2: Update default output-dir parameter

Change:

      - name: output-dir
        value: /data/feature-cache

to:

      - name: output-dir
        value: /feature-cache

• Step 3: Mount feature-cache-pvc + make training-data readOnly

Update volumeMounts:

        volumeMounts:
          - name: training-data
            mountPath: /data
            readOnly: true
          - name: feature-cache
            mountPath: /feature-cache

• Step 4: Update binary path to check both PVC and workspace

Update the args script:

          - |
            set -e
            # Binary on PVC (deployed by compile-and-train) or fallback
            BINARY=/data/bin/precompute_features
            if [ ! -x "$BINARY" ]; then
              echo "ERROR: $BINARY not found. Run compile-and-train first to deploy binaries."
              exit 1
            fi
            ...

The OUTPUT_DIR variable already uses the parameter which now defaults to /feature-cache.

• Step 5: Apply and commit

kubectl apply -f infra/k8s/argo/precompute-features-template.yaml -n foxhunt
git add infra/k8s/argo/precompute-features-template.yaml
git commit -m "infra: precompute writes to feature-cache-pvc (not training-data PVC)"

---

Task 4: Add --feature-cache-dir to training binaries

Files:

• Modify: crates/ml/examples/train_baseline_rl.rs

• Modify: crates/ml/examples/hyperopt_baseline_rl.rs

• Step 1: Add CLI arg to train_baseline_rl

In the Args struct (find with struct Args), add:

    /// Feature cache directory (overrides FOXHUNT_FEATURE_CACHE_DIR and auto-discovery)
    #[arg(long)]
    feature_cache_dir: Option<String>,

In the fxcache discovery code (around line 541-555), prepend:

    let fxcache_dir: Option<PathBuf> = args.feature_cache_dir.as_ref()
        .map(PathBuf::from)
        .filter(|p| p.exists())
        .or_else(|| {
            std::env::var("FOXHUNT_FEATURE_CACHE_DIR").ok()
                .map(PathBuf::from)
                .filter(|p| p.exists())
        })
        .or_else(|| {
            // walk-up sibling from data_dir
            ...existing walk-up code...
        });

• Step 2: Add CLI arg to hyperopt_baseline_rl

Same pattern in the Args struct. Then in the adapter setup, call:

    if let Some(ref dir) = args.feature_cache_dir {
        adapter = adapter.with_feature_cache(PathBuf::from(dir));
    }

• Step 3: Build and verify

SQLX_OFFLINE=true cargo check -p ml --examples

• Step 4: Commit

git add crates/ml/examples/train_baseline_rl.rs crates/ml/examples/hyperopt_baseline_rl.rs
git commit -m "feat: --feature-cache-dir CLI arg for training binaries"

---

Task 5: Add --baseline mode to argo-train.sh

Files:

• Modify: scripts/argo-train.sh

• Step 1: Add --baseline flag

In the argument parsing section, add:

BASELINE=false

In the while loop:

    --baseline)   BASELINE=true; shift ;;

• Step 2: Use different template when --baseline

After building the CMD:

if $BASELINE; then
  # Skip hyperopt — run baseline training directly
  CMD="$CMD -p hyperopt-trials=0"
  echo "  mode:   baseline (hyperopt skipped)"
fi

• Step 3: Add --feature-cache-dir passthrough

Add to argument parsing:

    --cache-dir)  CACHE_DIR="$2"; shift 2 ;;

And in CMD building:

[[ -n "$CACHE_DIR" ]] && CMD="$CMD -p feature-cache-dir=$CACHE_DIR"

• Step 4: Commit

git add scripts/argo-train.sh
git commit -m "feat: argo-train.sh --baseline flag (skips hyperopt) + --cache-dir"

---

Task 6: Clean up stale fxcache on PVC + deploy binaries

Files: No code changes — cluster operations only

• Step 1: Deploy updated binaries to PVC via compile-and-train

The compile step in compile-and-train already copies binaries to $BUILD/bin/training/ on the cargo-target PVC. The precompute template reads from /data/bin/precompute_features.

We need to copy binaries from cargo-target PVC to training-data PVC. Create a one-time cleanup job:

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: pvc-cleanup
  namespace: foxhunt
spec:
  restartPolicy: Never
  nodeSelector:
    k8s.scaleway.com/pool-name: platform
  containers:
    - name: cleanup
      image: ubuntu:24.04
      command: ["/bin/bash", "-c"]
      args:
        - |
          echo "=== Cleaning stale fxcache ==="
          rm -f /data/feature-cache/*.fxcache /data/feature-cache/*.json
          ls -la /data/feature-cache/ 2>/dev/null || echo "dir clean"
          
          echo "=== Deploying binary from cargo-target ==="
          mkdir -p /data/bin
          cp /cargo-target/src/bin/training/precompute_features /data/bin/ 2>/dev/null && echo "Binary deployed" || echo "No binary on cargo-target"
          chmod +x /data/bin/* 2>/dev/null
          ls -la /data/bin/
          
          echo "=== Done ==="
      volumeMounts:
        - name: training-data
          mountPath: /data
        - name: cargo-target
          mountPath: /cargo-target
          readOnly: true
  volumes:
    - name: training-data
      persistentVolumeClaim:
        claimName: training-data-pvc
    - name: cargo-target
      persistentVolumeClaim:
        claimName: cargo-target-cuda
        readOnly: true
EOF

Wait for completion:

kubectl wait --for=condition=Ready pod/pvc-cleanup -n foxhunt --timeout=60s || true
kubectl logs pvc-cleanup -n foxhunt
kubectl delete pod pvc-cleanup -n foxhunt

• Step 2: Push all changes and rebuild

git push origin main

Wait for CI to apply templates, then run compile to build new binaries:

./scripts/argo-compile-deploy.sh --sha $(git rev-parse HEAD)

• Step 3: Run precompute (standalone) to populate feature-cache-pvc

./scripts/argo-precompute.sh ES.FUT --watch

Verify: cache written to /feature-cache/ on the new PVC.

• Step 4: Run baseline training

./scripts/argo-train.sh dqn --baseline --watch

Expected: precompute step says "Cache already exists — skipping", hyperopt skipped (trials=0), train-best uses fxcache from feature-cache-pvc.

---

Task 7: Verify end-to-end

• Step 1: Check precompute skips on second run

./scripts/argo-train.sh dqn --baseline

The precompute step should output: Cache already exists: /feature-cache/xxx.fxcache (470 MB) — skipping extraction

• Step 2: Check training uses fxcache

In the train-best logs, verify:

fxcache hit: 1119777 bars from "/feature-cache/xxx.fxcache"
init_from_fxcache: 1119777 bars uploaded to GPU

NOT:

No fxcache found, falling back to DBN loading

• Step 3: Check no NaN in training

Verify epoch logs show finite loss, no NaN/Inf at step.