# Argo Workflows Phase 1 — Install & Validate Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Install Argo Workflows controller in the `foxhunt` Kapsule cluster and validate a manual DQN training workflow end-to-end.

**Architecture:** Argo Workflows controller runs on the `platform` node (alongside GitLab). A single parameterized WorkflowTemplate orchestrates the fetch→hyperopt→train→evaluate→upload pipeline. MinIO serves as the native Argo artifact repository. The Argo Server UI is exposed via `argo.fxhnt.ai` through the existing Tailscale nginx proxy.

**Tech Stack:** Argo Workflows v3.6 (Helm), Kubernetes 1.31 (Scaleway Kapsule), MinIO S3, nginx reverse proxy, Terraform (Scaleway DNS)

---

### Task 1: Create Argo Workflows Helm values file

This task creates the Helm values file that configures the Argo Workflows controller for the `foxhunt` namespace. The controller runs on the `platform` node pool alongside GitLab (low resource usage). Artifact repository points to the existing in-cluster MinIO.

**Files:**
- Create: `infra/k8s/argo/values.yaml`

**Step 1: Create the directory**

```bash
mkdir -p infra/k8s/argo
```

**Step 2: Write the Helm values file**

Create `infra/k8s/argo/values.yaml`:

```yaml
# Argo Workflows Helm values — foxhunt cluster
# Chart: argo/argo-workflows (v0.45.x maps to Argo v3.6.x)
# Install: helm install argo-workflows argo/argo-workflows -n foxhunt -f infra/k8s/argo/values.yaml

# Controller runs on platform node (low overhead, ~100Mi idle)
controller:
  nodeSelector:
    k8s.scaleway.com/pool-name: platform
  tolerations:
    - key: gitlab
      operator: Equal
      value: "true"
      effect: NoSchedule
  resources:
    requests:
      cpu: 50m
      memory: 128Mi
    limits:
      cpu: 200m
      memory: 256Mi
  # Only watch foxhunt namespace (not cluster-wide)
  workflowNamespaces:
    - foxhunt

# Argo Server — UI + API
server:
  nodeSelector:
    k8s.scaleway.com/pool-name: platform
  tolerations:
    - key: gitlab
      operator: Equal
      value: "true"
      effect: NoSchedule
  resources:
    requests:
      cpu: 25m
      memory: 64Mi
    limits:
      cpu: 100m
      memory: 128Mi
  # No external ingress — exposed via tailscale proxy
  ingress:
    enabled: false
  # No SSO for now — server mode allows UI access without auth
  extraArgs:
    - --auth-mode=server
    - --secure=false

# Default artifact repository — MinIO (in-cluster)
artifactRepository:
  s3:
    endpoint: minio.foxhunt.svc.cluster.local:9000
    insecure: true
    bucket: foxhunt-training-results
    accessKeySecret:
      name: minio-credentials
      key: access-key
    secretKeySecret:
      name: minio-credentials
      key: secret-key

# Disable components we don't need
executor:
  resources:
    requests:
      cpu: 10m
      memory: 32Mi
    limits:
      cpu: 100m
      memory: 64Mi

# Use existing service account for workflows (or create minimal one)
workflow:
  serviceAccount:
    create: true
    name: argo-workflow

# CRDs — install with Helm
crds:
  install: true
```

**Step 3: Verify the file is valid YAML**

```bash
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/values.yaml'))" && echo "Valid YAML"
```

Expected: `Valid YAML`

**Step 4: Commit**

```bash
git add infra/k8s/argo/values.yaml
git commit -m "feat(argo): add Helm values for Argo Workflows controller"
```

---

### Task 2: Create the MinIO `foxhunt-training-results` bucket

Argo Workflows needs a dedicated bucket for artifacts (hyperopt results, trained model checkpoints, eval reports). The existing `minio-init-buckets` job in `infra/k8s/minio/minio.yaml` only creates `foxhunt-binaries` and `foxhunt-models`. We need to add `foxhunt-training-results`.

**Files:**
- Modify: `infra/k8s/minio/minio.yaml` (the init-buckets Job, around line 170)

**Step 1: Add the bucket to the init job**

In `infra/k8s/minio/minio.yaml`, find the line:

```
              mc mb --ignore-existing foxhunt/foxhunt-models
```

Add after it:

```
              mc mb --ignore-existing foxhunt/foxhunt-training-results
```

**Step 2: Create the bucket on the running cluster immediately**

We don't want to re-run the init job. Just create it directly:

```bash
kubectl run minio-create-bucket --rm -it --restart=Never -n foxhunt \
  --image=minio/mc:latest \
  --overrides='{
    "spec": {
      "nodeSelector": {"k8s.scaleway.com/pool-name": "platform"},
      "containers": [{
        "name": "mc",
        "image": "minio/mc:latest",
        "command": ["/bin/sh", "-c"],
        "args": ["mkdir -p /root/.mc/certs/CAs && cp /etc/ssl/minio-ca/ca.crt /root/.mc/certs/CAs/minio-ca.crt && mc alias set foxhunt https://minio.foxhunt.svc.cluster.local:9000 ${MINIO_ROOT_USER} ${MINIO_ROOT_PASSWORD} && mc mb --ignore-existing foxhunt/foxhunt-training-results && mc ls foxhunt/"],
        "env": [
          {"name": "MINIO_ROOT_USER", "valueFrom": {"secretKeyRef": {"name": "minio-credentials", "key": "root-user"}}},
          {"name": "MINIO_ROOT_PASSWORD", "valueFrom": {"secretKeyRef": {"name": "minio-credentials", "key": "root-password"}}}
        ],
        "volumeMounts": [{"name": "ca-cert", "mountPath": "/etc/ssl/minio-ca", "readOnly": true}]
      }],
      "volumes": [{"name": "ca-cert", "configMap": {"name": "minio-ca-cert"}}]
    }
  }' -- true
```

Expected: Shows `foxhunt-binaries/`, `foxhunt-models/`, `foxhunt-training-results/`

**Step 3: Commit**

```bash
git add infra/k8s/minio/minio.yaml
git commit -m "feat(minio): add foxhunt-training-results bucket for Argo artifacts"
```

---

### Task 3: Install Argo Workflows via Helm

Install the Argo Workflows Helm chart using the values from Task 1. This runs the controller and server on the `platform` node.

**Files:** None (Helm install only — no file changes)

**Step 1: Add the Argo Helm repo**

```bash
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
```

**Step 2: Install Argo Workflows**

```bash
helm install argo-workflows argo/argo-workflows \
  -n foxhunt \
  -f infra/k8s/argo/values.yaml \
  --wait --timeout 120s
```

Expected: `STATUS: deployed`

**Step 3: Verify controller and server are running**

```bash
kubectl get pods -n foxhunt -l app.kubernetes.io/name=argo-workflows --no-headers
```

Expected: Two pods in `Running` state (controller + server)

**Step 4: Verify CRDs are installed**

```bash
kubectl get crd | grep argoproj
```

Expected: Shows `workflows.argoproj.io`, `workflowtemplates.argoproj.io`, `cronworkflows.argoproj.io`, etc.

**Step 5: Port-forward to test Argo Server locally**

```bash
kubectl port-forward -n foxhunt svc/argo-workflows-server 2746:2746 &
curl -s http://localhost:2746/api/v1/info | head -20
kill %1
```

Expected: JSON response with Argo version info

---

### Task 4: Add DNS record for `argo.fxhnt.ai`

Add a Terraform DNS A record pointing `argo.fxhnt.ai` to the same Tailscale IP as all other subdomains. This follows the identical pattern used for git, grafana, prometheus, api, monitor, and s3 subdomains.

**Files:**
- Modify: `infra/modules/dns/main.tf`

**Step 1: Add the DNS record**

Add after the existing `s3` record block (end of file):

```hcl
resource "scaleway_domain_record" "argo" {
  dns_zone = var.dns_zone
  name     = "argo"
  type     = "A"
  data     = var.git_ip
  ttl      = 300
}
```

**Step 2: Apply Terraform**

```bash
cd infra && terraform plan -target=module.dns.scaleway_domain_record.argo
```

Expected: Plan shows 1 resource to add

```bash
terraform apply -target=module.dns.scaleway_domain_record.argo -auto-approve
```

Expected: `Apply complete! Resources: 1 added`

**Step 3: Verify DNS resolution**

```bash
dig +short argo.fxhnt.ai
```

Expected: Returns the Tailscale IP (same as `dig +short git.fxhnt.ai`)

**Step 4: Commit**

```bash
git add infra/modules/dns/main.tf
git commit -m "feat(dns): add argo.fxhnt.ai DNS record"
```

---

### Task 5: Add nginx proxy block for Argo Server

Add an nginx server block to the existing Tailscale proxy ConfigMap so `argo.fxhnt.ai` proxies to the Argo Server service (port 2746). The Argo Server serves both the API and the web UI over HTTP.

**Files:**
- Modify: `infra/k8s/gitlab/tailscale-proxy.yaml` (the `tailscale-gitlab-nginx` ConfigMap, `default.conf` section)

**Step 1: Add the Argo server block**

In the ConfigMap `tailscale-gitlab-nginx`, in `default.conf`, add a new server block after the MinIO S3 block (end of the nginx config):

```nginx
    # Argo Workflows UI — argo.fxhnt.ai
    server {
        listen 443 ssl;
        server_name argo.fxhnt.ai;

        ssl_certificate /etc/nginx/certs/tls.crt;
        ssl_certificate_key /etc/nginx/certs/tls.key;
        ssl_protocols TLSv1.2 TLSv1.3;

        location / {
            proxy_pass http://argo-workflows-server.foxhunt.svc.cluster.local:2746;
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto https;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
        }
    }
```

**Important:** The service name `argo-workflows-server` is the default Helm chart service name. Verify with:
```bash
kubectl get svc -n foxhunt | grep argo
```

**Step 2: Restart the nginx pod to pick up the new ConfigMap**

```bash
kubectl rollout restart deployment/tailscale-gitlab-proxy -n foxhunt
kubectl rollout status deployment/tailscale-gitlab-proxy -n foxhunt --timeout=60s
```

Expected: `deployment "tailscale-gitlab-proxy" successfully rolled out`

**Step 3: Verify Argo UI is accessible**

```bash
curl -sk https://argo.fxhnt.ai/api/v1/info | head -20
```

Expected: JSON response with Argo version info (same as port-forward test in Task 3)

**Step 4: Commit**

```bash
git add infra/k8s/gitlab/tailscale-proxy.yaml
git commit -m "feat(proxy): add argo.fxhnt.ai reverse proxy to Argo Server"
```

---

### Task 6: Create the training WorkflowTemplate

This is the core of the Argo integration. A single parameterized WorkflowTemplate that runs the full training pipeline: fetch binary → hyperopt → train with best params → evaluate → upload results to MinIO.

The template must:
- Accept parameters for model, gpu-pool, hyperopt trials/epochs, train epochs, symbol
- Map model names to the correct binary (rl vs supervised)
- Use the existing `foxhunt-training-runtime` image (has rclone, CUDA, nvrtc)
- Schedule GPU steps on the correct node pool with proper tolerations
- Use the existing `training-data-pvc` for training data
- Upload results to MinIO `foxhunt-training-results` bucket

**Files:**
- Create: `infra/k8s/argo/training-workflow-template.yaml`

**Step 1: Write the WorkflowTemplate**

Create `infra/k8s/argo/training-workflow-template.yaml`:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: training-pipeline
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: training-pipeline
    app.kubernetes.io/part-of: foxhunt
spec:
  entrypoint: train-model
  serviceAccountName: argo-workflow
  # Default TTL — clean up completed workflows after 1 hour
  ttlStrategy:
    secondsAfterCompletion: 3600
  # 6 hour deadline for the entire workflow
  activeDeadlineSeconds: 21600

  arguments:
    parameters:
      - name: model
        # Required: dqn, ppo, tft, mamba2, tggn, tlob, liquid, kan, xlstm, diffusion
      - name: gpu-pool
        value: ci-training
      - name: hyperopt-trials
        value: "20"
      - name: hyperopt-epochs
        value: "8"
      - name: train-epochs
        value: "50"
      - name: symbol
        value: ES.FUT
      - name: data-dir
        value: /data/futures-baseline
      - name: tx-cost-bps
        value: "0.1"
      - name: tick-size
        value: "0.25"
      - name: spread-ticks
        value: "1.0"
      - name: initial-capital
        value: "35000"

  # Shared volumes across all steps
  volumeClaimTemplates:
    - metadata:
        name: workspace
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: scw-bssd-retain
        resources:
          requests:
            storage: 5Gi

  volumes:
    - name: training-data
      persistentVolumeClaim:
        claimName: training-data-pvc

  templates:
    # ── Entrypoint DAG ─────────────────────────────────────
    - name: train-model
      dag:
        tasks:
          - name: fetch-binary
            template: fetch-binary

          - name: hyperopt
            template: hyperopt
            dependencies: [fetch-binary]

          - name: train-best
            template: train-best
            dependencies: [hyperopt]

          - name: evaluate
            template: evaluate
            dependencies: [train-best]

          - name: upload-results
            template: upload-results
            dependencies: [evaluate]

    # ── Step 1: Fetch training binaries from MinIO ─────────
    - name: fetch-binary
      nodeSelector:
        k8s.scaleway.com/pool-name: platform
      tolerations:
        - key: gitlab
          operator: Equal
          value: "true"
          effect: NoSchedule
      container:
        image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
        command: ["/bin/sh", "-c"]
        args:
          - |
            set -e
            mkdir -p /workspace/bin
            echo "Fetching training binaries from MinIO..."
            rclone copy :s3:foxhunt-binaries/training/ /workspace/bin/ \
              --s3-provider=Minio \
              --s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
              --s3-access-key-id="${MINIO_ACCESS_KEY}" \
              --s3-secret-access-key="${MINIO_SECRET_KEY}" \
              --s3-no-check-bucket \
              --no-check-certificate
            chmod +x /workspace/bin/*
            echo "Fetched binaries:"
            ls -lh /workspace/bin/
        env:
          - name: MINIO_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: minio-credentials
                key: access-key
          - name: MINIO_SECRET_KEY
            valueFrom:
              secretKeyRef:
                name: minio-credentials
                key: secret-key
        volumeMounts:
          - name: workspace
            mountPath: /workspace
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 256Mi

    # ── Step 2: Hyperopt ───────────────────────────────────
    - name: hyperopt
      nodeSelector:
        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
        - key: node.cilium.io/agent-not-ready
          operator: Exists
          effect: NoSchedule
      container:
        image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest
        command: ["/bin/sh", "-c"]
        args:
          - |
            set -e
            export PATH="/workspace/bin:$PATH"
            export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')
            nvidia-smi
            mkdir -p /workspace/output/hyperopt

            # Select binary based on model
            MODEL="{{workflow.parameters.model}}"
            case "$MODEL" in
              dqn|ppo) BINARY=hyperopt_baseline_rl ;;
              *)       BINARY=hyperopt_baseline_supervised ;;
            esac

            echo "Running $BINARY --model $MODEL ({{workflow.parameters.hyperopt-trials}} trials × {{workflow.parameters.hyperopt-epochs}} epochs)"
            $BINARY \
              --model "$MODEL" \
              --trials {{workflow.parameters.hyperopt-trials}} \
              --n-initial 5 \
              --epochs {{workflow.parameters.hyperopt-epochs}} \
              --parallel 0 \
              --symbol {{workflow.parameters.symbol}} \
              --tx-cost-bps {{workflow.parameters.tx-cost-bps}} \
              --tick-size {{workflow.parameters.tick-size}} \
              --spread-ticks {{workflow.parameters.spread-ticks}} \
              --initial-capital {{workflow.parameters.initial-capital}} \
              --data-dir {{workflow.parameters.data-dir}} \
              --base-dir /workspace/output/hyperopt \
              --output /workspace/output/${MODEL}_hyperopt_results.json

            echo "=== Hyperopt results ==="
            cat /workspace/output/${MODEL}_hyperopt_results.json 2>/dev/null || echo "No results file"
        env:
          - name: RUST_LOG
            value: info
          - name: SQLX_OFFLINE
            value: "true"
          - name: OTEL_EXPORTER_OTLP_ENDPOINT
            value: "http://tempo.foxhunt.svc.cluster.local:4317"
          - name: CUBLAS_WORKSPACE_CONFIG
            value: ":4096:8"
        volumeMounts:
          - name: workspace
            mountPath: /workspace
          - name: training-data
            mountPath: /data
            readOnly: true
        resources:
          requests:
            nvidia.com/gpu: "1"
            cpu: "7"
            memory: 16Gi
          limits:
            nvidia.com/gpu: "1"
            cpu: "8"
            memory: 48Gi

    # ── Step 3: Train with best hyperparams ────────────────
    - name: train-best
      nodeSelector:
        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
        - key: node.cilium.io/agent-not-ready
          operator: Exists
          effect: NoSchedule
      container:
        image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest
        command: ["/bin/sh", "-c"]
        args:
          - |
            set -e
            export PATH="/workspace/bin:$PATH"
            export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')
            nvidia-smi

            MODEL="{{workflow.parameters.model}}"
            case "$MODEL" in
              dqn|ppo) BINARY=train_baseline_rl ;;
              *)       BINARY=train_baseline_supervised ;;
            esac

            HYPEROPT_FILE="/workspace/output/${MODEL}_hyperopt_results.json"
            HYPEROPT_FLAG=""
            if [ -f "$HYPEROPT_FILE" ]; then
              HYPEROPT_FLAG="--hyperopt-params $HYPEROPT_FILE"
            fi

            echo "Training $MODEL with best params ({{workflow.parameters.train-epochs}} epochs)"
            $BINARY \
              --model "$MODEL" \
              --symbol {{workflow.parameters.symbol}} \
              --tx-cost-bps {{workflow.parameters.tx-cost-bps}} \
              --tick-size {{workflow.parameters.tick-size}} \
              --spread-ticks {{workflow.parameters.spread-ticks}} \
              --data-dir {{workflow.parameters.data-dir}} \
              --output-dir /workspace/output \
              --max-steps-per-epoch {{workflow.parameters.train-epochs}} \
              $HYPEROPT_FLAG

            echo "=== Training complete ==="
            ls -lh /workspace/output/
        env:
          - name: RUST_LOG
            value: info
          - name: SQLX_OFFLINE
            value: "true"
          - name: OTEL_EXPORTER_OTLP_ENDPOINT
            value: "http://tempo.foxhunt.svc.cluster.local:4317"
          - name: CUBLAS_WORKSPACE_CONFIG
            value: ":4096:8"
        volumeMounts:
          - name: workspace
            mountPath: /workspace
          - name: training-data
            mountPath: /data
            readOnly: true
        resources:
          requests:
            nvidia.com/gpu: "1"
            cpu: "7"
            memory: 16Gi
          limits:
            nvidia.com/gpu: "1"
            cpu: "8"
            memory: 48Gi

    # ── Step 4: Evaluate trained model ─────────────────────
    - name: evaluate
      nodeSelector:
        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
        - key: node.cilium.io/agent-not-ready
          operator: Exists
          effect: NoSchedule
      container:
        image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest
        command: ["/bin/sh", "-c"]
        args:
          - |
            set -e
            export PATH="/workspace/bin:$PATH"
            export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')

            MODEL="{{workflow.parameters.model}}"
            HYPEROPT_FILE="/workspace/output/${MODEL}_hyperopt_results.json"
            HYPEROPT_FLAG=""
            if [ -f "$HYPEROPT_FILE" ]; then
              HYPEROPT_FLAG="--hyperopt-params $HYPEROPT_FILE"
            fi

            echo "Evaluating $MODEL"
            evaluate_baseline \
              --model "$MODEL" \
              --models-dir /workspace/output \
              --data-dir {{workflow.parameters.data-dir}} \
              --output /workspace/output/${MODEL}_eval_report.json \
              --symbol {{workflow.parameters.symbol}} \
              --tx-cost-bps {{workflow.parameters.tx-cost-bps}} \
              --tick-size {{workflow.parameters.tick-size}} \
              --spread-ticks {{workflow.parameters.spread-ticks}} \
              $HYPEROPT_FLAG \
            || true

            echo "=== Eval report ==="
            cat /workspace/output/${MODEL}_eval_report.json 2>/dev/null || echo "No eval report"
        env:
          - name: RUST_LOG
            value: info
          - name: SQLX_OFFLINE
            value: "true"
        volumeMounts:
          - name: workspace
            mountPath: /workspace
          - name: training-data
            mountPath: /data
            readOnly: true
        resources:
          requests:
            nvidia.com/gpu: "1"
            cpu: "4"
            memory: 16Gi
          limits:
            nvidia.com/gpu: "1"
            cpu: "8"
            memory: 32Gi

    # ── Step 5: Upload results to MinIO ────────────────────
    - name: upload-results
      nodeSelector:
        k8s.scaleway.com/pool-name: platform
      tolerations:
        - key: gitlab
          operator: Equal
          value: "true"
          effect: NoSchedule
      container:
        image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
        command: ["/bin/sh", "-c"]
        args:
          - |
            set -e
            MODEL="{{workflow.parameters.model}}"
            TIMESTAMP=$(date +%Y%m%d-%H%M%S)
            DEST="s3:foxhunt-training-results/${MODEL}/${TIMESTAMP}/"

            echo "Uploading results to MinIO: $DEST"
            rclone copy /workspace/output/ ":${DEST}" \
              --s3-provider=Minio \
              --s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
              --s3-access-key-id="${MINIO_ACCESS_KEY}" \
              --s3-secret-access-key="${MINIO_SECRET_KEY}" \
              --s3-no-check-bucket \
              --no-check-certificate \
              --include "*.json" \
              --include "*.safetensors"

            echo "Upload complete. Files in $DEST:"
            rclone ls ":${DEST}" \
              --s3-provider=Minio \
              --s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
              --s3-access-key-id="${MINIO_ACCESS_KEY}" \
              --s3-secret-access-key="${MINIO_SECRET_KEY}" \
              --s3-no-check-bucket \
              --no-check-certificate
        env:
          - name: MINIO_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: minio-credentials
                key: access-key
          - name: MINIO_SECRET_KEY
            valueFrom:
              secretKeyRef:
                name: minio-credentials
                key: secret-key
        volumeMounts:
          - name: workspace
            mountPath: /workspace
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 256Mi
```

**Step 2: Validate the YAML is syntactically correct**

```bash
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/training-workflow-template.yaml'))" && echo "Valid YAML"
```

Expected: `Valid YAML`

**Step 3: Commit**

```bash
git add infra/k8s/argo/training-workflow-template.yaml
git commit -m "feat(argo): add training-pipeline WorkflowTemplate"
```

---

### Task 7: Apply the WorkflowTemplate and test with a smoke workflow

Apply the WorkflowTemplate to the cluster and submit a minimal test workflow (just the `fetch-binary` step) to verify Argo is working before committing GPU resources.

**Files:** None (kubectl only)

**Step 1: Apply the WorkflowTemplate**

```bash
kubectl apply -f infra/k8s/argo/training-workflow-template.yaml
```

Expected: `workflowtemplate.argoproj.io/training-pipeline created`

**Step 2: Verify the template is registered**

```bash
kubectl get workflowtemplates -n foxhunt
```

Expected: Shows `training-pipeline`

**Step 3: Submit a smoke test — fetch-binary only**

Create a minimal workflow that only runs the fetch-binary step to validate plumbing:

```bash
cat <<'EOF' | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: smoke-test-fetch-
  namespace: foxhunt
spec:
  serviceAccountName: argo-workflow
  entrypoint: fetch-only
  volumes:
    - name: training-data
      persistentVolumeClaim:
        claimName: training-data-pvc
  templates:
    - name: fetch-only
      nodeSelector:
        k8s.scaleway.com/pool-name: platform
      tolerations:
        - key: gitlab
          operator: Equal
          value: "true"
          effect: NoSchedule
      container:
        image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
        command: ["/bin/sh", "-c"]
        args:
          - |
            set -e
            mkdir -p /tmp/bin
            rclone copy :s3:foxhunt-binaries/training/ /tmp/bin/ \
              --s3-provider=Minio \
              --s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
              --s3-access-key-id="${MINIO_ACCESS_KEY}" \
              --s3-secret-access-key="${MINIO_SECRET_KEY}" \
              --s3-no-check-bucket \
              --no-check-certificate
            echo "Fetched binaries:"
            ls -lh /tmp/bin/
        env:
          - name: MINIO_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: minio-credentials
                key: access-key
          - name: MINIO_SECRET_KEY
            valueFrom:
              secretKeyRef:
                name: minio-credentials
                key: secret-key
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 256Mi
EOF
```

**Step 4: Watch the workflow**

```bash
kubectl get workflows -n foxhunt -w
```

Expected: Workflow transitions from `Pending` → `Running` → `Succeeded`

**Step 5: Check logs**

```bash
WORKFLOW=$(kubectl get workflows -n foxhunt --sort-by=.metadata.creationTimestamp -o name | tail -1)
kubectl logs -n foxhunt ${WORKFLOW##*/} -c main
```

Expected: Lists the fetched training binaries (hyperopt_baseline_rl, train_baseline_rl, etc.)

**Step 6: Clean up the smoke test**

```bash
kubectl delete workflows -n foxhunt -l workflows.argoproj.io/workflow-template-name!=training-pipeline --field-selector=status.phase=Succeeded
```

---

### Task 8: Submit a full DQN training workflow

The final validation: submit the full training-pipeline WorkflowTemplate for DQN with reduced trial count (2 trials × 3 epochs) to verify the complete pipeline end-to-end without burning hours of H100 time.

**Files:** None (kubectl/argo submit only)

**Step 1: Submit the DQN workflow with minimal params**

```bash
kubectl create -f - <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: dqn-test-
  namespace: foxhunt
spec:
  workflowTemplateRef:
    name: training-pipeline
  arguments:
    parameters:
      - name: model
        value: dqn
      - name: gpu-pool
        value: ci-training
      - name: hyperopt-trials
        value: "2"
      - name: hyperopt-epochs
        value: "3"
      - name: train-epochs
        value: "50"
      - name: symbol
        value: ES.FUT
EOF
```

**Step 2: Watch the workflow progress**

```bash
kubectl get workflows -n foxhunt -w
```

The workflow runs as a DAG:
1. `fetch-binary` (CPU, ~30s) — fetches binaries from MinIO
2. `hyperopt` (GPU, ~10-15 min with 2 trials × 3 epochs) — finds best hyperparams
3. `train-best` (GPU, ~5 min with 50 steps) — trains with best params
4. `evaluate` (GPU, ~2 min) — evaluates on test window
5. `upload-results` (CPU, ~30s) — uploads to MinIO

**Step 3: Monitor individual step logs**

```bash
# Get workflow name
WF=$(kubectl get workflows -n foxhunt --sort-by=.metadata.creationTimestamp -o name | tail -1)
WF_NAME=${WF##*/}

# Watch hyperopt logs (once step starts)
kubectl logs -n foxhunt ${WF_NAME}-hyperopt-* -c main -f 2>/dev/null || echo "Hyperopt step not started yet"
```

**Step 4: Verify results in MinIO after completion**

```bash
kubectl run check-results --rm -it --restart=Never -n foxhunt \
  --image=rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest \
  --overrides='{"spec":{"nodeSelector":{"k8s.scaleway.com/pool-name":"platform"},"tolerations":[{"key":"gitlab","operator":"Equal","value":"true","effect":"NoSchedule"}]}}' \
  -- /bin/sh -c '
    rclone ls :s3:foxhunt-training-results/dqn/ \
      --s3-provider=Minio \
      --s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
      --s3-access-key-id="${MINIO_ACCESS_KEY}" \
      --s3-secret-access-key="${MINIO_SECRET_KEY}" \
      --s3-no-check-bucket \
      --no-check-certificate
  '
```

Expected: Shows `dqn_hyperopt_results.json`, `dqn_eval_report.json`, and `.safetensors` checkpoint files

**Step 5: Verify via Argo UI**

Open `https://argo.fxhnt.ai` in browser. The workflow should show the DAG with all steps completed (green). Click on individual steps to see logs.

---

### Task 9: Final commit — add kustomization and README

Bundle all Argo manifests into a kustomization for easy `kubectl apply -k` deployment, and document the setup.

**Files:**
- Create: `infra/k8s/argo/kustomization.yaml`

**Step 1: Write the kustomization file**

Create `infra/k8s/argo/kustomization.yaml`:

```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - training-workflow-template.yaml
```

Note: `values.yaml` is NOT a k8s resource — it's consumed by Helm. Only the WorkflowTemplate goes in kustomization.

**Step 2: Verify kustomize builds cleanly**

```bash
kubectl kustomize infra/k8s/argo/
```

Expected: Outputs the WorkflowTemplate YAML

**Step 3: Commit everything**

```bash
git add infra/k8s/argo/kustomization.yaml
git commit -m "feat(argo): add kustomization for Argo training manifests"
```