foxhunt/docs/plans/2026-03-04-argo-workflows-phase1.md

Argo Workflows Phase 1 — Install & Validate Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Install Argo Workflows controller in the foxhunt Kapsule cluster and validate a manual DQN training workflow end-to-end.

Architecture: Argo Workflows controller runs on the platform node (alongside GitLab). A single parameterized WorkflowTemplate orchestrates the fetch→hyperopt→train→evaluate→upload pipeline. MinIO serves as the native Argo artifact repository. The Argo Server UI is exposed via argo.fxhnt.ai through the existing Tailscale nginx proxy.

Tech Stack: Argo Workflows v3.6 (Helm), Kubernetes 1.31 (Scaleway Kapsule), MinIO S3, nginx reverse proxy, Terraform (Scaleway DNS)

---

Task 1: Create Argo Workflows Helm values file

This task creates the Helm values file that configures the Argo Workflows controller for the foxhunt namespace. The controller runs on the platform node pool alongside GitLab (low resource usage). Artifact repository points to the existing in-cluster MinIO.

Files:

• Create: infra/k8s/argo/values.yaml

Step 1: Create the directory

mkdir -p infra/k8s/argo

Step 2: Write the Helm values file

Create infra/k8s/argo/values.yaml:

# Argo Workflows Helm values — foxhunt cluster
# Chart: argo/argo-workflows (v0.45.x maps to Argo v3.6.x)
# Install: helm install argo-workflows argo/argo-workflows -n foxhunt -f infra/k8s/argo/values.yaml

# Controller runs on platform node (low overhead, ~100Mi idle)
controller:
  nodeSelector:
    k8s.scaleway.com/pool-name: platform
  tolerations:
    - key: gitlab
      operator: Equal
      value: "true"
      effect: NoSchedule
  resources:
    requests:
      cpu: 50m
      memory: 128Mi
    limits:
      cpu: 200m
      memory: 256Mi
  # Only watch foxhunt namespace (not cluster-wide)
  workflowNamespaces:
    - foxhunt

# Argo Server — UI + API
server:
  nodeSelector:
    k8s.scaleway.com/pool-name: platform
  tolerations:
    - key: gitlab
      operator: Equal
      value: "true"
      effect: NoSchedule
  resources:
    requests:
      cpu: 25m
      memory: 64Mi
    limits:
      cpu: 100m
      memory: 128Mi
  # No external ingress — exposed via tailscale proxy
  ingress:
    enabled: false
  # No SSO for now — server mode allows UI access without auth
  extraArgs:
    - --auth-mode=server
    - --secure=false

# Default artifact repository — MinIO (in-cluster)
artifactRepository:
  s3:
    endpoint: minio.foxhunt.svc.cluster.local:9000
    insecure: true
    bucket: foxhunt-training-results
    accessKeySecret:
      name: minio-credentials
      key: access-key
    secretKeySecret:
      name: minio-credentials
      key: secret-key

# Disable components we don't need
executor:
  resources:
    requests:
      cpu: 10m
      memory: 32Mi
    limits:
      cpu: 100m
      memory: 64Mi

# Use existing service account for workflows (or create minimal one)
workflow:
  serviceAccount:
    create: true
    name: argo-workflow

# CRDs — install with Helm
crds:
  install: true

Step 3: Verify the file is valid YAML

python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/values.yaml'))" && echo "Valid YAML"

Expected: Valid YAML

Step 4: Commit

git add infra/k8s/argo/values.yaml
git commit -m "feat(argo): add Helm values for Argo Workflows controller"

---

Task 2: Create the MinIO foxhunt-training-results bucket

Argo Workflows needs a dedicated bucket for artifacts (hyperopt results, trained model checkpoints, eval reports). The existing minio-init-buckets job in infra/k8s/minio/minio.yaml only creates foxhunt-binaries and foxhunt-models. We need to add foxhunt-training-results.

Files:

• Modify: infra/k8s/minio/minio.yaml (the init-buckets Job, around line 170)

Step 1: Add the bucket to the init job

In infra/k8s/minio/minio.yaml, find the line:

              mc mb --ignore-existing foxhunt/foxhunt-models

Add after it:

              mc mb --ignore-existing foxhunt/foxhunt-training-results

Step 2: Create the bucket on the running cluster immediately

We don't want to re-run the init job. Just create it directly:

kubectl run minio-create-bucket --rm -it --restart=Never -n foxhunt \
  --image=minio/mc:latest \
  --overrides='{
    "spec": {
      "nodeSelector": {"k8s.scaleway.com/pool-name": "platform"},
      "containers": [{
        "name": "mc",
        "image": "minio/mc:latest",
        "command": ["/bin/sh", "-c"],
        "args": ["mkdir -p /root/.mc/certs/CAs && cp /etc/ssl/minio-ca/ca.crt /root/.mc/certs/CAs/minio-ca.crt && mc alias set foxhunt https://minio.foxhunt.svc.cluster.local:9000 ${MINIO_ROOT_USER} ${MINIO_ROOT_PASSWORD} && mc mb --ignore-existing foxhunt/foxhunt-training-results && mc ls foxhunt/"],
        "env": [
          {"name": "MINIO_ROOT_USER", "valueFrom": {"secretKeyRef": {"name": "minio-credentials", "key": "root-user"}}},
          {"name": "MINIO_ROOT_PASSWORD", "valueFrom": {"secretKeyRef": {"name": "minio-credentials", "key": "root-password"}}}
        ],
        "volumeMounts": [{"name": "ca-cert", "mountPath": "/etc/ssl/minio-ca", "readOnly": true}]
      }],
      "volumes": [{"name": "ca-cert", "configMap": {"name": "minio-ca-cert"}}]
    }
  }' -- true

Expected: Shows foxhunt-binaries/, foxhunt-models/, foxhunt-training-results/

Step 3: Commit

git add infra/k8s/minio/minio.yaml
git commit -m "feat(minio): add foxhunt-training-results bucket for Argo artifacts"

---

Task 3: Install Argo Workflows via Helm

Install the Argo Workflows Helm chart using the values from Task 1. This runs the controller and server on the platform node.

Files: None (Helm install only — no file changes)

Step 1: Add the Argo Helm repo

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

Step 2: Install Argo Workflows

helm install argo-workflows argo/argo-workflows \
  -n foxhunt \
  -f infra/k8s/argo/values.yaml \
  --wait --timeout 120s

Expected: STATUS: deployed

Step 3: Verify controller and server are running

kubectl get pods -n foxhunt -l app.kubernetes.io/name=argo-workflows --no-headers

Expected: Two pods in Running state (controller + server)

Step 4: Verify CRDs are installed

kubectl get crd | grep argoproj

Expected: Shows workflows.argoproj.io, workflowtemplates.argoproj.io, cronworkflows.argoproj.io, etc.

Step 5: Port-forward to test Argo Server locally

kubectl port-forward -n foxhunt svc/argo-workflows-server 2746:2746 &
curl -s http://localhost:2746/api/v1/info | head -20
kill %1

Expected: JSON response with Argo version info

---

Task 4: Add DNS record for argo.fxhnt.ai

Add a Terraform DNS A record pointing argo.fxhnt.ai to the same Tailscale IP as all other subdomains. This follows the identical pattern used for git, grafana, prometheus, api, monitor, and s3 subdomains.

Files:

• Modify: infra/modules/dns/main.tf

Step 1: Add the DNS record

Add after the existing s3 record block (end of file):

resource "scaleway_domain_record" "argo" {
  dns_zone = var.dns_zone
  name     = "argo"
  type     = "A"
  data     = var.git_ip
  ttl      = 300
}

Step 2: Apply Terraform

cd infra && terraform plan -target=module.dns.scaleway_domain_record.argo

Expected: Plan shows 1 resource to add

terraform apply -target=module.dns.scaleway_domain_record.argo -auto-approve

Expected: Apply complete! Resources: 1 added

Step 3: Verify DNS resolution

dig +short argo.fxhnt.ai

Expected: Returns the Tailscale IP (same as dig +short git.fxhnt.ai)

Step 4: Commit

git add infra/modules/dns/main.tf
git commit -m "feat(dns): add argo.fxhnt.ai DNS record"

---

Task 5: Add nginx proxy block for Argo Server

Add an nginx server block to the existing Tailscale proxy ConfigMap so argo.fxhnt.ai proxies to the Argo Server service (port 2746). The Argo Server serves both the API and the web UI over HTTP.

Files:

• Modify: infra/k8s/gitlab/tailscale-proxy.yaml (the tailscale-gitlab-nginx ConfigMap, default.conf section)

Step 1: Add the Argo server block

In the ConfigMap tailscale-gitlab-nginx, in default.conf, add a new server block after the MinIO S3 block (end of the nginx config):

    # Argo Workflows UI — argo.fxhnt.ai
    server {
        listen 443 ssl;
        server_name argo.fxhnt.ai;

        ssl_certificate /etc/nginx/certs/tls.crt;
        ssl_certificate_key /etc/nginx/certs/tls.key;
        ssl_protocols TLSv1.2 TLSv1.3;

        location / {
            proxy_pass http://argo-workflows-server.foxhunt.svc.cluster.local:2746;
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto https;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
        }
    }

Important: The service name argo-workflows-server is the default Helm chart service name. Verify with:

kubectl get svc -n foxhunt | grep argo

Step 2: Restart the nginx pod to pick up the new ConfigMap

kubectl rollout restart deployment/tailscale-gitlab-proxy -n foxhunt
kubectl rollout status deployment/tailscale-gitlab-proxy -n foxhunt --timeout=60s

Expected: deployment "tailscale-gitlab-proxy" successfully rolled out

Step 3: Verify Argo UI is accessible

curl -sk https://argo.fxhnt.ai/api/v1/info | head -20

Expected: JSON response with Argo version info (same as port-forward test in Task 3)

Step 4: Commit

git add infra/k8s/gitlab/tailscale-proxy.yaml
git commit -m "feat(proxy): add argo.fxhnt.ai reverse proxy to Argo Server"

---

Task 6: Create the training WorkflowTemplate

This is the core of the Argo integration. A single parameterized WorkflowTemplate that runs the full training pipeline: fetch binary → hyperopt → train with best params → evaluate → upload results to MinIO.

The template must:

• Accept parameters for model, gpu-pool, hyperopt trials/epochs, train epochs, symbol
• Map model names to the correct binary (rl vs supervised)
• Use the existing foxhunt-training-runtime image (has rclone, CUDA, nvrtc)
• Schedule GPU steps on the correct node pool with proper tolerations
• Use the existing training-data-pvc for training data
• Upload results to MinIO foxhunt-training-results bucket

Files:

• Create: infra/k8s/argo/training-workflow-template.yaml

Step 1: Write the WorkflowTemplate

Create infra/k8s/argo/training-workflow-template.yaml:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: training-pipeline
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: training-pipeline
    app.kubernetes.io/part-of: foxhunt
spec:
  entrypoint: train-model
  serviceAccountName: argo-workflow
  # Default TTL — clean up completed workflows after 1 hour
  ttlStrategy:
    secondsAfterCompletion: 3600
  # 6 hour deadline for the entire workflow
  activeDeadlineSeconds: 21600

  arguments:
    parameters:
      - name: model
        # Required: dqn, ppo, tft, mamba2, tggn, tlob, liquid, kan, xlstm, diffusion
      - name: gpu-pool
        value: ci-training
      - name: hyperopt-trials
        value: "20"
      - name: hyperopt-epochs
        value: "8"
      - name: train-epochs
        value: "50"
      - name: symbol
        value: ES.FUT
      - name: data-dir
        value: /data/futures-baseline
      - name: tx-cost-bps
        value: "0.1"
      - name: tick-size
        value: "0.25"
      - name: spread-ticks
        value: "1.0"
      - name: initial-capital
        value: "35000"

  # Shared volumes across all steps
  volumeClaimTemplates:
    - metadata:
        name: workspace
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: scw-bssd-retain
        resources:
          requests:
            storage: 5Gi

  volumes:
    - name: training-data
      persistentVolumeClaim:
        claimName: training-data-pvc

  templates:
    # ── Entrypoint DAG ─────────────────────────────────────
    - name: train-model
      dag:
        tasks:
          - name: fetch-binary
            template: fetch-binary

          - name: hyperopt
            template: hyperopt
            dependencies: [fetch-binary]

          - name: train-best
            template: train-best
            dependencies: [hyperopt]

          - name: evaluate
            template: evaluate
            dependencies: [train-best]

          - name: upload-results
            template: upload-results
            dependencies: [evaluate]

    # ── Step 1: Fetch training binaries from MinIO ─────────
    - name: fetch-binary
      nodeSelector:
        k8s.scaleway.com/pool-name: platform
      tolerations:
        - key: gitlab
          operator: Equal
          value: "true"
          effect: NoSchedule
      container:
        image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
        command: ["/bin/sh", "-c"]
        args:
          - |
            set -e
            mkdir -p /workspace/bin
            echo "Fetching training binaries from MinIO..."
            rclone copy :s3:foxhunt-binaries/training/ /workspace/bin/ \
              --s3-provider=Minio \
              --s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
              --s3-access-key-id="${MINIO_ACCESS_KEY}" \
              --s3-secret-access-key="${MINIO_SECRET_KEY}" \
              --s3-no-check-bucket \
              --no-check-certificate
            chmod +x /workspace/bin/*
            echo "Fetched binaries:"
            ls -lh /workspace/bin/
        env:
          - name: MINIO_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: minio-credentials
                key: access-key
          - name: MINIO_SECRET_KEY
            valueFrom:
              secretKeyRef:
                name: minio-credentials
                key: secret-key
        volumeMounts:
          - name: workspace
            mountPath: /workspace
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 256Mi

    # ── Step 2: Hyperopt ───────────────────────────────────
    - name: hyperopt
      nodeSelector:
        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
        - key: node.cilium.io/agent-not-ready
          operator: Exists
          effect: NoSchedule
      container:
        image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest
        command: ["/bin/sh", "-c"]
        args:
          - |
            set -e
            export PATH="/workspace/bin:$PATH"
            export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')
            nvidia-smi
            mkdir -p /workspace/output/hyperopt

            # Select binary based on model
            MODEL="{{workflow.parameters.model}}"
            case "$MODEL" in
              dqn|ppo) BINARY=hyperopt_baseline_rl ;;
              *)       BINARY=hyperopt_baseline_supervised ;;
            esac

            echo "Running $BINARY --model $MODEL ({{workflow.parameters.hyperopt-trials}} trials × {{workflow.parameters.hyperopt-epochs}} epochs)"
            $BINARY \
              --model "$MODEL" \
              --trials {{workflow.parameters.hyperopt-trials}} \
              --n-initial 5 \
              --epochs {{workflow.parameters.hyperopt-epochs}} \
              --parallel 0 \
              --symbol {{workflow.parameters.symbol}} \
              --tx-cost-bps {{workflow.parameters.tx-cost-bps}} \
              --tick-size {{workflow.parameters.tick-size}} \
              --spread-ticks {{workflow.parameters.spread-ticks}} \
              --initial-capital {{workflow.parameters.initial-capital}} \
              --data-dir {{workflow.parameters.data-dir}} \
              --base-dir /workspace/output/hyperopt \
              --output /workspace/output/${MODEL}_hyperopt_results.json

            echo "=== Hyperopt results ==="
            cat /workspace/output/${MODEL}_hyperopt_results.json 2>/dev/null || echo "No results file"
        env:
          - name: RUST_LOG
            value: info
          - name: SQLX_OFFLINE
            value: "true"
          - name: OTEL_EXPORTER_OTLP_ENDPOINT
            value: "http://tempo.foxhunt.svc.cluster.local:4317"
          - name: CUBLAS_WORKSPACE_CONFIG
            value: ":4096:8"
        volumeMounts:
          - name: workspace
            mountPath: /workspace
          - name: training-data
            mountPath: /data
            readOnly: true
        resources:
          requests:
            nvidia.com/gpu: "1"
            cpu: "7"
            memory: 16Gi
          limits:
            nvidia.com/gpu: "1"
            cpu: "8"
            memory: 48Gi

    # ── Step 3: Train with best hyperparams ────────────────
    - name: train-best
      nodeSelector:
        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
        - key: node.cilium.io/agent-not-ready
          operator: Exists
          effect: NoSchedule
      container:
        image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest
        command: ["/bin/sh", "-c"]
        args:
          - |
            set -e
            export PATH="/workspace/bin:$PATH"
            export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')
            nvidia-smi

            MODEL="{{workflow.parameters.model}}"
            case "$MODEL" in
              dqn|ppo) BINARY=train_baseline_rl ;;
              *)       BINARY=train_baseline_supervised ;;
            esac

            HYPEROPT_FILE="/workspace/output/${MODEL}_hyperopt_results.json"
            HYPEROPT_FLAG=""
            if [ -f "$HYPEROPT_FILE" ]; then
              HYPEROPT_FLAG="--hyperopt-params $HYPEROPT_FILE"
            fi

            echo "Training $MODEL with best params ({{workflow.parameters.train-epochs}} epochs)"
            $BINARY \
              --model "$MODEL" \
              --symbol {{workflow.parameters.symbol}} \
              --tx-cost-bps {{workflow.parameters.tx-cost-bps}} \
              --tick-size {{workflow.parameters.tick-size}} \
              --spread-ticks {{workflow.parameters.spread-ticks}} \
              --data-dir {{workflow.parameters.data-dir}} \
              --output-dir /workspace/output \
              --max-steps-per-epoch {{workflow.parameters.train-epochs}} \
              $HYPEROPT_FLAG

            echo "=== Training complete ==="
            ls -lh /workspace/output/
        env:
          - name: RUST_LOG
            value: info
          - name: SQLX_OFFLINE
            value: "true"
          - name: OTEL_EXPORTER_OTLP_ENDPOINT
            value: "http://tempo.foxhunt.svc.cluster.local:4317"
          - name: CUBLAS_WORKSPACE_CONFIG
            value: ":4096:8"
        volumeMounts:
          - name: workspace
            mountPath: /workspace
          - name: training-data
            mountPath: /data
            readOnly: true
        resources:
          requests:
            nvidia.com/gpu: "1"
            cpu: "7"
            memory: 16Gi
          limits:
            nvidia.com/gpu: "1"
            cpu: "8"
            memory: 48Gi

    # ── Step 4: Evaluate trained model ─────────────────────
    - name: evaluate
      nodeSelector:
        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
        - key: node.cilium.io/agent-not-ready
          operator: Exists
          effect: NoSchedule
      container:
        image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest
        command: ["/bin/sh", "-c"]
        args:
          - |
            set -e
            export PATH="/workspace/bin:$PATH"
            export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')

            MODEL="{{workflow.parameters.model}}"
            HYPEROPT_FILE="/workspace/output/${MODEL}_hyperopt_results.json"
            HYPEROPT_FLAG=""
            if [ -f "$HYPEROPT_FILE" ]; then
              HYPEROPT_FLAG="--hyperopt-params $HYPEROPT_FILE"
            fi

            echo "Evaluating $MODEL"
            evaluate_baseline \
              --model "$MODEL" \
              --models-dir /workspace/output \
              --data-dir {{workflow.parameters.data-dir}} \
              --output /workspace/output/${MODEL}_eval_report.json \
              --symbol {{workflow.parameters.symbol}} \
              --tx-cost-bps {{workflow.parameters.tx-cost-bps}} \
              --tick-size {{workflow.parameters.tick-size}} \
              --spread-ticks {{workflow.parameters.spread-ticks}} \
              $HYPEROPT_FLAG \
            || true

            echo "=== Eval report ==="
            cat /workspace/output/${MODEL}_eval_report.json 2>/dev/null || echo "No eval report"
        env:
          - name: RUST_LOG
            value: info
          - name: SQLX_OFFLINE
            value: "true"
        volumeMounts:
          - name: workspace
            mountPath: /workspace
          - name: training-data
            mountPath: /data
            readOnly: true
        resources:
          requests:
            nvidia.com/gpu: "1"
            cpu: "4"
            memory: 16Gi
          limits:
            nvidia.com/gpu: "1"
            cpu: "8"
            memory: 32Gi

    # ── Step 5: Upload results to MinIO ────────────────────
    - name: upload-results
      nodeSelector:
        k8s.scaleway.com/pool-name: platform
      tolerations:
        - key: gitlab
          operator: Equal
          value: "true"
          effect: NoSchedule
      container:
        image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
        command: ["/bin/sh", "-c"]
        args:
          - |
            set -e
            MODEL="{{workflow.parameters.model}}"
            TIMESTAMP=$(date +%Y%m%d-%H%M%S)
            DEST="s3:foxhunt-training-results/${MODEL}/${TIMESTAMP}/"

            echo "Uploading results to MinIO: $DEST"
            rclone copy /workspace/output/ ":${DEST}" \
              --s3-provider=Minio \
              --s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
              --s3-access-key-id="${MINIO_ACCESS_KEY}" \
              --s3-secret-access-key="${MINIO_SECRET_KEY}" \
              --s3-no-check-bucket \
              --no-check-certificate \
              --include "*.json" \
              --include "*.safetensors"

            echo "Upload complete. Files in $DEST:"
            rclone ls ":${DEST}" \
              --s3-provider=Minio \
              --s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
              --s3-access-key-id="${MINIO_ACCESS_KEY}" \
              --s3-secret-access-key="${MINIO_SECRET_KEY}" \
              --s3-no-check-bucket \
              --no-check-certificate
        env:
          - name: MINIO_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: minio-credentials
                key: access-key
          - name: MINIO_SECRET_KEY
            valueFrom:
              secretKeyRef:
                name: minio-credentials
                key: secret-key
        volumeMounts:
          - name: workspace
            mountPath: /workspace
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 256Mi

Step 2: Validate the YAML is syntactically correct

python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/training-workflow-template.yaml'))" && echo "Valid YAML"

Expected: Valid YAML

Step 3: Commit

git add infra/k8s/argo/training-workflow-template.yaml
git commit -m "feat(argo): add training-pipeline WorkflowTemplate"

---

Task 7: Apply the WorkflowTemplate and test with a smoke workflow

Apply the WorkflowTemplate to the cluster and submit a minimal test workflow (just the fetch-binary step) to verify Argo is working before committing GPU resources.

Files: None (kubectl only)

Step 1: Apply the WorkflowTemplate

kubectl apply -f infra/k8s/argo/training-workflow-template.yaml

Expected: workflowtemplate.argoproj.io/training-pipeline created

Step 2: Verify the template is registered

kubectl get workflowtemplates -n foxhunt

Expected: Shows training-pipeline

Step 3: Submit a smoke test — fetch-binary only

Create a minimal workflow that only runs the fetch-binary step to validate plumbing:

cat <<'EOF' | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: smoke-test-fetch-
  namespace: foxhunt
spec:
  serviceAccountName: argo-workflow
  entrypoint: fetch-only
  volumes:
    - name: training-data
      persistentVolumeClaim:
        claimName: training-data-pvc
  templates:
    - name: fetch-only
      nodeSelector:
        k8s.scaleway.com/pool-name: platform
      tolerations:
        - key: gitlab
          operator: Equal
          value: "true"
          effect: NoSchedule
      container:
        image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
        command: ["/bin/sh", "-c"]
        args:
          - |
            set -e
            mkdir -p /tmp/bin
            rclone copy :s3:foxhunt-binaries/training/ /tmp/bin/ \
              --s3-provider=Minio \
              --s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
              --s3-access-key-id="${MINIO_ACCESS_KEY}" \
              --s3-secret-access-key="${MINIO_SECRET_KEY}" \
              --s3-no-check-bucket \
              --no-check-certificate
            echo "Fetched binaries:"
            ls -lh /tmp/bin/
        env:
          - name: MINIO_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: minio-credentials
                key: access-key
          - name: MINIO_SECRET_KEY
            valueFrom:
              secretKeyRef:
                name: minio-credentials
                key: secret-key
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 256Mi
EOF

Step 4: Watch the workflow

kubectl get workflows -n foxhunt -w

Expected: Workflow transitions from Pending → Running → Succeeded

Step 5: Check logs

WORKFLOW=$(kubectl get workflows -n foxhunt --sort-by=.metadata.creationTimestamp -o name | tail -1)
kubectl logs -n foxhunt ${WORKFLOW##*/} -c main

Expected: Lists the fetched training binaries (hyperopt_baseline_rl, train_baseline_rl, etc.)

Step 6: Clean up the smoke test

kubectl delete workflows -n foxhunt -l workflows.argoproj.io/workflow-template-name!=training-pipeline --field-selector=status.phase=Succeeded

---

Task 8: Submit a full DQN training workflow

The final validation: submit the full training-pipeline WorkflowTemplate for DQN with reduced trial count (2 trials × 3 epochs) to verify the complete pipeline end-to-end without burning hours of H100 time.

Files: None (kubectl/argo submit only)

Step 1: Submit the DQN workflow with minimal params

kubectl create -f - <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: dqn-test-
  namespace: foxhunt
spec:
  workflowTemplateRef:
    name: training-pipeline
  arguments:
    parameters:
      - name: model
        value: dqn
      - name: gpu-pool
        value: ci-training
      - name: hyperopt-trials
        value: "2"
      - name: hyperopt-epochs
        value: "3"
      - name: train-epochs
        value: "50"
      - name: symbol
        value: ES.FUT
EOF

Step 2: Watch the workflow progress

kubectl get workflows -n foxhunt -w

The workflow runs as a DAG:

1. fetch-binary (CPU, ~30s) — fetches binaries from MinIO
2. hyperopt (GPU, ~10-15 min with 2 trials × 3 epochs) — finds best hyperparams
3. train-best (GPU, ~5 min with 50 steps) — trains with best params
4. evaluate (GPU, ~2 min) — evaluates on test window
5. upload-results (CPU, ~30s) — uploads to MinIO

Step 3: Monitor individual step logs

# Get workflow name
WF=$(kubectl get workflows -n foxhunt --sort-by=.metadata.creationTimestamp -o name | tail -1)
WF_NAME=${WF##*/}

# Watch hyperopt logs (once step starts)
kubectl logs -n foxhunt ${WF_NAME}-hyperopt-* -c main -f 2>/dev/null || echo "Hyperopt step not started yet"

Step 4: Verify results in MinIO after completion

kubectl run check-results --rm -it --restart=Never -n foxhunt \
  --image=rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest \
  --overrides='{"spec":{"nodeSelector":{"k8s.scaleway.com/pool-name":"platform"},"tolerations":[{"key":"gitlab","operator":"Equal","value":"true","effect":"NoSchedule"}]}}' \
  -- /bin/sh -c '
    rclone ls :s3:foxhunt-training-results/dqn/ \
      --s3-provider=Minio \
      --s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
      --s3-access-key-id="${MINIO_ACCESS_KEY}" \
      --s3-secret-access-key="${MINIO_SECRET_KEY}" \
      --s3-no-check-bucket \
      --no-check-certificate
  '

Expected: Shows dqn_hyperopt_results.json, dqn_eval_report.json, and .safetensors checkpoint files

Step 5: Verify via Argo UI

Open https://argo.fxhnt.ai in browser. The workflow should show the DAG with all steps completed (green). Click on individual steps to see logs.

---

Task 9: Final commit — add kustomization and README

Bundle all Argo manifests into a kustomization for easy kubectl apply -k deployment, and document the setup.

Files:

• Create: infra/k8s/argo/kustomization.yaml

Step 1: Write the kustomization file

Create infra/k8s/argo/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - training-workflow-template.yaml

Note: values.yaml is NOT a k8s resource — it's consumed by Helm. Only the WorkflowTemplate goes in kustomization.

Step 2: Verify kustomize builds cleanly

kubectl kustomize infra/k8s/argo/

Expected: Outputs the WorkflowTemplate YAML

Step 3: Commit everything

git add infra/k8s/argo/kustomization.yaml
git commit -m "feat(argo): add kustomization for Argo training manifests"