a2468846e1
- Helm values (controller + server on platform node, MinIO artifact repo) - WorkflowTemplate: parameterized 5-step DAG (fetch→hyperopt→train→eval→upload) - Nginx proxy for argo.fxhnt.ai → Argo Server :2746 - DNS A record for argo.fxhnt.ai - MinIO bucket foxhunt-training-results for Argo artifacts - Kustomization for kubectl apply -k Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1022 lines
31 KiB
Markdown
1022 lines
31 KiB
Markdown
# Argo Workflows Phase 1 — Install & Validate Implementation Plan
|
||
|
||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||
|
||
**Goal:** Install Argo Workflows controller in the `foxhunt` Kapsule cluster and validate a manual DQN training workflow end-to-end.
|
||
|
||
**Architecture:** Argo Workflows controller runs on the `platform` node (alongside GitLab). A single parameterized WorkflowTemplate orchestrates the fetch→hyperopt→train→evaluate→upload pipeline. MinIO serves as the native Argo artifact repository. The Argo Server UI is exposed via `argo.fxhnt.ai` through the existing Tailscale nginx proxy.
|
||
|
||
**Tech Stack:** Argo Workflows v3.6 (Helm), Kubernetes 1.31 (Scaleway Kapsule), MinIO S3, nginx reverse proxy, Terraform (Scaleway DNS)
|
||
|
||
---
|
||
|
||
### Task 1: Create Argo Workflows Helm values file
|
||
|
||
This task creates the Helm values file that configures the Argo Workflows controller for the `foxhunt` namespace. The controller runs on the `platform` node pool alongside GitLab (low resource usage). Artifact repository points to the existing in-cluster MinIO.
|
||
|
||
**Files:**
|
||
- Create: `infra/k8s/argo/values.yaml`
|
||
|
||
**Step 1: Create the directory**
|
||
|
||
```bash
|
||
mkdir -p infra/k8s/argo
|
||
```
|
||
|
||
**Step 2: Write the Helm values file**
|
||
|
||
Create `infra/k8s/argo/values.yaml`:
|
||
|
||
```yaml
|
||
# Argo Workflows Helm values — foxhunt cluster
|
||
# Chart: argo/argo-workflows (v0.45.x maps to Argo v3.6.x)
|
||
# Install: helm install argo-workflows argo/argo-workflows -n foxhunt -f infra/k8s/argo/values.yaml
|
||
|
||
# Controller runs on platform node (low overhead, ~100Mi idle)
|
||
controller:
|
||
nodeSelector:
|
||
k8s.scaleway.com/pool-name: platform
|
||
tolerations:
|
||
- key: gitlab
|
||
operator: Equal
|
||
value: "true"
|
||
effect: NoSchedule
|
||
resources:
|
||
requests:
|
||
cpu: 50m
|
||
memory: 128Mi
|
||
limits:
|
||
cpu: 200m
|
||
memory: 256Mi
|
||
# Only watch foxhunt namespace (not cluster-wide)
|
||
workflowNamespaces:
|
||
- foxhunt
|
||
|
||
# Argo Server — UI + API
|
||
server:
|
||
nodeSelector:
|
||
k8s.scaleway.com/pool-name: platform
|
||
tolerations:
|
||
- key: gitlab
|
||
operator: Equal
|
||
value: "true"
|
||
effect: NoSchedule
|
||
resources:
|
||
requests:
|
||
cpu: 25m
|
||
memory: 64Mi
|
||
limits:
|
||
cpu: 100m
|
||
memory: 128Mi
|
||
# No external ingress — exposed via tailscale proxy
|
||
ingress:
|
||
enabled: false
|
||
# No SSO for now — server mode allows UI access without auth
|
||
extraArgs:
|
||
- --auth-mode=server
|
||
- --secure=false
|
||
|
||
# Default artifact repository — MinIO (in-cluster)
|
||
artifactRepository:
|
||
s3:
|
||
endpoint: minio.foxhunt.svc.cluster.local:9000
|
||
insecure: true
|
||
bucket: foxhunt-training-results
|
||
accessKeySecret:
|
||
name: minio-credentials
|
||
key: access-key
|
||
secretKeySecret:
|
||
name: minio-credentials
|
||
key: secret-key
|
||
|
||
# Disable components we don't need
|
||
executor:
|
||
resources:
|
||
requests:
|
||
cpu: 10m
|
||
memory: 32Mi
|
||
limits:
|
||
cpu: 100m
|
||
memory: 64Mi
|
||
|
||
# Use existing service account for workflows (or create minimal one)
|
||
workflow:
|
||
serviceAccount:
|
||
create: true
|
||
name: argo-workflow
|
||
|
||
# CRDs — install with Helm
|
||
crds:
|
||
install: true
|
||
```
|
||
|
||
**Step 3: Verify the file is valid YAML**
|
||
|
||
```bash
|
||
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/values.yaml'))" && echo "Valid YAML"
|
||
```
|
||
|
||
Expected: `Valid YAML`
|
||
|
||
**Step 4: Commit**
|
||
|
||
```bash
|
||
git add infra/k8s/argo/values.yaml
|
||
git commit -m "feat(argo): add Helm values for Argo Workflows controller"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 2: Create the MinIO `foxhunt-training-results` bucket
|
||
|
||
Argo Workflows needs a dedicated bucket for artifacts (hyperopt results, trained model checkpoints, eval reports). The existing `minio-init-buckets` job in `infra/k8s/minio/minio.yaml` only creates `foxhunt-binaries` and `foxhunt-models`. We need to add `foxhunt-training-results`.
|
||
|
||
**Files:**
|
||
- Modify: `infra/k8s/minio/minio.yaml` (the init-buckets Job, around line 170)
|
||
|
||
**Step 1: Add the bucket to the init job**
|
||
|
||
In `infra/k8s/minio/minio.yaml`, find the line:
|
||
|
||
```
|
||
mc mb --ignore-existing foxhunt/foxhunt-models
|
||
```
|
||
|
||
Add after it:
|
||
|
||
```
|
||
mc mb --ignore-existing foxhunt/foxhunt-training-results
|
||
```
|
||
|
||
**Step 2: Create the bucket on the running cluster immediately**
|
||
|
||
We don't want to re-run the init job. Just create it directly:
|
||
|
||
```bash
|
||
kubectl run minio-create-bucket --rm -it --restart=Never -n foxhunt \
|
||
--image=minio/mc:latest \
|
||
--overrides='{
|
||
"spec": {
|
||
"nodeSelector": {"k8s.scaleway.com/pool-name": "platform"},
|
||
"containers": [{
|
||
"name": "mc",
|
||
"image": "minio/mc:latest",
|
||
"command": ["/bin/sh", "-c"],
|
||
"args": ["mkdir -p /root/.mc/certs/CAs && cp /etc/ssl/minio-ca/ca.crt /root/.mc/certs/CAs/minio-ca.crt && mc alias set foxhunt https://minio.foxhunt.svc.cluster.local:9000 ${MINIO_ROOT_USER} ${MINIO_ROOT_PASSWORD} && mc mb --ignore-existing foxhunt/foxhunt-training-results && mc ls foxhunt/"],
|
||
"env": [
|
||
{"name": "MINIO_ROOT_USER", "valueFrom": {"secretKeyRef": {"name": "minio-credentials", "key": "root-user"}}},
|
||
{"name": "MINIO_ROOT_PASSWORD", "valueFrom": {"secretKeyRef": {"name": "minio-credentials", "key": "root-password"}}}
|
||
],
|
||
"volumeMounts": [{"name": "ca-cert", "mountPath": "/etc/ssl/minio-ca", "readOnly": true}]
|
||
}],
|
||
"volumes": [{"name": "ca-cert", "configMap": {"name": "minio-ca-cert"}}]
|
||
}
|
||
}' -- true
|
||
```
|
||
|
||
Expected: Shows `foxhunt-binaries/`, `foxhunt-models/`, `foxhunt-training-results/`
|
||
|
||
**Step 3: Commit**
|
||
|
||
```bash
|
||
git add infra/k8s/minio/minio.yaml
|
||
git commit -m "feat(minio): add foxhunt-training-results bucket for Argo artifacts"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 3: Install Argo Workflows via Helm
|
||
|
||
Install the Argo Workflows Helm chart using the values from Task 1. This runs the controller and server on the `platform` node.
|
||
|
||
**Files:** None (Helm install only — no file changes)
|
||
|
||
**Step 1: Add the Argo Helm repo**
|
||
|
||
```bash
|
||
helm repo add argo https://argoproj.github.io/argo-helm
|
||
helm repo update
|
||
```
|
||
|
||
**Step 2: Install Argo Workflows**
|
||
|
||
```bash
|
||
helm install argo-workflows argo/argo-workflows \
|
||
-n foxhunt \
|
||
-f infra/k8s/argo/values.yaml \
|
||
--wait --timeout 120s
|
||
```
|
||
|
||
Expected: `STATUS: deployed`
|
||
|
||
**Step 3: Verify controller and server are running**
|
||
|
||
```bash
|
||
kubectl get pods -n foxhunt -l app.kubernetes.io/name=argo-workflows --no-headers
|
||
```
|
||
|
||
Expected: Two pods in `Running` state (controller + server)
|
||
|
||
**Step 4: Verify CRDs are installed**
|
||
|
||
```bash
|
||
kubectl get crd | grep argoproj
|
||
```
|
||
|
||
Expected: Shows `workflows.argoproj.io`, `workflowtemplates.argoproj.io`, `cronworkflows.argoproj.io`, etc.
|
||
|
||
**Step 5: Port-forward to test Argo Server locally**
|
||
|
||
```bash
|
||
kubectl port-forward -n foxhunt svc/argo-workflows-server 2746:2746 &
|
||
curl -s http://localhost:2746/api/v1/info | head -20
|
||
kill %1
|
||
```
|
||
|
||
Expected: JSON response with Argo version info
|
||
|
||
---
|
||
|
||
### Task 4: Add DNS record for `argo.fxhnt.ai`
|
||
|
||
Add a Terraform DNS A record pointing `argo.fxhnt.ai` to the same Tailscale IP as all other subdomains. This follows the identical pattern used for git, grafana, prometheus, api, monitor, and s3 subdomains.
|
||
|
||
**Files:**
|
||
- Modify: `infra/modules/dns/main.tf`
|
||
|
||
**Step 1: Add the DNS record**
|
||
|
||
Add after the existing `s3` record block (end of file):
|
||
|
||
```hcl
|
||
resource "scaleway_domain_record" "argo" {
|
||
dns_zone = var.dns_zone
|
||
name = "argo"
|
||
type = "A"
|
||
data = var.git_ip
|
||
ttl = 300
|
||
}
|
||
```
|
||
|
||
**Step 2: Apply Terraform**
|
||
|
||
```bash
|
||
cd infra && terraform plan -target=module.dns.scaleway_domain_record.argo
|
||
```
|
||
|
||
Expected: Plan shows 1 resource to add
|
||
|
||
```bash
|
||
terraform apply -target=module.dns.scaleway_domain_record.argo -auto-approve
|
||
```
|
||
|
||
Expected: `Apply complete! Resources: 1 added`
|
||
|
||
**Step 3: Verify DNS resolution**
|
||
|
||
```bash
|
||
dig +short argo.fxhnt.ai
|
||
```
|
||
|
||
Expected: Returns the Tailscale IP (same as `dig +short git.fxhnt.ai`)
|
||
|
||
**Step 4: Commit**
|
||
|
||
```bash
|
||
git add infra/modules/dns/main.tf
|
||
git commit -m "feat(dns): add argo.fxhnt.ai DNS record"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 5: Add nginx proxy block for Argo Server
|
||
|
||
Add an nginx server block to the existing Tailscale proxy ConfigMap so `argo.fxhnt.ai` proxies to the Argo Server service (port 2746). The Argo Server serves both the API and the web UI over HTTP.
|
||
|
||
**Files:**
|
||
- Modify: `infra/k8s/gitlab/tailscale-proxy.yaml` (the `tailscale-gitlab-nginx` ConfigMap, `default.conf` section)
|
||
|
||
**Step 1: Add the Argo server block**
|
||
|
||
In the ConfigMap `tailscale-gitlab-nginx`, in `default.conf`, add a new server block after the MinIO S3 block (end of the nginx config):
|
||
|
||
```nginx
|
||
# Argo Workflows UI — argo.fxhnt.ai
|
||
server {
|
||
listen 443 ssl;
|
||
server_name argo.fxhnt.ai;
|
||
|
||
ssl_certificate /etc/nginx/certs/tls.crt;
|
||
ssl_certificate_key /etc/nginx/certs/tls.key;
|
||
ssl_protocols TLSv1.2 TLSv1.3;
|
||
|
||
location / {
|
||
proxy_pass http://argo-workflows-server.foxhunt.svc.cluster.local:2746;
|
||
proxy_set_header Host $http_host;
|
||
proxy_set_header X-Real-IP $remote_addr;
|
||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||
proxy_set_header X-Forwarded-Proto https;
|
||
proxy_http_version 1.1;
|
||
proxy_set_header Upgrade $http_upgrade;
|
||
proxy_set_header Connection "upgrade";
|
||
}
|
||
}
|
||
```
|
||
|
||
**Important:** The service name `argo-workflows-server` is the default Helm chart service name. Verify with:
|
||
```bash
|
||
kubectl get svc -n foxhunt | grep argo
|
||
```
|
||
|
||
**Step 2: Restart the nginx pod to pick up the new ConfigMap**
|
||
|
||
```bash
|
||
kubectl rollout restart deployment/tailscale-gitlab-proxy -n foxhunt
|
||
kubectl rollout status deployment/tailscale-gitlab-proxy -n foxhunt --timeout=60s
|
||
```
|
||
|
||
Expected: `deployment "tailscale-gitlab-proxy" successfully rolled out`
|
||
|
||
**Step 3: Verify Argo UI is accessible**
|
||
|
||
```bash
|
||
curl -sk https://argo.fxhnt.ai/api/v1/info | head -20
|
||
```
|
||
|
||
Expected: JSON response with Argo version info (same as port-forward test in Task 3)
|
||
|
||
**Step 4: Commit**
|
||
|
||
```bash
|
||
git add infra/k8s/gitlab/tailscale-proxy.yaml
|
||
git commit -m "feat(proxy): add argo.fxhnt.ai reverse proxy to Argo Server"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 6: Create the training WorkflowTemplate
|
||
|
||
This is the core of the Argo integration. A single parameterized WorkflowTemplate that runs the full training pipeline: fetch binary → hyperopt → train with best params → evaluate → upload results to MinIO.
|
||
|
||
The template must:
|
||
- Accept parameters for model, gpu-pool, hyperopt trials/epochs, train epochs, symbol
|
||
- Map model names to the correct binary (rl vs supervised)
|
||
- Use the existing `foxhunt-training-runtime` image (has rclone, CUDA, nvrtc)
|
||
- Schedule GPU steps on the correct node pool with proper tolerations
|
||
- Use the existing `training-data-pvc` for training data
|
||
- Upload results to MinIO `foxhunt-training-results` bucket
|
||
|
||
**Files:**
|
||
- Create: `infra/k8s/argo/training-workflow-template.yaml`
|
||
|
||
**Step 1: Write the WorkflowTemplate**
|
||
|
||
Create `infra/k8s/argo/training-workflow-template.yaml`:
|
||
|
||
```yaml
|
||
apiVersion: argoproj.io/v1alpha1
|
||
kind: WorkflowTemplate
|
||
metadata:
|
||
name: training-pipeline
|
||
namespace: foxhunt
|
||
labels:
|
||
app.kubernetes.io/name: training-pipeline
|
||
app.kubernetes.io/part-of: foxhunt
|
||
spec:
|
||
entrypoint: train-model
|
||
serviceAccountName: argo-workflow
|
||
# Default TTL — clean up completed workflows after 1 hour
|
||
ttlStrategy:
|
||
secondsAfterCompletion: 3600
|
||
# 6 hour deadline for the entire workflow
|
||
activeDeadlineSeconds: 21600
|
||
|
||
arguments:
|
||
parameters:
|
||
- name: model
|
||
# Required: dqn, ppo, tft, mamba2, tggn, tlob, liquid, kan, xlstm, diffusion
|
||
- name: gpu-pool
|
||
value: ci-training
|
||
- name: hyperopt-trials
|
||
value: "20"
|
||
- name: hyperopt-epochs
|
||
value: "8"
|
||
- name: train-epochs
|
||
value: "50"
|
||
- name: symbol
|
||
value: ES.FUT
|
||
- name: data-dir
|
||
value: /data/futures-baseline
|
||
- name: tx-cost-bps
|
||
value: "0.1"
|
||
- name: tick-size
|
||
value: "0.25"
|
||
- name: spread-ticks
|
||
value: "1.0"
|
||
- name: initial-capital
|
||
value: "35000"
|
||
|
||
# Shared volumes across all steps
|
||
volumeClaimTemplates:
|
||
- metadata:
|
||
name: workspace
|
||
spec:
|
||
accessModes: ["ReadWriteOnce"]
|
||
storageClassName: scw-bssd-retain
|
||
resources:
|
||
requests:
|
||
storage: 5Gi
|
||
|
||
volumes:
|
||
- name: training-data
|
||
persistentVolumeClaim:
|
||
claimName: training-data-pvc
|
||
|
||
templates:
|
||
# ── Entrypoint DAG ─────────────────────────────────────
|
||
- name: train-model
|
||
dag:
|
||
tasks:
|
||
- name: fetch-binary
|
||
template: fetch-binary
|
||
|
||
- name: hyperopt
|
||
template: hyperopt
|
||
dependencies: [fetch-binary]
|
||
|
||
- name: train-best
|
||
template: train-best
|
||
dependencies: [hyperopt]
|
||
|
||
- name: evaluate
|
||
template: evaluate
|
||
dependencies: [train-best]
|
||
|
||
- name: upload-results
|
||
template: upload-results
|
||
dependencies: [evaluate]
|
||
|
||
# ── Step 1: Fetch training binaries from MinIO ─────────
|
||
- name: fetch-binary
|
||
nodeSelector:
|
||
k8s.scaleway.com/pool-name: platform
|
||
tolerations:
|
||
- key: gitlab
|
||
operator: Equal
|
||
value: "true"
|
||
effect: NoSchedule
|
||
container:
|
||
image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
|
||
command: ["/bin/sh", "-c"]
|
||
args:
|
||
- |
|
||
set -e
|
||
mkdir -p /workspace/bin
|
||
echo "Fetching training binaries from MinIO..."
|
||
rclone copy :s3:foxhunt-binaries/training/ /workspace/bin/ \
|
||
--s3-provider=Minio \
|
||
--s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
|
||
--s3-access-key-id="${MINIO_ACCESS_KEY}" \
|
||
--s3-secret-access-key="${MINIO_SECRET_KEY}" \
|
||
--s3-no-check-bucket \
|
||
--no-check-certificate
|
||
chmod +x /workspace/bin/*
|
||
echo "Fetched binaries:"
|
||
ls -lh /workspace/bin/
|
||
env:
|
||
- name: MINIO_ACCESS_KEY
|
||
valueFrom:
|
||
secretKeyRef:
|
||
name: minio-credentials
|
||
key: access-key
|
||
- name: MINIO_SECRET_KEY
|
||
valueFrom:
|
||
secretKeyRef:
|
||
name: minio-credentials
|
||
key: secret-key
|
||
volumeMounts:
|
||
- name: workspace
|
||
mountPath: /workspace
|
||
resources:
|
||
requests:
|
||
cpu: 100m
|
||
memory: 128Mi
|
||
limits:
|
||
cpu: 500m
|
||
memory: 256Mi
|
||
|
||
# ── Step 2: Hyperopt ───────────────────────────────────
|
||
- name: hyperopt
|
||
nodeSelector:
|
||
k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
|
||
tolerations:
|
||
- key: nvidia.com/gpu
|
||
operator: Exists
|
||
effect: NoSchedule
|
||
- key: node.cilium.io/agent-not-ready
|
||
operator: Exists
|
||
effect: NoSchedule
|
||
container:
|
||
image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest
|
||
command: ["/bin/sh", "-c"]
|
||
args:
|
||
- |
|
||
set -e
|
||
export PATH="/workspace/bin:$PATH"
|
||
export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')
|
||
nvidia-smi
|
||
mkdir -p /workspace/output/hyperopt
|
||
|
||
# Select binary based on model
|
||
MODEL="{{workflow.parameters.model}}"
|
||
case "$MODEL" in
|
||
dqn|ppo) BINARY=hyperopt_baseline_rl ;;
|
||
*) BINARY=hyperopt_baseline_supervised ;;
|
||
esac
|
||
|
||
echo "Running $BINARY --model $MODEL ({{workflow.parameters.hyperopt-trials}} trials × {{workflow.parameters.hyperopt-epochs}} epochs)"
|
||
$BINARY \
|
||
--model "$MODEL" \
|
||
--trials {{workflow.parameters.hyperopt-trials}} \
|
||
--n-initial 5 \
|
||
--epochs {{workflow.parameters.hyperopt-epochs}} \
|
||
--parallel 0 \
|
||
--symbol {{workflow.parameters.symbol}} \
|
||
--tx-cost-bps {{workflow.parameters.tx-cost-bps}} \
|
||
--tick-size {{workflow.parameters.tick-size}} \
|
||
--spread-ticks {{workflow.parameters.spread-ticks}} \
|
||
--initial-capital {{workflow.parameters.initial-capital}} \
|
||
--data-dir {{workflow.parameters.data-dir}} \
|
||
--base-dir /workspace/output/hyperopt \
|
||
--output /workspace/output/${MODEL}_hyperopt_results.json
|
||
|
||
echo "=== Hyperopt results ==="
|
||
cat /workspace/output/${MODEL}_hyperopt_results.json 2>/dev/null || echo "No results file"
|
||
env:
|
||
- name: RUST_LOG
|
||
value: info
|
||
- name: SQLX_OFFLINE
|
||
value: "true"
|
||
- name: OTEL_EXPORTER_OTLP_ENDPOINT
|
||
value: "http://tempo.foxhunt.svc.cluster.local:4317"
|
||
- name: CUBLAS_WORKSPACE_CONFIG
|
||
value: ":4096:8"
|
||
volumeMounts:
|
||
- name: workspace
|
||
mountPath: /workspace
|
||
- name: training-data
|
||
mountPath: /data
|
||
readOnly: true
|
||
resources:
|
||
requests:
|
||
nvidia.com/gpu: "1"
|
||
cpu: "7"
|
||
memory: 16Gi
|
||
limits:
|
||
nvidia.com/gpu: "1"
|
||
cpu: "8"
|
||
memory: 48Gi
|
||
|
||
# ── Step 3: Train with best hyperparams ────────────────
|
||
- name: train-best
|
||
nodeSelector:
|
||
k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
|
||
tolerations:
|
||
- key: nvidia.com/gpu
|
||
operator: Exists
|
||
effect: NoSchedule
|
||
- key: node.cilium.io/agent-not-ready
|
||
operator: Exists
|
||
effect: NoSchedule
|
||
container:
|
||
image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest
|
||
command: ["/bin/sh", "-c"]
|
||
args:
|
||
- |
|
||
set -e
|
||
export PATH="/workspace/bin:$PATH"
|
||
export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')
|
||
nvidia-smi
|
||
|
||
MODEL="{{workflow.parameters.model}}"
|
||
case "$MODEL" in
|
||
dqn|ppo) BINARY=train_baseline_rl ;;
|
||
*) BINARY=train_baseline_supervised ;;
|
||
esac
|
||
|
||
HYPEROPT_FILE="/workspace/output/${MODEL}_hyperopt_results.json"
|
||
HYPEROPT_FLAG=""
|
||
if [ -f "$HYPEROPT_FILE" ]; then
|
||
HYPEROPT_FLAG="--hyperopt-params $HYPEROPT_FILE"
|
||
fi
|
||
|
||
echo "Training $MODEL with best params ({{workflow.parameters.train-epochs}} epochs)"
|
||
$BINARY \
|
||
--model "$MODEL" \
|
||
--symbol {{workflow.parameters.symbol}} \
|
||
--tx-cost-bps {{workflow.parameters.tx-cost-bps}} \
|
||
--tick-size {{workflow.parameters.tick-size}} \
|
||
--spread-ticks {{workflow.parameters.spread-ticks}} \
|
||
--data-dir {{workflow.parameters.data-dir}} \
|
||
--output-dir /workspace/output \
|
||
--max-steps-per-epoch {{workflow.parameters.train-epochs}} \
|
||
$HYPEROPT_FLAG
|
||
|
||
echo "=== Training complete ==="
|
||
ls -lh /workspace/output/
|
||
env:
|
||
- name: RUST_LOG
|
||
value: info
|
||
- name: SQLX_OFFLINE
|
||
value: "true"
|
||
- name: OTEL_EXPORTER_OTLP_ENDPOINT
|
||
value: "http://tempo.foxhunt.svc.cluster.local:4317"
|
||
- name: CUBLAS_WORKSPACE_CONFIG
|
||
value: ":4096:8"
|
||
volumeMounts:
|
||
- name: workspace
|
||
mountPath: /workspace
|
||
- name: training-data
|
||
mountPath: /data
|
||
readOnly: true
|
||
resources:
|
||
requests:
|
||
nvidia.com/gpu: "1"
|
||
cpu: "7"
|
||
memory: 16Gi
|
||
limits:
|
||
nvidia.com/gpu: "1"
|
||
cpu: "8"
|
||
memory: 48Gi
|
||
|
||
# ── Step 4: Evaluate trained model ─────────────────────
|
||
- name: evaluate
|
||
nodeSelector:
|
||
k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
|
||
tolerations:
|
||
- key: nvidia.com/gpu
|
||
operator: Exists
|
||
effect: NoSchedule
|
||
- key: node.cilium.io/agent-not-ready
|
||
operator: Exists
|
||
effect: NoSchedule
|
||
container:
|
||
image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest
|
||
command: ["/bin/sh", "-c"]
|
||
args:
|
||
- |
|
||
set -e
|
||
export PATH="/workspace/bin:$PATH"
|
||
export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')
|
||
|
||
MODEL="{{workflow.parameters.model}}"
|
||
HYPEROPT_FILE="/workspace/output/${MODEL}_hyperopt_results.json"
|
||
HYPEROPT_FLAG=""
|
||
if [ -f "$HYPEROPT_FILE" ]; then
|
||
HYPEROPT_FLAG="--hyperopt-params $HYPEROPT_FILE"
|
||
fi
|
||
|
||
echo "Evaluating $MODEL"
|
||
evaluate_baseline \
|
||
--model "$MODEL" \
|
||
--models-dir /workspace/output \
|
||
--data-dir {{workflow.parameters.data-dir}} \
|
||
--output /workspace/output/${MODEL}_eval_report.json \
|
||
--symbol {{workflow.parameters.symbol}} \
|
||
--tx-cost-bps {{workflow.parameters.tx-cost-bps}} \
|
||
--tick-size {{workflow.parameters.tick-size}} \
|
||
--spread-ticks {{workflow.parameters.spread-ticks}} \
|
||
$HYPEROPT_FLAG \
|
||
|| true
|
||
|
||
echo "=== Eval report ==="
|
||
cat /workspace/output/${MODEL}_eval_report.json 2>/dev/null || echo "No eval report"
|
||
env:
|
||
- name: RUST_LOG
|
||
value: info
|
||
- name: SQLX_OFFLINE
|
||
value: "true"
|
||
volumeMounts:
|
||
- name: workspace
|
||
mountPath: /workspace
|
||
- name: training-data
|
||
mountPath: /data
|
||
readOnly: true
|
||
resources:
|
||
requests:
|
||
nvidia.com/gpu: "1"
|
||
cpu: "4"
|
||
memory: 16Gi
|
||
limits:
|
||
nvidia.com/gpu: "1"
|
||
cpu: "8"
|
||
memory: 32Gi
|
||
|
||
# ── Step 5: Upload results to MinIO ────────────────────
|
||
- name: upload-results
|
||
nodeSelector:
|
||
k8s.scaleway.com/pool-name: platform
|
||
tolerations:
|
||
- key: gitlab
|
||
operator: Equal
|
||
value: "true"
|
||
effect: NoSchedule
|
||
container:
|
||
image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
|
||
command: ["/bin/sh", "-c"]
|
||
args:
|
||
- |
|
||
set -e
|
||
MODEL="{{workflow.parameters.model}}"
|
||
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
|
||
DEST="s3:foxhunt-training-results/${MODEL}/${TIMESTAMP}/"
|
||
|
||
echo "Uploading results to MinIO: $DEST"
|
||
rclone copy /workspace/output/ ":${DEST}" \
|
||
--s3-provider=Minio \
|
||
--s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
|
||
--s3-access-key-id="${MINIO_ACCESS_KEY}" \
|
||
--s3-secret-access-key="${MINIO_SECRET_KEY}" \
|
||
--s3-no-check-bucket \
|
||
--no-check-certificate \
|
||
--include "*.json" \
|
||
--include "*.safetensors"
|
||
|
||
echo "Upload complete. Files in $DEST:"
|
||
rclone ls ":${DEST}" \
|
||
--s3-provider=Minio \
|
||
--s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
|
||
--s3-access-key-id="${MINIO_ACCESS_KEY}" \
|
||
--s3-secret-access-key="${MINIO_SECRET_KEY}" \
|
||
--s3-no-check-bucket \
|
||
--no-check-certificate
|
||
env:
|
||
- name: MINIO_ACCESS_KEY
|
||
valueFrom:
|
||
secretKeyRef:
|
||
name: minio-credentials
|
||
key: access-key
|
||
- name: MINIO_SECRET_KEY
|
||
valueFrom:
|
||
secretKeyRef:
|
||
name: minio-credentials
|
||
key: secret-key
|
||
volumeMounts:
|
||
- name: workspace
|
||
mountPath: /workspace
|
||
resources:
|
||
requests:
|
||
cpu: 100m
|
||
memory: 128Mi
|
||
limits:
|
||
cpu: 500m
|
||
memory: 256Mi
|
||
```
|
||
|
||
**Step 2: Validate the YAML is syntactically correct**
|
||
|
||
```bash
|
||
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/training-workflow-template.yaml'))" && echo "Valid YAML"
|
||
```
|
||
|
||
Expected: `Valid YAML`
|
||
|
||
**Step 3: Commit**
|
||
|
||
```bash
|
||
git add infra/k8s/argo/training-workflow-template.yaml
|
||
git commit -m "feat(argo): add training-pipeline WorkflowTemplate"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 7: Apply the WorkflowTemplate and test with a smoke workflow
|
||
|
||
Apply the WorkflowTemplate to the cluster and submit a minimal test workflow (just the `fetch-binary` step) to verify Argo is working before committing GPU resources.
|
||
|
||
**Files:** None (kubectl only)
|
||
|
||
**Step 1: Apply the WorkflowTemplate**
|
||
|
||
```bash
|
||
kubectl apply -f infra/k8s/argo/training-workflow-template.yaml
|
||
```
|
||
|
||
Expected: `workflowtemplate.argoproj.io/training-pipeline created`
|
||
|
||
**Step 2: Verify the template is registered**
|
||
|
||
```bash
|
||
kubectl get workflowtemplates -n foxhunt
|
||
```
|
||
|
||
Expected: Shows `training-pipeline`
|
||
|
||
**Step 3: Submit a smoke test — fetch-binary only**
|
||
|
||
Create a minimal workflow that only runs the fetch-binary step to validate plumbing:
|
||
|
||
```bash
|
||
cat <<'EOF' | kubectl apply -f -
|
||
apiVersion: argoproj.io/v1alpha1
|
||
kind: Workflow
|
||
metadata:
|
||
generateName: smoke-test-fetch-
|
||
namespace: foxhunt
|
||
spec:
|
||
serviceAccountName: argo-workflow
|
||
entrypoint: fetch-only
|
||
volumes:
|
||
- name: training-data
|
||
persistentVolumeClaim:
|
||
claimName: training-data-pvc
|
||
templates:
|
||
- name: fetch-only
|
||
nodeSelector:
|
||
k8s.scaleway.com/pool-name: platform
|
||
tolerations:
|
||
- key: gitlab
|
||
operator: Equal
|
||
value: "true"
|
||
effect: NoSchedule
|
||
container:
|
||
image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
|
||
command: ["/bin/sh", "-c"]
|
||
args:
|
||
- |
|
||
set -e
|
||
mkdir -p /tmp/bin
|
||
rclone copy :s3:foxhunt-binaries/training/ /tmp/bin/ \
|
||
--s3-provider=Minio \
|
||
--s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
|
||
--s3-access-key-id="${MINIO_ACCESS_KEY}" \
|
||
--s3-secret-access-key="${MINIO_SECRET_KEY}" \
|
||
--s3-no-check-bucket \
|
||
--no-check-certificate
|
||
echo "Fetched binaries:"
|
||
ls -lh /tmp/bin/
|
||
env:
|
||
- name: MINIO_ACCESS_KEY
|
||
valueFrom:
|
||
secretKeyRef:
|
||
name: minio-credentials
|
||
key: access-key
|
||
- name: MINIO_SECRET_KEY
|
||
valueFrom:
|
||
secretKeyRef:
|
||
name: minio-credentials
|
||
key: secret-key
|
||
resources:
|
||
requests:
|
||
cpu: 100m
|
||
memory: 128Mi
|
||
limits:
|
||
cpu: 500m
|
||
memory: 256Mi
|
||
EOF
|
||
```
|
||
|
||
**Step 4: Watch the workflow**
|
||
|
||
```bash
|
||
kubectl get workflows -n foxhunt -w
|
||
```
|
||
|
||
Expected: Workflow transitions from `Pending` → `Running` → `Succeeded`
|
||
|
||
**Step 5: Check logs**
|
||
|
||
```bash
|
||
WORKFLOW=$(kubectl get workflows -n foxhunt --sort-by=.metadata.creationTimestamp -o name | tail -1)
|
||
kubectl logs -n foxhunt ${WORKFLOW##*/} -c main
|
||
```
|
||
|
||
Expected: Lists the fetched training binaries (hyperopt_baseline_rl, train_baseline_rl, etc.)
|
||
|
||
**Step 6: Clean up the smoke test**
|
||
|
||
```bash
|
||
kubectl delete workflows -n foxhunt -l workflows.argoproj.io/workflow-template-name!=training-pipeline --field-selector=status.phase=Succeeded
|
||
```
|
||
|
||
---
|
||
|
||
### Task 8: Submit a full DQN training workflow
|
||
|
||
The final validation: submit the full training-pipeline WorkflowTemplate for DQN with reduced trial count (2 trials × 3 epochs) to verify the complete pipeline end-to-end without burning hours of H100 time.
|
||
|
||
**Files:** None (kubectl/argo submit only)
|
||
|
||
**Step 1: Submit the DQN workflow with minimal params**
|
||
|
||
```bash
|
||
kubectl create -f - <<'EOF'
|
||
apiVersion: argoproj.io/v1alpha1
|
||
kind: Workflow
|
||
metadata:
|
||
generateName: dqn-test-
|
||
namespace: foxhunt
|
||
spec:
|
||
workflowTemplateRef:
|
||
name: training-pipeline
|
||
arguments:
|
||
parameters:
|
||
- name: model
|
||
value: dqn
|
||
- name: gpu-pool
|
||
value: ci-training
|
||
- name: hyperopt-trials
|
||
value: "2"
|
||
- name: hyperopt-epochs
|
||
value: "3"
|
||
- name: train-epochs
|
||
value: "50"
|
||
- name: symbol
|
||
value: ES.FUT
|
||
EOF
|
||
```
|
||
|
||
**Step 2: Watch the workflow progress**
|
||
|
||
```bash
|
||
kubectl get workflows -n foxhunt -w
|
||
```
|
||
|
||
The workflow runs as a DAG:
|
||
1. `fetch-binary` (CPU, ~30s) — fetches binaries from MinIO
|
||
2. `hyperopt` (GPU, ~10-15 min with 2 trials × 3 epochs) — finds best hyperparams
|
||
3. `train-best` (GPU, ~5 min with 50 steps) — trains with best params
|
||
4. `evaluate` (GPU, ~2 min) — evaluates on test window
|
||
5. `upload-results` (CPU, ~30s) — uploads to MinIO
|
||
|
||
**Step 3: Monitor individual step logs**
|
||
|
||
```bash
|
||
# Get workflow name
|
||
WF=$(kubectl get workflows -n foxhunt --sort-by=.metadata.creationTimestamp -o name | tail -1)
|
||
WF_NAME=${WF##*/}
|
||
|
||
# Watch hyperopt logs (once step starts)
|
||
kubectl logs -n foxhunt ${WF_NAME}-hyperopt-* -c main -f 2>/dev/null || echo "Hyperopt step not started yet"
|
||
```
|
||
|
||
**Step 4: Verify results in MinIO after completion**
|
||
|
||
```bash
|
||
kubectl run check-results --rm -it --restart=Never -n foxhunt \
|
||
--image=rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest \
|
||
--overrides='{"spec":{"nodeSelector":{"k8s.scaleway.com/pool-name":"platform"},"tolerations":[{"key":"gitlab","operator":"Equal","value":"true","effect":"NoSchedule"}]}}' \
|
||
-- /bin/sh -c '
|
||
rclone ls :s3:foxhunt-training-results/dqn/ \
|
||
--s3-provider=Minio \
|
||
--s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
|
||
--s3-access-key-id="${MINIO_ACCESS_KEY}" \
|
||
--s3-secret-access-key="${MINIO_SECRET_KEY}" \
|
||
--s3-no-check-bucket \
|
||
--no-check-certificate
|
||
'
|
||
```
|
||
|
||
Expected: Shows `dqn_hyperopt_results.json`, `dqn_eval_report.json`, and `.safetensors` checkpoint files
|
||
|
||
**Step 5: Verify via Argo UI**
|
||
|
||
Open `https://argo.fxhnt.ai` in browser. The workflow should show the DAG with all steps completed (green). Click on individual steps to see logs.
|
||
|
||
---
|
||
|
||
### Task 9: Final commit — add kustomization and README
|
||
|
||
Bundle all Argo manifests into a kustomization for easy `kubectl apply -k` deployment, and document the setup.
|
||
|
||
**Files:**
|
||
- Create: `infra/k8s/argo/kustomization.yaml`
|
||
|
||
**Step 1: Write the kustomization file**
|
||
|
||
Create `infra/k8s/argo/kustomization.yaml`:
|
||
|
||
```yaml
|
||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||
kind: Kustomization
|
||
resources:
|
||
- training-workflow-template.yaml
|
||
```
|
||
|
||
Note: `values.yaml` is NOT a k8s resource — it's consumed by Helm. Only the WorkflowTemplate goes in kustomization.
|
||
|
||
**Step 2: Verify kustomize builds cleanly**
|
||
|
||
```bash
|
||
kubectl kustomize infra/k8s/argo/
|
||
```
|
||
|
||
Expected: Outputs the WorkflowTemplate YAML
|
||
|
||
**Step 3: Commit everything**
|
||
|
||
```bash
|
||
git add infra/k8s/argo/kustomization.yaml
|
||
git commit -m "feat(argo): add kustomization for Argo training manifests"
|
||
```
|