foxhunt/docs/superpowers/plans/2026-03-31-databento-download-workflow.md

Databento Download Argo WorkflowTemplate Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Replace the hardcoded K8s Job in databento-download-streaming.yaml with a parameterized Argo WorkflowTemplate and a helper script, matching the existing argo-train.sh / train-dqn pattern.

Architecture: One Argo WorkflowTemplate (databento-download) with parameters for schema, output dir, parallelism, and node pool. One shell script (scripts/argo-download.sh) wraps argo submit with ergonomic CLI flags. The old K8s Job file becomes the WorkflowTemplate. The configmap databento-universe-all already has all schema configs — no changes needed there.

Tech Stack: Argo Workflows, K8s Job (inside Argo), Bash, existing download_baseline Rust binary on PVC.

---

File Structure

File	Action	Responsibility
infra/k8s/argo/databento-download-template.yaml	Create	Argo WorkflowTemplate with parameters
scripts/argo-download.sh	Create	CLI wrapper for argo submit
infra/k8s/jobs/databento-download-streaming.yaml	Modify	Add deprecation comment pointing to new template

---

Task 1: Create Argo WorkflowTemplate

Files:

• Create: infra/k8s/argo/databento-download-template.yaml

• Step 1: Write the WorkflowTemplate

# Databento data download via streaming API (get_range).
#
# Parameterized Argo WorkflowTemplate — replaces the hardcoded K8s Job.
# Binary pre-uploaded to /data/bin/download_baseline on training-data-pvc.
# Universe configs loaded from configmap databento-universe-all.
#
# Usage:
#   argo submit -n foxhunt --from=wftmpl/databento-download
#   argo submit -n foxhunt --from=wftmpl/databento-download -p schema=trades -p output-dir=/data/futures-baseline-trades
#   ./scripts/argo-download.sh mbp-10 --parallel 9 --pool ci-compile-cpu
#
# Databento rate limits: 100 concurrent connections per IP, 100 req/s.
---
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: databento-download
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: databento-download
    app.kubernetes.io/part-of: foxhunt
spec:
  entrypoint: download
  serviceAccountName: argo-workflow
  podMetadata:
    labels:
      app.kubernetes.io/part-of: foxhunt
      app.kubernetes.io/component: databento-download
  securityContext:
    fsGroup: 1000
  ttlStrategy:
    secondsAfterCompletion: 3600
  activeDeadlineSeconds: 86400  # 24 hours

  arguments:
    parameters:
      - name: schema
        value: mbp-10
        description: "Databento schema: ohlcv-1m, ohlcv-1s, trades, mbp-10"
      - name: output-dir
        value: /data/futures-baseline-mbp10
        description: "Output directory on training-data PVC"
      - name: parallel
        value: "9"
        description: "Number of parallel quarterly downloads (max ~100 per Databento rate limit)"
      - name: node-pool
        value: ci-compile-cpu
        description: "K8s node pool: ci-compile-cpu (32c/64G) or platform (4c/8G)"
      - name: cpu-request
        value: "4"
        description: "CPU request cores"
      - name: cpu-limit
        value: "16"
        description: "CPU limit cores"
      - name: mem-request
        value: 8Gi
        description: "Memory request"
      - name: mem-limit
        value: 32Gi
        description: "Memory limit"

  volumes:
    - name: training-data
      persistentVolumeClaim:
        claimName: training-data-pvc
    - name: universe-configs
      configMap:
        name: databento-universe-all

  templates:
    - name: download
      inputs:
        parameters:
          - name: schema
          - name: output-dir
          - name: parallel
          - name: node-pool
          - name: cpu-request
          - name: cpu-limit
          - name: mem-request
          - name: mem-limit
      nodeSelector:
        k8s.scaleway.com/pool-name: "{{inputs.parameters.node-pool}}"
      container:
        image: ubuntu:24.04
        command: ["/bin/bash", "-c"]
        env:
          - name: RUST_LOG
            value: info
          - name: DATABENTO_API_KEY
            valueFrom:
              secretKeyRef:
                name: databento-credentials
                key: api-key
        resources:
          requests:
            cpu: "{{inputs.parameters.cpu-request}}"
            memory: "{{inputs.parameters.mem-request}}"
          limits:
            cpu: "{{inputs.parameters.cpu-limit}}"
            memory: "{{inputs.parameters.mem-limit}}"
        volumeMounts:
          - name: training-data
            mountPath: /data
          - name: universe-configs
            mountPath: /configs
        args:
          - |
            set -e
            apt-get update -qq > /dev/null 2>&1 && apt-get install -y -qq ca-certificates > /dev/null 2>&1

            BINARY=/data/bin/download_baseline
            chmod +x "$BINARY"

            SCHEMA="{{inputs.parameters.schema}}"
            OUTPUT_DIR="{{inputs.parameters.output-dir}}"
            PARALLEL="{{inputs.parameters.parallel}}"

            # Map schema name to configmap key
            CONFIG="/configs/${SCHEMA}.toml"
            if [ ! -f "$CONFIG" ]; then
              echo "ERROR: Config not found: $CONFIG"
              echo "Available configs:"
              ls /configs/
              exit 1
            fi

            mkdir -p "$OUTPUT_DIR"

            echo "=== Databento Download ==="
            echo "Schema:    $SCHEMA"
            echo "Config:    $CONFIG"
            echo "Output:    $OUTPUT_DIR"
            echo "Parallel:  $PARALLEL"
            echo "Node:      $(hostname), CPUs: $(nproc), RAM: $(free -g | awk '/Mem:/{print $2}')G"
            echo ""

            $BINARY \
              --universe-config "$CONFIG" \
              --output-dir "$OUTPUT_DIR" \
              --parallel "$PARALLEL" \
              --yes

            echo ""
            echo "=== Download complete ==="
            echo "Files:"
            find "$OUTPUT_DIR" -name '*.dbn.zst' -exec ls -lh {} \;
            echo ""
            du -sh "$OUTPUT_DIR"
            df -h /data

• Step 2: Validate YAML syntax

Run: python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/databento-download-template.yaml'))" && echo OK Expected: OK

• Step 3: Apply the template to the cluster

Run: kubectl apply -f infra/k8s/argo/databento-download-template.yaml Expected: workflowtemplate.argoproj.io/databento-download created

• Step 4: Verify the template exists

Run: argo template get databento-download -n foxhunt Expected: Shows template with all parameters listed

• Step 5: Commit

git add infra/k8s/argo/databento-download-template.yaml
git commit -m "feat: Argo WorkflowTemplate for parameterized Databento downloads"

---

Task 2: Create CLI helper script

Files:

• Create: scripts/argo-download.sh

• Step 1: Write the helper script

#!/usr/bin/env bash
# Download Databento data via Argo Workflows.
#
# Usage:
#   ./scripts/argo-download.sh mbp-10                          # MBP-10 with defaults
#   ./scripts/argo-download.sh trades --output /data/futures-baseline-trades
#   ./scripts/argo-download.sh ohlcv-1s --parallel 4 --pool platform
#   ./scripts/argo-download.sh mbp-10 --watch                  # follow logs
#
# Schemas: ohlcv-1m, ohlcv-1s, trades, mbp-10
#
# Requires: argo CLI
set -euo pipefail

TEMPLATE="databento-download"
PARALLEL=""
OUTPUT=""
POOL=""
CPU_REQ=""
CPU_LIM=""
MEM_REQ=""
MEM_LIM=""
WATCH=false

usage() {
  cat <<EOF
Usage: $(basename "$0") <schema> [OPTIONS]

Schemas:
  ohlcv-1m    OHLCV 1-minute bars
  ohlcv-1s    OHLCV 1-second bars
  trades      Individual trade executions
  mbp-10      10-level order book snapshots

Options:
  --output <dir>       Output dir on PVC (default: schema-based)
  --parallel <n>       Parallel quarterly downloads (default: 9)
  --pool <name>        Node pool (default: ci-compile-cpu)
  --cpu-request <n>    CPU request (default: 4)
  --cpu-limit <n>      CPU limit (default: 16)
  --mem-request <s>    Memory request (default: 8Gi)
  --mem-limit <s>      Memory limit (default: 32Gi)
  --watch              Follow workflow logs after submission
  -h, --help           Show this help
EOF
  exit 0
}

[[ $# -eq 0 ]] && { echo "Error: schema argument required"; usage; }

SCHEMA="$1"; shift

# Validate schema
case "$SCHEMA" in
  ohlcv-1m|ohlcv-1s|trades|mbp-10) ;;
  *) echo "Error: unknown schema '$SCHEMA'"; usage ;;
esac

while [[ $# -gt 0 ]]; do
  case $1 in
    --output)      OUTPUT="$2"; shift 2 ;;
    --parallel)    PARALLEL="$2"; shift 2 ;;
    --pool)        POOL="$2"; shift 2 ;;
    --cpu-request) CPU_REQ="$2"; shift 2 ;;
    --cpu-limit)   CPU_LIM="$2"; shift 2 ;;
    --mem-request) MEM_REQ="$2"; shift 2 ;;
    --mem-limit)   MEM_LIM="$2"; shift 2 ;;
    --watch)       WATCH=true; shift ;;
    -h|--help)     usage ;;
    *)             echo "Unknown option: $1"; usage ;;
  esac
done

# Default output dir based on schema
if [[ -z "$OUTPUT" ]]; then
  case "$SCHEMA" in
    ohlcv-1m) OUTPUT="/data/futures-baseline" ;;
    ohlcv-1s) OUTPUT="/data/futures-baseline-1s" ;;
    trades)   OUTPUT="/data/futures-baseline-trades" ;;
    mbp-10)   OUTPUT="/data/futures-baseline-mbp10" ;;
  esac
fi

# Build argo submit command
CMD="argo submit -n foxhunt --from=wftmpl/$TEMPLATE"
CMD="$CMD -p schema=$SCHEMA"
CMD="$CMD -p output-dir=$OUTPUT"

[[ -n "$PARALLEL" ]] && CMD="$CMD -p parallel=$PARALLEL"
[[ -n "$POOL" ]]     && CMD="$CMD -p node-pool=$POOL"
[[ -n "$CPU_REQ" ]]  && CMD="$CMD -p cpu-request=$CPU_REQ"
[[ -n "$CPU_LIM" ]]  && CMD="$CMD -p cpu-limit=$CPU_LIM"
[[ -n "$MEM_REQ" ]]  && CMD="$CMD -p mem-request=$MEM_REQ"
[[ -n "$MEM_LIM" ]]  && CMD="$CMD -p mem-limit=$MEM_LIM"

if $WATCH; then
  CMD="$CMD --watch"
fi

echo "Submitting Databento download workflow..."
echo "  schema:   $SCHEMA"
echo "  output:   $OUTPUT"
[[ -n "$PARALLEL" ]] && echo "  parallel: $PARALLEL"
[[ -n "$POOL" ]]     && echo "  pool:     $POOL"
echo ""
eval "$CMD"

• Step 2: Make executable

Run: chmod +x scripts/argo-download.sh

• Step 3: Test help output

Run: ./scripts/argo-download.sh --help Expected: Shows usage with schemas and options

• Step 4: Commit

git add scripts/argo-download.sh
git commit -m "feat: argo-download.sh CLI wrapper for Databento download workflow"

---

Task 3: Deprecate old K8s Job file

Files:

• Modify: infra/k8s/jobs/databento-download-streaming.yaml:1-2

• Step 1: Add deprecation header

Add to the top of the file:

# DEPRECATED: Use the Argo WorkflowTemplate instead:
#   argo submit -n foxhunt --from=wftmpl/databento-download -p schema=mbp-10
#   ./scripts/argo-download.sh mbp-10 --parallel 9
#
# This file is kept for reference. The parameterized version is at:
#   infra/k8s/argo/databento-download-template.yaml

• Step 2: Commit

git add infra/k8s/jobs/databento-download-streaming.yaml
git commit -m "chore: deprecate hardcoded download job in favor of Argo template"

---

Task 4: Submit MBP-10 download

• Step 1: Run the download

./scripts/argo-download.sh mbp-10 --parallel 9 --pool ci-compile-cpu --watch

Expected: Argo workflow starts, autoscales a ci-compile-cpu node (32c/64G), downloads 9 quarterly MBP-10 files to /data/futures-baseline-mbp10/ES.FUT/.

• Step 2: Verify data on PVC after completion

kubectl run pvc-verify --rm -it --restart=Never -n foxhunt \
  --image=ubuntu:24.04 \
  --overrides='{"spec":{"nodeSelector":{"k8s.scaleway.com/pool-name":"platform"},"containers":[{"name":"v","image":"ubuntu:24.04","command":["bash","-c","ls -lah /data/futures-baseline-mbp10/ES.FUT/ && du -sh /data/futures-baseline-mbp10/ && df -h /data"],"volumeMounts":[{"name":"d","mountPath":"/data"}]}],"volumes":[{"name":"d","persistentVolumeClaim":{"claimName":"training-data-pvc"}}]}}'

Expected: 9 files (ES.FUT_2024-Q1.dbn.zst through ES.FUT_2026-Q1.dbn.zst), several GB each.