abd7e533bc
## Two architectural cleanups, both surfaced by the wgdc8 experiment ### Part 1: volume_bar_size in cache key Mirrors the imbalance_bar_threshold/ewma_alpha fix from `f7718b376`. The volume bar size constant (100 contracts/bar) was previously hardcoded and not in the fxcache key. Tuning it would have hit the same fossilization bug as imbalance_bar_threshold did pre-fix. Changes: - `Hyperparams.volume_bar_size: u64` field added (default 100, matches `DEFAULT_VOLUME_BAR_SIZE` for backwards compat). - TrainingProfile loader reads `volume_bar_size` TOML key. - `calculate_dbn_cache_key_full` signature 7 → 8 args. Hashed via `to_le_bytes()`. Test `test_cache_key_includes_volume_bar_size` added; passes alongside the 5 existing tests. - 4 callers updated atomically (per `feedback_no_partial_refactor`): `discover_and_load`, `data_loading.rs:146`, `train_baseline_rl.rs:599`, `precompute_features.rs:259,720`. - `data_loading.rs:279` now passes `self.hyperparams.volume_bar_size` to `build_volume_bars` instead of the hardcoded `DEFAULT_VOLUME_BAR_SIZE`. - New `--volume-bar-size` CLI arg on both binaries (default 100). - New Argo workflow params `volume-bar-size` (default "100") and `data-source` (default "mbp10") on both `train-template.yaml` and `train-multi-seed-template.yaml`. Threaded into precompute + trainer invocations. - `scripts/argo-train.sh` exposes `--volume-bar-size <n>` and `--data-source <s>` for ad-hoc overrides. ### Part 2: OFI front-month filter (latent bug fix) `crates/ml/examples/precompute_features.rs:539-557` (the OFI/VPIN/Kyle's Lambda computation branch when MBP-10 + trades data is available) was loading trades unfiltered for per-bar microstructure feature computation. The volume bar formation path filters front-month per-file (line 354), but the OFI path did not. Effect pre-fix: during contract rollover windows (e.g., ESZ24 → ESH25), OFI per-bar microstructure features included trades from BOTH contracts simultaneously, distorting VPIN, Kyle's Lambda, and trade imbalance signals. Severity in production: small (front-month dominates ES.FUT volume by 10-100×) but real and present in every prior MBP-10+trades production run. Fix: mirror the per-file `filter_front_month` call from the volume bar path. Volume bar formation and OFI computation now both see the same in-month trade tape. Added log line shows raw vs filtered count per file for transparency. ## Why bundled Both fixes touch trade-data plumbing in `precompute_features.rs` and the fxcache key contract. Per `feedback_no_partial_refactor`, related architectural cleanups land atomically. Both surfaced from the same wgdc8 audit; bundling avoids two cache-key-invalidating commits in sequence (each would force full fxcache regen). ## Compatibility - `volume_bar_size` defaults to 100 → existing wgdc7-equivalent runs reproduce, but with a *new* fxcache key (the f7718b376-era cache file is unreachable; harmless, can GC manually). - OFI fix is strictly more correct; no opt-out needed. Existing models trained on contaminated OFI features may show slight feature distribution drift on first cache regen — expected, not a regression. - `data_source = "ohlcv"` Argo param now possible; routes precompute through volume bar branch directly. wgdc8 experiment uses this to test bar resolution sensitivity at volume_bar_size=500 (5× DEFAULT). Tests: 6/6 feature_cache tests pass. Workspace + examples compile clean. Audit-doc: `docs/dqn-wire-up-audit.md` updated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
312 lines
13 KiB
Bash
Executable File
312 lines
13 KiB
Bash
Executable File
#!/usr/bin/env bash
|
||
# Train a model via Argo Workflows.
|
||
#
|
||
# Usage:
|
||
# ./scripts/argo-train.sh dqn # defaults: HEAD, H100, 50 epochs
|
||
# ./scripts/argo-train.sh dqn --sha abc1234 # specific commit
|
||
# ./scripts/argo-train.sh dqn --epochs 100 --trials 40 # override training params
|
||
# ./scripts/argo-train.sh ppo --gpu-pool ci-training # L40S instead of H100
|
||
# ./scripts/argo-train.sh dqn --baseline # skip hyperopt
|
||
# ./scripts/argo-train.sh dqn --watch # follow logs
|
||
#
|
||
# Supported models:
|
||
# RL: dqn, ppo
|
||
# Supervised: tft, mamba2, tggn, tlob, liquid, kan, xlstm, diffusion
|
||
#
|
||
# Requires: argo CLI
|
||
set -euo pipefail
|
||
|
||
SHA="HEAD"
|
||
BRANCH="main"
|
||
TRIALS=""
|
||
EPOCHS=""
|
||
GPU_POOL=""
|
||
SYMBOL=""
|
||
WATCH=false
|
||
BASELINE=false
|
||
CAPITAL=""
|
||
SANITIZER="none"
|
||
MULTI_SEED=1
|
||
FOLDS=1
|
||
DRY_RUN=false
|
||
TAG=""
|
||
PROFILE=false
|
||
IMBALANCE_BAR_THRESHOLD=""
|
||
IMBALANCE_BAR_EWMA_ALPHA=""
|
||
VOLUME_BAR_SIZE=""
|
||
DATA_SOURCE=""
|
||
|
||
usage() {
|
||
cat <<EOF
|
||
Usage: $(basename "$0") <model> [OPTIONS]
|
||
|
||
Models:
|
||
dqn, ppo (RL)
|
||
tft, mamba2, tggn, tlob, liquid, kan, xlstm (supervised)
|
||
|
||
Options:
|
||
--sha <commit> Git commit SHA (default: HEAD)
|
||
--branch <branch> Git branch (default: main)
|
||
--trials <n> Hyperopt trials (default: 20)
|
||
--epochs <n> Training epochs (default: 50)
|
||
--gpu-pool <pool> GPU node pool (default: ci-training-h100)
|
||
--symbol <sym> Trading symbol (default: ES.FUT)
|
||
--capital <n> Initial capital (default: 35000)
|
||
--baseline Skip hyperopt (trials=0)
|
||
--sanitizer <tool> Run under compute-sanitizer (memcheck|racecheck|synccheck)
|
||
--watch Follow workflow logs
|
||
--multi-seed <n> Run N seeds in parallel (default: 1, fans out via DAG when >1)
|
||
--folds <k> Walk-forward fold count (default: 1, fans out via DAG when >1)
|
||
--tag <t> Label workflow with foxhunt-tag=<t> for log aggregation
|
||
--imbalance-bar-threshold <f> Override imbalance bar threshold (default: 0.5)
|
||
MUST match between precompute_features and trainer.
|
||
--imbalance-bar-ewma-alpha <f> Override imbalance bar EWMA alpha (default: 0.1).
|
||
α=1.0 disables adaptation (this codebase's reversed
|
||
convention). MUST match between precompute and trainer.
|
||
--volume-bar-size <n> Override volume bar size in contracts (default: 100).
|
||
Used when --data-source != "mbp10". Cache key includes.
|
||
--data-source <s> Data source mode: "mbp10" or "ohlcv" (default: "mbp10").
|
||
Threaded into both precompute and trainer.
|
||
--profile Wrap training under nsys (NVIDIA Nsight Systems) and
|
||
upload .nsys-rep artefacts to MinIO bucket
|
||
foxhunt-training-artifacts/profiles/<sha>/. Forces the
|
||
multi-seed render path (template rendered locally) so
|
||
the nsys wrapper is visible in --dry-run output without
|
||
cluster contact. Plan 5 Task 3 (A.4.1).
|
||
--dry-run Print rendered workflow YAML to stdout, do not submit
|
||
-h, --help Show this help
|
||
EOF
|
||
exit 0
|
||
}
|
||
|
||
[[ $# -eq 0 ]] && { echo "Error: model argument required"; usage; }
|
||
|
||
MODEL="$1"; shift
|
||
|
||
case "$MODEL" in
|
||
dqn|ppo|tft|mamba2|tggn|tlob|liquid|kan|xlstm|diffusion) ;;
|
||
*) echo "Error: unknown model '$MODEL'"; usage ;;
|
||
esac
|
||
|
||
while [[ $# -gt 0 ]]; do
|
||
case $1 in
|
||
--sha) SHA="$2"; shift 2 ;;
|
||
--branch) BRANCH="$2"; shift 2 ;;
|
||
--trials) TRIALS="$2"; shift 2 ;;
|
||
--epochs) EPOCHS="$2"; shift 2 ;;
|
||
--gpu-pool) GPU_POOL="$2"; shift 2 ;;
|
||
--symbol) SYMBOL="$2"; shift 2 ;;
|
||
--capital) CAPITAL="$2"; shift 2 ;;
|
||
--baseline) BASELINE=true; shift ;;
|
||
--sanitizer) SANITIZER="$2"; shift 2 ;;
|
||
--watch) WATCH=true; shift ;;
|
||
--multi-seed) MULTI_SEED="$2"; shift 2 ;;
|
||
--folds) FOLDS="$2"; shift 2 ;;
|
||
--tag) TAG="$2"; shift 2 ;;
|
||
--profile) PROFILE=true; shift ;;
|
||
--imbalance-bar-threshold) IMBALANCE_BAR_THRESHOLD="$2"; shift 2 ;;
|
||
--imbalance-bar-ewma-alpha) IMBALANCE_BAR_EWMA_ALPHA="$2"; shift 2 ;;
|
||
--volume-bar-size) VOLUME_BAR_SIZE="$2"; shift 2 ;;
|
||
--data-source) DATA_SOURCE="$2"; shift 2 ;;
|
||
--dry-run) DRY_RUN=true; shift ;;
|
||
-h|--help) usage ;;
|
||
*) echo "Unknown option: $1"; usage ;;
|
||
esac
|
||
done
|
||
|
||
# Validate seed/fold counts are positive integers.
|
||
if ! [[ "$MULTI_SEED" =~ ^[0-9]+$ ]] || [[ "$MULTI_SEED" -lt 1 ]]; then
|
||
echo "Error: --multi-seed must be a positive integer, got '$MULTI_SEED'"
|
||
exit 1
|
||
fi
|
||
if ! [[ "$FOLDS" =~ ^[0-9]+$ ]] || [[ "$FOLDS" -lt 1 ]]; then
|
||
echo "Error: --folds must be a positive integer, got '$FOLDS'"
|
||
exit 1
|
||
fi
|
||
|
||
# Auto-derive cuda-compute-cap from GPU pool — cubins must match device sm_XX.
|
||
# Default pool is ci-training-h100 (sm_90). Override for other architectures:
|
||
# ci-training-h100* → sm_90 (Hopper)
|
||
# ci-training-l40s → sm_89 (Ada Lovelace)
|
||
# ci-training → sm_89 (alias for L40S — pool is named bare in some
|
||
# clusters; fixed 2026-05-04 after train-mnpf7 deployed
|
||
# with sm_90 cubins on L40S device, requiring terminate
|
||
# + resubmit with explicit --gpu-pool ci-training-l40s).
|
||
case "${GPU_POOL:-ci-training-h100}" in
|
||
*l40s*|ci-training) CUDA_COMPUTE_CAP="89" ;;
|
||
*h100*|*) CUDA_COMPUTE_CAP="90" ;; # default Hopper
|
||
esac
|
||
|
||
# ── Route: single-job (existing template) vs multi-seed DAG (new template) ──
|
||
# Backward compat: --multi-seed 1 --folds 1 keeps the existing single-template
|
||
# call path verbatim. Only when N>1 OR K>1 do we render the matrix DAG.
|
||
#
|
||
# --profile forces the multi-seed render path even at N=K=1 because the
|
||
# single-job dry-run goes through `argo submit --dry-run -o yaml` which
|
||
# resolves the WorkflowTemplate against a live cluster — not available in
|
||
# local CI. The multi-seed renderer reads the template file directly so
|
||
# `--profile --dry-run` works without cluster access. Plan 5 Task 3.
|
||
USE_MULTI_SEED=false
|
||
if [[ "$MULTI_SEED" -gt 1 || "$FOLDS" -gt 1 || "$PROFILE" == "true" ]]; then
|
||
USE_MULTI_SEED=true
|
||
fi
|
||
|
||
if [[ "$USE_MULTI_SEED" == "false" ]]; then
|
||
CMD="argo submit -n foxhunt --from=wftmpl/train"
|
||
CMD="$CMD -p commit-sha=$SHA"
|
||
CMD="$CMD -p git-branch=$BRANCH"
|
||
CMD="$CMD -p model=$MODEL"
|
||
CMD="$CMD -p cuda-compute-cap=$CUDA_COMPUTE_CAP"
|
||
|
||
[[ -n "$TRIALS" ]] && CMD="$CMD -p hyperopt-trials=$TRIALS"
|
||
[[ -n "$EPOCHS" ]] && CMD="$CMD -p train-epochs=$EPOCHS"
|
||
[[ -n "$GPU_POOL" ]] && CMD="$CMD -p gpu-pool=$GPU_POOL"
|
||
[[ -n "$SYMBOL" ]] && CMD="$CMD -p symbol=$SYMBOL"
|
||
[[ -n "$CAPITAL" ]] && CMD="$CMD -p initial-capital=$CAPITAL"
|
||
[[ "$SANITIZER" != "none" ]] && CMD="$CMD -p sanitizer=$SANITIZER"
|
||
[[ -n "$IMBALANCE_BAR_THRESHOLD" ]] && CMD="$CMD -p imbalance-bar-threshold=$IMBALANCE_BAR_THRESHOLD"
|
||
[[ -n "$IMBALANCE_BAR_EWMA_ALPHA" ]] && CMD="$CMD -p imbalance-bar-ewma-alpha=$IMBALANCE_BAR_EWMA_ALPHA"
|
||
[[ -n "$VOLUME_BAR_SIZE" ]] && CMD="$CMD -p volume-bar-size=$VOLUME_BAR_SIZE"
|
||
[[ -n "$DATA_SOURCE" ]] && CMD="$CMD -p data-source=$DATA_SOURCE"
|
||
|
||
$BASELINE && CMD="$CMD -p hyperopt-trials=0"
|
||
$WATCH && CMD="$CMD --watch"
|
||
|
||
if [[ "$DRY_RUN" == "true" ]]; then
|
||
# `argo submit --dry-run -o yaml` renders the resolved Workflow without
|
||
# contacting the cluster. Single-template path emits one Workflow object
|
||
# (no `kind: WorkflowTask` lines — that string only appears in the
|
||
# multi-seed DAG matrix).
|
||
eval "$CMD --dry-run -o yaml"
|
||
exit 0
|
||
fi
|
||
|
||
echo "Submitting $MODEL training workflow..."
|
||
echo " sha: $SHA"
|
||
echo " branch: $BRANCH"
|
||
echo " model: $MODEL"
|
||
$BASELINE && echo " mode: baseline (no hyperopt)"
|
||
[[ -n "$TRIALS" ]] && echo " trials: $TRIALS"
|
||
[[ -n "$EPOCHS" ]] && echo " epochs: $EPOCHS"
|
||
[[ -n "$GPU_POOL" ]] && echo " gpu: $GPU_POOL"
|
||
echo " sm: $CUDA_COMPUTE_CAP"
|
||
echo ""
|
||
eval "$CMD"
|
||
exit 0
|
||
fi
|
||
|
||
# ── Multi-seed path (one job per seed; folds run inside the binary) ──
|
||
# Render the train-multi-seed-template.yaml with the per-seed matrix
|
||
# expanded inline. Plan 5 Task 5 Phase B pivot: was N*K (seed,fold) jobs;
|
||
# is now N (seed-only) jobs because `train_baseline_rl` is a multi-fold
|
||
# walk-forward executor — it accepts `--max-folds K`, NOT `--fold K`. Each
|
||
# rendered task invokes the binary with `--seed N --max-folds K` so the
|
||
# walk-forward sweep happens inside the single training process.
|
||
#
|
||
# The base template ships with a placeholder marker (`# __MATRIX_TASKS__`)
|
||
# which we replace with N generated WorkflowTask stanzas.
|
||
TEMPLATE_SRC="infra/k8s/argo/train-multi-seed-template.yaml"
|
||
if [[ ! -f "$TEMPLATE_SRC" ]]; then
|
||
echo "Error: multi-seed template not found at $TEMPLATE_SRC"
|
||
exit 1
|
||
fi
|
||
|
||
# Build matrix YAML. Each task is a dag.tasks[] entry that targets the
|
||
# `train-single` template with the `seed` parameter bound from inputs.
|
||
# The template forwards `seed` as `--seed` and reads `--max-folds` from the
|
||
# workflow-scoped `folds` parameter, so each task internally runs all K folds.
|
||
build_matrix_tasks() {
|
||
local seeds="$1"
|
||
# 10 spaces — matches the sibling `- name: ensure-binary` list-item indent
|
||
# under `dag.tasks:` (which is itself at 8 spaces). Wrong indent here
|
||
# produces a YAML parse error in the rendered template.
|
||
local indent=" "
|
||
local s
|
||
for ((s=0; s<seeds; s++)); do
|
||
cat <<EOF
|
||
${indent}- name: train-s${s}
|
||
${indent} template: train-single
|
||
${indent} dependencies: [ensure-fxcache, gpu-warmup]
|
||
${indent} arguments:
|
||
${indent} parameters:
|
||
${indent} - name: seed
|
||
${indent} value: "${s}"
|
||
EOF
|
||
done
|
||
}
|
||
|
||
MATRIX_TASKS=$(build_matrix_tasks "$MULTI_SEED")
|
||
|
||
# Substitute the placeholder. Match the *exact* placeholder line (leading
|
||
# whitespace + the marker as the only content on the line) — the marker
|
||
# string also appears in doc comments above and we must not replace those.
|
||
# Use awk (not sed) — multi-line replacement with sed is fragile across
|
||
# platforms.
|
||
RENDERED=$(awk -v repl="$MATRIX_TASKS" '
|
||
/^[[:space:]]*# __MATRIX_TASKS__[[:space:]]*$/ { print repl; next }
|
||
{ print }
|
||
' "$TEMPLATE_SRC")
|
||
|
||
if [[ "$DRY_RUN" == "true" ]]; then
|
||
# Emit the rendered template + a synthetic per-task marker line so test
|
||
# harnesses can count generated jobs without piping through `argo submit`
|
||
# (which would require a live cluster). The `kind: WorkflowTask` marker
|
||
# is what the test_multi_seed_harness.sh asserts against. Plan 5 Task 5
|
||
# Phase B: one marker per seed (not per (seed, fold) pair) because each
|
||
# job now sweeps all K folds via `--max-folds`.
|
||
echo "$RENDERED"
|
||
for ((s=0; s<MULTI_SEED; s++)); do
|
||
echo "# kind: WorkflowTask seed=${s} max_folds=${FOLDS}"
|
||
done
|
||
exit 0
|
||
fi
|
||
|
||
# Submit: write the rendered template to a temp file, apply it as a
|
||
# WorkflowTemplate, then submit a Workflow that references it.
|
||
TMP_TEMPLATE=$(mktemp -t train-multi-seed.XXXXXX.yaml)
|
||
trap 'rm -f "$TMP_TEMPLATE"' EXIT
|
||
echo "$RENDERED" > "$TMP_TEMPLATE"
|
||
|
||
echo "Submitting multi-seed $MODEL training workflow..."
|
||
echo " sha: $SHA"
|
||
echo " branch: $BRANCH"
|
||
echo " model: $MODEL"
|
||
echo " multi-seed: $MULTI_SEED"
|
||
echo " folds: $FOLDS (each job runs all folds via --max-folds)"
|
||
echo " total jobs: $MULTI_SEED"
|
||
[[ "$PROFILE" == "true" ]] && echo " profile: nsys (per-job .nsys-rep upload)"
|
||
[[ -n "$TAG" ]] && echo " tag: $TAG"
|
||
[[ -n "$EPOCHS" ]] && echo " epochs: $EPOCHS"
|
||
[[ -n "$GPU_POOL" ]] && echo " gpu: $GPU_POOL"
|
||
echo " sm: $CUDA_COMPUTE_CAP"
|
||
echo ""
|
||
|
||
# Apply the rendered WorkflowTemplate, then submit a workflow from it.
|
||
kubectl apply -n foxhunt -f "$TMP_TEMPLATE"
|
||
|
||
CMD="argo submit -n foxhunt --from=wftmpl/train-multi-seed"
|
||
CMD="$CMD -p commit-sha=$SHA"
|
||
CMD="$CMD -p git-branch=$BRANCH"
|
||
CMD="$CMD -p model=$MODEL"
|
||
CMD="$CMD -p cuda-compute-cap=$CUDA_COMPUTE_CAP"
|
||
CMD="$CMD -p multi-seed=$MULTI_SEED"
|
||
CMD="$CMD -p folds=$FOLDS"
|
||
CMD="$CMD -p profile=$PROFILE"
|
||
|
||
[[ -n "$TRIALS" ]] && CMD="$CMD -p hyperopt-trials=$TRIALS"
|
||
[[ -n "$EPOCHS" ]] && CMD="$CMD -p train-epochs=$EPOCHS"
|
||
[[ -n "$GPU_POOL" ]] && CMD="$CMD -p gpu-pool=$GPU_POOL"
|
||
[[ -n "$SYMBOL" ]] && CMD="$CMD -p symbol=$SYMBOL"
|
||
[[ -n "$CAPITAL" ]] && CMD="$CMD -p initial-capital=$CAPITAL"
|
||
[[ -n "$TAG" ]] && CMD="$CMD --labels foxhunt-tag=$TAG"
|
||
[[ -n "$IMBALANCE_BAR_THRESHOLD" ]] && CMD="$CMD -p imbalance-bar-threshold=$IMBALANCE_BAR_THRESHOLD"
|
||
[[ -n "$IMBALANCE_BAR_EWMA_ALPHA" ]] && CMD="$CMD -p imbalance-bar-ewma-alpha=$IMBALANCE_BAR_EWMA_ALPHA"
|
||
[[ -n "$VOLUME_BAR_SIZE" ]] && CMD="$CMD -p volume-bar-size=$VOLUME_BAR_SIZE"
|
||
[[ -n "$DATA_SOURCE" ]] && CMD="$CMD -p data-source=$DATA_SOURCE"
|
||
[[ "$SANITIZER" != "none" ]] && CMD="$CMD -p sanitizer=$SANITIZER"
|
||
|
||
$BASELINE && CMD="$CMD -p hyperopt-trials=0"
|
||
$WATCH && CMD="$CMD --watch"
|
||
|
||
eval "$CMD"
|