submit-gpu-test wait sidecar had 64Mi limit — OOMKilled when tracking
large child workflow status JSON. Bumped to 256Mi.
Fixed YAML indentation errors in compile-and-train and training-pipeline
templates (misaligned labels, duplicate component keys).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1 writes ${MODEL}_phase1_results.json to PVC, Phase 2 reads it
via --hyperopt-params. Phase 2 uses half the trials but double epochs.
train-best downstream reads final ${MODEL}_hyperopt_results.json unchanged.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
5 templates were missing app.kubernetes.io/component=gpu-test label,
causing MinIO log archival to fail (port 9000 blocked by network policy).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Argo workflow archive via PostgreSQL — persistent logs after pod GC
- Allow Argo controller to reach PostgreSQL via network policy
- Allow Argo server to reach MinIO for S3 log archival
- Replace foxhunt-runtime with lightweight images (alpine:3.21, curlimages/curl)
in orchestration steps: fetch-binary, upload-results, detect-changes, gpu-warmup
- Use ci-builder for training steps (hyperopt, train-best, evaluate) — nvcc required
- Fix --hyperopt-results → --hyperopt-params flag in train-best step
- Remove invalid podGCGracePeriod from Argo Helm values
- Archive RBAC, kustomization updates
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Keep gpu-warmup only in compile-and-train-template where it runs
parallel with the CPU compile step. The other templates don't need
it — CI pipeline doesn't always train, and training-workflow already
has the GPU node available from the triggering event.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GPU experience collector was falling back to CPU because it required a
curiosity VarMap, but curiosity_weight=0.0 means the module is never
created. Fix: make curiosity optional (CuriosityWeightSet::zeros() for
GPU buffers, curiosity_scale=0.0 in kernel config). Also require
explicit binary-tag SHA in Argo training workflow (no "latest" fallback).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Rename node-dns-fix → node-bootstrap, fix nvidia gate race condition
(always create conf.d/ and write 50-registry.toml, don't gate on
99-nvidia.toml which doesn't exist on fresh autoscaled GPU nodes)
- Update busybox 1.36 → 1.37
- Fix fetch-binary: use PRIVATE-TOKEN/gitlab-pat (not DEPLOY-TOKEN)
- Fix upload-results: use gitlab-pat (gitlab-ci-token didn't exist)
- Add 'latest' rolling package version in both compile-services and
compile-training (re-upload after CalVer upload)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The training-data PVC already has all data (OHLCV, MBP-10, trades).
Mount it read-only at /data in hyperopt/train/evaluate steps instead
of rcloning ~50GB from MinIO every step.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- CI pipeline: add gpu-warmup step parallel to compile-training,
triggers GPU node autoscale so it's ready when training starts
- Training workflow: add mbp10-data-dir and trades-data-dir parameters,
download MBP-10 + trades data in all steps (hyperopt, train, evaluate)
- Use parameterized paths consistently (no hardcoded /tmp paths)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
nvidia-smi is driver-mounted by the GPU operator, which may not be
ready when the warmup pod starts on a fresh autoscaled node. The
warmup's purpose is just triggering autoscale, not GPU validation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previous fix removed ALL trailing backslashes from --s3-no-check-bucket,
but 3 of 6 occurrences need the continuation for --transfers=8 on the
next line. Restores \ on lines where --transfers follows.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Same bug as service manifests — \\ after --s3-no-check-bucket caused
chmod to be parsed as rclone args. Fixed in 6 places.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Kubernetes defaults to Always for :latest tags, forcing registry
round-trips that fail on fresh GPU nodes where containerd HTTP-only
registry config has a race condition with HTTPS fallback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Multi-window backtest: splits validation data into 3 non-overlapping
windows and aggregates with mean(Sharpe) - 0.5*std(Sharpe), penalizing
inconsistency and reducing overfit to a single data segment.
Top-K ensemble: hyperopt now emits top_k_params (top 5 trials) in JSON
output. train_baseline_rl gains --ensemble-top-k flag to train multiple
models per fold from different hyperopt configs, saving checkpoints as
dqn_ensemble_{k}_fold_{n}.safetensors.
Workflow template: adds ensemble-top-k parameter (default 5) and passes
--ensemble-top-k to the train-best step.
2720 tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Runs nvidia-smi on GPU node in parallel with fetch-binary, triggering
H100 autoscale during compilation so the node is ready when hyperopt
starts. Exits immediately to free GPU resources.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
containerd on Kapsule nodes uses host DNS which can't resolve
.svc.cluster.local names. Switch all image references from
gitlab-registry.foxhunt.svc.cluster.local:5000 to localhost:30500
(NodePort on the GitLab registry). Also make training runtime image
configurable via TRAINING_RUNTIME_IMAGE env var.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training data PVCs (ReadWriteOnce) don't work with ephemeral GPU nodes
that autoscale across availability zones. Replace with rclone download
from MinIO to /tmp at the start of each GPU step.
- Remove training-data-pvc parameter and volume mount
- Add rclone download from s3:foxhunt-training-data/futures-baseline/
to /tmp/futures-baseline/ in hyperopt, train-best, and evaluate steps
- Add MINIO_ACCESS_KEY/SECRET_KEY env vars to all GPU steps
- Add cuda-compute-cap parameter (default "90")
- Change data-dir default to /tmp/futures-baseline
- 36 .dbn.zst files (53MB total) uploaded to MinIO bucket
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
T=0.1 was too conservative for degenerate configs — trials with uniform
Q-values still collapsed to 1 unique action. Making temperature a
hyperopt parameter lets TPE learn the optimal exploration-exploitation
balance per config. Range [0.01, 2.0] log-scale.
Also sets training-workflow default gpu-pool to ci-training-h100.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Hyperopt requires trials > n_initial. When running quick smoke tests
with few trials, auto-reduce n_initial to trials-1 so workflows don't
fail with "trials must be greater than n_initial".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The default-deny-all NetworkPolicy blocks all egress for foxhunt-labeled
pods. Argo workflow pods need:
- Egress to MinIO (9000) for binary fetch and result upload
- Egress to K8s API (443/6443) for executor task result reporting
- Egress to Tempo (4317) for OTLP traces
- DNS already covered by allow-dns policy
Also fix PVC permission denied by setting fsGroup: 1000 to match the
foxhunt user (UID 1000) in the runtime images.
Smoke test fetch-binary step: Succeeded.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Helm values (controller + server on platform node, MinIO artifact repo)
- WorkflowTemplate: parameterized 5-step DAG (fetch→hyperopt→train→eval→upload)
- Nginx proxy for argo.fxhnt.ai → Argo Server :2746
- DNS A record for argo.fxhnt.ai
- MinIO bucket foxhunt-training-results for Argo artifacts
- Kustomization for kubectl apply -k
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
