Per-mount readOnly: true is still set on hyperopt/train/evaluate steps.
Only the precompute step gets write access to generate fxcache.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PVC is ReadWriteOnce — can't mount on compile node (different pool).
Precompute step now:
- runs on GPU node (same as training, shares PVC mount)
- depends on fetch-binary (gets binary from GitLab to /workspace/)
- copies binary to /data/bin/ on PVC for future use
- skips if cache already exists (data unchanged)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- precompute_features early-exits if {cache_key}.fxcache already exists
(cache key = SHA256 of filenames + sizes, only changes when data changes)
- compile-and-train copies binaries to PVC /data/bin/ after compile
- precompute step added to compile-and-train DAG (runs after compile,
before training, parallel with GPU warmup)
- precompute template reads binary from PVC (no GitLab download)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Was relying on stale binary at /data/bin/precompute_features on PVC.
Now downloads latest from GitLab Package Registry (uploaded by
compile-and-train). Falls back to PVC binary if download fails.
Also added precompute_features to compile-and-train examples list.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
cuda-nvcc-12-4 + cuda-cudart-dev-12-4 (~400MB) added to the CPU builder.
Reverts compile-and-deploy back to ci-builder-cpu (was briefly switched
to full ci-builder which is 6GB+ and unnecessary).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
cudarc build.rs requires nvcc to precompile CUDA kernels to .cubin files.
The ci-builder-cpu image doesn't have the CUDA toolkit installed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previously appended /${SYMBOL} to data-dir AND passed --symbol, causing
double nesting. Now data-dir stays as-is and --symbol handles filtering
via collect_dbn_files_filtered.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rust tracing with JSON subscriber in a container without TTY uses
fully-buffered stdout — logs only flush on buffer full or process exit.
This made H100 training appear stuck for 28+ minutes with no output.
stdbuf -oL forces line buffering so epoch logs appear immediately.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Epoch duration self-balances: bigger GPU → bigger auto-scaled batch →
fewer steps per epoch. The manual cap created 7 different values
(0, 8, 64, 100, 200, 300, 2000) across configs/tests/examples, making
behavior inconsistent between environments.
Removed from: DQNHyperparameters, training profiles (smoketest,
localdev, production), CLI args, Argo templates, hyperopt adapter,
all test overrides, supervised example.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
precompute passes data-dir/SYMBOL to match how train_baseline_rl
calls load_all_bars(data_dir, symbol). compile-and-train gets
FOXHUNT_FEATURE_CACHE_DIR env var and --epochs fix.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reuses compile-training, gpu-warmup, fetch-binary templates from
compile-and-train. Own train step runs train_baseline_rl directly
with --epochs, FOXHUNT_FEATURE_CACHE_DIR, and fxcache validation.
No hyperopt — pure baseline with default params.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- DAG: compile (ci-compile-cpu) + gpu-warmup → train (H100)
- Compiles from source targeting compute cap 90
- fxcache REQUIRED — fails if no .fxcache found
- Error chains printed with {:#} for full cause visibility
- Uses cargo-target-pvc for compile cache + sccache
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dedicated workflow for baseline RL training without hyperopt.
Runs train_baseline_rl binary on GPU node with fxcache.
Fetches pre-compiled binary from MinIO, falls back to PVC.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All pods with app.kubernetes.io/part-of=foxhunt get base infra
access. Fixes MinIO log archival timeout for precompute-features
and databento-download workflows. Component-specific policies
still add extra rules (GitLab SSH, external HTTPS, etc.).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previous 32Gi OOM'd with 148GB of MBP-10 data (61M snapshots ≈ 15GB
in memory + trades + OHLCV). ci-compile-cpu node has 64GB.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds feature-cache-dir parameter (default: /data/feature-cache) so
the training binary can load pre-computed .fxcache files.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All downloads now use the Rust download_baseline binary via the
Argo WorkflowTemplate (databento-download).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
500m/512Mi request fits alongside existing platform workloads.
Download is I/O-bound, not CPU-bound.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove runAsUser:1000 so apt-get install ca-certificates works.
Fix double-nested ES.FUT path (output-dir is the schema root, not ES.FUT subdir).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: ubuntu:24.04 has no CA certificates, rustls-native-certs
fails TLS handshake. Fix: apt-get install ca-certificates before
running download binary. Also add port 80 to network policy for apt.
Downloads OHLCV-1m, Trades, OHLCV-1s, MBP-10 via streaming API
(get_range) — parallel quarterly downloads, individual .dbn.zst files.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Set HOME=/tmp to fix pip cache ownership, --no-cache-dir for clean install.
Label compile-and-train matches network policy for HTTPS egress.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Downloads completed batch jobs (mbp-10, trades, ohlcv-1m, ohlcv-1s, mbp-1)
to training-data-pvc. Cleans old data per schema before downloading.
Uses existing databento-credentials secret.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add --min-hold-bars CLI arg to train_baseline_rl and hyperopt_baseline_rl.
Wire through Argo workflow as parameter. Default 5 (TOML), override via
CLI for quick A/B experiments without config changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
templateRef doesn't inherit workflow-level volumes from the referenced
template. All 3 train wrappers (DQN, PPO, supervised) now define
git-ssh-key, training-data, cargo-target-cuda volumes and workspace
volumeClaimTemplate locally.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Migrate from training-pipeline (requires pre-built binary-tag) to
compile-and-train (compiles from source). All 3 train templates now
use the same pattern: thin wrapper → compile-and-train → pipeline.
Consistent params: commit-sha + git-branch instead of binary-tag.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Was using commit-ref which doesn't exist in compile-and-train template.
Now passes commit-sha=HEAD + git-branch=main matching the parent template.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Line 18 had 6-space indent instead of 4 under metadata.labels.
This caused all CI pipelines to fail at apply-argo-templates since our
compile-and-train changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
train_baseline_rl now accepts --initial-capital (default $35K) matching
hyperopt. Argo compile-and-train passes the workflow parameter to the
train-best step. Both hyperopt and training now use consistent capital.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
With family-based 14D search space, Phase Fast already searches all dims.
Phases 2-4 (Full/Reward/Risk) were designed for the old 30D space where
each phase explored different parameter subsets. Now redundant — saves
3× hyperopt compute per training run.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Was referencing training-pipeline which needs binary-tag for pre-built
binaries from GitLab packages. Now references compile-and-train which
compiles from source then trains. No stale binary risk.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
