Rust tracing with JSON subscriber in a container without TTY uses
fully-buffered stdout — logs only flush on buffer full or process exit.
This made H100 training appear stuck for 28+ minutes with no output.
stdbuf -oL forces line buffering so epoch logs appear immediately.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Epoch duration self-balances: bigger GPU → bigger auto-scaled batch →
fewer steps per epoch. The manual cap created 7 different values
(0, 8, 64, 100, 200, 300, 2000) across configs/tests/examples, making
behavior inconsistent between environments.
Removed from: DQNHyperparameters, training profiles (smoketest,
localdev, production), CLI args, Argo templates, hyperopt adapter,
all test overrides, supervised example.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
precompute passes data-dir/SYMBOL to match how train_baseline_rl
calls load_all_bars(data_dir, symbol). compile-and-train gets
FOXHUNT_FEATURE_CACHE_DIR env var and --epochs fix.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reuses compile-training, gpu-warmup, fetch-binary templates from
compile-and-train. Own train step runs train_baseline_rl directly
with --epochs, FOXHUNT_FEATURE_CACHE_DIR, and fxcache validation.
No hyperopt — pure baseline with default params.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- DAG: compile (ci-compile-cpu) + gpu-warmup → train (H100)
- Compiles from source targeting compute cap 90
- fxcache REQUIRED — fails if no .fxcache found
- Error chains printed with {:#} for full cause visibility
- Uses cargo-target-pvc for compile cache + sccache
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dedicated workflow for baseline RL training without hyperopt.
Runs train_baseline_rl binary on GPU node with fxcache.
Fetches pre-compiled binary from MinIO, falls back to PVC.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All pods with app.kubernetes.io/part-of=foxhunt get base infra
access. Fixes MinIO log archival timeout for precompute-features
and databento-download workflows. Component-specific policies
still add extra rules (GitLab SSH, external HTTPS, etc.).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previous 32Gi OOM'd with 148GB of MBP-10 data (61M snapshots ≈ 15GB
in memory + trades + OHLCV). ci-compile-cpu node has 64GB.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds feature-cache-dir parameter (default: /data/feature-cache) so
the training binary can load pre-computed .fxcache files.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All downloads now use the Rust download_baseline binary via the
Argo WorkflowTemplate (databento-download).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
500m/512Mi request fits alongside existing platform workloads.
Download is I/O-bound, not CPU-bound.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove runAsUser:1000 so apt-get install ca-certificates works.
Fix double-nested ES.FUT path (output-dir is the schema root, not ES.FUT subdir).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: ubuntu:24.04 has no CA certificates, rustls-native-certs
fails TLS handshake. Fix: apt-get install ca-certificates before
running download binary. Also add port 80 to network policy for apt.
Downloads OHLCV-1m, Trades, OHLCV-1s, MBP-10 via streaming API
(get_range) — parallel quarterly downloads, individual .dbn.zst files.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Set HOME=/tmp to fix pip cache ownership, --no-cache-dir for clean install.
Label compile-and-train matches network policy for HTTPS egress.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Downloads completed batch jobs (mbp-10, trades, ohlcv-1m, ohlcv-1s, mbp-1)
to training-data-pvc. Cleans old data per schema before downloading.
Uses existing databento-credentials secret.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add --min-hold-bars CLI arg to train_baseline_rl and hyperopt_baseline_rl.
Wire through Argo workflow as parameter. Default 5 (TOML), override via
CLI for quick A/B experiments without config changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
templateRef doesn't inherit workflow-level volumes from the referenced
template. All 3 train wrappers (DQN, PPO, supervised) now define
git-ssh-key, training-data, cargo-target-cuda volumes and workspace
volumeClaimTemplate locally.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Migrate from training-pipeline (requires pre-built binary-tag) to
compile-and-train (compiles from source). All 3 train templates now
use the same pattern: thin wrapper → compile-and-train → pipeline.
Consistent params: commit-sha + git-branch instead of binary-tag.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Was using commit-ref which doesn't exist in compile-and-train template.
Now passes commit-sha=HEAD + git-branch=main matching the parent template.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Line 18 had 6-space indent instead of 4 under metadata.labels.
This caused all CI pipelines to fail at apply-argo-templates since our
compile-and-train changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
train_baseline_rl now accepts --initial-capital (default $35K) matching
hyperopt. Argo compile-and-train passes the workflow parameter to the
train-best step. Both hyperopt and training now use consistent capital.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
With family-based 14D search space, Phase Fast already searches all dims.
Phases 2-4 (Full/Reward/Risk) were designed for the old 30D space where
each phase explored different parameter subsets. Now redundant — saves
3× hyperopt compute per training run.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Was referencing training-pipeline which needs binary-tag for pre-built
binaries from GitLab packages. Now references compile-and-train which
compiles from source then trains. No stale binary risk.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
argo-training-workflow was missing DNS egress (port 53), causing
MinIO hostname resolution to timeout. Also missing GitLab SSH (2222),
GitLab API (8181), Pushgateway (9091), and Registry (5000).
Now matches the gpu-test-workflow netpol which works correctly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- train-dqn: hyperopt-trials 30→50, hyperopt-epochs 8→20,
train-epochs 80→200 (matches dqn-production.toml, 24D search space)
- training-workflow: fixed --max-steps-per-epoch bug (was passing
train-epochs as max-steps, limiting training to 80 steps/epoch
instead of 80 epochs). Changed to --epochs.
- Added FOXHUNT_TRAINING_PROFILE env var pointing to
dqn-production.toml so the training binary uses the correct config
(adam_epsilon=1e-8, cosine LR, drawdown penalty, etc.)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The feature cache auto-invalidates via content hash — manual rm -f
wasted ~2 minutes of MBP-10 + OFI recomputation per trial start.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- DSR warm-up: skip first 50 steps when EMA has insufficient history
- Phase Fast num_atoms: 51 → 101 (H100 can afford finer resolution,
1.19 per atom vs 2.35 — critical for distinguishing Q-values)
- Argo template: clear stale feature cache before hyperopt (ensures
fresh computation with VPIN/trades enrichment)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Without CARGO_TARGET_DIR, cubin cache goes to /tmp (ephemeral).
With it, cubins persist at /workspace/.cubin_cache/ on the PVC
across runs — saves ~30s of nvcc compilation per trial.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- test_embedded_h100_parses: update assertions to match h100.toml values
(gpu_n_episodes=2048, gpu_timesteps_per_episode=100)
- dqn_training_smoke_test: apply dqn-smoketest profile to cap hidden_dim=32.
H100's gpu profile sets hidden_dim_base=256 which causes loss explosion
(375x in 3 epochs) with lr=0.001.
- Revert gpu-test-pipeline DAG to compile-and-test (RWO PVC constraint)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
TOML training profiles are include_str!() into the ml binary at compile
time. Changes to config/training/ or config/gpu/ must trigger GPU test
rebuild, otherwise the binary runs with stale embedded defaults.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Argo 'main' (emissary) container was the OOMKilled one, not 'wait'.
podSpecPatch now targets both containers to prevent executor OOM when
tracking large child workflow status JSON.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
submit-gpu-test wait sidecar had 64Mi limit — OOMKilled when tracking
large child workflow status JSON. Bumped to 256Mi.
Fixed YAML indentation errors in compile-and-train and training-pipeline
templates (misaligned labels, duplicate component keys).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1 writes ${MODEL}_phase1_results.json to PVC, Phase 2 reads it
via --hyperopt-params. Phase 2 uses half the trials but double epochs.
train-best downstream reads final ${MODEL}_hyperopt_results.json unchanged.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
