sccache forces CARGO_INCREMENTAL=0, causing all 37 workspace crates to
recompile from scratch every CI run (~20 min). Only upstream deps were
cached (635 hits); 109 workspace rlib crates were non-cacheable.
Changes:
- Add cargo-target-cpu and cargo-target-cuda PVCs (30Gi each)
- Mount persistent target dir at /cargo-target via CARGO_TARGET_DIR
- Drop RUSTC_WRAPPER=sccache and SCCACHE_DIR from both compile steps
- Drop hardcoded CARGO_BUILD_JOBS=14 (let cargo auto-detect from nproc)
- Add 25GB cleanup guard to prevent unbounded PVC growth
- Update binary copy paths to use $CARGO_TARGET_DIR/release/
First build (cold PVC) is same speed. Subsequent builds with typical
3-5 file changes should drop from ~20 min to ~2-3 min via incremental.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace monolithic compile-all with granular per-binary change detection.
detect-changes now outputs space-separated package/example lists based on
a dependency map from source directories to binary targets:
- Shared crates (common, config, Cargo.toml) → all binaries
- Service-specific dirs → only that service binary
- Domain crates (trading_engine, risk) → dependent service subset
- ML crates → ml-training-service + trading-service + all training
- ML subdirs (trainers/, hyperopt/, evaluation/) → specific training binaries
compile-services and compile-training accept package lists and build only
affected binaries, saving ~20-30s link time per skipped binary.
deploy-services restarts only affected deployments (trading-service
excluded from auto-deploy for safety).
Fix: 'latest' package update now replaces individual files instead of
deleting the entire package, preventing corruption during partial builds.
compile-and-train-template: derive needed training binaries from model
parameter (3 instead of 7), drop unused training_uploader build.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ubuntu 24.04 minimal doesn't include passwd (useradd/groupadd).
Debian bookworm-slim had it by default. Required for foxhunt user.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Cargo defaults to /proc/cpuinfo CPU count (32 host cores) but each
compile pod is cgroup-limited to 14 CPU request. Without this, both
pods spawn 32 threads each (64 total) causing heavy throttling on
the 32-core POP2 node. Setting CARGO_BUILD_JOBS=14 per pod avoids
context-switch overhead and uses cgroup-aware parallelism.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Align all 4 Docker images with local dev environment:
- ci-builder: CUDA 12.4.1/Ubuntu 22.04 → 12.9.1/Ubuntu 24.04
- ci-builder-cpu: Debian bookworm → Ubuntu 24.04 (+ explicit rustup)
- foxhunt-runtime: Debian bookworm → Ubuntu 24.04
- foxhunt-training-runtime: CUDA 12.6.3 → 12.9.1, nvrtc 12-6 → 12-9
All images now have glibc 2.39, matching the local build machine.
Binaries compiled locally or in CI will run in any of these containers
without GLIBC_2.3x version mismatch errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch sccache backend from S3 (MinIO) to persistent local storage.
Two 20Gi PVCs (sccache-cpu, sccache-cuda) on scw-bssd-retain eliminate
network I/O for cache reads/writes. Both compile steps always run on
the same node so RWO is sufficient.
Removes 13 S3 env vars per compile step, replaces with SCCACHE_DIR=/sccache.
Updates ci-pipeline-template, compile-and-train-template, kustomization.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Argo Events labels workflows with events.argoproj.io/trigger, not
workflows.argoproj.io/workflow-template. Fixed find_ci_workflow() so
ci-train preset can actually find CI pipeline runs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New `ci-train` preset monitors the Argo CI workflow for a commit,
polls until completion, then auto-submits the training job (defaults
to hyperopt). Prints Prometheus/log links during monitoring.
Usage: ./infra/scripts/train.sh ci-train --model dqn [--commit SHA]
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three root causes for "no metrics" on the training dashboard:
1. Dashboard template variables ($model, $fold) sourced from
foxhunt_training_current_epoch which isn't emitted until the first
epoch completes. Switch to foxhunt_training_step which fires from
step 500 onward.
2. train.sh pod template missing Prometheus annotations
(prometheus.io/scrape, port, path). Also add the
app.kubernetes.io/component label to the eval manifest so
evaluation pods are discoverable too.
3. DQN and PPO trainers only called set_epoch() at the END of each
epoch. Move the call to the TOP of the epoch loop so the gauge
exists from the first training iteration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove phantom foxhunt_training_total_epochs reference (never registered)
and replace with foxhunt_training_step from the recent metrics commit.
Add full coverage for all 52 Tier-1 Prometheus metrics across 5 new rows:
RL diagnostics (Q-overestimation, PPO entropy/KL/advantage), training
health (NaN/grad explosion/feature errors, checkpoint ops, data load
latency), epoch returns, supervised eval (accuracy/precision/recall/F1),
and hyperopt details (mode, trial epoch, failed trials).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The train_baseline_rl binary expects --output-dir, not --output.
Also adds prometheus.io scrape annotations to job pod templates
so training metrics appear in Grafana.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes validated by 20/20 hyperopt trials on H100 (zero OOM):
1. Workspace default-features: ml-core, ml-dqn, ml-ppo, ml-supervised
workspace deps now have default-features=false. Prevents cudarc
(which requires nvcc) from leaking into CPU service builds via
Cargo feature unification. CI compile-services was failing with
"Failed to execute nvcc: No such file or directory" (exit 101).
2. BF16 comparison fix: Candle's gt()/le() don't support BF16 operands.
Cast ADX/CUSUM features to F32 before threshold comparison in
regime classification. Previous approach (cast threshold to BF16)
failed due to Candle broadcast_as reverting dtype.
3. CI pipeline: expand ML change detection to all 14 sub-crates,
add component:compile labels for sccache network policy matching,
bump training runtime to CUDA 12.6 + Ubuntu 24.04 (glibc 2.39).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bring Branching Dueling Q-Network (Tavakoli 2018) to full Rainbow parity
with the existing GPU hotpath. 3 independent advantage heads (exposure=5,
order=3, urgency=3) decompose the 45-action space into learnable branches.
H1 - CUDA fallback: gate GpuExperienceCollector when use_branching=true
(fused kernel hardcodes NUM_ACTIONS=5, incompatible with 45 factored)
H2 - Per-branch C51 distributional: each branch outputs [batch, n_d, atoms]
log-softmax, loss = avg of D cross-entropies vs projected Bellman target
M1 - NoisyNet: MaybeNoisyLinear enum in branch heads, reset_noise/disable_noise
wired through select_action, compute_loss, and set_eval_mode
M2 - Regime-conditional IS weights: Trending=1.2, Ranging=0.8, Volatile=0.6
applied to branching loss via ADX/CUSUM features at state[40:41]
M3 - State dim alignment: align_dim_for_tensor_cores() in from_dqn_params()
for H100 HMMA dispatch (8-byte alignment)
L1 - Fill simulator: splitmix64 replaces golden ratio hash (chi-squared tested)
L2 - Hyperopt 29D: branch_hidden_dim [64,256] added to PSO search space
Config plumbing: branch_hidden_dim, v_min/v_max/num_atoms, use_distributional,
use_noisy, noisy_sigma_init all flow from DQNConfig → BranchingConfig.
10 files, +3207/-125 lines, 33 branching tests + 387 ml-dqn + 284 ml-core pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a `source` template variable to the training dashboard that controls
which Prometheus job labels are queried:
- Live: only `training-pods` (running pods scraped directly)
- CI History: only `.*_baseline.*` (completed CI runs via pushgateway)
- All: both sources combined
All 37 panel queries now use `job=~"$source"` for consistent filtering.
Active Workers panel stays hardcoded to `job="training-pods"` since it
only makes sense for live pods. Template variable queries (model/fold
dropdowns) also respect the source selector.
Default is "Live" — dashboard shows only the currently running training
session with no pushgateway stale data contamination.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Stat panels (Active Workers, Current Epoch, Trial Progress, Elapsed, Best
Objective) showed conflicting values from pushgateway stale gauges overlapping
with live training pod metrics. Scoped all stat/gauge panels and template
variable queries to job="training-pods" so the dashboard reflects only the
currently running training session. Time-series panels remain unfiltered via
$model template variable scoping — dropdown only lists live models, so
historical pushgateway data is naturally excluded without explicit filtering.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove cuda from default features in ml-core, ml-dqn, ml-ppo, ml
- Propagate cuda feature from ml → ml-core/ml-dqn/ml-ppo
- CI compile-training already uses --features ml/cuda explicitly
- Fix MaxDD log format: {:.1}% → {:.3}% (was rounding 0.033% to 0.0%)
- Suppress unused_labels/unused_variables warnings for cfg(cuda) code
- Add CALLBACK_ENDPOINT env to ml-training-service deployment
- Fix Grafana active_workers query to use sum() with fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prometheus was configured to scrape pods via prometheus.io/scrape
annotations, but all services and training jobs used gitlab.com/
prefix from legacy GitLab-managed Prometheus — causing Prometheus
to never discover any foxhunt pods. This resulted in stale/missing
metrics on the Grafana training dashboard.
12 files updated across services/, gpu-overlays/, training/, and
monitoring/ (node-exporter, dcgm-exporter cleanup).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The nodeSelector pointed to non-existent 'ci-training' pool. The actual
Scaleway pool is 'ci-training-h100', matching the CI pipeline template.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three bugs in the GPU experience collection hot path:
1. gpu_batch_to_experiences() had hardcoded state_dim=43 but the CUDA kernel
outputs states at the ALIGNED dimension (56 with OFI, 48 without). After
sample 0, every replay buffer entry had corrupted state vectors — the
network was learning from garbage data.
2. GPU path never called monitor.track_reward(), so mean_reward was always
reported as 0.0 in epoch logs despite the agent generating real rewards.
3. Action tracking was double-counted (direct array write + track_action_by_exposure),
inflating diversity metrics by 2x. Consolidated into single bounded call.
Also adds missing app.kubernetes.io/component label to job-template.yaml
so Prometheus training-pods scrape job discovers training pods.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GPU experience collector was falling back to CPU because it required a
curiosity VarMap, but curiosity_weight=0.0 means the module is never
created. Fix: make curiosity optional (CuriosityWeightSet::zeros() for
GPU buffers, curiosity_scale=0.0 in kernel config). Also require
explicit binary-tag SHA in Argo training workflow (no "latest" fallback).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add QuestDB ILP sink for training metrics, update Prometheus scrape
configs, and fix network policies for monitoring stack connectivity.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New compile-and-train-template.yaml: compile + GPU warmup in parallel,
then fetch-binary → hyperopt → train-best → evaluate → upload-results
- Compile step outputs SHA tag via Argo output parameter, fetch-binary
uses it directly (no 'latest' package indirection)
- Remove trading-service from CI deploy step — must be explicitly enabled
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: preload_data() called loader.ofi_features.take() on the internal
DQN trainer, but load_training_data() only loads OHLCV bars — it never
populates ofi_features. The OFI loading is done by the hyperopt adapter's
own load_ofi_features() method.
Fixes:
- preload_data() now calls self.load_ofi_features() directly
- load_ofi_features() uses self.mbp10_data_dir when set (was hardcoded ../mbp10)
- input_dim pre-computation uses OFI-aware size (51 when enabled, 43 otherwise)
- CI 'latest' package: delete-then-upload to avoid GitLab duplicate file issue
- Warn when mbp10_data_dir is set but no OFI features loaded
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Rename node-dns-fix → node-bootstrap, fix nvidia gate race condition
(always create conf.d/ and write 50-registry.toml, don't gate on
99-nvidia.toml which doesn't exist on fresh autoscaled GPU nodes)
- Update busybox 1.36 → 1.37
- Fix fetch-binary: use PRIVATE-TOKEN/gitlab-pat (not DEPLOY-TOKEN)
- Fix upload-results: use gitlab-pat (gitlab-ci-token didn't exist)
- Add 'latest' rolling package version in both compile-services and
compile-training (re-upload after CalVer upload)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Service pods' initContainers need egress to gitlab-webservice-default:8181
to fetch release binaries from the Generic Package Registry. Without this,
curl hangs for 2+ minutes then times out (default-deny-all blocks it).
Covers all app.kubernetes.io/part-of=foxhunt pods in one policy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Argo Workflows can't resolve {{tasks.X.outputs}} inside template
bodies — only in DAG task arguments. Pass tag via arguments →
inputs.parameters in compile-services, compile-training,
upload-release, and deploy-services templates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace MinIO binary distribution with GitLab Generic Package Registry.
Every code push to main auto-creates a CalVer tag (vYYYY.MM.N),
compiles, uploads binaries to GitLab packages, creates a Release
with auto-generated notes, and deploys via deployment patching.
- New CI templates: create-tag, upload-release
- Modified: compile-services/training upload to GitLab packages
- Modified: deploy-services patches FOXHUNT_RELEASE on deployments
- All 7 service initContainers fetch from GitLab (curl, deploy token)
- Training job-template binary fetch from GitLab (data stays MinIO)
- MinIO retains: sccache, training data, model checkpoints
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The training-data PVC already has all data (OHLCV, MBP-10, trades).
Mount it read-only at /data in hyperopt/train/evaluate steps instead
of rcloning ~50GB from MinIO every step.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- CI pipeline: add gpu-warmup step parallel to compile-training,
triggers GPU node autoscale so it's ready when training starts
- Training workflow: add mbp10-data-dir and trades-data-dir parameters,
download MBP-10 + trades data in all steps (hyperopt, train, evaluate)
- Use parameterized paths consistently (no hardcoded /tmp paths)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add --mbp10-data-dir and --trades-data-dir CLI args to
hyperopt_baseline_rl binary so hyperopt trials can use real
order book and trade data for VPIN/Kyle's Lambda features.
- DQNTrainer: add mbp10_data_dir/trades_data_dir fields + with_ofi_data_dirs() builder
- DQNHyperparameters: pipe through from trainer instead of hardcoded None
- download-trades-job: fix nodeSelector to ci-compile-cpu (platform pool full)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Expand training-data-pvc from 10Gi → 200Gi (OHLCV + MBP-10 + headroom)
- Add data-sync-job: rclone sync from MinIO → PVC (delta-only, fast repeats)
- Add sync-training-data init container to training job template
(auto-syncs data from MinIO before each training run)
- Add training-job + data-sync-job NetworkPolicies (MinIO, Tempo, Pushgateway)
- Add app.kubernetes.io/part-of: foxhunt to training pod template
- download_baseline: add --parallel N flag for concurrent Databento downloads
- download_baseline: delete local files after MinIO upload (saves scratch space)
- Bump download job scratch volume to 100Gi
Architecture: Databento → MinIO (source of truth) → PVC (local cache).
First sync pulls everything; subsequent runs only sync deltas (seconds).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The download job pod was missing app.kubernetes.io/part-of: foxhunt,
so the default-deny-all egress policy blocked it and MinIO's ingress
policy rejected it. Also removed hostname pinning (was a misdiagnosis).
Added data-download-job NetworkPolicy allowing egress to MinIO:9000
and external HTTPS:443 (Databento API).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- download_baseline now reads schema from universe TOML config
(was hardcoded to ohlcv-1m, now supports mbp-10 and other schemas)
- Add --rclone-dest flag for direct upload to MinIO after each download
- Add config/universe-es-mbp10.toml: ES.FUT MBP-10, same date range
- Add infra/k8s/training/download-mbp10-job.yaml: K8s Job to download
ES.FUT MBP-10 data in-cluster and upload to MinIO
Usage:
kubectl apply -f infra/k8s/training/download-mbp10-job.yaml
Prereqs: databento-credentials secret, download_baseline binary in MinIO
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove all hardcoded Y-axis min/max bounds (41 removals) — dynamic scaling
- Eliminate Hyperopt section: merge into Training Status + Run Summary
- Deduplicate trial progress (3 panels → 1 gauge + 1 stat)
- Combine ML Jobs + Errors into single panel
- Add Current Trial and Hyperopt Elapsed to status bar
- Add Best Objective Δ (deriv) trend indicator
- Smooth line interpolation + gradient fills on all timeseries
- Stacked area for Action Distribution, gradient fill for Replay Buffer
- Shared crosshair tooltips, consistent threshold colors
- Status bar: stat panels with sparklines, trial before epoch
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
nvidia-smi is driver-mounted by the GPU operator, which may not be
ready when the warmup pod starts on a fresh autoscaled node. The
warmup's purpose is just triggering autoscale, not GPU validation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previous fix removed ALL trailing backslashes from --s3-no-check-bucket,
but 3 of 6 occurrences need the continuation for --transfers=8 on the
next line. Restores \ on lines where --transfers follows.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Same bug as service manifests — \\ after --s3-no-check-bucket caused
chmod to be parsed as rclone args. Fixed in 6 places.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both Dockerfiles already install mold v2.35.1 but the registry images
are stale builds without it. This change triggers rebuild-ci-builder
and rebuild-ci-builder-cpu pipeline steps via detect-changes.
.cargo/config.toml uses -fuse-ld=mold — without mold in the image,
linking falls back to the system default (slower).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Kubernetes defaults to Always for :latest tags, forcing registry
round-trips that fail on fresh GPU nodes where containerd HTTP-only
registry config has a race condition with HTTPS fallback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Argo Events sensor had no nodeSelector and randomly landed on the
H100 GPU node, preventing autoscale-down. Pin it to DEV1-L to avoid
wasting expensive GPU node hours on a lightweight event listener.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Multi-window backtest: splits validation data into 3 non-overlapping
windows and aggregates with mean(Sharpe) - 0.5*std(Sharpe), penalizing
inconsistency and reducing overfit to a single data segment.
Top-K ensemble: hyperopt now emits top_k_params (top 5 trials) in JSON
output. train_baseline_rl gains --ensemble-top-k flag to train multiple
models per fold from different hyperopt configs, saving checkpoints as
dqn_ensemble_{k}_fold_{n}.safetensors.
Workflow template: adds ensemble-top-k parameter (default 5) and passes
--ensemble-top-k to the train-best step.
2720 tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add clamp_max() to PromQL queries and hard Y-axis limits to prevent
early-epoch degenerate values from blowing up panel scaling (Sharpe
showing 12 instead of 1.3, Profit Factor at 30k).
Run Summary: clamp_max on Best Val Loss (5), Sharpe (5), Profit Factor (20)
Training Quality: clamp_max on Epoch Sharpe (5), Sortino (10), PF (20)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace stat panels with timeseries using smooth line interpolation,
gradient fill, and multi-tooltip. Layout changed from 6x1 to 3x2 grid.
Legends show model/fold only when multiple series exist.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 6 stat panels showing overall training run metrics: Best Val Loss,
Best Sharpe, Best Win Rate, Min Max Drawdown, Total Epochs, and Best
Profit Factor. Placed between Training Status and Training Curves rows.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
