The cargo-target PVC retains the source checkout between workflow runs.
When the container UID differs from the PVC owner, git refuses to operate
with "dubious ownership" error. Adding safe.directory resolves this.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
No cudnn feature flags remain after 6ba52425 — the devel headers
and libs were dead weight (~1.5 GB).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training data is already on the PVC — the rclone sync step was
wasting 2-3 minutes per job. Also fixes --output-dir → --output
to match the actual CLI flag.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: `option_env!("FOXHUNT_BUILD_VERSION")` in common/build_info.rs
was a compile-time macro tracked by cargo fingerprints (Rust 1.80+). Every
pipeline run with a new tag invalidated `common` → cascading rebuild of all
38 dependent workspace crates, even when zero source files changed.
Fix: replace `option_env!()` with `std::env::var()` (runtime LazyLock). Cargo
no longer tracks the version env var, so `common` only recompiles when its
source actually changes.
Also: skip git checkout when HEAD already matches target SHA (zero mtime
changes), and drop `-x` from git clean to preserve gitignored files.
Expected: ~3.7min → <1min for unchanged-crate rebuilds on warm PVC.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Keep gpu-warmup only in compile-and-train-template where it runs
parallel with the CPU compile step. The other templates don't need
it — CI pipeline doesn't always train, and training-workflow already
has the GPU node available from the triggering event.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
git clone gives all files the current timestamp, causing cargo to
recompile every workspace crate even when source is unchanged. Switch
to persistent checkout on the PVC: git fetch + git checkout --force
only updates files that actually changed between commits, preserving
mtimes on unchanged files so cargo correctly skips them.
- compile-services: /cargo-target/src persistent checkout
- compile-training: /cargo-target/src persistent checkout
- compile-and-train: /cargo-target/src persistent checkout + CARGO_HOME
- First run: full clone (fallback), subsequent: fetch + checkout
Expected: ~40 crate recompile → only changed crates (~1-2 min)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
With 37+ workspace crates, cargo's incremental fingerprinting already
skips unchanged crates — sccache adds per-rustc wrapping overhead for
no benefit. The real speedup is CARGO_HOME on PVC (eliminates crate
re-downloads every build).
Kept: CARGO_HOME=/cargo-target/cargo-home on PVC
Removed: RUSTC_WRAPPER=sccache, SCCACHE_DIR, CARGO_INCREMENTAL=0
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both compile-services and compile-training were doing full rebuilds
every run because:
1. No RUSTC_WRAPPER=sccache — sccache was installed but never used
2. No persistent CARGO_HOME — crates.io registry re-downloaded each time
3. Incremental compilation doesn't work across git clones (mtime-based)
Fix: enable sccache (content-hash based, clone-agnostic), persist
CARGO_HOME on the PVC, disable incremental (conflicts with sccache).
Expected: first run populates cache, subsequent runs ~2-3min vs ~8min.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
apply-argo-templates failed because ci-deploy role was missing
permissions for eventbus (argoproj.io), services (core), and
roles/rolebindings (rbac) needed to apply events/*.yaml files.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Argo forbids mixing `dependencies` and `depends` in the same DAG.
Converted all tasks to `depends` with explicit state conditions so
infra steps can wait for image rebuilds while still running when
rebuilds are skipped.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Uses Argo `depends` conditions so apply-argo-templates and
terragrunt-apply wait for rebuild-ci-builder-cpu to finish (or skip)
before running, ensuring they always get the latest image.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
apply-argo-templates needs git (clone repo) + kubectl (apply manifests).
ci-builder-cpu has git but lacked kubectl — added it. foxhunt-runtime
has kubectl but no git and runs as non-root, so can't be used here.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pipeline now auto-applies WorkflowTemplates, EventSources, Sensors,
and RBAC when infra/k8s/argo/ files change — no more manual kubectl.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add terragrunt-apply step to Argo CI pipeline (plan+apply on main push
when infra/live/ or infra/modules/ change)
- Bake OpenTofu 1.9.0 + Terragrunt 0.77.12 into ci-builder-cpu image
with SHA256 checksum verification
- Remove 3 ghost node pools (foxhunt, gitlab, h100-sxm8) from kapsule
module to match Scaleway reality
- Make terragrunt.hcl single source of truth (remove variable defaults)
- Fix GitLab TF state lock methods (POST/DELETE for HTTP backend)
- Harden PAT rotation: more retries, verification step, recovery docs
- Add weekly PAT expiry check CronJob (warns 14 days before expiry)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace fragile `git fetch --depth=1 origin "$SHA"` (fails with short SHAs)
with `git clone --no-checkout --filter=blob:none` + `git checkout "$SHA"`
across all 3 Argo templates (5 clone blocks total)
- Upgrade CI compile pool from POP2-32C-128G to POP2-HC-32C-64G
(higher clock EPYC cores, 28% cheaper at €0.851/hr vs €1.18/hr)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
sccache forces CARGO_INCREMENTAL=0, causing all 37 workspace crates to
recompile from scratch every CI run (~20 min). Only upstream deps were
cached (635 hits); 109 workspace rlib crates were non-cacheable.
Changes:
- Add cargo-target-cpu and cargo-target-cuda PVCs (30Gi each)
- Mount persistent target dir at /cargo-target via CARGO_TARGET_DIR
- Drop RUSTC_WRAPPER=sccache and SCCACHE_DIR from both compile steps
- Drop hardcoded CARGO_BUILD_JOBS=14 (let cargo auto-detect from nproc)
- Add 25GB cleanup guard to prevent unbounded PVC growth
- Update binary copy paths to use $CARGO_TARGET_DIR/release/
First build (cold PVC) is same speed. Subsequent builds with typical
3-5 file changes should drop from ~20 min to ~2-3 min via incremental.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace monolithic compile-all with granular per-binary change detection.
detect-changes now outputs space-separated package/example lists based on
a dependency map from source directories to binary targets:
- Shared crates (common, config, Cargo.toml) → all binaries
- Service-specific dirs → only that service binary
- Domain crates (trading_engine, risk) → dependent service subset
- ML crates → ml-training-service + trading-service + all training
- ML subdirs (trainers/, hyperopt/, evaluation/) → specific training binaries
compile-services and compile-training accept package lists and build only
affected binaries, saving ~20-30s link time per skipped binary.
deploy-services restarts only affected deployments (trading-service
excluded from auto-deploy for safety).
Fix: 'latest' package update now replaces individual files instead of
deleting the entire package, preventing corruption during partial builds.
compile-and-train-template: derive needed training binaries from model
parameter (3 instead of 7), drop unused training_uploader build.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ubuntu 24.04 minimal doesn't include passwd (useradd/groupadd).
Debian bookworm-slim had it by default. Required for foxhunt user.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Cargo defaults to /proc/cpuinfo CPU count (32 host cores) but each
compile pod is cgroup-limited to 14 CPU request. Without this, both
pods spawn 32 threads each (64 total) causing heavy throttling on
the 32-core POP2 node. Setting CARGO_BUILD_JOBS=14 per pod avoids
context-switch overhead and uses cgroup-aware parallelism.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Align all 4 Docker images with local dev environment:
- ci-builder: CUDA 12.4.1/Ubuntu 22.04 → 12.9.1/Ubuntu 24.04
- ci-builder-cpu: Debian bookworm → Ubuntu 24.04 (+ explicit rustup)
- foxhunt-runtime: Debian bookworm → Ubuntu 24.04
- foxhunt-training-runtime: CUDA 12.6.3 → 12.9.1, nvrtc 12-6 → 12-9
All images now have glibc 2.39, matching the local build machine.
Binaries compiled locally or in CI will run in any of these containers
without GLIBC_2.3x version mismatch errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch sccache backend from S3 (MinIO) to persistent local storage.
Two 20Gi PVCs (sccache-cpu, sccache-cuda) on scw-bssd-retain eliminate
network I/O for cache reads/writes. Both compile steps always run on
the same node so RWO is sufficient.
Removes 13 S3 env vars per compile step, replaces with SCCACHE_DIR=/sccache.
Updates ci-pipeline-template, compile-and-train-template, kustomization.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Argo Events labels workflows with events.argoproj.io/trigger, not
workflows.argoproj.io/workflow-template. Fixed find_ci_workflow() so
ci-train preset can actually find CI pipeline runs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New `ci-train` preset monitors the Argo CI workflow for a commit,
polls until completion, then auto-submits the training job (defaults
to hyperopt). Prints Prometheus/log links during monitoring.
Usage: ./infra/scripts/train.sh ci-train --model dqn [--commit SHA]
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three root causes for "no metrics" on the training dashboard:
1. Dashboard template variables ($model, $fold) sourced from
foxhunt_training_current_epoch which isn't emitted until the first
epoch completes. Switch to foxhunt_training_step which fires from
step 500 onward.
2. train.sh pod template missing Prometheus annotations
(prometheus.io/scrape, port, path). Also add the
app.kubernetes.io/component label to the eval manifest so
evaluation pods are discoverable too.
3. DQN and PPO trainers only called set_epoch() at the END of each
epoch. Move the call to the TOP of the epoch loop so the gauge
exists from the first training iteration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove phantom foxhunt_training_total_epochs reference (never registered)
and replace with foxhunt_training_step from the recent metrics commit.
Add full coverage for all 52 Tier-1 Prometheus metrics across 5 new rows:
RL diagnostics (Q-overestimation, PPO entropy/KL/advantage), training
health (NaN/grad explosion/feature errors, checkpoint ops, data load
latency), epoch returns, supervised eval (accuracy/precision/recall/F1),
and hyperopt details (mode, trial epoch, failed trials).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The train_baseline_rl binary expects --output-dir, not --output.
Also adds prometheus.io scrape annotations to job pod templates
so training metrics appear in Grafana.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes validated by 20/20 hyperopt trials on H100 (zero OOM):
1. Workspace default-features: ml-core, ml-dqn, ml-ppo, ml-supervised
workspace deps now have default-features=false. Prevents cudarc
(which requires nvcc) from leaking into CPU service builds via
Cargo feature unification. CI compile-services was failing with
"Failed to execute nvcc: No such file or directory" (exit 101).
2. BF16 comparison fix: Candle's gt()/le() don't support BF16 operands.
Cast ADX/CUSUM features to F32 before threshold comparison in
regime classification. Previous approach (cast threshold to BF16)
failed due to Candle broadcast_as reverting dtype.
3. CI pipeline: expand ML change detection to all 14 sub-crates,
add component:compile labels for sccache network policy matching,
bump training runtime to CUDA 12.6 + Ubuntu 24.04 (glibc 2.39).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bring Branching Dueling Q-Network (Tavakoli 2018) to full Rainbow parity
with the existing GPU hotpath. 3 independent advantage heads (exposure=5,
order=3, urgency=3) decompose the 45-action space into learnable branches.
H1 - CUDA fallback: gate GpuExperienceCollector when use_branching=true
(fused kernel hardcodes NUM_ACTIONS=5, incompatible with 45 factored)
H2 - Per-branch C51 distributional: each branch outputs [batch, n_d, atoms]
log-softmax, loss = avg of D cross-entropies vs projected Bellman target
M1 - NoisyNet: MaybeNoisyLinear enum in branch heads, reset_noise/disable_noise
wired through select_action, compute_loss, and set_eval_mode
M2 - Regime-conditional IS weights: Trending=1.2, Ranging=0.8, Volatile=0.6
applied to branching loss via ADX/CUSUM features at state[40:41]
M3 - State dim alignment: align_dim_for_tensor_cores() in from_dqn_params()
for H100 HMMA dispatch (8-byte alignment)
L1 - Fill simulator: splitmix64 replaces golden ratio hash (chi-squared tested)
L2 - Hyperopt 29D: branch_hidden_dim [64,256] added to PSO search space
Config plumbing: branch_hidden_dim, v_min/v_max/num_atoms, use_distributional,
use_noisy, noisy_sigma_init all flow from DQNConfig → BranchingConfig.
10 files, +3207/-125 lines, 33 branching tests + 387 ml-dqn + 284 ml-core pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a `source` template variable to the training dashboard that controls
which Prometheus job labels are queried:
- Live: only `training-pods` (running pods scraped directly)
- CI History: only `.*_baseline.*` (completed CI runs via pushgateway)
- All: both sources combined
All 37 panel queries now use `job=~"$source"` for consistent filtering.
Active Workers panel stays hardcoded to `job="training-pods"` since it
only makes sense for live pods. Template variable queries (model/fold
dropdowns) also respect the source selector.
Default is "Live" — dashboard shows only the currently running training
session with no pushgateway stale data contamination.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Stat panels (Active Workers, Current Epoch, Trial Progress, Elapsed, Best
Objective) showed conflicting values from pushgateway stale gauges overlapping
with live training pod metrics. Scoped all stat/gauge panels and template
variable queries to job="training-pods" so the dashboard reflects only the
currently running training session. Time-series panels remain unfiltered via
$model template variable scoping — dropdown only lists live models, so
historical pushgateway data is naturally excluded without explicit filtering.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove cuda from default features in ml-core, ml-dqn, ml-ppo, ml
- Propagate cuda feature from ml → ml-core/ml-dqn/ml-ppo
- CI compile-training already uses --features ml/cuda explicitly
- Fix MaxDD log format: {:.1}% → {:.3}% (was rounding 0.033% to 0.0%)
- Suppress unused_labels/unused_variables warnings for cfg(cuda) code
- Add CALLBACK_ENDPOINT env to ml-training-service deployment
- Fix Grafana active_workers query to use sum() with fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prometheus was configured to scrape pods via prometheus.io/scrape
annotations, but all services and training jobs used gitlab.com/
prefix from legacy GitLab-managed Prometheus — causing Prometheus
to never discover any foxhunt pods. This resulted in stale/missing
metrics on the Grafana training dashboard.
12 files updated across services/, gpu-overlays/, training/, and
monitoring/ (node-exporter, dcgm-exporter cleanup).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The nodeSelector pointed to non-existent 'ci-training' pool. The actual
Scaleway pool is 'ci-training-h100', matching the CI pipeline template.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three bugs in the GPU experience collection hot path:
1. gpu_batch_to_experiences() had hardcoded state_dim=43 but the CUDA kernel
outputs states at the ALIGNED dimension (56 with OFI, 48 without). After
sample 0, every replay buffer entry had corrupted state vectors — the
network was learning from garbage data.
2. GPU path never called monitor.track_reward(), so mean_reward was always
reported as 0.0 in epoch logs despite the agent generating real rewards.
3. Action tracking was double-counted (direct array write + track_action_by_exposure),
inflating diversity metrics by 2x. Consolidated into single bounded call.
Also adds missing app.kubernetes.io/component label to job-template.yaml
so Prometheus training-pods scrape job discovers training pods.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GPU experience collector was falling back to CPU because it required a
curiosity VarMap, but curiosity_weight=0.0 means the module is never
created. Fix: make curiosity optional (CuriosityWeightSet::zeros() for
GPU buffers, curiosity_scale=0.0 in kernel config). Also require
explicit binary-tag SHA in Argo training workflow (no "latest" fallback).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add QuestDB ILP sink for training metrics, update Prometheus scrape
configs, and fix network policies for monitoring stack connectivity.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New compile-and-train-template.yaml: compile + GPU warmup in parallel,
then fetch-binary → hyperopt → train-best → evaluate → upload-results
- Compile step outputs SHA tag via Argo output parameter, fetch-binary
uses it directly (no 'latest' package indirection)
- Remove trading-service from CI deploy step — must be explicitly enabled
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: preload_data() called loader.ofi_features.take() on the internal
DQN trainer, but load_training_data() only loads OHLCV bars — it never
populates ofi_features. The OFI loading is done by the hyperopt adapter's
own load_ofi_features() method.
Fixes:
- preload_data() now calls self.load_ofi_features() directly
- load_ofi_features() uses self.mbp10_data_dir when set (was hardcoded ../mbp10)
- input_dim pre-computation uses OFI-aware size (51 when enabled, 43 otherwise)
- CI 'latest' package: delete-then-upload to avoid GitLab duplicate file issue
- Warn when mbp10_data_dir is set but no OFI features loaded
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Rename node-dns-fix → node-bootstrap, fix nvidia gate race condition
(always create conf.d/ and write 50-registry.toml, don't gate on
99-nvidia.toml which doesn't exist on fresh autoscaled GPU nodes)
- Update busybox 1.36 → 1.37
- Fix fetch-binary: use PRIVATE-TOKEN/gitlab-pat (not DEPLOY-TOKEN)
- Fix upload-results: use gitlab-pat (gitlab-ci-token didn't exist)
- Add 'latest' rolling package version in both compile-services and
compile-training (re-upload after CalVer upload)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Service pods' initContainers need egress to gitlab-webservice-default:8181
to fetch release binaries from the Generic Package Registry. Without this,
curl hangs for 2+ minutes then times out (default-deny-all blocks it).
Covers all app.kubernetes.io/part-of=foxhunt pods in one policy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Argo Workflows can't resolve {{tasks.X.outputs}} inside template
bodies — only in DAG task arguments. Pass tag via arguments →
inputs.parameters in compile-services, compile-training,
upload-release, and deploy-services templates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace MinIO binary distribution with GitLab Generic Package Registry.
Every code push to main auto-creates a CalVer tag (vYYYY.MM.N),
compiles, uploads binaries to GitLab packages, creates a Release
with auto-generated notes, and deploys via deployment patching.
- New CI templates: create-tag, upload-release
- Modified: compile-services/training upload to GitLab packages
- Modified: deploy-services patches FOXHUNT_RELEASE on deployments
- All 7 service initContainers fetch from GitLab (curl, deploy token)
- Training job-template binary fetch from GitLab (data stays MinIO)
- MinIO retains: sccache, training data, model checkpoints
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The training-data PVC already has all data (OHLCV, MBP-10, trades).
Mount it read-only at /data in hyperopt/train/evaluate steps instead
of rcloning ~50GB from MinIO every step.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- CI pipeline: add gpu-warmup step parallel to compile-training,
triggers GPU node autoscale so it's ready when training starts
- Training workflow: add mbp10-data-dir and trades-data-dir parameters,
download MBP-10 + trades data in all steps (hyperopt, train, evaluate)
- Use parameterized paths consistently (no hardcoded /tmp paths)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add --mbp10-data-dir and --trades-data-dir CLI args to
hyperopt_baseline_rl binary so hyperopt trials can use real
order book and trade data for VPIN/Kyle's Lambda features.
- DQNTrainer: add mbp10_data_dir/trades_data_dir fields + with_ofi_data_dirs() builder
- DQNHyperparameters: pipe through from trainer instead of hardcoded None
- download-trades-job: fix nodeSelector to ci-compile-cpu (platform pool full)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
