Three root causes for "no metrics" on the training dashboard:
1. Dashboard template variables ($model, $fold) sourced from
foxhunt_training_current_epoch which isn't emitted until the first
epoch completes. Switch to foxhunt_training_step which fires from
step 500 onward.
2. train.sh pod template missing Prometheus annotations
(prometheus.io/scrape, port, path). Also add the
app.kubernetes.io/component label to the eval manifest so
evaluation pods are discoverable too.
3. DQN and PPO trainers only called set_epoch() at the END of each
epoch. Move the call to the TOP of the epoch loop so the gauge
exists from the first training iteration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove phantom foxhunt_training_total_epochs reference (never registered)
and replace with foxhunt_training_step from the recent metrics commit.
Add full coverage for all 52 Tier-1 Prometheus metrics across 5 new rows:
RL diagnostics (Q-overestimation, PPO entropy/KL/advantage), training
health (NaN/grad explosion/feature errors, checkpoint ops, data load
latency), epoch returns, supervised eval (accuracy/precision/recall/F1),
and hyperopt details (mode, trial epoch, failed trials).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The train_baseline_rl binary expects --output-dir, not --output.
Also adds prometheus.io scrape annotations to job pod templates
so training metrics appear in Grafana.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes validated by 20/20 hyperopt trials on H100 (zero OOM):
1. Workspace default-features: ml-core, ml-dqn, ml-ppo, ml-supervised
workspace deps now have default-features=false. Prevents cudarc
(which requires nvcc) from leaking into CPU service builds via
Cargo feature unification. CI compile-services was failing with
"Failed to execute nvcc: No such file or directory" (exit 101).
2. BF16 comparison fix: Candle's gt()/le() don't support BF16 operands.
Cast ADX/CUSUM features to F32 before threshold comparison in
regime classification. Previous approach (cast threshold to BF16)
failed due to Candle broadcast_as reverting dtype.
3. CI pipeline: expand ML change detection to all 14 sub-crates,
add component:compile labels for sccache network policy matching,
bump training runtime to CUDA 12.6 + Ubuntu 24.04 (glibc 2.39).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bring Branching Dueling Q-Network (Tavakoli 2018) to full Rainbow parity
with the existing GPU hotpath. 3 independent advantage heads (exposure=5,
order=3, urgency=3) decompose the 45-action space into learnable branches.
H1 - CUDA fallback: gate GpuExperienceCollector when use_branching=true
(fused kernel hardcodes NUM_ACTIONS=5, incompatible with 45 factored)
H2 - Per-branch C51 distributional: each branch outputs [batch, n_d, atoms]
log-softmax, loss = avg of D cross-entropies vs projected Bellman target
M1 - NoisyNet: MaybeNoisyLinear enum in branch heads, reset_noise/disable_noise
wired through select_action, compute_loss, and set_eval_mode
M2 - Regime-conditional IS weights: Trending=1.2, Ranging=0.8, Volatile=0.6
applied to branching loss via ADX/CUSUM features at state[40:41]
M3 - State dim alignment: align_dim_for_tensor_cores() in from_dqn_params()
for H100 HMMA dispatch (8-byte alignment)
L1 - Fill simulator: splitmix64 replaces golden ratio hash (chi-squared tested)
L2 - Hyperopt 29D: branch_hidden_dim [64,256] added to PSO search space
Config plumbing: branch_hidden_dim, v_min/v_max/num_atoms, use_distributional,
use_noisy, noisy_sigma_init all flow from DQNConfig → BranchingConfig.
10 files, +3207/-125 lines, 33 branching tests + 387 ml-dqn + 284 ml-core pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a `source` template variable to the training dashboard that controls
which Prometheus job labels are queried:
- Live: only `training-pods` (running pods scraped directly)
- CI History: only `.*_baseline.*` (completed CI runs via pushgateway)
- All: both sources combined
All 37 panel queries now use `job=~"$source"` for consistent filtering.
Active Workers panel stays hardcoded to `job="training-pods"` since it
only makes sense for live pods. Template variable queries (model/fold
dropdowns) also respect the source selector.
Default is "Live" — dashboard shows only the currently running training
session with no pushgateway stale data contamination.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Stat panels (Active Workers, Current Epoch, Trial Progress, Elapsed, Best
Objective) showed conflicting values from pushgateway stale gauges overlapping
with live training pod metrics. Scoped all stat/gauge panels and template
variable queries to job="training-pods" so the dashboard reflects only the
currently running training session. Time-series panels remain unfiltered via
$model template variable scoping — dropdown only lists live models, so
historical pushgateway data is naturally excluded without explicit filtering.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove cuda from default features in ml-core, ml-dqn, ml-ppo, ml
- Propagate cuda feature from ml → ml-core/ml-dqn/ml-ppo
- CI compile-training already uses --features ml/cuda explicitly
- Fix MaxDD log format: {:.1}% → {:.3}% (was rounding 0.033% to 0.0%)
- Suppress unused_labels/unused_variables warnings for cfg(cuda) code
- Add CALLBACK_ENDPOINT env to ml-training-service deployment
- Fix Grafana active_workers query to use sum() with fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prometheus was configured to scrape pods via prometheus.io/scrape
annotations, but all services and training jobs used gitlab.com/
prefix from legacy GitLab-managed Prometheus — causing Prometheus
to never discover any foxhunt pods. This resulted in stale/missing
metrics on the Grafana training dashboard.
12 files updated across services/, gpu-overlays/, training/, and
monitoring/ (node-exporter, dcgm-exporter cleanup).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The nodeSelector pointed to non-existent 'ci-training' pool. The actual
Scaleway pool is 'ci-training-h100', matching the CI pipeline template.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three bugs in the GPU experience collection hot path:
1. gpu_batch_to_experiences() had hardcoded state_dim=43 but the CUDA kernel
outputs states at the ALIGNED dimension (56 with OFI, 48 without). After
sample 0, every replay buffer entry had corrupted state vectors — the
network was learning from garbage data.
2. GPU path never called monitor.track_reward(), so mean_reward was always
reported as 0.0 in epoch logs despite the agent generating real rewards.
3. Action tracking was double-counted (direct array write + track_action_by_exposure),
inflating diversity metrics by 2x. Consolidated into single bounded call.
Also adds missing app.kubernetes.io/component label to job-template.yaml
so Prometheus training-pods scrape job discovers training pods.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GPU experience collector was falling back to CPU because it required a
curiosity VarMap, but curiosity_weight=0.0 means the module is never
created. Fix: make curiosity optional (CuriosityWeightSet::zeros() for
GPU buffers, curiosity_scale=0.0 in kernel config). Also require
explicit binary-tag SHA in Argo training workflow (no "latest" fallback).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add QuestDB ILP sink for training metrics, update Prometheus scrape
configs, and fix network policies for monitoring stack connectivity.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New compile-and-train-template.yaml: compile + GPU warmup in parallel,
then fetch-binary → hyperopt → train-best → evaluate → upload-results
- Compile step outputs SHA tag via Argo output parameter, fetch-binary
uses it directly (no 'latest' package indirection)
- Remove trading-service from CI deploy step — must be explicitly enabled
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: preload_data() called loader.ofi_features.take() on the internal
DQN trainer, but load_training_data() only loads OHLCV bars — it never
populates ofi_features. The OFI loading is done by the hyperopt adapter's
own load_ofi_features() method.
Fixes:
- preload_data() now calls self.load_ofi_features() directly
- load_ofi_features() uses self.mbp10_data_dir when set (was hardcoded ../mbp10)
- input_dim pre-computation uses OFI-aware size (51 when enabled, 43 otherwise)
- CI 'latest' package: delete-then-upload to avoid GitLab duplicate file issue
- Warn when mbp10_data_dir is set but no OFI features loaded
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Rename node-dns-fix → node-bootstrap, fix nvidia gate race condition
(always create conf.d/ and write 50-registry.toml, don't gate on
99-nvidia.toml which doesn't exist on fresh autoscaled GPU nodes)
- Update busybox 1.36 → 1.37
- Fix fetch-binary: use PRIVATE-TOKEN/gitlab-pat (not DEPLOY-TOKEN)
- Fix upload-results: use gitlab-pat (gitlab-ci-token didn't exist)
- Add 'latest' rolling package version in both compile-services and
compile-training (re-upload after CalVer upload)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Service pods' initContainers need egress to gitlab-webservice-default:8181
to fetch release binaries from the Generic Package Registry. Without this,
curl hangs for 2+ minutes then times out (default-deny-all blocks it).
Covers all app.kubernetes.io/part-of=foxhunt pods in one policy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Argo Workflows can't resolve {{tasks.X.outputs}} inside template
bodies — only in DAG task arguments. Pass tag via arguments →
inputs.parameters in compile-services, compile-training,
upload-release, and deploy-services templates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace MinIO binary distribution with GitLab Generic Package Registry.
Every code push to main auto-creates a CalVer tag (vYYYY.MM.N),
compiles, uploads binaries to GitLab packages, creates a Release
with auto-generated notes, and deploys via deployment patching.
- New CI templates: create-tag, upload-release
- Modified: compile-services/training upload to GitLab packages
- Modified: deploy-services patches FOXHUNT_RELEASE on deployments
- All 7 service initContainers fetch from GitLab (curl, deploy token)
- Training job-template binary fetch from GitLab (data stays MinIO)
- MinIO retains: sccache, training data, model checkpoints
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The training-data PVC already has all data (OHLCV, MBP-10, trades).
Mount it read-only at /data in hyperopt/train/evaluate steps instead
of rcloning ~50GB from MinIO every step.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- CI pipeline: add gpu-warmup step parallel to compile-training,
triggers GPU node autoscale so it's ready when training starts
- Training workflow: add mbp10-data-dir and trades-data-dir parameters,
download MBP-10 + trades data in all steps (hyperopt, train, evaluate)
- Use parameterized paths consistently (no hardcoded /tmp paths)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add --mbp10-data-dir and --trades-data-dir CLI args to
hyperopt_baseline_rl binary so hyperopt trials can use real
order book and trade data for VPIN/Kyle's Lambda features.
- DQNTrainer: add mbp10_data_dir/trades_data_dir fields + with_ofi_data_dirs() builder
- DQNHyperparameters: pipe through from trainer instead of hardcoded None
- download-trades-job: fix nodeSelector to ci-compile-cpu (platform pool full)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Expand training-data-pvc from 10Gi → 200Gi (OHLCV + MBP-10 + headroom)
- Add data-sync-job: rclone sync from MinIO → PVC (delta-only, fast repeats)
- Add sync-training-data init container to training job template
(auto-syncs data from MinIO before each training run)
- Add training-job + data-sync-job NetworkPolicies (MinIO, Tempo, Pushgateway)
- Add app.kubernetes.io/part-of: foxhunt to training pod template
- download_baseline: add --parallel N flag for concurrent Databento downloads
- download_baseline: delete local files after MinIO upload (saves scratch space)
- Bump download job scratch volume to 100Gi
Architecture: Databento → MinIO (source of truth) → PVC (local cache).
First sync pulls everything; subsequent runs only sync deltas (seconds).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The download job pod was missing app.kubernetes.io/part-of: foxhunt,
so the default-deny-all egress policy blocked it and MinIO's ingress
policy rejected it. Also removed hostname pinning (was a misdiagnosis).
Added data-download-job NetworkPolicy allowing egress to MinIO:9000
and external HTTPS:443 (Databento API).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- download_baseline now reads schema from universe TOML config
(was hardcoded to ohlcv-1m, now supports mbp-10 and other schemas)
- Add --rclone-dest flag for direct upload to MinIO after each download
- Add config/universe-es-mbp10.toml: ES.FUT MBP-10, same date range
- Add infra/k8s/training/download-mbp10-job.yaml: K8s Job to download
ES.FUT MBP-10 data in-cluster and upload to MinIO
Usage:
kubectl apply -f infra/k8s/training/download-mbp10-job.yaml
Prereqs: databento-credentials secret, download_baseline binary in MinIO
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove all hardcoded Y-axis min/max bounds (41 removals) — dynamic scaling
- Eliminate Hyperopt section: merge into Training Status + Run Summary
- Deduplicate trial progress (3 panels → 1 gauge + 1 stat)
- Combine ML Jobs + Errors into single panel
- Add Current Trial and Hyperopt Elapsed to status bar
- Add Best Objective Δ (deriv) trend indicator
- Smooth line interpolation + gradient fills on all timeseries
- Stacked area for Action Distribution, gradient fill for Replay Buffer
- Shared crosshair tooltips, consistent threshold colors
- Status bar: stat panels with sparklines, trial before epoch
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
nvidia-smi is driver-mounted by the GPU operator, which may not be
ready when the warmup pod starts on a fresh autoscaled node. The
warmup's purpose is just triggering autoscale, not GPU validation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previous fix removed ALL trailing backslashes from --s3-no-check-bucket,
but 3 of 6 occurrences need the continuation for --transfers=8 on the
next line. Restores \ on lines where --transfers follows.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Same bug as service manifests — \\ after --s3-no-check-bucket caused
chmod to be parsed as rclone args. Fixed in 6 places.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both Dockerfiles already install mold v2.35.1 but the registry images
are stale builds without it. This change triggers rebuild-ci-builder
and rebuild-ci-builder-cpu pipeline steps via detect-changes.
.cargo/config.toml uses -fuse-ld=mold — without mold in the image,
linking falls back to the system default (slower).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Kubernetes defaults to Always for :latest tags, forcing registry
round-trips that fail on fresh GPU nodes where containerd HTTP-only
registry config has a race condition with HTTPS fallback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Argo Events sensor had no nodeSelector and randomly landed on the
H100 GPU node, preventing autoscale-down. Pin it to DEV1-L to avoid
wasting expensive GPU node hours on a lightweight event listener.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Multi-window backtest: splits validation data into 3 non-overlapping
windows and aggregates with mean(Sharpe) - 0.5*std(Sharpe), penalizing
inconsistency and reducing overfit to a single data segment.
Top-K ensemble: hyperopt now emits top_k_params (top 5 trials) in JSON
output. train_baseline_rl gains --ensemble-top-k flag to train multiple
models per fold from different hyperopt configs, saving checkpoints as
dqn_ensemble_{k}_fold_{n}.safetensors.
Workflow template: adds ensemble-top-k parameter (default 5) and passes
--ensemble-top-k to the train-best step.
2720 tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add clamp_max() to PromQL queries and hard Y-axis limits to prevent
early-epoch degenerate values from blowing up panel scaling (Sharpe
showing 12 instead of 1.3, Profit Factor at 30k).
Run Summary: clamp_max on Best Val Loss (5), Sharpe (5), Profit Factor (20)
Training Quality: clamp_max on Epoch Sharpe (5), Sortino (10), PF (20)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace stat panels with timeseries using smooth line interpolation,
gradient fill, and multi-tooltip. Layout changed from 6x1 to 3x2 grid.
Legends show model/fold only when multiple series exist.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 6 stat panels showing overall training run metrics: Best Val Loss,
Best Sharpe, Best Win Rate, Min Max Drawdown, Total Epochs, and Best
Profit Factor. Placed between Training Status and Training Curves rows.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Runs nvidia-smi on GPU node in parallel with fetch-binary, triggering
H100 autoscale during compilation so the node is ready when hyperopt
starts. Exits immediately to free GPU resources.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The foxhunt-runtime image runs as non-root user 'foxhunt' and uses
/bin/sh (dash), not bash. Two issues:
1. &>/dev/null is bash-only — use >/dev/null 2>&1 for POSIX sh
2. Fallback download to /tmp (writable), not /usr/local/bin (root-only)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The build-ci-image template used volumeClaimTemplates (PVC) which
weren't inherited when called via templateRef from ci-pipeline.
Restructured to single pod: init container (alpine/git) clones repo
into emptyDir, main container (kaniko) builds and pushes image from
the same volume. No PVC needed, works correctly via templateRef.
Tested: foxhunt-runtime image rebuilt successfully with kubectl.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
build-ci-image used volumeClaimTemplates (PVC) shared between a
git-clone pod and a kaniko-build pod in a DAG. When called via
templateRef from ci-pipeline, the VCT wasn't inherited, causing
"volume 'workspace' not found" errors.
Fixed by merging into a single pod: init container (git clone) +
main container (kaniko build) sharing an emptyDir volume. No PVC
needed, works correctly when called via templateRef.
Also removed stale compile-training-template.yaml reference from
kustomization (file doesn't exist, template is inline in ci-pipeline).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add kubectl v1.31.4 to foxhunt-runtime Dockerfile so deploy step
doesn't need to download it each run. Deploy step falls back to
curl download if kubectl not found (for current image version).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Deploy step: use foxhunt-runtime image (cached on platform node) +
curl kubectl, replacing bitnami/kubectl:1.31 which doesn't exist.
- Sensor: add podGC:OnPodCompletion and ttlStrategy to workflow spec
(not inherited from WorkflowTemplate via workflowTemplateRef).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three issues prevented services from being deployed after push:
1. No deploy step: pipeline compiled + uploaded binaries to MinIO but
never restarted service deployments. Added deploy-services step
(bitnami/kubectl) that runs rollout restart after compile-services.
2. compile-services couldn't schedule: targeted platform node (4 CPU,
6Gi allocatable) but requested 6Gi memory. Moved to ci-compile-cpu
(POP2-32C-128G) alongside compile-training — both fit simultaneously
(28/32 CPU, 48/128Gi).
3. No podGC: completed Argo pods held resources indefinitely, blocking
subsequent pipeline steps. Added podGC:OnPodCompletion.
Also: detect-changes now triggers service rebuild when CI pipeline
template or k8s service manifests change.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DaemonSet init container now:
- Detects GPU nodes (99-nvidia.toml in conf.d)
- Creates v3-compatible registry config drop-in
- Restarts containerd via chroot if config files changed
- Idempotent: skips restart if files already exist
Requires hostPID + privileged for chroot /proc/1/root.
On new nodes, first run creates files + restarts containerd
(kills pod), second run finds files unchanged (no restart).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Hyperopt: catch early stopping errors and return penalty metrics
instead of aborting entire run. Trials that stop early are scored
as poor (objective=-500) so optimizer avoids those configs.
- DaemonSet: detect GPU nodes (NVIDIA conf.d overlay) and create
v3-compatible registry config drop-in for containerd v2.1+.
Old grpc.v1.cri.registry path is silently ignored by containerd v2.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
