Split the monolithic ml crate (260K lines, 55s compile) into 5 crates:
ml-core, ml-rl, ml-supervised, ml-infra, ml (facade).
15-task plan with full module inventory and import migration guide.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three bugs in the GPU experience collection hot path:
1. gpu_batch_to_experiences() had hardcoded state_dim=43 but the CUDA kernel
outputs states at the ALIGNED dimension (56 with OFI, 48 without). After
sample 0, every replay buffer entry had corrupted state vectors — the
network was learning from garbage data.
2. GPU path never called monitor.track_reward(), so mean_reward was always
reported as 0.0 in epoch logs despite the agent generating real rewards.
3. Action tracking was double-counted (direct array write + track_action_by_exposure),
inflating diversity metrics by 2x. Consolidated into single bounded call.
Also adds missing app.kubernetes.io/component label to job-template.yaml
so Prometheus training-pods scrape job discovers training pods.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PSO can now tune sharpe_weight in [0.0, 0.5] instead of hardcoded 0.3.
This lets hyperopt discover the optimal Sharpe ratio blending weight
in the composite reward signal per-symbol.
Also updates docstring to reflect current C1-C4 search space state.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The GPU experience kernel had hardcoded STATE_DIM=54, MARKET_DIM=51,
NUM_ATOMS_MAX=51 but the host-side feature buffer uploads 40 features
per bar and networks use state_dim=56 with up to 100 atoms. Past ~702K
bars the stride mismatch caused out-of-bounds GPU reads → ILLEGAL_ADDRESS.
All dimension constants now use #ifndef guards so the NVRTC JIT compiler
receives the actual values via #define injection — zero runtime overhead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Hyperopt varies hidden_dim_base (256–1024) but the CUDA experience
kernel hardcoded SHARED_H1=256, SHARED_H2=256. When hyperopt picked
larger dims, the kernel did matvec with wrong strides → illegal memory
access → training crash on first epoch.
Fix: wrap CUDA #defines in #ifndef guards and inject actual dims from
the dueling network config at NVRTC compile time. The kernel is now
specialized per-trial with the exact weight matrix dimensions — zero
runtime overhead, no dimension mismatch possible.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
from_dqn_params() was stripping the last hidden_dim, creating only 1
shared layer (shared_0) when hidden_dims=[256,256]. The CUDA experience
kernel hardcodes 2 shared layers and gpu_weights.rs expects shared_1.*
weight keys. This caused "Missing weight: shared_1.weight" every epoch,
forcing CPU fallback for all experience collection on H100.
Fix: use all hidden_dims as shared layers. Default [256,256] now creates
shared_0 + shared_1, matching the CUDA kernel architecture exactly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The DQN hyperopt adapter has an explicit trades_data_dir field from CLI
args, but load_ofi_features() was only using the derived sibling path.
Now uses the explicit field when available, falling back to derivation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The OFI calculator's compute_ofi_from_file() never called feed_trade(),
leaving 3/8 features (VPIN, Kyle's Lambda, trade_imbalance) at zero in
production. Added compute_ofi_with_trades() that interleaves trades by
timestamp into the MBP-10 streaming callback. Updated DQN and PPO
hyperopt adapters to derive trades_dir as sibling of dbn_data_dir.
Smoke test validates: without trades VPIN=0/5000, with trades VPIN=5000/5000.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests require a CUDA GPU and are #[ignore]d by default. Run with:
cargo test -p ml --test gpu_kernel_parity_test -- --ignored
Covers:
- Standard dueling forward: finite Q-values, valid action range [0,4]
- C51 distributional forward: Q-values bounded by atom support [-25,25]
- Candle vs kernel Q-value parity: argmax action consistency
- NoisyNet exploration: noise injection produces action diversity
- Weight extraction roundtrip: VarMap → CudaSlice → sync
- Distributional weight shapes: value_out [51,128], advantage_out [255,128]
- RMSNorm gamma extraction: initialized to 1.0
- Repeated kernel launches: 5 consecutive runs all produce finite output
All 8 tests pass on RTX 3050 Ti (4.49s total).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Port factorized Gaussian noise exploration and 51-atom categorical value
distribution to the CUDA experience collection kernel, fixing a correctness
bug where GPU Q-values were wrong when use_distributional=true (production
default) due to misinterpreted distributional weight shapes.
- D5: NoisyNet factorized noise (Box-Muller + f(x)=sign(x)*sqrt(|x|)) on
all 6 dueling layers, online network only — target stays deterministic
- D6: C51 distributional dueling forward with per-action atom softmax,
RMSNorm after shared/value/advantage layers, correct [51,128]/[255,128]
weight interpretation
- RmsNormWeightSet extraction and post-epoch sync (GPU-to-GPU)
- Fix get_effective_epsilon() to report actual 2% noisy floor instead of 0.0
- Proportional diversity entropy penalty (continuous gradient vs cliff)
- 2758 tests passing, 0 failures
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GPU experience collector was falling back to CPU because it required a
curiosity VarMap, but curiosity_weight=0.0 means the module is never
created. Fix: make curiosity optional (CuriosityWeightSet::zeros() for
GPU buffers, curiosity_scale=0.0 in kernel config). Also require
explicit binary-tag SHA in Argo training workflow (no "latest" fallback).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The walk-forward backtest in the DQN hyperopt adapter constructed batch
tensors with raw state_dim (51/43) but the model expects aligned
state_dim (56/48). This caused shape mismatch errors during backtest
evaluation: matmul [1024, 51] vs [56, 1024].
Fix: use align_dim_for_tensor_cores() and zero-pad each state vector
before tensor construction, matching the same pattern used in
compute_loss_internal.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Additional padding in get_q_values and convert_to_state — these are
currently unused in the training pipeline but would crash if called.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Validation, action selection, and Q-value monitoring paths were using
raw state dimensions (51) while the model expected aligned dims (56).
Training epoch 1 passed because the GPU pipeline pads correctly, but
validation crashed: shape mismatch [1000,51] vs [56,1024].
Fixed: validation batch, select_actions_batch CPU fallback,
estimate_avg_q_value, compute_q_gap — all now zero-pad to aligned dim.
Also fixed model size estimation log to show aligned dims.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add QuestDB ILP sink for training metrics, update Prometheus scrape
configs, and fix network policies for monitoring stack connectivity.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move BF16 tensor core alignment from per-forward-pass allocation to
data pipeline boundaries. On H100, state_dim 43→48 and 51→56 (8-aligned)
so cuBLAS dispatches HMMA instructions instead of falling back to scalar FMA.
Architecture:
- Trainer computes aligned state_dim at source (align_dim_for_tensor_cores)
- GPU path: DqnGpuData.pad_state_tensor() pads once at upload boundary
- CPU path: train_batch() fold zero-pads Experience.state vectors
- Networks receive pre-aligned tensors — zero per-step overhead
All state_dim defaults updated to aligned values (43→48, 51→56).
Removed pad_to_aligned() from all network forward() methods.
2758 tests pass, 0 failures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
cuBLAS requires M/N/K dimensions to be multiples of 8 for BF16 tensor
core dispatch. state_dim=51 (with OFI) caused silent fallback to scalar
FMA ops, leaving tensor cores at 0.1% utilization despite BF16 enabled.
Pad state_dim to next 8-multiple (51→56, 43→48) at network construction
and zero-pad input tensors in forward pass across all DQN network types:
Sequential, DuelingQNetwork, DistributionalDuelingQNetwork, NetworkLayers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix NaN detection: replace non-existent isnan() with ne(&self) (NaN≠NaN)
- C51 gradient strip: copy().detach() replaces to_vec2→from_vec GPU→CPU→GPU roundtrip
- GPU batch fast path: skip CPU fold + 5× from_vec when GpuBatch available
- Action validation: GPU clamp() replaces CPU loop in GPU batch path
- PER TD errors: keep as GPU Tensor when GpuPrioritized replay active
- PER indices: use GpuBatch.indices directly instead of CPU Vec→Tensor
- IS weights: cache GPU tensor from GpuBatch, reuse in both loss paths
- Logging: Q-values 10→500 steps, diagnostics 100→1000 steps
2758 tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both adapters now call load_ofi_features_parallel from mbp10_loader
instead of duplicating the rayon + streaming logic inline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes:
- Streaming MBP-10 parser (parse_mbp10_streaming): computes OFI inline
during decode — eliminates Vec<Mbp10Snapshot> allocation (41.9M clones)
- Parallel file processing: rayon par_iter across 9 MBP-10 files
(778s sequential → ~90s expected on H100 24-core)
- Fix hardcoded state_dim=54 in walk-forward backtest tensor creation
that caused panic "range end index 55296 out of range for slice of
length 52224" — now uses dynamic state_dim (43 or 51 with OFI)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New compile-and-train-template.yaml: compile + GPU warmup in parallel,
then fetch-binary → hyperopt → train-best → evaluate → upload-results
- Compile step outputs SHA tag via Argo output parameter, fetch-binary
uses it directly (no 'latest' package indirection)
- Remove trading-service from CI deploy step — must be explicitly enabled
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: preload_data() called loader.ofi_features.take() on the internal
DQN trainer, but load_training_data() only loads OHLCV bars — it never
populates ofi_features. The OFI loading is done by the hyperopt adapter's
own load_ofi_features() method.
Fixes:
- preload_data() now calls self.load_ofi_features() directly
- load_ofi_features() uses self.mbp10_data_dir when set (was hardcoded ../mbp10)
- input_dim pre-computation uses OFI-aware size (51 when enabled, 43 otherwise)
- CI 'latest' package: delete-then-upload to avoid GitLab duplicate file issue
- Warn when mbp10_data_dir is set but no OFI features loaded
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Rename node-dns-fix → node-bootstrap, fix nvidia gate race condition
(always create conf.d/ and write 50-registry.toml, don't gate on
99-nvidia.toml which doesn't exist on fresh autoscaled GPU nodes)
- Update busybox 1.36 → 1.37
- Fix fetch-binary: use PRIVATE-TOKEN/gitlab-pat (not DEPLOY-TOKEN)
- Fix upload-results: use gitlab-pat (gitlab-ci-token didn't exist)
- Add 'latest' rolling package version in both compile-services and
compile-training (re-upload after CalVer upload)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Service pods' initContainers need egress to gitlab-webservice-default:8181
to fetch release binaries from the Generic Package Registry. Without this,
curl hangs for 2+ minutes then times out (default-deny-all blocks it).
Covers all app.kubernetes.io/part-of=foxhunt pods in one policy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Argo Workflows can't resolve {{tasks.X.outputs}} inside template
bodies — only in DAG task arguments. Pass tag via arguments →
inputs.parameters in compile-services, compile-training,
upload-release, and deploy-services templates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The hyperopt preload_data() created a loader with default hyperparams
(mbp10_data_dir=None), so MBP-10/trades data was never loaded. Each
trial then set mbp10_data_dir → state_dim=51, but the preloaded data
had no OFI features → shape mismatch [128,43] vs [51,1024].
- Pass mbp10_data_dir/trades_data_dir to preload hyperparams
- Extract ofi_features from loader after preload
- Store as preloaded_ofi_features: Option<Arc<Vec<[f64;8]>>>
- Inject into each trial's DQNTrainer before training
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace MinIO binary distribution with GitLab Generic Package Registry.
Every code push to main auto-creates a CalVer tag (vYYYY.MM.N),
compiles, uploads binaries to GitLab packages, creates a Release
with auto-generated notes, and deploys via deployment patching.
- New CI templates: create-tag, upload-release
- Modified: compile-services/training upload to GitLab packages
- Modified: deploy-services patches FOXHUNT_RELEASE on deployments
- All 7 service initContainers fetch from GitLab (curl, deploy token)
- Training job-template binary fetch from GitLab (data stays MinIO)
- MinIO retains: sccache, training data, model checkpoints
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The GPU path (build_batch_states / build_state_tensor) only concatenated
market[40] + portfolio[3] = 43 dims, while state_dim was set to 51 when
OFI was enabled. This caused shape mismatch [128,43] vs [51,1024].
- Add ofi_features: Option<Tensor> field to DqnGpuData
- Add upload_ofi() method for MBP-10 order book features (8 dims/bar)
- Concatenate OFI in build_batch_states: [count,40]+[count,3]+[count,8]=[count,51]
- Concatenate OFI in build_state_tensor: [1,40]+[1,3]+[1,8]=[1,51]
- Wire trainer to call upload_ofi() after GPU data upload
- Fix CPU path to zero-pad regime_features when OFI enabled but data missing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The training-data PVC already has all data (OHLCV, MBP-10, trades).
Mount it read-only at /data in hyperopt/train/evaluate steps instead
of rcloning ~50GB from MinIO every step.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- CI pipeline: add gpu-warmup step parallel to compile-training,
triggers GPU node autoscale so it's ready when training starts
- Training workflow: add mbp10-data-dir and trades-data-dir parameters,
download MBP-10 + trades data in all steps (hyperopt, train, evaluate)
- Use parameterized paths consistently (no hardcoded /tmp paths)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add --mbp10-data-dir and --trades-data-dir CLI args to
hyperopt_baseline_rl binary so hyperopt trials can use real
order book and trade data for VPIN/Kyle's Lambda features.
- DQNTrainer: add mbp10_data_dir/trades_data_dir fields + with_ofi_data_dirs() builder
- DQNHyperparameters: pipe through from trainer instead of hardcoded None
- download-trades-job: fix nodeSelector to ci-compile-cpu (platform pool full)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Expand training-data-pvc from 10Gi → 200Gi (OHLCV + MBP-10 + headroom)
- Add data-sync-job: rclone sync from MinIO → PVC (delta-only, fast repeats)
- Add sync-training-data init container to training job template
(auto-syncs data from MinIO before each training run)
- Add training-job + data-sync-job NetworkPolicies (MinIO, Tempo, Pushgateway)
- Add app.kubernetes.io/part-of: foxhunt to training pod template
- download_baseline: add --parallel N flag for concurrent Databento downloads
- download_baseline: delete local files after MinIO upload (saves scratch space)
- Bump download job scratch volume to 100Gi
Architecture: Databento → MinIO (source of truth) → PVC (local cache).
First sync pulls everything; subsequent runs only sync deltas (seconds).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The download job pod was missing app.kubernetes.io/part-of: foxhunt,
so the default-deny-all egress policy blocked it and MinIO's ingress
policy rejected it. Also removed hostname pinning (was a misdiagnosis).
Added data-download-job NetworkPolicy allowing egress to MinIO:9000
and external HTTPS:443 (Databento API).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire 8 OFI features (OFI L1/L5, depth imbalance, VPIN, Kyle's lambda,
bid/ask slopes, trade imbalance) through the DQN training pipeline:
- Add mbp10_data_dir config field to DQNHyperparameters
- Dynamic state_dim: 43 (no OFI) or 51 (with OFI) based on config
- Compute OFI per bar during data loading, store on trainer
- Pass OFI features through regime_features slot in TradingState
- Configurable MBP-10 path with recursive .dbn/.dbn.zst discovery
- Add zstd auto-detection to DbnParser::parse_mbp10_file()
- Add --mbp10-data-dir CLI flag to train_baseline_rl
- Fix hardcoded [f64; 51] → FeatureVector51 ([f64; 40]) across
examples, walk_forward, GPU memory profile, and test fixtures
- Fix stale state_dim=51 in dqn_config_2025() and DQN tests
2747 tests pass, 0 failures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- download_baseline now reads schema from universe TOML config
(was hardcoded to ohlcv-1m, now supports mbp-10 and other schemas)
- Add --rclone-dest flag for direct upload to MinIO after each download
- Add config/universe-es-mbp10.toml: ES.FUT MBP-10, same date range
- Add infra/k8s/training/download-mbp10-job.yaml: K8s Job to download
ES.FUT MBP-10 data in-cluster and upload to MinIO
Usage:
kubectl apply -f infra/k8s/training/download-mbp10-job.yaml
Prereqs: databento-credentials secret, download_baseline binary in MinIO
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
batch_greedy_actions, batch_softmax_actions, and
batch_hierarchical_softmax_actions all used self.forward() which routes
through the dist_dueling/dueling/standard Q-network. When use_iqn=true,
the training loss trains the IQN QuantileNetwork but inference never
consulted it — the trained IQN weights were ignored at evaluation time.
Add q_values_for_batch() helper that dispatches to the IQN network
(with CVaR or expected-Q reduction) when use_iqn=true, and wire all
three batch methods through it. select_action, select_action_with_confidence,
and select_action_inference already had correct IQN branches.
Add test_iqn_batch_greedy_actions_uses_iqn_network covering training,
batch greedy, batch softmax, and inference paths with IQN enabled.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add RMSNorm (root mean square normalization) after each hidden layer
in the distributional-dueling Q-network: shared backbone layers, value
stream, and advantage stream. RMSNorm stabilizes activations and
gradients without the overhead of full LayerNorm (no mean centering),
making the network less sensitive to input scale during training.
Architecture per layer: Linear -> LeakyReLU -> RMSNorm
RMSNorm weights are automatically tracked in the existing VarMap since
they are created via VarBuilder::from_varmap with the same shared map.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove all hardcoded Y-axis min/max bounds (41 removals) — dynamic scaling
- Eliminate Hyperopt section: merge into Training Status + Run Summary
- Deduplicate trial progress (3 panels → 1 gauge + 1 stat)
- Combine ML Jobs + Errors into single panel
- Add Current Trial and Hyperopt Elapsed to status bar
- Add Best Objective Δ (deriv) trend indicator
- Smooth line interpolation + gradient fills on all timeseries
- Stacked area for Action Distribution, gradient fill for Replay Buffer
- Shared crosshair tooltips, consistent threshold colors
- Status bar: stat panels with sparklines, trial before epoch
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. Walk-forward windows: replaced 3 non-overlapping with sliding (50% overlap, ~5 windows).
Aggregation changed from mean-0.5*std to median-0.5*IQR for outlier robustness.
2. Composite score: tanh normalization prevents Calmar ratio scale dominance
(0.02% drawdowns → values in thousands drowning out Sharpe/Sortino).
3. Q-value overestimation: new Prometheus gauge foxhunt_training_q_overestimation_ratio,
warning log when ratio>10 or q_mean>5, adaptive tau doubles when Q-mean growth>0.5/epoch
(capped at 0.01), decays back when stable.
2742 tests pass, 0 failures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
nvidia-smi is driver-mounted by the GPU operator, which may not be
ready when the warmup pod starts on a fresh autoscaled node. The
warmup's purpose is just triggering autoscale, not GPU validation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previous fix removed ALL trailing backslashes from --s3-no-check-bucket,
but 3 of 6 occurrences need the continuation for --transfers=8 on the
next line. Restores \ on lines where --transfers follows.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Same bug as service manifests — \\ after --s3-no-check-bucket caused
chmod to be parsed as rclone args. Fixed in 6 places.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both Dockerfiles already install mold v2.35.1 but the registry images
are stale builds without it. This change triggers rebuild-ci-builder
and rebuild-ci-builder-cpu pipeline steps via detect-changes.
.cargo/config.toml uses -fuse-ld=mold — without mold in the image,
linking falls back to the system default (slower).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
