- Move UnifiedTrainable trait, TrainingMetrics, CheckpointMetadata, and
checkpoint helpers from ml to ml-core (zero new dependencies — ml-core
already had candle-core + serde_json)
- Wrap Mamba2SSM in Mamba2TrainableAdapter to satisfy orphan rule (trait
in ml-core, type in ml-supervised — all other 9 models already used
wrapper pattern)
- Make Mamba2SSM::validate() pub for cross-crate adapter access
- Delete 5 permanently disabled deployment modules (cfg(any()) — never
compiled): registry, hot_swap, validation, monitoring, endpoints
(-5,842 lines)
- Delete 2 undeclared dead files in training/: dqn_trainer.rs,
transformer_trainer.rs (-138 lines)
- Fix pre-existing compute_loss test shape mismatch in mamba adapter
UnifiedTrainable in ml-core unblocks future trainers/ extraction (17.8K
lines) since model-specific trainers can now depend on ml-core for the
trait without pulling in the full ml monolith.
14 files changed, +128 -6,497 (net -6,369 lines)
Tests: 274 ml-core + 948 ml = 1,222 passed, 0 failed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add QuestDB ILP sink for training metrics, update Prometheus scrape
configs, and fix network policies for monitoring stack connectivity.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add --mbp10-data-dir and --trades-data-dir CLI args to
hyperopt_baseline_rl binary so hyperopt trials can use real
order book and trade data for VPIN/Kyle's Lambda features.
- DQNTrainer: add mbp10_data_dir/trades_data_dir fields + with_ofi_data_dirs() builder
- DQNHyperparameters: pipe through from trainer instead of hardcoded None
- download-trades-job: fix nodeSelector to ci-compile-cpu (platform pool full)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Expand training-data-pvc from 10Gi → 200Gi (OHLCV + MBP-10 + headroom)
- Add data-sync-job: rclone sync from MinIO → PVC (delta-only, fast repeats)
- Add sync-training-data init container to training job template
(auto-syncs data from MinIO before each training run)
- Add training-job + data-sync-job NetworkPolicies (MinIO, Tempo, Pushgateway)
- Add app.kubernetes.io/part-of: foxhunt to training pod template
- download_baseline: add --parallel N flag for concurrent Databento downloads
- download_baseline: delete local files after MinIO upload (saves scratch space)
- Bump download job scratch volume to 100Gi
Architecture: Databento → MinIO (source of truth) → PVC (local cache).
First sync pulls everything; subsequent runs only sync deltas (seconds).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire 8 OFI features (OFI L1/L5, depth imbalance, VPIN, Kyle's lambda,
bid/ask slopes, trade imbalance) through the DQN training pipeline:
- Add mbp10_data_dir config field to DQNHyperparameters
- Dynamic state_dim: 43 (no OFI) or 51 (with OFI) based on config
- Compute OFI per bar during data loading, store on trainer
- Pass OFI features through regime_features slot in TradingState
- Configurable MBP-10 path with recursive .dbn/.dbn.zst discovery
- Add zstd auto-detection to DbnParser::parse_mbp10_file()
- Add --mbp10-data-dir CLI flag to train_baseline_rl
- Fix hardcoded [f64; 51] → FeatureVector51 ([f64; 40]) across
examples, walk_forward, GPU memory profile, and test fixtures
- Fix stale state_dim=51 in dqn_config_2025() and DQN tests
2747 tests pass, 0 failures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- download_baseline now reads schema from universe TOML config
(was hardcoded to ohlcv-1m, now supports mbp-10 and other schemas)
- Add --rclone-dest flag for direct upload to MinIO after each download
- Add config/universe-es-mbp10.toml: ES.FUT MBP-10, same date range
- Add infra/k8s/training/download-mbp10-job.yaml: K8s Job to download
ES.FUT MBP-10 data in-cluster and upload to MinIO
Usage:
kubectl apply -f infra/k8s/training/download-mbp10-job.yaml
Prereqs: databento-credentials secret, download_baseline binary in MinIO
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Critical bug: all 3 DQN action selection methods (select_action,
select_action_with_confidence, select_action_inference) used
FactoredAction::from_index() which maps indices 0-4 to exposure_idx=0
(Short100) via division by 9. This is the root cause of action
diversity collapse during both training and production inference.
Fix: ExposureLevel::from_index() + OrderRouter::route_default() in all
DQN paths. Also fixes hyperopt objective thresholds (<10 → <3 for
5-action degenerate detection), stale defaults/comments, integration
test configs.
Files: dqn.rs (3 methods), trainer.rs (validation + select_action),
hyperopt/adapters/dqn.rs (thresholds), dqn_model.rs (comments),
train_baseline_rl.rs (default), reward.rs (comment),
dqn_integration.rs + ensemble_integration.rs (num_actions).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Multi-window backtest: splits validation data into 3 non-overlapping
windows and aggregates with mean(Sharpe) - 0.5*std(Sharpe), penalizing
inconsistency and reducing overfit to a single data segment.
Top-K ensemble: hyperopt now emits top_k_params (top 5 trials) in JSON
output. train_baseline_rl gains --ensemble-top-k flag to train multiple
models per fold from different hyperopt configs, saving checkpoints as
dqn_ensemble_{k}_fold_{n}.safetensors.
Workflow template: adds ensemble-top-k parameter (default 5) and passes
--ensemble-top-k to the train-best step.
2720 tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ephemeral Argo workflow pods terminate after training completes, causing
Prometheus to lose all scraped metrics. Add push_to_gateway() to POST
final metrics to the existing pushgateway service so they persist on the
Grafana training dashboard after pod completion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace Silverman's bandwidth (h = 1.06σn^(-1/5)) with Scott's rule
(h = 0.7σn^(-1/(d+4))) for tighter kernels in high-D parameter spaces
- Add best-trial injection: always evaluate EI at best known point plus
5 small perturbations (±5%), preventing optimizer from forgetting peaks
- Scale n_candidates dynamically: max(256, 8*n_dims) instead of fixed 100
- Reduce gamma from 0.25 to 0.15 when trials < 50 for tighter exploitation
- Wire model_name through PSO/TPE paths for per-trial Prometheus metrics
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fixes from deep investigation audit (LOW/MEDIUM priority):
1. CVaR penalty: hard cliff (0 or 10) → smooth ramp with gradient signal
for PSO. Formula: min(10, max(0, -cvar-0.05)*200).
2. Clip outliers leakage: data_loading.rs now computes clip bounds from
training portion only (first 80%), then applies to full series.
Log returns and windowed normalize are causal (no leakage).
3. Noisy sigma scheduler: hyperopt now matches conservative() defaults
(enabled, initial=0.8, final=0.4) so hyperopt-found params
generalize to train_best without scheduler mismatch.
4. evaluate_supervised.rs: NormStats fallback from test data (leakage)
replaced with bail! matching evaluate_baseline.rs behavior.
5. Doc comments: stale 27D references updated to 31D (4 locations).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add device_pool to DQN/PPO hyperopt trainers for round-robin GPU assignment
per trial. Binary detects all CUDA devices, scales VRAM budget by GPU count.
Single-GPU: no behavior change (pool of 1).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
NoisyLinear layers have different parameter names (sigma_weight/sigma_bias)
than standard Linear layers. Hardcoding use_noisy_nets=false caused
checkpoint loading to silently fail when the training run used noisy nets,
resulting in empty folds in the eval report.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
num_atoms, dueling_hidden_dim, v_min/v_max, gamma were using defaults
instead of hyperopt values — causing tensor shape mismatch on checkpoint
load (e.g. output layer 45×200=9000 vs 45×51=2295). Also fixed use_iqn
to read from use_qr_dqn key (matching trainer's field mapping).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Hyperopt found hold_penalty_weight=1.818, curiosity_weight=0.402, etc.
but walk-forward training used tiny defaults (0.01, 0.1), causing the
agent to learn "holding is optimal" → 0 trades across all 50 epochs.
Now passes: hold_penalty_weight, max_position_absolute, huber_delta,
entropy_coefficient, curiosity_weight, weight_decay, kelly_fractional,
kelly_max_fraction from hyperopt JSON to DQNHyperparameters.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Walk-forward DQN training hardcoded epsilon_start=1.0 with noisy_nets=true,
forcing pure random exploration for all 50 epochs (0 trades, 0 Sharpe).
Now reads epsilon, PER, dueling, distributional, noisy nets, and QR-DQN
params from the hyperopt JSON. When noisy_nets=true, defaults epsilon to
0.05 instead of 1.0.
Evaluator now falls back to highest dqn_fold{N}_epoch{E}.safetensors when
_best.safetensors doesn't exist (early stopping saves epoch checkpoints).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire LR scheduler to actually update the Adam optimizer (was logged but
never applied). Add update_learning_rate to DQN, RegimeConditionalDQN,
and DQNAgentType so decay_factor propagates through all agent variants.
Fix train/eval parity: CLI defaults 51→54 features, 3→45 actions;
DQN eval always uses 3-layer hidden_dims; PPO eval uses 5-layer value
network matching trainer. enhanced_ml.rs hardcoded config updated from
state_dim=16/num_actions=3 to 54/45.
Fix Candle F32/BF16 traps: replace `Tensor * 0.5` (f64 literal) with
broadcast_mul(Tensor::full(0.5_f32)) in quantile_regression.rs (2x),
dqn.rs Huber loss, and IQN gamma multiplication. Prevents panics on
Ampere+ BF16 GPUs.
Add urgency_weight() multiplier to training cost model in reward.rs
and portfolio_tracker.rs — urgency dimension (Patient/Normal/Aggressive)
now affects learned value function, matching evaluate_baseline.rs.
Fix equity tracking: additive (equity += ret) → multiplicative
(equity *= 1.0 + ret) in compute_metrics. Fix total return calc.
Fix PER beta annealing: epochs*70 → epochs*1000 (~1015 actual steps
per epoch for 130k bars / batch 128).
Fix silent target network freeze: mutex lock failure now propagates
error instead of silently skipping weight update.
Make save_checkpoint atomic (write .tmp then rename). Make NormStats
write atomic with error logging instead of silent discard.
Convert EnsembleConfig::new assert! → Result<Self, MLError> with
14 call site updates. Fix hyperopt result serialization to warn
instead of silently dropping to Value::Null.
Fix clippy MSRV mismatch: clippy.toml 1.75 → 1.85 matching Cargo.toml.
2732 tests pass, 0 clippy warnings across workspace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- DQNAgent::load_from_safetensors: validate architecture hash before
loading weights (was bypassing validation entirely - P0 production gap)
- DQNEnsemble::save_to_directory: embed architecture metadata via
safetensors::serialize_to_file instead of bare VarMap::save (save/load
round-trip was guaranteed to fail)
- architecture_hash: hash hidden_dims.len() before values to prevent
theoretical collision between different-length configs
- train_baseline_rl: NormStats write now atomic (write .tmp then rename)
with cleanup guard on rename failure; serialization error is now loud
(error! + return None) instead of silently discarded
- train_baseline_rl: checkpoint rename failure cleans up orphaned .tmp
- hyperopt_baseline_rl: fix clippy single_match_else (match on equality
check -> if/else)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Wire position-delta cost model: costs only apply on position changes,
proportional to delta (not flat per-bar)
- Differentiate order types: Market=15bps, LimitMaker=5bps, IoC=10bps
- Apply urgency weight: Patient=0.5x, Normal=1.0x, Aggressive=1.5x
- Append 3 portfolio features (position, unrealized PnL, drawdown) to match
training's 54-dim state vector (was 51, causing shape mismatch on load)
- Fix default num_actions 3->45 to match training's factored action space
- Add --bars-per-year CLI flag for per-symbol annualization
(ES/NQ=347760, 6E=345000, ZN=105840)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Atomic checkpoint writes: write to .tmp then fs::rename() (POSIX atomic)
- Remove redundant thread::sleep(50ms) after CUDA tensor readback sync
- NormStats: bail on missing file instead of computing from test data (lookahead bias)
- q_value_std: warn when missing from training metrics instead of silent 0.0 fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change --optimizer default from "pso" to "tpe" since TPE has better
sample efficiency in 25D spaces. Fix clippy let_underscore_must_use
on CUDA sync tensor readback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add optimize_with_tpe() function that uses Tree-Parzen Estimator for
Bayesian hyperparameter optimization. Add --optimizer=tpe CLI flag
to hyperopt_baseline_rl binary (default: pso for backward compat).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split model overhead into two constants: MODEL_OVERHEAD_MB (pure
model weights for batch-size capping) and TRIAL_VRAM_MB (total
per-trial VRAM for concurrent hyperopt planning). DQN trials
empirically consume ~7 GB each on L40S (model + GPU replay buffer +
experience collector + CUDA allocations + fragmentation), not the
200 MB previously estimated. This caused plan_hyperopt to compute
128 concurrent trials instead of the actual 5, inflating PSO
particles from 20→128 and total trials from 20→384 via .max()
instead of .min(), guaranteeing a 4h timeout kill.
Fix auto-scaling to: (1) match particles to GPU concurrency for
maximum hardware utilization on any node, (2) cap particles at
max_trials to never inflate the trial budget, (3) never auto-inflate
the total trial count.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The auto-detect heuristic used cpus/2 ("smart cap"), designed for
CPU-bound workloads. DQN/PPO trials are GPU-bound — each rayon
thread submits CUDA kernels and waits on cudaDeviceSynchronize(),
using minimal CPU. cpus-1 is the correct cap.
On L40S-1-48G (8 vCPU): 3 threads → 7 threads (2.3× more trials).
Also bumps CI CPU limit 7500m→8000m to expose all 8 cores.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The batch span processor needs a tokio runtime for gRPC transport and
periodic flush. Async services already have one via #[tokio::main], but
sync training binaries (hyperopt, train, evaluate) don't.
Previous approach (making binaries async with #[tokio::main]) caused
"Cannot start a runtime from within a runtime" panics because the ML
crate's internal code creates its own tokio runtimes for block_on().
New approach: build_otel_tracer() detects runtime context via
Handle::try_current(). If absent, it creates a dedicated 1-worker
multi-thread runtime stored in a process-lifetime OnceLock. The worker
thread actively polls the OTLP batch export task.
Reverts training binaries to sync fn main() so internal runtime creation
(hyperopt adapters, DQN/PPO trainers) continues working as before.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All 6 training binaries (hyperopt_baseline_rl, hyperopt_baseline_supervised,
train_baseline_rl, train_baseline_supervised, evaluate_baseline,
evaluate_supervised) used sync fn main() but the OTLP batch exporter
requires a tokio runtime (tonic/hyper-util gRPC transport). This caused
an immediate panic on CI when OTEL_EXPORTER_OTLP_ENDPOINT was set.
Fix: #[tokio::main(flavor = "current_thread")] on all 6 binaries.
Also fix pre-existing clippy warnings (shadow, let_underscore_must_use,
doc_markdown, cognitive_complexity, integer_division, unsafe_code).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both DQN and PPO eval paths used old 3-action index matching (0=Buy,
1=Sell, 2=Hold). Now uses FactoredAction.target_exposure() for
exposure-weighted returns and order-type-specific transaction costs.
PPO path had .to_int() which doesn't exist on FactoredAction.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- DQN PER: defer td_errors to_vec1() after loss.to_scalar() — piggyback on
existing pipeline flush instead of forcing premature GPU→CPU stall
- PPO trajectories: capacity-hint Vec allocations, extend_flat_states methods,
states_flat field on TrajectoryBatch for zero-copy GPU upload
- TGGN validate(): batch N per-sample losses on GPU → single to_scalar() sync
(was N GPU→CPU syncs)
- Liquid backward(): batch grad-norm per-param sqr().sum_all() on GPU → single
to_scalar() sync (was N GPU→CPU syncs per optimizer step)
- Liquid validate(): same N→1 GPU sync reduction as TGGN
- DQN trainer: restore EpochPrefetcher/DoubleBufferedLoader API (wrongly deleted)
- train_baseline_rl: wire DoubleBuffer GPU pre-upload — after CPU prefetch
completes, immediately upload next fold to GPU via DqnGpuData::upload() so
next fold starts with data already resident on GPU
2478 tests pass, 0 clippy warnings, 0 compile errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Emit set_eval_metrics (directional_accuracy, sharpe, profit_factor, return)
per fold for supervised model evaluation. Start metrics server on :9094.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Emit set_eval_metrics (win_rate, sharpe, profit_factor, return) per fold
for DQN/PPO evaluation. Start metrics server on :9094. Feeds training
cockpit Grafana dashboard eval panels.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace tracing_subscriber::fmt() with init_observability() which adds
JSON structured logging + optional OTLP export to Tempo. When
OTEL_EXPORTER_OTLP_ENDPOINT env var is unset, falls back to fmt-only.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Emit set_epoch, set_epoch_loss, set_validation_loss, set_iteration_seconds
after each DQN/PPO fold completes. Feeds training cockpit Grafana dashboard.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously train_baseline_rl.rs only passed commission (tx_cost_bps) to
DQN/PPO trainers, ignoring bid-ask spread slippage. Now computes per-fold
average spread via spread_cost_bps() (same as evaluate_baseline) and passes
total cost (commission + spread) to both trainers.
Removes #[allow(dead_code)] — function is now used by all 4 example binaries.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace raw DQN::new() + manual training loop in the walk-forward
training binary with DQNTrainer, which automatically activates:
- Mixed precision (BF16/F16 auto-detected from GPU)
- Dynamic batch sizing (AutoBatchSizer + HardwareBudget)
- Gradient accumulation
- Full Rainbow DQN (PER, dueling, C51, noisy nets, n-step)
- Regime-conditional Q-networks
- Portfolio tracking, Kelly sizing, entropy regularization
The walk-forward fold structure (data loading, feature extraction,
window generation, normalization) stays in the binary — only per-fold
training delegates to DQNTrainer::train_with_preloaded_data().
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Quality audit found 2 dead code paths in the GPU optimization commit:
1. Data caching: preload_data() was defined on all 10 hyperopt adapters
but never called. Now wired in both hyperopt binaries (RL + supervised)
before the trial loop. Each model preloads training data once into
Arc<Vec<...>>, eliminating per-trial disk I/O.
2. PPO mixed precision: config.mixed_precision was stored but never used
in forward passes. Added forward_mixed() to PolicyNetwork and
ValueNetwork (same BF16/FP16 pattern as DQN's NetworkLayers). Stored
on network structs and auto-applied via forward(). Wired in
PPO::with_device() for MLP networks.
Also fixes missing mixed_precision field in 2 test files and
trading_service PPOConfig literal.
5 files changed, +152/-20. 2418 tests pass, workspace compiles clean.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
