48aac3c601
Three critical stability fixes for PPO training: 1. Gradient clipping: Added clip_grads() that scales gradients by max_norm/norm when L2 norm exceeds max_grad_norm (0.5). Previously the code only logged a warning — gradient norms of 27-171x above the threshold were applied raw to weights, causing divergence. Fixed in all 8 locations across PpoTrainer (6) and ContinuousPPO (2). 2. Return normalization: Value loss now normalizes returns to N(0,1) before computing MSE. Raw cumulative returns (±1000s) caused enormous value gradients that destabilized the critic network. 3. Hyperopt search space: Tightened value LR upper bound from 1e-3 to 1e-4 (1e-3 is documented unstable), policy LR from 1e-3 to 3e-4. Also fixed LSTM path where optimizer.step() ran BEFORE gradient norm check — now clip → step (not step → warn). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ml
Machine learning models for Foxhunt.
Models
- DQN (Rainbow) -- Deep Q-Network with prioritized experience replay, dueling heads, noisy nets, double Q-learning
- PPO -- Proximal Policy Optimization with GAE, LSTM policies, clip-higher option
- TFT -- Temporal Fusion Transformer for multi-horizon time series forecasting
- Mamba2 -- State space model for efficient sequence prediction
- Liquid Networks -- Biologically inspired neural networks for non-stationary data
- TLOB -- Transformer-based Limit Order Book analysis
- Flash Attention -- Optimized attention implementation
Training
Two paths per model:
- Standalone trainer -- direct training loop (e.g.,
DQN::train,PpoTrainer) - UnifiedTrainable adapter -- wraps models for the hyperopt pipeline (e.g.,
DQNTrainableAdapter,UnifiedTrainablePPO)
Inference
InferenceAdapterBridge connects models to the ensemble coordinator in adaptive-strategy. Each model exposes an InferenceAdapter trait for prediction.
Backend
- Candle v0.9.1 --
VarMap,AdamW,loss.backward(),GradStore,opt.step(&grads) - CUDA required for training -- tested on RTX 3050 Ti 4GB, max batch size 230
- CPU inference supported
Hyperopt
ArgminOptimizer (Particle Swarm Optimization) with per-model adapters:
DQN, PPO, ContinuousPPO, TFT, Mamba2. Uses ParameterSpace trait for continuous parameter mapping.
ModelType Enum
15 variants: CompactDQN, DistilledMicroNet, DQN, RainbowDQN, MAMBA, TFT, TGGN, LNN, TLOB, PPO, Transformer, Mamba, LiquidNet, TGNN, Ensemble.
Key Modules
dqn, ppo, tft, mamba, liquid, tlob, flash_attention, ensemble, evaluation, inference, trainers, hyperopt, checkpoint, preprocessing, data_loaders, features, model_factory, training_pipeline, regime_detection, stress_testing, validation, bridge, common, metrics.
Testing
SQLX_OFFLINE=true cargo test -p ml --lib # ~2009 tests