623ebfcf71
Previous workflow train-4qwtc hit memory-pressure thrash (~56Gi
cgroup.current sitting at the 56Gi pod limit, kernel reclaim
hammering page cache) right after OFI completed on the 9-quarter
17.8M-bar dataset. Two refactors reduce peak by ~12GB:
(1) walk_forward.rs: new `normalize_batch_in_place(&mut features)`
that rewrites the slice in place. The previous `normalize_batch`
`.collect()`s a new Vec — at this dataset size that's a
transient ~6GB peak while both pre- and post-normalised arrays
are alive.
(2) precompute_features.rs:
- call `normalize_batch_in_place` instead of the rebinding form.
- explicit `drop(feature_vectors)` after copying the slice into
`features` — `feature_vectors` would otherwise stay alive
until end-of-main shadowing the ~6GB allocation through
every downstream step.
Combined with the prior `t.into_iter()` refactor (a27cb40a9), the
peak transient drops from ~56GB to ~44GB — well under the 56Gi pod
limit on the existing ci-compile-cpu pool (POP2-HC-32C-64G).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ml
10-model ML ensemble for the Foxhunt HFT system, built on Candle v0.9.1.
Models
- DQN (Rainbow) — deep Q-network with prioritized replay, dueling heads, noisy nets
- PPO — proximal policy optimization with GAE, LSTM policies, clip-higher
- TFT — temporal fusion transformer for multi-horizon forecasting
- Mamba2 — state space model for sequence prediction
- Liquid Networks — biologically inspired networks for non-stationary data
- TLOB — transformer-based limit order book analysis
- KAN — Kolmogorov-Arnold networks
- xLSTM — extended LSTM architecture
- TGGN — temporal graph neural network
- Diffusion — diffusion-based generative model
Key Modules
ensemble— model ensemble coordination and confidence aggregationhyperopt— PSO-based hyperparameter optimization with per-model adapterstrainers— unified training loops (DQN, PPO, supervised)inference—InferenceAdaptertrait for predictioncheckpoint— model checkpointing and restorationevaluation— walk-forward evaluation pipeline
Usage
use ml::dqn::DQN;
use ml::ppo::PpoTrainer;