74c7a80114
Closes the patience-based early-stopping bug that ran xmd6b 30 epochs
past peak val performance (epoch 2: val_Sharpe=90, total_pnl=0.44 →
epoch 30: val_Sharpe=26, total_pnl=0.16) — a 70% loss of alpha to
training-induced overfitting.
Two intertwined bugs, fixed atomically:
T1.1a (wrong-source) at training_loop.rs:7234:
- was: self.early_stopping.should_stop(-log_output.epoch_sharpe, epoch)
reading the TRAINING ROLLOUT Sharpe (Thompson-noisy, in-sample,
oscillates even when the model is frozen)
- now: self.early_stopping.should_stop(log_output.val_loss, min_delta, epoch)
reading the deterministic-backtest val_loss
- The comment 5733 lines earlier (line 1499) explicitly says "Use
val_Sharpe (deterministic backtest), NOT epoch_sharpe" — patience
path was the inconsistency, backtracking already honored it.
T1.1b (hardcoded threshold) in early_stopping.rs:
- was: EarlyStopping::new(patience, min_delta) with min_delta=0.001
constructor-set, struct field, structurally meaningless
against the val_loss noise floor (typical val_sharpe deltas
are O(1-10), so 0.001 essentially never gates)
- now: EarlyStopping::new(patience), should_stop(val_loss, min_delta,
epoch) with min_delta computed per-call from
ISV[VAL_SHARPE_VAR_EMA_INDEX=351] as
sqrt(var_ema).max(0.5)
- Floor 0.5 covers cold-start before var_ema bootstraps from sentinel
per pearl_blend_formulas_must_have_permanent_floor.
- Per feedback_isv_for_adaptive_bounds + feedback_adaptive_not_tuned.
T2.3 (test signature update) absorbed:
- 6 existing unit tests migrated to new should_stop signature.
- 1 NEW test (test_min_delta_can_change_per_call) verifying per-call
threshold change works correctly.
- EarlyStopping::min_delta struct field deleted.
- Atomic per feedback_no_partial_refactor.
Affected files:
- crates/ml/src/trainers/dqn/early_stopping.rs (struct + tests)
- crates/ml/src/trainers/dqn/trainer/constructor.rs (new() arg)
- crates/ml/src/trainers/dqn/trainer/training_loop.rs (call site)
Verification:
- cargo check -p ml --tests: passes
- cargo test -p ml --lib early_stopping: 8/8 pass
Behavioral expectation post-fix: xmd6b-shape runs (val_Sharpe rising
31→90 epochs 0-2, declining 90→26 epochs 3-30) will trigger early-stop
near the peak. With patience=5 and var_ema bootstrapping by epoch 2-3,
the controller detects "no improvement of ≥ 1σ for 5 consecutive
epochs" by ~epoch 7-8 and stops, saving ~22 epochs of overfitting.
Plan reference: docs/plans/2026-05-10-sp21-train-eval-coherence-isv-defrost.md
Tier 1 status: T1.1a ✓, T1.1b ✓, T2.3 ✓ (this commit). T1.2 + T1.4 next.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Foxhunt
Production HFT trading system in Rust.
Architecture
The workspace contains 32 crates organized as follows:
Core Libraries (16)
| Crate | Purpose |
|---|---|
trading_engine |
Order processing, FIX 4.4, IB TWS, SIMD, RDTSC timing |
risk |
VaR, Kelly, circuit breakers, kill switches, compliance |
risk-data |
Risk data types and shared structures |
trading-data |
Trading data types |
ml |
DQN Rainbow, PPO, TFT, Mamba2, ensemble inference |
ml-data |
ML data types and feature definitions |
data |
Market data ingestion and storage |
backtesting |
Replay engine, strategy tester |
adaptive-strategy |
Ensemble execution, microstructure analysis |
common |
Shared types, resilience, error handling |
storage |
S3 and local model storage |
model_loader |
Model serialization and loading |
market-data |
Market data feed handlers |
database |
PostgreSQL access layer (SQLx) |
config |
Configuration management |
tli |
CLI commands and tooling |
Services (8)
| Service | Purpose |
|---|---|
backtesting_service |
gRPC backtesting service |
broker_gateway_service |
FIX routing, broker connectivity |
trading_service |
Core trading operations |
ml_training_service |
Model training orchestration |
data_acquisition_service |
Market data acquisition |
trading_agent_service |
Autonomous trading agents |
api_gateway |
gRPC API gateway with auth |
web-gateway |
Axum REST + WebSocket gateway |
Frontend
web-dashboard/ -- React 19 + TypeScript + Vite + TradingView charts.
Building
# Check compilation (no PostgreSQL required)
SQLX_OFFLINE=true cargo check --workspace
# Run tests for a specific crate
SQLX_OFFLINE=true cargo test -p <crate> --lib
# Clippy
SQLX_OFFLINE=true cargo clippy --workspace
ML Models
Four production model architectures on Candle v0.9.1 with CUDA:
- DQN Rainbow -- Deep Q-Network with prioritized replay, dueling heads, noisy nets
- PPO -- Proximal Policy Optimization with GAE and LSTM policies
- TFT -- Temporal Fusion Transformer for multi-horizon forecasting
- Mamba2 -- State space model for sequence prediction
Each model has a standalone trainer and a UnifiedTrainable adapter for the hyperopt pipeline.
Infrastructure
- Git: Gitea at
git.fxhnt.ai(Tailscale-only), Scaleway DEV1-S - Observability: OpenTelemetry OTLP (env
OTEL_EXPORTER_OTLP_ENDPOINT) - Database: PostgreSQL with SQLx offline mode for CI
License
Proprietary. All rights reserved.