foxhunt/docs/plans/2026-03-01-ml-inference-production-cleanup-design.md

ML Inference Production Cleanup Design

Date: 2026-03-01 Branch: feature/training-prometheus-metrics Scope: Remove legacy ML code, refactor SharedMLStrategy to use real model inference, complete training metrics instrumentation

Problem Statement

The ML inference layer has accumulated legacy scaffolding that was useful during development but is now dead code in production:

1. SimpleDQNAdapter — a fake linear model with hardcoded weights, never used by any production path (only tests)
2. MLFeatureExtractor — a legacy 66-feature extractor superseded by ProductionFeatureExtractorAdapter (225 features)
3. TrainingMetricsPusher — a Pushgateway client with zero consumers
4. SharedMLStrategy hardcodes SimpleDQNAdapter in both constructors, making it impossible to inject real models
5. Hyperopt binaries emit only 4/18 Prometheus metrics (lifecycle signals only, zero training loop metrics)

Both the backtesting service and trading service already use SharedMLStrategy with ProductionFeatureExtractorAdapter, but predictions come from a hardcoded linear model instead of real trained checkpoints.

Design

Section 1: Dead Code Removal (~1,500 lines)

Delete from crates/common/src/ml_strategy.rs:

• SimpleDQNAdapter struct + all impls (lines 1310-1543)
• MLFeatureExtractor struct + all impls (lines 1-1293 approximately)
• All inline #[cfg(test)] tests for both (lines 2494-2671)

Delete test/bench files:

• crates/common/tests/volume_indicators_test.rs
• crates/common/tests/volume_indicators_integration_test.rs
• crates/common/tests/macd_tests.rs
• crates/common/benches/ml_strategy_bench.rs
• services/backtesting_service/tests/ml_strategy_backtest_test.rs

Delete:

• crates/ml/src/training/push_metrics.rs
• Remove pub mod push_metrics; from crates/ml/src/training.rs

Fix remaining test files:

• crates/common/tests/ml_strategy_integration_tests.rs — rewrite to test SharedMLStrategy with mock adapters
• services/trading_service/tests/ml_order_service_tests.rs — remove SimpleDQNAdapter usage

Fix re-exports:

• crates/common/src/lib.rs — remove MLFeatureExtractor + SimpleDQNAdapter re-exports
• Doc comments in crates/ml/src/features/config.rs referencing MLFeatureExtractor

Section 2: Refactor SharedMLStrategy

Keep: MLModelAdapter trait (correct abstraction).

Change constructor:

impl SharedMLStrategy {
    /// Create strategy with injected models and production feature extractor
    pub fn new_with_models(
        extractor: Box<dyn ProductionFeatureExtractor225>,
        models: Vec<Box<dyn MLModelAdapter>>,
        min_confidence_threshold: f64,
    ) -> Result<Self, CommonError>
}

Delete: new() constructor (legacy 66-feature path) and new_with_production_extractor() (hardcodes SimpleDQNAdapter).

Section 3: Real Model Adapter in ml/

New file: crates/ml/src/ensemble/model_adapter.rs

Create EnsembleModelAdapter that implements common::ml_strategy::MLModelAdapter:

• Wraps the model registry (get_global_registry())
• Calls real model inference via loaded checkpoints
• Returns graceful empty predictions if no models are loaded

Factory function:

pub fn build_production_strategy(
    min_confidence_threshold: f64,
) -> Result<SharedMLStrategy> {
    let extractor = Box::new(ProductionFeatureExtractorAdapter::new());
    let registry = get_global_registry();
    let models: Vec<Box<dyn MLModelAdapter>> = registry.available_models()
        .into_iter()
        .map(|id| Box::new(EnsembleModelAdapter::new(id, registry.clone())) as _)
        .collect();
    SharedMLStrategy::new_with_models(extractor, models, min_confidence_threshold)
}

Section 4: Update Callers

• services/backtesting_service/src/ml_strategy_engine.rs — use ml::ensemble::build_production_strategy()
• crates/backtesting/src/strategy_runner.rs — same
• services/trading_agent_service/src/service.rs — replace MLFeatureExtractor::new() with ProductionFeatureExtractorAdapter
• services/trading_agent_service/src/assets.rs — remove unused _feature_extractor parameter

Section 5: Complete Hyperopt Instrumentation

Add training_metrics:: calls inside optimization loops:

• hyperopt_baseline_supervised.rs — 8 model hyperopt functions (TFT, Mamba2, Liquid, TGGN, TLOB, KAN, xLSTM, Diffusion)
• hyperopt_baseline_rl.rs — DQN and PPO hyperopt functions

Fill missing metrics in training binaries:

• set_batches_per_second() in supervised trainer
• record_gradient_explosion() detection in all 4 binaries (check for inf/very large loss)
• record_feature_error() in data loading paths

Section 6: Metrics Server Hardening

In crates/common/src/metrics/server.rs:

• Set 5-second read timeout: stream.set_read_timeout(Some(Duration::from_secs(5)))
• Limit request line to 8KB: reader.take(8192).read_line(...)
• Fix Content-Type: text/plain; version=0.0.4; charset=utf-8

Verification

1. SQLX_OFFLINE=true cargo check --workspace — zero errors
2. SQLX_OFFLINE=true cargo clippy -p common -p ml --all-targets -- -D warnings — zero warnings
3. SQLX_OFFLINE=true cargo test -p common --lib — all pass
4. SQLX_OFFLINE=true cargo test -p ml --lib — all pass
5. All 4 training examples compile
6. curl localhost:9094/metrics shows all 18 foxhunt_training_* metrics

Risk Assessment

Change	Risk	Mitigation
Delete MLFeatureExtractor	LOW	Only test/trading_agent consumers, both legacy
Delete SimpleDQNAdapter	LOW	Zero production consumers
Refactor SharedMLStrategy constructor	MEDIUM	Update all callers (4 files)
EnsembleModelAdapter + model registry	MEDIUM	Graceful fallback if no checkpoints loaded
Hyperopt instrumentation	LOW	Additive changes only
Metrics server hardening	LOW	Additive safety checks