# ML Inference Production Cleanup Design

**Date:** 2026-03-01
**Branch:** `feature/training-prometheus-metrics`
**Scope:** Remove legacy ML code, refactor SharedMLStrategy to use real model inference, complete training metrics instrumentation

## Problem Statement

The ML inference layer has accumulated legacy scaffolding that was useful during development but is now dead code in production:

1. **SimpleDQNAdapter** — a fake linear model with hardcoded weights, never used by any production path (only tests)
2. **MLFeatureExtractor** — a legacy 66-feature extractor superseded by ProductionFeatureExtractorAdapter (225 features)
3. **TrainingMetricsPusher** — a Pushgateway client with zero consumers
4. **SharedMLStrategy** hardcodes `SimpleDQNAdapter` in both constructors, making it impossible to inject real models
5. **Hyperopt binaries** emit only 4/18 Prometheus metrics (lifecycle signals only, zero training loop metrics)

Both the backtesting service and trading service already use `SharedMLStrategy` with `ProductionFeatureExtractorAdapter`, but predictions come from a hardcoded linear model instead of real trained checkpoints.

## Design

### Section 1: Dead Code Removal (~1,500 lines)

**Delete from `crates/common/src/ml_strategy.rs`:**
- `SimpleDQNAdapter` struct + all impls (lines 1310-1543)
- `MLFeatureExtractor` struct + all impls (lines 1-1293 approximately)
- All inline `#[cfg(test)]` tests for both (lines 2494-2671)

**Delete test/bench files:**
- `crates/common/tests/volume_indicators_test.rs`
- `crates/common/tests/volume_indicators_integration_test.rs`
- `crates/common/tests/macd_tests.rs`
- `crates/common/benches/ml_strategy_bench.rs`
- `services/backtesting_service/tests/ml_strategy_backtest_test.rs`

**Delete:**
- `crates/ml/src/training/push_metrics.rs`
- Remove `pub mod push_metrics;` from `crates/ml/src/training.rs`

**Fix remaining test files:**
- `crates/common/tests/ml_strategy_integration_tests.rs` — rewrite to test SharedMLStrategy with mock adapters
- `services/trading_service/tests/ml_order_service_tests.rs` — remove SimpleDQNAdapter usage

**Fix re-exports:**
- `crates/common/src/lib.rs` — remove MLFeatureExtractor + SimpleDQNAdapter re-exports
- Doc comments in `crates/ml/src/features/config.rs` referencing MLFeatureExtractor

### Section 2: Refactor SharedMLStrategy

**Keep:** `MLModelAdapter` trait (correct abstraction).

**Change constructor:**
```rust
impl SharedMLStrategy {
    /// Create strategy with injected models and production feature extractor
    pub fn new_with_models(
        extractor: Box<dyn ProductionFeatureExtractor225>,
        models: Vec<Box<dyn MLModelAdapter>>,
        min_confidence_threshold: f64,
    ) -> Result<Self, CommonError>
}
```

**Delete:** `new()` constructor (legacy 66-feature path) and `new_with_production_extractor()` (hardcodes SimpleDQNAdapter).

### Section 3: Real Model Adapter in ml/

**New file:** `crates/ml/src/ensemble/model_adapter.rs`

Create `EnsembleModelAdapter` that implements `common::ml_strategy::MLModelAdapter`:
- Wraps the model registry (`get_global_registry()`)
- Calls real model inference via loaded checkpoints
- Returns graceful empty predictions if no models are loaded

**Factory function:**
```rust
pub fn build_production_strategy(
    min_confidence_threshold: f64,
) -> Result<SharedMLStrategy> {
    let extractor = Box::new(ProductionFeatureExtractorAdapter::new());
    let registry = get_global_registry();
    let models: Vec<Box<dyn MLModelAdapter>> = registry.available_models()
        .into_iter()
        .map(|id| Box::new(EnsembleModelAdapter::new(id, registry.clone())) as _)
        .collect();
    SharedMLStrategy::new_with_models(extractor, models, min_confidence_threshold)
}
```

### Section 4: Update Callers

- `services/backtesting_service/src/ml_strategy_engine.rs` — use `ml::ensemble::build_production_strategy()`
- `crates/backtesting/src/strategy_runner.rs` — same
- `services/trading_agent_service/src/service.rs` — replace `MLFeatureExtractor::new()` with `ProductionFeatureExtractorAdapter`
- `services/trading_agent_service/src/assets.rs` — remove unused `_feature_extractor` parameter

### Section 5: Complete Hyperopt Instrumentation

Add `training_metrics::` calls inside optimization loops:
- `hyperopt_baseline_supervised.rs` — 8 model hyperopt functions (TFT, Mamba2, Liquid, TGGN, TLOB, KAN, xLSTM, Diffusion)
- `hyperopt_baseline_rl.rs` — DQN and PPO hyperopt functions

Fill missing metrics in training binaries:
- `set_batches_per_second()` in supervised trainer
- `record_gradient_explosion()` detection in all 4 binaries (check for inf/very large loss)
- `record_feature_error()` in data loading paths

### Section 6: Metrics Server Hardening

In `crates/common/src/metrics/server.rs`:
- Set 5-second read timeout: `stream.set_read_timeout(Some(Duration::from_secs(5)))`
- Limit request line to 8KB: `reader.take(8192).read_line(...)`
- Fix Content-Type: `text/plain; version=0.0.4; charset=utf-8`

## Verification

1. `SQLX_OFFLINE=true cargo check --workspace` — zero errors
2. `SQLX_OFFLINE=true cargo clippy -p common -p ml --all-targets -- -D warnings` — zero warnings
3. `SQLX_OFFLINE=true cargo test -p common --lib` — all pass
4. `SQLX_OFFLINE=true cargo test -p ml --lib` — all pass
5. All 4 training examples compile
6. `curl localhost:9094/metrics` shows all 18 `foxhunt_training_*` metrics

## Risk Assessment

| Change | Risk | Mitigation |
|--------|------|------------|
| Delete MLFeatureExtractor | LOW | Only test/trading_agent consumers, both legacy |
| Delete SimpleDQNAdapter | LOW | Zero production consumers |
| Refactor SharedMLStrategy constructor | MEDIUM | Update all callers (4 files) |
| EnsembleModelAdapter + model registry | MEDIUM | Graceful fallback if no checkpoints loaded |
| Hyperopt instrumentation | LOW | Additive changes only |
| Metrics server hardening | LOW | Additive safety checks |
