# AGENT WIRE-23: Master Feature Integration Roadmap **Date**: 2025-10-19 **Status**: ✅ COMPLETE - Synthesis of WIRE-01 through WIRE-22 **Priority**: ðŸ”ī **CRITICAL** - Blocks production deployment --- ## ðŸŽŊ Executive Summary **CRITICAL FINDING**: Wave D implementation is **99.4% complete at component level** but **0-30% integrated at system level**. All 24 regime features (indices 201-224) are implemented and tested, but the trading pipeline uses NONE of them. ### Integration Status by Feature Category | Category | Implementation | Integration | Gap Severity | |----------|---------------|-------------|--------------| | **Kelly Criterion** | ✅ 100% (3 implementations) | ❌ 0% - Not wired | ðŸ”ī CRITICAL | | **Adaptive Position Sizer** | ✅ 100% (1,643 lines) | ❌ 0% - Not wired | ðŸ”ī CRITICAL | | **Regime Detection** | ✅ 100% (8 modules) | ❌ 0% - Not extracted | ðŸ”ī CRITICAL | | **CUSUM Integration** | ✅ 100% (10 features) | ❌ 0% - Not used for decisions | ðŸ”ī CRITICAL | | **ADX Integration** | ✅ 100% (5 features) | ✅ 100% - Fully wired | ✅ READY | | **Transition Probabilities** | ✅ 100% (5 features) | ❌ 0% - Not in pipeline | ðŸ”ī CRITICAL | | **SharedMLStrategy** | ✅ 100% (2,395 lines) | ❌ 0% - Uses 30 features, not 225 | ðŸ”ī CRITICAL | | **Triple Barrier Labeling** | ✅ 100% (315 lines) | ❌ 0% - Not used in training | ðŸŸĄ HIGH | | **Fractional Differencing** | ✅ 100% (379 lines) | ❌ 0% - Stub returns zeros | ðŸŸĒ LOW | ### Overall System Integration: **23% COMPLETE** - ✅ **Implemented**: 100% (all components built and tested) - ❌ **Integrated**: 23% (only ADX + basic feature extraction working) - ðŸ”ī **Production Ready**: **NO** - Critical gaps block deployment --- ## 📋 Feature Integration Matrix ### Priority 0: CRITICAL (Must Fix Before Deployment) | Feature | Implementation Status | Integration Status | Blocker? | Effort | |---------|----------------------|-------------------|----------|--------| | **Kelly Criterion** | ✅ WIRE-01 | ❌ Not in `allocate_portfolio()` | YES | 3h | | **Adaptive Position Sizer** | ✅ WIRE-02 | ❌ Not in allocation flow | YES | 3h | | **Regime Detection** | ✅ WIRE-03 | ❌ Not in decision pipeline | YES | 6h | | **CUSUM → Regime Transitions** | ✅ WIRE-07 | ❌ Not triggering regime changes | YES | 8h | | **Transition Probabilities** | ✅ WIRE-09 | ❌ Not in feature pipeline | YES | 3h | | **SharedMLStrategy (225 features)** | ✅ WIRE-12 | ❌ Hardcoded to 30 features | YES | 12h | **Total P0 Effort**: 35 hours (4.4 days) ### Priority 1: HIGH (Should Fix for Full Wave D Value) | Feature | Implementation Status | Integration Status | Blocker? | Effort | |---------|----------------------|-------------------|----------|--------| | **Triple Barrier Labeling** | ✅ WIRE-05 | ❌ Not in ML training pipeline | NO | 6h | | **PPO Position Sizer** | ✅ WIRE-04 | ❌ Disabled (Kelly default) | NO | 8h | | **Meta-Labeling** | ⚠ïļ WIRE-05 | ❌ Stub implementation | NO | 8h | **Total P1 Effort**: 22 hours (2.75 days) ### Priority 2: NICE-TO-HAVE (Polish) | Feature | Implementation Status | Integration Status | Blocker? | Effort | |---------|----------------------|-------------------|----------|--------| | **Fractional Differencing** | ✅ WIRE-06 | ❌ Stub returns zeros | NO | 4h | | **TLI Commands** | ✅ Implemented | ✅ Operational | NO | 0h | | **Grafana Dashboards** | ⚠ïļ Partial | ❌ Need regime metrics | NO | 6h | **Total P2 Effort**: 10 hours (1.25 days) --- ## 🚀 3-Phase Integration Roadmap ### Phase 1: CRITICAL WIRING (35 hours / 4.4 days) - IMMEDIATE **Goal**: Wire P0 features to unblock deployment #### Task 1.1: SharedMLStrategy Refactor (12 hours) **Owner**: WIRE-12 findings **Priority**: P0 - Blocks everything **Changes Required**: 1. Replace hardcoded 30-feature extraction with `FeatureConfig` system 2. Add `kelly_sizer`, `regime_detector`, `adaptive_sizer` fields to struct 3. Register all 4 models (DQN, MAMBA-2, PPO, TFT) by default 4. Implement `generate_trade_signal()` with full orchestration 5. Update all service instantiations **Files**: - `/home/jgrusewski/Work/foxhunt/common/src/ml_strategy.rs` (2,395 lines - MODIFY) - `/home/jgrusewski/Work/foxhunt/services/trading_service/src/paper_trading_executor.rs` (MODIFY) - `/home/jgrusewski/Work/foxhunt/services/backtesting_service/src/ml_strategy_engine.rs` (MODIFY) **Validation**: ```rust #[test] fn test_shared_ml_uses_225_features() { let config = FeatureConfig::from_wave(WaveLevel::WaveD); let strategy = SharedMLStrategy::new(config, ...)?; let signal = strategy.generate_trade_signal(...).await?; assert_eq!(signal.features.len(), 213); // Wave D = 213 features assert!(signal.position_size > 0.0); assert!(!signal.regime.is_empty()); } ``` --- #### Task 1.2: Wire Kelly Criterion (3 hours) **Owner**: WIRE-01 findings **Priority**: P0 - Core value proposition **Changes Required**: 1. Implement `allocate_portfolio()` in Trading Agent Service 2. Add Kelly selection logic based on regime (Trending → Kelly, else MLOptimized) 3. Query `asset_statistics` table for win_rate, avg_win, avg_loss 4. Create `asset_statistics` table migration **Files**: - `/home/jgrusewski/Work/foxhunt/services/trading_agent_service/src/service.rs:285` (allocate_portfolio - IMPLEMENT) - `/home/jgrusewski/Work/foxhunt/services/trading_agent_service/src/allocation.rs` (USE existing AllocationMethod::KellyCriterion) **Integration Point**: ```rust async fn allocate_portfolio(request: AllocatePortfolioRequest) -> Result { let regime = self.get_current_regime(&req.strategy_id).await?; let allocation_method = match regime.regime_type { RegimeType::Trending => AllocationMethod::KellyCriterion { fraction: 0.25 }, RegimeType::Volatile => AllocationMethod::MeanVariance { lambda: 2.0 }, _ => AllocationMethod::MLOptimized, }; let allocator = PortfolioAllocator::new(allocation_method); let allocations = allocator.allocate(&assets, total_capital)?; // ... return allocations } ``` **Database Migration**: ```sql CREATE TABLE asset_statistics ( symbol TEXT PRIMARY KEY, win_rate DOUBLE PRECISION NOT NULL, avg_win DOUBLE PRECISION NOT NULL, avg_loss DOUBLE PRECISION NOT NULL, volatility DOUBLE PRECISION NOT NULL, last_updated TIMESTAMPTZ NOT NULL DEFAULT NOW() ); ``` --- #### Task 1.3: Wire Adaptive Position Sizer (3 hours) **Owner**: WIRE-02 findings **Priority**: P0 - Regime-adaptive sizing **Changes Required**: 1. Add `RegimeDetector` to Trading Agent Service struct 2. Create `regime.rs` module with database query layer 3. Apply regime multipliers (0.2x-1.5x) in `allocate_portfolio()` **Files**: - `/home/jgrusewski/Work/foxhunt/services/trading_agent_service/src/regime.rs` (NEW - 200 lines) - `/home/jgrusewski/Work/foxhunt/services/trading_agent_service/src/service.rs` (MODIFY) - `/home/jgrusewski/Work/foxhunt/services/trading_agent_service/src/allocation.rs` (MODIFY - add RegimeAdaptive method) **Integration Point**: ```rust // After base allocation: let regime_state = self.regime_detector.get_regime(symbol).await?; let adjusted = base_allocation * regime_state.position_multiplier; // Apply stop-loss multiplier let atr = calculate_atr(symbol, 14).await?; let stop_loss_distance = atr * regime_state.stop_loss_multiplier; ``` --- #### Task 1.4: Wire CUSUM to Regime Transitions (8 hours) **Owner**: WIRE-07 findings **Priority**: P0 - Core regime detection **Changes Required**: 1. Create `RegimeOrchestrator` to coordinate CUSUM + classifiers 2. Wire CUSUM breaks to trigger regime re-evaluation 3. Update `regime_transitions` table with `cusum_alert_triggered` **Files**: - `/home/jgrusewski/Work/foxhunt/ml/src/regime/orchestrator.rs` (NEW - 400 lines) - `/home/jgrusewski/Work/foxhunt/ml/src/regime/trending.rs` (MODIFY - accept CUSUM input) - `/home/jgrusewski/Work/foxhunt/ml/src/regime/ranging.rs` (MODIFY - accept CUSUM input) - `/home/jgrusewski/Work/foxhunt/ml/src/regime/volatile.rs` (MODIFY - accept CUSUM input) **Architecture**: ```rust pub struct RegimeOrchestrator { cusum_detector: CUSUMDetector, trending: TrendingClassifier, ranging: RangingClassifier, volatile: VolatileClassifier, current_regime: MarketRegime, } impl RegimeOrchestrator { pub fn classify(&mut self, bar: OHLCVBar) -> (MarketRegime, RegimeMetrics) { // 1. Check for structural breaks let break_signal = self.cusum_detector.update(bar.close); // 2. If break detected, force re-evaluation if break_signal.is_some() { let new_regime = self.resolve_regime(...); if new_regime != self.current_regime { self.record_transition(break_signal, new_regime); } } (self.current_regime, self.get_metrics()) } } ``` --- #### Task 1.5: Wire Transition Probabilities (3 hours) **Owner**: WIRE-09 findings **Priority**: P0 - Anticipatory position adjustments **Changes Required**: 1. Add `RegimeTransitionFeatures` to feature pipeline 2. Implement `extract_stage6_regime_features()` in pipeline.rs 3. Use previous bar's regime for current feature extraction **Files**: - `/home/jgrusewski/Work/foxhunt/ml/src/features/pipeline.rs` (MODIFY - add Stage 6) - `/home/jgrusewski/Work/foxhunt/ml/src/features/regime_transition.rs` (USE existing) **Integration Point**: ```rust // In FeatureExtractionPipeline: pub struct FeatureExtractionPipeline { transition_features: RegimeTransitionFeatures, current_regime: MarketRegime, } fn extract_stage6_regime_features(&mut self, regime: MarketRegime) -> Result<()> { self.transition_features.update(regime); let features = self.transition_features.compute_features(); // 5 features (216-220) self.feature_buffer.extend_from_slice(&features); Ok(()) } ``` --- #### Task 1.6: Wire Regime Detection to Decision Flow (6 hours) **Owner**: WIRE-03 findings **Priority**: P0 - Core Wave D value **Changes Required**: 1. Add regime detection BEFORE asset selection (filter universe) 2. Add regime detection BEFORE allocation (strategy selection) 3. Add regime state persistence to database **Files**: - `/home/jgrusewski/Work/foxhunt/services/trading_agent_service/src/service.rs` (MODIFY - all endpoints) **Integration Points**: **Point A: Before Asset Selection** ```rust let regime = self.get_regime_state("MARKET").await?; match regime.regime_type { RegimeType::Trending => { universe_criteria.min_momentum_score = 0.6; // Momentum assets }, RegimeType::Ranging => { universe_criteria.max_momentum_score = 0.4; // Mean-reversion }, RegimeType::Volatile => { universe_criteria.max_volatility = 0.15; // Stable assets }, } ``` **Point B: During Allocation (shown in Task 1.2)** **Point C: After Allocation (shown in Task 1.3)** --- ### Phase 2: HIGH-VALUE FEATURES (22 hours / 2.75 days) - SHORT-TERM **Goal**: Complete Wave D value proposition #### Task 2.1: Wire Triple Barrier Labeling (6 hours) **Owner**: WIRE-05 findings **Priority**: P1 - ML training quality **Changes Required**: 1. Modify `data/src/training_pipeline.rs` to use `TripleBarrierEngine` 2. Update training examples to use classification labels (not regression) 3. Add sample weighting based on `quality_score` **Files**: - `/home/jgrusewski/Work/foxhunt/data/src/training_pipeline.rs` (MODIFY) - `/home/jgrusewski/Work/foxhunt/ml/examples/train_mamba2_dbn.rs` (MODIFY) - `/home/jgrusewski/Work/foxhunt/ml/examples/train_dqn.rs` (MODIFY) - `/home/jgrusewski/Work/foxhunt/ml/examples/train_ppo.rs` (MODIFY) - `/home/jgrusewski/Work/foxhunt/ml/examples/train_tft_dbn.rs` (MODIFY) **Expected Impact**: +10-15% win rate, -40-60% label noise --- #### Task 2.2: Enable PPO Position Sizer (8 hours) **Owner**: WIRE-04 findings **Priority**: P1 - RL-based sizing **Changes Required**: 1. Train PPO model with real market data 2. Replace stub inference with real model 3. Add config option to enable PPO (default: Kelly) **Files**: - `/home/jgrusewski/Work/foxhunt/adaptive-strategy/src/risk/ppo_position_sizer.rs` (MODIFY - remove stubs) - `/home/jgrusewski/Work/foxhunt/adaptive-strategy/src/config.rs` (MODIFY - add PPO option) **Note**: Lower priority than Kelly - can deploy without this --- #### Task 2.3: Complete Meta-Labeling (8 hours) **Owner**: WIRE-05 findings **Priority**: P1 - Bet sizing filter **Changes Required**: 1. Implement production `apply_meta_labeling()` (remove stub) 2. Train secondary betting model 3. Integrate into Trading Agent Service **Files**: - `/home/jgrusewski/Work/foxhunt/ml/src/labeling/meta_labeling_engine.rs` (MODIFY) - `/home/jgrusewski/Work/foxhunt/ml/src/labeling/meta_labeling/secondary_model.rs` (USE) **Expected Impact**: +15-25% risk-adjusted returns --- ### Phase 3: POLISH (10 hours / 1.25 days) - MEDIUM-TERM **Goal**: Complete feature coverage #### Task 3.1: Enable Fractional Differencing (4 hours) **Owner**: WIRE-06 findings **Priority**: P2 - Signal quality improvement **Changes Required**: 1. Replace stub in `dbn_sequence_loader.rs` with real implementation 2. Add `StreamingDifferentiator` usage **Files**: - `/home/jgrusewski/Work/foxhunt/ml/src/data_loaders/dbn_sequence_loader.rs:1176-1180` (MODIFY) **Expected Impact**: +5-10% Sharpe (stationarity improvement) --- #### Task 3.2: Add Regime Metrics to Grafana (6 hours) **Owner**: Monitoring requirements **Priority**: P2 - Operational visibility **Changes Required**: 1. Add Prometheus metrics for regime transitions 2. Create Grafana dashboard for regime metrics 3. Add alerts for flip-flopping (>50/hour) **Files**: - `/home/jgrusewski/Work/foxhunt/services/trading_agent_service/src/metrics.rs` (MODIFY) - `grafana/dashboards/regime_detection.json` (NEW) --- ## 📊 Integration Impact Analysis ### Expected Performance Gains (After Full Integration) | Metric | Current (Wave C) | Wave D (Fully Integrated) | Improvement | |--------|------------------|---------------------------|-------------| | **Sharpe Ratio** | 1.2 (baseline) | 1.8-2.2 | **+50-83%** | | **Win Rate** | 52% | 58-62% | **+12-19%** | | **Max Drawdown** | -25% | -15-18% | **-28-40%** | | **Position Sizing** | Static (1.0x) | Adaptive (0.2x-1.5x) | **Dynamic** | | **Risk-Adjusted Return** | Baseline | +25-50% | **Target** | ### Expected Latency Budget (After Integration) | Component | Current | Target | Status | |-----------|---------|--------|--------| | Feature Extraction (225 features) | 30 features (~10Ξs) | 225 features (<50Ξs) | âģ PENDING | | Regime Detection | N/A | <5Ξs | âģ PENDING | | Kelly Sizing | N/A | <100Ξs | âģ PENDING | | ML Ensemble (4 models) | DQN only (~200Ξs) | All models (~4ms) | âģ PENDING | | **Total E2E Latency** | ~210Ξs | **<5ms** | âģ PENDING | **Target Met**: Yes (5ms << 3s budget) --- ## 🛠ïļ Deployment Strategy ### Pre-Deployment Checklist #### P0 Tasks (MUST COMPLETE) - [ ] Task 1.1: SharedMLStrategy uses 225 features (**12h**) - [ ] Task 1.2: Kelly Criterion wired to allocation (**3h**) - [ ] Task 1.3: Adaptive Position Sizer wired (**3h**) - [ ] Task 1.4: CUSUM triggers regime transitions (**8h**) - [ ] Task 1.5: Transition probabilities in pipeline (**3h**) - [ ] Task 1.6: Regime detection in decision flow (**6h**) - [ ] E2E integration test: Market data → Orders (**6h**) - [ ] Performance validation: <5ms latency (**2h**) **Total P0 Effort**: 43 hours (5.4 days) #### P1 Tasks (SHOULD COMPLETE) - [ ] Task 2.1: Triple Barrier labeling in training (**6h**) - [ ] Task 2.2: PPO Position Sizer enabled (**8h** - OPTIONAL) - [ ] Task 2.3: Meta-labeling completed (**8h**) **Total P1 Effort**: 22 hours (2.75 days) #### P2 Tasks (CAN DEFER) - [ ] Task 3.1: Fractional differencing enabled (**4h**) - [ ] Task 3.2: Grafana dashboards (**6h**) **Total P2 Effort**: 10 hours (1.25 days) --- ### Rollback Plan #### Level 1: Feature Flag (IMMEDIATE) ```rust const ENABLE_WAVE_D_FEATURES: bool = false; // Set to true after validation if ENABLE_WAVE_D_FEATURES { // Use 225 features, regime detection, Kelly, etc. } else { // Fall back to Wave C (201 features, static allocation) } ``` #### Level 2: Database Rollback (5 minutes) ```sql -- Disable regime tables (keep data) REVOKE SELECT ON regime_states FROM foxhunt; REVOKE SELECT ON adaptive_strategy_metrics FROM foxhunt; ``` #### Level 3: Code Rollback (10 minutes) ```bash git revert cargo build --release --workspace systemctl restart trading_agent_service systemctl restart trading_service ``` --- ## 📅 Timeline Summary ### Option A: CRITICAL ONLY (P0) - **Effort**: 43 hours (5.4 days) - **Deliverable**: Minimum viable Wave D deployment - **Risk**: Medium - skips triple barrier, meta-labeling ### Option B: FULL VALUE (P0 + P1) - **Effort**: 65 hours (8.1 days) - **Deliverable**: Complete Wave D value proposition - **Risk**: Low - includes all high-value features ### Option C: COMPLETE (P0 + P1 + P2) - **Effort**: 75 hours (9.4 days) - **Deliverable**: Fully polished Wave D deployment - **Risk**: Very Low - includes all features + monitoring **RECOMMENDED**: **Option B** (P0 + P1) - 8.1 days for full Wave D value --- ## ðŸŽŊ Success Criteria ### Definition of Done #### System-Level Integration 1. ✅ SharedMLStrategy uses `FeatureConfig` system (NOT hardcoded 30 features) 2. ✅ All 4 ML models (DQN, MAMBA-2, PPO, TFT) registered by default 3. ✅ `generate_trade_signal()` returns `TradeRecommendation` with: - 213 features (Wave D) - Position size (Kelly-sized) - Regime classification - Risk multipliers #### Feature Integration 4. ✅ Kelly Criterion active in `allocate_portfolio()` (Trending regime) 5. ✅ Adaptive Position Sizer applies regime multipliers (0.2x-1.5x) 6. ✅ Regime Detection runs BEFORE asset selection and allocation 7. ✅ CUSUM breaks trigger regime transitions in database 8. ✅ Transition probabilities (features 216-220) in feature pipeline #### Validation 9. ✅ E2E test: Market data → 225 features → Regime → Kelly → Orders 10. ✅ Performance test: <5ms E2E latency (P99) 11. ✅ Backtest: Wave D outperforms Wave C (+25-50% Sharpe) 12. ✅ Paper trading: 2 weeks validation before real capital --- ## 📚 Reference Documentation ### Agent Reports Analyzed - **WIRE-01**: Kelly Criterion integration (❌ 0% wired) - **WIRE-02**: Adaptive Position Sizer integration (❌ 0% wired) - **WIRE-03**: Regime Detection integration (❌ 0% wired) - **WIRE-04**: PPO Position Sizer status (⚠ïļ Disabled) - **WIRE-05**: Triple Barrier labeling status (❌ Not in training) - **WIRE-06**: Fractional Differencing status (⚠ïļ Stub returns zeros) - **WIRE-07**: CUSUM integration (❌ Not used for decisions) - **WIRE-08**: ADX integration (✅ 100% operational - ONLY success) - **WIRE-09**: Transition Probabilities (❌ Not in pipeline) - **WIRE-11**: Trading Agent decision flow (❌ Placeholders) - **WIRE-12**: SharedMLStrategy completeness (❌ 0% integration) ### Key Files Referenced - `/home/jgrusewski/Work/foxhunt/common/src/ml_strategy.rs` (2,395 lines - CRITICAL) - `/home/jgrusewski/Work/foxhunt/services/trading_agent_service/src/service.rs` (675 lines - CRITICAL) - `/home/jgrusewski/Work/foxhunt/ml/src/features/pipeline.rs` (CRITICAL - add Stage 6) - `/home/jgrusewski/Work/foxhunt/ml/src/regime/orchestrator.rs` (NEW - 400 lines) ### Database Tables - `regime_states` (✅ Created, ❌ Empty) - `regime_transitions` (✅ Created, ❌ Empty) - `adaptive_strategy_metrics` (✅ Created, ❌ Empty) - `asset_statistics` (❌ MISSING - required for Kelly) --- ## ðŸšĻ Critical Warnings ### Deployment Without Integration = FAILURE **Risk**: Deploying "Wave D" without integration will: 1. ✅ Train ML models on 225 features 2. ❌ **CRASH** when live trading provides only 30 features 3. ❌ No Kelly sizing → suboptimal position sizes 4. ❌ No regime detection → no adaptive strategies 5. ❌ No CUSUM → delayed regime transitions ($2K-3K loss/contract) 6. ❌ Wave D value proposition **COMPLETELY UNREALIZED** **BLOCKER**: This gap renders Wave D **UNDEPLOYABLE** despite "99.4% test pass rate". --- ## ✅ Recommended Next Steps ### Immediate Actions (Today) 1. **APPROVE** integration roadmap (this document) 2. **ASSIGN** agents to P0 tasks (WIRE-24 through WIRE-29) 3. **CREATE** feature flag for Wave D integration (Task 1.1) 4. **SCHEDULE** 2-week integration sprint ### Week 1: Critical Wiring (P0 Tasks 1.1-1.6) - Days 1-3: SharedMLStrategy refactor (Task 1.1) - Days 4-5: Kelly + Adaptive Sizer + Regime wiring (Tasks 1.2-1.6) ### Week 2: Validation + High-Value Features (P0 + P1) - Days 1-2: E2E testing + performance validation - Days 3-5: Triple Barrier + Meta-labeling (Tasks 2.1, 2.3) ### Production Deployment (Week 3) - Days 1-2: Final smoke tests + dry-run deployment - Days 3-5: Monitoring setup + production rollout - **MILESTONE**: Wave D production deployment COMPLETE --- ## 🎉 Conclusion **Master Integration Roadmap**: ✅ COMPLETE **Status**: Wave D is **99.4% implemented** but **23% integrated**. All 24 regime features (indices 201-224) exist, are tested, and perform 432x faster than targets. However, **ZERO** of these features are used in production trading decisions. **Recommended Path**: Execute **Option B** (P0 + P1) for **8.1 days** to achieve full Wave D value proposition. **Expected Outcome**: +25-50% Sharpe improvement, +10-15% win rate, -20-30% drawdown. **Next Agent**: WIRE-24 (SharedMLStrategy refactor - 12 hours) --- **AGENT WIRE-23: MISSION COMPLETE** *"The components are ready. The wiring begins now."*