# Agent WIRE-02: Wave D Adaptive Position Sizer Integration Analysis **Date**: 2025-10-19 **Agent**: WIRE-02 **Mission**: Investigate why AdaptivePositionSizer is implemented but not integrated into trading flow **Status**: ✅ **COMPLETE** - Gap identified, integration plan ready --- ## Executive Summary **CRITICAL FINDING**: Wave D's regime-adaptive position sizing (Features 221-224) is **IMPLEMENTED but NOT INTEGRATED** into the trading decision flow. **Impact**: The 225-feature ML models will train with regime-adaptive features, but **production trading will not apply regime-based position sizing** unless we complete the missing integration. **Root Cause**: 1. `RegimeAdaptiveFeatures` exists in `ml/src/features/regime_adaptive.rs` (644 lines, 12/12 tests passing) 2. Database schema exists (`regime_states`, `regime_transitions`, `adaptive_strategy_metrics`) 3. gRPC endpoints exist (`GetRegimeState`, `GetRegimeTransitions`) 4. **BUT**: `services/trading_agent_service/src/allocation.rs` has NO imports/usage of regime detection **Gap**: Position sizing multipliers (0.2x-1.5x) and stop-loss multipliers (1.5x-4.0x ATR) are computed as ML features but **never applied to actual order sizing**. --- ## Investigation Results ### 1. Implementation Status ✅ #### ✅ Feature Extraction (`ml/src/features/regime_adaptive.rs`) **Location**: `/home/jgrusewski/Work/foxhunt/ml/src/features/regime_adaptive.rs` (644 lines) **Struct**: `RegimeAdaptiveFeatures` ```rust pub struct RegimeAdaptiveFeatures { current_regime: MarketRegime, returns_window: VecDeque, window_size: usize, current_position_size: f64, max_position_size: f64, atr_period: usize, } ``` **Key Methods**: ```rust pub fn update( &mut self, regime: MarketRegime, return_value: f64, current_position: f64, bars: &[OHLCVBar], ) -> [f64; 4] { // Returns: // [0]: Position multiplier (0.2x-1.5x based on regime) // [1]: Stop-loss multiplier (1.5x-4.0x ATR based on regime) // [2]: Regime-adjusted Sharpe ratio // [3]: Risk budget utilization } ``` **Position Multipliers** (Feature 221): ```rust const POSITION_MULTIPLIERS: [(MarketRegime, f64); 7] = [ (MarketRegime::Normal, 1.0), // Baseline (MarketRegime::Trending, 1.5), // Increase size in trends (MarketRegime::Sideways, 0.8), // Reduce in choppy markets (MarketRegime::Bull, 1.2), // Moderate increase (MarketRegime::Bear, 0.7), // Reduce in downtrends (MarketRegime::HighVolatility, 0.5), // Reduce risk (MarketRegime::Crisis, 0.2), // Extreme risk reduction ]; ``` **Stop-Loss Multipliers** (Feature 222): ```rust const STOPLOSS_MULTIPLIERS: [(MarketRegime, f64); 7] = [ (MarketRegime::Normal, 2.0), // 2x ATR (MarketRegime::Trending, 2.5), // Wider for trends (MarketRegime::Sideways, 1.5), // Tighter in ranges (MarketRegime::Bull, 2.0), // Standard (MarketRegime::Bear, 2.5), // Wider in bear (MarketRegime::HighVolatility, 3.0), // Wide for volatility (MarketRegime::Crisis, 4.0), // Very wide to avoid panic exits ]; ``` **Test Status**: 12/12 tests passing (100%) --- #### ✅ Database Schema (`migrations/045_wave_d_regime_tracking.sql`) **Tables Created**: 1. **`regime_states`** (14 columns): - `symbol`, `event_timestamp`, `regime`, `confidence` - CUSUM metrics: `cusum_s_plus`, `cusum_s_minus`, `cusum_alert_count` - ADX metrics: `adx`, `plus_di`, `minus_di` - Stability: `stability`, `entropy` - Indexes: `idx_regime_states_symbol_timestamp`, `idx_regime_states_regime`, `idx_regime_states_confidence` 2. **`regime_transitions`** (10 columns): - `symbol`, `event_timestamp`, `from_regime`, `to_regime` - `duration_bars`, `transition_probability`, `adx_at_transition` - Indexes: `idx_regime_transitions_symbol_timestamp`, `idx_regime_transitions_from_to` 3. **`adaptive_strategy_metrics`** (12 columns): - `symbol`, `event_timestamp`, `regime` - **`position_multiplier`** (0.0-2.0) - **`stop_loss_multiplier`** (1.0-5.0) - `regime_sharpe`, `risk_budget_utilization` - Performance: `total_trades`, `winning_trades`, `total_pnl` **Functions**: - `get_latest_regime(p_symbol TEXT)`: Fetch current regime - `get_regime_transition_matrix(p_symbol TEXT, p_window_hours INTEGER)`: Transition probabilities - `get_regime_performance(p_symbol TEXT, p_window_hours INTEGER)`: Regime-specific performance **Migration Status**: Applied (verified in docs) --- #### ✅ gRPC API Endpoints **Proto Definitions** (confirmed in 100+ doc references): ```protobuf rpc GetRegimeState(GetRegimeStateRequest) returns (GetRegimeStateResponse); rpc GetRegimeTransitions(GetRegimeTransitionsRequest) returns (GetRegimeTransitionsResponse); message GetRegimeStateRequest { string symbol = 1; } message GetRegimeStateResponse { string regime = 1; double confidence = 2; int64 event_timestamp = 3; double cusum_s_plus = 4; double cusum_s_minus = 5; double adx = 6; double stability = 7; double entropy = 8; } ``` **Implementation Status**: Endpoints defined and routed through API Gateway (confirmed in `AGENT_F8_REGIME_ROUTING_VALIDATION_REPORT.md`) --- ### 2. Integration Gap ❌ #### ❌ Trading Agent Service Allocation (`services/trading_agent_service/src/allocation.rs`) **Current State**: 716 lines, 5 allocation methods, **ZERO regime integration** **File**: `/home/jgrusewski/Work/foxhunt/services/trading_agent_service/src/allocation.rs` **Existing Allocation Methods**: 1. `EqualWeight` - 1/N allocation 2. `RiskParity` - Inverse volatility weighting 3. `MeanVariance` - Markowitz optimization 4. `MLOptimized` - ML scores as expected returns 5. `KellyCriterion` - Edge-based sizing **Missing**: ```rust // ❌ NO IMPORTS // use ml::features::regime_adaptive::RegimeAdaptiveFeatures; // use ml::ensemble::MarketRegime; // ❌ NO REGIME DETECTION pub fn allocate(&self, assets: &[AssetInfo], total_capital: Decimal) -> Result> { match &self.method { AllocationMethod::EqualWeight => self.equal_weight(assets, total_capital), AllocationMethod::RiskParity => self.risk_parity(assets, total_capital), // ... NO REGIME ADAPTATION } } ``` **Impact**: All allocation methods compute static weights WITHOUT regime-based adjustments. --- #### ❌ Service Implementation (`services/trading_agent_service/src/service.rs`) **File**: `/home/jgrusewski/Work/foxhunt/services/trading_agent_service/src/service.rs` (675 lines) **Current `allocate_portfolio` Implementation**: ```rust async fn allocate_portfolio( &self, _request: Request, ) -> Result, Status> { info!("AllocatePortfolio called (placeholder)"); Ok(Response::new(AllocatePortfolioResponse { allocations: vec![], metrics: Some(AllocationMetrics { total_weight: 0.0, portfolio_volatility: 0.0, portfolio_sharpe: 0.0, var_95: 0.0, max_drawdown_estimate: 0.0, }), timestamp: chrono::Utc::now().timestamp_nanos_opt().unwrap_or(0), allocation_id: uuid::Uuid::new_v4().to_string(), })) } ``` **Status**: **PLACEHOLDER** - no actual allocation logic, no regime detection --- ### 3. Data Flow Analysis #### Current Flow (Wave C - 201 Features) ``` 1. Universe Selection → [ES.FUT, NQ.FUT, ZN.FUT] 2. Asset Selection → AssetInfo[] (with ML scores) 3. PortfolioAllocator::allocate() → Static weights (e.g., 1/N) 4. Order Generation → Fixed position sizes 5. Trading Service → Execute orders ``` #### Missing Flow (Wave D - 225 Features) ``` 1. Universe Selection → [ES.FUT, NQ.FUT, ZN.FUT] 2. FOR EACH symbol: a. Query regime_states table → get_latest_regime(symbol) b. Retrieve MarketRegime (Normal/Trending/Volatile/Crisis) c. Look up position_multiplier (0.2x-1.5x) d. Look up stop_loss_multiplier (1.5x-4.0x ATR) 3. Asset Selection → AssetInfo[] (with ML scores) 4. ❌ MISSING: Apply regime multipliers to allocation weights 5. ❌ MISSING: Adjust position sizes by regime 6. Order Generation → ❌ Uses unadjusted sizes 7. Trading Service → ❌ Executes with wrong position sizes ``` --- ## Integration Plan ### Phase 1: Database Query Layer (2 hours) **File**: `services/trading_agent_service/src/regime.rs` (NEW) ```rust //! Regime Detection Integration //! //! Provides regime state queries for position sizing adjustments. use sqlx::PgPool; use anyhow::{Context, Result}; /// Market regime classification #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum MarketRegime { Normal, Trending, Sideways, Bull, Bear, HighVolatility, Crisis, } /// Regime state with metrics #[derive(Debug, Clone)] pub struct RegimeState { pub regime: MarketRegime, pub confidence: f64, pub event_timestamp: chrono::DateTime, pub position_multiplier: f64, // From adaptive_strategy_metrics pub stop_loss_multiplier: f64, // From adaptive_strategy_metrics } /// Regime detection client pub struct RegimeDetector { db_pool: PgPool, } impl RegimeDetector { pub fn new(db_pool: PgPool) -> Self { Self { db_pool } } /// Get current regime state for symbol pub async fn get_regime(&self, symbol: &str) -> Result { // Query database using get_latest_regime() function let row = sqlx::query!( r#" SELECT regime, confidence, event_timestamp FROM get_latest_regime($1) "#, symbol ) .fetch_one(&self.db_pool) .await .context("Failed to fetch regime state")?; // Query adaptive metrics let metrics = sqlx::query!( r#" SELECT position_multiplier, stop_loss_multiplier FROM adaptive_strategy_metrics WHERE symbol = $1 AND event_timestamp = $2 ORDER BY event_timestamp DESC LIMIT 1 "#, symbol, row.event_timestamp ) .fetch_one(&self.db_pool) .await .context("Failed to fetch adaptive metrics")?; Ok(RegimeState { regime: parse_regime(&row.regime.unwrap_or_default())?, confidence: row.confidence.unwrap_or(0.0), event_timestamp: row.event_timestamp.unwrap(), position_multiplier: metrics.position_multiplier, stop_loss_multiplier: metrics.stop_loss_multiplier, }) } /// Get regime for multiple symbols (batch query) pub async fn get_regimes_batch(&self, symbols: &[String]) -> Result> { let mut results = Vec::new(); for symbol in symbols { if let Ok(state) = self.get_regime(symbol).await { results.push((symbol.clone(), state)); } } Ok(results) } } fn parse_regime(s: &str) -> Result { match s { "Normal" => Ok(MarketRegime::Normal), "Trending" => Ok(MarketRegime::Trending), "Ranging" | "Sideways" => Ok(MarketRegime::Sideways), "Bull" => Ok(MarketRegime::Bull), "Bear" => Ok(MarketRegime::Bear), "Volatile" | "HighVolatility" => Ok(MarketRegime::HighVolatility), "Crisis" => Ok(MarketRegime::Crisis), _ => Ok(MarketRegime::Normal), // Default fallback } } ``` --- ### Phase 2: Allocation Integration (3 hours) **File**: `services/trading_agent_service/src/allocation.rs` (MODIFY) **Changes**: 1. **Add regime-aware allocation method**: ```rust #[derive(Debug, Clone)] pub enum AllocationMethod { EqualWeight, RiskParity, MeanVariance { lambda: f64 }, MLOptimized, KellyCriterion { fraction: f64 }, // ✅ NEW: Regime-adaptive allocation RegimeAdaptive { base_method: Box, regime_detector: Arc, }, } ``` 2. **Implement regime-adaptive wrapper**: ```rust async fn regime_adaptive( &self, assets: &[AssetInfo], total_capital: Decimal, base_method: &AllocationMethod, regime_detector: &RegimeDetector, ) -> Result> { // Step 1: Get base allocation (e.g., from MLOptimized) let base_allocator = PortfolioAllocator::new((**base_method).clone()); let base_allocation = base_allocator.allocate(assets, total_capital)?; // Step 2: Fetch regime states for all symbols let symbols: Vec = assets.iter().map(|a| a.symbol.clone()).collect(); let regime_states = regime_detector.get_regimes_batch(&symbols).await?; // Step 3: Apply regime multipliers let mut adjusted_allocation = HashMap::new(); for (symbol, base_capital) in base_allocation { if let Some((_, regime_state)) = regime_states.iter().find(|(s, _)| s == &symbol) { // Apply position multiplier (0.2x-1.5x) let adjusted_capital = base_capital * Decimal::from_f64_retain(regime_state.position_multiplier) .unwrap_or(Decimal::ONE); adjusted_allocation.insert(symbol, adjusted_capital); } else { // No regime data → use base allocation adjusted_allocation.insert(symbol, base_capital); } } // Step 4: Renormalize to total_capital (multipliers may exceed 100%) let sum: Decimal = adjusted_allocation.values().sum(); if sum > total_capital { for capital in adjusted_allocation.values_mut() { *capital = (*capital / sum) * total_capital; } } Ok(adjusted_allocation) } ``` 3. **Update `allocate()` to support regime method**: ```rust pub async fn allocate_async( &self, assets: &[AssetInfo], total_capital: Decimal, ) -> Result> { match &self.method { AllocationMethod::EqualWeight => self.equal_weight(assets, total_capital), AllocationMethod::RiskParity => self.risk_parity(assets, total_capital), AllocationMethod::MeanVariance { lambda } => self.mean_variance(assets, total_capital, *lambda), AllocationMethod::MLOptimized => self.ml_optimized(assets, total_capital), AllocationMethod::KellyCriterion { fraction } => self.kelly_criterion(assets, total_capital, *fraction), // ✅ NEW AllocationMethod::RegimeAdaptive { base_method, regime_detector } => { self.regime_adaptive(assets, total_capital, base_method, regime_detector).await }, } } ``` --- ### Phase 3: Service Wiring (2 hours) **File**: `services/trading_agent_service/src/service.rs` (MODIFY) **Changes**: 1. **Add RegimeDetector to service state**: ```rust pub struct TradingAgentServiceImpl { db_pool: PgPool, universe_selector: UniverseSelector, strategy_coordinator: StrategyCoordinator, metrics: TradingAgentMetrics, regime_detector: Arc, // ✅ NEW } impl TradingAgentServiceImpl { pub fn new(db_pool: PgPool) -> Self { Self { universe_selector: UniverseSelector::new(db_pool.clone()), strategy_coordinator: StrategyCoordinator::new(db_pool.clone()), metrics: TradingAgentMetrics::new(), regime_detector: Arc::new(RegimeDetector::new(db_pool.clone())), // ✅ NEW db_pool, } } } ``` 2. **Implement `allocate_portfolio` with regime adaptation**: ```rust async fn allocate_portfolio( &self, request: Request, ) -> Result, Status> { let req = request.into_inner(); // Convert proto assets to AssetInfo let assets: Vec = req.assets.iter().map(|a| { AssetInfo { symbol: a.symbol.clone(), expected_return: a.expected_return, volatility: a.volatility, ml_score: a.ml_score, // ... other fields } }).collect(); let total_capital = Decimal::from_f64_retain(req.total_capital) .ok_or_else(|| Status::invalid_argument("Invalid total capital"))?; // ✅ Use regime-adaptive allocation let base_method = AllocationMethod::MLOptimized; let allocator = PortfolioAllocator::new(AllocationMethod::RegimeAdaptive { base_method: Box::new(base_method), regime_detector: self.regime_detector.clone(), }); let allocations = allocator.allocate_async(&assets, total_capital) .await .map_err(|e| Status::internal(format!("Allocation failed: {}", e)))?; // Convert to proto let proto_allocations: Vec = allocations.iter().map(|(symbol, capital)| { Allocation { symbol: symbol.clone(), capital: capital.to_f64().unwrap_or(0.0), weight: (capital / total_capital).to_f64().unwrap_or(0.0), } }).collect(); Ok(Response::new(AllocatePortfolioResponse { allocations: proto_allocations, // ... metrics })) } ``` --- ### Phase 4: Order Generation Integration (1 hour) **File**: `services/trading_agent_service/src/orders.rs` (MODIFY) **Changes**: 1. **Add stop-loss multiplier to order generation**: ```rust pub async fn generate_order( &self, allocation: &PortfolioAllocation, regime_state: &RegimeState, // ✅ NEW parameter ) -> Result { let price = self.get_current_price(&allocation.symbol).await?; let quantity = (allocation.capital / price).floor(); // ✅ Apply regime-based stop-loss let atr = self.calculate_atr(&allocation.symbol, 14).await?; let stop_loss_distance = atr * regime_state.stop_loss_multiplier; let stop_loss_price = if allocation.direction == Direction::Long { price - stop_loss_distance } else { price + stop_loss_distance }; Ok(Order { symbol: allocation.symbol.clone(), quantity, price, stop_loss: Some(stop_loss_price), // ... other fields }) } ``` --- ### Phase 5: Testing (3 hours) **File**: `services/trading_agent_service/tests/regime_allocation_test.rs` (NEW) **Test Cases**: 1. **Test regime multiplier application**: - Crisis regime (0.2x) → $100K allocation → $20K actual - Trending regime (1.5x) → $100K allocation → $150K actual (then renormalized) - Normal regime (1.0x) → $100K allocation → $100K actual 2. **Test stop-loss adjustment**: - Volatile regime (3.0x ATR) → ATR=$10 → Stop=$30 away - Sideways regime (1.5x ATR) → ATR=$10 → Stop=$15 away 3. **Test database fallback**: - No regime data → Use base allocation without multipliers - Stale regime data (>1 hour old) → Fall back to Normal regime 4. **Test end-to-end flow**: - Mock regime_states table with test data - Call `allocate_portfolio` gRPC endpoint - Verify allocations reflect regime multipliers --- ## Validation Checklist ### Pre-Integration ✅ - [x] RegimeAdaptiveFeatures implemented (644 lines, 12/12 tests) - [x] Database schema created (regime_states, regime_transitions, adaptive_strategy_metrics) - [x] gRPC endpoints defined (GetRegimeState, GetRegimeTransitions) - [x] Migration 045 applied ### Post-Integration (To Be Verified) - [ ] `services/trading_agent_service/src/regime.rs` created - [ ] `services/trading_agent_service/src/allocation.rs` updated with RegimeAdaptive method - [ ] `services/trading_agent_service/src/service.rs` wired with RegimeDetector - [ ] `services/trading_agent_service/src/orders.rs` applies stop-loss multipliers - [ ] Test suite validates regime multipliers applied correctly - [ ] End-to-end test: Crisis regime → 0.2x position size - [ ] End-to-end test: Trending regime → 1.5x position size - [ ] End-to-end test: Volatile regime → 3.0x ATR stop-loss --- ## Risk Assessment ### Risk 1: Database Query Latency - **Impact**: Regime queries add latency to allocation decisions - **Mitigation**: - Use database indexes (already created: `idx_regime_states_symbol_timestamp`) - Batch queries for multiple symbols (`get_regimes_batch`) - Cache regime states (TTL: 1 minute) - **Acceptable Latency**: <5ms per symbol (P99) ### Risk 2: Stale Regime Data - **Impact**: Trading with outdated regime classifications - **Mitigation**: - Check `event_timestamp` (reject data >1 hour old) - Fall back to Normal regime (1.0x multiplier) if stale - Monitor `regime_states` freshness via Grafana ### Risk 3: Over-Leverage in Trending Regimes - **Impact**: 1.5x multiplier could exceed risk limits - **Mitigation**: - Renormalize allocations to 100% after applying multipliers - Hard cap: No single position >20% (already in allocation.rs) - Risk budget monitoring (Feature 224) ### Risk 4: Whipsaw in Crisis Regimes - **Impact**: 0.2x multiplier during false alarms → missed opportunities - **Mitigation**: - Require high confidence (>0.8) for Crisis regime - Monitor regime flip-flopping (>50/hour alert) - Manual override capability --- ## Performance Impact ### Latency Breakdown (Estimated) ``` Current (Wave C - 201 features): - Universe Selection: 50ms - Asset Selection: 20ms - Allocation (MLOptimized): 0.5ms - Order Generation: 10ms - TOTAL: 80.5ms With Regime Integration (Wave D - 225 features): - Universe Selection: 50ms - Asset Selection: 20ms - Regime Detection (batch query): +3ms ← NEW - Allocation (RegimeAdaptive): 0.5ms - Order Generation (with stop-loss): 10ms - TOTAL: 83.5ms (+3.7% overhead) ``` **Conclusion**: +3ms overhead is acceptable for 25-50% Sharpe improvement. --- ## Code Statistics ### Files to Create 1. `services/trading_agent_service/src/regime.rs` (~200 lines) 2. `services/trading_agent_service/tests/regime_allocation_test.rs` (~300 lines) ### Files to Modify 1. `services/trading_agent_service/src/allocation.rs` (+100 lines) 2. `services/trading_agent_service/src/service.rs` (+50 lines) 3. `services/trading_agent_service/src/orders.rs` (+30 lines) 4. `services/trading_agent_service/src/lib.rs` (+1 line for module export) ### Total Code Changes - **New Code**: ~500 lines - **Modified Code**: ~180 lines - **Total Effort**: ~11 hours (2+3+2+1+3) --- ## Rollback Plan ### Level 1: Feature Flag (Immediate) ```rust const ENABLE_REGIME_ADAPTIVE: bool = false; // Set to true to enable if ENABLE_REGIME_ADAPTIVE { AllocationMethod::RegimeAdaptive { ... } } else { AllocationMethod::MLOptimized // Fall back to Wave C behavior } ``` ### Level 2: Database Rollback (5 minutes) ```sql -- Disable regime tables (keep data) REVOKE SELECT ON regime_states FROM foxhunt; REVOKE SELECT ON adaptive_strategy_metrics FROM foxhunt; ``` ### Level 3: Code Rollback (10 minutes) ```bash git revert cargo build --release -p trading_agent_service systemctl restart trading_agent_service ``` --- ## Next Steps ### Immediate (Priority 1) 1. ✅ **This Report**: Document integration gap 2. ⏳ **User Decision**: Approve integration plan (11 hours effort) 3. ⏳ **Implementation**: Execute 5-phase integration plan ### Short-term (After Integration) 4. ⏳ **Testing**: Run regime allocation tests (3 hours) 5. ⏳ **Validation**: Paper trading with regime-adaptive sizing (1 week) 6. ⏳ **Monitoring**: Set up Grafana alerts for regime metrics ### Medium-term (Production) 7. ⏳ **Performance Tuning**: Optimize database queries (<5ms P99) 8. ⏳ **Caching**: Add 1-minute TTL cache for regime states 9. ⏳ **Documentation**: Update production runbooks --- ## Conclusion **Gap Confirmed**: Wave D's adaptive position sizing (Features 221-224) is **fully implemented** but **NOT integrated** into the trading decision flow. **Impact**: Without integration, the 225-feature ML models will include regime-adaptive features in training, but **production trading will ignore regime-based position sizing**. **Recommendation**: **PROCEED WITH INTEGRATION** before ML retraining. **Rationale**: 1. **Effort**: 11 hours (manageable) 2. **Risk**: Low (feature flag + rollback plan) 3. **Benefit**: +25-50% Sharpe improvement (per Wave D hypothesis) 4. **Urgency**: Must complete before 225-feature ML retraining (4-6 weeks) **Decision Required**: Approve integration plan and proceed with implementation? --- **Files Analyzed**: - `/home/jgrusewski/Work/foxhunt/ml/src/features/regime_adaptive.rs` (644 lines) - `/home/jgrusewski/Work/foxhunt/services/trading_agent_service/src/allocation.rs` (716 lines) - `/home/jgrusewski/Work/foxhunt/services/trading_agent_service/src/service.rs` (675 lines) - `/home/jgrusewski/Work/foxhunt/migrations/045_wave_d_regime_tracking.sql` (12,819 bytes) - `/home/jgrusewski/Work/foxhunt/AGENT_D11_PORTFOLIO_ALLOCATION_IMPLEMENTATION_REPORT.md` (analysis) **Agent WIRE-02 Complete** | 2025-10-19