diff --git a/docs/plans/2026-03-03-rl-agent-gaps-implementation.md b/docs/plans/2026-03-03-rl-agent-gaps-implementation.md new file mode 100644 index 000000000..2ba75222c --- /dev/null +++ b/docs/plans/2026-03-03-rl-agent-gaps-implementation.md @@ -0,0 +1,457 @@ +# RL Agent Gaps Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Implement 7 features to close training quality and production readiness gaps in DQN/PPO agents. + +**Architecture:** Vertical slices in dependency order. Each task is a self-contained feature that builds on previous ones. State vector expands from 54 → 57 → 185 dims across tasks. + +**Tech Stack:** Rust, Candle (ML), tokio (async), Prometheus (metrics), gRPC (service integration). Build: `SQLX_OFFLINE=true cargo check --workspace`. Test: `SQLX_OFFLINE=true cargo test -p ml --lib`. + +**Key Codebase Facts:** +- State vector: 51 market features (`features/extraction.rs:51 FeatureVector`) + 3 portfolio (`[value, position_size, spread]`) = **54 dims** +- Portfolio features already include `position_size` (signed). Missing: `unrealized_pnl`, `bars_in_position`, `cost_basis`. +- `RegimeType::classify_from_features()` in `regime_conditional.rs:101-139` already works (ADX+CUSUM). Only `RegimeDetectionEngine::detect_regime()` is a stub. +- `deployment/ab_testing.rs` already exists (26KB, 730+ lines) with TrafficSplitter. Needs Thompson Sampling extension, not rebuild. +- Feature dim constant: `walk_forward.rs` has `FEATURE_DIM = 51` (market only, separate from portfolio). +- Clippy: `#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::indexing_slicing)]` +- All struct literals must use `..Default::default()` for forward compat. + +--- + +### Task 1: Position-Aware State Encoding + +**Files:** +- Create: `crates/ml/src/features/position_features.rs` +- Modify: `crates/ml/src/features/mod.rs` — add `pub mod position_features;` +- Modify: `crates/ml/src/dqn/reward.rs` — update `TradingState` to carry new features +- Modify: `crates/ml/src/walk_forward.rs` — update NormStats if needed +- Modify: `crates/ml/src/trainers/dqn/trainer.rs` — wire position features into state construction +- Modify: `crates/ml/src/trainers/ppo.rs` — same for PPO trajectory state +- Test: inline `#[cfg(test)]` in `position_features.rs` + +**Context:** The state vector is currently 54 dims (51 market + 3 portfolio: `[value, position_size, spread]`). We add 3 more portfolio features: `unrealized_pnl` (normalized by EMA σ), `bars_in_position` (log-scaled), `cost_basis` (bps relative to price). New total: **57 dims**. + +**Step 1: Write the failing test** + +Create `crates/ml/src/features/position_features.rs` with test: + +```rust +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_position_features_flat() { + let pf = PositionFeatures::new(); + let features = pf.extract(0.0, 0.0, 0, 0.0, 100.0); + assert_eq!(features.len(), 3); + assert!((features[0]).abs() < 1e-8); // unrealized_pnl = 0 when flat + assert!((features[1]).abs() < 1e-8); // bars_in_position = 0 + assert!((features[2]).abs() < 1e-8); // cost_basis = 0 when flat + } + + #[test] + fn test_position_features_long() { + let mut pf = PositionFeatures::new(); + // Feed some volatility history for normalization + for _ in 0..100 { pf.update_volatility(0.02); } + let features = pf.extract(1.0, 500.0, 50, 99.0, 100.0); + assert_eq!(features.len(), 3); + assert!(features[0] > 0.0); // positive unrealized_pnl + assert!(features[1] > 0.0); // bars_in_position > 0 + assert!(features[2] > 0.0); // cost_basis: price rose from 99 to 100 + } +} +``` + +**Step 2: Run test to verify it fails** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib -- features::position_features` +Expected: FAIL (module doesn't exist yet or struct not defined) + +**Step 3: Write minimal implementation** + +```rust +//! Position-aware features for RL agent state encoding. +//! Adds unrealized_pnl, bars_in_position, and cost_basis to the state vector. + +/// Extracts position-aware features for the RL state vector. +#[derive(Debug)] +pub struct PositionFeatures { + vol_ema: f64, + vol_alpha: f64, +} + +impl PositionFeatures { + pub fn new() -> Self { + Self { vol_ema: 0.02, vol_alpha: 0.01 } + } + + pub fn update_volatility(&mut self, abs_return: f64) { + self.vol_ema = self.vol_alpha * abs_return + (1.0 - self.vol_alpha) * self.vol_ema; + } + + /// Extract 3 position features: + /// [0] unrealized_pnl: normalized by EMA volatility + /// [1] bars_in_position: log(1 + bars) / log(1001), in [0, 1] + /// [2] cost_basis: (current_price - entry_price) / entry_price * 10000 (bps), clamped + pub fn extract( + &self, + position_size: f64, + unrealized_pnl: f64, + bars_held: u64, + entry_price: f64, + current_price: f64, + ) -> Vec { + let vol = self.vol_ema.max(1e-8); + let norm_pnl = if position_size.abs() < 1e-8 { 0.0 } else { unrealized_pnl / vol }; + let norm_bars = (1.0 + bars_held as f64).ln() / (1001.0_f64).ln(); + let cost_bps = if position_size.abs() < 1e-8 || entry_price.abs() < 1e-8 { + 0.0 + } else { + ((current_price - entry_price) / entry_price * 10000.0).clamp(-500.0, 500.0) + }; + vec![norm_pnl, norm_bars, cost_bps] + } +} + +impl Default for PositionFeatures { + fn default() -> Self { Self::new() } +} +``` + +**Step 4: Run test to verify it passes** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib -- features::position_features` +Expected: PASS + +**Step 5: Wire into DQN trainer** + +In `crates/ml/src/trainers/dqn/trainer.rs`, add `position_features: PositionFeatures` field to `DQNTrainer`. In the state construction path (where `portfolio_features` is built), append the 3 new features. Update `DQNConfig.state_dim` default from 54 to 57 where it's hardcoded. + +**Step 6: Wire into PPO trainer** + +Same integration in `crates/ml/src/trainers/ppo.rs` trajectory collection. + +**Step 7: Update feature dim references** + +Grep for `54` and `state_dim` across the crate. Update hyperopt bounds, walk-forward NormStats, and any hardcoded dims. Use `..Default::default()` in all config struct literals. + +**Step 8: Run full test suite** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib` +Expected: All pass (2503+) + +**Step 9: Clippy + commit** + +Run: `SQLX_OFFLINE=true cargo clippy -p ml -- -D warnings` +Commit: `feat(ml): add position-aware state encoding (54→57 dims)` + +--- + +### Task 2: Regime Detection (Feature-Based + HMM) + +**Files:** +- Modify: `crates/ml/src/regime_detection.rs` — replace stub with real feature-based classifier +- Create: `crates/ml/src/regime_detection/` directory (split module) + - `mod.rs` — re-exports + - `feature_classifier.rs` — real-time classifier + - `hmm.rs` — offline HMM +- Modify: `crates/ml/src/dqn/regime_conditional.rs` — wire `RegimeDetectionEngine` to use real classifier +- Test: inline `#[cfg(test)]` in each new file + +**Context:** `RegimeType::classify_from_features()` at `regime_conditional.rs:101-139` already classifies using ADX (feature 211) and CUSUM (feature 203). The `RegimeDetectionEngine` wrapping it is the stub. We need to: +1. Move classification logic into `feature_classifier.rs` with configurable thresholds +2. Add HMM for offline validation +3. Wire the engine to use the real classifier + +**Step 1: Write failing test for feature classifier** + +```rust +#[test] +fn test_trending_regime_high_adx() { + let classifier = FeatureClassifier::new(ClassifierConfig::default()); + // ADX > 25, Hurst > 0.55 + let regime = classifier.classify(30.0, 0.6, 0.02, 0.02); + assert_eq!(regime, RegimeType::Trending); +} + +#[test] +fn test_volatile_regime_high_vol() { + let classifier = FeatureClassifier::new(ClassifierConfig::default()); + // vol > 2σ above mean + let regime = classifier.classify(10.0, 0.5, 0.08, 0.02); + assert_eq!(regime, RegimeType::Volatile); +} +``` + +**Step 2-4: Implement FeatureClassifier** + +Implement `FeatureClassifier` with configurable thresholds (ADX trending threshold, Hurst threshold, vol z-score threshold). The `classify()` method takes computed indicators and returns `RegimeType`. + +**Step 5: Implement HMM (3-state)** + +Simple Baum-Welch HMM with 3 states. `fit()` takes daily returns, `predict()` returns most likely state. Store transition matrix for persistence estimation. + +**Step 6: Wire into RegimeDetectionEngine** + +Replace stub `detect_regime()` with call to `FeatureClassifier::classify()`. Add `fit_hmm()` and `hmm_validate()` methods. + +**Step 7: Wire into RegimeConditionalDQN** + +In `regime_conditional.rs`, use the engine instead of inline `classify_from_features()`. + +**Step 8: Tests + clippy + commit** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib -- regime` +Commit: `feat(ml): real regime detection with feature classifier + HMM` + +--- + +### Task 3: Slippage Modeling + +**Files:** +- Create: `crates/ml/src/risk/slippage.rs` +- Modify: `crates/ml/src/risk/mod.rs` — add `pub mod slippage;` +- Modify: `crates/ml/src/dqn/reward.rs` — integrate slippage into cost penalty +- Test: inline `#[cfg(test)]` in `slippage.rs` + +**Context:** Current costs are fixed per `FactoredAction::transaction_cost()` (15/5/10 bps). We add a `SlippageModel` trait with `LinearImpactModel` (volume-dependent) and `FixedCostModel` (backward compat wrapper). + +**Step 1: Write failing test** + +```rust +#[test] +fn test_linear_impact_higher_volume() { + let model = LinearImpactModel::new(LinearImpactConfig::default()); + let cost_low = model.estimate_cost(1.0, 1000.0, 100_000.0, OrderType::Market); + let cost_high = model.estimate_cost(1.0, 10_000.0, 100_000.0, OrderType::Market); + assert!(cost_high > cost_low, "Higher volume should cost more"); +} +``` + +**Step 2-4: Implement SlippageModel trait + LinearImpactModel + FixedCostModel** + +```rust +pub trait SlippageModel: Send + Sync { + fn estimate_cost(&self, position_change: f64, volume: f64, adv: f64, order_type: OrderType) -> f64; +} +``` + +LinearImpactModel: `cost = spread/2 + sqrt_impact * sqrt(volume / adv)` + order type premium. + +**Step 5: Wire into reward.rs** + +In the transaction cost penalty section (`reward.rs:818-876`), replace `action.transaction_cost()` with `slippage_model.estimate_cost()` when a model is configured. Add `slippage_model: Option>` to `RewardFunction`. + +**Step 6: Tests + clippy + commit** + +Commit: `feat(ml): add volume-dependent slippage model` + +--- + +### Task 4: Curriculum Learning + +**Files:** +- Create: `crates/ml/src/trainers/curriculum.rs` +- Modify: `crates/ml/src/trainers/mod.rs` — add `pub mod curriculum;` +- Modify: `crates/ml/src/trainers/dqn/trainer.rs` — integrate curriculum phase into training loop +- Modify: `crates/ml/src/dqn/action_space.rs` — add `get_curriculum_action_mask(phase, max_position)` +- Test: inline `#[cfg(test)]` + +**Context:** Uses existing `get_valid_action_mask()` in `action_space.rs:32-58` as foundation. Adds phase-based filtering that restricts action space and regime exposure during early training. + +**Step 1: Write failing test** + +```rust +#[test] +fn test_phase1_masks_extreme_exposure() { + let mask = get_curriculum_action_mask(CurriculumPhase::Basic, 2.0); + // Phase 1: only Flat, Long50, Short50 → exposure indices 1,2,3 valid (not 0=Short100, 4=Long100) + // Short100 actions (indices 0-8) should be masked + assert!(!mask[0]); // Short100/Market/Patient + // Flat actions (indices 18-26) should be valid + assert!(mask[18]); // Flat/Market/Patient +} +``` + +**Step 2-4: Implement CurriculumScheduler** + +```rust +pub enum CurriculumPhase { Basic, FullPosition, AllRegimes } + +pub struct CurriculumScheduler { + current_phase: CurriculumPhase, + sharpe_history: VecDeque, + config: CurriculumConfig, +} +``` + +Phase transitions gated by Sharpe thresholds over consecutive folds. Phase tracked in checkpoint metadata. + +**Step 5: Wire into DQN trainer** + +At epoch boundary, check phase transitions. Apply curriculum mask in addition to position mask. + +**Step 6: Tests + clippy + commit** + +Commit: `feat(ml): add curriculum learning with 3-phase schedule` + +--- + +### Task 5: Multi-Timeframe Feature Fusion (Default On) + +**Files:** +- Create: `crates/ml/src/features/multi_timeframe.rs` +- Create: `crates/ml/src/features/bar_resampler.rs` +- Modify: `crates/ml/src/features/mod.rs` — add modules +- Modify: `crates/ml/src/trainers/dqn/trainer.rs` — fuse multi-timeframe into state +- Modify: `crates/ml/src/trainers/ppo.rs` — same +- Test: inline `#[cfg(test)]` + +**Context:** Currently 1-minute bars only. Resample on-the-fly to 5m/15m/1h. Per-timeframe LSTM encoder → concat → linear projection → 128-dim macro context. Total state: 57 (position-aware) + 128 (multi-tf) = **185 dims**. Default on. + +**Step 1: Implement BarResampler** + +Aggregates 1m OHLCV bars into higher timeframes (5m, 15m, 1h) with correct OHLC aggregation. + +**Step 2: Implement MultiTimeframeEncoder** + +Candle LSTM encoder per timeframe (shared weights optional). Concat 4×64=256, linear project to 128. + +```rust +pub struct MultiTimeframeEncoder { + encoders: [LSTMEncoder; 4], // 1m, 5m, 15m, 1h + projection: Linear, // 256 → 128 + resampler: BarResampler, +} +``` + +**Step 3: Wire into trainers** + +Append 128-dim macro context after 57-dim micro features. Update state_dim configs. + +**Step 4: Tests + clippy + commit** + +Commit: `feat(ml): add multi-timeframe feature fusion (default on, 57→185 dims)` + +--- + +### Task 6: Online Learning with EWC + +**Files:** +- Create: `crates/ml/src/trainers/online_learning.rs` +- Modify: `crates/ml/src/trainers/mod.rs` — add module +- Modify: `crates/ml/src/trainers/dqn/config.rs` — add `OnlineLearningConfig` +- Test: inline `#[cfg(test)]` + +**Context:** After walk-forward training, the model receives live trade data. Every 100 trades, perform a mini-update with EWC regularization to prevent catastrophic forgetting. + +**Step 1: Implement EWC (Elastic Weight Consolidation)** + +```rust +pub struct EWCRegularizer { + fisher_diagonal: HashMap, // per-parameter Fisher info + optimal_params: HashMap, // θ* from initial training + lambda: f64, // EWC strength (default 1000.0) +} +``` + +Compute Fisher Information Matrix (diagonal approx) from initial training data. EWC loss = `task_loss + λ * Σ F_i * (θ_i - θ*_i)²`. + +**Step 2: Implement OnlineLearner** + +```rust +pub struct OnlineLearner { + rolling_buffer: VecDeque, + buffer_capacity: usize, // 10,000 + update_interval: usize, // 100 trades + trade_count: usize, + ewc: EWCRegularizer, + lr_multiplier: f64, // 0.1 + max_grad_norm: f64, // 1.0 + sharpe_monitor: RollingSharpe, // 500-trade window + checkpoint_path: PathBuf, +} +``` + +**Step 3: Safety rails** + +- Gradient clipping: `max_grad_norm = 1.0` +- Learning rate: `base_lr * 0.1` +- Auto-rollback: if rolling Sharpe degrades > 20% over 500 trades +- Kill switch: if rolling Sharpe < -1.0, freeze model + +**Step 4: Tests + clippy + commit** + +Commit: `feat(ml): add online learning with EWC regularization` + +--- + +### Task 7: A/B Testing — Thompson Sampling Extension + +**Files:** +- Modify: `crates/ml/src/deployment/ab_testing.rs` — add Thompson Sampling strategy +- Test: extend existing tests in same file + +**Context:** `ab_testing.rs` already has 730+ lines with `TrafficSplitter`, `ABTestConfig`, `TestGroup`, and 4 splitting strategies (HashBased, Random, RoundRobin, WeightedRandom). We ADD `ThompsonSampling` as a 5th strategy with Bayesian promotion criteria. + +**Step 1: Add ThompsonSampling variant to TrafficSplittingStrategy** + +```rust +pub enum TrafficSplittingStrategy { + HashBased, + Random, + RoundRobin, + WeightedRandom, + ThompsonSampling, // NEW +} +``` + +**Step 2: Implement Thompson Sampling logic** + +```rust +pub struct ThompsonSamplingState { + control_alpha: f64, // Beta distribution α for champion + control_beta: f64, // Beta distribution β for champion + treatment_alpha: f64, // Beta distribution α for challenger + treatment_beta: f64, // Beta distribution β for challenger +} +``` + +Sample from Beta(α, β) for each arm; assign trade to higher sample. Update α/β after trade outcome. + +**Step 3: Add Bayesian promotion criteria** + +```rust +pub struct PromotionCriteria { + pub min_trades: usize, // 500 + pub bayesian_confidence: f64, // 0.95 + pub max_drawdown_ratio: f64, // 1.2 (challenger DD ≤ 1.2× champion DD) + pub require_manual_approval: bool, // true +} +``` + +**Step 4: Wire into TrafficSplitter::assign_group()** + +Add `ThompsonSampling` arm to the match in `assign_group()` (currently at line 443-448). + +**Step 5: Tests + clippy + commit** + +Commit: `feat(ml): add Thompson Sampling and Bayesian promotion to A/B testing` + +--- + +## Execution Notes + +**Build command:** `SQLX_OFFLINE=true cargo check --workspace` +**Test command:** `SQLX_OFFLINE=true cargo test -p ml --lib` +**Clippy command:** `SQLX_OFFLINE=true cargo clippy -p ml -- -D warnings` + +**Critical rules:** +- Never use `.unwrap()`, `.expect()`, `.panic!()`, or `[index]` — use `.get()`, `?`, `.ok_or()` +- All struct literals use `..Default::default()` +- Candle f32 trap: `Tensor * 0.5` fails. Use `tensor.broadcast_mul(&Tensor::new(0.5_f32, device)?)` +- Each task should be committed individually before starting the next +- Run full `cargo test -p ml --lib` after each task to catch regressions