# RL Agent Gaps Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Implement 7 features to close training quality and production readiness gaps in DQN/PPO agents.

**Architecture:** Vertical slices in dependency order. Each task is a self-contained feature that builds on previous ones. State vector expands from 54 → 57 → 185 dims across tasks.

**Tech Stack:** Rust, Candle (ML), tokio (async), Prometheus (metrics), gRPC (service integration). Build: `SQLX_OFFLINE=true cargo check --workspace`. Test: `SQLX_OFFLINE=true cargo test -p ml --lib`.

**Key Codebase Facts:**
- State vector: 51 market features (`features/extraction.rs:51 FeatureVector`) + 3 portfolio (`[value, position_size, spread]`) = **54 dims**
- Portfolio features already include `position_size` (signed). Missing: `unrealized_pnl`, `bars_in_position`, `cost_basis`.
- `RegimeType::classify_from_features()` in `regime_conditional.rs:101-139` already works (ADX+CUSUM). Only `RegimeDetectionEngine::detect_regime()` is a stub.
- `deployment/ab_testing.rs` already exists (26KB, 730+ lines) with TrafficSplitter. Needs Thompson Sampling extension, not rebuild.
- Feature dim constant: `walk_forward.rs` has `FEATURE_DIM = 51` (market only, separate from portfolio).
- Clippy: `#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::indexing_slicing)]`
- All struct literals must use `..Default::default()` for forward compat.

---

### Task 1: Position-Aware State Encoding

**Files:**
- Create: `crates/ml/src/features/position_features.rs`
- Modify: `crates/ml/src/features/mod.rs` — add `pub mod position_features;`
- Modify: `crates/ml/src/dqn/reward.rs` — update `TradingState` to carry new features
- Modify: `crates/ml/src/walk_forward.rs` — update NormStats if needed
- Modify: `crates/ml/src/trainers/dqn/trainer.rs` — wire position features into state construction
- Modify: `crates/ml/src/trainers/ppo.rs` — same for PPO trajectory state
- Test: inline `#[cfg(test)]` in `position_features.rs`

**Context:** The state vector is currently 54 dims (51 market + 3 portfolio: `[value, position_size, spread]`). We add 3 more portfolio features: `unrealized_pnl` (normalized by EMA σ), `bars_in_position` (log-scaled), `cost_basis` (bps relative to price). New total: **57 dims**.

**Step 1: Write the failing test**

Create `crates/ml/src/features/position_features.rs` with test:

```rust
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_position_features_flat() {
        let pf = PositionFeatures::new();
        let features = pf.extract(0.0, 0.0, 0, 0.0, 100.0);
        assert_eq!(features.len(), 3);
        assert!((features[0]).abs() < 1e-8); // unrealized_pnl = 0 when flat
        assert!((features[1]).abs() < 1e-8); // bars_in_position = 0
        assert!((features[2]).abs() < 1e-8); // cost_basis = 0 when flat
    }

    #[test]
    fn test_position_features_long() {
        let mut pf = PositionFeatures::new();
        // Feed some volatility history for normalization
        for _ in 0..100 { pf.update_volatility(0.02); }
        let features = pf.extract(1.0, 500.0, 50, 99.0, 100.0);
        assert_eq!(features.len(), 3);
        assert!(features[0] > 0.0); // positive unrealized_pnl
        assert!(features[1] > 0.0); // bars_in_position > 0
        assert!(features[2] > 0.0); // cost_basis: price rose from 99 to 100
    }
}
```

**Step 2: Run test to verify it fails**

Run: `SQLX_OFFLINE=true cargo test -p ml --lib -- features::position_features`
Expected: FAIL (module doesn't exist yet or struct not defined)

**Step 3: Write minimal implementation**

```rust
//! Position-aware features for RL agent state encoding.
//! Adds unrealized_pnl, bars_in_position, and cost_basis to the state vector.

/// Extracts position-aware features for the RL state vector.
#[derive(Debug)]
pub struct PositionFeatures {
    vol_ema: f64,
    vol_alpha: f64,
}

impl PositionFeatures {
    pub fn new() -> Self {
        Self { vol_ema: 0.02, vol_alpha: 0.01 }
    }

    pub fn update_volatility(&mut self, abs_return: f64) {
        self.vol_ema = self.vol_alpha * abs_return + (1.0 - self.vol_alpha) * self.vol_ema;
    }

    /// Extract 3 position features:
    /// [0] unrealized_pnl: normalized by EMA volatility
    /// [1] bars_in_position: log(1 + bars) / log(1001), in [0, 1]
    /// [2] cost_basis: (current_price - entry_price) / entry_price * 10000 (bps), clamped
    pub fn extract(
        &self,
        position_size: f64,
        unrealized_pnl: f64,
        bars_held: u64,
        entry_price: f64,
        current_price: f64,
    ) -> Vec<f64> {
        let vol = self.vol_ema.max(1e-8);
        let norm_pnl = if position_size.abs() < 1e-8 { 0.0 } else { unrealized_pnl / vol };
        let norm_bars = (1.0 + bars_held as f64).ln() / (1001.0_f64).ln();
        let cost_bps = if position_size.abs() < 1e-8 || entry_price.abs() < 1e-8 {
            0.0
        } else {
            ((current_price - entry_price) / entry_price * 10000.0).clamp(-500.0, 500.0)
        };
        vec![norm_pnl, norm_bars, cost_bps]
    }
}

impl Default for PositionFeatures {
    fn default() -> Self { Self::new() }
}
```

**Step 4: Run test to verify it passes**

Run: `SQLX_OFFLINE=true cargo test -p ml --lib -- features::position_features`
Expected: PASS

**Step 5: Wire into DQN trainer**

In `crates/ml/src/trainers/dqn/trainer.rs`, add `position_features: PositionFeatures` field to `DQNTrainer`. In the state construction path (where `portfolio_features` is built), append the 3 new features. Update `DQNConfig.state_dim` default from 54 to 57 where it's hardcoded.

**Step 6: Wire into PPO trainer**

Same integration in `crates/ml/src/trainers/ppo.rs` trajectory collection.

**Step 7: Update feature dim references**

Grep for `54` and `state_dim` across the crate. Update hyperopt bounds, walk-forward NormStats, and any hardcoded dims. Use `..Default::default()` in all config struct literals.

**Step 8: Run full test suite**

Run: `SQLX_OFFLINE=true cargo test -p ml --lib`
Expected: All pass (2503+)

**Step 9: Clippy + commit**

Run: `SQLX_OFFLINE=true cargo clippy -p ml -- -D warnings`
Commit: `feat(ml): add position-aware state encoding (54→57 dims)`

---

### Task 2: Regime Detection (Feature-Based + HMM)

**Files:**
- Modify: `crates/ml/src/regime_detection.rs` — replace stub with real feature-based classifier
- Create: `crates/ml/src/regime_detection/` directory (split module)
  - `mod.rs` — re-exports
  - `feature_classifier.rs` — real-time classifier
  - `hmm.rs` — offline HMM
- Modify: `crates/ml/src/dqn/regime_conditional.rs` — wire `RegimeDetectionEngine` to use real classifier
- Test: inline `#[cfg(test)]` in each new file

**Context:** `RegimeType::classify_from_features()` at `regime_conditional.rs:101-139` already classifies using ADX (feature 211) and CUSUM (feature 203). The `RegimeDetectionEngine` wrapping it is the stub. We need to:
1. Move classification logic into `feature_classifier.rs` with configurable thresholds
2. Add HMM for offline validation
3. Wire the engine to use the real classifier

**Step 1: Write failing test for feature classifier**

```rust
#[test]
fn test_trending_regime_high_adx() {
    let classifier = FeatureClassifier::new(ClassifierConfig::default());
    // ADX > 25, Hurst > 0.55
    let regime = classifier.classify(30.0, 0.6, 0.02, 0.02);
    assert_eq!(regime, RegimeType::Trending);
}

#[test]
fn test_volatile_regime_high_vol() {
    let classifier = FeatureClassifier::new(ClassifierConfig::default());
    // vol > 2σ above mean
    let regime = classifier.classify(10.0, 0.5, 0.08, 0.02);
    assert_eq!(regime, RegimeType::Volatile);
}
```

**Step 2-4: Implement FeatureClassifier**

Implement `FeatureClassifier` with configurable thresholds (ADX trending threshold, Hurst threshold, vol z-score threshold). The `classify()` method takes computed indicators and returns `RegimeType`.

**Step 5: Implement HMM (3-state)**

Simple Baum-Welch HMM with 3 states. `fit()` takes daily returns, `predict()` returns most likely state. Store transition matrix for persistence estimation.

**Step 6: Wire into RegimeDetectionEngine**

Replace stub `detect_regime()` with call to `FeatureClassifier::classify()`. Add `fit_hmm()` and `hmm_validate()` methods.

**Step 7: Wire into RegimeConditionalDQN**

In `regime_conditional.rs`, use the engine instead of inline `classify_from_features()`.

**Step 8: Tests + clippy + commit**

Run: `SQLX_OFFLINE=true cargo test -p ml --lib -- regime`
Commit: `feat(ml): real regime detection with feature classifier + HMM`

---

### Task 3: Slippage Modeling

**Files:**
- Create: `crates/ml/src/risk/slippage.rs`
- Modify: `crates/ml/src/risk/mod.rs` — add `pub mod slippage;`
- Modify: `crates/ml/src/dqn/reward.rs` — integrate slippage into cost penalty
- Test: inline `#[cfg(test)]` in `slippage.rs`

**Context:** Current costs are fixed per `FactoredAction::transaction_cost()` (15/5/10 bps). We add a `SlippageModel` trait with `LinearImpactModel` (volume-dependent) and `FixedCostModel` (backward compat wrapper).

**Step 1: Write failing test**

```rust
#[test]
fn test_linear_impact_higher_volume() {
    let model = LinearImpactModel::new(LinearImpactConfig::default());
    let cost_low = model.estimate_cost(1.0, 1000.0, 100_000.0, OrderType::Market);
    let cost_high = model.estimate_cost(1.0, 10_000.0, 100_000.0, OrderType::Market);
    assert!(cost_high > cost_low, "Higher volume should cost more");
}
```

**Step 2-4: Implement SlippageModel trait + LinearImpactModel + FixedCostModel**

```rust
pub trait SlippageModel: Send + Sync {
    fn estimate_cost(&self, position_change: f64, volume: f64, adv: f64, order_type: OrderType) -> f64;
}
```

LinearImpactModel: `cost = spread/2 + sqrt_impact * sqrt(volume / adv)` + order type premium.

**Step 5: Wire into reward.rs**

In the transaction cost penalty section (`reward.rs:818-876`), replace `action.transaction_cost()` with `slippage_model.estimate_cost()` when a model is configured. Add `slippage_model: Option<Box<dyn SlippageModel>>` to `RewardFunction`.

**Step 6: Tests + clippy + commit**

Commit: `feat(ml): add volume-dependent slippage model`

---

### Task 4: Curriculum Learning

**Files:**
- Create: `crates/ml/src/trainers/curriculum.rs`
- Modify: `crates/ml/src/trainers/mod.rs` — add `pub mod curriculum;`
- Modify: `crates/ml/src/trainers/dqn/trainer.rs` — integrate curriculum phase into training loop
- Modify: `crates/ml/src/dqn/action_space.rs` — add `get_curriculum_action_mask(phase, max_position)`
- Test: inline `#[cfg(test)]`

**Context:** Uses existing `get_valid_action_mask()` in `action_space.rs:32-58` as foundation. Adds phase-based filtering that restricts action space and regime exposure during early training.

**Step 1: Write failing test**

```rust
#[test]
fn test_phase1_masks_extreme_exposure() {
    let mask = get_curriculum_action_mask(CurriculumPhase::Basic, 2.0);
    // Phase 1: only Flat, Long50, Short50 → exposure indices 1,2,3 valid (not 0=Short100, 4=Long100)
    // Short100 actions (indices 0-8) should be masked
    assert!(!mask[0]); // Short100/Market/Patient
    // Flat actions (indices 18-26) should be valid
    assert!(mask[18]); // Flat/Market/Patient
}
```

**Step 2-4: Implement CurriculumScheduler**

```rust
pub enum CurriculumPhase { Basic, FullPosition, AllRegimes }

pub struct CurriculumScheduler {
    current_phase: CurriculumPhase,
    sharpe_history: VecDeque<f64>,
    config: CurriculumConfig,
}
```

Phase transitions gated by Sharpe thresholds over consecutive folds. Phase tracked in checkpoint metadata.

**Step 5: Wire into DQN trainer**

At epoch boundary, check phase transitions. Apply curriculum mask in addition to position mask.

**Step 6: Tests + clippy + commit**

Commit: `feat(ml): add curriculum learning with 3-phase schedule`

---

### Task 5: Multi-Timeframe Feature Fusion (Default On)

**Files:**
- Create: `crates/ml/src/features/multi_timeframe.rs`
- Create: `crates/ml/src/features/bar_resampler.rs`
- Modify: `crates/ml/src/features/mod.rs` — add modules
- Modify: `crates/ml/src/trainers/dqn/trainer.rs` — fuse multi-timeframe into state
- Modify: `crates/ml/src/trainers/ppo.rs` — same
- Test: inline `#[cfg(test)]`

**Context:** Currently 1-minute bars only. Resample on-the-fly to 5m/15m/1h. Per-timeframe LSTM encoder → concat → linear projection → 128-dim macro context. Total state: 57 (position-aware) + 128 (multi-tf) = **185 dims**. Default on.

**Step 1: Implement BarResampler**

Aggregates 1m OHLCV bars into higher timeframes (5m, 15m, 1h) with correct OHLC aggregation.

**Step 2: Implement MultiTimeframeEncoder**

Candle LSTM encoder per timeframe (shared weights optional). Concat 4×64=256, linear project to 128.

```rust
pub struct MultiTimeframeEncoder {
    encoders: [LSTMEncoder; 4], // 1m, 5m, 15m, 1h
    projection: Linear,         // 256 → 128
    resampler: BarResampler,
}
```

**Step 3: Wire into trainers**

Append 128-dim macro context after 57-dim micro features. Update state_dim configs.

**Step 4: Tests + clippy + commit**

Commit: `feat(ml): add multi-timeframe feature fusion (default on, 57→185 dims)`

---

### Task 6: Online Learning with EWC

**Files:**
- Create: `crates/ml/src/trainers/online_learning.rs`
- Modify: `crates/ml/src/trainers/mod.rs` — add module
- Modify: `crates/ml/src/trainers/dqn/config.rs` — add `OnlineLearningConfig`
- Test: inline `#[cfg(test)]`

**Context:** After walk-forward training, the model receives live trade data. Every 100 trades, perform a mini-update with EWC regularization to prevent catastrophic forgetting.

**Step 1: Implement EWC (Elastic Weight Consolidation)**

```rust
pub struct EWCRegularizer {
    fisher_diagonal: HashMap<String, Tensor>,  // per-parameter Fisher info
    optimal_params: HashMap<String, Tensor>,   // θ* from initial training
    lambda: f64,                                // EWC strength (default 1000.0)
}
```

Compute Fisher Information Matrix (diagonal approx) from initial training data. EWC loss = `task_loss + λ * Σ F_i * (θ_i - θ*_i)²`.

**Step 2: Implement OnlineLearner**

```rust
pub struct OnlineLearner {
    rolling_buffer: VecDeque<Experience>,
    buffer_capacity: usize,        // 10,000
    update_interval: usize,        // 100 trades
    trade_count: usize,
    ewc: EWCRegularizer,
    lr_multiplier: f64,            // 0.1
    max_grad_norm: f64,            // 1.0
    sharpe_monitor: RollingSharpe, // 500-trade window
    checkpoint_path: PathBuf,
}
```

**Step 3: Safety rails**

- Gradient clipping: `max_grad_norm = 1.0`
- Learning rate: `base_lr * 0.1`
- Auto-rollback: if rolling Sharpe degrades > 20% over 500 trades
- Kill switch: if rolling Sharpe < -1.0, freeze model

**Step 4: Tests + clippy + commit**

Commit: `feat(ml): add online learning with EWC regularization`

---

### Task 7: A/B Testing — Thompson Sampling Extension

**Files:**
- Modify: `crates/ml/src/deployment/ab_testing.rs` — add Thompson Sampling strategy
- Test: extend existing tests in same file

**Context:** `ab_testing.rs` already has 730+ lines with `TrafficSplitter`, `ABTestConfig`, `TestGroup`, and 4 splitting strategies (HashBased, Random, RoundRobin, WeightedRandom). We ADD `ThompsonSampling` as a 5th strategy with Bayesian promotion criteria.

**Step 1: Add ThompsonSampling variant to TrafficSplittingStrategy**

```rust
pub enum TrafficSplittingStrategy {
    HashBased,
    Random,
    RoundRobin,
    WeightedRandom,
    ThompsonSampling,  // NEW
}
```

**Step 2: Implement Thompson Sampling logic**

```rust
pub struct ThompsonSamplingState {
    control_alpha: f64,    // Beta distribution α for champion
    control_beta: f64,     // Beta distribution β for champion
    treatment_alpha: f64,  // Beta distribution α for challenger
    treatment_beta: f64,   // Beta distribution β for challenger
}
```

Sample from Beta(α, β) for each arm; assign trade to higher sample. Update α/β after trade outcome.

**Step 3: Add Bayesian promotion criteria**

```rust
pub struct PromotionCriteria {
    pub min_trades: usize,           // 500
    pub bayesian_confidence: f64,    // 0.95
    pub max_drawdown_ratio: f64,     // 1.2 (challenger DD ≤ 1.2× champion DD)
    pub require_manual_approval: bool, // true
}
```

**Step 4: Wire into TrafficSplitter::assign_group()**

Add `ThompsonSampling` arm to the match in `assign_group()` (currently at line 443-448).

**Step 5: Tests + clippy + commit**

Commit: `feat(ml): add Thompson Sampling and Bayesian promotion to A/B testing`

---

## Execution Notes

**Build command:** `SQLX_OFFLINE=true cargo check --workspace`
**Test command:** `SQLX_OFFLINE=true cargo test -p ml --lib`
**Clippy command:** `SQLX_OFFLINE=true cargo clippy -p ml -- -D warnings`

**Critical rules:**
- Never use `.unwrap()`, `.expect()`, `.panic!()`, or `[index]` — use `.get()`, `?`, `.ok_or()`
- All struct literals use `..Default::default()`
- Candle f32 trap: `Tensor * 0.5` fails. Use `tensor.broadcast_mul(&Tensor::new(0.5_f32, device)?)`
- Each task should be committed individually before starting the next
- Run full `cargo test -p ml --lib` after each task to catch regressions