foxhunt/docs/codebase-cleanup/DQN_STRESS_TESTING_ANALYSIS.md

# DQN Stress Testing Implementation Analysis

**Analysis Date:** 2025-11-27
**Evaluated Against:** 2025 Industry Standards for Financial ML Stress Testing
**File Analyzed:** `/home/jgrusewski/Work/foxhunt/ml/src/dqn/stress_testing.rs` (521 lines)

---

## Executive Summary

The DQN stress testing framework provides a **solid foundation** but falls short of 2025 institutional standards in several critical areas. While basic scenario coverage exists, the implementation lacks:
- **Real market data integration** (currently uses simulated metrics)
- **Advanced liquidity stress testing** (spread widening only)
- **Tail risk modeling** (EVT, copulas, fat-tailed distributions)
- **Multi-regime correlation breakdown scenarios**
- **Recovery path validation** (time-to-recovery metrics incomplete)

**Overall Grade:** C+ (Functional but incomplete for production deployment)

---

## 1. Scenario Generation (Current: Basic, Target: Advanced)

### Current Implementation ✅

**Strengths:**
- **8 predefined scenarios** covering standard stress events:
  ```rust
  flash_crash_scenario()           // -10% in 5 minutes
  liquidity_crisis_scenario()      // 50x spread widening
  vix_spike_scenario()             // 5x volatility
  trending_market_scenario()       // +8% strong trend
  whipsaw_scenario()               // rapid reversals
  gap_risk_scenario()              // -7% gap down
  correlation_breakdown_scenario() // multi-asset stress
  multi_asset_stress_scenario()    // combined stress (-12%)
  ```

- **Parameterized design** allows custom scenarios:
  ```rust
  pub struct StressScenario {
      price_shock_pct: f64,
      volatility_multiplier: f64,
      spread_multiplier: f64,
      duration_steps: usize,
      max_drawdown_threshold: f64,
      min_action_diversity: f64,
  }
  ```

- **Synthetic data generation** (`apply_stress()` method):
  ```rust
  fn apply_stress(&self, scenario: &StressScenario) -> Result<Vec<f64>> {
      // Gradual shock over first 20% of duration
      // Random walk with scaled volatility
      // Returns stressed price series
  }
  ```

### Critical Gaps ❌

**1. No Historical Scenario Replay**
- Missing ability to replay actual historical crises:
  - Flash Crash (May 6, 2010)
  - COVID-19 crash (March 2020)
  - Silicon Valley Bank collapse (March 2023)
  - FTX collapse (November 2022)

**2025 Standard:** Historical scenario libraries with tick-level data

**2. No Monte Carlo Scenario Generation**
- No stochastic scenario generation using:
  - Jump-diffusion processes
  - Regime-switching models
  - GARCH-based volatility paths
  - Copula-based multi-asset scenarios

**2025 Standard:** 10,000+ Monte Carlo paths per scenario

**3. No Conditional Scenarios**
- Missing conditional stress tests:
  - "Given VIX > 40, what if correlation → 1.0?"
  - "Given position size > 80%, what if liquidity dries up?"
  - "Given trending regime, what if sudden reversal?"

**2025 Standard:** Bayesian scenario trees with conditional probabilities

**4. Limited Severity Levels**
- Only single severity per scenario
- No stress ladder: mild → moderate → severe → extreme

**2025 Standard:** 4-5 severity levels per scenario with probability weights

---

## 2. Market Regime Stress Tests (Current: Basic, Target: Advanced)

### Current Implementation ⚠️

**Present but Incomplete:**
- **Basic regime scenarios**:
  - `trending_market_scenario()` - +8% trend
  - `whipsaw_scenario()` - rapid reversals
  - `vix_spike_scenario()` - volatility regime

- **Regime-conditional architecture exists** (separate file):
  ```rust
  // From ml/src/dqn/regime_conditional.rs
  pub enum RegimeType {
      Trending,   // High ADX, directional moves
      Ranging,    // Low ADX, mean reversion
      Volatile,   // High entropy, unpredictable
  }
  ```

### Critical Gaps ❌

**1. No Regime Transition Stress**
- Missing tests for regime shifts:
  - Trending → Volatile (breakdown)
  - Ranging → Trending (breakout)
  - Volatile → Ranging (normalization)

**2025 Standard:** Hidden Markov Model regime transitions with transition probabilities

**2. No Regime-Specific Thresholds**
- All scenarios use same thresholds regardless of regime:
  ```rust
  // Current: Same threshold for all regimes
  max_drawdown_threshold: 20.0,

  // 2025 Standard: Regime-adaptive thresholds
  // Trending: 15% drawdown acceptable
  // Ranging: 8% drawdown max
  // Volatile: 25% drawdown expected
  ```

**3. No Multi-Regime Scenarios**
- No tests combining multiple regimes:
  - Morning trending + afternoon whipsaw
  - Pre-market gap + intraday ranging
  - Volatile open + trending close

**2025 Standard:** Sequential regime chains with realistic durations

**4. No Regime Detection Under Stress**
- No validation that regime classifier remains accurate during stress
- Risk: Model may misclassify regime during crisis → wrong action selection

**2025 Standard:** Regime classification robustness tests (accuracy > 80% even at 3σ events)

---

## 3. Liquidity Stress Tests (Current: Primitive, Target: Advanced)

### Current Implementation ⚠️

**Basic Coverage:**
- **Spread widening** simulation:
  ```rust
  liquidity_crisis_scenario() {
      spread_multiplier: 50.0,  // 50x normal spread
      price_shock_pct: -2.0,
      duration_steps: 600,
  }
  ```

- **Circuit breaker integration** (separate module):
  ```rust
  // From ml/src/dqn/circuit_breaker.rs
  pub struct CircuitBreaker {
      failure_threshold: usize,
      timeout_duration: Duration,
      // Halts trading during consecutive failures
  }
  ```

### Critical Gaps ❌

**1. No Market Depth Modeling**
- Missing order book depth simulation:
  - No bid/ask queue sizes
  - No level 2 liquidity curves
  - No market depth degradation

**2025 Standard:** Full L2 order book replay with depth-dependent slippage

**Example Missing:**
```rust
// 2025 Standard
pub struct LiquidityStressParams {
    depth_levels: Vec<(Price, Quantity)>,  // Bid/ask ladder
    depth_decay_rate: f64,                 // How fast depth evaporates
    replenishment_rate: f64,               // How fast it recovers
    toxic_flow_intensity: f64,             // Informed trader activity
}
```

**2. No Market Impact Modeling**
- Missing dynamic market impact:
  - Temporary impact (recovers over time)
  - Permanent impact (price dislocation)
  - No Kyle's Lambda or Almgren-Chriss models

**Current Risk:** DQN may learn to trade large sizes without penalty

**2025 Standard:** Square-root market impact + transient impact decay

**3. No Liquidity Recovery Modeling**
- No simulation of liquidity returning after stress
- Recovery metrics incomplete:
  ```rust
  // Current: Simple heuristic
  let recovery_steps = if max_drawdown < 15.0 {
      Some(scenario.duration_steps / 2)
  } else {
      None
  };

  // 2025 Standard: Exponential recovery with half-life
  // recovery_time = f(shock_magnitude, market_regime, time_of_day)
  ```

**4. No Fragmentation Scenarios**
- Missing multi-venue liquidity tests:
  - Primary exchange vs dark pools
  - Cross-venue arbitrage during stress
  - Smart order routing under fragmentation

**2025 Standard:** Multi-venue liquidity aggregation with venue-specific stress

**5. No Adverse Selection**
- No toxic flow detection or informed trader scenarios
- DQN may learn exploitable patterns without adversarial testing

**2025 Standard:** Adversarial agents with information asymmetry

---

## 4. Correlation Breakdown Tests (Current: Minimal, Target: Comprehensive)

### Current Implementation ⚠️

**Nominal Coverage:**
- **Single scenario** for correlation breakdown:
  ```rust
  correlation_breakdown_scenario() {
      name: "Correlation Breakdown",
      price_shock_pct: -6.0,
      volatility_multiplier: 3.5,
      spread_multiplier: 7.0,
      duration_steps: 900,
  }
  ```

- **Multi-asset infrastructure exists** (separate module):
  ```rust
  // From ml/src/dqn/multi_asset.rs
  pub struct MultiAssetPortfolio {
      correlation_matrix: Array2<f64>,  // NxN correlation matrix
      positions: HashMap<String, Position>,
  }
  ```

### Critical Gaps ❌

**1. No Dynamic Correlation Modeling**
- Correlation matrix is **static** (identity matrix by default)
- No time-varying correlations (DCC-GARCH, EWMA)
- No correlation regime shifts

**2025 Standard:** Dynamic Conditional Correlation (DCC) with regime-dependent correlations

**Example Missing:**
```rust
// 2025 Standard
pub struct CorrelationStressParams {
    normal_correlation: Array2<f64>,    // Calm markets: ρ ≈ 0.3
    stress_correlation: Array2<f64>,    // Panic: ρ ≈ 0.9
    transition_speed: f64,              // How fast correlation converges
    asymmetry: bool,                    // Downside correlation > upside
}
```

**2. No Tail Dependence Modeling**
- Missing copula-based tail correlation:
  - Assets can be uncorrelated normally but correlated in tails
  - Critical for portfolio VaR estimation

**Current Risk:** Portfolio diversification may fail during crisis

**2025 Standard:** t-copulas or Archimedean copulas for tail dependence

**3. No Cross-Asset Contagion**
- No stress propagation across asset classes:
  - Equity crash → credit spread widening
  - FX volatility → commodity dislocation
  - Liquidity contagion across markets

**2025 Standard:** Multi-layer correlation networks with shock propagation

**4. No Factor Model Stress**
- Missing factor-based stress tests:
  - What if market beta → 1.5 (all stocks move together)?
  - What if size factor reverses?
  - What if momentum crashes?

**2025 Standard:** Barra-style factor risk model with factor shocks

---

## 5. Recovery Testing (Current: Incomplete, Target: Rigorous)

### Current Implementation ⚠️

**Basic Metrics:**
- **Recovery steps calculation**:
  ```rust
  pub struct StressResult {
      recovery_steps: Option<usize>,  // Time to 95% recovery
      // ...
  }
  ```

- **Simple heuristic**:
  ```rust
  let recovery_steps = if max_drawdown < 15.0 {
      Some(scenario.duration_steps / 2)
  } else {
      None
  };
  ```

### Critical Gaps ❌

**1. No Recovery Path Validation**
- Missing time-series validation:
  - Is recovery monotonic or oscillating?
  - Are there secondary drawdowns?
  - What's the recovery volatility?

**2025 Standard:** Full recovery trajectory analysis with confidence bands

**2. No Adaptive Recovery Modeling**
- Recovery time should depend on:
  - Market regime during recovery
  - Volatility levels
  - Liquidity conditions
  - Position size

**Current:** Fixed heuristic (duration / 2)

**2025 Standard:** Empirically-calibrated recovery models from historical data

**3. No Scarring Effects**
- Missing tests for permanent damage:
  - Reduced risk appetite after stress
  - Lower position sizes
  - Increased epsilon (more exploration)

**2025 Standard:** Agent behavior shift metrics (pre-stress vs post-stress)

**4. No Multiple Recovery Scenarios**
- Only tests single recovery path
- Missing:
  - V-shaped recovery (fast bounce)
  - U-shaped recovery (slow normalization)
  - W-shaped recovery (double dip)
  - L-shaped recovery (no recovery)

**2025 Standard:** Recovery pattern classification with probability distribution

**5. No Compounding Stress**
- What if second shock occurs during recovery?
- No tests for clustered volatility (GARCH effects)

**2025 Standard:** Compound stress scenarios with recovery interruption

---

## 6. Integration with Risk Framework

### Current Capabilities ✅

**Well-Integrated:**
1. **Circuit Breaker** (`ml/src/dqn/circuit_breaker.rs`)
   - Halts trading after consecutive failures
   - Cooldown periods
   - Half-open testing state

2. **Risk-Adjusted Rewards** (referenced in code)
   - Sharpe ratio integration
   - Drawdown penalties
   - VaR-based risk metrics

3. **Action Masking** (risk constraints)
   - Position limits
   - Drawdown limits
   - Compliance rules

### Missing Integrations ❌

**1. No VaR Backtesting**
- Missing Kupiec POF test (Proportion of Failures)
- No Christoffersen test (clustering of violations)
- No traffic light system (Basel III)

**2025 Standard:** Automated VaR model validation with regulatory tests

**2. No Expected Shortfall (ES) Calculation**
- VaR tells "how often" but not "how bad"
- ES = average loss beyond VaR threshold

**2025 Standard:** ES calculation for all scenarios (Basel III requirement)

**3. No Stress VaR**
- Missing stressed VaR (using crisis-period calibration)
- No incremental VaR (marginal contribution analysis)

**2025 Standard:** Stressed VaR + Incremental VaR for position sizing

---

## 7. Code Quality Assessment

### Strengths ✅

1. **Clean Architecture**
   - Clear separation of concerns
   - Builder pattern for scenarios
   - Composable stress tests

2. **Good Documentation**
   - Comprehensive module docs
   - Example usage provided
   - Clear parameter explanations

3. **Type Safety**
   - Strong typing throughout
   - Serde serialization for results
   - Result types for error handling

4. **Testability**
   - Unit tests for scenario definitions
   - Validation tests for thresholds
   - Smoke tests for basic functionality

### Weaknesses ❌

1. **Simulated Metrics** (Lines 153-170)
   ```rust
   // NOTE: Full DQN training integration would go here
   // For now, we simulate metrics based on scenario severity
   let severity_factor = (scenario.price_shock_pct.abs() +
                          scenario.volatility_multiplier) / 15.0;
   ```
   **Impact:** Not actually running DQN through stress scenarios

2. **Hardcoded Parameters**
   - Base price = 4000.0 (line 299)
   - Volatility = 0.02 (line 312)
   - Recovery heuristic = duration / 2 (line 166)

3. **No Parallelization**
   - Scenarios run sequentially (`run_all_scenarios()`)
   - Could leverage Rayon for parallel execution

4. **Limited Metrics**
   - Missing key metrics:
     - Maximum adverse excursion (MAE)
     - Time underwater
     - Ulcer index
     - Calmar ratio
     - Recovery factor

---

## 8. Recommended Improvements (Prioritized)

### Tier 1: Critical (Must-Have for Production) 🔴

1. **Integrate Real DQN Execution** (Effort: 3 days)
   ```rust
   // Replace simulation with actual DQN rollout
   pub fn run_scenario(&mut self, scenario: &StressScenario) -> Result<StressResult> {
       let stressed_data = self.apply_stress(scenario)?;

       // ACTUAL DQN evaluation on stressed data
       let mut cumulative_reward = 0.0;
       let mut max_drawdown = 0.0;
       let mut actions = Vec::new();

       for (i, price) in stressed_data.iter().enumerate() {
           let state = self.build_state(i, price, &stressed_data);
           let action = self.trainer.select_action(&state)?;
           let reward = self.trainer.step(action, &state)?;

           actions.push(action);
           cumulative_reward += reward;
           max_drawdown = max_drawdown.max(compute_drawdown(...));
       }

       // Compute real metrics from actual execution
   }
   ```

2. **Add Historical Scenario Replay** (Effort: 5 days)
   ```rust
   pub fn load_historical_scenario(
       scenario_name: &str,  // "flash_crash_2010", "covid_march_2020"
       data_path: &Path,
   ) -> Result<StressScenario> {
       // Load tick-level historical data
       // Replay exact price/volume/spread dynamics
   }
   ```

3. **Implement Market Depth Modeling** (Effort: 4 days)
   ```rust
   pub struct LiquidityProfile {
       bid_depth: Vec<(Price, Quantity)>,
       ask_depth: Vec<(Price, Quantity)>,
       depth_decay_halflife: Duration,
       replenishment_rate: f64,
   }

   pub fn apply_liquidity_stress(
       &mut self,
       profile: &LiquidityProfile,
       order_size: Quantity,
   ) -> (Price, Slippage, MarketImpact) {
       // Walk the book, compute slippage
   }
   ```

4. **Add Expected Shortfall** (Effort: 1 day)
   ```rust
   pub struct StressResult {
       // Existing fields...
       pub expected_shortfall: f64,  // CVaR / ES
       pub var_95: f64,
       pub var_99: f64,
       pub worst_1pct_avg: f64,
   }
   ```

### Tier 2: Important (Needed for Robustness) 🟡

5. **Dynamic Correlation Modeling** (Effort: 3 days)
   ```rust
   pub struct DynamicCorrelationModel {
       ewma_lambda: f64,
       correlation_regime: RegimeType,
       stress_correlation_matrix: Array2<f64>,
       normal_correlation_matrix: Array2<f64>,
   }
   ```

6. **Monte Carlo Scenario Generation** (Effort: 4 days)
   ```rust
   pub fn generate_monte_carlo_scenarios(
       num_scenarios: usize,
       process: StochasticProcess,  // JumpDiffusion, GARCH, etc.
   ) -> Vec<StressScenario> {
       // Generate 10,000+ scenarios
   }
   ```

7. **Recovery Path Validation** (Effort: 2 days)
   ```rust
   pub struct RecoveryMetrics {
       recovery_time: Option<Duration>,
       recovery_pattern: RecoveryPattern,  // V, U, W, L
       secondary_drawdowns: Vec<f64>,
       recovery_volatility: f64,
   }
   ```

8. **Regime Transition Stress** (Effort: 3 days)
   ```rust
   pub fn regime_transition_scenario(
       from: RegimeType,
       to: RegimeType,
       transition_speed: Duration,
   ) -> StressScenario {
       // Model regime shifts
   }
   ```

### Tier 3: Enhancement (Nice-to-Have) 🟢

9. **Parallel Execution** (Effort: 1 day)
   ```rust
   use rayon::prelude::*;

   pub fn run_all_scenarios_parallel(&mut self) -> Vec<StressResult> {
       self.scenarios
           .par_iter()
           .map(|scenario| self.run_scenario(scenario))
           .collect()
   }
   ```

10. **Adversarial Agents** (Effort: 5 days)
    ```rust
    pub struct AdversarialScenario {
        informed_trader_intensity: f64,
        front_running_probability: f64,
        spoofing_rate: f64,
    }
    ```

11. **Multi-Venue Fragmentation** (Effort: 4 days)
    ```rust
    pub struct VenueStressParams {
        venues: Vec<TradingVenue>,
        venue_correlations: Array2<f64>,
        cross_venue_arb_delay: Duration,
    }
    ```

---

## 9. Comparison with Industry Standards

| Feature | Current Status | 2025 Standard | Gap |
|---------|---------------|---------------|-----|
| **Scenario Coverage** | 8 predefined | 50+ historical + Monte Carlo | ❌ Large |
| **Severity Levels** | 1 per scenario | 4-5 levels per scenario | ❌ Large |
| **Market Depth** | None | Full L2 order book | ❌ Critical |
| **Market Impact** | None | Kyle's Lambda + Almgren-Chriss | ❌ Critical |
| **Correlation Dynamics** | Static | DCC-GARCH | ❌ Large |
| **Tail Dependence** | None | Copula-based | ❌ Large |
| **Recovery Modeling** | Simple heuristic | Empirical calibration | ❌ Moderate |
| **VaR Backtesting** | None | Kupiec + Christoffersen | ❌ Critical |
| **Expected Shortfall** | None | Full ES calculation | ❌ Critical |
| **Regime Transitions** | None | HMM-based | ❌ Moderate |
| **Execution** | Simulated | Real DQN rollout | ❌ Critical |
| **Parallelization** | Sequential | Rayon parallel | ⚠️ Minor |

---

## 10. Risk Assessment

### Production Deployment Risks 🔴

**If deployed as-is, the following risks exist:**

1. **False Confidence** (Severity: HIGH)
   - Simulated metrics may not reflect actual DQN behavior
   - Could pass stress tests in simulation but fail in production

2. **Liquidity Blindness** (Severity: HIGH)
   - No market impact modeling → DQN may learn unrealistic strategies
   - Large orders may cause unmodeled slippage

3. **Correlation Failure** (Severity: MEDIUM)
   - Static correlations → diversification may fail in crisis
   - Portfolio VaR underestimated

4. **Incomplete Recovery** (Severity: MEDIUM)
   - Simple recovery heuristic may miss scarring effects
   - Agent may not adapt after extreme stress

5. **Regulatory Risk** (Severity: HIGH for regulated firms)
   - Missing Basel III requirements (ES, Stressed VaR)
   - No VaR backtesting → cannot validate risk model

---

## 11. Suggested Next Steps

### Phase 1: Make It Real (2 weeks)
1. Integrate actual DQN execution into stress scenarios
2. Add market depth and slippage modeling
3. Implement Expected Shortfall calculation

### Phase 2: Add Realism (2 weeks)
4. Load historical crisis scenarios
5. Implement dynamic correlation modeling
6. Add regime transition stress tests

### Phase 3: Robustness (1 week)
7. Implement recovery path validation
8. Add VaR backtesting (Kupiec/Christoffersen)
9. Parallelize scenario execution

### Phase 4: Advanced Features (2 weeks)
10. Monte Carlo scenario generation
11. Adversarial agent testing
12. Multi-venue fragmentation modeling

---

## 12. Conclusion

The current DQN stress testing implementation provides a **good architectural foundation** with clean code and extensible design. However, it requires **significant enhancements** to meet 2025 institutional standards:

**Key Strengths:**
- ✅ Well-structured code with clear separation of concerns
- ✅ Comprehensive scenario coverage for basic stress events
- ✅ Integration with circuit breaker and risk management
- ✅ Good documentation and testability

**Critical Weaknesses:**
- ❌ **Simulated metrics instead of real DQN execution**
- ❌ **No market depth or liquidity modeling**
- ❌ **Missing Expected Shortfall and VaR backtesting**
- ❌ **Static correlations without tail dependence**
- ❌ **Incomplete recovery modeling**

**Recommended Action:** Prioritize **Tier 1 improvements** (real DQN execution, market depth, ES calculation) before considering production deployment.

**Estimated Effort:** 4-6 weeks of focused development to reach production-grade status.

---

## Appendix A: Code Examples from Analysis

### Example 1: Current Simulated Metrics (Lines 153-170)
```rust
// Apply stress scenario
let _stressed_data = self.apply_stress(scenario)?;

// Simulate stress test (NOTE: Full DQN training integration would go here)
// For now, we simulate metrics based on scenario severity
let severity_factor = (scenario.price_shock_pct.abs() + scenario.volatility_multiplier) / 15.0;

// Simulate portfolio performance under stress
let initial_portfolio = 100_000.0;
let max_drawdown = scenario.price_shock_pct.abs() * (1.0 + severity_factor * 0.5);
let final_portfolio = initial_portfolio * (1.0 + scenario.price_shock_pct / 100.0 * 0.8);

// Simulate action diversity (reduces under extreme stress)
let action_diversity = (70.0 - severity_factor * 15.0).max(20.0);
```

### Example 2: Synthetic Stress Data Generation (Lines 294-319)
```rust
fn apply_stress(&self, scenario: &StressScenario) -> Result<Vec<f64>> {
    let mut stressed_prices = Vec::new();
    let base_price = 4000.0; // ES futures typical price

    for step in 0..scenario.duration_steps {
        let progress = step as f64 / scenario.duration_steps as f64;

        // Apply price shock (gradual over first 20% of duration)
        let shock_factor = if progress < 0.2 {
            1.0 + (scenario.price_shock_pct / 100.0) * (progress / 0.2)
        } else {
            1.0 + (scenario.price_shock_pct / 100.0)
        };

        // Add volatility (random walk scaled by volatility multiplier)
        let volatility = 0.02 * scenario.volatility_multiplier * (rand::random::<f64>() - 0.5);

        let price = base_price * shock_factor * (1.0 + volatility);
        stressed_prices.push(price);
    }

    Ok(stressed_prices)
}
```

### Example 3: Recovery Heuristic (Line 166)
```rust
let recovery_steps = if max_drawdown < 15.0 {
    Some(scenario.duration_steps / 2)
} else {
    None
};
```

---

**Analyst:** Code Analyzer Agent
**Review Status:** Complete
**Confidence Level:** High (based on comprehensive codebase analysis)