foxhunt/docs/codebase-cleanup/DQN_STRESS_TESTING_ANALYSIS.md

DQN Stress Testing Implementation Analysis

Analysis Date: 2025-11-27 Evaluated Against: 2025 Industry Standards for Financial ML Stress Testing File Analyzed: /home/jgrusewski/Work/foxhunt/ml/src/dqn/stress_testing.rs (521 lines)

---

Executive Summary

The DQN stress testing framework provides a solid foundation but falls short of 2025 institutional standards in several critical areas. While basic scenario coverage exists, the implementation lacks:

• Real market data integration (currently uses simulated metrics)
• Advanced liquidity stress testing (spread widening only)
• Tail risk modeling (EVT, copulas, fat-tailed distributions)
• Multi-regime correlation breakdown scenarios
• Recovery path validation (time-to-recovery metrics incomplete)

Overall Grade: C+ (Functional but incomplete for production deployment)

---

1. Scenario Generation (Current: Basic, Target: Advanced)

Current Implementation ✅

Strengths:

• 8 predefined scenarios covering standard stress events:

  flash_crash_scenario()           // -10% in 5 minutes
  liquidity_crisis_scenario()      // 50x spread widening
  vix_spike_scenario()             // 5x volatility
  trending_market_scenario()       // +8% strong trend
  whipsaw_scenario()               // rapid reversals
  gap_risk_scenario()              // -7% gap down
  correlation_breakdown_scenario() // multi-asset stress
  multi_asset_stress_scenario()    // combined stress (-12%)
  

• Parameterized design allows custom scenarios:

  pub struct StressScenario {
      price_shock_pct: f64,
      volatility_multiplier: f64,
      spread_multiplier: f64,
      duration_steps: usize,
      max_drawdown_threshold: f64,
      min_action_diversity: f64,
  }
  

• Synthetic data generation (apply_stress() method):

  fn apply_stress(&self, scenario: &StressScenario) -> Result<Vec<f64>> {
      // Gradual shock over first 20% of duration
      // Random walk with scaled volatility
      // Returns stressed price series
  }
  

Critical Gaps ❌

1. No Historical Scenario Replay

• Missing ability to replay actual historical crises:
  • Flash Crash (May 6, 2010)
  • COVID-19 crash (March 2020)
  • Silicon Valley Bank collapse (March 2023)
  • FTX collapse (November 2022)

2025 Standard: Historical scenario libraries with tick-level data

2. No Monte Carlo Scenario Generation

• No stochastic scenario generation using:
  • Jump-diffusion processes
  • Regime-switching models
  • GARCH-based volatility paths
  • Copula-based multi-asset scenarios

2025 Standard: 10,000+ Monte Carlo paths per scenario

3. No Conditional Scenarios

• Missing conditional stress tests:
  • "Given VIX > 40, what if correlation → 1.0?"
  • "Given position size > 80%, what if liquidity dries up?"
  • "Given trending regime, what if sudden reversal?"

2025 Standard: Bayesian scenario trees with conditional probabilities

4. Limited Severity Levels

• Only single severity per scenario
• No stress ladder: mild → moderate → severe → extreme

2025 Standard: 4-5 severity levels per scenario with probability weights

---

2. Market Regime Stress Tests (Current: Basic, Target: Advanced)

Current Implementation ⚠️

Present but Incomplete:

• Basic regime scenarios:

  • trending_market_scenario() - +8% trend
  • whipsaw_scenario() - rapid reversals
  • vix_spike_scenario() - volatility regime

• Regime-conditional architecture exists (separate file):

  // From ml/src/dqn/regime_conditional.rs
  pub enum RegimeType {
      Trending,   // High ADX, directional moves
      Ranging,    // Low ADX, mean reversion
      Volatile,   // High entropy, unpredictable
  }
  

Critical Gaps ❌

1. No Regime Transition Stress

• Missing tests for regime shifts:
  • Trending → Volatile (breakdown)
  • Ranging → Trending (breakout)
  • Volatile → Ranging (normalization)

2025 Standard: Hidden Markov Model regime transitions with transition probabilities

2. No Regime-Specific Thresholds

• All scenarios use same thresholds regardless of regime:

  // Current: Same threshold for all regimes
  max_drawdown_threshold: 20.0,
  
  // 2025 Standard: Regime-adaptive thresholds
  // Trending: 15% drawdown acceptable
  // Ranging: 8% drawdown max
  // Volatile: 25% drawdown expected
  

3. No Multi-Regime Scenarios

• No tests combining multiple regimes:
  • Morning trending + afternoon whipsaw
  • Pre-market gap + intraday ranging
  • Volatile open + trending close

2025 Standard: Sequential regime chains with realistic durations

4. No Regime Detection Under Stress

• No validation that regime classifier remains accurate during stress
• Risk: Model may misclassify regime during crisis → wrong action selection

2025 Standard: Regime classification robustness tests (accuracy > 80% even at 3σ events)

---

3. Liquidity Stress Tests (Current: Primitive, Target: Advanced)

Current Implementation ⚠️

Basic Coverage:

• Spread widening simulation:

  liquidity_crisis_scenario() {
      spread_multiplier: 50.0,  // 50x normal spread
      price_shock_pct: -2.0,
      duration_steps: 600,
  }
  

• Circuit breaker integration (separate module):

  // From ml/src/dqn/circuit_breaker.rs
  pub struct CircuitBreaker {
      failure_threshold: usize,
      timeout_duration: Duration,
      // Halts trading during consecutive failures
  }
  

Critical Gaps ❌

1. No Market Depth Modeling

• Missing order book depth simulation:
  • No bid/ask queue sizes
  • No level 2 liquidity curves
  • No market depth degradation

2025 Standard: Full L2 order book replay with depth-dependent slippage

Example Missing:

// 2025 Standard
pub struct LiquidityStressParams {
    depth_levels: Vec<(Price, Quantity)>,  // Bid/ask ladder
    depth_decay_rate: f64,                 // How fast depth evaporates
    replenishment_rate: f64,               // How fast it recovers
    toxic_flow_intensity: f64,             // Informed trader activity
}

2. No Market Impact Modeling

• Missing dynamic market impact:
  • Temporary impact (recovers over time)
  • Permanent impact (price dislocation)
  • No Kyle's Lambda or Almgren-Chriss models

Current Risk: DQN may learn to trade large sizes without penalty

2025 Standard: Square-root market impact + transient impact decay

3. No Liquidity Recovery Modeling

• No simulation of liquidity returning after stress
• Recovery metrics incomplete:

  // Current: Simple heuristic
  let recovery_steps = if max_drawdown < 15.0 {
      Some(scenario.duration_steps / 2)
  } else {
      None
  };
  
  // 2025 Standard: Exponential recovery with half-life
  // recovery_time = f(shock_magnitude, market_regime, time_of_day)
  

4. No Fragmentation Scenarios

• Missing multi-venue liquidity tests:
  • Primary exchange vs dark pools
  • Cross-venue arbitrage during stress
  • Smart order routing under fragmentation

2025 Standard: Multi-venue liquidity aggregation with venue-specific stress

5. No Adverse Selection

• No toxic flow detection or informed trader scenarios
• DQN may learn exploitable patterns without adversarial testing

2025 Standard: Adversarial agents with information asymmetry

---

4. Correlation Breakdown Tests (Current: Minimal, Target: Comprehensive)

Current Implementation ⚠️

Nominal Coverage:

• Single scenario for correlation breakdown:

  correlation_breakdown_scenario() {
      name: "Correlation Breakdown",
      price_shock_pct: -6.0,
      volatility_multiplier: 3.5,
      spread_multiplier: 7.0,
      duration_steps: 900,
  }
  

• Multi-asset infrastructure exists (separate module):

  // From ml/src/dqn/multi_asset.rs
  pub struct MultiAssetPortfolio {
      correlation_matrix: Array2<f64>,  // NxN correlation matrix
      positions: HashMap<String, Position>,
  }
  

Critical Gaps ❌

1. No Dynamic Correlation Modeling

• Correlation matrix is static (identity matrix by default)
• No time-varying correlations (DCC-GARCH, EWMA)
• No correlation regime shifts

2025 Standard: Dynamic Conditional Correlation (DCC) with regime-dependent correlations

Example Missing:

// 2025 Standard
pub struct CorrelationStressParams {
    normal_correlation: Array2<f64>,    // Calm markets: ρ ≈ 0.3
    stress_correlation: Array2<f64>,    // Panic: ρ ≈ 0.9
    transition_speed: f64,              // How fast correlation converges
    asymmetry: bool,                    // Downside correlation > upside
}

2. No Tail Dependence Modeling

• Missing copula-based tail correlation:
  • Assets can be uncorrelated normally but correlated in tails
  • Critical for portfolio VaR estimation

Current Risk: Portfolio diversification may fail during crisis

2025 Standard: t-copulas or Archimedean copulas for tail dependence

3. No Cross-Asset Contagion

• No stress propagation across asset classes:
  • Equity crash → credit spread widening
  • FX volatility → commodity dislocation
  • Liquidity contagion across markets

2025 Standard: Multi-layer correlation networks with shock propagation

4. No Factor Model Stress

• Missing factor-based stress tests:
  • What if market beta → 1.5 (all stocks move together)?
  • What if size factor reverses?
  • What if momentum crashes?

2025 Standard: Barra-style factor risk model with factor shocks

---

5. Recovery Testing (Current: Incomplete, Target: Rigorous)

Current Implementation ⚠️

Basic Metrics:

• Recovery steps calculation:

  pub struct StressResult {
      recovery_steps: Option<usize>,  // Time to 95% recovery
      // ...
  }
  

• Simple heuristic:

  let recovery_steps = if max_drawdown < 15.0 {
      Some(scenario.duration_steps / 2)
  } else {
      None
  };
  

Critical Gaps ❌

1. No Recovery Path Validation

• Missing time-series validation:
  • Is recovery monotonic or oscillating?
  • Are there secondary drawdowns?
  • What's the recovery volatility?

2025 Standard: Full recovery trajectory analysis with confidence bands

2. No Adaptive Recovery Modeling

• Recovery time should depend on:
  • Market regime during recovery
  • Volatility levels
  • Liquidity conditions
  • Position size

Current: Fixed heuristic (duration / 2)

2025 Standard: Empirically-calibrated recovery models from historical data

3. No Scarring Effects

• Missing tests for permanent damage:
  • Reduced risk appetite after stress
  • Lower position sizes
  • Increased epsilon (more exploration)

2025 Standard: Agent behavior shift metrics (pre-stress vs post-stress)

4. No Multiple Recovery Scenarios

• Only tests single recovery path
• Missing:
  • V-shaped recovery (fast bounce)
  • U-shaped recovery (slow normalization)
  • W-shaped recovery (double dip)
  • L-shaped recovery (no recovery)

2025 Standard: Recovery pattern classification with probability distribution

5. No Compounding Stress

• What if second shock occurs during recovery?
• No tests for clustered volatility (GARCH effects)

2025 Standard: Compound stress scenarios with recovery interruption

---

6. Integration with Risk Framework

Current Capabilities ✅

Well-Integrated:

1. Circuit Breaker (ml/src/dqn/circuit_breaker.rs)

   • Halts trading after consecutive failures
   • Cooldown periods
   • Half-open testing state

2. Risk-Adjusted Rewards (referenced in code)

   • Sharpe ratio integration
   • Drawdown penalties
   • VaR-based risk metrics

3. Action Masking (risk constraints)

   • Position limits
   • Drawdown limits
   • Compliance rules

Missing Integrations ❌

1. No VaR Backtesting

• Missing Kupiec POF test (Proportion of Failures)
• No Christoffersen test (clustering of violations)
• No traffic light system (Basel III)

2025 Standard: Automated VaR model validation with regulatory tests

2. No Expected Shortfall (ES) Calculation

• VaR tells "how often" but not "how bad"
• ES = average loss beyond VaR threshold

2025 Standard: ES calculation for all scenarios (Basel III requirement)

3. No Stress VaR

• Missing stressed VaR (using crisis-period calibration)
• No incremental VaR (marginal contribution analysis)

2025 Standard: Stressed VaR + Incremental VaR for position sizing

---

7. Code Quality Assessment

Strengths ✅

1. Clean Architecture

   • Clear separation of concerns
   • Builder pattern for scenarios
   • Composable stress tests

2. Good Documentation

   • Comprehensive module docs
   • Example usage provided
   • Clear parameter explanations

3. Type Safety

   • Strong typing throughout
   • Serde serialization for results
   • Result types for error handling

4. Testability

   • Unit tests for scenario definitions
   • Validation tests for thresholds
   • Smoke tests for basic functionality

Weaknesses ❌

1. Simulated Metrics (Lines 153-170)

   // NOTE: Full DQN training integration would go here
   // For now, we simulate metrics based on scenario severity
   let severity_factor = (scenario.price_shock_pct.abs() +
                          scenario.volatility_multiplier) / 15.0;
   

   Impact: Not actually running DQN through stress scenarios

2. Hardcoded Parameters

   • Base price = 4000.0 (line 299)
   • Volatility = 0.02 (line 312)
   • Recovery heuristic = duration / 2 (line 166)

3. No Parallelization

   • Scenarios run sequentially (run_all_scenarios())
   • Could leverage Rayon for parallel execution

4. Limited Metrics

   • Missing key metrics:
     • Maximum adverse excursion (MAE)
     • Time underwater
     • Ulcer index
     • Calmar ratio
     • Recovery factor

---

8. Recommended Improvements (Prioritized)

Tier 1: Critical (Must-Have for Production) 🔴

1. Integrate Real DQN Execution (Effort: 3 days)

   // Replace simulation with actual DQN rollout
   pub fn run_scenario(&mut self, scenario: &StressScenario) -> Result<StressResult> {
       let stressed_data = self.apply_stress(scenario)?;
   
       // ACTUAL DQN evaluation on stressed data
       let mut cumulative_reward = 0.0;
       let mut max_drawdown = 0.0;
       let mut actions = Vec::new();
   
       for (i, price) in stressed_data.iter().enumerate() {
           let state = self.build_state(i, price, &stressed_data);
           let action = self.trainer.select_action(&state)?;
           let reward = self.trainer.step(action, &state)?;
   
           actions.push(action);
           cumulative_reward += reward;
           max_drawdown = max_drawdown.max(compute_drawdown(...));
       }
   
       // Compute real metrics from actual execution
   }
   

2. Add Historical Scenario Replay (Effort: 5 days)

   pub fn load_historical_scenario(
       scenario_name: &str,  // "flash_crash_2010", "covid_march_2020"
       data_path: &Path,
   ) -> Result<StressScenario> {
       // Load tick-level historical data
       // Replay exact price/volume/spread dynamics
   }
   

3. Implement Market Depth Modeling (Effort: 4 days)

   pub struct LiquidityProfile {
       bid_depth: Vec<(Price, Quantity)>,
       ask_depth: Vec<(Price, Quantity)>,
       depth_decay_halflife: Duration,
       replenishment_rate: f64,
   }
   
   pub fn apply_liquidity_stress(
       &mut self,
       profile: &LiquidityProfile,
       order_size: Quantity,
   ) -> (Price, Slippage, MarketImpact) {
       // Walk the book, compute slippage
   }
   

4. Add Expected Shortfall (Effort: 1 day)

   pub struct StressResult {
       // Existing fields...
       pub expected_shortfall: f64,  // CVaR / ES
       pub var_95: f64,
       pub var_99: f64,
       pub worst_1pct_avg: f64,
   }
   

Tier 2: Important (Needed for Robustness) 🟡

5. Dynamic Correlation Modeling (Effort: 3 days)

   pub struct DynamicCorrelationModel {
       ewma_lambda: f64,
       correlation_regime: RegimeType,
       stress_correlation_matrix: Array2<f64>,
       normal_correlation_matrix: Array2<f64>,
   }
   

6. Monte Carlo Scenario Generation (Effort: 4 days)

   pub fn generate_monte_carlo_scenarios(
       num_scenarios: usize,
       process: StochasticProcess,  // JumpDiffusion, GARCH, etc.
   ) -> Vec<StressScenario> {
       // Generate 10,000+ scenarios
   }
   

7. Recovery Path Validation (Effort: 2 days)

   pub struct RecoveryMetrics {
       recovery_time: Option<Duration>,
       recovery_pattern: RecoveryPattern,  // V, U, W, L
       secondary_drawdowns: Vec<f64>,
       recovery_volatility: f64,
   }
   

8. Regime Transition Stress (Effort: 3 days)

   pub fn regime_transition_scenario(
       from: RegimeType,
       to: RegimeType,
       transition_speed: Duration,
   ) -> StressScenario {
       // Model regime shifts
   }
   

Tier 3: Enhancement (Nice-to-Have) 🟢

9. Parallel Execution (Effort: 1 day)

   use rayon::prelude::*;
   
   pub fn run_all_scenarios_parallel(&mut self) -> Vec<StressResult> {
       self.scenarios
           .par_iter()
           .map(|scenario| self.run_scenario(scenario))
           .collect()
   }
   

10. Adversarial Agents (Effort: 5 days)

    pub struct AdversarialScenario {
        informed_trader_intensity: f64,
        front_running_probability: f64,
        spoofing_rate: f64,
    }
    

11. Multi-Venue Fragmentation (Effort: 4 days)

    pub struct VenueStressParams {
        venues: Vec<TradingVenue>,
        venue_correlations: Array2<f64>,
        cross_venue_arb_delay: Duration,
    }
    

---

9. Comparison with Industry Standards

Feature	Current Status	2025 Standard	Gap
Scenario Coverage	8 predefined	50+ historical + Monte Carlo	❌ Large
Severity Levels	1 per scenario	4-5 levels per scenario	❌ Large
Market Depth	None	Full L2 order book	❌ Critical
Market Impact	None	Kyle's Lambda + Almgren-Chriss	❌ Critical
Correlation Dynamics	Static	DCC-GARCH	❌ Large
Tail Dependence	None	Copula-based	❌ Large
Recovery Modeling	Simple heuristic	Empirical calibration	❌ Moderate
VaR Backtesting	None	Kupiec + Christoffersen	❌ Critical
Expected Shortfall	None	Full ES calculation	❌ Critical
Regime Transitions	None	HMM-based	❌ Moderate
Execution	Simulated	Real DQN rollout	❌ Critical
Parallelization	Sequential	Rayon parallel	⚠️ Minor

---

10. Risk Assessment

Production Deployment Risks 🔴

If deployed as-is, the following risks exist:

1. False Confidence (Severity: HIGH)

   • Simulated metrics may not reflect actual DQN behavior
   • Could pass stress tests in simulation but fail in production

2. Liquidity Blindness (Severity: HIGH)

   • No market impact modeling → DQN may learn unrealistic strategies
   • Large orders may cause unmodeled slippage

3. Correlation Failure (Severity: MEDIUM)

   • Static correlations → diversification may fail in crisis
   • Portfolio VaR underestimated

4. Incomplete Recovery (Severity: MEDIUM)

   • Simple recovery heuristic may miss scarring effects
   • Agent may not adapt after extreme stress

5. Regulatory Risk (Severity: HIGH for regulated firms)

   • Missing Basel III requirements (ES, Stressed VaR)
   • No VaR backtesting → cannot validate risk model

---

11. Suggested Next Steps

Phase 1: Make It Real (2 weeks)

1. Integrate actual DQN execution into stress scenarios
2. Add market depth and slippage modeling
3. Implement Expected Shortfall calculation

Phase 2: Add Realism (2 weeks)

4. Load historical crisis scenarios
5. Implement dynamic correlation modeling
6. Add regime transition stress tests

Phase 3: Robustness (1 week)

7. Implement recovery path validation
8. Add VaR backtesting (Kupiec/Christoffersen)
9. Parallelize scenario execution

Phase 4: Advanced Features (2 weeks)

10. Monte Carlo scenario generation
11. Adversarial agent testing
12. Multi-venue fragmentation modeling

---

12. Conclusion

The current DQN stress testing implementation provides a good architectural foundation with clean code and extensible design. However, it requires significant enhancements to meet 2025 institutional standards:

Key Strengths:

• ✅ Well-structured code with clear separation of concerns
• ✅ Comprehensive scenario coverage for basic stress events
• ✅ Integration with circuit breaker and risk management
• ✅ Good documentation and testability

Critical Weaknesses:

• ❌ Simulated metrics instead of real DQN execution
• ❌ No market depth or liquidity modeling
• ❌ Missing Expected Shortfall and VaR backtesting
• ❌ Static correlations without tail dependence
• ❌ Incomplete recovery modeling

Recommended Action: Prioritize Tier 1 improvements (real DQN execution, market depth, ES calculation) before considering production deployment.

Estimated Effort: 4-6 weeks of focused development to reach production-grade status.

---

Appendix A: Code Examples from Analysis

Example 1: Current Simulated Metrics (Lines 153-170)

// Apply stress scenario
let _stressed_data = self.apply_stress(scenario)?;

// Simulate stress test (NOTE: Full DQN training integration would go here)
// For now, we simulate metrics based on scenario severity
let severity_factor = (scenario.price_shock_pct.abs() + scenario.volatility_multiplier) / 15.0;

// Simulate portfolio performance under stress
let initial_portfolio = 100_000.0;
let max_drawdown = scenario.price_shock_pct.abs() * (1.0 + severity_factor * 0.5);
let final_portfolio = initial_portfolio * (1.0 + scenario.price_shock_pct / 100.0 * 0.8);

// Simulate action diversity (reduces under extreme stress)
let action_diversity = (70.0 - severity_factor * 15.0).max(20.0);

Example 2: Synthetic Stress Data Generation (Lines 294-319)

fn apply_stress(&self, scenario: &StressScenario) -> Result<Vec<f64>> {
    let mut stressed_prices = Vec::new();
    let base_price = 4000.0; // ES futures typical price

    for step in 0..scenario.duration_steps {
        let progress = step as f64 / scenario.duration_steps as f64;

        // Apply price shock (gradual over first 20% of duration)
        let shock_factor = if progress < 0.2 {
            1.0 + (scenario.price_shock_pct / 100.0) * (progress / 0.2)
        } else {
            1.0 + (scenario.price_shock_pct / 100.0)
        };

        // Add volatility (random walk scaled by volatility multiplier)
        let volatility = 0.02 * scenario.volatility_multiplier * (rand::random::<f64>() - 0.5);

        let price = base_price * shock_factor * (1.0 + volatility);
        stressed_prices.push(price);
    }

    Ok(stressed_prices)
}

Example 3: Recovery Heuristic (Line 166)

let recovery_steps = if max_drawdown < 15.0 {
    Some(scenario.duration_steps / 2)
} else {
    None
};

---

Analyst: Code Analyzer Agent Review Status: Complete Confidence Level: High (based on comprehensive codebase analysis)