foxhunt/docs/risk_reward_integration_analysis.md

Risk Management & Reward Shaping Integration Analysis

Date: 2025-11-27 Location: /home/jgrusewski/Work/foxhunt Scope: DQN Risk-Reward Integration

---

Executive Summary

This analysis examines how risk management integrates with reward shaping in the DQN system. The architecture shows sophisticated multi-layer integration with proper separation of concerns, but reveals critical signal leakage risks and unit mismatches that could cause reward overfitting to risk patterns.

Key Findings:

• ✅ Strong Architecture: EMA-based normalization, Kelly sizing, circuit breakers all properly implemented
• ⚠️ Signal Leakage Risk: Reward function directly observes risk penalties → agent may learn to game risk metrics
• ⚠️ Unit Mismatch Fixed: Hold penalty scaling bug fixed (1/1000 scale factor), but monitoring needed
• ✅ Training Stability: Percentage-based rewards + normalization prevents gradient explosion
• ⚠️ Kelly Integration: Not directly in reward function (good!), but requires 10+ trades minimum

---

1. Reward Normalization & Training Stability

Implementation: /ml/src/dqn/reward.rs (Lines 14-165)

Architecture:

pub struct RewardNormalizer {
    mean: f64,              // EMA of rewards
    variance: f64,          // EMA of squared deviations
    alpha: 0.01,           // Mean decay (100-step window)
    beta: 0.01,            // Variance decay (100-step window)
    epsilon: 1e-8,         // Numerical stability
}

// EMA update (NOT Welford's algorithm):
mean = alpha * value + (1 - alpha) * mean
variance = beta * (value - mean)^2 + (1 - beta) * variance

Why EMA vs. Welford's Algorithm?

• Welford's: Equal weight to all history → mean drift in non-stationary markets
• EMA: Exponential decay → adapts to regime changes (old samples decay as (1-α)^t)
• Window: α=0.01 → 100-step effective window (63% decay after 100 steps)

Normalization Process:

1. Calculate raw reward (percentage-based P&L)
2. Normalize using PREVIOUS stats (critical: avoids zeroing first reward)
3. Update EMA with current reward (for NEXT normalization)
4. Clip to ±3.0 std (prevents extreme outliers, preserves signal)

Expected Impact:

• Rewards centered at 0.0, scaled to std=1.0
• Q-values stabilize at: r/(1-γ) ≈ 3.0/(1-0.9626) ≈ 80
• Gradient variance reduced significantly (BUG #41 fix)

Validation:

// Line 233-252: Mandatory percentage P&L validation
if !use_percentage_pnl {
    return Err(MLError::ConfigError {
        reason: "use_percentage_pnl=false causes gradient explosion. \
                 Q-values explode to ±50,000..."
    });
}

Status: ✅ Robust & Well-Documented

---

2. Risk Penalties in Reward Function

Implementation: /ml/src/dqn/reward.rs (Lines 478-573)

Reward Components (Weighted Sum):

final_reward = pnl_weight * pnl_reward          // P&L (%)
             - risk_weight * risk_penalty       // Position size penalty
             - cost_weight * cost_penalty       // Transaction costs
             + diversity_bonus                  // Entropy regularization

2.1 P&L Reward (Lines 640-712)

Formula:

pnl_reward = (next_value - current_value) / current_value  // Percentage return

Range: ±0.02 (±2% typical per step) Scale: Already in percentage units → no 100x amplification Status: ✅ Properly scaled for C51 distributional RL

2.2 Risk Penalty (Lines 713-741)

Formula:

position_size = portfolio_features[1]  // Signed: +Long, -Short, 0=Flat
risk_penalty = if position_size > 0.8 {
    (position_size - 0.8) * 5.0  // Escalating penalty for >80% exposure
} else { 0.0 }

Issues:

1. Hardcoded threshold (0.8): Not tied to portfolio volatility or market regime
2. Linear escalation: 5x multiplier may be too aggressive or too weak depending on scale
3. No drawdown integration: Doesn't use actual drawdown data from RiskMetrics

Signal Leakage Risk:

• Reward directly observes position size → agent learns to hover at 79% to avoid penalty
• Recommendation: Use Kelly-suggested position sizes as reference, not absolute thresholds

2.3 Transaction Cost Penalty (Lines 743-835)

Formula (FIXED):

// BUG FIX: Use action.transaction_cost() instead of spread estimation
tx_cost_rate = action.transaction_cost()  // Market: 0.15%, Limit: 0.05%, IoC: 0.10%
cost_penalty = position_change * tx_cost_rate

Before Fix: Spread × 0.5 ≈ 0.05% (3x underestimated market orders) After Fix: Actual exchange fees (0.05-0.15%) Status: ✅ Fixed in BUG #24

2.4 Hold Reward / Penalty (Lines 866-907)

Formula (FIXED):

// BUG FIX: Scale hold_penalty_weight to percentage units
hold_penalty_pct = hold_penalty_weight / 1000.0  // 0.5-2.0 → 0.0005-0.002 (0.05-0.2%)

volatility = |next_log_return|  // Absolute log return as price velocity
hold_reward = if volatility < movement_threshold {
    +0.001  // Low volatility: reward holding
} else {
    -hold_penalty_pct  // High volatility: penalize holding
}

Before Fix: 0.5-2.0 scalar (333-4000x mismatch with tx costs) After Fix: 0.0005-0.002 (5-20 basis points, aligned with 0.05-0.15% tx costs) Ratio: hold_penalty / tx_cost ≈ 0.001 / 0.001 = 1.0 (balanced) Status: ✅ Fixed in Phase 1 scaling adjustment

2.5 Diversity Penalty (Lines 344-386, 517-526)

Formula:

entropy = -Σ(p_i * log2(p_i))  // Shannon entropy of last 100 actions
diversity_bonus = if entropy < 0.5 {
    -0.1  // Penalty for low diversity (< 50% of max entropy)
} else { 0.0 }

Range: Entropy ∈ [0, 1.585] (log2(3) for 3 actions) Threshold: 0.5 (31.5% of max) triggers penalty Weight: -0.1 (100x stronger than hold_reward) Purpose: Prevent action collapse (e.g., 98% SELL bug) Status: ✅ Effective in-training regularization

---

3. Kelly Sizing Integration

Implementation: /risk/src/kelly_sizing.rs (Lines 1-342)

Kelly Criterion Formula:

// f* = (bp - q) / b
// where:
//   b = odds ratio (average_win / average_loss)
//   p = win_rate
//   q = loss_rate (1 - p)
//
// Example:
//   win_rate = 0.6, avg_win = $50, avg_loss = $30
//   b = 50/30 = 1.667
//   f* = (1.667 * 0.6 - 0.4) / 1.667 = 0.36 (36% of capital)

Adjustments:

1. Fractional Kelly: Multiply by 0.25-0.5 (reduces risk, prevents over-betting)
2. Confidence Scaling: Based on sample size + win rate distribution
3. Hard Caps: Min 1%, Max 10% (prevents extreme bets)

Integration Point:

• NOT in reward function (good! avoids circular dependency)
• Used in position sizing after action selection
• Requires minimum 10 trades to calculate (lines 140-149)

Critical Issue:

// Line 140-149: Insufficient data handling
if trades.len() < 10 {
    return Err(RiskError::DataUnavailable {
        reason: format!(
            "Insufficient trade history: {} trades (minimum 10 required)",
            trades.len()
        ),
    });
}

Problem: Early training (< 10 trades) will FAIL with error, not default sizing Recommendation: Use conservative default (e.g., 2% position size) until 10+ trades accumulated

Status: ✅ Kelly sizing isolated from rewards, but needs warmup handling

---

4. Circuit Breaker Integration

4.1 Simplified Circuit Breaker: /ml/src/dqn/circuit_breaker.rs

Purpose: Lightweight training throttling (no Redis, no broker) States: Closed → Open → HalfOpen → Closed

Thresholds:

• Failure threshold: 5 consecutive losses → OPEN
• Success threshold: 3 consecutive wins → CLOSED (from HalfOpen)
• Timeout: 60 seconds cooldown
• Half-open calls: 2 test trades allowed

Trigger Logic:

// Open circuit after 5 consecutive failures
if consecutive_failures >= 5 {
    state = Open;  // Block all requests
    open_timestamp = now();
}

// After 60s timeout, transition to HalfOpen
if elapsed >= 60s {
    state = HalfOpen;  // Allow 2 test trades
}

// Close after 3 consecutive successes in HalfOpen
if consecutive_successes >= 3 {
    state = Closed;  // Resume normal trading
}

Integration with Reward:

• Circuit breaker does NOT affect reward calculation
• Only blocks action execution (prevents runaway losses)
• Proper separation of concerns

Status: ✅ Well-isolated from reward function

4.2 Risk Crate Circuit Breaker: /ml/src/dqn/risk_integration.rs

Purpose: Full production-grade circuit breaker with Redis coordination Components:

1. TrainingBrokerService: Tracks portfolio value + daily P&L
2. DQNRiskCircuitBreaker: Wraps risk::RealCircuitBreaker
3. Redis Coordination: Multi-process safe (not needed for single-process training)

Configuration:

CircuitBreakerConfig {
    daily_loss_percentage: 10.0%,           // $10K on $100K capital
    position_limit_percentage: 5.0%,        // Max 5% portfolio per position
    max_consecutive_violations: 5,
    auto_recovery_enabled: false,           // Manual recovery for safety
    cooldown_period_secs: 300,              // 5 minutes
}

Reward Integration:

// Line 222-238: Record reward and check circuit breaker
pub async fn record_reward(&self, reward: f64) -> RiskResult<()> {
    self.broker_service.record_reward(reward).await;

    // Check circuit breaker ONLY on losses
    if reward < 0.0 {
        let is_open = self.check().await?;
        if is_open {
            error!("Circuit breaker TRIGGERED");
            error!("Daily Loss: ${:.2}", state.current_daily_loss);
        }
    }
    Ok(())
}

Status: ✅ Properly isolated, but requires Redis (may not be needed for training)

---

5. Signal Leakage Analysis

5.1 What is Signal Leakage?

Definition: Agent learns to game risk metrics instead of learning profitable trading strategies.

Example:

• Agent observes: risk_penalty = 0 when position_size < 0.8
• Agent learns: "Stay at 79% position to maximize reward"
• Problem: Agent avoids legitimate risk management instead of learning optimal position sizing

5.2 Current Leakage Points

Component	Leakage Risk	Mechanism	Severity
Position Size Penalty	HIGH	Hardcoded 0.8 threshold → agent learns to hover at 79%	🔴 Critical
Transaction Costs	LOW	Agent learns to minimize trading (good!)	🟢 Benign
Hold Penalty	MEDIUM	Agent may learn to avoid holding during high volatility even when correct	🟡 Moderate
Diversity Penalty	LOW	Entropy threshold only prevents collapse, doesn't leak strategy	🟢 Benign
Kelly Sizing	NONE	Not in reward function → no leakage	🟢 Safe
Circuit Breaker	NONE	Only blocks execution, doesn't affect rewards	🟢 Safe

5.3 Detected Leakage Example

Hypothesis: Agent learns to maintain 79% position to avoid risk penalty

Evidence to Check:

1. Distribution of position sizes in trained agent (should see spike at 79%)
2. Correlation between position size and reward (should be independent after removing P&L)
3. Agent behavior when position_size = 0.81 (should immediately reduce to 0.79)

Recommendation:

// Instead of hardcoded threshold, use Kelly-suggested position
let kelly_position = kelly_sizer.calculate_kelly_fraction(symbol, strategy)?;
let risk_penalty = if position_size > kelly_position * 1.2 {
    (position_size - kelly_position) * adaptive_multiplier
} else { 0.0 };

This makes the penalty dynamic and tied to actual risk analysis, not arbitrary thresholds.

---

6. Integration Point Summary

6.1 Reward Function Architecture

┌─────────────────────────────────────────────────────┐
│         DQN Reward Function (reward.rs)             │
├─────────────────────────────────────────────────────┤
│                                                     │
│  Base Reward = pnl_weight * pnl_reward            │
│              - risk_weight * risk_penalty         │  ← LEAKAGE RISK
│              - cost_weight * cost_penalty         │  ← Safe (realistic)
│              + hold_reward / -hold_penalty        │  ← Scaled correctly
│                                                     │
│  Final Reward = Base + diversity_bonus            │
│                                                     │
│  Normalized Reward = EMA_normalize(Final)         │  ← Stability layer
│                      .clamp(-3.0, 3.0)            │
│                                                     │
└─────────────────────────────────────────────────────┘
                        ↓
┌─────────────────────────────────────────────────────┐
│   Risk Management (Separate Layer)                  │
├─────────────────────────────────────────────────────┤
│                                                     │
│  Kelly Sizing (kelly_sizing.rs)                   │  ← NOT in rewards
│    - Position size optimization                    │
│    - Requires 10+ trade history                    │
│    - Fractional Kelly (0.25-0.5)                  │
│                                                     │
│  Circuit Breakers (circuit_breaker.rs)            │  ← Execution layer
│    - Simplified: 5 failures → OPEN                │
│    - Risk Crate: Daily loss % → OPEN              │
│    - Does NOT modify rewards                       │
│                                                     │
└─────────────────────────────────────────────────────┘

6.2 Data Flow

Training Step:
1. Select action (DQN policy network)
2. Calculate raw reward (P&L + penalties)
3. Normalize reward (EMA + clipping)
4. Update Q-network (TD error)
5. [Separate] Kelly sizing for next position
6. [Separate] Circuit breaker check

Execution Step:
1. Select action (trained policy)
2. [Pre-check] Circuit breaker allow?
3. [Pre-check] Kelly position size
4. Execute trade
5. [Post-check] Record outcome for Kelly
6. [Post-check] Update circuit breaker

Key Observation: Kelly and circuit breakers are post-decision, not pre-decision factors. They don't directly influence reward calculation during training.

---

7. Potential Issues & Recommendations

Issue 1: Position Size Penalty (Hardcoded Threshold)

Severity: 🔴 High Location: /ml/src/dqn/reward.rs:718-741

Problem:

let risk_penalty = if position_size > 0.8 {
    (position_size - 0.8) * 5.0  // Hardcoded 0.8 threshold
} else { 0.0 }

Signal Leakage: Agent learns to maintain 79% position regardless of market conditions.

Recommendation:

// Use Kelly-suggested position as dynamic threshold
let kelly_fraction = self.kelly_sizer.calculate_kelly_fraction(symbol, strategy)?;
let safe_threshold = kelly_fraction.adjusted_kelly_fraction * 1.2; // 20% buffer

let risk_penalty = if position_size > safe_threshold {
    (position_size - safe_threshold).powi(2) * adaptive_weight  // Quadratic penalty
} else { 0.0 }

Issue 2: Drawdown Not Used in Penalties

Severity: 🟡 Medium Location: /ml/src/dqn/reward.rs:319-329

Problem: RiskMetrics struct includes max_drawdown, but it's never used in risk penalty calculation.

Current Usage:

pub struct RiskMetrics {
    pub var_95: Decimal,         // ❌ Not used
    pub max_drawdown: Decimal,   // ❌ Not used
    pub sharpe_ratio: Decimal,   // ❌ Not used
    pub volatility: Decimal,     // ❌ Not used
}

Recommendation:

// Add drawdown penalty component
let drawdown_penalty = if max_drawdown > threshold {
    (max_drawdown - threshold) * drawdown_weight
} else { 0.0 };

final_reward = pnl_reward - risk_penalty - cost_penalty - drawdown_penalty;

Issue 3: Kelly Sizing Warmup

Severity: 🟡 Medium Location: /risk/src/kelly_sizing.rs:140-149

Problem: Training fails with error when < 10 trades recorded.

Current Behavior:

if trades.len() < 10 {
    return Err(RiskError::DataUnavailable { ... });
}

Recommendation:

if trades.len() < 10 {
    // Use conservative default during warmup
    return Ok(KellyResult {
        position_fraction: 0.02,  // 2% default
        use_kelly: false,
        confidence: 0.0,
        sample_size: trades.len(),
        ...
    });
}

Issue 4: EMA Normalization Cold Start

Severity: 🟢 Low (already handled correctly) Location: /ml/src/dqn/reward.rs:548-552

Correct Implementation:

// CRITICAL FIX: Normalize BEFORE update to avoid zeroing first reward
let norm = normalizer.normalize(final_reward_f64);  // Use PREVIOUS stats
normalizer.update(final_reward_f64);                // Update for NEXT call

Status: ✅ Fixed (proper order prevents first-reward zeroing)

Issue 5: Hold Penalty Unit Mismatch

Severity: 🟢 Low (fixed, but monitor) Location: /ml/src/dqn/reward.rs:881-887

Fixed Implementation:

let hold_penalty_scale = Decimal::try_from(1000.0).unwrap_or(Decimal::ONE);
let hold_penalty_pct = self.config.hold_penalty_weight / hold_penalty_scale;
// Result: 0.5-2.0 → 0.0005-0.002 (5-20 basis points)

Ratio Check:

• Hold penalty: 0.0005-0.002 (0.05-0.2%)
• Transaction costs: 0.0005-0.0015 (0.05-0.15%)
• Ratio: 0.33-1.33 (balanced)

Status: ✅ Fixed in Phase 1, monitor for over/under-trading patterns

---

8. Testing & Validation Recommendations

8.1 Unit Tests (Add These)

Test 1: Position Size Leakage Detection

#[test]
fn test_no_position_size_gaming() {
    let mut reward_fn = RewardFunction::new(RewardConfig::default());

    // Test position sizes around threshold
    let positions = vec![0.75, 0.79, 0.80, 0.81, 0.85];
    let rewards: Vec<_> = positions.iter().map(|&pos| {
        let state = create_state_with_position(pos);
        reward_fn.calculate_reward(action, &state, &next_state, &[])
    }).collect();

    // Check for discontinuity at 0.8 threshold
    assert!(
        (rewards[2] - rewards[1]).abs() < 0.1,
        "Reward should be smooth across threshold, not create gaming incentive"
    );
}

Test 2: Kelly Warmup Handling

#[tokio::test]
async fn test_kelly_warmup_does_not_fail() {
    let sizer = KellySizer::new(KellyConfig::default());

    // Should return default sizing, not error
    let result = sizer.calculate_kelly_fraction(&Symbol::from("AAPL"), "strategy");
    assert!(result.is_ok());

    let kelly = result.unwrap();
    assert_eq!(kelly.position_fraction, 0.02); // Conservative default
    assert!(!kelly.use_kelly);
}

Test 3: Hold Penalty Scaling

#[test]
fn test_hold_penalty_magnitude() {
    let config = RewardConfig {
        hold_penalty_weight: Decimal::try_from(1.0).unwrap(),
        ..Default::default()
    };
    let reward_fn = RewardFunction::new(config);

    // High volatility state
    let volatile_state = create_state_with_log_return(0.05); // 5% move
    let hold_reward = reward_fn.calculate_hold_reward(&state, &volatile_state)?;

    // Check penalty is in basis points (not dollars)
    assert!(
        hold_reward.abs() < Decimal::try_from(0.01).unwrap(),
        "Hold penalty should be in percentage units (< 1%)"
    );
}

8.2 Integration Tests (Add These)

Test 4: End-to-End Reward + Risk Flow

#[tokio::test]
async fn test_reward_risk_integration() {
    let mut reward_fn = RewardFunction::new(RewardConfig::default());
    let kelly_sizer = KellySizer::new(KellyConfig::default());
    let circuit_breaker = CircuitBreaker::new(CircuitBreakerConfig::default());

    // Simulate 50 trading steps
    for step in 0..50 {
        // 1. Calculate reward
        let reward = reward_fn.calculate_reward(action, &state, &next_state, &actions)?;

        // 2. Record for Kelly (post-decision)
        if step >= 10 {
            kelly_sizer.add_trade_outcome(TradeOutcome { ... });
        }

        // 3. Check circuit breaker (post-decision)
        if reward < 0.0 {
            circuit_breaker.record_failure();
            assert!(circuit_breaker.allow_request()); // Should not block training
        }
    }

    // Verify Kelly sizing became available after warmup
    let kelly_result = kelly_sizer.calculate_kelly_fraction(&symbol, "strategy");
    assert!(kelly_result.is_ok());
    assert!(kelly_result.unwrap().sample_size >= 10);
}

8.3 Monitoring Metrics (Add These)

Metric 1: Position Size Distribution

// Track histogram of position sizes during training
// Alert if spike at 0.79 (indicates gaming)
let position_histogram = agent.get_position_histogram();
assert!(
    position_histogram[79] < 0.2,  // < 20% of actions at 79%
    "Agent may be gaming position size threshold"
);

Metric 2: Reward Component Variance

// Track variance of each reward component
let component_stats = reward_fn.get_component_stats();
assert!(
    component_stats.risk_penalty_variance < component_stats.pnl_variance,
    "Risk penalty should not dominate reward signal"
);

Metric 3: Kelly vs. Actual Position Divergence

// Compare Kelly-suggested vs. actual positions
let kelly_fraction = kelly_sizer.calculate_kelly_fraction(&symbol, "strategy")?;
let actual_position = agent.current_position_size();

let divergence = (actual_position - kelly_fraction).abs();
assert!(
    divergence < 0.2,
    "Agent should approximately follow Kelly sizing, divergence: {:.2}%",
    divergence * 100.0
);

---

9. Conclusion

Strengths ✅

1. EMA-based normalization prevents mean drift in non-stationary markets
2. Percentage-based P&L achieves scale-invariance (BUG #17 fix)
3. Transaction cost fix uses actual exchange fees (BUG #24 fix)
4. Hold penalty scaling fix aligns units with transaction costs (Phase 1 fix)
5. Kelly sizing isolated from reward function (no circular dependency)
6. Circuit breakers separated from reward calculation (proper layering)

Weaknesses ⚠️

1. Position size penalty uses hardcoded threshold → signal leakage risk
2. Drawdown not used in penalties despite being in RiskMetrics struct
3. Kelly sizing requires 10+ trades → early training failures
4. No validation tests for leakage detection or component interaction

Critical Recommendations 🔴

1. Replace hardcoded 0.8 threshold with Kelly-based dynamic thresholds
2. Add drawdown penalty to reward function (use RiskMetrics.max_drawdown)
3. Implement Kelly warmup (use 2% default for first 10 trades)
4. Add leakage detection tests (position size distribution, reward component variance)
5. Monitor training metrics for position gaming and reward component dominance

Expected Impact

• Training Stability: Current EMA + percentage P&L is robust ✅
• Risk Management: Kelly + circuit breakers properly isolated ✅
• Reward Overfitting: Moderate risk due to hardcoded thresholds ⚠️
• Signal Leakage: High risk if agent learns to game position size threshold 🔴

---

File Locations

Component	File Path	Lines
Reward Normalization	/ml/src/dqn/reward.rs	14-165
Reward Function	/ml/src/dqn/reward.rs	389-941
Risk Penalties	/ml/src/dqn/reward.rs	713-907
Kelly Sizing	/risk/src/kelly_sizing.rs	1-342
Circuit Breaker (Simple)	/ml/src/dqn/circuit_breaker.rs	1-435
Risk Integration	/ml/src/dqn/risk_integration.rs	1-355
Reward Coordinator	/ml/src/dqn/reward_coordinator.rs	1-569

---

Appendix: Reward Formula Breakdown

Complete Reward Calculation:

// Step 1: Calculate base components
pnl_reward = (next_value - current_value) / current_value  // ±0.02 typical
risk_penalty = if position_size > 0.8 { (position_size - 0.8) * 5.0 } else { 0.0 }
cost_penalty = position_change * tx_cost_rate  // 0.05-0.15%
hold_reward = if volatility < threshold { +0.001 } else { -hold_penalty_pct }
diversity_bonus = if entropy < 0.5 { -0.1 } else { 0.0 }

// Step 2: Weighted sum
base_reward = pnl_weight * pnl_reward          // Default: 1.0
            - risk_weight * risk_penalty       // Default: 0.1
            - cost_weight * cost_penalty       // Default: 1.0
            + hold_reward                      // Conditional ±0.001

final_reward = base_reward + diversity_bonus

// Step 3: Normalize and clip
normalized = (final_reward - mean) / std
clipped = normalized.clamp(-3.0, 3.0)

// Step 4: Update EMA for next step
mean = 0.01 * final_reward + 0.99 * mean
variance = 0.01 * (final_reward - mean)^2 + 0.99 * variance

Expected Ranges:

• Raw reward: -0.05 to +0.05 (±5% moves rare)
• Normalized reward: -3.0 to +3.0 (clipped)
• Q-values: -80 to +80 (r/(1-γ) with γ=0.9626)
• TD errors: ±10 typical (after normalization)

---

End of Analysis