2df1ea92e1
BREAKING CHANGES: - Removed orphaned dqn.rs monolithic trainer (4,975 lines) - Removed orphaned dqn_ensemble.rs module (816 lines) - Removed orphaned tft.rs and tft_complete_int8_integration_test.rs - TFT trainer split into modular directory structure DQN Module Refactoring: - Split trainers/dqn.rs into modular structure (config.rs, statistics.rs, trainer.rs) - Fixed hyperopt 39D search space (continuous params only) - Boolean flags (use_dueling, use_double_dqn, use_per, use_noisy_nets) are now FIXED architectural decisions - use_distributional defaults to false (Candle BUG #36 - scatter_add gradient issues) Clean Module Structure: - ml/src/trainers/dqn/ directory with proper mod.rs exports - ml/src/trainers/tft/ directory with config.rs, types.rs, model.rs, trainer.rs, tests.rs - All P0 features validated: TD-error clamping, batch diversity, LR scheduler, priority staleness Documentation: - Added comprehensive docs in docs/codebase-cleanup/ - ADR-001 for DQN refactoring decisions - Rainbow DQN component matrix and quick reference guides Build Status: Compiles with zero errors 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
723 lines
25 KiB
Markdown
723 lines
25 KiB
Markdown
# Risk Management & Reward Shaping Integration Analysis
|
||
**Date**: 2025-11-27
|
||
**Location**: `/home/jgrusewski/Work/foxhunt`
|
||
**Scope**: DQN Risk-Reward Integration
|
||
|
||
---
|
||
|
||
## Executive Summary
|
||
|
||
This analysis examines how risk management integrates with reward shaping in the DQN system. The architecture shows **sophisticated multi-layer integration** with proper separation of concerns, but reveals **critical signal leakage risks** and **unit mismatches** that could cause reward overfitting to risk patterns.
|
||
|
||
**Key Findings:**
|
||
- ✅ **Strong Architecture**: EMA-based normalization, Kelly sizing, circuit breakers all properly implemented
|
||
- ⚠️ **Signal Leakage Risk**: Reward function directly observes risk penalties → agent may learn to game risk metrics
|
||
- ⚠️ **Unit Mismatch Fixed**: Hold penalty scaling bug fixed (1/1000 scale factor), but monitoring needed
|
||
- ✅ **Training Stability**: Percentage-based rewards + normalization prevents gradient explosion
|
||
- ⚠️ **Kelly Integration**: Not directly in reward function (good!), but requires 10+ trades minimum
|
||
|
||
---
|
||
|
||
## 1. Reward Normalization & Training Stability
|
||
|
||
### Implementation: `/ml/src/dqn/reward.rs` (Lines 14-165)
|
||
|
||
**Architecture:**
|
||
```rust
|
||
pub struct RewardNormalizer {
|
||
mean: f64, // EMA of rewards
|
||
variance: f64, // EMA of squared deviations
|
||
alpha: 0.01, // Mean decay (100-step window)
|
||
beta: 0.01, // Variance decay (100-step window)
|
||
epsilon: 1e-8, // Numerical stability
|
||
}
|
||
|
||
// EMA update (NOT Welford's algorithm):
|
||
mean = alpha * value + (1 - alpha) * mean
|
||
variance = beta * (value - mean)^2 + (1 - beta) * variance
|
||
```
|
||
|
||
**Why EMA vs. Welford's Algorithm?**
|
||
- **Welford's**: Equal weight to all history → mean drift in non-stationary markets
|
||
- **EMA**: Exponential decay → adapts to regime changes (old samples decay as (1-α)^t)
|
||
- **Window**: α=0.01 → 100-step effective window (63% decay after 100 steps)
|
||
|
||
**Normalization Process:**
|
||
1. **Calculate raw reward** (percentage-based P&L)
|
||
2. **Normalize using PREVIOUS stats** (critical: avoids zeroing first reward)
|
||
3. **Update EMA** with current reward (for NEXT normalization)
|
||
4. **Clip to ±3.0 std** (prevents extreme outliers, preserves signal)
|
||
|
||
**Expected Impact:**
|
||
- Rewards centered at 0.0, scaled to std=1.0
|
||
- Q-values stabilize at: `r/(1-γ) ≈ 3.0/(1-0.9626) ≈ 80`
|
||
- Gradient variance reduced significantly (BUG #41 fix)
|
||
|
||
**Validation:**
|
||
```rust
|
||
// Line 233-252: Mandatory percentage P&L validation
|
||
if !use_percentage_pnl {
|
||
return Err(MLError::ConfigError {
|
||
reason: "use_percentage_pnl=false causes gradient explosion. \
|
||
Q-values explode to ±50,000..."
|
||
});
|
||
}
|
||
```
|
||
|
||
**Status**: ✅ **Robust & Well-Documented**
|
||
|
||
---
|
||
|
||
## 2. Risk Penalties in Reward Function
|
||
|
||
### Implementation: `/ml/src/dqn/reward.rs` (Lines 478-573)
|
||
|
||
**Reward Components (Weighted Sum):**
|
||
```rust
|
||
final_reward = pnl_weight * pnl_reward // P&L (%)
|
||
- risk_weight * risk_penalty // Position size penalty
|
||
- cost_weight * cost_penalty // Transaction costs
|
||
+ diversity_bonus // Entropy regularization
|
||
```
|
||
|
||
### 2.1 P&L Reward (Lines 640-712)
|
||
**Formula:**
|
||
```rust
|
||
pnl_reward = (next_value - current_value) / current_value // Percentage return
|
||
```
|
||
|
||
**Range**: ±0.02 (±2% typical per step)
|
||
**Scale**: Already in percentage units → no 100x amplification
|
||
**Status**: ✅ **Properly scaled for C51 distributional RL**
|
||
|
||
### 2.2 Risk Penalty (Lines 713-741)
|
||
**Formula:**
|
||
```rust
|
||
position_size = portfolio_features[1] // Signed: +Long, -Short, 0=Flat
|
||
risk_penalty = if position_size > 0.8 {
|
||
(position_size - 0.8) * 5.0 // Escalating penalty for >80% exposure
|
||
} else { 0.0 }
|
||
```
|
||
|
||
**Issues:**
|
||
1. **Hardcoded threshold (0.8)**: Not tied to portfolio volatility or market regime
|
||
2. **Linear escalation**: 5x multiplier may be too aggressive or too weak depending on scale
|
||
3. **No drawdown integration**: Doesn't use actual drawdown data from `RiskMetrics`
|
||
|
||
**Signal Leakage Risk:**
|
||
- Reward directly observes position size → agent learns to hover at 79% to avoid penalty
|
||
- **Recommendation**: Use Kelly-suggested position sizes as reference, not absolute thresholds
|
||
|
||
### 2.3 Transaction Cost Penalty (Lines 743-835)
|
||
**Formula (FIXED):**
|
||
```rust
|
||
// BUG FIX: Use action.transaction_cost() instead of spread estimation
|
||
tx_cost_rate = action.transaction_cost() // Market: 0.15%, Limit: 0.05%, IoC: 0.10%
|
||
cost_penalty = position_change * tx_cost_rate
|
||
```
|
||
|
||
**Before Fix**: Spread × 0.5 ≈ 0.05% (3x underestimated market orders)
|
||
**After Fix**: Actual exchange fees (0.05-0.15%)
|
||
**Status**: ✅ **Fixed in BUG #24**
|
||
|
||
### 2.4 Hold Reward / Penalty (Lines 866-907)
|
||
**Formula (FIXED):**
|
||
```rust
|
||
// BUG FIX: Scale hold_penalty_weight to percentage units
|
||
hold_penalty_pct = hold_penalty_weight / 1000.0 // 0.5-2.0 → 0.0005-0.002 (0.05-0.2%)
|
||
|
||
volatility = |next_log_return| // Absolute log return as price velocity
|
||
hold_reward = if volatility < movement_threshold {
|
||
+0.001 // Low volatility: reward holding
|
||
} else {
|
||
-hold_penalty_pct // High volatility: penalize holding
|
||
}
|
||
```
|
||
|
||
**Before Fix**: 0.5-2.0 scalar (333-4000x mismatch with tx costs)
|
||
**After Fix**: 0.0005-0.002 (5-20 basis points, aligned with 0.05-0.15% tx costs)
|
||
**Ratio**: `hold_penalty / tx_cost ≈ 0.001 / 0.001 = 1.0` (balanced)
|
||
**Status**: ✅ **Fixed in Phase 1 scaling adjustment**
|
||
|
||
### 2.5 Diversity Penalty (Lines 344-386, 517-526)
|
||
**Formula:**
|
||
```rust
|
||
entropy = -Σ(p_i * log2(p_i)) // Shannon entropy of last 100 actions
|
||
diversity_bonus = if entropy < 0.5 {
|
||
-0.1 // Penalty for low diversity (< 50% of max entropy)
|
||
} else { 0.0 }
|
||
```
|
||
|
||
**Range**: Entropy ∈ [0, 1.585] (log2(3) for 3 actions)
|
||
**Threshold**: 0.5 (31.5% of max) triggers penalty
|
||
**Weight**: -0.1 (100x stronger than hold_reward)
|
||
**Purpose**: Prevent action collapse (e.g., 98% SELL bug)
|
||
**Status**: ✅ **Effective in-training regularization**
|
||
|
||
---
|
||
|
||
## 3. Kelly Sizing Integration
|
||
|
||
### Implementation: `/risk/src/kelly_sizing.rs` (Lines 1-342)
|
||
|
||
**Kelly Criterion Formula:**
|
||
```rust
|
||
// f* = (bp - q) / b
|
||
// where:
|
||
// b = odds ratio (average_win / average_loss)
|
||
// p = win_rate
|
||
// q = loss_rate (1 - p)
|
||
//
|
||
// Example:
|
||
// win_rate = 0.6, avg_win = $50, avg_loss = $30
|
||
// b = 50/30 = 1.667
|
||
// f* = (1.667 * 0.6 - 0.4) / 1.667 = 0.36 (36% of capital)
|
||
```
|
||
|
||
**Adjustments:**
|
||
1. **Fractional Kelly**: Multiply by 0.25-0.5 (reduces risk, prevents over-betting)
|
||
2. **Confidence Scaling**: Based on sample size + win rate distribution
|
||
3. **Hard Caps**: Min 1%, Max 10% (prevents extreme bets)
|
||
|
||
**Integration Point:**
|
||
- **NOT in reward function** (good! avoids circular dependency)
|
||
- Used in **position sizing** after action selection
|
||
- Requires **minimum 10 trades** to calculate (lines 140-149)
|
||
|
||
**Critical Issue:**
|
||
```rust
|
||
// Line 140-149: Insufficient data handling
|
||
if trades.len() < 10 {
|
||
return Err(RiskError::DataUnavailable {
|
||
reason: format!(
|
||
"Insufficient trade history: {} trades (minimum 10 required)",
|
||
trades.len()
|
||
),
|
||
});
|
||
}
|
||
```
|
||
|
||
**Problem**: Early training (< 10 trades) will FAIL with error, not default sizing
|
||
**Recommendation**: Use conservative default (e.g., 2% position size) until 10+ trades accumulated
|
||
|
||
**Status**: ✅ **Kelly sizing isolated from rewards, but needs warmup handling**
|
||
|
||
---
|
||
|
||
## 4. Circuit Breaker Integration
|
||
|
||
### 4.1 Simplified Circuit Breaker: `/ml/src/dqn/circuit_breaker.rs`
|
||
|
||
**Purpose**: Lightweight training throttling (no Redis, no broker)
|
||
**States**: Closed → Open → HalfOpen → Closed
|
||
|
||
**Thresholds:**
|
||
- **Failure threshold**: 5 consecutive losses → OPEN
|
||
- **Success threshold**: 3 consecutive wins → CLOSED (from HalfOpen)
|
||
- **Timeout**: 60 seconds cooldown
|
||
- **Half-open calls**: 2 test trades allowed
|
||
|
||
**Trigger Logic:**
|
||
```rust
|
||
// Open circuit after 5 consecutive failures
|
||
if consecutive_failures >= 5 {
|
||
state = Open; // Block all requests
|
||
open_timestamp = now();
|
||
}
|
||
|
||
// After 60s timeout, transition to HalfOpen
|
||
if elapsed >= 60s {
|
||
state = HalfOpen; // Allow 2 test trades
|
||
}
|
||
|
||
// Close after 3 consecutive successes in HalfOpen
|
||
if consecutive_successes >= 3 {
|
||
state = Closed; // Resume normal trading
|
||
}
|
||
```
|
||
|
||
**Integration with Reward:**
|
||
- Circuit breaker does **NOT affect reward calculation**
|
||
- Only blocks action execution (prevents runaway losses)
|
||
- Proper separation of concerns
|
||
|
||
**Status**: ✅ **Well-isolated from reward function**
|
||
|
||
### 4.2 Risk Crate Circuit Breaker: `/ml/src/dqn/risk_integration.rs`
|
||
|
||
**Purpose**: Full production-grade circuit breaker with Redis coordination
|
||
**Components:**
|
||
1. **TrainingBrokerService**: Tracks portfolio value + daily P&L
|
||
2. **DQNRiskCircuitBreaker**: Wraps `risk::RealCircuitBreaker`
|
||
3. **Redis Coordination**: Multi-process safe (not needed for single-process training)
|
||
|
||
**Configuration:**
|
||
```rust
|
||
CircuitBreakerConfig {
|
||
daily_loss_percentage: 10.0%, // $10K on $100K capital
|
||
position_limit_percentage: 5.0%, // Max 5% portfolio per position
|
||
max_consecutive_violations: 5,
|
||
auto_recovery_enabled: false, // Manual recovery for safety
|
||
cooldown_period_secs: 300, // 5 minutes
|
||
}
|
||
```
|
||
|
||
**Reward Integration:**
|
||
```rust
|
||
// Line 222-238: Record reward and check circuit breaker
|
||
pub async fn record_reward(&self, reward: f64) -> RiskResult<()> {
|
||
self.broker_service.record_reward(reward).await;
|
||
|
||
// Check circuit breaker ONLY on losses
|
||
if reward < 0.0 {
|
||
let is_open = self.check().await?;
|
||
if is_open {
|
||
error!("Circuit breaker TRIGGERED");
|
||
error!("Daily Loss: ${:.2}", state.current_daily_loss);
|
||
}
|
||
}
|
||
Ok(())
|
||
}
|
||
```
|
||
|
||
**Status**: ✅ **Properly isolated, but requires Redis (may not be needed for training)**
|
||
|
||
---
|
||
|
||
## 5. Signal Leakage Analysis
|
||
|
||
### 5.1 What is Signal Leakage?
|
||
|
||
**Definition**: Agent learns to game risk metrics instead of learning profitable trading strategies.
|
||
|
||
**Example:**
|
||
- Agent observes: `risk_penalty = 0 when position_size < 0.8`
|
||
- Agent learns: "Stay at 79% position to maximize reward"
|
||
- **Problem**: Agent avoids legitimate risk management instead of learning optimal position sizing
|
||
|
||
### 5.2 Current Leakage Points
|
||
|
||
| Component | Leakage Risk | Mechanism | Severity |
|
||
|-----------|-------------|-----------|----------|
|
||
| **Position Size Penalty** | HIGH | Hardcoded 0.8 threshold → agent learns to hover at 79% | 🔴 Critical |
|
||
| **Transaction Costs** | LOW | Agent learns to minimize trading (good!) | 🟢 Benign |
|
||
| **Hold Penalty** | MEDIUM | Agent may learn to avoid holding during high volatility even when correct | 🟡 Moderate |
|
||
| **Diversity Penalty** | LOW | Entropy threshold only prevents collapse, doesn't leak strategy | 🟢 Benign |
|
||
| **Kelly Sizing** | NONE | Not in reward function → no leakage | 🟢 Safe |
|
||
| **Circuit Breaker** | NONE | Only blocks execution, doesn't affect rewards | 🟢 Safe |
|
||
|
||
### 5.3 Detected Leakage Example
|
||
|
||
**Hypothesis**: Agent learns to maintain 79% position to avoid risk penalty
|
||
|
||
**Evidence to Check:**
|
||
1. Distribution of position sizes in trained agent (should see spike at 79%)
|
||
2. Correlation between position size and reward (should be independent after removing P&L)
|
||
3. Agent behavior when position_size = 0.81 (should immediately reduce to 0.79)
|
||
|
||
**Recommendation:**
|
||
```rust
|
||
// Instead of hardcoded threshold, use Kelly-suggested position
|
||
let kelly_position = kelly_sizer.calculate_kelly_fraction(symbol, strategy)?;
|
||
let risk_penalty = if position_size > kelly_position * 1.2 {
|
||
(position_size - kelly_position) * adaptive_multiplier
|
||
} else { 0.0 };
|
||
```
|
||
|
||
This makes the penalty **dynamic** and tied to actual risk analysis, not arbitrary thresholds.
|
||
|
||
---
|
||
|
||
## 6. Integration Point Summary
|
||
|
||
### 6.1 Reward Function Architecture
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────┐
|
||
│ DQN Reward Function (reward.rs) │
|
||
├─────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ Base Reward = pnl_weight * pnl_reward │
|
||
│ - risk_weight * risk_penalty │ ← LEAKAGE RISK
|
||
│ - cost_weight * cost_penalty │ ← Safe (realistic)
|
||
│ + hold_reward / -hold_penalty │ ← Scaled correctly
|
||
│ │
|
||
│ Final Reward = Base + diversity_bonus │
|
||
│ │
|
||
│ Normalized Reward = EMA_normalize(Final) │ ← Stability layer
|
||
│ .clamp(-3.0, 3.0) │
|
||
│ │
|
||
└─────────────────────────────────────────────────────┘
|
||
↓
|
||
┌─────────────────────────────────────────────────────┐
|
||
│ Risk Management (Separate Layer) │
|
||
├─────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ Kelly Sizing (kelly_sizing.rs) │ ← NOT in rewards
|
||
│ - Position size optimization │
|
||
│ - Requires 10+ trade history │
|
||
│ - Fractional Kelly (0.25-0.5) │
|
||
│ │
|
||
│ Circuit Breakers (circuit_breaker.rs) │ ← Execution layer
|
||
│ - Simplified: 5 failures → OPEN │
|
||
│ - Risk Crate: Daily loss % → OPEN │
|
||
│ - Does NOT modify rewards │
|
||
│ │
|
||
└─────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### 6.2 Data Flow
|
||
|
||
```
|
||
Training Step:
|
||
1. Select action (DQN policy network)
|
||
2. Calculate raw reward (P&L + penalties)
|
||
3. Normalize reward (EMA + clipping)
|
||
4. Update Q-network (TD error)
|
||
5. [Separate] Kelly sizing for next position
|
||
6. [Separate] Circuit breaker check
|
||
|
||
Execution Step:
|
||
1. Select action (trained policy)
|
||
2. [Pre-check] Circuit breaker allow?
|
||
3. [Pre-check] Kelly position size
|
||
4. Execute trade
|
||
5. [Post-check] Record outcome for Kelly
|
||
6. [Post-check] Update circuit breaker
|
||
```
|
||
|
||
**Key Observation**: Kelly and circuit breakers are **post-decision**, not pre-decision factors. They don't directly influence reward calculation during training.
|
||
|
||
---
|
||
|
||
## 7. Potential Issues & Recommendations
|
||
|
||
### Issue 1: Position Size Penalty (Hardcoded Threshold)
|
||
**Severity**: 🔴 High
|
||
**Location**: `/ml/src/dqn/reward.rs:718-741`
|
||
|
||
**Problem:**
|
||
```rust
|
||
let risk_penalty = if position_size > 0.8 {
|
||
(position_size - 0.8) * 5.0 // Hardcoded 0.8 threshold
|
||
} else { 0.0 }
|
||
```
|
||
|
||
**Signal Leakage**: Agent learns to maintain 79% position regardless of market conditions.
|
||
|
||
**Recommendation**:
|
||
```rust
|
||
// Use Kelly-suggested position as dynamic threshold
|
||
let kelly_fraction = self.kelly_sizer.calculate_kelly_fraction(symbol, strategy)?;
|
||
let safe_threshold = kelly_fraction.adjusted_kelly_fraction * 1.2; // 20% buffer
|
||
|
||
let risk_penalty = if position_size > safe_threshold {
|
||
(position_size - safe_threshold).powi(2) * adaptive_weight // Quadratic penalty
|
||
} else { 0.0 }
|
||
```
|
||
|
||
### Issue 2: Drawdown Not Used in Penalties
|
||
**Severity**: 🟡 Medium
|
||
**Location**: `/ml/src/dqn/reward.rs:319-329`
|
||
|
||
**Problem**: `RiskMetrics` struct includes `max_drawdown`, but it's **never used** in risk penalty calculation.
|
||
|
||
**Current Usage**:
|
||
```rust
|
||
pub struct RiskMetrics {
|
||
pub var_95: Decimal, // ❌ Not used
|
||
pub max_drawdown: Decimal, // ❌ Not used
|
||
pub sharpe_ratio: Decimal, // ❌ Not used
|
||
pub volatility: Decimal, // ❌ Not used
|
||
}
|
||
```
|
||
|
||
**Recommendation**:
|
||
```rust
|
||
// Add drawdown penalty component
|
||
let drawdown_penalty = if max_drawdown > threshold {
|
||
(max_drawdown - threshold) * drawdown_weight
|
||
} else { 0.0 };
|
||
|
||
final_reward = pnl_reward - risk_penalty - cost_penalty - drawdown_penalty;
|
||
```
|
||
|
||
### Issue 3: Kelly Sizing Warmup
|
||
**Severity**: 🟡 Medium
|
||
**Location**: `/risk/src/kelly_sizing.rs:140-149`
|
||
|
||
**Problem**: Training fails with error when < 10 trades recorded.
|
||
|
||
**Current Behavior**:
|
||
```rust
|
||
if trades.len() < 10 {
|
||
return Err(RiskError::DataUnavailable { ... });
|
||
}
|
||
```
|
||
|
||
**Recommendation**:
|
||
```rust
|
||
if trades.len() < 10 {
|
||
// Use conservative default during warmup
|
||
return Ok(KellyResult {
|
||
position_fraction: 0.02, // 2% default
|
||
use_kelly: false,
|
||
confidence: 0.0,
|
||
sample_size: trades.len(),
|
||
...
|
||
});
|
||
}
|
||
```
|
||
|
||
### Issue 4: EMA Normalization Cold Start
|
||
**Severity**: 🟢 Low (already handled correctly)
|
||
**Location**: `/ml/src/dqn/reward.rs:548-552`
|
||
|
||
**Correct Implementation**:
|
||
```rust
|
||
// CRITICAL FIX: Normalize BEFORE update to avoid zeroing first reward
|
||
let norm = normalizer.normalize(final_reward_f64); // Use PREVIOUS stats
|
||
normalizer.update(final_reward_f64); // Update for NEXT call
|
||
```
|
||
|
||
**Status**: ✅ Fixed (proper order prevents first-reward zeroing)
|
||
|
||
### Issue 5: Hold Penalty Unit Mismatch
|
||
**Severity**: 🟢 Low (fixed, but monitor)
|
||
**Location**: `/ml/src/dqn/reward.rs:881-887`
|
||
|
||
**Fixed Implementation**:
|
||
```rust
|
||
let hold_penalty_scale = Decimal::try_from(1000.0).unwrap_or(Decimal::ONE);
|
||
let hold_penalty_pct = self.config.hold_penalty_weight / hold_penalty_scale;
|
||
// Result: 0.5-2.0 → 0.0005-0.002 (5-20 basis points)
|
||
```
|
||
|
||
**Ratio Check**:
|
||
- Hold penalty: 0.0005-0.002 (0.05-0.2%)
|
||
- Transaction costs: 0.0005-0.0015 (0.05-0.15%)
|
||
- Ratio: 0.33-1.33 (balanced)
|
||
|
||
**Status**: ✅ Fixed in Phase 1, monitor for over/under-trading patterns
|
||
|
||
---
|
||
|
||
## 8. Testing & Validation Recommendations
|
||
|
||
### 8.1 Unit Tests (Add These)
|
||
|
||
**Test 1: Position Size Leakage Detection**
|
||
```rust
|
||
#[test]
|
||
fn test_no_position_size_gaming() {
|
||
let mut reward_fn = RewardFunction::new(RewardConfig::default());
|
||
|
||
// Test position sizes around threshold
|
||
let positions = vec![0.75, 0.79, 0.80, 0.81, 0.85];
|
||
let rewards: Vec<_> = positions.iter().map(|&pos| {
|
||
let state = create_state_with_position(pos);
|
||
reward_fn.calculate_reward(action, &state, &next_state, &[])
|
||
}).collect();
|
||
|
||
// Check for discontinuity at 0.8 threshold
|
||
assert!(
|
||
(rewards[2] - rewards[1]).abs() < 0.1,
|
||
"Reward should be smooth across threshold, not create gaming incentive"
|
||
);
|
||
}
|
||
```
|
||
|
||
**Test 2: Kelly Warmup Handling**
|
||
```rust
|
||
#[tokio::test]
|
||
async fn test_kelly_warmup_does_not_fail() {
|
||
let sizer = KellySizer::new(KellyConfig::default());
|
||
|
||
// Should return default sizing, not error
|
||
let result = sizer.calculate_kelly_fraction(&Symbol::from("AAPL"), "strategy");
|
||
assert!(result.is_ok());
|
||
|
||
let kelly = result.unwrap();
|
||
assert_eq!(kelly.position_fraction, 0.02); // Conservative default
|
||
assert!(!kelly.use_kelly);
|
||
}
|
||
```
|
||
|
||
**Test 3: Hold Penalty Scaling**
|
||
```rust
|
||
#[test]
|
||
fn test_hold_penalty_magnitude() {
|
||
let config = RewardConfig {
|
||
hold_penalty_weight: Decimal::try_from(1.0).unwrap(),
|
||
..Default::default()
|
||
};
|
||
let reward_fn = RewardFunction::new(config);
|
||
|
||
// High volatility state
|
||
let volatile_state = create_state_with_log_return(0.05); // 5% move
|
||
let hold_reward = reward_fn.calculate_hold_reward(&state, &volatile_state)?;
|
||
|
||
// Check penalty is in basis points (not dollars)
|
||
assert!(
|
||
hold_reward.abs() < Decimal::try_from(0.01).unwrap(),
|
||
"Hold penalty should be in percentage units (< 1%)"
|
||
);
|
||
}
|
||
```
|
||
|
||
### 8.2 Integration Tests (Add These)
|
||
|
||
**Test 4: End-to-End Reward + Risk Flow**
|
||
```rust
|
||
#[tokio::test]
|
||
async fn test_reward_risk_integration() {
|
||
let mut reward_fn = RewardFunction::new(RewardConfig::default());
|
||
let kelly_sizer = KellySizer::new(KellyConfig::default());
|
||
let circuit_breaker = CircuitBreaker::new(CircuitBreakerConfig::default());
|
||
|
||
// Simulate 50 trading steps
|
||
for step in 0..50 {
|
||
// 1. Calculate reward
|
||
let reward = reward_fn.calculate_reward(action, &state, &next_state, &actions)?;
|
||
|
||
// 2. Record for Kelly (post-decision)
|
||
if step >= 10 {
|
||
kelly_sizer.add_trade_outcome(TradeOutcome { ... });
|
||
}
|
||
|
||
// 3. Check circuit breaker (post-decision)
|
||
if reward < 0.0 {
|
||
circuit_breaker.record_failure();
|
||
assert!(circuit_breaker.allow_request()); // Should not block training
|
||
}
|
||
}
|
||
|
||
// Verify Kelly sizing became available after warmup
|
||
let kelly_result = kelly_sizer.calculate_kelly_fraction(&symbol, "strategy");
|
||
assert!(kelly_result.is_ok());
|
||
assert!(kelly_result.unwrap().sample_size >= 10);
|
||
}
|
||
```
|
||
|
||
### 8.3 Monitoring Metrics (Add These)
|
||
|
||
**Metric 1: Position Size Distribution**
|
||
```rust
|
||
// Track histogram of position sizes during training
|
||
// Alert if spike at 0.79 (indicates gaming)
|
||
let position_histogram = agent.get_position_histogram();
|
||
assert!(
|
||
position_histogram[79] < 0.2, // < 20% of actions at 79%
|
||
"Agent may be gaming position size threshold"
|
||
);
|
||
```
|
||
|
||
**Metric 2: Reward Component Variance**
|
||
```rust
|
||
// Track variance of each reward component
|
||
let component_stats = reward_fn.get_component_stats();
|
||
assert!(
|
||
component_stats.risk_penalty_variance < component_stats.pnl_variance,
|
||
"Risk penalty should not dominate reward signal"
|
||
);
|
||
```
|
||
|
||
**Metric 3: Kelly vs. Actual Position Divergence**
|
||
```rust
|
||
// Compare Kelly-suggested vs. actual positions
|
||
let kelly_fraction = kelly_sizer.calculate_kelly_fraction(&symbol, "strategy")?;
|
||
let actual_position = agent.current_position_size();
|
||
|
||
let divergence = (actual_position - kelly_fraction).abs();
|
||
assert!(
|
||
divergence < 0.2,
|
||
"Agent should approximately follow Kelly sizing, divergence: {:.2}%",
|
||
divergence * 100.0
|
||
);
|
||
```
|
||
|
||
---
|
||
|
||
## 9. Conclusion
|
||
|
||
### Strengths ✅
|
||
1. **EMA-based normalization** prevents mean drift in non-stationary markets
|
||
2. **Percentage-based P&L** achieves scale-invariance (BUG #17 fix)
|
||
3. **Transaction cost fix** uses actual exchange fees (BUG #24 fix)
|
||
4. **Hold penalty scaling fix** aligns units with transaction costs (Phase 1 fix)
|
||
5. **Kelly sizing isolated** from reward function (no circular dependency)
|
||
6. **Circuit breakers separated** from reward calculation (proper layering)
|
||
|
||
### Weaknesses ⚠️
|
||
1. **Position size penalty uses hardcoded threshold** → signal leakage risk
|
||
2. **Drawdown not used in penalties** despite being in `RiskMetrics` struct
|
||
3. **Kelly sizing requires 10+ trades** → early training failures
|
||
4. **No validation tests** for leakage detection or component interaction
|
||
|
||
### Critical Recommendations 🔴
|
||
1. **Replace hardcoded 0.8 threshold** with Kelly-based dynamic thresholds
|
||
2. **Add drawdown penalty** to reward function (use `RiskMetrics.max_drawdown`)
|
||
3. **Implement Kelly warmup** (use 2% default for first 10 trades)
|
||
4. **Add leakage detection tests** (position size distribution, reward component variance)
|
||
5. **Monitor training metrics** for position gaming and reward component dominance
|
||
|
||
### Expected Impact
|
||
- **Training Stability**: Current EMA + percentage P&L is robust ✅
|
||
- **Risk Management**: Kelly + circuit breakers properly isolated ✅
|
||
- **Reward Overfitting**: Moderate risk due to hardcoded thresholds ⚠️
|
||
- **Signal Leakage**: High risk if agent learns to game position size threshold 🔴
|
||
|
||
---
|
||
|
||
## File Locations
|
||
|
||
| Component | File Path | Lines |
|
||
|-----------|-----------|-------|
|
||
| Reward Normalization | `/ml/src/dqn/reward.rs` | 14-165 |
|
||
| Reward Function | `/ml/src/dqn/reward.rs` | 389-941 |
|
||
| Risk Penalties | `/ml/src/dqn/reward.rs` | 713-907 |
|
||
| Kelly Sizing | `/risk/src/kelly_sizing.rs` | 1-342 |
|
||
| Circuit Breaker (Simple) | `/ml/src/dqn/circuit_breaker.rs` | 1-435 |
|
||
| Risk Integration | `/ml/src/dqn/risk_integration.rs` | 1-355 |
|
||
| Reward Coordinator | `/ml/src/dqn/reward_coordinator.rs` | 1-569 |
|
||
|
||
---
|
||
|
||
## Appendix: Reward Formula Breakdown
|
||
|
||
**Complete Reward Calculation:**
|
||
```rust
|
||
// Step 1: Calculate base components
|
||
pnl_reward = (next_value - current_value) / current_value // ±0.02 typical
|
||
risk_penalty = if position_size > 0.8 { (position_size - 0.8) * 5.0 } else { 0.0 }
|
||
cost_penalty = position_change * tx_cost_rate // 0.05-0.15%
|
||
hold_reward = if volatility < threshold { +0.001 } else { -hold_penalty_pct }
|
||
diversity_bonus = if entropy < 0.5 { -0.1 } else { 0.0 }
|
||
|
||
// Step 2: Weighted sum
|
||
base_reward = pnl_weight * pnl_reward // Default: 1.0
|
||
- risk_weight * risk_penalty // Default: 0.1
|
||
- cost_weight * cost_penalty // Default: 1.0
|
||
+ hold_reward // Conditional ±0.001
|
||
|
||
final_reward = base_reward + diversity_bonus
|
||
|
||
// Step 3: Normalize and clip
|
||
normalized = (final_reward - mean) / std
|
||
clipped = normalized.clamp(-3.0, 3.0)
|
||
|
||
// Step 4: Update EMA for next step
|
||
mean = 0.01 * final_reward + 0.99 * mean
|
||
variance = 0.01 * (final_reward - mean)^2 + 0.99 * variance
|
||
```
|
||
|
||
**Expected Ranges:**
|
||
- Raw reward: -0.05 to +0.05 (±5% moves rare)
|
||
- Normalized reward: -3.0 to +3.0 (clipped)
|
||
- Q-values: -80 to +80 (r/(1-γ) with γ=0.9626)
|
||
- TD errors: ±10 typical (after normalization)
|
||
|
||
---
|
||
|
||
**End of Analysis**
|