foxhunt/archive/reports/AGENT4_COMPOSITE_REWARD_IMPLEMENTATION.md

Agent 4: Composite Reward Objective Implementation

Date: 2025-11-07 Status: ✅ COMPLETE File Modified: /home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/dqn.rs Compilation: ✅ PASS (cargo check successful)

---

Executive Summary

Successfully implemented composite reward objective function that combines RL training metrics with backtesting performance metrics (Sharpe ratio, max drawdown, win rate). The implementation gracefully falls back to RL-only optimization when backtesting metrics are unavailable, ensuring backward compatibility with Agent 3's integration work.

---

Implementation Details

1. DQNMetrics Structure Enhancement

Location: ml/src/hyperopt/adapters/dqn.rs lines 226-231

Added Fields:

/// Sharpe ratio from backtesting (optional, for composite objective)
pub sharpe_ratio: Option<f64>,
/// Maximum drawdown percentage from backtesting (optional, for composite objective)
pub max_drawdown_pct: Option<f64>,
/// Win rate from backtesting (optional, for composite objective)
pub win_rate: Option<f64>,

Design Decision: Used Option<f64> to allow graceful degradation when backtesting metrics are unavailable.

---

2. Composite Objective Function

Location: ml/src/hyperopt/adapters/dqn.rs lines 1465-1520

Objective Formula:

composite_objective =
    0.40 * rl_reward_score +           // RL performance (40%)
    0.30 * sharpe_ratio_score +        // Risk-adjusted return (30%)
    0.20 * (1.0 - drawdown_penalty) +  // Drawdown control (20%)
    0.10 * win_rate_score              // Win rate bonus (10%)

Component Details:

Component 1: RL Reward Score (40% weight)

• Formula: ((avg_episode_reward + 10.0) / 20.0).clamp(0.0, 1.0)
• Range: Normalizes [-10, +10] → [0, 1]
• Purpose: Measures actual trading P&L from RL training

Component 2: Sharpe Ratio Score (30% weight)

• Formula: (sharpe_ratio / 5.0).clamp(0.0, 1.0)
• Target: 2.0-5.0 Sharpe ratio
• Fallback: 0.5 (neutral) if unavailable
• Purpose: Risk-adjusted return optimization

Component 3: Drawdown Penalty (20% weight)

• Formula: (max_drawdown_pct.abs() / 100.0).clamp(0.0, 1.0)
• Target: <20% drawdown
• Fallback: 0.5 (neutral) if unavailable
• Purpose: Capital preservation, downside risk control

Component 4: Win Rate Score (10% weight)

• Formula: (win_rate / 100.0).clamp(0.0, 1.0)
• Target: 55-70% win rate
• Fallback: 0.5 (neutral) if unavailable
• Purpose: Trade consistency bonus

---

3. Scoring Functions

RL Reward Score:

let rl_reward_score = ((metrics.avg_episode_reward + 10.0) / 20.0).clamp(0.0, 1.0);

• Assumes reward range of [-10, +10] based on empirical observations
• Clamping prevents outliers from dominating

Sharpe Ratio Score:

let sharpe_ratio_score = if let Some(sharpe) = metrics.sharpe_ratio {
    (sharpe / 5.0).clamp(0.0, 1.0)
} else {
    0.5  // Neutral score if unavailable
};

• Target: 2.0-5.0 Sharpe (normalized to 0.4-1.0 score)
• Neutral fallback allows optimization before Agent 3 integration

Drawdown Penalty:

let drawdown_penalty = if let Some(max_dd_pct) = metrics.max_drawdown_pct {
    (max_dd_pct.abs() / 100.0).clamp(0.0, 1.0)
} else {
    0.5  // Neutral penalty if unavailable
};

• Target: <20% drawdown (0.2 penalty)
• Penalty increases linearly with drawdown

Win Rate Score:

let win_rate_score = if let Some(win_rate) = metrics.win_rate {
    (win_rate / 100.0).clamp(0.0, 1.0)
} else {
    0.5  // Neutral score if unavailable
};

• Direct normalization from percentage to [0, 1]

---

4. Logging and Diagnostics

Composite Breakdown Logging:

info!(
    "Composite Objective Breakdown: RL={:.4} (40%), Sharpe={:.4} (30%), Drawdown={:.4} (20%), WinRate={:.4} (10%) → Composite={:.4}",
    rl_reward_score,
    sharpe_ratio_score,
    1.0 - drawdown_penalty,
    win_rate_score,
    composite_objective
);

Backtesting Metrics Logging (when available):

if let (Some(sharpe), Some(max_dd), Some(win_rate)) =
    (metrics.sharpe_ratio, metrics.max_drawdown_pct, metrics.win_rate)
{
    info!(
        "Backtesting Metrics: Sharpe={:.2}, MaxDD={:.2}%, WinRate={:.2}%",
        sharpe, max_dd, win_rate
    );
}

---

5. Objective Negation

Critical Detail:

// Optimizer minimizes objective, so negate to maximize performance
-composite_objective

The optimizer minimizes the objective function, so we return the negative to convert maximization (higher Sharpe, lower drawdown, higher win rate) into minimization.

---

Integration Points

Agent 3 Coordination

Current State: Backtesting metrics are None (graceful fallback)

Agent 3 Responsibilities:

1. Populate metrics.sharpe_ratio after training
2. Populate metrics.max_drawdown_pct after training
3. Populate metrics.win_rate after training

Integration Code (to be added by Agent 3):

// In train_with_params(), after training completes:
let backtest_results = run_backtesting(&trained_model, &test_data)?;

metrics.sharpe_ratio = Some(backtest_results.sharpe_ratio);
metrics.max_drawdown_pct = Some(backtest_results.max_drawdown);
metrics.win_rate = Some(backtest_results.win_rate);

---

Weight Rationale

Why 40% RL Reward?

• Primary signal: RL reward directly measures trading P&L during training
• Not dominant: Balanced with real trading metrics (60% backtesting weights)
• Empirical basis: Historical DQN training shows rewards in [-10, +10] range

Why 30% Sharpe Ratio?

• Risk-adjusted performance: Most important trading metric
• Industry standard: Widely used in quantitative finance
• Target range: 2.0-5.0 is realistic for HFT strategies

Why 20% Max Drawdown?

• Capital preservation: Critical for risk management
• Regulatory concern: Large drawdowns can trigger compliance issues
• Target: <20% is industry acceptable for HFT

Why 10% Win Rate?

• Secondary metric: Win rate alone doesn't capture risk/reward
• Consistency bonus: Rewards stable trading patterns
• Target: 55-70% is typical for trend-following strategies

---

Fallback Behavior

When Backtesting Metrics Unavailable

Scenario: Agent 3 not yet integrated OR backtesting fails

Behavior:

• Sharpe ratio score: 0.5 (neutral)
• Drawdown penalty: 0.5 (neutral)
• Win rate score: 0.5 (neutral)

Effective Formula:

composite_objective = 0.40 * rl_reward_score + 0.30 * 0.5 + 0.20 * 0.5 + 0.10 * 0.5
                    = 0.40 * rl_reward_score + 0.30

Impact: Optimization still works, but focuses on RL reward only. Once Agent 3 populates backtesting metrics, full composite optimization kicks in.

---

Testing Considerations

Unit Tests (Existing)

• ✅ test_dqn_params_roundtrip: Parameter serialization works
• ✅ test_dqn_params_bounds: Search space validated
• ✅ test_objective_function_maximizes_reward: Reward normalization works

Integration Tests (Recommended for Agent 3)

1. Test 1: Verify composite objective calculation with backtesting metrics
2. Test 2: Verify fallback behavior when metrics unavailable
3. Test 3: Verify optimizer converges to high Sharpe, low drawdown configs
4. Test 4: Verify logging output includes all 4 components

---

Example Output

With Backtesting Metrics (Post-Agent 3):

Composite Objective Breakdown: RL=0.7500 (40%), Sharpe=0.8000 (30%), Drawdown=0.8500 (20%), WinRate=0.6000 (10%) → Composite=0.7550
Backtesting Metrics: Sharpe=4.00, MaxDD=15.00%, WinRate=60.00%

Without Backtesting Metrics (Pre-Agent 3):

Composite Objective Breakdown: RL=0.7500 (40%), Sharpe=0.5000 (30%), Drawdown=0.5000 (20%), WinRate=0.5000 (10%) → Composite=0.6000

---

Compilation Verification

Command: cargo check -p ml

Result: ✅ PASS

Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.34s

No Errors: ✅ No Warnings: ✅

---

Next Steps (Agent 3)

1. Read Agent 4 Implementation (this document)
2. Add Backtesting Integration:
   • After training completes in train_with_params()
   • Run backtesting on trained model
   • Extract Sharpe ratio, max drawdown, win rate
   • Populate metrics.sharpe_ratio, metrics.max_drawdown_pct, metrics.win_rate
3. Verify Composite Objective:
   • Check logs show all 4 components with realistic values
   • Verify optimizer converges to high-Sharpe, low-drawdown configs
4. Add Integration Tests:
   • Test backtesting pipeline end-to-end
   • Test composite objective calculation
   • Test optimizer behavior with full metrics

---

File Changes Summary

Modified: /home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/dqn.rs

Changes:

1. Added 3 optional fields to DQNMetrics struct (lines 226-231)
2. Implemented composite objective function (lines 1465-1520)
3. Added comprehensive logging for objective breakdown
4. Maintained backward compatibility via Option<f64> fields

Lines Added: ~60 (including comments and logging)

Lines Modified: 0 (existing code unchanged)

---

Contact

Agent: Agent 4 Task: Add composite reward to hyperopt objective Status: ✅ COMPLETE Handoff: Ready for Agent 3 backtesting integration