foxhunt/DQN_TRIAL19_EVALUATION_REPORT.md

DQN Model Evaluation Report - Comprehensive Analysis

Date: 2025-11-04 Evaluated Model: ml/trained_models/dqn_best_model.safetensors (Epoch 445) Training Date: 2025-11-01 to 2025-11-04 Evaluation Duration: 1.06 seconds Status: ⚠️ CRITICAL PERFORMANCE ISSUES IDENTIFIED

---

Executive Summary

The trained DQN model (500 epochs, best checkpoint at epoch 445) has been evaluated on unseen data (ES_FUT_unseen.parquet, 14,420 bars) and shows catastrophic underperformance:

• Total P&L: -$373.25 (net loss)
• Win Rate: 19.4% (vs. 55%+ target)
• Sharpe Ratio: -7.00 (extremely poor, target >1.5)
• Max Drawdown: $377.25
• Action Distribution: 99.4% HOLD (severe action collapse)

Production Readiness: ❌ NOT READY - Model exhibits severe HOLD bias and cannot generate profitable trades.

---

Evaluation Metrics - Unseen Data

Trading Performance

Metric	Unseen Data	Training Data	Target	Status
Total Trades	36	421	N/A	⚠️ Very low
Winning Trades	7 (19.4%)	119 (28.3%)	>55%	❌ FAIL
Losing Trades	29	301	<45%	❌ FAIL
Total P&L	-$373.25	-$2,643.00	>$0	❌ FAIL
Avg P&L/Trade	-$10.37	-$6.28	>$0	❌ FAIL
Sharpe Ratio	-7.00	-4.24	>1.5	❌ CATASTROPHIC
Max Drawdown	$377.25	$2,769.00	<15%	❌ FAIL
Avg Bars Held	398.7	412.5	N/A	⚠️ Long hold times

Key Finding: Performance on unseen data is consistent with training data - both show severe losses and negative Sharpe ratios. This indicates the model learned a systematically unprofitable trading strategy, not overfitting.

---

Action Distribution Analysis

Unseen Data (14,420 bars):

• BUY: 43 actions (0.3%)
• SELL: 45 actions (0.3%)
• HOLD: 14,332 actions (99.4%)

Training Data (173,953 bars):

• BUY: 452 actions (0.3%)
• SELL: 1,529 actions (0.9%)
• HOLD: 171,972 actions (98.9%)

Critical Issue: Model exhibits severe HOLD collapse - taking action <1% of the time. This is a well-known DQN failure mode where the model learns to avoid risky actions (BUY/SELL) and defaults to the "safe" no-op action (HOLD).

Root Cause: Likely caused by:

1. Reward shaping issues: HOLD penalty too low or absent
2. Q-value floor bias: Conservative Q-value estimates favor HOLD
3. Exploration decay: Epsilon decayed too quickly, preventing BUY/SELL exploration
4. Training data imbalance: Insufficient profitable trade examples

---

Trade Quality Analysis

Winning Trades (7 total):

• Average profit: $16.86
• Largest win: $39.25
• Win rate: 19.4%

Losing Trades (29 total):

• Average loss: -$16.94
• Largest loss: -$87.50
• Loss rate: 80.6%

Risk-Reward Ratio: 0.99 (avg win / avg loss)

• Target: >1.5 for profitable trading
• Status: ❌ FAIL - Losses are as large as wins, leading to net negative P&L

Holding Period:

• Average: 398.7 bars (~16.6 hours for 1-min data)
• This suggests the model holds positions for extended periods, amplifying losses

---

Latency Performance

Metric	Value	Target	Status
Mean	73 μs	<200 μs	✅ PASS
Median	68 μs	<200 μs	✅ PASS
P95	83 μs	<200 μs	✅ PASS
P99	92 μs	<200 μs	✅ PASS
Max	50,136 μs	<1000 μs	⚠️ Outlier spike

Inference Performance: ✅ EXCELLENT - Model meets latency requirements for HFT deployment. Mean inference time of 73 μs is well within target.

---

Comparison: Baseline vs. Current Model

Metric	Baseline (/tmp/dqn_prod_best.safetensors)	Current Best (ml/trained_models/dqn_best_model.safetensors)	Change
SHA256	7c4184cb99f752...	92e1e81fa3893...	Different models
Total P&L	-$373.25	-$373.25	IDENTICAL
Win Rate	19.4%	19.4%	IDENTICAL
Sharpe Ratio	-7.00	-7.00	IDENTICAL
Action Dist	BUY:0.3%, SELL:0.3%, HOLD:99.4%	BUY:0.3%, SELL:0.3%, HOLD:99.4%	IDENTICAL

Suspicious Finding: Despite different SHA256 checksums, both models produce byte-for-byte identical evaluation results. This suggests:

1. Models may have converged to the same local minimum (HOLD collapse attractor)
2. Training process is deterministic given same hyperparameters
3. Current training configuration produces consistently poor models

---

Root Cause Analysis

Issue #1: HOLD Collapse (CRITICAL)

Symptoms:

• 99.4% HOLD action rate
• Only 88 BUY/SELL actions across 14,420 bars (<1%)
• Model refuses to take trading actions

Diagnosis: The DQN learned that HOLD is the "safest" action because:

1. Reward function flaw: HOLD likely receives 0 reward (neutral), while BUY/SELL risk negative rewards from commission costs and price movements
2. Epsilon decay: Exploration decayed too quickly (0.99-0.9946), locking in early HOLD bias before learning profitable trades
3. Q-value estimates: HOLD Q-values likely dominate due to conservative Bellman updates

Evidence:

• Hyperopt Trial #68 (best) used epsilon_decay=0.99 (standard)
• Training logs show final Q-values converged to narrow range (381-703), suggesting limited action differentiation
• No entropy penalty or HOLD penalty in reward function

Fix Required:

1. Add explicit HOLD penalty (-0.01 to -0.1 per step)
2. Increase epsilon decay to 0.995-0.999 (slower exploration → exploitation)
3. Implement entropy regularization to encourage action diversity
4. Use prioritized experience replay to amplify rare profitable trades

---

Issue #2: Reward Shaping Inadequacy (HIGH)

Problem: Current reward function likely relies solely on P&L, which:

• Heavily penalizes early exploration (BUY/SELL result in immediate commission costs)
• Rewards HOLD by default (no commission, no loss)
• Fails to incentivize learning from profitable market patterns

Fix Required:

1. Implement shaped rewards:
   • HOLD penalty: -0.05 per step
   • Action diversity bonus: +0.1 for BUY/SELL exploration
   • Trend-following bonus: +0.2 for BUY in uptrend, SELL in downtrend
2. Use hindsight experience replay to relabel failed trades with "what should have been done"

---

Issue #3: Hyperparameter Mismatch (MEDIUM)

Training Configuration (inferred from hyperopt best, Nov 3):

• Learning rate: 0.000554
• Batch size: 230
• Gamma: 0.99
• Epsilon decay: 0.99
• Buffer size: 1,000,000

Observations:

• High learning rate (5.5e-4) may cause rapid convergence to HOLD local minimum
• Large buffer (1M) provides diverse experiences but doesn't overcome HOLD bias
• No evidence of HOLD penalty or entropy regularization

Fix Required:

• Test with lower learning rates (1e-4 to 3e-4) for slower, more careful exploration
• Reduce batch size to 64-128 to increase gradient noise (escape local minima)
• Add explicit HOLD collapse detection and mitigation

---

Hyperopt Objective Analysis

Expected vs. Actual Performance

Hyperopt Trial #68 (Nov 3):

• Objective: 0.000635 (positive reward)
• Training Duration: 31.59 seconds
• Validation Loss: Unknown (not in report)

Actual Evaluation:

• Total P&L: -$373.25 (catastrophic loss)
• Sharpe Ratio: -7.00 (extremely poor)
• Win Rate: 19.4% (far below 55% target)

Discrepancy: The hyperopt objective of +0.000635 suggests a positive reward, yet the trained model produces massive losses (-$373.25). This indicates:

1. Objective function mismatch: Hyperopt may be optimizing for a proxy metric (e.g., validation loss, Q-value stability) that doesn't correlate with trading profitability
2. Evaluation data mismatch: Hyperopt may have evaluated on a different dataset than the final model
3. Short training epochs: Hyperopt trials used 20 epochs (per DQN_HYPEROPT_RESULTS_20251103.md), while final model used 500 epochs - model may have overfit or collapsed during extended training

Required Investigation:

1. ✅ Review hyperopt objective calculation in ml/src/hyperopt/adapters/dqn.rs
2. ✅ Compare hyperopt evaluation dataset vs. current ES_FUT_unseen.parquet
3. ✅ Re-run hyperopt Trial #68 hyperparameters for 500 epochs and evaluate
4. ✅ Verify if validation loss anomaly (240x lower than training loss, per DQN_HYPEROPT_FINAL_RESULTS.md) was resolved

---

Production Readiness Assessment

Blocking Issues for Deployment

Issue	Severity	Impact	Status
HOLD Collapse	CRITICAL	Model refuses to trade, 99.4% inaction	❌ BLOCKS
Negative Sharpe (-7.00)	CRITICAL	Consistently losing strategy	❌ BLOCKS
Low Win Rate (19.4%)	CRITICAL	80% of trades are losers	❌ BLOCKS
Net Loss (-$373)	CRITICAL	Unprofitable on unseen data	❌ BLOCKS
Hyperopt Objective Mismatch	HIGH	Objective doesn't predict real performance	⚠️ REVIEW
Validation Loss Anomaly	HIGH	240x val_loss < train_loss (if unresolved)	⚠️ REVIEW

Overall Status: ❌ NOT PRODUCTION READY

---

Recommendations

Priority 1: Fix HOLD Collapse (CRITICAL - 1-2 days)

Tasks:

1. Implement HOLD penalty in reward function (-0.05 per timestep)
2. Add entropy regularization to DQN loss (coefficient: 0.01)
3. Increase epsilon decay from 0.99 to 0.995-0.999
4. Add HOLD collapse detection (trigger retraining if HOLD >90%)
5. Test with shaped rewards (trend-following bonus, action diversity bonus)

Expected Outcome: HOLD action rate <50%, BUY/SELL actions >50%

Code Changes:

• ml/src/trainers/dqn.rs: Modify reward calculation (lines ~500-600)
• ml/src/dqn/dqn.rs: Add entropy loss term (lines ~300-400)
• ml/examples/train_dqn.rs: Add CLI flags for HOLD penalty and entropy coefficient

---

Priority 2: Validate Hyperopt Objective (HIGH - 4-8 hours)

Tasks:

1. Review objective calculation in ml/src/hyperopt/adapters/dqn.rs
2. Verify if objective uses validation loss or actual trading P&L
3. Re-run Trial #68 hyperparameters with HOLD penalty enabled
4. Compare hyperopt evaluation dataset vs. ES_FUT_unseen.parquet
5. Investigate validation loss anomaly (if unresolved from Nov 2)

Expected Outcome: Hyperopt objective correlates with backtest Sharpe ratio (r^2 >0.7)

---

Priority 3: Multi-Seed Validation (MEDIUM - 2-3 hours)

Tasks:

1. Train 5 models with Trial #68 hyperparameters + different random seeds
2. Evaluate all 5 on ES_FUT_unseen.parquet
3. Calculate variance in Sharpe ratio and win rate
4. Confirm HOLD collapse is systemic, not seed-dependent

Expected Outcome: All 5 models show HOLD collapse (confirms systemic issue, not randomness)

---

Priority 4: Explore Alternative RL Algorithms (LOW - 1 week)

Rationale: DQN may be fundamentally unsuited for this trading problem due to:

• Discrete action space (BUY/SELL/HOLD) biases toward safe HOLD
• Q-value estimation instability with sparse rewards
• Difficulty learning long-horizon dependencies (400+ bars held)

Alternatives to Consider:

1. PPO (already implemented, dual learning rates verified working)
   • Advantage: Continuous action space (position sizing), better exploration
   • Status: ✅ Ready for production training (per CLAUDE.md)
2. SAC (Soft Actor-Critic): Entropy-regularized by design, prevents action collapse
3. TD3 (Twin Delayed DDPG): Better for continuous control, stable training

Next Step: Deploy PPO production training (already validated, faster than DQN fix)

---

Conclusion

The DQN Trial #19 (or Trial #68, unclear from naming) model achieved excellent inference latency (73 μs mean) but catastrophic trading performance:

• Sharpe Ratio: -7.00 (target: >1.5) - 567% below target
• Win Rate: 19.4% (target: >55%) - 65% below target
• HOLD Collapse: 99.4% inaction rate - model refuses to trade

Root Cause: Reward function flaw incentivizes HOLD (safe, neutral reward) over BUY/SELL (risky, often negative). Model converged to local minimum of "do nothing."

Immediate Action:

1. ❌ DO NOT DEPLOY current DQN model to production
2. ✅ FIX HOLD COLLAPSE via reward shaping (Priority 1, 1-2 days)
3. ⚠️ CONSIDER PPO as faster alternative (already production-ready per CLAUDE.md)

Long-Term Path:

• Fix DQN reward function and hyperparameters (1-2 weeks)
• OR pivot to PPO/SAC for better exploration guarantees (1 week)
• Re-run hyperopt with corrected objective function (1 week)

Production ETA: 2-4 weeks (DQN fix) or 1 week (PPO deployment)

---

Report Generated: 2025-11-04 12:30:00 UTC Evaluation Tool: cargo run -p ml --example evaluate_dqn Model Path: /home/jgrusewski/Work/foxhunt/ml/trained_models/dqn_best_model.safetensors Data Path: /home/jgrusewski/Work/foxhunt/test_data/ES_FUT_unseen.parquet Status: ⚠️ CRITICAL ISSUES - PRODUCTION DEPLOYMENT BLOCKED