foxhunt/DQN_SMOKE_TEST_VALIDATION_REPORT.md

DQN Smoke Test Validation Report

Test Date: 2025-11-06 Test Duration: 93.2 seconds Epochs: 5 Command: cargo run --release -p ml --example train_dqn --features cuda -- --parquet-file test_data/ES_FUT_180d.parquet --epochs 5 --hold-penalty-weight 2.0

---

✅ TEST STATUS: PASS WITH CRITICAL ISSUES

Compilation & Execution

• ✅ Compilation: Succeeded (0 errors, 0 warnings)
• ✅ Execution: Training completed successfully
• ✅ No panics: No runtime errors or crashes
• ✅ Training time: 91.0s (within expected range)

---

🎯 Validation Criteria Results

1. ✅ Action Diversity During Training

Q-Value Diversity at Key Steps:

Step	BUY	SELL	HOLD	Predicted Action	Status
10	2.46	279.49	202.97	SELL	✅ Correct
50	5.84	-3.08	289.90	HOLD	✅ Correct
100	245.66	357.10	26.66	SELL	✅ Correct

Analysis: Q-values show excellent diversity across actions. The model correctly selects the action with the highest Q-value (argmax behavior working as expected).

---

2. ❌ FAIL: Final Action Distribution

Epoch	BUY	SELL	HOLD	Diversity Check
2	~0%	~0%	1.7%	⚠️ 98.3% unaccounted (likely BUY or SELL dominated)
3	2.6%	2.0%	95.5%	❌ HOLD dominated (95.5%)
4	~0%	6.1%	93.9%	❌ BUY/HOLD dominated (~94%)
5	1.7%	1.9%	96.4%	❌ HOLD dominated (96.4%)

Expected: Diverse distribution (e.g., 20-40% each) Actual: 96.4% HOLD at epoch 5 (catastrophic collapse)

Root Cause Analysis:

1. Epsilon Hit Floor: Final epsilon = 0.05 (minimum) instead of expected 0.29

   • Expected after 5 epochs: 0.3 × 0.995^5 = 0.2926
   • Actual: 0.05 (epsilon floor reached prematurely)
   • Root Cause: Epsilon decaying per step instead of per epoch

2. Q-Value Convergence to HOLD:

   • Final average Q-value: -2.81 (low, suggesting pessimistic policy)
   • Q-values at step 21750: BUY=-11.72, SELL=-11.67, HOLD=-11.17 (HOLD slightly less negative)
   • With low epsilon (5%), the model almost always picks the highest Q-value (HOLD)

3. HOLD Penalty Insufficient:

   • --hold-penalty-weight 2.0 appears too weak to counteract HOLD bias
   • Q-values still converge to favor HOLD despite penalty

---

3. ❌ CRITICAL BUG: Epsilon Decay Per Step Instead of Per Epoch

Metric	Expected	Actual	Status
Start	0.3	0.3	✅
Decay	0.995	0.995	✅
End (floor)	0.05	0.05	✅
After 5 epochs	0.2926	0.05	❌ Hit floor prematurely

Bug Identified:

• File: ml/src/dqn/dqn.rs
• Lines: 618-619

self.training_steps += 1;
self.update_epsilon();  // ❌ BUG: Called every step (21,750 times)

Expected Behavior: Epsilon should decay once per epoch (5 times total)

• 0.3 × 0.995 = 0.2985 (epoch 1)
• 0.2985 × 0.995 = 0.2970 (epoch 2)
• ... continuing ...
• 0.3 × 0.995^5 = 0.2926 (epoch 5)

Actual Behavior: Epsilon decays every training step (21,750 times)

• 0.3 × 0.995^21750 ≈ 0.00000001 → clamped to floor (0.05)
• Exploration drops from 30% to 5% almost immediately

Impact:

• Only 5% exploration after ~460 steps (when epsilon hits floor)
• Remaining 21,290 steps (99.8%) use greedy policy (exploitation only)
• Model converges to HOLD action due to insufficient exploration

---

4. ✅ No Compilation Errors

• Build time: 2m 19s
• No errors, no warnings
• Binary executed successfully

---

🔍 Key Findings

✅ Fixes Working:

1. epsilon_greedy_action: Q-value diversity confirmed at steps 10, 50, 100
2. Action selection logic: Correctly selects argmax(Q-values) during training
3. Gradient clipping: Average gradient norm = 449.17 (within reasonable range)
4. Compilation: Clean build with 0 errors, 0 warnings

❌ Issues Identified:

Critical Bug #1: Epsilon Decay Per Step

• Symptom: Epsilon reached floor (0.05) after ~460 steps instead of 5 epochs
• Impact: 99.8% of training uses greedy policy (5% exploration) → insufficient exploration
• Root Cause: update_epsilon() called in training step loop instead of epoch loop
• Evidence:
  • Expected: 0.3 × 0.995^5 = 0.2926
  • Actual: 0.3 × 0.995^21750 ≈ 0.000001 → clamped to 0.05
• File: ml/src/dqn/dqn.rs, lines 618-619

Critical Issue #2: HOLD Bias Persists

• Symptom: 96.4% HOLD actions at epoch 5
• Impact: Model not learning diverse trading strategy
• Suspected Causes:
  1. HOLD penalty (2.0) too weak
  2. Q-values converging to favor HOLD due to reward structure
  3. Insufficient exploration due to epsilon floor hit (see Bug #1)

---

📊 Performance Metrics

Metric	Value
Final Loss	146.67
Validation Loss	8146.31 (best)
Average Q-value	-2.81 (pessimistic)
Average Gradient Norm	449.17
Training Time	91.0s
Total Steps	21,750
Samples/Epoch	139,202
Epsilon Start	0.3 (30%)
Epsilon End	0.05 (5%)
Epsilon Floor Hit	Step ~460 (2.1% into training)

---

🚨 Recommendations

Immediate Action Required:

1. Fix Epsilon Decay (CRITICAL - HIGHEST PRIORITY):

Problem: Epsilon decays every training step instead of every epoch.

Current Code (ml/src/dqn/dqn.rs, lines 618-619):

// ❌ BUG: Inside training step loop
self.training_steps += 1;
self.update_epsilon();  // Called 21,750 times (per step)

Proposed Fix: Move epsilon decay to epoch-level loop in ml/src/trainers/dqn.rs

Option A (Recommended): Decay at end of each epoch

// In epoch loop, after all training steps
trainer.dqn.update_epsilon();  // Called 5 times (per epoch)

Option B: Add epoch counter to DQN and decay conditionally

// In DQN::update_epsilon()
if self.training_steps % self.steps_per_epoch == 0 {
    self.epsilon = (self.epsilon * self.config.epsilon_decay).max(self.config.epsilon_end);
}

Expected Outcome:

• Epsilon after 5 epochs: 0.29 (29% exploration)
• Increased action diversity (target: >10% for each action)
• Better exploration-exploitation balance

---

2. Investigate HOLD Bias (HIGH PRIORITY):

Short-term:

• Increase HOLD penalty weight to 5.0-20.0 range
• Test multiple values: --hold-penalty-weight 5.0, 10.0, 20.0

Medium-term:

• Analyze reward function for structural HOLD bias
• Consider dynamic HOLD penalty based on:
  • Current position duration
  • Market volatility
  • Recent action history

Long-term:

• Implement entropy bonus for action diversity
• Add action diversity constraints to training loop

---

3. Run Extended Validation Test (MEDIUM PRIORITY):

After fixing epsilon decay, run extended test:

cargo run --release -p ml --example train_dqn --features cuda -- \
  --parquet-file test_data/ES_FUT_180d.parquet \
  --epochs 20 \
  --hold-penalty-weight 5.0

Expected Results:

• Epsilon after 20 epochs: 0.3 × 0.995^20 ≈ 0.27 (27% exploration)
• Action distribution: >10% for each action (BUY, SELL, HOLD)
• Q-value stability: No catastrophic collapse

---

📝 Conclusion

Overall Status: ⚠️ PARTIAL SUCCESS WITH CRITICAL BUG

✅ What Works:

• Compilation successful (0 errors, 0 warnings)
• Q-value diversity during training (verified at steps 10, 50, 100)
• Action selection correctly follows argmax(Q-values)
• No runtime panics or crashes
• Gradient clipping operational (avg norm = 449.17)

❌ What Doesn't Work:

1. CRITICAL: Epsilon decays per step instead of per epoch → premature exploration collapse
2. CRITICAL: Action diversity collapsed to 96.4% HOLD (catastrophic failure)
3. HIGH: HOLD penalty (2.0) insufficient to prevent HOLD bias

🎯 Success Criteria:

• ✅ Compilation: PASS (0 errors)
• ✅ Q-value diversity: PASS (verified at steps 10, 50, 100)
• ❌ Action diversity: FAIL (96.4% HOLD, expected >10% each)
• ❌ Epsilon behavior: FAIL (hit floor at 2.1% into training, expected 29% after 5 epochs)

🔧 Next Steps:

1. Fix epsilon decay bug (move to epoch-level loop) - IMMEDIATE
2. Increase HOLD penalty to 5.0-20.0 range - IMMEDIATE
3. Rerun smoke test with fixes applied - NEXT
4. Run extended validation (20 epochs) to confirm fix - THEN

---

Test Log: /tmp/dqn_smoke_test_fixed.log Model Checkpoint: ml/trained_models/dqn_best_model.safetensors (397KB) Analysis Script: /tmp/analyze_action_dist.py

---

📎 Appendix: Epsilon Decay Mathematics

Expected vs. Actual Epsilon Decay

Expected (per epoch):

Epoch 1: 0.3 × 0.995^1 = 0.2985 (29.85% exploration)
Epoch 2: 0.3 × 0.995^2 = 0.2970 (29.70% exploration)
Epoch 3: 0.3 × 0.995^3 = 0.2955 (29.55% exploration)
Epoch 4: 0.3 × 0.995^4 = 0.2940 (29.40% exploration)
Epoch 5: 0.3 × 0.995^5 = 0.2926 (29.26% exploration)

Actual (per step):

Step 1:     0.3 × 0.995^1 = 0.2985
Step 460:   0.3 × 0.995^460 ≈ 0.0500 (hit floor)
Step 21750: 0.3 × 0.995^21750 ≈ 0.000001 (clamped to 0.05)

Time to Floor:

0.3 × 0.995^n = 0.05
n = log(0.05/0.3) / log(0.995)
n ≈ 460 steps (2.1% of 21,750 steps)

Impact:

• Only 2.1% of training uses intended exploration rate (30% → 5%)
• 97.9% of training uses minimum exploration (5%)
• Insufficient exploration leads to premature convergence (96.4% HOLD)