17d94e654c
Wave 10 Summary: - A1-A4: Architecture upgrades (4x network, LeakyReLU, Xavier init, diagnostics) - A5-A6: Integration testing and production validation - A7: Research hyperopt vs manual tuning (manual recommended) - A8-A12: HOLD penalty tuning and critical bug fixes Architecture Changes: - Network expansion: [128,64,32] → [256,128,64] (2.5x parameters) - LeakyReLU activation (alpha=0.01) to prevent dead neurons - Xavier/Glorot initialization for better gradient flow - Real-time diagnostic monitoring (Q-values, dead neurons, gradients) Critical Bugs Fixed: - Bug #1: HOLD penalty not wired to reward calculation - Bug #2: Zero price error in calculate_hold_reward (velocity-based fix) - Huber loss default enabled (Wave 9) - Shape mismatch fix (Wave 8) Test Results: - Integration tests: 149/152 passing (98%) - New tests: 40+ tests added across 15 files - Xavier init: 5/5 tests passing - HOLD penalty wiring: 4/4 tests passing - Zero price fix: 4/4 tests passing Known Issues: - HOLD bias persists at ~100% despite penalties - Gradient collapse: 217 instances per training run (norm=0.0) - Reversed penalty effect: Higher penalties → worse Q-spread - Root cause: Gradient clipping bottleneck (max_norm=10.0 vs penalty signal) Phase 1 Trials (all completed without crashes): - Penalty 0.5: Q-spread 250 pts, HOLD 100% - Penalty 1.0: Q-spread 251 pts, HOLD 100% - Penalty 2.0: Q-spread 255 pts, HOLD 100% (+ Q-value explosion) Next Steps: Architectural investigation via parallel agent debugging 🤖 Generated with Claude Code (https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
108 lines
3.5 KiB
Plaintext
108 lines
3.5 KiB
Plaintext
WAVE 9-A4: HUBER LOSS VALIDATION - QUICK REFERENCE
|
|
==================================================
|
|
Date: 2025-11-05
|
|
Objective: Validate Huber loss default reduces action bias
|
|
Result: ❌ FAILED - Huber loss does NOT fix action bias
|
|
|
|
CRITICAL FINDINGS
|
|
-----------------
|
|
1. Huber loss CORRECTLY implemented and enabled
|
|
2. Action bias UNCHANGED: 96.7% BUY (vs 96.6% SELL in Wave 8)
|
|
3. Gradient collapse at step 210: 36,341 → 0.80 (4,561x drop)
|
|
4. Q-values collapsed to 0.0000 for all actions
|
|
5. Loss became NEGATIVE at step 1010 (mathematically impossible)
|
|
|
|
TRAINING METRICS (10 EPOCHS)
|
|
-----------------------------
|
|
Training steps: 43,500 ✅ Target reached
|
|
Training loss: 0.006215 ✅ Decreasing
|
|
Validation loss: 0.000290 ✅ Low
|
|
Avg Q-value: 0.0156 ❌ COLLAPSED
|
|
Gradient norm: 0.0158 ❌ COLLAPSED
|
|
Duration: 100.88s ✅ Fast
|
|
|
|
ACTION DISTRIBUTION (EPOCH 10)
|
|
-------------------------------
|
|
BUY: 96.7% (134,596) ❌ EXTREME BIAS
|
|
SELL: 1.6% (2,255) ❌ SUPPRESSED
|
|
HOLD: 1.7% (2,351) ❌ SUPPRESSED
|
|
|
|
GRADIENT COLLAPSE TIMELINE
|
|
---------------------------
|
|
Step 1-200: grad=30,000-40,000 ✅ Healthy
|
|
Step 210: grad=0.80 ❌ COLLAPSED (99.998% drop)
|
|
Step 1000+: grad=0.001-0.03 ❌ Near-zero
|
|
|
|
LOSS TRAJECTORY
|
|
---------------
|
|
Step 1-1000: loss=3000-500 ✅ Positive, decreasing
|
|
Step 1010: loss=-0.0822 ❌ NEGATIVE (impossible)
|
|
Step 1010+: loss=-0.08 ❌ Stuck negative
|
|
|
|
ROOT CAUSE ANALYSIS
|
|
-------------------
|
|
Problem: ARCHITECTURAL, not algorithmic
|
|
- Network too small (64, 32 neurons)
|
|
- ReLU neurons dying after step 210
|
|
- Numerical instability causing negative loss
|
|
- Q-values collapsing to exactly zero
|
|
|
|
What Huber Loss DOES:
|
|
- ✅ Reduces outlier impact
|
|
- ✅ Stabilizes gradients from extreme TD errors
|
|
- ✅ More robust to noisy rewards
|
|
|
|
What Huber Loss CANNOT DO:
|
|
- ❌ Fix network capacity issues
|
|
- ❌ Prevent dead ReLU units
|
|
- ❌ Solve vanishing gradients
|
|
- ❌ Fix numerical instability
|
|
|
|
RECOMMENDATIONS (PRIORITY ORDER)
|
|
--------------------------------
|
|
P1: ARCHITECTURAL CHANGES (IMMEDIATE)
|
|
1. Increase network size: [256, 128, 64] vs [64, 32]
|
|
2. Use LeakyReLU(0.01) instead of ReLU
|
|
3. Xavier/Glorot initialization for all layers
|
|
|
|
P2: HYPERPARAMETER TUNING
|
|
1. Lower learning rate: 0.00001 (10x lower)
|
|
2. Increase target update freq: 100 (vs 1000)
|
|
3. Slow epsilon decay: 0.999 (vs 0.995)
|
|
|
|
P3: TRAINING DYNAMICS
|
|
1. Add LR scheduling: 0.0001 → 0.00001
|
|
2. Clip gradients by value: 1.0
|
|
3. Add batch normalization after each layer
|
|
|
|
P4: DIAGNOSTIC TOOLS
|
|
1. Monitor Q-values per action every 10 steps
|
|
2. Track % dead ReLU units every 100 steps
|
|
3. Log separate losses for BUY/SELL/HOLD
|
|
|
|
CONCLUSION
|
|
----------
|
|
Huber loss is WORKING but does NOT address the root cause.
|
|
Action bias persists because the Q-network is COLLAPSING due to:
|
|
- Insufficient capacity (too few neurons)
|
|
- Dead ReLU units (neurons stuck at zero)
|
|
- Vanishing gradients (too small to propagate)
|
|
|
|
NEXT STEPS: Implement P1 architectural changes IMMEDIATELY.
|
|
Abandon Huber loss investigation - it's not the problem.
|
|
|
|
FILES REFERENCED
|
|
----------------
|
|
- /home/jgrusewski/Work/foxhunt/ml/src/dqn/dqn.rs (lines 91, 539-567)
|
|
- /home/jgrusewski/Work/foxhunt/ml/examples/train_dqn.rs (line 285)
|
|
- /tmp/wave9_huber_production.log (training output)
|
|
|
|
COMPARISON TO WAVE 8
|
|
--------------------
|
|
Wave 8 (MSE loss): 96.6% SELL bias
|
|
Wave 9 (Huber loss): 96.7% BUY bias
|
|
Delta: +0.1% bias, action flipped
|
|
|
|
Conclusion: Loss function does NOT affect action bias.
|
|
Problem is deeper (architectural/capacity/dynamics).
|