17d94e654c
Wave 10 Summary: - A1-A4: Architecture upgrades (4x network, LeakyReLU, Xavier init, diagnostics) - A5-A6: Integration testing and production validation - A7: Research hyperopt vs manual tuning (manual recommended) - A8-A12: HOLD penalty tuning and critical bug fixes Architecture Changes: - Network expansion: [128,64,32] → [256,128,64] (2.5x parameters) - LeakyReLU activation (alpha=0.01) to prevent dead neurons - Xavier/Glorot initialization for better gradient flow - Real-time diagnostic monitoring (Q-values, dead neurons, gradients) Critical Bugs Fixed: - Bug #1: HOLD penalty not wired to reward calculation - Bug #2: Zero price error in calculate_hold_reward (velocity-based fix) - Huber loss default enabled (Wave 9) - Shape mismatch fix (Wave 8) Test Results: - Integration tests: 149/152 passing (98%) - New tests: 40+ tests added across 15 files - Xavier init: 5/5 tests passing - HOLD penalty wiring: 4/4 tests passing - Zero price fix: 4/4 tests passing Known Issues: - HOLD bias persists at ~100% despite penalties - Gradient collapse: 217 instances per training run (norm=0.0) - Reversed penalty effect: Higher penalties → worse Q-spread - Root cause: Gradient clipping bottleneck (max_norm=10.0 vs penalty signal) Phase 1 Trials (all completed without crashes): - Penalty 0.5: Q-spread 250 pts, HOLD 100% - Penalty 1.0: Q-spread 251 pts, HOLD 100% - Penalty 2.0: Q-spread 255 pts, HOLD 100% (+ Q-value explosion) Next Steps: Architectural investigation via parallel agent debugging 🤖 Generated with Claude Code (https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
118 lines
4.5 KiB
Plaintext
118 lines
4.5 KiB
Plaintext
================================================================================
|
|
W1-A2: DQN REWARD NORMALIZATION TESTS - QUICK REFERENCE
|
|
================================================================================
|
|
Date: 2025-11-05
|
|
Status: ✅ COMPLETE
|
|
Agent: W1-A2
|
|
Task: Unit tests for Fix #1 (Portfolio value normalization)
|
|
|
|
================================================================================
|
|
TEST RESULTS
|
|
================================================================================
|
|
✅ 7/7 tests PASSING (100% success rate)
|
|
✅ 0 regressions (132/132 existing DQN tests passing)
|
|
✅ Compilation clean (23.77s build time)
|
|
✅ Execution instant (0.00s test runtime)
|
|
|
|
================================================================================
|
|
FILE CREATED
|
|
================================================================================
|
|
Location: /home/jgrusewski/Work/foxhunt/ml/tests/dqn_reward_normalization_test.rs
|
|
Size: 310 lines
|
|
Tests: 7 comprehensive unit tests
|
|
|
|
================================================================================
|
|
TEST COVERAGE SUMMARY
|
|
================================================================================
|
|
1. test_buy_sell_reward_symmetry ✅
|
|
- BUY with +100 P&L → reward = 0.01
|
|
- SELL with +100 P&L → reward = 0.01
|
|
- Verifies: BUY/SELL symmetry
|
|
|
|
2. test_zero_pnl_symmetry ✅
|
|
- BUY with 0 P&L → reward = 0.0
|
|
- SELL with 0 P&L → reward = 0.0
|
|
- Verifies: Zero P&L handling
|
|
|
|
3. test_negative_pnl_symmetry ✅
|
|
- BUY with -100 P&L → reward = -0.01
|
|
- SELL with -100 P&L → reward = -0.01
|
|
- Verifies: Negative P&L symmetry
|
|
|
|
4. test_large_pnl_normalization ✅
|
|
- BUY with +1,000 P&L → reward = 0.1
|
|
- SELL with +1,000 P&L → reward = 0.1
|
|
- Verifies: Correct scaling (10% gain)
|
|
|
|
5. test_hold_action_reward ✅
|
|
- HOLD action → reward = 0.01 (configured)
|
|
- Verifies: HOLD independent of P&L
|
|
|
|
6. test_normalization_eliminates_portfolio_value_bias ✅
|
|
- Small portfolio (5,000) + 100 P&L → reward = 0.01
|
|
- Large portfolio (20,000) + 100 P&L → reward = 0.01
|
|
- Verifies: NO SIZE BIAS (critical!)
|
|
|
|
7. test_percentage_based_normalization ✅
|
|
- 5% gain (500/10,000) → reward = 0.05
|
|
- Verifies: Percentage-based rewards
|
|
|
|
================================================================================
|
|
FIX #1 VERIFICATION
|
|
================================================================================
|
|
BEFORE (BROKEN):
|
|
pnl_change / current_value ❌
|
|
Problem: BUY/SELL asymmetry, portfolio size bias
|
|
|
|
AFTER (FIXED):
|
|
pnl_change / INITIAL_CAPITAL (10,000) ✅
|
|
Result: BUY/SELL symmetry, no bias
|
|
|
|
Implementation: ml/src/dqn/reward.rs lines 133-136
|
|
|
|
================================================================================
|
|
KEY FINDINGS
|
|
================================================================================
|
|
✅ Fix #1 correctly implemented
|
|
✅ BUY/SELL rewards are now symmetric
|
|
✅ Normalization eliminates portfolio value bias
|
|
✅ Rewards scale as percentage of initial capital (10,000)
|
|
✅ Zero impact on existing DQN functionality
|
|
|
|
================================================================================
|
|
RUN TESTS
|
|
================================================================================
|
|
# Run normalization tests only
|
|
cargo test -p ml --test dqn_reward_normalization_test
|
|
|
|
# Run all DQN tests (verify no regressions)
|
|
cargo test -p ml --lib dqn
|
|
|
|
================================================================================
|
|
NEXT STEPS
|
|
================================================================================
|
|
1. ✅ Tests created (Agent W1-A2 complete)
|
|
2. ⏳ Merge test file into main branch
|
|
3. ⏳ Add to CI/CD pipeline
|
|
4. ⏳ Production deployment validation
|
|
|
|
================================================================================
|
|
SUCCESS CRITERIA
|
|
================================================================================
|
|
✅ 4/4 required tests → 7/7 tests (exceeds requirement)
|
|
✅ BUY/SELL symmetry verified
|
|
✅ Positive, zero, negative P&L covered
|
|
✅ Clear assertions with expected values
|
|
✅ Zero regressions
|
|
|
|
================================================================================
|
|
DOCUMENTATION
|
|
================================================================================
|
|
Full Report: W1_A2_REWARD_NORMALIZATION_TEST_REPORT.md
|
|
Test File: ml/tests/dqn_reward_normalization_test.rs
|
|
Fix Reference: ml/src/dqn/reward.rs (lines 133-136)
|
|
|
|
================================================================================
|
|
AGENT W1-A2 STATUS: ✅ COMPLETE
|
|
================================================================================
|