00ef9e2866
Major Changes: - Migrated from 3-action TradingAction to 45-action FactoredAction - 45 actions: 5 exposure × 3 order types × 3 urgency levels - Absolute exposure model (target positions -1.0 to +1.0) - Transaction cost differentiation (Market 0.15%, LimitMaker 0.05%, IoC 0.10%) - Fixed action diversity threshold (1.11% → 0.5% for 45-action space) Bug Fixes: - Bug #15: Incomplete FactoredAction integration (code existed but unused) - Bug #16: Runtime crash in action diversity checking (hardcoded 3-action match) Code Changes (13 files, ~464 lines): - ml/src/dqn/action_space.rs: Core FactoredAction + 4 helper methods - ml/src/trainers/dqn.rs: Action diversity refactored (3→45 dynamic) - ml/src/dqn/reward.rs: calculate_reward() signature updated - ml/src/dqn/portfolio_tracker.rs: execute_action() absolute exposure - ml/src/dqn/dqn.rs: WorkingDQN action selection migrated - ml/tests/*.rs: 9 test files updated with FactoredAction assertions Test Results: - 1-epoch smoke test: 100% action diversity (45/45 actions, 80.2s) - 10-epoch production: 87.8% readiness (79/90 scorecard, 14.0 min) - Loss convergence: 96.9% reduction (119K → 3.6K) - Action diversity: 100% → 44% (healthy specialization) - Checkpoint reliability: 12/12 files saved (100%) - DQN tests: 195/195 passing (100%) - ML baseline: 1,514/1,515 passing (99.93%) Production Status: ✅ CERTIFIED (87.8% readiness) Go/No-Go: ✅ GO FOR 100-EPOCH PRODUCTION TRAINING 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
112 lines
4.5 KiB
Markdown
112 lines
4.5 KiB
Markdown
# Gamma 0.90 Test Campaign - Executive Summary
|
||
|
||
## Test Objective
|
||
Investigate if reducing gamma from 0.9626 to 0.90 (56% noise amplification reduction) resolves gradient collapse and training instability issues.
|
||
|
||
## Key Finding: **GAMMA IS NOT THE ROOT CAUSE**
|
||
|
||
### Critical Evidence
|
||
1. **Gradient collapse timing**: Identical at step 20 for both gamma values
|
||
2. **Collapse persistence**: 100% zero gradients from step 20 onwards (both gammas)
|
||
3. **Q-value behavior**: Identical -333.33 stuck value after epoch 2
|
||
4. **Loss stagnation**: Identical 9.4-9.42 range, no decreasing trend
|
||
|
||
## Test Results Summary
|
||
|
||
| Metric | Gamma 0.9626 (Baseline) | Gamma 0.90 (Test) | Improvement |
|
||
|--------|-------------------------|-------------------|-------------|
|
||
| Gradient collapse step | 20 | 20 | ❌ NONE |
|
||
| Epochs with grad=0 | 3-10 | 3-5+ | ❌ NONE |
|
||
| Q-value at epoch 3 | -333.33 | -333.33 | ❌ IDENTICAL |
|
||
| Loss convergence | NO | NO | ❌ NO CHANGE |
|
||
| Action diversity | Collapsed | Collapsed | ❌ NO CHANGE |
|
||
|
||
### Detailed Epoch Progression (Gamma 0.90)
|
||
```
|
||
Epoch 1: Q=+5.74, grad=0.225, loss=9.405 ✅ Healthy
|
||
Epoch 2: Q=-282.76, grad=0.232, loss=9.414 ⚠️ Degrading
|
||
Epoch 3: Q=-333.33, grad=0.000, loss=9.398 ❌ COLLAPSED
|
||
Epoch 4: Q=-333.33, grad=0.000, loss=9.413 ❌ COLLAPSED
|
||
Epoch 5: Q=-333.33, grad=0.000, loss=9.420 ❌ COLLAPSED
|
||
```
|
||
|
||
## What This Rules Out
|
||
- ❌ Gamma amplifying noise over long horizons
|
||
- ❌ Future value estimation instability
|
||
- ❌ Bootstrapping feedback loop issues
|
||
|
||
## Actual Root Causes (Prioritized)
|
||
|
||
### 1. **Reward Scale Mismatch** 🔥 CRITICAL
|
||
- Current: 3197.23× adaptive scaling (from 0.0313% typical move)
|
||
- Issue: May not match actual P&L variance distribution
|
||
- Elite reward system may compound scaling problems
|
||
- **Test next**: SimplePnL reward (no multi-component complexity)
|
||
|
||
### 2. **Optimizer Instability** 🔥 HIGH PRIORITY
|
||
- Learning rate: 0.0001 (may be 10× too high)
|
||
- Gradient clipping: max_norm=10.0 (not preventing collapse)
|
||
- Adam epsilon: 1.5e-4 (Rainbow DQN standard, may need adjustment)
|
||
- **Test next**: LR=1e-5 (10× reduction)
|
||
|
||
### 3. **TD-Error Explosion** 🔥 HIGH PRIORITY
|
||
- Current clipping: ±10.0
|
||
- May allow Bellman update explosions
|
||
- Target network: Hard updates every 10K steps (sudden Q-value shifts)
|
||
- **Test next**: TD-error clip=±1.0 (10× tighter)
|
||
|
||
### 4. **Network Architecture** ⚠️ MEDIUM PRIORITY
|
||
- Hidden dims: [512, 256, 128, 64] (may be over-parameterized)
|
||
- Dead neurons: 0.00% (may be false negative - no activation monitoring)
|
||
- LeakyReLU alpha: 0.01 (may saturate)
|
||
- **Test next**: Simplified architecture [128, 64]
|
||
|
||
## Recommended Next Steps
|
||
|
||
### Immediate Priority (Next 2 Hours)
|
||
1. **SimplePnL Reward Test** (30 min)
|
||
```bash
|
||
cargo run --release --example train_dqn --features cuda -- \
|
||
--parquet-file test_data/ES_FUT_180d.parquet \
|
||
--epochs 10 --reward-system SimplePnL --output-dir /tmp/ml_training/simplepnl_test
|
||
```
|
||
|
||
2. **Learning Rate Reduction** (15 min)
|
||
```bash
|
||
cargo run --release --example train_dqn --features cuda -- \
|
||
--parquet-file test_data/ES_FUT_180d.parquet \
|
||
--epochs 10 --learning-rate 0.00001 --output-dir /tmp/ml_training/lr_1e5_test
|
||
```
|
||
|
||
3. **Reward Scale Override** (30 min)
|
||
```bash
|
||
# Test 10× smaller scaling
|
||
cargo run --release --example train_dqn --features cuda -- \
|
||
--parquet-file test_data/ES_FUT_180d.parquet \
|
||
--epochs 10 --reward-scale 300.0 --output-dir /tmp/ml_training/reward_scale_300_test
|
||
```
|
||
|
||
### Success Criteria for Follow-Up Tests
|
||
- ✅ Gradient norm >1.0 at epoch 5
|
||
- ✅ Q-values stay in ±100 range
|
||
- ✅ Loss decreases by 50%+ from epoch 1 to 10
|
||
- ✅ No single action >50% at any epoch
|
||
- ✅ No gradient collapse before epoch 5
|
||
|
||
## Files Generated
|
||
1. **GAMMA_0.90_TEST_RESULTS.md** - Complete diagnostic report
|
||
2. **diagnostic_data/** - Extracted metrics (Q-values, gradients, epochs)
|
||
- q_value_progression_gamma_0.90.txt
|
||
- gradient_progression_gamma_0.90.txt
|
||
- epoch_metrics_gamma_0.90.txt
|
||
- gradient_collapse_count_gamma_0.90.txt
|
||
|
||
## Conclusion
|
||
Gamma reduction from 0.9626 to 0.90 provides **ZERO improvement**. Gradient collapse persists identically, confirming that the discount factor is NOT the root cause. Focus investigation on reward system complexity (SimplePnL test) and optimizer instability (learning rate reduction).
|
||
|
||
**DO NOT pursue further gamma tuning** - resources better spent on reward/optimizer investigation.
|
||
|
||
---
|
||
Generated: 2025-11-10
|
||
Test Duration: ~8 minutes (10 epochs with gamma=0.90)
|