foxhunt/GAMMA_TEST_SUMMARY.md

Gamma 0.90 Test Campaign - Executive Summary

Test Objective

Investigate if reducing gamma from 0.9626 to 0.90 (56% noise amplification reduction) resolves gradient collapse and training instability issues.

Key Finding: GAMMA IS NOT THE ROOT CAUSE

Critical Evidence

1. Gradient collapse timing: Identical at step 20 for both gamma values
2. Collapse persistence: 100% zero gradients from step 20 onwards (both gammas)
3. Q-value behavior: Identical -333.33 stuck value after epoch 2
4. Loss stagnation: Identical 9.4-9.42 range, no decreasing trend

Test Results Summary

Metric	Gamma 0.9626 (Baseline)	Gamma 0.90 (Test)	Improvement
Gradient collapse step	20	20	❌ NONE
Epochs with grad=0	3-10	3-5+	❌ NONE
Q-value at epoch 3	-333.33	-333.33	❌ IDENTICAL
Loss convergence	NO	NO	❌ NO CHANGE
Action diversity	Collapsed	Collapsed	❌ NO CHANGE

Detailed Epoch Progression (Gamma 0.90)

Epoch 1: Q=+5.74,    grad=0.225,  loss=9.405  ✅ Healthy
Epoch 2: Q=-282.76,  grad=0.232,  loss=9.414  ⚠️ Degrading  
Epoch 3: Q=-333.33,  grad=0.000,  loss=9.398  ❌ COLLAPSED
Epoch 4: Q=-333.33,  grad=0.000,  loss=9.413  ❌ COLLAPSED
Epoch 5: Q=-333.33,  grad=0.000,  loss=9.420  ❌ COLLAPSED

What This Rules Out

• ❌ Gamma amplifying noise over long horizons
• ❌ Future value estimation instability
• ❌ Bootstrapping feedback loop issues

Actual Root Causes (Prioritized)

1. Reward Scale Mismatch 🔥 CRITICAL

• Current: 3197.23× adaptive scaling (from 0.0313% typical move)
• Issue: May not match actual P&L variance distribution
• Elite reward system may compound scaling problems
• Test next: SimplePnL reward (no multi-component complexity)

2. Optimizer Instability 🔥 HIGH PRIORITY

• Learning rate: 0.0001 (may be 10× too high)
• Gradient clipping: max_norm=10.0 (not preventing collapse)
• Adam epsilon: 1.5e-4 (Rainbow DQN standard, may need adjustment)
• Test next: LR=1e-5 (10× reduction)

3. TD-Error Explosion 🔥 HIGH PRIORITY

• Current clipping: ±10.0
• May allow Bellman update explosions
• Target network: Hard updates every 10K steps (sudden Q-value shifts)
• Test next: TD-error clip=±1.0 (10× tighter)

4. Network Architecture ⚠️ MEDIUM PRIORITY

• Hidden dims: [512, 256, 128, 64] (may be over-parameterized)
• Dead neurons: 0.00% (may be false negative - no activation monitoring)
• LeakyReLU alpha: 0.01 (may saturate)
• Test next: Simplified architecture [128, 64]

Recommended Next Steps

Immediate Priority (Next 2 Hours)

1. SimplePnL Reward Test (30 min)

   cargo run --release --example train_dqn --features cuda -- \
     --parquet-file test_data/ES_FUT_180d.parquet \
     --epochs 10 --reward-system SimplePnL --output-dir /tmp/ml_training/simplepnl_test
   

2. Learning Rate Reduction (15 min)

   cargo run --release --example train_dqn --features cuda -- \
     --parquet-file test_data/ES_FUT_180d.parquet \
     --epochs 10 --learning-rate 0.00001 --output-dir /tmp/ml_training/lr_1e5_test
   

3. Reward Scale Override (30 min)

   # Test 10× smaller scaling
   cargo run --release --example train_dqn --features cuda -- \
     --parquet-file test_data/ES_FUT_180d.parquet \
     --epochs 10 --reward-scale 300.0 --output-dir /tmp/ml_training/reward_scale_300_test
   

Success Criteria for Follow-Up Tests

• ✅ Gradient norm >1.0 at epoch 5
• ✅ Q-values stay in ±100 range
• ✅ Loss decreases by 50%+ from epoch 1 to 10
• ✅ No single action >50% at any epoch
• ✅ No gradient collapse before epoch 5

Files Generated

1. GAMMA_0.90_TEST_RESULTS.md - Complete diagnostic report
2. diagnostic_data/ - Extracted metrics (Q-values, gradients, epochs)
   • q_value_progression_gamma_0.90.txt
   • gradient_progression_gamma_0.90.txt
   • epoch_metrics_gamma_0.90.txt
   • gradient_collapse_count_gamma_0.90.txt

Conclusion

Gamma reduction from 0.9626 to 0.90 provides ZERO improvement. Gradient collapse persists identically, confirming that the discount factor is NOT the root cause. Focus investigation on reward system complexity (SimplePnL test) and optimizer instability (learning rate reduction).

DO NOT pursue further gamma tuning - resources better spent on reward/optimizer investigation.

---

Generated: 2025-11-10 Test Duration: ~8 minutes (10 epochs with gamma=0.90)