foxhunt/WAVE_16L_POLYAK_SOFT_UPDATES.md

Wave 16L: Polyak Soft Updates Investigation - Gradient Collapse NOT FIXED

Date: 2025-11-10 Duration: ~1 hour Status: ❌ FAILED - Polyak does NOT fix gradient collapse Conclusion: The gradient collapse issue is NOT caused by target update strategy

---

Executive Summary

Polyak soft target updates were ALREADY IMPLEMENTED in Wave 16 (Agent 36) but disabled by default. Investigation reveals that enabling Polyak averaging (tau=0.005) does NOT fix the gradient collapse issue. Gradients remain at exactly 0.000000 throughout all training steps, regardless of hard vs soft target updates.

Critical Finding: The gradient collapse is caused by a different root cause - likely related to reward signals, loss computation, or optimizer configuration.

---

Investigation Results

1. Polyak Implementation Status: ✅ ALREADY IMPLEMENTED

Code locations:

• /home/jgrusewski/Work/foxhunt/ml/src/dqn/target_update.rs (276 lines, full implementation + unit tests)
• /home/jgrusewski/Work/foxhunt/ml/src/dqn/dqn.rs (lines 87-91, 209-211, 912-931)
• /home/jgrusewski/Work/foxhunt/ml/examples/train_dqn.rs (lines 178-188, 306-313, 506-512)

Functions:

• polyak_update(online_vars, target_vars, tau) - Soft target blending (θ_target = τ*θ_online + (1-τ)*θ_target)
• hard_update(online_vars, target_vars) - Complete replacement (legacy)
• convergence_half_life(tau) - Calculate convergence speed

CLI Flags:

• --soft-updates: Enable Polyak averaging (default: false)
• --tau <float>: Polyak coefficient (default: 1.0 for hard updates)
  • Rainbow DQN standard: 0.001 (693-step half-life)
  • Moderate: 0.005 (138-step half-life)
  • Aggressive: 0.01 (69-step half-life)

Unit Tests: 6/6 passing (ml/src/dqn/target_update.rs lines 143-276)

• test_polyak_single_update
• test_gradual_convergence
• test_hard_update_correctness
• test_convergence_half_life_calculation
• test_invalid_tau_negative
• test_invalid_tau_too_large

---

2. Test Results: ❌ POLYAK DOES NOT FIX GRADIENT COLLAPSE

Test Configuration:

cargo run --release --package ml --example train_dqn --features cuda -- \
  --parquet-file test_data/ES_FUT_180d.parquet \
  --epochs 10 \
  --soft-updates \
  --tau 0.005 \
  --output-dir /tmp/ml_training/polyak_soft_updates_test

Logs: /tmp/ml_training/polyak_soft_updates_test.log

Observed Behavior:

INFO: 🎯 WAVE 16: Using soft target updates (Polyak averaging)
INFO:   • Tau: 0.005
INFO:   • Convergence half-life: 138 steps
INFO:   • Strategy: Smooth Q-value tracking (50-70% variance reduction)

Step 10:  grad=0.0000, loss=9.3970, Q=[BUY:104.49, SELL:-46.96, HOLD:-549.77]
Step 20:  grad=0.0000, loss=9.3970, Q=[BUY:95.53, SELL:103.77, HOLD:-579.90]
Step 100: grad=0.0000, loss=9.3970, Q=[BUY:88.70, SELL:199.22, HOLD:-593.74]
Step 200: grad=0.0000, loss=9.3970, Q=[BUY:80.77, SELL:197.46, HOLD:-551.05]
Step 300: grad=0.0000, loss=9.3970, Q=[BUY:85.65, SELL:198.24, HOLD:-576.82]
Step 600: grad=0.0000, loss=9.3970, Q=[BUY:88.52, SELL:198.37, HOLD:-591.42]

WARN: ⚠️  GRADIENT COLLAPSE: norm=0.000000 at step 100
WARN: ⚠️  GRADIENT COLLAPSE: norm=0.000000 at step 200
WARN: ⚠️  GRADIENT COLLAPSE: norm=0.000000 at step 300
WARN: ⚠️  GRADIENT COLLAPSE: norm=0.000000 at step 600

Comparison: Hard vs Soft Updates:

Metric	Hard Updates (tau=1.0)	Soft Updates (tau=0.005)	Change
Gradient Norm	0.000000	0.000000	❌ NO CHANGE
Loss	9.3970 (stuck)	9.3970 (stuck)	❌ NO CHANGE
Q-values	Wild swings	Wild swings	❌ NO CHANGE
Action Distribution	Unknown	Unknown	❌ NO CHANGE
Dead Neurons	0.00%	0.00%	❌ NO CHANGE

---

Root Cause Analysis

❌ Ruled Out: Target Update Strategy

Evidence:

1. Polyak soft updates (tau=0.005) produce IDENTICAL gradient collapse
2. Loss stuck at 9.3970 regardless of update mode
3. Gradient norm=0.000000 in BOTH hard and soft update modes
4. Q-values fluctuate but no gradients flow backwards

Conclusion: Target update strategy (hard vs soft) is NOT the cause of gradient collapse.

---

🔍 Likely Root Causes (Investigation Required)

1. Reward Signal Issues (🔴 HIGHEST PRIORITY)

Hypothesis: Elite reward system may be generating zero or constant rewards, leading to zero TD-errors.

Evidence:

• Loss stuck at 9.3970 (no learning signal)
• Q-values fluctuate wildly but gradients are zero
• Reward normalization scale: 3197.23x (very high scaling factor)
• Elite reward system uses 5 components (extrinsic, intrinsic, entropy, curiosity, ensemble)

Investigation Script:

# Test with SimplePnL reward system (pure P&L, no multi-component complexity)
cargo run --release --package ml --example train_dqn --features cuda -- \
  --parquet-file test_data/ES_FUT_180d.parquet \
  --epochs 10 \
  --reward-system simplepnl \
  --output-dir /tmp/ml_training/simplepnl_gradient_test \
  2>&1 | tee /tmp/ml_training/simplepnl_gradient_test.log

# Check for non-zero gradients:
grep "grad_norm" /tmp/ml_training/simplepnl_gradient_test.log | head -20

Expected: If SimplePnL restores gradients, Elite reward system is the culprit.

---

2. TD-Error Clipping Too Aggressive (🟠 HIGH PRIORITY)

Hypothesis: TD-error clipping (default 10.0) may be zeroing out all gradients before backpropagation.

Evidence:

• td_error_clip: 10.0 (Wave 4 Agent 1: prevents noise amplification)
• Gradients clipped to exactly 0.000000 (not just small)
• Loss never changes (9.3970 constant across all steps)

Investigation Script:

# Test with TD-error clipping disabled (set to 1000.0)
cargo run --release --package ml --example train_dqn --features cuda -- \
  --parquet-file test_data/ES_FUT_180d.parquet \
  --epochs 10 \
  --td-error-clip 1000.0 \
  --output-dir /tmp/ml_training/no_td_clip_test \
  2>&1 | tee /tmp/ml_training/no_td_clip_test.log

# Check gradient norms:
grep "grad_norm" /tmp/ml_training/no_td_clip_test.log | head -20

Expected: If TD-error clipping is too aggressive, disabling it should restore non-zero gradients.

---

3. Optimizer Configuration (🟡 MEDIUM PRIORITY)

Hypothesis: Adam optimizer epsilon (1.5e-4, from Wave 16H) may be suppressing small gradients.

Evidence:

• Adam eps: 1.5e-4 (Rainbow DQN standard for numerical stability)
• Standard PyTorch eps: 1e-8 (10,000x smaller)
• Large epsilon can suppress gradient updates when gradient magnitudes are small

Code Change Required (ml/src/dqn/dqn.rs:726):

let adam_params = ParamsAdam {
    lr: self.config.learning_rate,
    beta_1: 0.9,
    beta_2: 0.999,
    eps: 1e-8,  // CHANGE FROM 1.5e-4 to 1e-8
};

Expected: Smaller epsilon should allow tiny gradients to propagate through Adam updates.

---

4. Gradient Clipping Configuration (🟡 LOW PRIORITY)

Hypothesis: Gradient clipping (max_norm=10.0, Wave D Bug #1 fix) may be incorrectly zeroing gradients.

Evidence:

• gradient_clip_norm: 10.0 (Wave D: prevents gradient explosions)
• Gradients are 0.000000 (not just reduced magnitude)
• Clipping should reduce magnitude, not zero out completely

Investigation Script:

# Test with gradient clipping disabled (set to 1000.0)
cargo run --release --package ml --example train_dqn --features cuda -- \
  --parquet-file test_data/ES_FUT_180d.parquet \
  --epochs 10 \
  --gradient-clip-norm 1000.0 \
  --output-dir /tmp/ml_training/no_grad_clip_test \
  2>&1 | tee /tmp/ml_training/no_grad_clip_test.log

Expected: Gradients should NOT change (clipping only reduces magnitude, should not zero out).

---

5. Loss Computation Bug (🟡 MEDIUM PRIORITY)

Hypothesis: Huber loss with delta=10.0 may have a bug that returns zero gradients.

Evidence:

• use_huber_loss: true (default)
• huber_delta: 10.0 (handles TD-errors up to ±10)
• Loss stuck at 9.3970 (constant, no learning)

Code Change Required (ml/examples/train_dqn.rs - ADD CLI FLAG):

/// Use MSE loss instead of Huber loss (for debugging)
#[arg(long)]
no_huber_loss: bool,

Then test with MSE loss (simpler, more standard):

cargo run --release --package ml --example train_dqn --features cuda -- \
  --parquet-file test_data/ES_FUT_180d.parquet \
  --epochs 10 \
  --no-huber-loss \
  --output-dir /tmp/ml_training/mse_loss_test

Expected: If MSE restores gradients, Huber loss implementation has a bug.

---

Recommended Investigation Sequence

Phase 1: Reward System (🔴 IMMEDIATE - 30 MIN)

Priority: CRITICAL - Most likely root cause

Steps:

1. Test with SimplePnL reward system (--reward-system simplepnl)
2. Test with reward_normalization_scale=1.0 (disable adaptive scaling)
3. Compare gradient norms and loss curves

Expected: One of these tests should restore non-zero gradients.

Success Criteria: grad_norm > 10.0 at step 100

---

Phase 2: TD-Error Clipping (🟠 30 MIN)

Priority: HIGH - Second most likely cause

Steps:

1. Test with td_error_clip=1000.0 (effectively disabled)
2. Test with td_error_clip=100.0 (10x larger)
3. Compare gradient norms at steps 10, 50, 100

Expected: Gradients should become non-zero if clipping is the issue.

Success Criteria: grad_norm > 10.0 consistently

---

Phase 3: Optimizer Configuration (🟡 30 MIN)

Priority: MEDIUM - Possible contributor

Steps:

1. Change Adam epsilon from 1.5e-4 to 1e-8 (PyTorch standard)
2. Test with learning_rate=0.001 (10x higher, more aggressive)
3. Compare gradient norms and Q-value convergence

Expected: Larger learning rate should amplify gradient signals.

Success Criteria: grad_norm > 50.0 (higher due to 10x LR)

---

Phase 4: Loss Function (🟡 30 MIN)

Priority: MEDIUM - Unlikely but possible

Steps:

1. Add --no-huber-loss CLI flag (requires small code change)
2. Test with MSE loss (simpler, more standard)
3. Compare loss curves and gradient norms

Expected: MSE should behave similarly (Huber unlikely culprit).

Success Criteria: Gradients should be similar to Huber (if bug, will differ)

---

Key Insights

✅ Polyak Implementation is Correct and Complete

• Full implementation with 6/6 unit tests passing
• Convergence half-life calculation accurate (693 steps for tau=0.001)
• Soft updates execute every step (not just every 1000 steps)
• CLI flags functional and well-documented
• Integration with training loop correct (ml/src/dqn/dqn.rs lines 912-931)

❌ Polyak Does NOT Fix Gradient Collapse

• Gradient norm=0.000000 in BOTH hard and soft update modes
• Loss stuck at 9.3970 regardless of target update strategy
• Q-values fluctuate wildly but no learning occurs
• Action distribution likely unchanged (unable to verify due to gradient collapse)

🔍 Root Cause is Elsewhere

The gradient collapse issue is NOT caused by target update strategy. Investigation must shift focus to:

1. Reward system (Elite multi-component may generate zero/constant rewards)
2. TD-error clipping (10.0 threshold may be too aggressive)
3. Optimizer configuration (Adam epsilon 1.5e-4 may suppress small gradients)
4. Loss computation (Huber loss with delta=10.0 may have bugs)

---

User Clarification

User Question: "I thought you already implemented the Polyak (with tau)"

Answer: YES, Polyak was already implemented in Wave 16 (Agent 36), but:

1. It was disabled by default (use_soft_updates: false, tau: 1.0)
2. The logs showed "WAVE 16: Using hard target updates (legacy mode)"
3. Investigation revealed that enabling Polyak does NOT fix the gradient collapse

Critical Finding: The gradient collapse is NOT related to target updates (hard vs soft). The problem lies elsewhere - likely in the reward system, TD-error clipping, or optimizer configuration.

---

Files Referenced

Implementation Files (No Changes Required)

• /home/jgrusewski/Work/foxhunt/ml/src/dqn/target_update.rs (276 lines)
• /home/jgrusewski/Work/foxhunt/ml/src/dqn/dqn.rs (1,019 lines)
• /home/jgrusewski/Work/foxhunt/ml/examples/train_dqn.rs (1,000+ lines)
• /home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn.rs (2,300+ lines)
• /home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/dqn.rs (600+ lines)

Test Logs

• /tmp/ml_training/polyak_soft_updates_test.log (Polyak tau=0.005 test)
• /tmp/ml_training/wave5_simple_pnl_test.log (Previous hard update test)

---

Next Actions

Immediate (TODAY)

1. Test SimplePnL reward system (30 min)

   cargo run --release --package ml --example train_dqn --features cuda -- \
     --parquet-file test_data/ES_FUT_180d.parquet \
     --epochs 10 \
     --reward-system simplepnl \
     --output-dir /tmp/ml_training/simplepnl_gradient_test
   

   Expected: Restore non-zero gradients if Elite reward system is the issue.

2. Test TD-error clipping disabled (30 min)

   cargo run --release --package ml --example train_dqn --features cuda -- \
     --parquet-file test_data/ES_FUT_180d.parquet \
     --epochs 10 \
     --td-error-clip 1000.0 \
     --output-dir /tmp/ml_training/no_td_clip_test
   

   Expected: Restore non-zero gradients if TD-error clipping is too aggressive.

3. Analyze reward signals (15 min)

   • Check: Are rewards non-zero in logs?
   • Check: Are TD-errors within [-10, +10] range?
   • Check: Is loss computation correct?

---

Conclusion

Polyak soft target updates are fully implemented and functional, but do NOT fix the gradient collapse issue. The root cause lies elsewhere - most likely in the reward system (Elite multi-component generating zero/constant rewards) or TD-error clipping (10.0 threshold too aggressive). Immediate investigation of reward signals and TD-error clipping required to restore gradient flow and enable actual learning.

Status: ❌ FAILED - Polyak implementation correct, but gradient collapse persists due to different root cause.

Next Priority: Reward system and TD-error clipping investigation (Phase 1-2, 60 minutes total).

---

References

Wave 16 Documentation:

• Agent 36: Polyak soft updates implementation
• ml/src/dqn/target_update.rs: Full implementation with 6 unit tests
• ml/src/dqn/dqn.rs: Integration with WorkingDQN (lines 912-931)
• ml/examples/train_dqn.rs: CLI flags (lines 178-188, 306-313, 506-512)

Related Waves:

• Wave 4 Agent 1: TD-error clipping (gradient collapse prevention)
• Wave 16H: Adam epsilon (1.5e-4 for numerical stability)
• Wave D Bug #1: Gradient clipping (max_norm=10.0)
• Wave 10: Elite reward system (5-component multi-objective)