fix: gradient clip 10.0→1.0 — prevents grad accumulation with complex reward

grad_norm was growing 5x per epoch (523→2673→13K→67K→NaN). The old
clip at 10.0 allowed gradients to accumulate. With clip=1.0, the
Adam optimizer receives bounded updates.

Trial 2 (TPE-guided params) trains cleanly for 4 epochs:
  train_loss: 4.5→3.3 (decreasing!)
  Q-value: 12-19 (stable)
  grad_norm: 162-567 (bounded)

Trial 1 (random initial params) still NaN's — this is expected and
handled by the hyperopt penalty (1M objective). The optimizer learns
to avoid unstable parameter regions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-03-23 14:40:30 +01:00
parent 1936c160b5
commit f5cb953082

View File

@@ -1262,7 +1262,7 @@ impl DQNHyperparameters {
use_huber_loss: true, // Default: Huber loss enabled (more robust)
huber_delta: 100.0, // BUG #12 FIX: Scale delta 100x for gradient explosion fix (was 1.0)
use_double_dqn: true, // Default: Double DQN enabled (prevents overestimation bias)
gradient_clip_norm: Some(10.0), // REVERTED: Back to 10.0 default (production standard)
gradient_clip_norm: Some(1.0), // Tight clip — prevents grad explosion with complex reward
hold_penalty_weight: 0.01, // Default: 1% penalty weight
movement_threshold: 0.02, // Default: 2% price movement threshold
enable_preprocessing: true, // Default: preprocessing enabled (Wave 14 Agent 32)