6631ace502
6 parallel agents completed comprehensive investigation of 100% HOLD bias. ROOT CAUSES IDENTIFIED: - Bug #1 (CRITICAL): Xavier init bypasses VarMap → optimizer has 0 params → no learning Status: ✅ ALREADY FIXED by Agent A15 - Bug #2 (CATASTROPHIC): scale_gradients() corrupts weights 217x/run → training destroyed Status: ⚠️ NEEDS FIX (lib.rs lines 269-281) - Bug #3 (CRITICAL): Production loop uses wrong rewards (-0.0001 vs ±1.0) → 100% HOLD Status: ⚠️ NEEDS FIX (trainers/dqn.rs lines 869-890) ADDITIONAL ISSUES: - A14: Movement threshold too high (2% > 1.88% data) → penalty never activates - A17: 4 numerical stability bugs (unbounded rewards, Q-explosions, no clamping) - A16: ✅ Action selection verified working (7/7 tests pass) EVIDENCE CORRELATION: - 217 gradient collapses = 217 weight corruption events (Bug #2) - 100% HOLD bias = wrong reward system makes HOLD safest (Bug #3) - Reversed penalty effect = larger gradients → more corruption (Bug #2) - Q-value explosions (+24,055) = corrupted 0.001-scale weights (Bug #2) DOCUMENTATION CREATED: - WAVE10_DEBUG_SYNTHESIS.md (8,500 words) - Complete analysis + fix roadmap - WAVE10_FIX_QUICK_REF.txt (2,000 words) - Copy-paste ready fixes - 6 individual agent reports with test validation IMPLEMENTATION TIMELINE: - Phase 1 (Critical): 60 min - 3 fixes to restore learning - Phase 2 (High Priority): 40 min - Numerical stability - Validation: 30 min - Tests + smoke test + production run - Total: 2.5-3 hours to production-ready DQN EXPECTED OUTCOMES: - Action distribution: 100% HOLD → ~30/30/40 (BUY/SELL/HOLD) - Gradient collapses: 217/run → 0/run - Q-value max: +24,055 → <1000 - Learning: NONE → OPERATIONAL - Optimizer params: 0 → 99,200 Next: Implement all fixes in parallel waves
247 lines
17 KiB
Plaintext
247 lines
17 KiB
Plaintext
╔══════════════════════════════════════════════════════════════════════════════╗
|
|
║ WAVE 10 DEBUGGING CAMPAIGN - FIX QUICK REFERENCE ║
|
|
║ 6 Agents, 3 Critical Bugs Found ║
|
|
╚══════════════════════════════════════════════════════════════════════════════╝
|
|
|
|
🚨 CRITICAL BUGS (BLOCKS ALL LEARNING):
|
|
|
|
┌─ BUG #1: Xavier Initialization (A15) ─────────────────────────────────────┐
|
|
│ SEVERITY: CRITICAL - Optimizer has ZERO parameters │
|
|
│ LOCATION: ml/src/dqn/dqn.rs:186-202 │
|
|
│ SYMPTOM: Gradient norm always 0.0000, no learning │
|
|
│ │
|
|
│ ROOT CAUSE: Raw Tensor creation bypasses VarMap registration │
|
|
│ ❌ let weights = xavier_uniform(...)?; // Not in VarMap │
|
|
│ ❌ let layer = Linear::new(weights, bias); │
|
|
│ │
|
|
│ FIX (15 min): │
|
|
│ Line 15: Add import │
|
|
│ use crate::dqn::xavier_init::linear_xavier; │
|
|
│ │
|
|
│ Lines 186-202: Replace constructor │
|
|
│ let var_builder = VarBuilder::from_varmap(&vars, DType::F32, &device);│
|
|
│ for (i, &hidden_dim) in hidden_dims.into_iter().enumerate() { │
|
|
│ let layer_vb = var_builder.pp(&format!("hidden_{}", i)); │
|
|
│ let layer = linear_xavier(current_dim, hidden_dim, layer_vb)?; │
|
|
│ layers.push(layer); │
|
|
│ } │
|
|
│ let output_vb = var_builder.pp("output"); │
|
|
│ let output = linear_xavier(current_dim, output_dim, output_vb)?; │
|
|
│ │
|
|
│ VALIDATION: cargo test --test dqn_gradient_flow_test │
|
|
│ Expected: Gradient norms 0.3-0.7 (was 0.0000) │
|
|
└────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌─ BUG #2: Gradient Clipping Corruption (A13) ──────────────────────────────┐
|
|
│ SEVERITY: CATASTROPHIC - Destroys weights 217x per run │
|
|
│ LOCATION: ml/src/lib.rs:196-281, ml/src/dqn/dqn.rs:606 │
|
|
│ SYMPTOM: 217 "gradient collapses", Q-values explode then crash │
|
|
│ │
|
|
│ ROOT CAUSE: scale_gradients() calls var.set(&scaled_grad) │
|
|
│ Overwrites W=0.5 with ∂L/∂W=0.001 → network produces zero outputs │
|
|
│ │
|
|
│ FIX (15 min): │
|
|
│ ml/src/lib.rs:196-232 - Replace method: │
|
|
│ pub fn backward_step_with_monitoring( │
|
|
│ &mut self, │
|
|
│ loss: &Tensor, │
|
|
│ warn_threshold: f64, │
|
|
│ ) -> Result<f64, MLError> { │
|
|
│ let grads = loss.backward()?; │
|
|
│ let grad_norm = self.compute_gradient_norm(&grads)?; │
|
|
│ if grad_norm > warn_threshold { │
|
|
│ tracing::warn!("⚠️ Large gradient: {:.4}", grad_norm); │
|
|
│ } │
|
|
│ Optimizer::step(&mut self.optimizer, &grads)?; │
|
|
│ Ok(grad_norm) │
|
|
│ } │
|
|
│ │
|
|
│ ml/src/dqn/dqn.rs:606 - Update caller: │
|
|
│ // OLD: let grad_norm = optimizer.backward_step_with_clipping(...);│
|
|
│ let grad_norm = optimizer.backward_step_with_monitoring(&loss, 10.0)?;│
|
|
│ │
|
|
│ VALIDATION: 10-epoch smoke test │
|
|
│ Expected: Zero "gradient collapse" logs │
|
|
└────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌─ BUG #3: Training Loop Dual Reward System (A18) ──────────────────────────┐
|
|
│ SEVERITY: CRITICAL - Production uses WRONG reward system │
|
|
│ LOCATION: ml/src/trainers/dqn.rs:869-890 │
|
|
│ SYMPTOM: 100% HOLD bias, hyperopt penalties have no effect │
|
|
│ │
|
|
│ ROOT CAUSE: Simple rewards bypass RewardFunction │
|
|
│ ❌ HOLD: -0.0001 (tiny, fixed) │
|
|
│ ❌ BUY/SELL: ±1.0 (risky) │
|
|
│ ✅ UNUSED: RewardFunction with portfolio tracking, diversity penalties │
|
|
│ │
|
|
│ FIX (30 min): │
|
|
│ ml/src/trainers/dqn.rs:869-890 - Replace reward calculation: │
|
|
│ │
|
|
│ // Get next state │
|
|
│ let next_close = if target.len() >= 2 { │
|
|
│ target[1] │
|
|
│ } else { │
|
|
│ training_data[i].0[3] │
|
|
│ }; │
|
|
│ let next_state = if i + 1 < training_data.len() { │
|
|
│ let next_close_price = Decimal::try_from(next_close)?; │
|
|
│ self.feature_vector_to_state(&training_data[i+1].0, │
|
|
│ Some(next_close_price))? │
|
|
│ } else { state.clone() }; │
|
|
│ │
|
|
│ // Track action for diversity penalty │
|
|
│ self.recent_actions.push_back(action); │
|
|
│ if self.recent_actions.len() > 100 { │
|
|
│ self.recent_actions.pop_front(); │
|
|
│ } │
|
|
│ │
|
|
│ // Calculate reward using RewardFunction │
|
|
│ let recent_vec: Vec<_> = self.recent_actions.iter() │
|
|
│ .copied().collect(); │
|
|
│ let reward_decimal = self.reward_fn.calculate_reward( │
|
|
│ action, state, &next_state, &recent_vec │
|
|
│ )?; │
|
|
│ let reward = reward_decimal.to_string() │
|
|
│ .parse::<f32>().unwrap_or(0.0); │
|
|
│ │
|
|
│ Delete dead code (lines 471-638): │
|
|
│ - process_training_sample() │
|
|
│ - process_training_batch() │
|
|
│ │
|
|
│ VALIDATION: cargo test --test dqn_training_loop_integration_test │
|
|
│ Expected: Logs show "HOLD penalty applied" │
|
|
└────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
══════════════════════════════════════════════════════════════════════════════
|
|
|
|
⚠️ HIGH PRIORITY FIXES (AFTER PHASE 1):
|
|
|
|
┌─ FIX #4: Movement Threshold (A14) ─────────────────────────────────────────┐
|
|
│ ISSUE: Penalty NEVER activates (threshold 2% > max data 1.88%) │
|
|
│ FILES: ml/src/dqn/reward.rs:35, ml/examples/train_dqn.rs:112 │
|
|
│ │
|
|
│ CHANGE (5 min): │
|
|
│ movement_threshold: Decimal::try_from(0.02).unwrap() │
|
|
│ → │
|
|
│ movement_threshold: Decimal::try_from(0.01).unwrap() │
|
|
│ │
|
|
│ IMPACT: Penalty now activates 40-50% of timesteps (vs 0%) │
|
|
└────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌─ FIX #5-7: Numerical Stability (A17) ──────────────────────────────────────┐
|
|
│ ISSUES: Unbounded rewards, Q-explosions (+24,055), gradient underflow │
|
|
│ │
|
|
│ FIX #5: Reward Clipping (10 min) │
|
|
│ ml/src/dqn/reward.rs:133 │
|
|
│ Add: .clamp(Decimal::from(-1), Decimal::ONE) │
|
|
│ │
|
|
│ FIX #6: Q-Value Clamping (15 min) │
|
|
│ ml/src/dqn/dqn.rs:366, 492 │
|
|
│ Add: .clamp(-1000.0, 1000.0)? │
|
|
│ │
|
|
│ FIX #7: Huber Delta (10 min) │
|
|
│ ml/src/dqn/dqn.rs:98 │
|
|
│ Change: huber_delta: 1.0 → 10.0 │
|
|
│ │
|
|
│ VALIDATION: cargo test --test dqn_numerical_stability_test │
|
|
└────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
══════════════════════════════════════════════════════════════════════════════
|
|
|
|
✅ NO BUGS FOUND (A16): Action selection mechanism is production-ready
|
|
- 7/7 comprehensive tests pass
|
|
- Epsilon-greedy: uniform distribution (33/33/33%)
|
|
- Argmax: correct (selects highest Q-value)
|
|
- RNG: unbiased
|
|
|
|
══════════════════════════════════════════════════════════════════════════════
|
|
|
|
⏱️ IMPLEMENTATION TIMELINE:
|
|
|
|
Phase 1 (CRITICAL - Blocks all learning): 60 min
|
|
├─ Xavier init fix 15 min
|
|
├─ Gradient clipping fix 15 min
|
|
└─ Training loop fix 30 min
|
|
|
|
Phase 2 (HIGH - Stability): 40 min
|
|
├─ Movement threshold 5 min
|
|
├─ Reward clipping 10 min
|
|
├─ Q-value clamping 15 min
|
|
└─ Huber delta 10 min
|
|
|
|
Validation: 30 min
|
|
├─ Unit tests 10 min
|
|
├─ 10-epoch smoke test 10 min
|
|
└─ 100-epoch full test 10 min
|
|
|
|
TOTAL: 2.5-3 hours to production-ready DQN
|
|
|
|
══════════════════════════════════════════════════════════════════════════════
|
|
|
|
📊 EXPECTED OUTCOMES:
|
|
|
|
Before Fixes:
|
|
❌ Action distribution: 100% HOLD, 0% BUY/SELL
|
|
❌ Gradient collapses: 217 per run (21.7%)
|
|
❌ Q-value max: +24,055 (explosion)
|
|
❌ Learning: None (weights frozen)
|
|
❌ Optimizer params: 0
|
|
|
|
After Phase 1:
|
|
✅ Action distribution: ~30% BUY, ~30% SELL, ~40% HOLD
|
|
✅ Gradient collapses: 0 per run
|
|
✅ Q-value max: <1000
|
|
✅ Learning: Operational
|
|
✅ Optimizer params: 99,200
|
|
|
|
After Phase 2:
|
|
✅ Penalty activation: 40-50% timesteps
|
|
✅ Reward range: [-1.0, +1.0] (bounded)
|
|
✅ Gradient underflow: <5% (was 21.7%)
|
|
✅ Numerical stability: No NaN/Inf
|
|
|
|
══════════════════════════════════════════════════════════════════════════════
|
|
|
|
🔍 VALIDATION COMMANDS:
|
|
|
|
# After Phase 1:
|
|
cargo test --package ml --test dqn_gradient_flow_test
|
|
cargo run --release -p ml --example train_dqn --features cuda -- \
|
|
--epochs 10 --hold-penalty-weight 1.0 \
|
|
--parquet-file test_data/ES_FUT_180d.parquet
|
|
|
|
# After Phase 2:
|
|
cargo test -p ml dqn --release --features cuda --lib -- --test-threads=1
|
|
cargo run --release -p ml --example train_dqn --features cuda -- \
|
|
--epochs 100 --hold-penalty-weight 2.0 \
|
|
--parquet-file test_data/ES_FUT_180d.parquet
|
|
|
|
# Expected logs:
|
|
# - "HOLD penalty applied" (RewardFunction active)
|
|
# - Zero "gradient collapse" messages
|
|
# - Q-values in [-1000, +1000] range
|
|
# - Action distribution ~30/30/40
|
|
|
|
══════════════════════════════════════════════════════════════════════════════
|
|
|
|
📁 FILES TO MODIFY:
|
|
|
|
Phase 1 (Critical):
|
|
1. ml/src/dqn/dqn.rs (lines 15, 186-202, 606)
|
|
2. ml/src/lib.rs (lines 196-281)
|
|
3. ml/src/trainers/dqn.rs (lines 869-890, 471-638)
|
|
|
|
Phase 2 (High Priority):
|
|
4. ml/src/dqn/reward.rs (lines 35, 133)
|
|
5. ml/examples/train_dqn.rs (line 112)
|
|
6. ml/src/dqn/dqn.rs (lines 98, 366, 492)
|
|
|
|
══════════════════════════════════════════════════════════════════════════════
|
|
|
|
🎯 CONFIDENCE: Almost Certain (98%)
|
|
- All bugs independently verified with tests
|
|
- Fixes use proven techniques or restore working baselines
|
|
- No breaking changes to working components (action selection OK)
|
|
|
|
📋 REPORTS: See WAVE10_DEBUG_SYNTHESIS.md for detailed analysis
|