foxhunt/docs/RAINBOW_DQN_QUICK_REF.md

Rainbow DQN Quick Reference

Full Analysis: See RAINBOW_DQN_COMPONENT_MATRIX.md

---

Component Status at a Glance

#	Component	File	Default	Hyperopt	Notes
1	Double DQN	dqn.rs	✅ Always ON	❌ Hardcoded	Target network with soft/hard updates
2	Dueling Networks	dueling.rs	✅ ON	✅ Tunable	Separate V(s) and A(s,a) streams
3	Prioritized Replay	prioritized_replay.rs	✅ ON	✅ Tunable	Segment tree, alpha/beta params
4	Multi-Step Returns	multi_step.rs	✅ ON (n=3)	✅ Tunable (1-5)	N-step TD learning
5	Distributional C51	distributional.rs	❌ OFF	✅ Tunable	BUG #36: Candle gradient issue
6	Noisy Networks	noisy_layers.rs	✅ ON	✅ Tunable	Learned exploration, factorized Gaussian

Current Production: 5/6 components enabled (C51 disabled due to library bug)

---

Critical Bug

BUG #36: C51 Gradient Flow Failure

• Impact: 40% training failure rate at epoch 2
• Root Cause: Candle library's scatter_add breaks autograd in project_distribution()
• Status: BLOCKED - external library issue
• Workaround: QR-DQN implemented as alternative (quantile_regression.rs)
• Performance: Standard DQN achieves Sharpe 0.77-2.0 WITHOUT C51

---

Hyperopt Search Space (39D)

Base (11D)

• learning_rate (1e-5 to 3e-4, log)
• batch_size (64-160)
• gamma (0.95-0.99)
• buffer_size (50K-100K, log)
• hold_penalty_weight (1.0-2.0)
• max_position_absolute (4.0-8.0)
• huber_delta (10-40, log)
• entropy_coefficient (0.0-0.1)
• transaction_cost_multiplier (0.5-2.0)
• per_alpha (0.4-0.8)
• per_beta_start (0.2-0.6)

Rainbow (6D)

• v_min (-3 to -1) - unused
• v_max (1 to 3) - unused
• noisy_sigma_init (0.1-1.0, log)
• dueling_hidden_dim (128-512, step=128)
• n_steps (1-5)
• num_atoms (51-201, step=50) - unused

Advanced (22D)

• Kelly risk (4D), ensemble uncertainty (5D), warmup, curiosity, tau, etc.

---

File Locations

Core Implementation

ml/src/dqn/
├── dqn.rs                    # Main agent + Double DQN
├── dueling.rs                # Dueling architecture
├── prioritized_replay.rs     # PER segment tree
├── multi_step.rs             # N-step returns
├── distributional.rs         # C51 (disabled)
├── noisy_layers.rs           # Noisy networks
└── rainbow_network.rs        # Integrated network

Configuration

ml/src/trainers/dqn/
├── config.rs                 # DQNHyperparameters (264+ lines)
└── trainer.rs                # Component integration logic

ml/src/hyperopt/adapters/
└── dqn.rs                    # DQNParams + search space

Tests

ml/src/dqn/tests/
├── target_update_comprehensive_tests.rs
└── factored_integration_tests.rs

ml/src/trainers/dqn/tests/
├── p0_integration_tests.rs
└── p1_integration_tests.rs

---

Quick Integration Check

# Search for component usage in trainer
grep -n "use_dueling\|use_distributional\|use_noisy\|use_per" ml/src/trainers/dqn/trainer.rs

# Check hyperopt defaults
grep -n "Default for DQNParams" ml/src/hyperopt/adapters/dqn.rs -A 100

---

Production Defaults

// ml/src/trainers/dqn/config.rs::DQNHyperparameters::default()
use_double_dqn: true,         // Always ON
use_per: true,                // PER enabled (25-40% speedup)
use_dueling: true,            // Dueling ON (Wave 8)
use_distributional: false,    // C51 OFF (BUG #36)
use_noisy_nets: true,         // Noisy ON (Wave 8)
n_steps: 3,                   // 3-step returns (Rainbow standard)

// Hyperparameters
per_alpha: 0.6,               // Rainbow standard
per_beta_start: 0.4,          // Rainbow standard
noisy_sigma_init: 0.5,        // Rainbow standard
dueling_hidden_dim: 128,
num_atoms: 51,                // Unused (C51 disabled)
v_min: -2.0,                  // Unused (C51 disabled)
v_max: 2.0,                   // Unused (C51 disabled)

---

Advanced Components (Beyond Rainbow)

Component	File	Wave	Status
QR-DQN (Quantile)	quantile_regression.rs	26 P1.13	✅ C51 alternative
Ensemble Uncertainty	ensemble_network.rs	26 P2.3	✅ 3-10 heads
Hindsight Replay	hindsight_replay.rs	26 P1.7	✅ 5-10x efficiency
Curiosity	curiosity.rs	26 P1.8	✅ Intrinsic rewards
GAE	gae.rs	26 P1.9	✅ Lower variance
Attention	attention.rs	26 P1.2	✅ Temporal patterns
Spectral Norm	spectral_norm.rs	-	✅ Q-stability
Residual	residual.rs	26 P0.4	✅ Gradient flow
RMSNorm	rmsnorm.rs	26 P2.4	✅ 15% faster
Mixed Precision	mixed_precision.rs	26 P2.1	✅ 2x speedup

---

Architecture Flow

State → Feature Extraction → Dueling Split
                                │
                    ┌───────────┴──────────┐
                    ▼                      ▼
              Value Stream          Advantage Stream
                    │                      │
                    └───────────┬──────────┘
                                ▼
                    Aggregate: Q(s,a) = V(s) + A(s,a) - mean(A)
                                │
                    ┌───────────┴──────────┐
                    ▼                      ▼
              [C51 disabled]        Noisy Layers (σ learned)
                                           │
                                           ▼
                                    Q-values (45 actions)
                                           │
                                           ▼
                    Double DQN: action = argmax Q(θ), eval = Q(θ')
                                           │
                                           ▼
                    PER: Sample by priority |TD_error|^α
                                           │
                                           ▼
                    Multi-Step: R_t^n = Σ γ^k r_{t+k} + γ^n Q(...)

---

Key Metrics

• Sharpe Ratio: 0.77-2.0 (without C51)
• PER Speedup: 25-40% faster convergence
• Noisy Nets: Better than ε-greedy (no manual schedule)
• Dueling: +10-20% sample efficiency
• Multi-Step: Faster credit assignment (n=3 optimal)
• Success Rate: 95%+ (with C51 disabled), 60% (with C51 enabled - BUG #36)

---

Common Tasks

Enable/Disable Components

// ml/src/hyperopt/adapters/dqn.rs::DQNParams::default()
use_dueling: true,           // Toggle dueling
use_distributional: false,   // Toggle C51 (keep false until BUG #36 fixed)
use_noisy_nets: true,        // Toggle noisy exploration
use_per: true,               // Toggle prioritized replay

Adjust Hyperparameters

// Hyperopt search space bounds:
// ml/src/hyperopt/adapters/dqn.rs::ParameterSpace::continuous_bounds()

// Line 407: learning_rate range
(1e-5_f64.ln(), 3e-4_f64.ln())

// Line 416: per_alpha range
(0.4, 0.8)

// Line 426: dueling_hidden_dim range
(128.0, 512.0)

Integrate New Component

1. Create implementation file in ml/src/dqn/
2. Add config struct with Default impl
3. Add boolean flag to DQNHyperparameters (config.rs)
4. Add param to DQNParams (hyperopt/adapters/dqn.rs)
5. Add search space bounds to continuous_bounds()
6. Update trainer integration logic (trainers/dqn/trainer.rs)
7. Add tests to ml/src/dqn/tests/

---

For full analysis, see: docs/RAINBOW_DQN_COMPONENT_MATRIX.md