# DQN Factored Actions - Debug Flowchart ## Current Broken Flow (Only 3 Actions Used) ``` ┌─────────────────────────────────────────────────────────────┐ │ cargo build --release │ │ (NO --features factored-actions flag) │ └─────────────────────┬───────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ Compilation: #[cfg(not(feature = "factored-actions"))] │ │ ▶ NUM_ACTIONS = 3 (line 232) │ │ ▶ num_actions: 3 in WorkingDQNConfig (line 619) │ └─────────────────────┬───────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ DQNTrainer::new() initialization (line 570) │ │ ▶ Creates WorkingDQNConfig with num_actions=3 │ │ ▶ factored_network = None (line 728) │ │ ▶ use_factored_actions = false (line 731) │ └─────────────────────┬───────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ WorkingDQN::new(config) - main/target networks │ │ ▶ Q-network output layer: num_actions=3 │ │ ▶ Sequential network: ... → 64 → [3 Q-values] │ │ ▶ Target network: ... → 64 → [3 Q-values] │ └─────────────────────┬───────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ Training Loop - Action Selection │ │ │ │ State Vector (128 dims) │ │ ▼ │ │ Q-network forward() │ │ ▼ │ │ [Q_BUY, Q_SELL, Q_HOLD] ◄── Only 3 Q-values! │ │ ▼ │ │ argmax() → action_idx ∈ {0, 1, 2} │ │ ▼ │ │ TradingAction::from_int(action_idx) │ │ ├─ 0 → Buy │ │ ├─ 1 → Sell │ │ └─ 2 → Hold │ │ │ │ RESULT: Only 3 actions selectable! ❌ │ └─────────────────────┬───────────────────────────────────────┘ │ ▼ Training logs show 3 actions only (BUY%, SELL%, HOLD%) ✗ ``` --- ## Expected Correct Flow (With All 45 Actions) ``` ┌─────────────────────────────────────────────────────────────┐ │ cargo build --release --features factored-actions │ │ (WITH feature flag) │ └─────────────────────┬───────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ Compilation: #[cfg(feature = "factored-actions")] │ │ ▶ NUM_ACTIONS = 45 (line 230) │ │ ▶ num_actions: 45 in WorkingDQNConfig (line 617) │ │ ▶ Import FactoredQNetwork (line 28) │ └─────────────────────┬───────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ CLI: train_dqn --use-factored-actions (NEW FLAG) │ │ ▶ Flag parsed and passed to DQNTrainer │ │ ▶ use_factored_actions = true in trainer │ └─────────────────────┬───────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ DQNTrainer::new() initialization (FIXED) │ │ ▶ Creates WorkingDQNConfig with num_actions=45 │ │ ▶ Creates FactoredQNetwork (128 → 64 → 5,3,3 heads) │ │ ▶ factored_network = Some(FactoredQNetwork) │ │ ▶ use_factored_actions = true │ └─────────────────────┬───────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ FactoredQNetwork initialization │ │ ▶ shared_encoder: 128 → 64 │ │ ▶ exposure_head: 64 → 5 (Short100, Short50, Flat, ... │ │ ▶ order_head: 64 → 3 (Market, LimitMaker, IoC) │ │ ▶ urgency_head: 64 → 3 (Patient, Normal, Aggressive) │ └─────────────────────┬───────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ Training Loop - Factored Action Selection (FIXED) │ │ │ │ State Vector (128 dims) │ │ ▼ │ │ FactoredQNetwork.forward() │ │ ▼ │ │ Q_exposure: [5 values] exp_idx ∈ {0,1,2,3,4} │ │ Q_order: [3 values] → ord_idx ∈ {0,1,2} │ │ Q_urgency: [3 values] urg_idx ∈ {0,1,2} │ │ ▼ │ │ compute_joint_q() broadcasts + sums │ │ → [batch, 5, 3, 3] → flatten → [batch, 45] │ │ ▼ │ │ argmax() → action_idx ∈ {0, 1, 2, ..., 44} ✅ │ │ ▼ │ │ FactoredAction::from_index(action_idx) │ │ → (exposure, order, urgency) tuple │ │ → All 45 combinations available! │ │ │ │ RESULT: All 45 actions selectable! ✅ │ └─────────────────────┬───────────────────────────────────────┘ │ ▼ Training logs show 45 actions distributed (Top 10 actions with proper factorization) ✓ ``` --- ## The Critical Gap: FactoredQNetwork Creation ``` ┌─ CODE AT LINE 728 ─┐ │ #[cfg(feature = "factored-actions")] │ factored_network: None, ◄── ALWAYS NONE! │ │ #[cfg(feature = "factored-actions")] │ use_factored_actions: false, ◄── ALWAYS FALSE! └────────────────────┘ The infrastructure exists but is never activated! WHAT SHOULD HAPPEN: ┌─────────────────────────────────────────┐ │ if cli_flag_use_factored_actions { │ │ let net = FactoredQNetwork::new( │ │ 128, // state_dim │ │ &device // GPU/CPU │ │ )?; │ │ self.factored_network = Some(net); │ │ self.use_factored_actions = true; │ │ } │ └─────────────────────────────────────────┘ ``` --- ## Action Selection Code Path Analysis ### Current (Broken) - Always 3 Actions ```rust // In WorkingDQN.select_action() let state_tensor = Tensor::from_vec(..., (1, 128), device)?; // Main network forward let q_values = self.q_network.forward(&state_tensor)?; // q_values shape: [1, 3] ◄── ONLY 3! // Epsilon-greedy if rng.gen::() < epsilon { // Random action action_idx = rng.gen_range(0..3); // 0, 1, or 2 } else { // Greedy action action_idx = q_values.argmax(1)?; // Returns 0, 1, or 2 } // Convert to enum let action = TradingAction::from_int(action_idx as u8)?; // Only 3 variants: Buy(0), Sell(1), Hold(2) ``` ### Expected (Fixed) - 45 Actions ```rust // In DQNTrainer with --use-factored-actions flag if self.use_factored_actions { // Use FactoredQNetwork let state_tensor = Tensor::from_vec(..., (1, 128), device)?; let action = self.factored_network .as_ref() .unwrap() .select_epsilon_greedy(&state_tensor, epsilon)?; // ▶ Inside select_epsilon_greedy(): // - Generate 3 random indices: exp (0-4), ord (0-2), urg (0-2) // - Combine: idx = exp*9 + ord*3 + urg // - Result: 0-44 (all 45 combinations) // Use action: FactoredAction { exposure, order, urgency } } else { // Use standard 3-action selection (current) let action = standard_dqn_select_action(); } ``` --- ## Why Only 3 Actions Shows Up in Logs ``` Config Initialization (Trainer): ├─ Compile without --features factored-actions │ └─ num_actions = 3 │ └─ Create Q-network with output_dim = 3 └─ Sequential { ... Linear(64 → 3) } └─ Forward returns [1, 3] tensor └─ argmax on 3 values returns 0, 1, or 2 └─ Only 3 actions selectable! └─ Training logs show BUY%, SELL%, HOLD% ``` --- ## Compile-Time Feature Flag Impact ### Without `--features factored-actions` ```rust // ml/src/trainers/dqn.rs line 228-232 #[cfg(feature = "factored-actions")] const NUM_ACTIONS: usize = 45; #[cfg(not(feature = "factored-actions"))] ◄── THIS BRANCH TAKEN const NUM_ACTIONS: usize = 3; // ml/src/trainers/dqn.rs line 615-619 #[cfg(feature = "factored-actions")] num_actions: 45, #[cfg(not(feature = "factored-actions"))] ◄── THIS BRANCH TAKEN num_actions: 3, ``` ### With `--features factored-actions` ```rust // ml/src/trainers/dqn.rs line 228-232 #[cfg(feature = "factored-actions")] ◄── THIS BRANCH TAKEN const NUM_ACTIONS: usize = 45; #[cfg(not(feature = "factored-actions"))] const NUM_ACTIONS: usize = 3; // ml/src/trainers/dqn.rs line 615-619 #[cfg(feature = "factored-actions")] ◄── THIS BRANCH TAKEN num_actions: 45, #[cfg(not(feature = "factored-actions"))] num_actions: 3, // BUT STILL BROKEN BECAUSE: #[cfg(feature = "factored-actions")] factored_network: None, ◄── STILL NEVER CREATED! use_factored_actions: false, ◄── STILL ALWAYS FALSE! ``` --- ## Summary: Why Only 3 Out of 45 | Component | Status | Issue | |-----------|--------|-------| | **Action Space Definition** | ✅ Correct | All 45 combinations defined (exposure × order × urgency) | | **FactoredQNetwork** | ✅ Correct | 5-head architecture properly implemented | | **Compile-Time Feature Flag** | ⚠️ Works but depends on `--features` | `NUM_ACTIONS` set correctly IF flag enabled | | **Trainer Initialization** | ❌ BROKEN | Sets `num_actions` to 3 by default, never enables factored network | | **Action Selection Logic** | ❌ BROKEN | Still uses TradingAction (3 variants) instead of FactoredAction | | **CLI Flag** | ❌ MISSING | `--use-factored-actions` doesn't exist, can't enable at runtime | | **Training Monitoring** | ❌ BROKEN | Assumes 3 actions, won't track 45 properly | **Result**: Only 3 actions selectable, regardless of feature flag compilation.