foxhunt/DQN_FACTORED_ACTIONS_DEBUG_FLOWCHART.md

DQN Factored Actions - Debug Flowchart

Current Broken Flow (Only 3 Actions Used)

┌─────────────────────────────────────────────────────────────┐
│ cargo build --release                                       │
│ (NO --features factored-actions flag)                       │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Compilation: #[cfg(not(feature = "factored-actions"))]      │
│ ▶ NUM_ACTIONS = 3 (line 232)                                │
│ ▶ num_actions: 3 in WorkingDQNConfig (line 619)            │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ DQNTrainer::new() initialization (line 570)                 │
│ ▶ Creates WorkingDQNConfig with num_actions=3              │
│ ▶ factored_network = None (line 728)                        │
│ ▶ use_factored_actions = false (line 731)                  │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ WorkingDQN::new(config) - main/target networks              │
│ ▶ Q-network output layer: num_actions=3                     │
│ ▶ Sequential network: ... → 64 → [3 Q-values]             │
│ ▶ Target network: ... → 64 → [3 Q-values]                 │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Training Loop - Action Selection                            │
│                                                             │
│ State Vector (128 dims)                                    │
│   ▼                                                        │
│ Q-network forward()                                        │
│   ▼                                                        │
│ [Q_BUY, Q_SELL, Q_HOLD]  ◄── Only 3 Q-values!           │
│   ▼                                                        │
│ argmax() → action_idx ∈ {0, 1, 2}                         │
│   ▼                                                        │
│ TradingAction::from_int(action_idx)                        │
│   ├─ 0 → Buy                                               │
│   ├─ 1 → Sell                                              │
│   └─ 2 → Hold                                              │
│                                                             │
│ RESULT: Only 3 actions selectable! ❌                      │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
          Training logs show 3 actions only
           (BUY%, SELL%, HOLD%) ✗

---

Expected Correct Flow (With All 45 Actions)

┌─────────────────────────────────────────────────────────────┐
│ cargo build --release --features factored-actions           │
│ (WITH feature flag)                                         │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Compilation: #[cfg(feature = "factored-actions")]           │
│ ▶ NUM_ACTIONS = 45 (line 230)                               │
│ ▶ num_actions: 45 in WorkingDQNConfig (line 617)           │
│ ▶ Import FactoredQNetwork (line 28)                         │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ CLI: train_dqn --use-factored-actions (NEW FLAG)            │
│ ▶ Flag parsed and passed to DQNTrainer                     │
│ ▶ use_factored_actions = true in trainer                   │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ DQNTrainer::new() initialization (FIXED)                    │
│ ▶ Creates WorkingDQNConfig with num_actions=45             │
│ ▶ Creates FactoredQNetwork (128 → 64 → 5,3,3 heads)       │
│ ▶ factored_network = Some(FactoredQNetwork)                │
│ ▶ use_factored_actions = true                              │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ FactoredQNetwork initialization                             │
│ ▶ shared_encoder: 128 → 64                                 │
│ ▶ exposure_head: 64 → 5 (Short100, Short50, Flat, ...    │
│ ▶ order_head: 64 → 3 (Market, LimitMaker, IoC)           │
│ ▶ urgency_head: 64 → 3 (Patient, Normal, Aggressive)     │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Training Loop - Factored Action Selection (FIXED)           │
│                                                             │
│ State Vector (128 dims)                                    │
│   ▼                                                        │
│ FactoredQNetwork.forward()                                 │
│   ▼                                                        │
│ Q_exposure: [5 values]    exp_idx ∈ {0,1,2,3,4}           │
│ Q_order: [3 values]   →   ord_idx ∈ {0,1,2}               │
│ Q_urgency: [3 values]     urg_idx ∈ {0,1,2}               │
│   ▼                                                        │
│ compute_joint_q() broadcasts + sums                         │
│   → [batch, 5, 3, 3] → flatten → [batch, 45]             │
│   ▼                                                        │
│ argmax() → action_idx ∈ {0, 1, 2, ..., 44}  ✅            │
│   ▼                                                        │
│ FactoredAction::from_index(action_idx)                     │
│   → (exposure, order, urgency) tuple                        │
│   → All 45 combinations available!                          │
│                                                             │
│ RESULT: All 45 actions selectable! ✅                      │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
    Training logs show 45 actions distributed
    (Top 10 actions with proper factorization) ✓

---

The Critical Gap: FactoredQNetwork Creation

┌─ CODE AT LINE 728 ─┐
│ #[cfg(feature = "factored-actions")]
│ factored_network: None,  ◄── ALWAYS NONE!
│
│ #[cfg(feature = "factored-actions")]
│ use_factored_actions: false,  ◄── ALWAYS FALSE!
└────────────────────┘

The infrastructure exists but is never activated!

WHAT SHOULD HAPPEN:
┌─────────────────────────────────────────┐
│ if cli_flag_use_factored_actions {      │
│   let net = FactoredQNetwork::new(      │
│       128,              // state_dim     │
│       &device           // GPU/CPU       │
│   )?;                                   │
│   self.factored_network = Some(net);    │
│   self.use_factored_actions = true;     │
│ }                                       │
└─────────────────────────────────────────┘

---

Action Selection Code Path Analysis

Current (Broken) - Always 3 Actions

// In WorkingDQN.select_action()
let state_tensor = Tensor::from_vec(..., (1, 128), device)?;

// Main network forward
let q_values = self.q_network.forward(&state_tensor)?;
// q_values shape: [1, 3] ◄── ONLY 3!

// Epsilon-greedy
if rng.gen::<f64>() < epsilon {
    // Random action
    action_idx = rng.gen_range(0..3);  // 0, 1, or 2
} else {
    // Greedy action
    action_idx = q_values.argmax(1)?;  // Returns 0, 1, or 2
}

// Convert to enum
let action = TradingAction::from_int(action_idx as u8)?;
// Only 3 variants: Buy(0), Sell(1), Hold(2)

Expected (Fixed) - 45 Actions

// In DQNTrainer with --use-factored-actions flag
if self.use_factored_actions {
    // Use FactoredQNetwork
    let state_tensor = Tensor::from_vec(..., (1, 128), device)?;
    
    let action = self.factored_network
        .as_ref()
        .unwrap()
        .select_epsilon_greedy(&state_tensor, epsilon)?;
        // ▶ Inside select_epsilon_greedy():
        //   - Generate 3 random indices: exp (0-4), ord (0-2), urg (0-2)
        //   - Combine: idx = exp*9 + ord*3 + urg
        //   - Result: 0-44 (all 45 combinations)
    
    // Use action: FactoredAction { exposure, order, urgency }
} else {
    // Use standard 3-action selection (current)
    let action = standard_dqn_select_action();
}

---

Why Only 3 Actions Shows Up in Logs

Config Initialization (Trainer):
  ├─ Compile without --features factored-actions
  │  └─ num_actions = 3
  │
  └─ Create Q-network with output_dim = 3
     └─ Sequential { ... Linear(64 → 3) }
        └─ Forward returns [1, 3] tensor
           └─ argmax on 3 values returns 0, 1, or 2
              └─ Only 3 actions selectable!
                 └─ Training logs show BUY%, SELL%, HOLD%

---

Compile-Time Feature Flag Impact

Without --features factored-actions

// ml/src/trainers/dqn.rs line 228-232
#[cfg(feature = "factored-actions")]
const NUM_ACTIONS: usize = 45;
#[cfg(not(feature = "factored-actions"))]  ◄── THIS BRANCH TAKEN
const NUM_ACTIONS: usize = 3;

// ml/src/trainers/dqn.rs line 615-619
#[cfg(feature = "factored-actions")]
num_actions: 45,
#[cfg(not(feature = "factored-actions"))]  ◄── THIS BRANCH TAKEN
num_actions: 3,

With --features factored-actions

// ml/src/trainers/dqn.rs line 228-232
#[cfg(feature = "factored-actions")]  ◄── THIS BRANCH TAKEN
const NUM_ACTIONS: usize = 45;
#[cfg(not(feature = "factored-actions"))]
const NUM_ACTIONS: usize = 3;

// ml/src/trainers/dqn.rs line 615-619
#[cfg(feature = "factored-actions")]  ◄── THIS BRANCH TAKEN
num_actions: 45,
#[cfg(not(feature = "factored-actions"))]
num_actions: 3,

// BUT STILL BROKEN BECAUSE:
#[cfg(feature = "factored-actions")]
factored_network: None,          ◄── STILL NEVER CREATED!
use_factored_actions: false,     ◄── STILL ALWAYS FALSE!

---

Summary: Why Only 3 Out of 45

Component	Status	Issue
Action Space Definition	✅ Correct	All 45 combinations defined (exposure × order × urgency)
FactoredQNetwork	✅ Correct	5-head architecture properly implemented
Compile-Time Feature Flag	⚠️ Works but depends on --features	NUM_ACTIONS set correctly IF flag enabled
Trainer Initialization	❌ BROKEN	Sets num_actions to 3 by default, never enables factored network
Action Selection Logic	❌ BROKEN	Still uses TradingAction (3 variants) instead of FactoredAction
CLI Flag	❌ MISSING	--use-factored-actions doesn't exist, can't enable at runtime
Training Monitoring	❌ BROKEN	Assumes 3 actions, won't track 45 properly

Result: Only 3 actions selectable, regardless of feature flag compilation.