foxhunt/DQN_FACTORED_ACTIONS_BUG_REPORT.md

DQN Factored Actions Bug Report - CRITICAL DISCOVERY

Status: CRITICAL BUG FOUND - Only 3 out of 45 factored actions are selectable
Date: 2025-11-11
Severity: CATASTROPHIC (45-action space is completely non-functional)
Impact: All training uses 3-action space (Buy/Sell/Hold) instead of 45-action factored space

---

Executive Summary

The DQN is hardcoded to use only 3 actions regardless of the feature flag setting for factored actions. The 45-action factored space infrastructure exists but is never activated because:

1. num_actions always defaults to 3 in the WorkingDQNConfig initialization
2. Feature flag compilation is broken: The condition checks compile num_actions: 45 OR num_actions: 3 but trainer defaults to num_actions: 3 regardless of flag
3. FactoredQNetwork is implemented correctly but completely bypassed (initialized as None in trainer, never used)
4. Action selection still uses TradingAction enum (Buy/Sell/Hold) instead of FactoredAction

---

Root Cause Analysis

Bug #1: Trainer Always Creates 3-Action Config (Line 614-619)

File: /home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn.rs

// Lines 614-619
let config = WorkingDQNConfig {
    state_dim: 128,
    #[cfg(feature = "factored-actions")]
    num_actions: 45, // ← SET TO 45 WHEN FEATURE FLAG ENABLED
    #[cfg(not(feature = "factored-actions"))]
    num_actions: 3,  // ← SET TO 3 WHEN FEATURE FLAG DISABLED
    // ... rest of config
};

PROBLEM: This code is CORRECT! The feature flag properly sets num_actions to either 3 or 45. BUT the trainer initialization always uses 3-action mode.

Bug #2: CLI Never Actually Enables Feature Flag

The --use-factored-actions CLI flag doesn't enable the factored-actions feature at compile time. The binary needs to be compiled with:

cargo build --features factored-actions

Without this compile-time flag, the code compiles with #[cfg(not(feature = "factored-actions"))], forcing num_actions: 3.

Bug #3: FactoredQNetwork Created But Never Used (Lines 728-731)

// Lines 728-731
#[cfg(feature = "factored-actions")]
factored_network: None,  // ← ALWAYS INITIALIZED AS NONE!

#[cfg(feature = "factored-actions")]
use_factored_actions: false,  // ← ALWAYS FALSE!

PROBLEM: Even if the feature flag was enabled:

• factored_network is initialized as None and never created
• use_factored_actions is hardcoded to false
• The trainer never calls FactoredQNetwork methods for action selection
• Instead, it continues using TradingAction (Buy/Sell/Hold)

Bug #4: Action Selection Still Uses 3-Action TradingAction (Lines 263-268)

// In TrainingMonitor (lines 263-268)
fn track_action(&mut self, action: &TradingAction) {
    let idx = match action {
        TradingAction::Buy => 0,
        TradingAction::Sell => 1,
        TradingAction::Hold => 2,
    };
    self.action_counts[idx] += 1;
}

PROBLEM: This only tracks 3 actions. When factored actions are enabled, we should be tracking FactoredAction with indices 0-44.

---

Evidence: Hardcoded Constants

File: /home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn.rs lines 228-232

// Action count depends on feature flag
#[cfg(feature = "factored-actions")]
const NUM_ACTIONS: usize = 45;
#[cfg(not(feature = "factored-actions"))]
const NUM_ACTIONS: usize = 3;

This is correct at compile time, but training data shows only 3 actions used, meaning the binary was compiled WITHOUT the factored-actions feature flag.

---

Why Only 3 Actions Are Selected

Scenario A: Feature Flag NOT Enabled (Current State)

If compiled without --features factored-actions:

1. NUM_ACTIONS = 3
2. num_actions = 3 (line 619)
3. Q-network outputs 3 Q-values (one per action)
4. Action selection argmax picks from 3 indices: [0, 1, 2] = [BUY, SELL, HOLD]
5. FactoredQNetwork never instantiated
6. Training produces 3-action distribution

Result: Only 3 actions available. ✅ Explains observed behavior.

Scenario B: Feature Flag Enabled But CLI Flag Not Propagated

If compiled WITH --features factored-actions but CLI --use-factored-actions not activated:

1. NUM_ACTIONS = 45
2. num_actions = 45 (line 617)
3. Q-network outputs 45 Q-values
4. BUT use_factored_actions = false (line 731)
5. Action selection still uses TradingAction enum (only 3 variants)
6. Argmax on 45 Q-values returns indices 0-44
7. BUT code tries to convert to TradingAction (only 3 valid)

Result: Runtime error or silent fallback to actions 0-2.

---

FactoredQNetwork Implementation Status

File: /home/jgrusewski/Work/foxhunt/ml/src/dqn/factored_q_network.rs

The FactoredQNetwork is fully implemented and correct:

✅ Structure (lines 44-59):

• shared_encoder: 128 → 64
• exposure_head: 64 → 5
• order_head: 64 → 3
• urgency_head: 64 → 3

✅ Forward Pass (lines 123-154):

• Computes 3 separate heads
• Returns (5, 3, 3) tensors

✅ Joint Q-Values (lines 161-201):

• Combines via additive factorization: Q(s,a) = Q_exp + Q_ord + Q_urg
• Returns [batch, 45] tensor ✅

✅ Action Selection (lines 204-260):

• select_greedy_action(): Takes argmax per head
• select_epsilon_greedy(): Random factored action exploration
• Both return FactoredAction (not TradingAction)

Problem: This network is created but NEVER INSTANTIATED in the trainer.

---

Action Space Mapping (Correct Implementation)

File: /home/jgrusewski/Work/foxhunt/ml/src/dqn/action_space.rs

Action mapping is fully correct:

Index = exposure_idx * 9 + order_idx * 3 + urgency_idx

Exposure (5 options): 0=Short100, 1=Short50, 2=Flat, 3=Long50, 4=Long100
Order (3 options):    0=Market, 1=LimitMaker, 2=IoC
Urgency (3 options):  0=Patient, 1=Normal, 2=Aggressive

Example: (Flat=2, Market=0, Normal=1) → 2*9 + 0*3 + 1 = 19 ✅
Example: (Long100=4, Market=0, Aggressive=2) → 4*9 + 0*3 + 2 = 38 ✅

All 45 combinations are unique and valid (verified by round-trip tests).

---

How to Fix

Short-term Fix: Enable Feature Flag at Compile Time

# Currently broken:
cargo build --release

# Must use:
cargo build --release --features factored-actions

# Or in Cargo.toml:
cargo run --features factored-actions --example train_dqn --release

Problem: This only addresses compilation. Action selection still broken (Bug #3).

Long-term Fix: Implement 45-Action Selection in Trainer

The trainer needs to be refactored to:

1. Actually create FactoredQNetwork instead of passing None

   #[cfg(feature = "factored-actions")]
   let factored_network = if use_factored_actions {
       Some(Arc::new(RwLock::new(
           FactoredQNetwork::new(128, &device)?
       )))
   } else {
       None
   };
   

2. Use FactoredQNetwork for action selection when enabled

   if self.use_factored_actions {
       // Use FactoredQNetwork.select_epsilon_greedy()
       let factored_action = self.factored_network
           .as_ref()
           .unwrap()
           .read()
           .await
           .select_epsilon_greedy(&state_tensor, epsilon)?;
       // Convert to action index for storage
   } else {
       // Use standard 3-action selection (current)
   }
   

3. Refactor action tracking to support both 3 and 45 actions

   // Current: hardcoded for 3 actions
   action_counts: vec![0; NUM_ACTIONS],
   
   // Already correct via feature flag!
   // But tracking logic needs to handle FactoredAction
   

4. Update action-to-reward mapping for factored actions

   • Current code maps TradingAction → reward
   • Need to map FactoredAction → exposure, order, urgency → reward

---

Test Cases Affected

Files that expect 3 actions but would break with 45:

File	Issue	Impact
ml/tests/dqn_factored_smoke_tests.rs	Tests factored-actions feature	Will fail with 45 actions until trainer is fixed
ml/src/dqn/tests/factored_integration_tests.rs	Integration tests	Needs updated action selection logic
ml/examples/train_dqn.rs	CLI training example	Works but uses 3-action fallback
ml/src/trainers/dqn.rs lines 263-281	TrainingMonitor	Hard-coded 3-action tracking

---

Current Training Status

What's Happening:

1. Binary compiled without factored-actions feature
2. num_actions = 3 (forced by #[cfg(not(feature = "factored-actions"))])
3. Q-network has 3 outputs (Buy, Sell, Hold)
4. Argmax selects from [0, 1, 2]
5. Actions stored as TradingAction variants

Result: Only 3 actions available ❌

---

Verification Commands

# Check if binary compiled with factored-actions feature
grep "const NUM_ACTIONS: usize = " ml/src/trainers/dqn.rs
# Expected: Should show NUM_ACTIONS = 45 if compiled with feature

# Check training logs
grep "Action Distribution" target/release/examples/train_dqn.log
# Current output: BUY=XX% SELL=XX% HOLD=XX%
# Expected with fix: Top 10 actions with index 0-44

# Compile with factored-actions (doesn't fully fix, but required step)
cargo build --release --features factored-actions

---

Recommendations

Priority 1: Implement Full 45-Action Support

• Effort: 2-4 hours
• Steps:
  1. Create FactoredQNetwork in trainer when feature enabled
  2. Route action selection to FactoredQNetwork.select_epsilon_greedy()
  3. Update TrainingMonitor to track 45 actions
  4. Update reward calculation for FactoredAction

Priority 2: Add --use-factored-actions CLI Flag

• Effort: 30 minutes
• Steps:
  1. Add --use-factored-actions flag to train_dqn.rs
  2. Pass flag to DQNTrainer::new_with_factored_actions()
  3. Set use_factored_actions = true in trainer

Priority 3: Validation Tests

• Effort: 1 hour
• Steps:
  1. Create test that verifies 45 actions are selectable
  2. Verify action-to-exposure-order-urgency mapping
  3. Validate that all combinations (0-44) can be reached

---

Files to Modify

ml/src/trainers/dqn.rs
  - Line 728: Initialize FactoredQNetwork properly
  - Line 731: Set use_factored_actions from CLI flag
  - Lines 263-281: Update track_action() for 45 actions
  - Lines 1600+: Update action selection logic

ml/examples/train_dqn.rs
  - Add --use-factored-actions flag
  - Pass to DQNTrainer initialization

ml/src/dqn/dqn.rs
  - Verify Q-network output dimension matches num_actions (should be automatic)

---

Conclusion

The 45-action factored space is fully implemented but completely disconnected from the training pipeline. The trainer:

1. ✅ Sets num_actions = 45 when feature flag enabled
2. ✅ FactoredQNetwork is fully functional
3. ❌ Never instantiates FactoredQNetwork
4. ❌ Still uses TradingAction for selection (3 variants only)
5. ❌ CLI flag --use-factored-actions doesn't exist

Result: Training always uses 3 actions, regardless of feature flag or infrastructure availability.

Expected behavior after fix: With --features factored-actions --use-factored-actions, should see all 45 actions selected with proper exposure/order/urgency combinations.