# DQN 45 Factored Actions Restoration Plan

> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Enable Branching DQN by default, restoring 45 factored actions (5 exposure × 3 order × 3 urgency) and removing OrderRouter from the decision path.

**Architecture:** The BranchingDuelingQNetwork already outputs per-branch Q-values [5, 3, 3]. The `select_action()` path already supports branching. Three other action selection paths (`select_action_with_confidence`, `select_action_inference`, `select_action_factored`) still use `OrderRouter::route_default` and need branching support. Monitoring stays exposure-focused (correct for branching). CountBonus expands to per-branch tracking.

**Tech Stack:** Rust, Candle ML, CUDA

---

## Chunk 1: Core DQN Config + Action Selection Paths

### Task 1: Enable branching by default in DQNConfig

**Files:**
- Modify: `crates/ml-dqn/src/dqn.rs:300` (default), `crates/ml-dqn/src/dqn.rs:744` (emergency defaults)

- [ ] **Step 1:** Change `use_branching: false` → `use_branching: true` in `DQNConfig::default()` at line 300
- [ ] **Step 2:** Change `use_branching: false` → `use_branching: true` in `emergency_safe_defaults()` at line 744
- [ ] **Step 3:** Run `SQLX_OFFLINE=true cargo check -p ml-dqn --lib` — expect success
- [ ] **Step 4:** Commit: "feat(dqn): enable branching DQN by default (45 factored actions)"

### Task 2: Add branching path to `select_action_with_confidence`

**Files:**
- Modify: `crates/ml-dqn/src/dqn.rs` — method `select_action_with_confidence` (~line 1798)

The current code at line 1798-1803 does:
```rust
let action_idx = rng.gen_range(0..self.config.num_actions);
let uniform_conf = (1.0_f32 / self.config.num_actions as f32).clamp(0.5, 0.95);
let exposure = ExposureLevel::from_index(action_idx)?;
(OrderRouter::route_default(exposure), uniform_conf)
```

- [ ] **Step 1:** Add branching random path: when `use_branching`, sample per-branch independently (like `select_action` does at lines 1623-1628), with confidence = 1/45 clamped
- [ ] **Step 2:** Add branching greedy path: when `use_branching`, use `branching_q_network.forward_branches_eval` + `greedy_branch_actions` (like `select_action` does at lines 1645-1656), with per-branch softmax confidence
- [ ] **Step 3:** Run `SQLX_OFFLINE=true cargo check -p ml-dqn --lib`
- [ ] **Step 4:** Commit

### Task 3: Add branching path to `select_action_inference`

**Files:**
- Modify: `crates/ml-dqn/src/dqn.rs` — method `select_action_inference` (~line 1946)

- [ ] **Step 1:** Add branching greedy path (inference is always greedy — no epsilon): use `branching_q_network.forward_branches_eval` + `greedy_branch_actions`, compute per-branch softmax confidence
- [ ] **Step 2:** Run `SQLX_OFFLINE=true cargo check -p ml-dqn --lib`
- [ ] **Step 3:** Commit

### Task 4: Fix `select_action_factored` in agent.rs

**Files:**
- Modify: `crates/ml-dqn/src/agent.rs:956-996`

The current code at line 993-995:
```rust
let exposure = super::action_space::ExposureLevel::from_index(action_idx)?;
Ok(super::order_router::OrderRouter::route_default(exposure))
```

- [ ] **Step 1:** When `self.dqn` has `use_branching: true`, delegate to `self.dqn.select_action()` instead of the flat Q-value path. The DQN's own `select_action` already handles branching correctly.
- [ ] **Step 2:** Run `SQLX_OFFLINE=true cargo check -p ml-dqn --lib`
- [ ] **Step 3:** Commit

## Chunk 2: Count Bonus + Monitoring + CUDA

### Task 5: Expand CountBonus to per-branch tracking

**Files:**
- Modify: `crates/ml-dqn/src/count_bonus.rs`

- [ ] **Step 1:** Change from flat `[u64; 5]` to 3 per-branch arrays: `exposure_counts: [u64; 5]`, `order_counts: [u64; 3]`, `urgency_counts: [u64; 3]`. Add `record_factored_action(exposure_idx, order_idx, urgency_idx)` method. The `bonuses()` method returns per-branch bonuses: `([f64; 5], [f64; 3], [f64; 3])`.
- [ ] **Step 2:** Keep backward-compatible `record_action(action_idx: usize)` that only tracks exposure (for non-branching path). Keep `bonuses()` returning `[f64; 5]` for exposure.
- [ ] **Step 3:** Add `bonuses_branched()` → `([f64; 5], [f64; 3], [f64; 3])` for branching path.
- [ ] **Step 4:** Update tests
- [ ] **Step 5:** Run `SQLX_OFFLINE=true cargo test -p ml-dqn --lib -- count_bonus`
- [ ] **Step 6:** Commit

### Task 6: Update CUDA header with branching constants

**Files:**
- Modify: `crates/ml/src/cuda_pipeline/common_device_functions.cuh:24-26`

- [ ] **Step 1:** Add `DQN_ORDER_ACTIONS 3`, `DQN_URGENCY_ACTIONS 3`, `DQN_TOTAL_ACTIONS 45` alongside existing `DQN_NUM_ACTIONS 5`
- [ ] **Step 2:** Commit

### Task 7: Update monitoring to track order_type and urgency distributions

**Files:**
- Modify: `crates/ml/src/trainers/dqn/monitoring.rs`

- [ ] **Step 1:** Add `order_counts: [usize; 3]` and `urgency_counts: [usize; 3]` fields
- [ ] **Step 2:** Add `track_order_type(idx: usize)` and `track_urgency(idx: usize)` methods
- [ ] **Step 3:** Update `track_action()` to also track order and urgency from FactoredAction
- [ ] **Step 4:** Update `log_action_distribution()` to include order type and urgency breakdowns
- [ ] **Step 5:** Run `SQLX_OFFLINE=true cargo check -p ml --lib`
- [ ] **Step 6:** Commit

### Task 8: Update financials.rs action_counts parameter

**Files:**
- Modify: `crates/ml/src/trainers/dqn/financials.rs:29`

- [ ] **Step 1:** Keep `&[usize; 5]` — financials correctly groups by exposure (Buy=Long100+Long50, Sell=Short100+Short50, Hold=Flat). This is correct even with 45 actions since branching decomposes to exposure.
- [ ] **Step 2:** No change needed — verified correct. Skip.

## Chunk 3: Tests + Hyperopt Adapter

### Task 9: Update test hardcoded num_actions

**Files:**
- Modify: Multiple test sites in `crates/ml-dqn/src/dqn.rs` (lines 4823, 4862, 4887, 4985)

- [ ] **Step 1:** Tests at lines 4821-4825, 4860-4864, 4885-4889: keep `num_actions = 5` — this is the exposure branch size, correct for branching. The `use_branching: true` default now applies.
- [ ] **Step 2:** Test at line 4983-4987 already sets `use_branching: true` — no change needed.
- [ ] **Step 3:** Run full test suite: `SQLX_OFFLINE=true cargo test -p ml-dqn --lib`
- [ ] **Step 4:** Commit all fixes

### Task 10: Update hyperopt DQN adapter

**Files:**
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs` — line 3218 (OrderRouter::route_default call)

- [ ] **Step 1:** Check if hyperopt adapter's `route_default` call is in action selection or evaluation. If in evaluation, it should respect `use_branching` from the config. If it's only used for creating a FactoredAction from exposure index during backtest evaluation, leave it — the backtest engine uses exposure for PnL, not order type/urgency.
- [ ] **Step 2:** Run `SQLX_OFFLINE=true cargo check -p ml --lib`
- [ ] **Step 3:** Commit

### Task 11: Run full workspace build + test

- [ ] **Step 1:** `SQLX_OFFLINE=true cargo check --workspace`
- [ ] **Step 2:** `SQLX_OFFLINE=true cargo test -p ml-dqn --lib`
- [ ] **Step 3:** `SQLX_OFFLINE=true cargo test -p ml --lib`
- [ ] **Step 4:** `SQLX_OFFLINE=true cargo clippy -p ml-dqn -p ml -- -D warnings`
- [ ] **Step 5:** Fix any issues found
- [ ] **Step 6:** Final commit
