# DQN Action Space & Evaluation Fix

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Fix three fundamental defects that make DQN hyperopt produce degenerate results: reward degeneracy across 45 actions, frozen epoch Sharpe, and action diversity collapse.

**Architecture:** Reduce DQN output from 45 to 5 exposure actions (matching what the simulation actually differentiates), add a deterministic order router for downstream FactoredAction construction, reset epoch metrics properly, and add fill simulation to the portfolio tracker so order type eventually matters.

**Tech Stack:** Rust, Candle ML, CUDA (NVRTC kernels), Argo Workflows

---

## Root Causes

### RC1: Reward degeneracy (45 actions → 5 effective)
The 45-action factored space (5 exposure × 3 order × 3 urgency) produces only 5 meaningfully distinct rewards. The CUDA kernel reward is `pnl = (next_value - current_value) / current_value` — purely portfolio mark-to-market. Order type only affects `tx_rate` (0.005%-0.015%), which is 1000-4000x weaker than PnL (~2%). Urgency is not modeled at all in the CUDA kernel. The Q-network correctly learns 40 of 45 actions are duplicates and collapses to ~2.

### RC2: Frozen epoch Sharpe
`pnl_history` is a ring buffer (cap 1000) that never clears between epochs. By epoch 2, it contains a mix of epoch 1+2 PnL. Since the model barely changes across 8 short hyperopt epochs, the PnL distribution is near-identical, and `compute_epoch_financials()` returns the same Sharpe ±0.001. This makes Sharpe-based early stopping fire at epoch 4 (plateau window) every single trial.

### RC3: Action diversity collapse
With only 5 effective actions and fast Q-value convergence (64k experiences/epoch), the network establishes a large Q-value gap within epoch 1. No exploration mechanism (noisy nets, epsilon floor, count bonus) can overcome a Q-gap of 0.2 when noise magnitude is ~0.01. The diversity entropy in the CUDA kernel further collapses 45 actions to 3 categories (Short/Flat/Long), making the diversity penalty unable to distinguish exposure levels.

---

## Fix Overview

| Phase | What | Files | Risk |
|-------|------|-------|------|
| A | 5-action DQN + order router | ~18 files | Medium (architecture change) |
| B | Per-epoch pnl_history reset + validation Sharpe | ~3 files | Low |
| C | Fill simulation in portfolio tracker + CUDA | ~6 files | Medium (simulation fidelity) |

Phases A and B are required for hyperopt to produce valid results. Phase C enables re-expanding to 45 actions in the future when fill simulation creates genuine reward differentiation.

---

## Phase A: 5-Action Exposure DQN + Order Router

### Task A1: Create OrderRouter module

**Files:**
- Create: `crates/ml/src/dqn/order_router.rs`
- Modify: `crates/ml/src/dqn/mod.rs` (add `pub mod order_router;`)

**Step 1: Write OrderRouter**

```rust
/// Deterministic order routing based on market microstructure.
/// DQN selects exposure level (0-4), OrderRouter selects order type + urgency.
pub struct OrderRouter;

impl OrderRouter {
    /// Route an exposure-level action to a full FactoredAction.
    ///
    /// Rules:
    /// - Spread < median → LimitMaker (save costs)
    /// - Spread > 2x median → Market (ensure fill in wide markets)
    /// - Otherwise → IoC (aggressive limit)
    /// - Volatility high → Aggressive, low → Patient, else Normal
    pub fn route(
        exposure: ExposureLevel,
        spread: f32,
        median_spread: f32,
        volatility: f32,
        median_volatility: f32,
    ) -> FactoredAction {
        let order = if spread < median_spread {
            OrderType::LimitMaker
        } else if spread > 2.0 * median_spread {
            OrderType::Market
        } else {
            OrderType::IoC
        };

        let urgency = if volatility > 1.5 * median_volatility {
            Urgency::Aggressive
        } else if volatility < 0.5 * median_volatility {
            Urgency::Patient
        } else {
            Urgency::Normal
        };

        FactoredAction::new(exposure, order, urgency)
    }

    /// Simple fallback: Market order, Normal urgency.
    pub fn route_default(exposure: ExposureLevel) -> FactoredAction {
        FactoredAction::new(exposure, OrderType::Market, Urgency::Normal)
    }
}
```

**Step 2: Add `ExposureLevel::from_action_index(idx: usize)` to `common/action.rs`**

```rust
impl ExposureLevel {
    /// Convert DQN action index (0-4) to ExposureLevel
    pub fn from_action_index(idx: usize) -> Result<Self, MLError> {
        Self::from_index(idx) // Already exists, same mapping
    }
}
```

**Step 3: Add module to `dqn/mod.rs`**

**Step 4: Write tests for OrderRouter**

---

### Task A2: Change DQN network output from 45 to 5

**Files:**
- Modify: `crates/ml/src/dqn/dqn.rs`
  - `DQNConfig::default()` — `num_actions: 45` → `num_actions: 5`
  - Test `test_dqn_default_config` — update assertion
  - Test `test_dqn_action_selection_basic` — update
  - Test `test_dqn_action_selection_noisy` — update
- Modify: `crates/ml/src/dqn/dueling.rs`
  - Test `test_dueling_default_config` — assertion `num_actions: 45` → `5`
- Modify: `crates/ml/src/dqn/distributional_dueling.rs`
  - Test assertion `num_actions: 45` → `5`
- Modify: `crates/ml/src/dqn/quantile_regression.rs`
  - `QuantileRegressionConfig::default()` — `num_actions: 45` → `5`
  - Test assertion update
- Modify: `crates/ml/src/dqn/factored_q_network.rs`
  - Loop `0..45` → `0..5` in test

**Key constraint:** The advantage head output changes from `[45, ADV_H]` to `[5, ADV_H]`. This affects weight matrix sizes for all DQN variants (standard, dueling, C51, QR-DQN).

---

### Task A3: Update CUDA kernel constants and action mapping

**Files:**
- Modify: `crates/ml/src/cuda_pipeline/common_device_functions.cuh`
  - `#define NUM_ACTIONS 45` → `#define NUM_ACTIONS 5`
  - `diversity_entropy()`: change category mapping from 3 categories to 5 (each action IS an exposure level)
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu`
  - `action_to_exposure()`: remove `/ 9`, direct switch on 0-4
  - `action_to_tx_cost()`: return fixed cost (Market rate) since order type is now determined by OrderRouter, not the DQN action. OR: accept that GPU kernel uses Market cost for all actions (order routing is CPU-side post-hoc).
- Modify: `crates/ml/src/cuda_pipeline/dqn_experience_kernel.cu`
  - Epsilon-greedy: `gpu_random(&rng) * 5.0f` instead of `* 45.0f`
  - Clamp: `if (action_idx >= 5) action_idx = 4;`
- Modify: `crates/ml/src/cuda_pipeline/ppo_experience_kernel.cu`
  - Same changes (PPO uses same NUM_ACTIONS define). NOTE: PPO action space change needs care — verify PPO still uses 45 or also moves to 5. If PPO should stay at 45, we need separate defines (`DQN_NUM_ACTIONS` vs `PPO_NUM_ACTIONS`).

**Critical decision: PPO action space**
PPO also uses `NUM_ACTIONS = 45` in `common_device_functions.cuh`. Options:
1. Move PPO to 5 actions too (consistent, but out of scope)
2. Split into `DQN_NUM_ACTIONS = 5` and `PPO_NUM_ACTIONS = 45` in the shared header
3. Make NUM_ACTIONS a kernel parameter instead of a compile-time constant

**Recommendation:** Option 2 — split defines. PPO change can be a separate effort.

---

### Task A4: Update GPU weight upload for 5 actions

**Files:**
- Modify: `crates/ml/src/cuda_pipeline/gpu_weights.rs`
  - All 4 occurrences of `num_actions: 45` → `num_actions: 5`
  - Weight matrix sizes for advantage head: `[45 × ADV_H]` → `[5 × ADV_H]`
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs`
  - Action clamping: `0..44` → `0..4`
  - Diversity window buffer allocation (if size depends on NUM_ACTIONS)
- Modify: `crates/ml/src/cuda_pipeline/mod.rs`
  - Test data: `(i % 45) as i32` → `(i % 5) as i32`

---

### Task A5: Update DQN agent action selection

**Files:**
- Modify: `crates/ml/src/dqn/agent.rs`
  - `select_action_factored()`: select from 0-4 (exposure index), then use OrderRouter to construct FactoredAction
  - `get_masked_q_values()`: mask 5 actions instead of 45. Profitability check by exposure level.
  - Random exploration: `rng.gen_range(0..5)` instead of `0..45`
  - Remove `is_trade_profitable` per-action-index loop (now loops 0..5)

---

### Task A6: Update count-based exploration for 5 actions

**Files:**
- Modify: `crates/ml/src/dqn/count_bonus.rs`
  - `action_counts: [u64; 45]` → `[u64; 5]`
  - `[0_u64; 45]` → `[0_u64; 5]`
  - Loop `0..45` → `0..5`
  - Bonus magnitude may need re-tuning (fewer actions = more visits per action = lower bonus)

---

### Task A7: Update training loop for 5 actions

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer.rs`
  - `num_actions: 45` → `num_actions: 5` (line 387)
  - `total_action_counts: [usize; 45]` → `[usize; 5]` (lines 1302, 1422)
  - Action clamping: `.clamp(0, 44)` → `.clamp(0, 4)` (line 1789)
  - `rng.gen_range(0..45)` → `0..5` (line 3324)
  - GPU batch action selection: argmax over 5 Q-values
  - After selecting exposure index, call `OrderRouter::route()` to get full FactoredAction
  - Pass FactoredAction to `portfolio_tracker.execute_action()` and `reward_function.calculate_reward()`
  - `select_actions_batch()`: update for 5-action Q-values
  - `select_actions_batch_gpu()`: update for 5-action Q-values

---

### Task A8: Update monitoring for 5 actions

**Files:**
- Modify: `crates/ml/src/trainers/dqn/monitoring.rs`
  - `action_counts: [usize; 45]` → `[usize; 5]`
  - `q_value_counts: [usize; 45]` → `[usize; 5]`
  - `[0; 45]` → `[0; 5]`
  - Diversity logging: "X/5 actions" instead of "X/45 actions"
  - Entropy threshold: adjust from 0.3 (for 45) to match 5-action space (~0.8 for uniform over 5)
  - Low diversity threshold: change from `< 20%` of 45 (=9 actions) to `< 60%` of 5 (=3 exposures)

---

### Task A9: Update financials for 5-action counts

**Files:**
- Modify: `crates/ml/src/trainers/dqn/financials.rs`
  - `action_counts: &[usize; 45]` → `&[usize; 5]`
  - Action distribution: exposure 0 = Short100 (sell), 1 = Short50 (sell), 2 = Flat (hold), 3 = Long50 (buy), 4 = Long100 (buy)
  - `exposure / 9` → direct index mapping
  - Tests: `[0; 45]` → `[0; 5]`, update index references

---

### Task A10: Update curriculum learning for 5 actions

**Files:**
- Modify: `crates/ml/src/trainers/curriculum.rs`
  - `const NUM_ACTIONS: usize = 45` → `5`
  - Mask generation: 5-element masks based on exposure level
  - Test assertions

---

### Task A11: Update hyperopt DQN adapter for 5 actions

**Files:**
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs`
  - `DQNHyperparameters` construction: `num_actions` not directly set here (comes from DQNConfig default)
  - Entropy threshold in search space / objective: adjust for 5-action entropy scale
  - Action diversity penalty in objective function: adjust threshold (was X/45, now X/5)
  - Verify `from_continuous()` / `to_continuous()` don't reference 45 anywhere
  - Update any test vectors that assume 45 actions

---

### Task A12: Update action_space.rs for 5-action mapping

**Files:**
- Modify: `crates/ml/src/dqn/action_space.rs`
  - `interpret_action()`: map 0-4 directly to ExposureLevel, use OrderRouter for full FactoredAction
  - Tests: update bidirectional mapping tests for 5 actions
  - Remove `FactoredAction::from_index(45).is_err()` → `from_index(5).is_err()`

**Note:** `FactoredAction` itself stays at 45 possible values. The change is that the DQN outputs 0-4 (exposure index), and `OrderRouter` constructs the full `FactoredAction`. The `FactoredAction::from_index()` method (0-44) is still used downstream.

---

### Task A13: Verify PPO is unaffected

**Files to check (read-only verification):**
- `crates/ml/src/ppo/ppo.rs` — uses `num_actions: 45` independently
- `crates/ml/src/ppo/adaptive_entropy.rs` — uses `num_actions: 45`
- `crates/ml/src/ppo/entropy_regularization.rs` — `const NUM_ACTIONS: usize = 45`
- `crates/ml/src/ppo/action_masking.rs` — checks `num_actions == 45`
- `crates/ml/src/ppo/action_space.rs` — loops `0..45`
- `crates/ml/src/cuda_pipeline/ppo_experience_kernel.cu` — uses `NUM_ACTIONS`

**Action:** If PPO shares `NUM_ACTIONS` from `common_device_functions.cuh`, split into `DQN_NUM_ACTIONS` and `PPO_NUM_ACTIONS`. PPO stays at 45.

---

## Phase B: Per-Epoch Metrics Reset + Validation Sharpe

### Task B1: Clear pnl_history between epochs

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer.rs`

At the start of each epoch (after `compute_epoch_financials`), clear the PnL history:

```rust
// After computing epoch financials and before starting next epoch's experience collection:
self.pnl_history.clear();
```

**Location:** After the `compute_epoch_financials()` call at line ~2899 and the early stopping check block, before the next epoch's `Starting epoch` log.

This ensures each epoch's Sharpe reflects only that epoch's trading performance, not a stale mix of all prior epochs.

---

### Task B2: Clear action_counts between epochs

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer.rs`

The `monitor.action_counts` is already reset per-epoch (TrainingMonitor is created inside the epoch loop at line ~1759). Verify this is correct.

The `total_action_counts` at line 1422 accumulates across epochs — this is fine for total training stats.

---

### Task B3: Fix Sharpe-based early stopping for non-degenerate case

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer.rs`
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs`

With per-epoch pnl_history reset (B1), Sharpe will now vary between epochs because each epoch trains on fresh experience collection with updated weights. Re-enable early stopping with sensible parameters:

```rust
// In hyperopt adapter:
early_stopping_enabled: true,
plateau_window: (self.epochs / 2).max(5),  // Need at least 5 epochs of data
min_epochs_before_stopping: (self.epochs / 3).max(3),  // Don't stop before epoch 3
```

The improvement threshold should also increase from 0.01 to 0.1 since Sharpe varies more with proper per-epoch reset:

```rust
// In check_early_stopping():
if improvement < 0.1 {  // Was 0.01 — too sensitive
```

---

## Phase C: Fill Simulation (Future — enables 45-action re-expansion)

### Task C1: Add fill probability to FactoredAction

**Files:**
- Modify: `crates/ml/src/common/action.rs`

```rust
impl OrderType {
    /// Base fill probability for this order type.
    /// Market: always fills. LimitMaker: depends on price movement. IoC: high fill rate.
    pub fn base_fill_probability(&self) -> f64 {
        match self {
            OrderType::Market => 1.0,
            OrderType::LimitMaker => 0.0, // Needs bar range check
            OrderType::IoC => 0.90,
        }
    }
}

impl FactoredAction {
    /// Compute fill probability given bar context.
    /// LimitMaker fills when bar range > spread (price moved enough to touch limit).
    /// Urgency scales: Patient=0.7x, Normal=1.0x, Aggressive=1.3x (capped at 1.0).
    pub fn fill_probability(&self, bar_range: f64, spread: f64) -> f64 {
        let base = match self.order {
            OrderType::Market => 1.0,
            OrderType::LimitMaker => {
                if spread <= 0.0 { return 0.5; }
                let range_ratio = bar_range / spread;
                (range_ratio * 0.3).min(0.85) // Cap at 85% (queue position uncertainty)
            },
            OrderType::IoC => 0.90,
        };
        let urgency_mult = match self.urgency {
            Urgency::Patient => 0.7,
            Urgency::Normal => 1.0,
            Urgency::Aggressive => 1.3,
        };
        (base * urgency_mult).min(1.0)
    }
}
```

---

### Task C2: Add fill simulation to PortfolioTracker

**Files:**
- Modify: `crates/ml/src/dqn/portfolio_tracker.rs`

Add `bar_high` and `bar_low` parameters to `execute_action_internal()`:

```rust
pub fn execute_action_with_fill(
    &mut self,
    action: FactoredAction,
    price: f32,
    bar_high: f32,
    bar_low: f32,
    max_position: f32,
) {
    let bar_range = (bar_high - bar_low) as f64;
    let spread = self.avg_spread as f64;
    let fill_prob = action.fill_probability(bar_range, spread);

    // Roll for fill
    let mut rng = rand::thread_rng();
    let roll: f64 = rng.gen();
    if roll > fill_prob {
        // Order didn't fill — no position change, no cost
        return;
    }

    // Price improvement for LimitMaker (earn half the spread instead of paying it)
    let fill_price = match action.order {
        OrderType::LimitMaker => {
            if action.exposure.target_exposure() > 0.0 {
                price - self.avg_spread * 0.5 // Buy at lower price
            } else {
                price + self.avg_spread * 0.5 // Sell at higher price
            }
        },
        _ => price, // Market/IoC fill at current price
    };

    self.execute_action_internal(action, fill_price, max_position, None);
}
```

---

### Task C3: Add fill simulation to CUDA kernel

**Files:**
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu`
- Modify: `crates/ml/src/cuda_pipeline/dqn_experience_kernel.cu`

Add `action_to_fill_prob()` device function:

```cuda
__device__ __forceinline__ float action_to_fill_prob(
    int action_idx, float bar_range, float spread
) {
    // Only relevant when NUM_ACTIONS = 45 (factored space)
    // For NUM_ACTIONS = 5, all actions are exposure-only, fill_prob = 1.0
    #if NUM_ACTIONS <= 5
        return 1.0f;
    #else
        int order_type = (action_idx / 3) % 3;
        float base;
        switch (order_type) {
            case 0: base = 1.0f; break;     // Market: always fills
            case 1: {                         // LimitMaker
                if (spread <= 0.0f) base = 0.5f;
                else base = fminf(bar_range / spread * 0.3f, 0.85f);
                break;
            }
            case 2: base = 0.90f; break;    // IoC
            default: base = 1.0f;
        }
        int urgency = action_idx % 3;
        float urg_mult = (urgency == 0) ? 0.7f : (urgency == 2) ? 1.3f : 1.0f;
        return fminf(base * urg_mult, 1.0f);
    #endif
}
```

In the kernel, after action selection and before portfolio simulation:

```cuda
float fill_prob = action_to_fill_prob(action_idx, bar_high - bar_low, spread);
float fill_roll = gpu_random(&rng);
if (fill_roll > fill_prob) {
    // No fill — skip portfolio update, zero reward for this step
    combined_reward = 0.0f;
    // ... write experience with no position change
    continue; // or set pnl_reward = 0 and skip portfolio logic
}
```

**Note:** This requires passing `bar_high` and `bar_low` to the kernel, which means updating the kernel signature and the Rust launcher.

---

### Task C4: Pass bar context to experience collection

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer.rs`
  - In the experience collection loop, extract bar high/low from training data targets
  - Pass to `portfolio_tracker.execute_action_with_fill()`
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs`
  - Add high/low buffer upload if CUDA kernel needs it

---

### Task C5: Tests for fill simulation

**Files:**
- Modify: `crates/ml/src/dqn/portfolio_tracker.rs` (add tests)
- Modify: `crates/ml/src/common/action.rs` (add fill_probability tests)

Test cases:
- Market order always fills regardless of bar range
- LimitMaker with tiny bar range (< spread) → very low fill probability
- LimitMaker with large bar range (> 3x spread) → ~85% fill probability
- IoC → ~90% fill probability
- Urgency scaling: Patient × 0.7, Aggressive × 1.3 (capped at 1.0)
- Portfolio unchanged when order doesn't fill
- LimitMaker price improvement: buy fills at price - spread/2

---

## Execution Order

```
Phase A (required):
  A1 → A2 → A3 → A4 → A5 → A6 → A7 → A8 → A9 → A10 → A11 → A12 → A13
  (sequential — each task depends on prior)

Phase B (required, can start after A7):
  B1 → B2 → B3

Phase C (future, after A+B validated on H100):
  C1 → C2 → C3 → C4 → C5
```

## Validation

After Phase A+B, run DQN hyperopt on H100:
- `--model dqn --trials 20 --epochs 30 --symbols ES.FUT`
- Expected: action diversity 3-5/5 (60-100%), Sharpe varies across epochs
- Expected: different trials produce different objectives (not all 44.6)
- Expected: early stopping fires only when Sharpe genuinely plateaus (epoch 15+, not epoch 4)
- Compare v10 results against v8 baseline (Sharpe=-284 catastrophe)

After Phase C, re-run with 45 actions:
- Expected: order type diversity (LimitMaker used in tight-spread bars, Market in wide-spread)
- Expected: urgency varies with volatility
- Expected: fill probability creates genuine reward differentiation
