foxhunt/docs/plans/2026-03-06-dqn-action-space-fix.md

DQN Action Space & Evaluation Fix

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Fix three fundamental defects that make DQN hyperopt produce degenerate results: reward degeneracy across 45 actions, frozen epoch Sharpe, and action diversity collapse.

Architecture: Reduce DQN output from 45 to 5 exposure actions (matching what the simulation actually differentiates), add a deterministic order router for downstream FactoredAction construction, reset epoch metrics properly, and add fill simulation to the portfolio tracker so order type eventually matters.

Tech Stack: Rust, Candle ML, CUDA (NVRTC kernels), Argo Workflows

---

Root Causes

RC1: Reward degeneracy (45 actions → 5 effective)

The 45-action factored space (5 exposure × 3 order × 3 urgency) produces only 5 meaningfully distinct rewards. The CUDA kernel reward is pnl = (next_value - current_value) / current_value — purely portfolio mark-to-market. Order type only affects tx_rate (0.005%-0.015%), which is 1000-4000x weaker than PnL (~2%). Urgency is not modeled at all in the CUDA kernel. The Q-network correctly learns 40 of 45 actions are duplicates and collapses to ~2.

RC2: Frozen epoch Sharpe

pnl_history is a ring buffer (cap 1000) that never clears between epochs. By epoch 2, it contains a mix of epoch 1+2 PnL. Since the model barely changes across 8 short hyperopt epochs, the PnL distribution is near-identical, and compute_epoch_financials() returns the same Sharpe ±0.001. This makes Sharpe-based early stopping fire at epoch 4 (plateau window) every single trial.

RC3: Action diversity collapse

With only 5 effective actions and fast Q-value convergence (64k experiences/epoch), the network establishes a large Q-value gap within epoch 1. No exploration mechanism (noisy nets, epsilon floor, count bonus) can overcome a Q-gap of 0.2 when noise magnitude is ~0.01. The diversity entropy in the CUDA kernel further collapses 45 actions to 3 categories (Short/Flat/Long), making the diversity penalty unable to distinguish exposure levels.

---

Fix Overview

Phase	What	Files	Risk
A	5-action DQN + order router	~18 files	Medium (architecture change)
B	Per-epoch pnl_history reset + validation Sharpe	~3 files	Low
C	Fill simulation in portfolio tracker + CUDA	~6 files	Medium (simulation fidelity)

Phases A and B are required for hyperopt to produce valid results. Phase C enables re-expanding to 45 actions in the future when fill simulation creates genuine reward differentiation.

---

Phase A: 5-Action Exposure DQN + Order Router

Task A1: Create OrderRouter module

Files:

• Create: crates/ml/src/dqn/order_router.rs
• Modify: crates/ml/src/dqn/mod.rs (add pub mod order_router;)

Step 1: Write OrderRouter

/// Deterministic order routing based on market microstructure.
/// DQN selects exposure level (0-4), OrderRouter selects order type + urgency.
pub struct OrderRouter;

impl OrderRouter {
    /// Route an exposure-level action to a full FactoredAction.
    ///
    /// Rules:
    /// - Spread < median → LimitMaker (save costs)
    /// - Spread > 2x median → Market (ensure fill in wide markets)
    /// - Otherwise → IoC (aggressive limit)
    /// - Volatility high → Aggressive, low → Patient, else Normal
    pub fn route(
        exposure: ExposureLevel,
        spread: f32,
        median_spread: f32,
        volatility: f32,
        median_volatility: f32,
    ) -> FactoredAction {
        let order = if spread < median_spread {
            OrderType::LimitMaker
        } else if spread > 2.0 * median_spread {
            OrderType::Market
        } else {
            OrderType::IoC
        };

        let urgency = if volatility > 1.5 * median_volatility {
            Urgency::Aggressive
        } else if volatility < 0.5 * median_volatility {
            Urgency::Patient
        } else {
            Urgency::Normal
        };

        FactoredAction::new(exposure, order, urgency)
    }

    /// Simple fallback: Market order, Normal urgency.
    pub fn route_default(exposure: ExposureLevel) -> FactoredAction {
        FactoredAction::new(exposure, OrderType::Market, Urgency::Normal)
    }
}

Step 2: Add ExposureLevel::from_action_index(idx: usize) to common/action.rs

impl ExposureLevel {
    /// Convert DQN action index (0-4) to ExposureLevel
    pub fn from_action_index(idx: usize) -> Result<Self, MLError> {
        Self::from_index(idx) // Already exists, same mapping
    }
}

Step 3: Add module to dqn/mod.rs

Step 4: Write tests for OrderRouter

---

Task A2: Change DQN network output from 45 to 5

Files:

• Modify: crates/ml/src/dqn/dqn.rs
  • DQNConfig::default() — num_actions: 45 → num_actions: 5
  • Test test_dqn_default_config — update assertion
  • Test test_dqn_action_selection_basic — update
  • Test test_dqn_action_selection_noisy — update
• Modify: crates/ml/src/dqn/dueling.rs
  • Test test_dueling_default_config — assertion num_actions: 45 → 5
• Modify: crates/ml/src/dqn/distributional_dueling.rs
  • Test assertion num_actions: 45 → 5
• Modify: crates/ml/src/dqn/quantile_regression.rs
  • QuantileRegressionConfig::default() — num_actions: 45 → 5
  • Test assertion update
• Modify: crates/ml/src/dqn/factored_q_network.rs
  • Loop 0..45 → 0..5 in test

Key constraint: The advantage head output changes from [45, ADV_H] to [5, ADV_H]. This affects weight matrix sizes for all DQN variants (standard, dueling, C51, QR-DQN).

---

Task A3: Update CUDA kernel constants and action mapping

Files:

• Modify: crates/ml/src/cuda_pipeline/common_device_functions.cuh
  • #define NUM_ACTIONS 45 → #define NUM_ACTIONS 5
  • diversity_entropy(): change category mapping from 3 categories to 5 (each action IS an exposure level)
• Modify: crates/ml/src/cuda_pipeline/experience_kernels.cu
  • action_to_exposure(): remove / 9, direct switch on 0-4
  • action_to_tx_cost(): return fixed cost (Market rate) since order type is now determined by OrderRouter, not the DQN action. OR: accept that GPU kernel uses Market cost for all actions (order routing is CPU-side post-hoc).
• Modify: crates/ml/src/cuda_pipeline/dqn_experience_kernel.cu
  • Epsilon-greedy: gpu_random(&rng) * 5.0f instead of * 45.0f
  • Clamp: if (action_idx >= 5) action_idx = 4;
• Modify: crates/ml/src/cuda_pipeline/ppo_experience_kernel.cu
  • Same changes (PPO uses same NUM_ACTIONS define). NOTE: PPO action space change needs care — verify PPO still uses 45 or also moves to 5. If PPO should stay at 45, we need separate defines (DQN_NUM_ACTIONS vs PPO_NUM_ACTIONS).

Critical decision: PPO action space PPO also uses NUM_ACTIONS = 45 in common_device_functions.cuh. Options:

1. Move PPO to 5 actions too (consistent, but out of scope)
2. Split into DQN_NUM_ACTIONS = 5 and PPO_NUM_ACTIONS = 45 in the shared header
3. Make NUM_ACTIONS a kernel parameter instead of a compile-time constant

Recommendation: Option 2 — split defines. PPO change can be a separate effort.

---

Task A4: Update GPU weight upload for 5 actions

Files:

• Modify: crates/ml/src/cuda_pipeline/gpu_weights.rs
  • All 4 occurrences of num_actions: 45 → num_actions: 5
  • Weight matrix sizes for advantage head: [45 × ADV_H] → [5 × ADV_H]
• Modify: crates/ml/src/cuda_pipeline/gpu_experience_collector.rs
  • Action clamping: 0..44 → 0..4
  • Diversity window buffer allocation (if size depends on NUM_ACTIONS)
• Modify: crates/ml/src/cuda_pipeline/mod.rs
  • Test data: (i % 45) as i32 → (i % 5) as i32

---

Task A5: Update DQN agent action selection

Files:

• Modify: crates/ml/src/dqn/agent.rs
  • select_action_factored(): select from 0-4 (exposure index), then use OrderRouter to construct FactoredAction
  • get_masked_q_values(): mask 5 actions instead of 45. Profitability check by exposure level.
  • Random exploration: rng.gen_range(0..5) instead of 0..45
  • Remove is_trade_profitable per-action-index loop (now loops 0..5)

---

Task A6: Update count-based exploration for 5 actions

Files:

• Modify: crates/ml/src/dqn/count_bonus.rs
  • action_counts: [u64; 45] → [u64; 5]
  • [0_u64; 45] → [0_u64; 5]
  • Loop 0..45 → 0..5
  • Bonus magnitude may need re-tuning (fewer actions = more visits per action = lower bonus)

---

Task A7: Update training loop for 5 actions

Files:

• Modify: crates/ml/src/trainers/dqn/trainer.rs
  • num_actions: 45 → num_actions: 5 (line 387)
  • total_action_counts: [usize; 45] → [usize; 5] (lines 1302, 1422)
  • Action clamping: .clamp(0, 44) → .clamp(0, 4) (line 1789)
  • rng.gen_range(0..45) → 0..5 (line 3324)
  • GPU batch action selection: argmax over 5 Q-values
  • After selecting exposure index, call OrderRouter::route() to get full FactoredAction
  • Pass FactoredAction to portfolio_tracker.execute_action() and reward_function.calculate_reward()
  • select_actions_batch(): update for 5-action Q-values
  • select_actions_batch_gpu(): update for 5-action Q-values

---

Task A8: Update monitoring for 5 actions

Files:

• Modify: crates/ml/src/trainers/dqn/monitoring.rs
  • action_counts: [usize; 45] → [usize; 5]
  • q_value_counts: [usize; 45] → [usize; 5]
  • [0; 45] → [0; 5]
  • Diversity logging: "X/5 actions" instead of "X/45 actions"
  • Entropy threshold: adjust from 0.3 (for 45) to match 5-action space (~0.8 for uniform over 5)
  • Low diversity threshold: change from < 20% of 45 (=9 actions) to < 60% of 5 (=3 exposures)

---

Task A9: Update financials for 5-action counts

Files:

• Modify: crates/ml/src/trainers/dqn/financials.rs
  • action_counts: &[usize; 45] → &[usize; 5]
  • Action distribution: exposure 0 = Short100 (sell), 1 = Short50 (sell), 2 = Flat (hold), 3 = Long50 (buy), 4 = Long100 (buy)
  • exposure / 9 → direct index mapping
  • Tests: [0; 45] → [0; 5], update index references

---

Task A10: Update curriculum learning for 5 actions

Files:

• Modify: crates/ml/src/trainers/curriculum.rs
  • const NUM_ACTIONS: usize = 45 → 5
  • Mask generation: 5-element masks based on exposure level
  • Test assertions

---

Task A11: Update hyperopt DQN adapter for 5 actions

Files:

• Modify: crates/ml/src/hyperopt/adapters/dqn.rs
  • DQNHyperparameters construction: num_actions not directly set here (comes from DQNConfig default)
  • Entropy threshold in search space / objective: adjust for 5-action entropy scale
  • Action diversity penalty in objective function: adjust threshold (was X/45, now X/5)
  • Verify from_continuous() / to_continuous() don't reference 45 anywhere
  • Update any test vectors that assume 45 actions

---

Task A12: Update action_space.rs for 5-action mapping

Files:

• Modify: crates/ml/src/dqn/action_space.rs
  • interpret_action(): map 0-4 directly to ExposureLevel, use OrderRouter for full FactoredAction
  • Tests: update bidirectional mapping tests for 5 actions
  • Remove FactoredAction::from_index(45).is_err() → from_index(5).is_err()

Note: FactoredAction itself stays at 45 possible values. The change is that the DQN outputs 0-4 (exposure index), and OrderRouter constructs the full FactoredAction. The FactoredAction::from_index() method (0-44) is still used downstream.

---

Task A13: Verify PPO is unaffected

Files to check (read-only verification):

• crates/ml/src/ppo/ppo.rs — uses num_actions: 45 independently
• crates/ml/src/ppo/adaptive_entropy.rs — uses num_actions: 45
• crates/ml/src/ppo/entropy_regularization.rs — const NUM_ACTIONS: usize = 45
• crates/ml/src/ppo/action_masking.rs — checks num_actions == 45
• crates/ml/src/ppo/action_space.rs — loops 0..45
• crates/ml/src/cuda_pipeline/ppo_experience_kernel.cu — uses NUM_ACTIONS

Action: If PPO shares NUM_ACTIONS from common_device_functions.cuh, split into DQN_NUM_ACTIONS and PPO_NUM_ACTIONS. PPO stays at 45.

---

Phase B: Per-Epoch Metrics Reset + Validation Sharpe

Task B1: Clear pnl_history between epochs

Files:

• Modify: crates/ml/src/trainers/dqn/trainer.rs

At the start of each epoch (after compute_epoch_financials), clear the PnL history:

// After computing epoch financials and before starting next epoch's experience collection:
self.pnl_history.clear();

Location: After the compute_epoch_financials() call at line ~2899 and the early stopping check block, before the next epoch's Starting epoch log.

This ensures each epoch's Sharpe reflects only that epoch's trading performance, not a stale mix of all prior epochs.

---

Task B2: Clear action_counts between epochs

Files:

• Modify: crates/ml/src/trainers/dqn/trainer.rs

The monitor.action_counts is already reset per-epoch (TrainingMonitor is created inside the epoch loop at line ~1759). Verify this is correct.

The total_action_counts at line 1422 accumulates across epochs — this is fine for total training stats.

---

Task B3: Fix Sharpe-based early stopping for non-degenerate case

Files:

• Modify: crates/ml/src/trainers/dqn/trainer.rs
• Modify: crates/ml/src/hyperopt/adapters/dqn.rs

With per-epoch pnl_history reset (B1), Sharpe will now vary between epochs because each epoch trains on fresh experience collection with updated weights. Re-enable early stopping with sensible parameters:

// In hyperopt adapter:
early_stopping_enabled: true,
plateau_window: (self.epochs / 2).max(5),  // Need at least 5 epochs of data
min_epochs_before_stopping: (self.epochs / 3).max(3),  // Don't stop before epoch 3

The improvement threshold should also increase from 0.01 to 0.1 since Sharpe varies more with proper per-epoch reset:

// In check_early_stopping():
if improvement < 0.1 {  // Was 0.01 — too sensitive

---

Phase C: Fill Simulation (Future — enables 45-action re-expansion)

Task C1: Add fill probability to FactoredAction

Files:

• Modify: crates/ml/src/common/action.rs

impl OrderType {
    /// Base fill probability for this order type.
    /// Market: always fills. LimitMaker: depends on price movement. IoC: high fill rate.
    pub fn base_fill_probability(&self) -> f64 {
        match self {
            OrderType::Market => 1.0,
            OrderType::LimitMaker => 0.0, // Needs bar range check
            OrderType::IoC => 0.90,
        }
    }
}

impl FactoredAction {
    /// Compute fill probability given bar context.
    /// LimitMaker fills when bar range > spread (price moved enough to touch limit).
    /// Urgency scales: Patient=0.7x, Normal=1.0x, Aggressive=1.3x (capped at 1.0).
    pub fn fill_probability(&self, bar_range: f64, spread: f64) -> f64 {
        let base = match self.order {
            OrderType::Market => 1.0,
            OrderType::LimitMaker => {
                if spread <= 0.0 { return 0.5; }
                let range_ratio = bar_range / spread;
                (range_ratio * 0.3).min(0.85) // Cap at 85% (queue position uncertainty)
            },
            OrderType::IoC => 0.90,
        };
        let urgency_mult = match self.urgency {
            Urgency::Patient => 0.7,
            Urgency::Normal => 1.0,
            Urgency::Aggressive => 1.3,
        };
        (base * urgency_mult).min(1.0)
    }
}

---

Task C2: Add fill simulation to PortfolioTracker

Files:

• Modify: crates/ml/src/dqn/portfolio_tracker.rs

Add bar_high and bar_low parameters to execute_action_internal():

pub fn execute_action_with_fill(
    &mut self,
    action: FactoredAction,
    price: f32,
    bar_high: f32,
    bar_low: f32,
    max_position: f32,
) {
    let bar_range = (bar_high - bar_low) as f64;
    let spread = self.avg_spread as f64;
    let fill_prob = action.fill_probability(bar_range, spread);

    // Roll for fill
    let mut rng = rand::thread_rng();
    let roll: f64 = rng.gen();
    if roll > fill_prob {
        // Order didn't fill — no position change, no cost
        return;
    }

    // Price improvement for LimitMaker (earn half the spread instead of paying it)
    let fill_price = match action.order {
        OrderType::LimitMaker => {
            if action.exposure.target_exposure() > 0.0 {
                price - self.avg_spread * 0.5 // Buy at lower price
            } else {
                price + self.avg_spread * 0.5 // Sell at higher price
            }
        },
        _ => price, // Market/IoC fill at current price
    };

    self.execute_action_internal(action, fill_price, max_position, None);
}

---

Task C3: Add fill simulation to CUDA kernel

Files:

• Modify: crates/ml/src/cuda_pipeline/experience_kernels.cu
• Modify: crates/ml/src/cuda_pipeline/dqn_experience_kernel.cu

Add action_to_fill_prob() device function:

__device__ __forceinline__ float action_to_fill_prob(
    int action_idx, float bar_range, float spread
) {
    // Only relevant when NUM_ACTIONS = 45 (factored space)
    // For NUM_ACTIONS = 5, all actions are exposure-only, fill_prob = 1.0
    #if NUM_ACTIONS <= 5
        return 1.0f;
    #else
        int order_type = (action_idx / 3) % 3;
        float base;
        switch (order_type) {
            case 0: base = 1.0f; break;     // Market: always fills
            case 1: {                         // LimitMaker
                if (spread <= 0.0f) base = 0.5f;
                else base = fminf(bar_range / spread * 0.3f, 0.85f);
                break;
            }
            case 2: base = 0.90f; break;    // IoC
            default: base = 1.0f;
        }
        int urgency = action_idx % 3;
        float urg_mult = (urgency == 0) ? 0.7f : (urgency == 2) ? 1.3f : 1.0f;
        return fminf(base * urg_mult, 1.0f);
    #endif
}

In the kernel, after action selection and before portfolio simulation:

float fill_prob = action_to_fill_prob(action_idx, bar_high - bar_low, spread);
float fill_roll = gpu_random(&rng);
if (fill_roll > fill_prob) {
    // No fill — skip portfolio update, zero reward for this step
    combined_reward = 0.0f;
    // ... write experience with no position change
    continue; // or set pnl_reward = 0 and skip portfolio logic
}

Note: This requires passing bar_high and bar_low to the kernel, which means updating the kernel signature and the Rust launcher.

---

Task C4: Pass bar context to experience collection

Files:

• Modify: crates/ml/src/trainers/dqn/trainer.rs
  • In the experience collection loop, extract bar high/low from training data targets
  • Pass to portfolio_tracker.execute_action_with_fill()
• Modify: crates/ml/src/cuda_pipeline/gpu_experience_collector.rs
  • Add high/low buffer upload if CUDA kernel needs it

---

Task C5: Tests for fill simulation

Files:

• Modify: crates/ml/src/dqn/portfolio_tracker.rs (add tests)
• Modify: crates/ml/src/common/action.rs (add fill_probability tests)

Test cases:

• Market order always fills regardless of bar range
• LimitMaker with tiny bar range (< spread) → very low fill probability
• LimitMaker with large bar range (> 3x spread) → ~85% fill probability
• IoC → ~90% fill probability
• Urgency scaling: Patient × 0.7, Aggressive × 1.3 (capped at 1.0)
• Portfolio unchanged when order doesn't fill
• LimitMaker price improvement: buy fills at price - spread/2

---

Execution Order

Phase A (required):
  A1 → A2 → A3 → A4 → A5 → A6 → A7 → A8 → A9 → A10 → A11 → A12 → A13
  (sequential — each task depends on prior)

Phase B (required, can start after A7):
  B1 → B2 → B3

Phase C (future, after A+B validated on H100):
  C1 → C2 → C3 → C4 → C5

Validation

After Phase A+B, run DQN hyperopt on H100:

• --model dqn --trials 20 --epochs 30 --symbols ES.FUT
• Expected: action diversity 3-5/5 (60-100%), Sharpe varies across epochs
• Expected: different trials produce different objectives (not all 44.6)
• Expected: early stopping fires only when Sharpe genuinely plateaus (epoch 15+, not epoch 4)
• Compare v10 results against v8 baseline (Sharpe=-284 catastrophe)

After Phase C, re-run with 45 actions:

• Expected: order type diversity (LimitMaker used in tight-spread bars, Market in wide-spread)
• Expected: urgency varies with volatility
• Expected: fill probability creates genuine reward differentiation