foxhunt/docs/plans/2026-03-04-dqn-action-collapse-implementation.md

# DQN Action Collapse Fix — Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Fix DQN action collapse (2/45 actions, Trades=0, Q-values stuck at [-0.48, -0.47]) by correcting broken defaults, reward bias, GPU metrics gap, and exploration deficiencies.

**Architecture:** 4 files modified. Config defaults fixed (CQL alpha 1.0→0.1 + configurable, IQN disabled, v_min/v_max widened). Reward hold bias eliminated. GPU pnl_history populated. Count bonus wired into batch selection. cql_alpha added to 26D hyperopt search space.

**Tech Stack:** Rust, Candle ML framework, CUDA experience collector

---

### Task 1: Fix DQNConfig Defaults

**Files:**
- Modify: `crates/ml/src/dqn/dqn.rs:238-259`

**Step 1: Edit the defaults**

Change lines 238-259 in `DQNConfig::default()`:

```rust
// FROM:
v_min: -2.0,   // Bug #5: Corrected from -10.0
v_max: 2.0,    // Bug #5: Corrected from 10.0
// ...
entropy_coefficient: 0.01,
// ...
use_cql: true,
cql_alpha: 1.0,
// ...
use_iqn: true,

// TO:
v_min: -10.0,  // Widened: [-2,2] compressed Q-values, [-10,10] gives C51 room to separate 45 actions
v_max: 10.0,
// ...
entropy_coefficient: 0.05,  // 5x stronger anti-collapse for 45-action space
// ...
use_cql: true,
cql_alpha: 0.1,  // Reduced: 1.0 added ~ln(45)=3.8 loss penalty, crushing Q-value differentiation
// ...
use_iqn: false,  // Disabled: IQN trains base q_network but inference uses dist_dueling (zero gradients)
```

**Step 2: Run tests to verify compilation**

Run: `SQLX_OFFLINE=true cargo check -p ml --lib 2>&1 | head -20`
Expected: Compiles (may show warnings, no errors)

**Step 3: Commit**

```bash
git add crates/ml/src/dqn/dqn.rs
git commit -m "fix(ml): correct DQNConfig defaults — widen v_min/v_max, reduce cql_alpha, disable IQN"
```

---

### Task 2: Add cql_alpha to DQNHyperparameters and Wire Through Trainer

**Files:**
- Modify: `crates/ml/src/trainers/dqn/config.rs:413+` (DQNHyperparameters struct)
- Modify: `crates/ml/src/trainers/dqn/trainer.rs:443-444` (hardcoded CQL config)

**Step 1: Add fields to DQNHyperparameters**

In `crates/ml/src/trainers/dqn/config.rs`, add two new fields to `DQNHyperparameters` after the existing fields (near the risk management section):

```rust
/// Enable Conservative Q-Learning (default: true, alpha controls strength)
pub use_cql: bool,
/// CQL regularization strength (default: 0.1, range 0.0-1.0)
/// 0.0 = disabled, 0.1 = mild conservatism, 1.0 = full offline-RL strength
pub cql_alpha: f64,
```

Find every place `DQNHyperparameters` is constructed with struct literals and add the new fields with defaults `use_cql: true, cql_alpha: 0.1`. Key locations:
- `DQNHyperparameters::default()` impl
- Any test fixtures creating DQNHyperparameters

**Step 2: Wire through trainer**

In `crates/ml/src/trainers/dqn/trainer.rs:443-444`, change:

```rust
// FROM:
use_cql: true,
cql_alpha: 1.0,

// TO:
use_cql: hyperparams.use_cql,
cql_alpha: hyperparams.cql_alpha as f32,
```

**Step 3: Run tests**

Run: `SQLX_OFFLINE=true cargo check -p ml --lib 2>&1 | head -20`
Expected: Compiles. Fix any struct literal missing-field errors by adding `use_cql: true, cql_alpha: 0.1`.

**Step 4: Commit**

```bash
git add crates/ml/src/trainers/dqn/config.rs crates/ml/src/trainers/dqn/trainer.rs
git commit -m "feat(ml): make CQL configurable via DQNHyperparameters (default alpha=0.1)"
```

---

### Task 3: Fix Hold Reward Bias

**Files:**
- Modify: `crates/ml/src/dqn/reward.rs:929-935` (hold_penalty /1000 divisor)
- Modify: `crates/ml/src/trainers/dqn/trainer.rs:492` (hold_reward default)

**Step 1: Remove /1000 divisor from hold_penalty**

In `crates/ml/src/dqn/reward.rs:929-935`, change:

```rust
// FROM:
// BUG FIX: Scale hold_penalty_weight to percentage units
// Config value: 0.5-2.0 (raw scalar from hyperopt)
// Scaled value: 0.0005-0.002 (50-200 basis points = 0.05-0.2%)
// This makes hold penalty comparable to transaction costs (0.05-0.15%), not 30x weaker
let hold_penalty_scale = Decimal::try_from(1000.0)
    .unwrap_or(Decimal::ONE);
let hold_penalty_pct = self.config.hold_penalty_weight / hold_penalty_scale;

// TO:
// Hold penalty weight used directly (0.01-2.0 range from hyperopt).
// Previously divided by 1000, making it 0.00001 — negligible vs tx costs (0.05-0.15%).
let hold_penalty_pct = self.config.hold_penalty_weight;
```

**Step 2: Zero hold_reward in trainer RewardConfig**

In `crates/ml/src/trainers/dqn/trainer.rs:492`, change:

```rust
// FROM:
hold_reward: Decimal::try_from(0.001).unwrap_or(Decimal::ZERO),

// TO:
hold_reward: Decimal::ZERO,  // Flat position = no edge = zero reward (was +0.001, 20x trade PnL)
```

**Step 3: Run tests**

Run: `SQLX_OFFLINE=true cargo test -p ml --lib -- reward 2>&1 | tail -20`
Expected: PASS (reward tests should still pass since they test relative behavior, not absolute values)

**Step 4: Commit**

```bash
git add crates/ml/src/dqn/reward.rs crates/ml/src/trainers/dqn/trainer.rs
git commit -m "fix(ml): eliminate hold reward bias — zero hold_reward, remove /1000 penalty divisor"
```

---

### Task 4: Populate pnl_history from GPU Experience Collection

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer.rs:1762-1777` (after GPU batch Ok)

**Step 1: Add pnl_history population**

In `crates/ml/src/trainers/dqn/trainer.rs`, inside the `Ok(batch) =>` arm (after line 1770, before `let experiences = gpu_batch_to_experiences`), add:

```rust
// Populate pnl_history from GPU-collected rewards for financial metrics.
// Without this, compute_epoch_financials() sees an empty deque and
// reports Trades=0, Sharpe=0.00 even when the agent is trading.
for &reward in &batch.rewards {
    self.pnl_history.push_back(reward as f64);
    if self.pnl_history.len() > 1000 {
        self.pnl_history.pop_front();
    }
}
```

**Step 2: Verify compilation**

Run: `SQLX_OFFLINE=true cargo check -p ml --lib 2>&1 | head -10`
Expected: Compiles

**Step 3: Commit**

```bash
git add crates/ml/src/trainers/dqn/trainer.rs
git commit -m "fix(ml): populate pnl_history from GPU experience collector for financial metrics"
```

---

### Task 5: Wire Count Bonus into select_actions_batch

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer.rs:3238-3254` (select_actions_batch)

**Step 1: Add count bonus to batch Q-values**

In `select_actions_batch()`, after the forward pass (line 3241) and before `drop(agent)` (line 3243), add count bonus application:

```rust
// Apply count bonus (UCB exploration) to Q-values before argmax.
// This was previously only applied in DQN::select_action() (single-sample path),
// meaning the batch training path had no UCB exploration — the count bonus was dead code.
let batch_q_values = if agent.config.use_count_bonus {
    let bonuses = agent.get_count_bonuses();  // Vec<f32> of length num_actions
    let bonus_tensor = Tensor::from_vec(bonuses, (1, agent.config.num_actions), batch_q_values.device())
        .map_err(|e| anyhow::anyhow!("Failed to create bonus tensor: {}", e))?
        .to_dtype(batch_q_values.dtype())
        .map_err(|e| anyhow::anyhow!("Failed to cast bonus tensor: {}", e))?;
    batch_q_values.broadcast_add(&bonus_tensor)
        .map_err(|e| anyhow::anyhow!("Failed to add count bonus to Q-values: {}", e))?
} else {
    batch_q_values
};
```

Note: You'll need to check if `get_count_bonuses()` exists on the DQN struct. If it only has `count_bonus.compute_bonuses(num_actions)`, expose a method:

```rust
// In crates/ml/src/dqn/dqn.rs, add to DQN impl:
pub fn get_count_bonuses(&self) -> Vec<f32> {
    self.count_bonus.compute_bonuses(self.config.num_actions)
}
```

**Step 2: Verify compilation**

Run: `SQLX_OFFLINE=true cargo check -p ml --lib 2>&1 | head -10`
Expected: Compiles

**Step 3: Commit**

```bash
git add crates/ml/src/dqn/dqn.rs crates/ml/src/trainers/dqn/trainer.rs
git commit -m "feat(ml): wire count bonus (UCB) into batch action selection for exploration"
```

---

### Task 6: Add cql_alpha to Hyperopt Search Space (25D → 26D)

**Files:**
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs` (multiple functions)

**Step 1: Expand continuous_bounds_for (25→26 params)**

In `continuous_bounds_for()` (~line 504), add after the last bound:

```rust
// CQL conservatism (1D)
(0.0, 0.5),        // 25: cql_alpha (linear, 0.0=disabled, 0.1=mild, 0.5=moderate)
```

**Step 2: Update from_continuous (25→26 params)**

Change the length check from 25 to 26:
```rust
if x.len() != 26 {
    return Err(MLError::ConfigError {
        reason: format!("Expected 26 continuous parameters, got {}", x.len()),
    });
}
```

Add parsing:
```rust
let cql_alpha = x[25].clamp(0.0, 0.5);
```

Add to the returned struct:
```rust
cql_alpha,
```

**Step 3: Update to_continuous (add emission)**

Add at end of the vec:
```rust
self.cql_alpha,                    // 25
```

**Step 4: Update param_names (add name)**

Add at end of the vec:
```rust
"cql_alpha",                // 25
```

**Step 5: Update v_min/v_max search ranges**

Change bounds:
```rust
// FROM:
(-3.0, -1.0),                   // 11: v_min (linear) - Bug #5 fix: center -2.0
(1.0, 3.0),                     // 12: v_max (linear) - Bug #5 fix: center +2.0

// TO:
(-15.0, -3.0),                  // 11: v_min (linear) - widened for C51 action separation
(3.0, 15.0),                    // 12: v_max (linear) - widened for C51 action separation
```

**Step 6: Add cql_alpha field to DQNParams struct**

```rust
pub cql_alpha: f64,
```

**Step 7: Add cql_alpha to DQNParams::default()**

```rust
cql_alpha: 0.1,
```

**Step 8: Wire cql_alpha into DQNHyperparameters construction**

In the function that builds `DQNHyperparameters` from `DQNParams` (~line 2246+):

```rust
use_cql: true,
cql_alpha: params.cql_alpha,
```

**Step 9: Fix IQN default in hyperopt adapter**

In the adapter defaults (~line 647):
```rust
// FROM:
use_qr_dqn: true,

// TO:
use_qr_dqn: false,  // IQN disabled: conflicts with C51 (train-test mismatch)
```

**Step 10: Run tests and fix assertions**

Run: `SQLX_OFFLINE=true cargo test -p ml --lib -- dqn 2>&1 | tail -40`

Fix the tests that assert on the old values:
- `test_continuous_bounds`: Update `assert_eq!(bounds.len(), 26)`, `bounds[11]` to `(-15.0, -3.0)`, `bounds[12]` to `(3.0, 15.0)`, add `bounds[25]` to `(0.0, 0.5)`
- `test_param_names`: Update `assert_eq!(names.len(), 26)`, add `names[25]` = `"cql_alpha"`
- `test_per_params_always_enabled` and similar: Add `0.1` (cql_alpha) to end of continuous vectors (now 26 elements)
- `test_validate_rejects_v_min_gte_v_max`: v_min/v_max test values may need adjustment
- `test_qr_dqn_params_in_default`: Now asserts `!params.use_qr_dqn`

**Step 11: Commit**

```bash
git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "feat(ml): add cql_alpha to 26D hyperopt search space, widen v_min/v_max ranges"
```

---

### Task 7: Run Full Test Suite and Fix Remaining Failures

**Files:**
- All 4 modified files

**Step 1: Run all ml tests**

Run: `SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -30`
Expected: PASS (all tests, fix any remaining failures from struct changes)

**Step 2: Run clippy**

Run: `SQLX_OFFLINE=true cargo clippy -p ml --lib -- -D warnings 2>&1 | tail -20`
Expected: 0 errors, 0 warnings

**Step 3: Run workspace check**

Run: `SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -10`
Expected: Compiles (check that trading_service, fxt, etc. still compile with new DQNHyperparameters fields)

**Step 4: Commit any test fixes**

```bash
git add -A
git commit -m "test(ml): update assertions for new DQN defaults and 26D search space"
```

---

### Task 8: Final Validation Commit

**Step 1: Verify git status is clean**

Run: `git status`

**Step 2: Squash or keep commits as-is**

Keep individual commits for traceability. The final commit history should be:
1. `fix(ml): correct DQNConfig defaults`
2. `feat(ml): make CQL configurable via DQNHyperparameters`
3. `fix(ml): eliminate hold reward bias`
4. `fix(ml): populate pnl_history from GPU experience collector`
5. `feat(ml): wire count bonus (UCB) into batch action selection`
6. `feat(ml): add cql_alpha to 26D hyperopt search space`
7. `test(ml): update assertions for new DQN defaults`

---

## Summary of Expected Behavioral Changes

| Metric | Before (job #8885) | After (expected) |
|--------|-------------------|------------------|
| Action diversity | 2/45 (4.4%) | >9/45 (>20%) |
| Entropy | 0.120 | >0.5 |
| Q-value range | [-0.48, -0.47] (0.01 spread) | >0.5 spread |
| Trades | 0 | >0 |
| Sharpe | 0.00 | Non-zero |
| Hold reward per bar | +0.001 | 0.0 |
| Hold penalty per bar | -0.00001 | -0.01 to -2.0 (direct from hyperopt) |
| CQL loss penalty | ~3.8 per step | ~0.38 per step |
| Count bonus in batch | Dead code | Active |
| GPU financial metrics | Dark (empty pnl_history) | Reporting |