# DQN Action Collapse Fix — Full Exploration Fix (Approach B)

**Date:** 2026-03-04
**Status:** Approved
**Triggered by:** GitLab job #8885 — DQN hyperopt on H100 showing action collapse

## Problem

DQN training collapses to 2/45 actions (4.4% diversity), entropy=0.120, Trades=0, Sharpe=0.00 across all epochs. Action A38 (Long100/Market/Aggressive) dominates. Q-values stuck at [-0.48, -0.47] — all 45 actions nearly identical.

## Root Causes

### RC1: IQN + C51 Train-Test Mismatch (CRITICAL)
Both `use_iqn: true` and `use_distributional: true` enabled by default. IQN loss trains base q_network; inference uses dist_dueling network which gets zero gradients.

### RC2: CQL + Tight v_min/v_max Squashes Q-Values (CRITICAL)
`use_cql: true` with `cql_alpha=1.0` adds ~ln(45)=3.8 to loss at every step, pushing all Q-values toward uniformity. `v_min=-2, v_max=2` constrains C51 distribution to a 4-unit range — insufficient to separate 45 actions.

### RC3: Hold Reward 20x Larger Than Trade PnL (CRITICAL)
`hold_reward=+0.001` per bar; trade PnL ~0.0001 minus tx costs ~0.00015 = net -0.00005. Hold penalty divided by 1000 (line 933), making it 0.00001 — negligible.

### RC4: GPU Path Never Populates pnl_history (HIGH)
When CUDA collects experiences, CPU loop is skipped entirely. `pnl_history` stays empty, so `compute_epoch_financials()` reports Trades=0, Sharpe=0.00 even if the agent is trading.

### Related Issues
- **R1:** Count bonus not used in `select_actions_batch()` — UCB exploration is dead code during training
- **R2:** Hyperopt PSO with 5 LHS samples across 25D space — only 1 PSO iteration with 20 trials
- **R3:** Hyperopt v_min/v_max search range [-3,-1] to [1,3] — too narrow for meaningful C51 separation

## Changes

### 1. Config Defaults (`crates/ml/src/dqn/dqn.rs`)

| Line | Current | New | Rationale |
|------|---------|-----|-----------|
| 238 | `v_min: -2.0` | `v_min: -10.0` | Room for C51 to separate actions |
| 239 | `v_max: 2.0` | `v_max: 10.0` | Same |
| 249 | `entropy_coefficient: 0.01` | `entropy_coefficient: 0.05` | 5x stronger anti-collapse for 45 actions |
| 254-255 | `use_cql: true, cql_alpha: 1.0` | `cql_alpha: 0.1` | Reduce from full-strength to mild conservatism (0.38 vs 3.8 loss penalty). Can disable entirely if still collapsing. |
| 258-259 | `use_iqn: true` | `use_iqn: false` | Eliminate train-test mismatch with C51 |

### 2. Trainer CQL Hardcode (`crates/ml/src/trainers/dqn/trainer.rs`)

| Line | Current | New |
|------|---------|-----|
| 443 | `use_cql: true` (hardcoded) | `use_cql: hyperparams.use_cql` (configurable, default true) |
| 444 | `cql_alpha: 1.0` (hardcoded) | `cql_alpha: hyperparams.cql_alpha` (configurable, default 0.1) |

Add `use_cql: bool` and `cql_alpha: f64` to `DQNHyperparameters` struct so it's tunable from CLI and hyperopt.

### 3. Reward Function (`crates/ml/src/dqn/reward.rs`)

| Change | Current | New |
|--------|---------|-----|
| `hold_reward` line 940 | `self.config.hold_reward` (+0.001) | Used as-is, but config changed to 0.0 |
| Hold penalty scale lines 933-935 | `hold_penalty_weight / 1000` | `hold_penalty_weight` directly (remove /1000) |

### 4. RewardConfig in Trainer (`crates/ml/src/trainers/dqn/trainer.rs`)

| Line | Current | New |
|------|---------|-----|
| 492 | `hold_reward: 0.001` | `hold_reward: 0.0` |

### 5. GPU Financial Metrics (`crates/ml/src/trainers/dqn/trainer.rs`)

After GPU experience collection (after line 1787), populate `pnl_history` from batch rewards for financial monitoring.

### 6. Count Bonus in Batch Selection (`crates/ml/src/trainers/dqn/trainer.rs`)

In `select_actions_batch()`, apply count bonus to Q-values before argmax (same as `select_action()` does for single actions).

### 7. Hyperopt Adapter (`crates/ml/src/hyperopt/adapters/dqn.rs`)

| Change | Current | New |
|--------|---------|-----|
| v_min bounds | `(-3.0, -1.0)` | `(-15.0, -3.0)` |
| v_max bounds | `(1.0, 3.0)` | `(3.0, 15.0)` |
| IQN default | not set (inherits true) | `use_iqn: false` explicitly |
| n_initial | Fixed 5 | `max(5, n_dims)` = 25 for DQN |

### 8. Hyperopt v_min/v_max test ranges update

Update `test_continuous_bounds` assertions to match new ranges.

## Files Modified (4)

1. `crates/ml/src/dqn/dqn.rs` — DQNConfig defaults (5 values)
2. `crates/ml/src/dqn/reward.rs` — hold_penalty scale (remove /1000)
3. `crates/ml/src/trainers/dqn/trainer.rs` — CQL default, hold_reward, GPU pnl_history, count bonus in batch
4. `crates/ml/src/hyperopt/adapters/dqn.rs` — v_min/v_max ranges, n_initial, IQN default, tests

## Validation

- All existing unit tests must pass (`SQLX_OFFLINE=true cargo test -p ml --lib`)
- Updated tests for new default values
- Next hyperopt run should show: action diversity >20%, non-zero trades, Q-value spread >0.5
- GPU path should report meaningful financial metrics (Trades>0, Sharpe!=0)

## Risk

- Low: all changes are config defaults and reward scaling. No architectural changes.
- CQL alpha reduced to 0.1, now configurable via hyperparams — can tune or disable without code changes.
- IQN disabled by default but code preserved — can re-enable via config.
- Hold reward change may increase trading frequency; hyperopt will find the right balance.
