feat(fxt,ml): add fxt train monitor command and update DQN tests for action collapse fix

Add live training metrics monitor CLI command (streaming & one-shot) using the monitoring gRPC service. Update DQN tests to match post-fix defaults: IQN disabled, CQL alpha=0.1, v_min/v_max widened, 26D search space. - train.rs: `fxt train monitor [--once] [--model X] [--interval N]` - Rewrite gradient collapse test for BF16 mixed precision awareness - Update inference test config to match trainer defaults (IQN off, CQL on) - Update production smoke test for 26D parameter space - Add dqn_action_collapse_fix_test.rs verifying all 6 root cause fixes - Add planning docs for monitoring service and epoch financial metrics Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 21:35:07 +01:00
parent f0e6845fb3
commit 77fe520e08
10 changed files with 3761 additions and 190 deletions
--- a/docs/plans/2026-03-02-monitoring-service.md
+++ b/docs/plans/2026-03-02-monitoring-service.md
--- a/docs/plans/2026-03-03-epoch-financial-metrics-design.md
+++ b/docs/plans/2026-03-03-epoch-financial-metrics-design.md
@@ -0,0 +1,230 @@
+# Epoch-Level Financial Metrics in fxt
+
+**Date**: 2026-03-03
+**Status**: Design
+
+## Problem
+
+Training produces per-epoch financial metrics (Sharpe, win rate, max DD) internally,
+but none are exposed through the Prometheus → monitoring service → fxt pipeline.
+The fxt watch TUI and `fxt train monitor` can only show loss, accuracy, and RL
+diagnostics — not the trading-relevant metrics that matter.
+
+## Scope
+
+- Add ~11 new Prometheus gauges for epoch-level financial metrics
+- Add epoch-level backtest evaluation for supervised models (TFT, Mamba2, etc.)
+- Add epoch history ring buffer (last 50 epochs) in monitoring service
+- Wire through proto → monitoring service → fxt state → fxt render
+- Fix: action distribution already tracked but not exposed
+
+## Architecture
+
+```
+Trainer (DQN/PPO/TFT/Mamba2/...)
+  │ Epoch end: run mini-backtest on validation window
+  │ set_epoch_financial_metrics("dqn", "fold_0", sharpe, sortino, ...)
+  │ set_action_distribution_pct("dqn", "fold_0", buy%, sell%, hold%)
+  v
+Prometheus (foxhunt_training_epoch_{sharpe,sortino,win_rate,...})
+  v
+Monitoring Service
+  │ scrapes + maps to proto fields
+  │ stores ring buffer: HashMap<session_key, VecDeque<EpochSnapshot>>
+  v
+GetLiveTrainingMetrics: latest snapshot (existing streaming)
+GetEpochHistory: full ring buffer for selected session (new RPC)
+  v
+fxt watch TUI: detail view shows financial table + sparklines
+fxt train monitor: new financial metrics section
+```
+
+## New Prometheus Gauges
+
+All with `{model, fold}` labels, pushed at epoch end:
+
+| Gauge | Description |
+|---|---|
+| `foxhunt_training_epoch_sharpe` | Sharpe ratio |
+| `foxhunt_training_epoch_sortino` | Sortino ratio |
+| `foxhunt_training_epoch_win_rate` | Win rate (0-1) |
+| `foxhunt_training_epoch_max_drawdown` | Max drawdown (0-1) |
+| `foxhunt_training_epoch_profit_factor` | Gross profit / gross loss |
+| `foxhunt_training_epoch_total_return` | Total return (fractional) |
+| `foxhunt_training_epoch_avg_return` | Avg return per trade |
+| `foxhunt_training_epoch_total_trades` | Trade count in eval |
+| `foxhunt_training_epoch_action_buy_pct` | BUY action % |
+| `foxhunt_training_epoch_action_sell_pct` | SELL action % |
+| `foxhunt_training_epoch_action_hold_pct` | HOLD action % |
+
+Convenience functions:
+- `set_epoch_financial_metrics(model, fold, sharpe, sortino, win_rate, max_dd, profit_factor, total_return, avg_return, total_trades)`
+- `set_action_distribution_pct(model, fold, buy_pct, sell_pct, hold_pct)`
+
+## Proto Changes
+
+### monitoring.proto — TrainingSession (new fields 36-46)
+
+```protobuf
+// Epoch-level financial metrics (from backtest evaluation)
+float epoch_sharpe = 36;
+float epoch_sortino = 37;
+float epoch_win_rate = 38;
+float epoch_max_drawdown = 39;
+float epoch_profit_factor = 40;
+float epoch_total_return = 41;
+float epoch_avg_return = 42;
+uint32 epoch_total_trades = 43;
+// Action distribution (aggregated %)
+float action_buy_pct = 44;
+float action_sell_pct = 45;
+float action_hold_pct = 46;
+```
+
+### monitoring.proto — New RPC + messages
+
+```protobuf
+rpc GetEpochHistory(GetEpochHistoryRequest)
+    returns (GetEpochHistoryResponse);
+
+message GetEpochHistoryRequest {
+  string model = 1;
+  string fold = 2;
+  uint32 max_epochs = 3;  // 0 = all available (up to 50)
+}
+
+message EpochFinancialSnapshot {
+  uint32 epoch = 1;
+  float sharpe = 2;
+  float sortino = 3;
+  float win_rate = 4;
+  float max_drawdown = 5;
+  float profit_factor = 6;
+  float total_return = 7;
+  float avg_return = 8;
+  uint32 total_trades = 9;
+  float loss = 10;
+  float val_loss = 11;
+  float learning_rate = 12;
+  float action_buy_pct = 13;
+  float action_sell_pct = 14;
+  float action_hold_pct = 15;
+}
+
+message GetEpochHistoryResponse {
+  string model = 1;
+  string fold = 2;
+  repeated EpochFinancialSnapshot epochs = 3;
+}
+```
+
+## Supervised Model Evaluation
+
+Supervised models (TFT, Mamba2, TGGN, TLOB, LNN, KAN, xLSTM, Diffusion) produce
+price/return predictions, not discrete actions.
+
+**Mini-backtest at epoch end:**
+1. Run inference on validation window bars
+2. Convert predictions to signals: prediction > threshold → BUY, < -threshold → SELL, else HOLD
+3. Simulate trades using existing `EvaluationEngine` (ml/src/evaluation/)
+4. Extract Sharpe, max DD, win rate, total return, trade count
+5. Push to Prometheus gauges
+
+This reuses `UnifiedTrainable::evaluate()` which already returns `EvaluationResult`
+with financial metrics. The trainers just need to call it at epoch end and push metrics.
+
+## Monitoring Service Changes
+
+### Metric mapper (service.rs)
+
+Add 11 new match arms:
+```rust
+"foxhunt_training_epoch_sharpe" => session.epoch_sharpe = s.value as f32,
+"foxhunt_training_epoch_sortino" => session.epoch_sortino = s.value as f32,
+// ... etc.
+```
+
+### Epoch history store
+
+```rust
+struct EpochHistoryStore {
+    histories: HashMap<String, VecDeque<EpochFinancialSnapshot>>,
+}
+```
+
+- Key: `"{model}/{fold}"`
+- Capacity: 50 epochs per session
+- On each Prometheus scrape that has new `current_epoch`: snapshot all financial
+  fields into the ring buffer
+- Served via `GetEpochHistory` RPC
+
+## fxt Changes
+
+### State (state.rs)
+
+Add 11 fields to `TrainingSession`:
+```rust
+pub epoch_sharpe: f32,
+pub epoch_sortino: f32,
+pub epoch_win_rate: f32,
+pub epoch_max_drawdown: f32,
+pub epoch_profit_factor: f32,
+pub epoch_total_return: f32,
+pub epoch_avg_return: f32,
+pub epoch_total_trades: u32,
+pub action_buy_pct: f32,
+pub action_sell_pct: f32,
+pub action_hold_pct: f32,
+```
+
+Add to `SessionHistory`:
+```rust
+pub sharpe: Vec<f64>,
+pub win_rate: Vec<f64>,
+pub max_drawdown: Vec<f64>,
+pub total_return: Vec<f64>,
+```
+
+### Render (render.rs)
+
+**List view**: Add Sharpe + Win Rate columns to the session table.
+
+**Detail view — Metrics sub-tab**: Full financial metrics table:
+| Metric | Value |
+|---|---|
+| Sharpe Ratio | 2.31 (green/yellow/red) |
+| Sortino Ratio | 3.12 |
+| Win Rate | 55.2% |
+| Max Drawdown | 8.1% |
+| Profit Factor | 1.84 |
+| Total Return | +12.4% |
+| Avg Return/Trade | +0.3% |
+| Total Trades | 142 |
+
+**Detail view — Loss sub-tab**: Add Sharpe sparkline.
+
+**Action distribution**: Bar chart in Metrics or RL sub-tab.
+
+### fxt train monitor
+
+Add `print_financial_metrics()` section after the session table:
+```
+Financial (dqn): Sharpe=2.31 WinRate=55.2% MaxDD=8.1% PF=1.84 Return=+12.4%
+```
+
+## Files Changed
+
+| File | Change |
+|---|---|
+| `crates/common/src/metrics/training_metrics.rs` | +11 gauges, +2 convenience fns |
+| `crates/ml/src/trainers/dqn/trainer.rs` | Push financial metrics at epoch end |
+| `crates/ml/src/trainers/ppo.rs` | Push financial metrics at epoch end |
+| `crates/ml/src/training_pipeline.rs` | Add eval step for supervised trainers |
+| `bin/fxt/proto/monitoring.proto` | +11 fields + new RPC + messages |
+| `services/monitoring_service/proto/monitoring.proto` | Same |
+| `services/monitoring_service/src/service.rs` | +11 match arms + epoch history store + new RPC |
+| `bin/fxt/src/commands/watch/state.rs` | +11 fields on TrainingSession, +4 on SessionHistory |
+| `bin/fxt/src/commands/watch/streams.rs` | Map new proto fields |
+| `bin/fxt/src/commands/watch/render.rs` | Financial table + sparklines + list columns |
+| `bin/fxt/src/commands/train/monitor.rs` | `print_financial_metrics()` section |
+| `bin/fxt/src/client/ml_training_client.rs` | `get_epoch_history()` client method |
--- a/docs/plans/2026-03-03-epoch-financial-metrics-implementation.md
+++ b/docs/plans/2026-03-03-epoch-financial-metrics-implementation.md
--- a/docs/plans/2026-03-04-dqn-action-collapse-fix.md
+++ b/docs/plans/2026-03-04-dqn-action-collapse-fix.md
@@ -0,0 +1,104 @@
+# DQN Action Collapse Fix — Full Exploration Fix (Approach B)
+
+**Date:** 2026-03-04
+**Status:** Approved
+**Triggered by:** GitLab job #8885 — DQN hyperopt on H100 showing action collapse
+
+## Problem
+
+DQN training collapses to 2/45 actions (4.4% diversity), entropy=0.120, Trades=0, Sharpe=0.00 across all epochs. Action A38 (Long100/Market/Aggressive) dominates. Q-values stuck at [-0.48, -0.47] — all 45 actions nearly identical.
+
+## Root Causes
+
+### RC1: IQN + C51 Train-Test Mismatch (CRITICAL)
+Both `use_iqn: true` and `use_distributional: true` enabled by default. IQN loss trains base q_network; inference uses dist_dueling network which gets zero gradients.
+
+### RC2: CQL + Tight v_min/v_max Squashes Q-Values (CRITICAL)
+`use_cql: true` with `cql_alpha=1.0` adds ~ln(45)=3.8 to loss at every step, pushing all Q-values toward uniformity. `v_min=-2, v_max=2` constrains C51 distribution to a 4-unit range — insufficient to separate 45 actions.
+
+### RC3: Hold Reward 20x Larger Than Trade PnL (CRITICAL)
+`hold_reward=+0.001` per bar; trade PnL ~0.0001 minus tx costs ~0.00015 = net -0.00005. Hold penalty divided by 1000 (line 933), making it 0.00001 — negligible.
+
+### RC4: GPU Path Never Populates pnl_history (HIGH)
+When CUDA collects experiences, CPU loop is skipped entirely. `pnl_history` stays empty, so `compute_epoch_financials()` reports Trades=0, Sharpe=0.00 even if the agent is trading.
+
+### Related Issues
+- **R1:** Count bonus not used in `select_actions_batch()` — UCB exploration is dead code during training
+- **R2:** Hyperopt PSO with 5 LHS samples across 25D space — only 1 PSO iteration with 20 trials
+- **R3:** Hyperopt v_min/v_max search range [-3,-1] to [1,3] — too narrow for meaningful C51 separation
+
+## Changes
+
+### 1. Config Defaults (`crates/ml/src/dqn/dqn.rs`)
+
+| Line | Current | New | Rationale |
+|------|---------|-----|-----------|
+| 238 | `v_min: -2.0` | `v_min: -10.0` | Room for C51 to separate actions |
+| 239 | `v_max: 2.0` | `v_max: 10.0` | Same |
+| 249 | `entropy_coefficient: 0.01` | `entropy_coefficient: 0.05` | 5x stronger anti-collapse for 45 actions |
+| 254-255 | `use_cql: true, cql_alpha: 1.0` | `cql_alpha: 0.1` | Reduce from full-strength to mild conservatism (0.38 vs 3.8 loss penalty). Can disable entirely if still collapsing. |
+| 258-259 | `use_iqn: true` | `use_iqn: false` | Eliminate train-test mismatch with C51 |
+
+### 2. Trainer CQL Hardcode (`crates/ml/src/trainers/dqn/trainer.rs`)
+
+| Line | Current | New |
+|------|---------|-----|
+| 443 | `use_cql: true` (hardcoded) | `use_cql: hyperparams.use_cql` (configurable, default true) |
+| 444 | `cql_alpha: 1.0` (hardcoded) | `cql_alpha: hyperparams.cql_alpha` (configurable, default 0.1) |
+
+Add `use_cql: bool` and `cql_alpha: f64` to `DQNHyperparameters` struct so it's tunable from CLI and hyperopt.
+
+### 3. Reward Function (`crates/ml/src/dqn/reward.rs`)
+
+| Change | Current | New |
+|--------|---------|-----|
+| `hold_reward` line 940 | `self.config.hold_reward` (+0.001) | Used as-is, but config changed to 0.0 |
+| Hold penalty scale lines 933-935 | `hold_penalty_weight / 1000` | `hold_penalty_weight` directly (remove /1000) |
+
+### 4. RewardConfig in Trainer (`crates/ml/src/trainers/dqn/trainer.rs`)
+
+| Line | Current | New |
+|------|---------|-----|
+| 492 | `hold_reward: 0.001` | `hold_reward: 0.0` |
+
+### 5. GPU Financial Metrics (`crates/ml/src/trainers/dqn/trainer.rs`)
+
+After GPU experience collection (after line 1787), populate `pnl_history` from batch rewards for financial monitoring.
+
+### 6. Count Bonus in Batch Selection (`crates/ml/src/trainers/dqn/trainer.rs`)
+
+In `select_actions_batch()`, apply count bonus to Q-values before argmax (same as `select_action()` does for single actions).
+
+### 7. Hyperopt Adapter (`crates/ml/src/hyperopt/adapters/dqn.rs`)
+
+| Change | Current | New |
+|--------|---------|-----|
+| v_min bounds | `(-3.0, -1.0)` | `(-15.0, -3.0)` |
+| v_max bounds | `(1.0, 3.0)` | `(3.0, 15.0)` |
+| IQN default | not set (inherits true) | `use_iqn: false` explicitly |
+| n_initial | Fixed 5 | `max(5, n_dims)` = 25 for DQN |
+
+### 8. Hyperopt v_min/v_max test ranges update
+
+Update `test_continuous_bounds` assertions to match new ranges.
+
+## Files Modified (4)
+
+1. `crates/ml/src/dqn/dqn.rs` — DQNConfig defaults (5 values)
+2. `crates/ml/src/dqn/reward.rs` — hold_penalty scale (remove /1000)
+3. `crates/ml/src/trainers/dqn/trainer.rs` — CQL default, hold_reward, GPU pnl_history, count bonus in batch
+4. `crates/ml/src/hyperopt/adapters/dqn.rs` — v_min/v_max ranges, n_initial, IQN default, tests
+
+## Validation
+
+- All existing unit tests must pass (`SQLX_OFFLINE=true cargo test -p ml --lib`)
+- Updated tests for new default values
+- Next hyperopt run should show: action diversity >20%, non-zero trades, Q-value spread >0.5
+- GPU path should report meaningful financial metrics (Trades>0, Sharpe!=0)
+
+## Risk
+
+- Low: all changes are config defaults and reward scaling. No architectural changes.
+- CQL alpha reduced to 0.1, now configurable via hyperparams — can tune or disable without code changes.
+- IQN disabled by default but code preserved — can re-enable via config.
+- Hold reward change may increase trading frequency; hyperopt will find the right balance.
--- a/docs/plans/2026-03-04-dqn-action-collapse-implementation.md
+++ b/docs/plans/2026-03-04-dqn-action-collapse-implementation.md
@@ -0,0 +1,421 @@
+# DQN Action Collapse Fix — Implementation Plan
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
+
+**Goal:** Fix DQN action collapse (2/45 actions, Trades=0, Q-values stuck at [-0.48, -0.47]) by correcting broken defaults, reward bias, GPU metrics gap, and exploration deficiencies.
+
+**Architecture:** 4 files modified. Config defaults fixed (CQL alpha 1.0→0.1 + configurable, IQN disabled, v_min/v_max widened). Reward hold bias eliminated. GPU pnl_history populated. Count bonus wired into batch selection. cql_alpha added to 26D hyperopt search space.
+
+**Tech Stack:** Rust, Candle ML framework, CUDA experience collector
+
+---
+
+### Task 1: Fix DQNConfig Defaults
+
+**Files:**
+- Modify: `crates/ml/src/dqn/dqn.rs:238-259`
+
+**Step 1: Edit the defaults**
+
+Change lines 238-259 in `DQNConfig::default()`:
+
+```rust
+// FROM:
+v_min: -2.0,   // Bug #5: Corrected from -10.0
+v_max: 2.0,    // Bug #5: Corrected from 10.0
+// ...
+entropy_coefficient: 0.01,
+// ...
+use_cql: true,
+cql_alpha: 1.0,
+// ...
+use_iqn: true,
+
+// TO:
+v_min: -10.0,  // Widened: [-2,2] compressed Q-values, [-10,10] gives C51 room to separate 45 actions
+v_max: 10.0,
+// ...
+entropy_coefficient: 0.05,  // 5x stronger anti-collapse for 45-action space
+// ...
+use_cql: true,
+cql_alpha: 0.1,  // Reduced: 1.0 added ~ln(45)=3.8 loss penalty, crushing Q-value differentiation
+// ...
+use_iqn: false,  // Disabled: IQN trains base q_network but inference uses dist_dueling (zero gradients)
+```
+
+**Step 2: Run tests to verify compilation**
+
+Run: `SQLX_OFFLINE=true cargo check -p ml --lib 2>&1 | head -20`
+Expected: Compiles (may show warnings, no errors)
+
+**Step 3: Commit**
+
+```bash
+git add crates/ml/src/dqn/dqn.rs
+git commit -m "fix(ml): correct DQNConfig defaults — widen v_min/v_max, reduce cql_alpha, disable IQN"
+```
+
+---
+
+### Task 2: Add cql_alpha to DQNHyperparameters and Wire Through Trainer
+
+**Files:**
+- Modify: `crates/ml/src/trainers/dqn/config.rs:413+` (DQNHyperparameters struct)
+- Modify: `crates/ml/src/trainers/dqn/trainer.rs:443-444` (hardcoded CQL config)
+
+**Step 1: Add fields to DQNHyperparameters**
+
+In `crates/ml/src/trainers/dqn/config.rs`, add two new fields to `DQNHyperparameters` after the existing fields (near the risk management section):
+
+```rust
+/// Enable Conservative Q-Learning (default: true, alpha controls strength)
+pub use_cql: bool,
+/// CQL regularization strength (default: 0.1, range 0.0-1.0)
+/// 0.0 = disabled, 0.1 = mild conservatism, 1.0 = full offline-RL strength
+pub cql_alpha: f64,
+```
+
+Find every place `DQNHyperparameters` is constructed with struct literals and add the new fields with defaults `use_cql: true, cql_alpha: 0.1`. Key locations:
+- `DQNHyperparameters::default()` impl
+- Any test fixtures creating DQNHyperparameters
+
+**Step 2: Wire through trainer**
+
+In `crates/ml/src/trainers/dqn/trainer.rs:443-444`, change:
+
+```rust
+// FROM:
+use_cql: true,
+cql_alpha: 1.0,
+
+// TO:
+use_cql: hyperparams.use_cql,
+cql_alpha: hyperparams.cql_alpha as f32,
+```
+
+**Step 3: Run tests**
+
+Run: `SQLX_OFFLINE=true cargo check -p ml --lib 2>&1 | head -20`
+Expected: Compiles. Fix any struct literal missing-field errors by adding `use_cql: true, cql_alpha: 0.1`.
+
+**Step 4: Commit**
+
+```bash
+git add crates/ml/src/trainers/dqn/config.rs crates/ml/src/trainers/dqn/trainer.rs
+git commit -m "feat(ml): make CQL configurable via DQNHyperparameters (default alpha=0.1)"
+```
+
+---
+
+### Task 3: Fix Hold Reward Bias
+
+**Files:**
+- Modify: `crates/ml/src/dqn/reward.rs:929-935` (hold_penalty /1000 divisor)
+- Modify: `crates/ml/src/trainers/dqn/trainer.rs:492` (hold_reward default)
+
+**Step 1: Remove /1000 divisor from hold_penalty**
+
+In `crates/ml/src/dqn/reward.rs:929-935`, change:
+
+```rust
+// FROM:
+// BUG FIX: Scale hold_penalty_weight to percentage units
+// Config value: 0.5-2.0 (raw scalar from hyperopt)
+// Scaled value: 0.0005-0.002 (50-200 basis points = 0.05-0.2%)
+// This makes hold penalty comparable to transaction costs (0.05-0.15%), not 30x weaker
+let hold_penalty_scale = Decimal::try_from(1000.0)
+    .unwrap_or(Decimal::ONE);
+let hold_penalty_pct = self.config.hold_penalty_weight / hold_penalty_scale;
+
+// TO:
+// Hold penalty weight used directly (0.01-2.0 range from hyperopt).
+// Previously divided by 1000, making it 0.00001 — negligible vs tx costs (0.05-0.15%).
+let hold_penalty_pct = self.config.hold_penalty_weight;
+```
+
+**Step 2: Zero hold_reward in trainer RewardConfig**
+
+In `crates/ml/src/trainers/dqn/trainer.rs:492`, change:
+
+```rust
+// FROM:
+hold_reward: Decimal::try_from(0.001).unwrap_or(Decimal::ZERO),
+
+// TO:
+hold_reward: Decimal::ZERO,  // Flat position = no edge = zero reward (was +0.001, 20x trade PnL)
+```
+
+**Step 3: Run tests**
+
+Run: `SQLX_OFFLINE=true cargo test -p ml --lib -- reward 2>&1 | tail -20`
+Expected: PASS (reward tests should still pass since they test relative behavior, not absolute values)
+
+**Step 4: Commit**
+
+```bash
+git add crates/ml/src/dqn/reward.rs crates/ml/src/trainers/dqn/trainer.rs
+git commit -m "fix(ml): eliminate hold reward bias — zero hold_reward, remove /1000 penalty divisor"
+```
+
+---
+
+### Task 4: Populate pnl_history from GPU Experience Collection
+
+**Files:**
+- Modify: `crates/ml/src/trainers/dqn/trainer.rs:1762-1777` (after GPU batch Ok)
+
+**Step 1: Add pnl_history population**
+
+In `crates/ml/src/trainers/dqn/trainer.rs`, inside the `Ok(batch) =>` arm (after line 1770, before `let experiences = gpu_batch_to_experiences`), add:
+
+```rust
+// Populate pnl_history from GPU-collected rewards for financial metrics.
+// Without this, compute_epoch_financials() sees an empty deque and
+// reports Trades=0, Sharpe=0.00 even when the agent is trading.
+for &reward in &batch.rewards {
+    self.pnl_history.push_back(reward as f64);
+    if self.pnl_history.len() > 1000 {
+        self.pnl_history.pop_front();
+    }
+}
+```
+
+**Step 2: Verify compilation**
+
+Run: `SQLX_OFFLINE=true cargo check -p ml --lib 2>&1 | head -10`
+Expected: Compiles
+
+**Step 3: Commit**
+
+```bash
+git add crates/ml/src/trainers/dqn/trainer.rs
+git commit -m "fix(ml): populate pnl_history from GPU experience collector for financial metrics"
+```
+
+---
+
+### Task 5: Wire Count Bonus into select_actions_batch
+
+**Files:**
+- Modify: `crates/ml/src/trainers/dqn/trainer.rs:3238-3254` (select_actions_batch)
+
+**Step 1: Add count bonus to batch Q-values**
+
+In `select_actions_batch()`, after the forward pass (line 3241) and before `drop(agent)` (line 3243), add count bonus application:
+
+```rust
+// Apply count bonus (UCB exploration) to Q-values before argmax.
+// This was previously only applied in DQN::select_action() (single-sample path),
+// meaning the batch training path had no UCB exploration — the count bonus was dead code.
+let batch_q_values = if agent.config.use_count_bonus {
+    let bonuses = agent.get_count_bonuses();  // Vec<f32> of length num_actions
+    let bonus_tensor = Tensor::from_vec(bonuses, (1, agent.config.num_actions), batch_q_values.device())
+        .map_err(|e| anyhow::anyhow!("Failed to create bonus tensor: {}", e))?
+        .to_dtype(batch_q_values.dtype())
+        .map_err(|e| anyhow::anyhow!("Failed to cast bonus tensor: {}", e))?;
+    batch_q_values.broadcast_add(&bonus_tensor)
+        .map_err(|e| anyhow::anyhow!("Failed to add count bonus to Q-values: {}", e))?
+} else {
+    batch_q_values
+};
+```
+
+Note: You'll need to check if `get_count_bonuses()` exists on the DQN struct. If it only has `count_bonus.compute_bonuses(num_actions)`, expose a method:
+
+```rust
+// In crates/ml/src/dqn/dqn.rs, add to DQN impl:
+pub fn get_count_bonuses(&self) -> Vec<f32> {
+    self.count_bonus.compute_bonuses(self.config.num_actions)
+}
+```
+
+**Step 2: Verify compilation**
+
+Run: `SQLX_OFFLINE=true cargo check -p ml --lib 2>&1 | head -10`
+Expected: Compiles
+
+**Step 3: Commit**
+
+```bash
+git add crates/ml/src/dqn/dqn.rs crates/ml/src/trainers/dqn/trainer.rs
+git commit -m "feat(ml): wire count bonus (UCB) into batch action selection for exploration"
+```
+
+---
+
+### Task 6: Add cql_alpha to Hyperopt Search Space (25D → 26D)
+
+**Files:**
+- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs` (multiple functions)
+
+**Step 1: Expand continuous_bounds_for (25→26 params)**
+
+In `continuous_bounds_for()` (~line 504), add after the last bound:
+
+```rust
+// CQL conservatism (1D)
+(0.0, 0.5),        // 25: cql_alpha (linear, 0.0=disabled, 0.1=mild, 0.5=moderate)
+```
+
+**Step 2: Update from_continuous (25→26 params)**
+
+Change the length check from 25 to 26:
+```rust
+if x.len() != 26 {
+    return Err(MLError::ConfigError {
+        reason: format!("Expected 26 continuous parameters, got {}", x.len()),
+    });
+}
+```
+
+Add parsing:
+```rust
+let cql_alpha = x[25].clamp(0.0, 0.5);
+```
+
+Add to the returned struct:
+```rust
+cql_alpha,
+```
+
+**Step 3: Update to_continuous (add emission)**
+
+Add at end of the vec:
+```rust
+self.cql_alpha,                    // 25
+```
+
+**Step 4: Update param_names (add name)**
+
+Add at end of the vec:
+```rust
+"cql_alpha",                // 25
+```
+
+**Step 5: Update v_min/v_max search ranges**
+
+Change bounds:
+```rust
+// FROM:
+(-3.0, -1.0),                   // 11: v_min (linear) - Bug #5 fix: center -2.0
+(1.0, 3.0),                     // 12: v_max (linear) - Bug #5 fix: center +2.0
+
+// TO:
+(-15.0, -3.0),                  // 11: v_min (linear) - widened for C51 action separation
+(3.0, 15.0),                    // 12: v_max (linear) - widened for C51 action separation
+```
+
+**Step 6: Add cql_alpha field to DQNParams struct**
+
+```rust
+pub cql_alpha: f64,
+```
+
+**Step 7: Add cql_alpha to DQNParams::default()**
+
+```rust
+cql_alpha: 0.1,
+```
+
+**Step 8: Wire cql_alpha into DQNHyperparameters construction**
+
+In the function that builds `DQNHyperparameters` from `DQNParams` (~line 2246+):
+
+```rust
+use_cql: true,
+cql_alpha: params.cql_alpha,
+```
+
+**Step 9: Fix IQN default in hyperopt adapter**
+
+In the adapter defaults (~line 647):
+```rust
+// FROM:
+use_qr_dqn: true,
+
+// TO:
+use_qr_dqn: false,  // IQN disabled: conflicts with C51 (train-test mismatch)
+```
+
+**Step 10: Run tests and fix assertions**
+
+Run: `SQLX_OFFLINE=true cargo test -p ml --lib -- dqn 2>&1 | tail -40`
+
+Fix the tests that assert on the old values:
+- `test_continuous_bounds`: Update `assert_eq!(bounds.len(), 26)`, `bounds[11]` to `(-15.0, -3.0)`, `bounds[12]` to `(3.0, 15.0)`, add `bounds[25]` to `(0.0, 0.5)`
+- `test_param_names`: Update `assert_eq!(names.len(), 26)`, add `names[25]` = `"cql_alpha"`
+- `test_per_params_always_enabled` and similar: Add `0.1` (cql_alpha) to end of continuous vectors (now 26 elements)
+- `test_validate_rejects_v_min_gte_v_max`: v_min/v_max test values may need adjustment
+- `test_qr_dqn_params_in_default`: Now asserts `!params.use_qr_dqn`
+
+**Step 11: Commit**
+
+```bash
+git add crates/ml/src/hyperopt/adapters/dqn.rs
+git commit -m "feat(ml): add cql_alpha to 26D hyperopt search space, widen v_min/v_max ranges"
+```
+
+---
+
+### Task 7: Run Full Test Suite and Fix Remaining Failures
+
+**Files:**
+- All 4 modified files
+
+**Step 1: Run all ml tests**
+
+Run: `SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -30`
+Expected: PASS (all tests, fix any remaining failures from struct changes)
+
+**Step 2: Run clippy**
+
+Run: `SQLX_OFFLINE=true cargo clippy -p ml --lib -- -D warnings 2>&1 | tail -20`
+Expected: 0 errors, 0 warnings
+
+**Step 3: Run workspace check**
+
+Run: `SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -10`
+Expected: Compiles (check that trading_service, fxt, etc. still compile with new DQNHyperparameters fields)
+
+**Step 4: Commit any test fixes**
+
+```bash
+git add -A
+git commit -m "test(ml): update assertions for new DQN defaults and 26D search space"
+```
+
+---
+
+### Task 8: Final Validation Commit
+
+**Step 1: Verify git status is clean**
+
+Run: `git status`
+
+**Step 2: Squash or keep commits as-is**
+
+Keep individual commits for traceability. The final commit history should be:
+1. `fix(ml): correct DQNConfig defaults`
+2. `feat(ml): make CQL configurable via DQNHyperparameters`
+3. `fix(ml): eliminate hold reward bias`
+4. `fix(ml): populate pnl_history from GPU experience collector`
+5. `feat(ml): wire count bonus (UCB) into batch action selection`
+6. `feat(ml): add cql_alpha to 26D hyperopt search space`
+7. `test(ml): update assertions for new DQN defaults`
+
+---
+
+## Summary of Expected Behavioral Changes
+
+| Metric | Before (job #8885) | After (expected) |
+|--------|-------------------|------------------|
+| Action diversity | 2/45 (4.4%) | >9/45 (>20%) |
+| Entropy | 0.120 | >0.5 |
+| Q-value range | [-0.48, -0.47] (0.01 spread) | >0.5 spread |
+| Trades | 0 | >0 |
+| Sharpe | 0.00 | Non-zero |
+| Hold reward per bar | +0.001 | 0.0 |
+| Hold penalty per bar | -0.00001 | -0.01 to -2.0 (direct from hyperopt) |
+| CQL loss penalty | ~3.8 per step | ~0.38 per step |
+| Count bonus in batch | Dead code | Active |
+| GPU financial metrics | Dark (empty pnl_history) | Reporting |