Files

jgrusewski 77fe520e08 feat(fxt,ml): add fxt train monitor command and update DQN tests for action collapse fix

Add live training metrics monitor CLI command (streaming & one-shot) using
the monitoring gRPC service. Update DQN tests to match post-fix defaults:
IQN disabled, CQL alpha=0.1, v_min/v_max widened, 26D search space.

- train.rs: `fxt train monitor [--once] [--model X] [--interval N]`
- Rewrite gradient collapse test for BF16 mixed precision awareness
- Update inference test config to match trainer defaults (IQN off, CQL on)
- Update production smoke test for 26D parameter space
- Add dqn_action_collapse_fix_test.rs verifying all 6 root cause fixes
- Add planning docs for monitoring service and epoch financial metrics

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-04 22:58:40 +01:00

5.1 KiB

Raw Blame History

DQN Action Collapse Fix — Full Exploration Fix (Approach B)

Date: 2026-03-04 Status: Approved Triggered by: GitLab job #8885 — DQN hyperopt on H100 showing action collapse

Problem

DQN training collapses to 2/45 actions (4.4% diversity), entropy=0.120, Trades=0, Sharpe=0.00 across all epochs. Action A38 (Long100/Market/Aggressive) dominates. Q-values stuck at [-0.48, -0.47] — all 45 actions nearly identical.

Root Causes

RC1: IQN + C51 Train-Test Mismatch (CRITICAL)

Both use_iqn: true and use_distributional: true enabled by default. IQN loss trains base q_network; inference uses dist_dueling network which gets zero gradients.

RC2: CQL + Tight v_min/v_max Squashes Q-Values (CRITICAL)

use_cql: true with cql_alpha=1.0 adds ~ln(45)=3.8 to loss at every step, pushing all Q-values toward uniformity. v_min=-2, v_max=2 constrains C51 distribution to a 4-unit range — insufficient to separate 45 actions.

RC3: Hold Reward 20x Larger Than Trade PnL (CRITICAL)

hold_reward=+0.001 per bar; trade PnL ~0.0001 minus tx costs ~0.00015 = net -0.00005. Hold penalty divided by 1000 (line 933), making it 0.00001 — negligible.

RC4: GPU Path Never Populates pnl_history (HIGH)

When CUDA collects experiences, CPU loop is skipped entirely. pnl_history stays empty, so compute_epoch_financials() reports Trades=0, Sharpe=0.00 even if the agent is trading.

R1: Count bonus not used in select_actions_batch() — UCB exploration is dead code during training
R2: Hyperopt PSO with 5 LHS samples across 25D space — only 1 PSO iteration with 20 trials
R3: Hyperopt v_min/v_max search range [-3,-1] to [1,3] — too narrow for meaningful C51 separation

Changes

1. Config Defaults (`crates/ml/src/dqn/dqn.rs`)

Line	Current	New	Rationale
238	`v_min: -2.0`	`v_min: -10.0`	Room for C51 to separate actions
239	`v_max: 2.0`	`v_max: 10.0`	Same
249	`entropy_coefficient: 0.01`	`entropy_coefficient: 0.05`	5x stronger anti-collapse for 45 actions
254-255	`use_cql: true, cql_alpha: 1.0`	`cql_alpha: 0.1`	Reduce from full-strength to mild conservatism (0.38 vs 3.8 loss penalty). Can disable entirely if still collapsing.
258-259	`use_iqn: true`	`use_iqn: false`	Eliminate train-test mismatch with C51

2. Trainer CQL Hardcode (`crates/ml/src/trainers/dqn/trainer.rs`)

Line	Current	New
443	`use_cql: true` (hardcoded)	`use_cql: hyperparams.use_cql` (configurable, default true)
444	`cql_alpha: 1.0` (hardcoded)	`cql_alpha: hyperparams.cql_alpha` (configurable, default 0.1)

Add use_cql: bool and cql_alpha: f64 to DQNHyperparameters struct so it's tunable from CLI and hyperopt.

3. Reward Function (`crates/ml/src/dqn/reward.rs`)

Change	Current	New
`hold_reward` line 940	`self.config.hold_reward` (+0.001)	Used as-is, but config changed to 0.0
Hold penalty scale lines 933-935	`hold_penalty_weight / 1000`	`hold_penalty_weight` directly (remove /1000)

4. RewardConfig in Trainer (`crates/ml/src/trainers/dqn/trainer.rs`)

Line	Current	New
492	`hold_reward: 0.001`	`hold_reward: 0.0`

5. GPU Financial Metrics (`crates/ml/src/trainers/dqn/trainer.rs`)

After GPU experience collection (after line 1787), populate pnl_history from batch rewards for financial monitoring.

6. Count Bonus in Batch Selection (`crates/ml/src/trainers/dqn/trainer.rs`)

In select_actions_batch(), apply count bonus to Q-values before argmax (same as select_action() does for single actions).

7. Hyperopt Adapter (`crates/ml/src/hyperopt/adapters/dqn.rs`)

Change	Current	New
v_min bounds	`(-3.0, -1.0)`	`(-15.0, -3.0)`
v_max bounds	`(1.0, 3.0)`	`(3.0, 15.0)`
IQN default	not set (inherits true)	`use_iqn: false` explicitly
n_initial	Fixed 5	`max(5, n_dims)` = 25 for DQN

8. Hyperopt v_min/v_max test ranges update

Update test_continuous_bounds assertions to match new ranges.

Files Modified (4)

crates/ml/src/dqn/dqn.rs — DQNConfig defaults (5 values)
crates/ml/src/dqn/reward.rs — hold_penalty scale (remove /1000)
crates/ml/src/trainers/dqn/trainer.rs — CQL default, hold_reward, GPU pnl_history, count bonus in batch
crates/ml/src/hyperopt/adapters/dqn.rs — v_min/v_max ranges, n_initial, IQN default, tests

Validation

All existing unit tests must pass (SQLX_OFFLINE=true cargo test -p ml --lib)
Updated tests for new default values
Next hyperopt run should show: action diversity >20%, non-zero trades, Q-value spread >0.5
GPU path should report meaningful financial metrics (Trades>0, Sharpe!=0)

Risk

Low: all changes are config defaults and reward scaling. No architectural changes.
CQL alpha reduced to 0.1, now configurable via hyperparams — can tune or disable without code changes.
IQN disabled by default but code preserved — can re-enable via config.
Hold reward change may increase trading frequency; hyperopt will find the right balance.

5.1 KiB Raw Blame History