foxhunt/docs/superpowers/plans/2026-03-24-master-trading-agent.md

# MASTER PLAN: Maximum Profitable DQN Trading Agent

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Activate all 25 DQN features into a unified, maximally profitable ES futures swing trading agent — nothing left on the shelf.

**Architecture:** Three-phase training pipeline (Behavioral Cloning → Full-Stack Online RL → Decision Transformer Refinement) with trade-aware hybrid reward, dynamic trade management as environment physics, ensemble consensus action selection, and IQN risk-aware position sizing. Every auxiliary head (IQN, ensemble, attention, curiosity, CQL, DT) injects gradient into the shared trunk via the fused CUDA training pipeline.

**Tech Stack:** CUDA kernels (nvcc precompiled), cuBLAS SGEMM, Rust async, cudarc 0.19.3, CUDA Graphs with split forward/adam phases.

---

## Design Philosophy

### We Are Building a Swing Trader, Not a Scalper

- **Timeframe**: 1-minute bars, holding 5-100+ bars (5 min to 2+ hours)
- **Edge**: Pattern recognition across 4 timeframes (5m/15m/1h/4h) + regime detection
- **Sizing**: Conviction-based via ensemble agreement + IQN risk quantiles + Kelly optimal
- **Exits**: Model learns when to exit from the reward signal, with dynamic guardrails
- **Goal**: Fewer, higher-quality trades that capture multi-bar moves

### The Agent Decides WHAT, the Environment Manages HOW

| Agent Controls | Environment Controls |
|----------------|---------------------|
| Entry timing (when to trade) | Transaction costs (real friction) |
| Exposure level (how much) | Capital floor (25% max drawdown) |
| Order type (limit vs market) | CVaR position scaling (tail risk) |
| Exit timing (when to close) | Kelly sizing (optimal fraction) |
| | Dynamic trailing stop (profit lock) |

### The Reward Teaches Trading, Not Just Returns

```
reward_t = trade_shaping(t) + exit_bonus(t)

trade_shaping(t):
  - IN TRADE: unrealized_change / equity × regime_quality  (dense, per-bar)
  - FLAT: 0.0  (doing nothing is free)

exit_bonus(t):
  - AT EXIT: trade_return × patience_multiplier  (sparse, strong)
  - patience_multiplier = sqrt(hold_time / regime_expected_hold)
  - Rewards patience: same return in fewer bars = less reward
```

---

## Feature Activation Map

### Tier 1: Foundation (14 features — already ACTIVE)
These run every step. No changes needed.

| # | Feature | Status |
|---|---------|--------|
| 1 | Branching DQN (9×3×3) | ACTIVE |
| 2 | C51 Distributional (51 atoms) | ACTIVE |
| 3 | MSE Warmup + Blended Ramp | ACTIVE |
| 4 | Multi-Timeframe Features (16-dim) | ACTIVE |
| 5 | 4-Head Attention on h_s2 | ACTIVE |
| 6 | Expert Demonstrations (MA crossover) | ACTIVE |
| 7 | N-Step Returns (n=3) | ACTIVE |
| 8 | PER Beta Annealing | ACTIVE |
| 9 | Noisy Networks (σ=0.5) | ACTIVE |
| 10 | Circuit Breaker | ACTIVE |
| 11 | Gradient Clipping | ACTIVE |
| 12 | Soft Target Updates (τ=0.005) | ACTIVE |
| 13 | Spectral Normalization | ACTIVE |
| 14 | Financial Metrics Pipeline | ACTIVE |

### Tier 2: Activate via Config (5 features — ACTIVE when flag > 0)
Just need hyperopt to search over non-zero values.

| # | Feature | Config Flag | Default |
|---|---------|-------------|---------|
| 15 | IQN Dual-Head | iqn_lambda | 0.25 |
| 16 | HER Counterfactual | her_ratio | 0.0 → **0.2** |
| 17 | Count Bonus (UCB) | count_bonus_coefficient | 0.0 → **0.1** |
| 18 | Q-Gap Conviction | q_gap_threshold | 0.0 → **0.05** |
| 19 | Regime PER Scaling | (hardcoded true) | ACTIVE |

### Tier 3: Wire into Pipeline (4 features — code exists, needs connection)
GPU kernels exist but aren't called from the training loop.

| # | Feature | What's Missing |
|---|---------|---------------|
| 20 | Kelly Sizing | Apply kelly_fraction to position in experience kernel |
| 21 | CVaR Action Selection | Remove hardcoded `false`, pass IQN quantiles to selector |
| 22 | Curiosity Reward | Wire GpuCuriosityTrainer into experience collector |
| 23 | Decision Transformer | Integrate DT pretrain into Phase 1 of training pipeline |

### Tier 4: Implement GPU Path (3 features — need new CUDA kernels)
Config flags exist but no GPU implementation.

| # | Feature | What's Needed |
|---|---------|--------------|
| 24 | CQL Conservative Loss | GPU kernel for conservative Q-penalty on OOD actions |
| 25 | Ensemble Value Backward | cuBLAS backward through value heads for KL diversity gradient |

### Tier 5: Reward & Trade Management Redesign (the core)

| # | Component | Description |
|---|-----------|-------------|
| 26 | Trade-Aware Hybrid Reward | Dense per-bar shaping + sparse trade-completion |
| 27 | Dynamic Trade Management | Regime-aware trailing stop as environment physics |
| 28 | Three-Phase Training Pipeline | BC → Online RL → DT Refinement |

---

## File Structure

| File | Action | Responsibility |
|------|--------|---------------|
| `crates/ml/src/cuda_pipeline/experience_kernels.cu` | **Rewrite reward section** | Trade-aware hybrid reward, dynamic trailing stop, Kelly/CVaR scaling |
| `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` | Modify | CQL loss kernel, ensemble backward, entropy in fused path |
| `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` | Modify | Wire curiosity reward into experience loop |
| `crates/ml/src/cuda_pipeline/gpu_action_selector.rs` | Modify | CVaR-aware action selection from IQN quantiles |
| `crates/ml/src/trainers/dqn/fused_training.rs` | Modify | DT auxiliary loss step, CQL step, curiosity training step |
| `crates/ml/src/trainers/dqn/config.rs` | Modify | Activate defaults for all Tier 2-3 features |
| `crates/ml/src/trainers/dqn/trainer/training_loop.rs` | Modify | Three-phase pipeline orchestration |
| `crates/ml/src/trainers/dqn/trainer/constructor.rs` | Modify | Remove hardcoded flags, respect config |
| `crates/ml/src/trainers/dqn/trainer/action.rs` | Modify | Ensemble consensus + IQN CVaR + Kelly sizing in action path |
| `crates/ml/src/hyperopt/adapters/dqn.rs` | Modify | Add search ranges for all new config fields |
| `crates/ml/src/trainers/dqn/smoke_tests/training_stability.rs` | Modify | Assert winning trades, positive Sharpe potential |

---

## Task 1: Trade-Aware Hybrid Reward (experience_kernels.cu)

**This is the most critical task.** The reward function determines what the model optimizes for.

**Files:**
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` — rewrite the reward section

### Reward Design

```cuda
/* ═══════════════════════════════════════════════════════════════════
 * REWARD v5: Trade-Aware Hybrid
 *
 * Two components, two timescales:
 *
 * 1. DENSE (every bar): Trade-level unrealized P&L change
 *    reward_dense = (unrealized_t - unrealized_{t-1}) / equity
 *    This is NOT portfolio return — it's specifically about THIS trade.
 *    When flat: reward_dense = 0.0 (no trade, no signal)
 *
 * 2. SPARSE (at trade exit): Full trade return with patience bonus
 *    reward_sparse = trade_return × sqrt(hold_time / expected_hold)
 *    expected_hold = base_hold × regime_factor(ADX)
 *    This rewards holding: same return over more bars = more reward
 *
 * The dense component keeps gradients flowing every bar.
 * The sparse component teaches the actual goal: profitable trades.
 *
 * Loss aversion: negative returns weighted 1.5x (both components)
 * ═══════════════════════════════════════════════════════════════════ */
```

### Dynamic Trade Management (Environment Physics)

```cuda
/* Dynamic trailing stop — activates when trade is profitable.
 *
 * trail_distance = base_trail × vol_scale(CUSUM) × trend_scale(ADX)
 *   - High vol (CUSUM > 0.5): wider trail (let noise pass)
 *   - Strong trend (ADX > 30): wider trail (let trend run)
 *   - Ranging (ADX < 20): tight trail (capture quick profits)
 *
 * The trailing stop is the ONLY auto-exit. No static SL/TP.
 * The model learns entry quality + exit timing through the reward.
 * The trailing stop only prevents giving back large gains.
 *
 * Minimum hold: regime-adaptive, learned via hold_time state feature
 *   - The model SEES hold_time in its state (slot +3 in state_gather)
 *   - It learns that exiting before min_hold means re-entering costs 2x tx
 *   - No hard override — just the natural tx cost penalty
 */
```

- [ ] **Step 1:** Remove the old reward v2 8-component block (DSR, normalized PnL, drawdown, idle, hold penalty, profit-take, trade completion)
- [ ] **Step 2:** Implement reward v5 dense component: per-bar unrealized P&L change for the current trade
- [ ] **Step 3:** Implement reward v5 sparse component: trade-completion return × patience multiplier at exit
- [ ] **Step 4:** Implement dynamic trailing stop: regime-aware trail distance, activates when unrealized > 0
- [ ] **Step 5:** Keep capital floor protection (25% drawdown = game over, done=1)
- [ ] **Step 6:** Keep CVaR/Kelly position scaling (environment physics, unchanged)
- [ ] **Step 7:** Run `SQLX_OFFLINE=true cargo check -p ml` — verify clean build
- [ ] **Step 8:** Commit: "feat: reward v5 — trade-aware hybrid with dynamic trailing stop"

---

## Task 2: Activate Tier 2 Config Defaults

**Files:**
- Modify: `crates/ml/src/trainers/dqn/config.rs`

Set non-zero defaults for features that are already wired but disabled by zero config values.

- [ ] **Step 1:** Set `her_ratio: 0.2` (was 0.0) — 20% of batch gets HER relabeling
- [ ] **Step 2:** Set `count_bonus_coefficient: Some(0.1)` (was 0.0) — UCB exploration active
- [ ] **Step 3:** Set `q_gap_threshold: 0.05` (was 0.0) — conviction filter active with warmup ramp
- [ ] **Step 4:** Set `curiosity_weight: 0.05` (was 0.0) — intrinsic motivation for novel states
- [ ] **Step 5:** Set `use_cql: true` (was false in practice) — conservative Q-learning
- [ ] **Step 6:** Set `ensemble_count: 3` (was 1) — 3-head ensemble for uncertainty
- [ ] **Step 7:** Set `dt_pretrain_epochs: 3` (was 0) — Decision Transformer pretraining
- [ ] **Step 8:** Set `use_cvar_action_selection: true` in hyperparams (was hardcoded false)
- [ ] **Step 9:** Set `enable_kelly_sizing: true, kelly_fractional: 0.5, kelly_max_fraction: 0.25`
- [ ] **Step 10:** Run `SQLX_OFFLINE=true cargo check -p ml` — verify clean build
- [ ] **Step 11:** Commit: "feat: activate all dormant features via config defaults"

---

## Task 3: Wire Kelly Sizing into Experience Kernel

**Files:**
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu`

Kelly criterion is already computed in the experience kernel (ps[14:19] track win/loss stats). It already scales `target_position` at line ~633. Verify it's working and not gated by a dead condition.

- [ ] **Step 1:** Read the Kelly block in the experience kernel. Verify it reads ps[14:17] (win_count, loss_count, sum_wins, sum_losses) and computes kelly_f
- [ ] **Step 2:** Verify the Kelly scaling is NOT inside a dead code branch (no `if use_kelly` guard that's always false)
- [ ] **Step 3:** Verify the Kelly half-Kelly safety multiplier (0.5×) is applied
- [ ] **Step 4:** Add Kelly fraction to the state_gather kernel as an observable feature — the model should SEE its own Kelly sizing to learn from it
- [ ] **Step 5:** Commit: "fix: verify Kelly sizing active in experience kernel + expose as state feature"

---

## Task 4: Wire CVaR Action Selection from IQN

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer/constructor.rs` — remove hardcoded `false`
- Modify: `crates/ml/src/trainers/dqn/trainer/action.rs` — use IQN CVaR for position scaling

The IQN head has `compute_cvar_q()` that returns CVaR (worst-case quantile) Q-values. Use these to scale position sizing: when CVaR is very negative (high downside risk), reduce position.

- [ ] **Step 1:** In constructor.rs, change `use_cvar_action_selection: false` to `use_cvar_action_selection: hyperparams.use_cvar_action_selection`
- [ ] **Step 2:** In action.rs, after Q-value selection but before action output: if IQN is active, compute CVaR Q-values for the selected action
- [ ] **Step 3:** Use CVaR to scale the selected exposure: `effective_exposure = selected_exposure × cvar_scale` where `cvar_scale = sigmoid(cvar_q / threshold)`
- [ ] **Step 4:** Pass the CVaR-scaled exposure to the GPU action selector
- [ ] **Step 5:** Commit: "feat: IQN CVaR-aware position scaling in action selection"

---

## Task 5: Wire Curiosity into GPU Experience Collector

**Files:**
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs`
- Modify: `crates/ml/src/trainers/dqn/fused_training.rs`

The `GpuCuriosityTrainer` exists and its forward model can predict next-state. The prediction error IS the curiosity reward. Wire it into the experience collection loop.

- [ ] **Step 1:** In gpu_experience_collector.rs, after the env_step kernel writes rewards, launch the curiosity forward model on (state, action) pairs
- [ ] **Step 2:** Compute prediction error: `curiosity_reward = ||predicted_next_state - actual_next_state||²`
- [ ] **Step 3:** Add curiosity_reward × curiosity_scale to the experience rewards buffer via SAXPY kernel
- [ ] **Step 4:** In fused_training.rs, after the main training step, train the curiosity forward model on the sampled batch (update the prediction model)
- [ ] **Step 5:** Commit: "feat: GPU curiosity reward in experience collection + forward model training"

---

## Task 6: Implement CQL Conservative Loss (GPU Kernel)

**Files:**
- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs`
- Modify: `crates/ml/src/trainers/dqn/fused_training.rs`

CQL adds a penalty for Q-values on out-of-distribution actions: `L_cql = α × (log_sum_exp(Q(s,a)) - Q(s, a_data))`. This prevents Q-value overestimation on actions the model hasn't tried.

- [ ] **Step 1:** Add inline CUDA kernel `cql_penalty_kernel` in gpu_dqn_trainer.rs:
  - Inputs: Q-values for all actions [B, total_actions], actions taken [B]
  - Computes: logsumexp(Q) - Q(s, a_taken) per sample
  - Writes: per-sample CQL penalty [B]
- [ ] **Step 2:** Add `cql_penalty_grad_kernel` — gradient of CQL penalty w.r.t. Q-values
- [ ] **Step 3:** Compile both kernels in GpuDqnTrainer::new()
- [ ] **Step 4:** Add `launch_cql_penalty()` and `launch_cql_grad()` methods
- [ ] **Step 5:** In fused_training.rs run_full_step(), add CQL as Step 5g (after ensemble):
  ```rust
  if self.hyperparams.use_cql {
      let cql_loss = self.trainer.launch_cql_penalty(cql_alpha)?;
      self.trainer.launch_cql_grad()?;
      // CQL gradient already in d_value/d_adv logits, flows through backward
  }
  ```
- [ ] **Step 6:** Commit: "feat: GPU CQL conservative loss kernel — prevents Q-value overestimation"

---

## Task 7: Implement Ensemble Value Backward

**Files:**
- Modify: `crates/ml/src/trainers/dqn/fused_training.rs`

The ensemble computes KL diversity gradient but doesn't backpropagate it through the value heads to the trunk. This means the trunk doesn't learn to produce diverse representations.

- [ ] **Step 1:** In run_ensemble_step(), after KL gradient is computed, perform cuBLAS SGEMM backward through each ensemble value head
- [ ] **Step 2:** Accumulate the trunk gradient from all K heads via SAXPY into grad_buf (scaled by diversity_weight)
- [ ] **Step 3:** The combined gradient (C51 + IQN + attention + ensemble_diversity) all flow into the single Adam update
- [ ] **Step 4:** Commit: "feat: ensemble value backward — KL diversity gradient flows to shared trunk"

---

## Task 8: Integrate Decision Transformer into Phase 1

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs`
- Modify: `crates/ml/src/trainers/dqn/fused_training.rs`

Decision Transformer pretraining uses historical trajectories to bootstrap the policy. In Phase 1 (first `dt_pretrain_epochs` epochs), the DT loss is an auxiliary objective alongside MSE.

- [ ] **Step 1:** In training_loop.rs, during the first `dt_pretrain_epochs` epochs:
  - Build trajectory sequences from the experience buffer: (state, action, return-to-go) tuples
  - Upload to GPU via DT's trajectory builder kernel
- [ ] **Step 2:** In fused_training.rs, add `run_dt_pretrain_step()`:
  - DT forward: embed → causal attention → action prediction
  - DT loss: cross-entropy on predicted vs actual actions
  - DT backward: gradient through DT layers → trunk embedding gradient
  - SAXPY into grad_buf (scaled by dt_lambda)
- [ ] **Step 3:** After dt_pretrain_epochs, disable DT auxiliary loss (let online RL take over)
- [ ] **Step 4:** Commit: "feat: Decision Transformer pretraining in Phase 1 — behavioral cloning from trajectories"

---

## Task 9: Ensemble Consensus Action Selection

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer/action.rs`

With K=3 ensemble heads, action selection should use the COMMITTEE, not just head 0:

- [ ] **Step 1:** For each state, forward through all K value heads to get K sets of Q-values
- [ ] **Step 2:** Compute mean Q (consensus) and variance (uncertainty) across heads
- [ ] **Step 3:** Action selection: argmax on mean Q (consensus trade)
- [ ] **Step 4:** Position sizing: scale by `1 / (1 + uncertainty)` — high disagreement → smaller position
- [ ] **Step 5:** If uncertainty > threshold: override to Flat (heads can't agree → sit out)
- [ ] **Step 6:** Commit: "feat: ensemble consensus action selection — uncertainty-based sizing and sit-out"

---

## Task 10: Three-Phase Training Pipeline

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs`

Wire the three training phases into the epoch loop:

### Phase 1: Behavioral Cloning + MSE Warmup (epochs 0..dt_pretrain_epochs)
- MSE loss (c51_alpha ≈ 0)
- Expert demonstrations at high ratio (expert_ratio = 0.5)
- Decision Transformer auxiliary loss
- HER relabeling (learn from counterfactuals)

### Phase 2: Full-Stack Online RL (epochs dt_pretrain_epochs..end)
- Blended MSE→C51 ramp (c51_alpha ramps to 1.0)
- Expert ratio decays to 0
- All auxiliary heads active: IQN, ensemble, attention, curiosity, CQL
- HER continues (counterfactual learning)
- Dynamic trade management (trailing stops, CVaR scaling)

### Phase 3: Refinement (last 20% of epochs)
- Pure C51 loss (c51_alpha = 1.0)
- No expert demonstrations
- Ensemble variance-based exploration (replace epsilon-greedy)
- Shrink-and-perturb for plasticity maintenance

- [ ] **Step 1:** Add phase detection logic at epoch boundary:
  ```rust
  let phase = if epoch < dt_pretrain_epochs { Phase::BehavioralCloning }
              else if epoch < total_epochs * 80 / 100 { Phase::OnlineRL }
              else { Phase::Refinement };
  ```
- [ ] **Step 2:** Configure each auxiliary head's weight based on phase
- [ ] **Step 3:** Configure exploration strategy based on phase (epsilon → noisy → ensemble)
- [ ] **Step 4:** Configure expert demo ratio based on phase (high → decay → zero)
- [ ] **Step 5:** Commit: "feat: three-phase training pipeline — BC → Online RL → Refinement"

---

## Task 11: Hyperopt Search Space for All Features

**Files:**
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs`

Add all new config fields to the hyperopt search space so PSO can optimize them:

- [ ] **Step 1:** Add `her_ratio` to search space: range [0.0, 0.5]
- [ ] **Step 2:** Add `curiosity_weight` to search space: range [0.0, 0.2]
- [ ] **Step 3:** Add `cql_alpha` to search space: range [0.0, 1.0]
- [ ] **Step 4:** Add `ensemble_count` to search space: range [1, 5] (integer)
- [ ] **Step 5:** Add `dt_pretrain_epochs` to search space: range [0, 10]
- [ ] **Step 6:** Add `base_trail_distance` to search space: range [0.002, 0.02]
- [ ] **Step 7:** Add `patience_multiplier_scale` to search space: range [0.5, 3.0]
- [ ] **Step 8:** Add `dense_reward_weight` to search space: range [0.01, 0.5]
- [ ] **Step 9:** Add `sparse_reward_weight` to search space: range [0.5, 5.0]
- [ ] **Step 10:** Update `from_continuous()`, `to_continuous()`, `param_names()`, `continuous_bounds()`
- [ ] **Step 11:** Update dimension assertions in tests
- [ ] **Step 12:** Commit: "feat: full hyperopt search space — all 25 features searchable"

---

## Task 12: Smoke Test Validation — Winning Trades

**Files:**
- Modify: `crates/ml/src/trainers/dqn/smoke_tests/training_stability.rs`

The ultimate test: does the model produce winning trades on real data?

- [ ] **Step 1:** In `test_50_epoch_convergence`, assert:
  ```rust
  // With 50 epochs of training on real ES data, the model MUST have some winning trades
  let wins = metrics.additional_metrics.get("winning_trades").copied().unwrap_or(0.0);
  let total = metrics.additional_metrics.get("total_trades").copied().unwrap_or(0.0);
  assert!(total > 50.0, "Must execute >50 trades over 50 epochs");
  assert!(wins > 0.0, "Must have at least 1 winning trade — 0 wins is a bug");
  let win_rate = wins / total.max(1.0);
  assert!(win_rate > 0.1, "Win rate must be >10% — random is 50%. Got {:.1}%", win_rate * 100.0);
  ```
- [ ] **Step 2:** In `test_trading_model_behavior`, assert:
  ```rust
  // Even in 3 epochs, the dynamic trailing stop should lock in some winners
  let wins = metrics.additional_metrics.get("winning_trades").copied().unwrap_or(0.0);
  assert!(wins > 0.0, "Even 3 epochs must produce at least 1 winning trade");
  ```
- [ ] **Step 3:** Run all smoke tests: `FOXHUNT_TEST_DATA=... cargo test -p ml --lib -- smoke_tests --ignored`
- [ ] **Step 4:** Commit: "test: assert winning trades — the model must learn to profit"

---

## Key Design Decisions

### Why Trade-Aware Reward Instead of Pure Per-Bar?

Per-bar mark-to-market (reward v4) gives identical signal for a 1-bar scalp and a 50-bar trend-follow. The model has no reason to prefer multi-bar holding. Trade-aware reward solves this by:
1. Dense component: unrealized P&L change per bar (not portfolio return) — specifically about THIS trade
2. Sparse component: trade return × patience bonus — rewards holding through noise

### Why Dynamic Trailing Stop Instead of Learned Exits?

The model CAN learn exit timing (it sees hold_time, unrealized P&L, ADX in its state). But learning exits requires 10,000+ trades of experience. The trailing stop provides a SAFETY NET while the model develops exit skill:
- Locks in profits when a trade reverses (prevents giving back winners)
- Width adapts to regime (tight in ranging, wide in trending)
- The model learns to ENTER positions that work WITH the trailing stop

As the model improves, the trailing stop triggers less often (the model exits BEFORE the trail). The trail becomes insurance, not the primary exit mechanism.

### Why All 25 Features at Once?

Ablation studies can happen on H100. The smoke test validates the PIPELINE works. Hyperopt on H100 finds the optimal CONFIG. We don't know which features the ES futures market rewards — that's hyperopt's job. Our job is to give hyperopt the maximum search space.

### Why Three Phases?

Phase 1 (BC + DT) bootstraps from expert knowledge — the model starts better than random.
Phase 2 (Online RL) refines from live experience — the model discovers patterns experts can't encode.
Phase 3 (Refinement) consolidates — shrink-and-perturb prevents dead neurons, ensemble exploration prevents local optima.

---

## Execution Order

Tasks 1-3 are the foundation (reward + configs + Kelly). Must be done first.
Tasks 4-8 activate auxiliary heads (can be parallelized).
Task 9 wires ensemble consensus (depends on Task 7).
Task 10 orchestrates the pipeline (depends on all above).
Task 11 enables hyperopt (depends on all above).
Task 12 validates everything (last).

```
Task 1 (reward) ──┐
Task 2 (configs) ──┤
Task 3 (Kelly) ────┤
                   ├──→ Tasks 4-8 (parallel) ──→ Task 9 ──→ Task 10 ──→ Task 11 ──→ Task 12
```