spec: 9-exposure action space design — expand from 5 to 9 levels

25% step increments: {-100%, -75%, -50%, -25%, 0%, +25%, +50%, +75%, +100%}
Total factored actions: 81 (was 45). Branching Q-values: 15 (was 11).
Implementation after reward v2 H100 validation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

docs/superpowers/specs/2026-03-23-9-exposure-action-space-design.md

@@ -0,0 +1,171 @@
+# Expanded Exposure Action Space: 5 → 9 Levels — Design Spec
+
+## Problem
+
+The current Branching DQN has 5 exposure levels: {-100%, -50%, 0%, +50%, +100%}. This is too coarse — the model can only choose between "full position" and "half position", with no ability to scale in/out gradually. With 50 features (42 market + 8 OFI microstructure) and C51 distributional RL, the model has rich state information but blunt position sizing tools.
+
+A trader with conviction in a trend would scale in gradually (25% → 50% → 75%) rather than jumping from flat to 50%. The 5-level discretization forces all-or-nothing decisions that increase transaction costs and drawdown.
+
+## Solution
+
+Expand the exposure branch from 5 to 9 discrete levels with 25% steps:
+
+```
+Index:   0      1      2      3      4     5     6     7     8
+Label: S100   S75    S50    S25   Flat  L25   L50   L75  L100
+Value: -1.00  -0.75  -0.50  -0.25  0.00  0.25  0.50  0.75  1.00
+```
+
+Total factored action space: 9 × 3 × 3 = 81 (was 5 × 3 × 3 = 45).
+Branching DQN outputs: 9 + 3 + 3 = 15 Q-values (was 5 + 3 + 3 = 11).
+
+### Why 9 Levels
+
+- **25% step size**: Each step = ~0.5 ES contracts at max_position=2. This is the minimum meaningful increment for single-instrument futures on 1-minute bars.
+- **Not 11 (20% steps)**: Produces fractional contract counts (0.4 at max_position=2). Since ES contracts are indivisible, this precision is wasted.
+- **Not continuous (DDPG/SAC)**: Continuous action spaces are unstable with non-stationary financial data. Actor-critic training doubles parameters and requires target policy smoothing that interacts badly with market regime changes.
+- **Not a 4th "scale" branch**: Exposure direction and scale are semantically coupled ("how much long" depends on "going long"). Tavakoli (2018) requires branch independence for the dueling decomposition to remain valid.
+
+### Why Not More
+
+Going beyond 9 (e.g., 13 levels at ~15% steps or 21 at 10%) causes:
+- Sample efficiency degradation: exploration needs ~2x more samples per doubling of action count
+- Diminishing returns: the difference between 75% and 85% exposure on ES futures is negligible
+- Q-value resolution: with 101 atoms and v_range=60, each atom covers 1.2 units. More actions = smaller per-action Q-value differences = harder for C51 to resolve
+
+## Impact Analysis
+
+### C51 Distributional RL
+
+- Branch atom tensor: `9 * 101 = 909` floats (was `5 * 101 = 505`). Per batch element.
+- Total branch atoms: `(9+3+3) * 101 = 1515` (was `(5+3+3) * 101 = 1111`). ~36% increase.
+- Shared memory in C51 kernel: `MAX_BRANCH_SIZE * NUM_ATOMS = 9 * 101 = 909 floats = 3636 bytes`. Under 4KB — no occupancy impact on H100 (164KB shmem/SM).
+- VRAM: negligible increase (~1.5MB for batch=1024).
+
+### Q-Gap Conviction Filter
+
+- Flat index changes from 2 to 4 (center of 9-element array).
+- All hardcoded `flat_idx = 2` must become `flat_idx = b0_size / 2`.
+- Filter logic unchanged: `if (Q_best - Q_flat < threshold) → flat`.
+- With more intermediate levels, the filter becomes more useful: the model can express partial conviction (25% long) without triggering the full-position threshold.
+
+### Training Sample Efficiency
+
+- Branching DQN explores 9 exposure actions independently (not 81 combinations).
+- Going from 5 → 9 increases exploration requirement by ~80% for the exposure branch.
+- Estimated convergence slowdown: **10-20% more epochs** (not 80%).
+- Count-bonus exploration (UCB) already tracks per-branch visitation and auto-adapts.
+
+### Network Architecture
+
+- Exposure advantage head: output dim changes from `5 * num_atoms` to `9 * num_atoms`.
+- Weight count increase: `adv_h * 9 * 101 = 116,364` (was `adv_h * 5 * 101 = 64,640`). +51K params. Negligible.
+- No hidden layer changes needed — the shared trunk and value head are unaffected.
+
+## Kernel Changes
+
+### `experience_kernels.cu`
+
+**`exposure_idx_to_fraction()`** — rewrite from switch-case to formula:
+
+```cuda
+__device__ __forceinline__ float exposure_idx_to_fraction(int idx) {
+    /* 9 levels: -1.0, -0.75, -0.50, -0.25, 0.0, +0.25, +0.50, +0.75, +1.0 */
+    return -1.0f + (float)idx * 0.25f;
+}
+```
+
+**Q-gap filter flat_idx** — generalize from hardcoded 2:
+
+```cuda
+/* Flat is always the center of the exposure array */
+int flat_idx = b0_size / 2;  /* 9/2 = 4 */
+```
+
+### `epsilon_greedy_kernel.cu`
+
+All three kernels: replace hardcoded `5` and `4` with runtime `b0_size`:
+
+```cuda
+/* Was: exposure = (int)(gpu_random(&rng) * 5.0f); if (exposure >= 5) exposure = 4; */
+exposure = (int)(gpu_random(&rng) * (float)b0_size);
+if (exposure >= b0_size) exposure = b0_size - 1;
+```
+
+### `common_device_functions.cuh`
+
+```cuda
+#define DQN_NUM_ACTIONS 9          /* was 5 */
+#define DQN_TOTAL_ACTIONS 81       /* was 45 (9 × 3 × 3) */
+#define BRANCH_EXPOSURE_ACTIONS 9  /* was 5 */
+```
+
+### `c51_loss_kernel.cu`
+
+```cuda
+#define MAX_BRANCH_SIZE 9          /* was 5 — shared memory sizing */
+```
+
+## Rust Changes
+
+### `ml-core/src/common/action.rs`
+
+Add 4 new `ExposureLevel` variants: `Short75`, `Short25`, `Long25`, `Long75`.
+Update `from_index()`, `target_exposure()`, `to_trading_action()`.
+All hardcoded `45` → `81`.
+
+### `ml/src/trainers/dqn/monitoring.rs`
+
+Arrays from `[_; 5]` to `[_; 9]`. Add exposure names.
+
+### `ml/src/trainers/dqn/trainer/metrics.rs`
+
+`DIRECTION_LUT: [f32; 9] = [-1.0, -0.75, -0.5, -0.25, 0.0, 0.25, 0.5, 0.75, 1.0]`
+
+### `ml/src/cuda_pipeline/gpu_experience_collector.rs`
+
+`branch_sizes: [9, 3, 3]` (was `[5, 3, 3]`).
+
+## TOML Configuration
+
+```toml
+[branching]
+branch_0_size = 9    # was 5
+branch_1_size = 3    # unchanged
+branch_2_size = 3    # unchanged
+```
+
+## What Stays the Same
+
+- **Order type branch (3 actions)**: LimitMaker / IoC / Market — unchanged.
+- **Urgency branch (3 actions)**: Patient / Normal / Aggressive — unchanged.
+- **Factored action formula**: `exposure * 9 + order * 3 + urgency` — the stride `3 × 3 = 9` is coincidentally unchanged! The divisor in `from_index()` stays 9.
+- **Branching DQN architecture**: shared trunk + 3 independent heads. Only the exposure head output dim changes.
+- **Replay buffer**: stores factored action index (u32) — range changes from 0-44 to 0-80, no structural change.
+- **Hyperopt search space**: branch sizes are not searched (fixed). No new dimensions.
+- **Backtest evaluator**: already parameterized via `DqnBacktestConfig.branch_0_size`.
+
+## Migration / Backward Compatibility
+
+- **Old checkpoints are incompatible** — the exposure head has different weight dimensions. This is expected. No backward compatibility shim needed.
+- **Emergency safe defaults**: keep `use_branching: false` with 5-action fallback for safety. Or update to 9.
+- **Tests**: all `45` → `81` assertions, all `[_; 5]` → `[_; 9]` arrays.
+
+## Testing Strategy
+
+1. **Unit test**: verify `exposure_idx_to_fraction()` produces correct values for 0-8
+2. **Roundtrip test**: `FactoredAction::from_index(i).to_index() == i` for 0-80
+3. **CUDA kernel test**: verify PTX compilation with `BRANCH_0_SIZE=9`
+4. **Smoke test**: 3 epochs with 9-action branching on ES.FUT — loss decreases, Q-values non-zero
+5. **A/B test on H100**: compare hyperopt results with 5 vs 9 exposure levels (same reward function)
+
+## Implementation Sequence
+
+1. Core types (`action.rs`) — add 4 variants, update all methods
+2. CUDA header (`common_device_functions.cuh`) — update macros
+3. CUDA kernels — `exposure_idx_to_fraction()` formula, flat_idx generalization
+4. Monitoring arrays — expand from 5 to 9
+5. Config / TOML — `branch_0_size = 9`
+6. Tests — update all assertions
+7. Compile + smoke test locally
+8. H100 validation