# Expanded Exposure Action Space: 5 → 9 Levels — Design Spec

## Problem

The current Branching DQN has 5 exposure levels: {-100%, -50%, 0%, +50%, +100%}. This is too coarse — the model can only choose between "full position" and "half position", with no ability to scale in/out gradually. With 50 features (42 market + 8 OFI microstructure) and C51 distributional RL, the model has rich state information but blunt position sizing tools.

A trader with conviction in a trend would scale in gradually (25% → 50% → 75%) rather than jumping from flat to 50%. The 5-level discretization forces all-or-nothing decisions that increase transaction costs and drawdown.

## Solution

Expand the exposure branch from 5 to 9 discrete levels with 25% steps:

```
Index:   0      1      2      3      4     5     6     7     8
Label: S100   S75    S50    S25   Flat  L25   L50   L75  L100
Value: -1.00  -0.75  -0.50  -0.25  0.00  0.25  0.50  0.75  1.00
```

Total factored action space: 9 × 3 × 3 = 81 (was 5 × 3 × 3 = 45).
Branching DQN outputs: 9 + 3 + 3 = 15 Q-values (was 5 + 3 + 3 = 11).

### Why 9 Levels

- **25% step size**: Each step = ~0.5 ES contracts at max_position=2. This is the minimum meaningful increment for single-instrument futures on 1-minute bars.
- **Not 11 (20% steps)**: Produces fractional contract counts (0.4 at max_position=2). Since ES contracts are indivisible, this precision is wasted.
- **Not continuous (DDPG/SAC)**: Continuous action spaces are unstable with non-stationary financial data. Actor-critic training doubles parameters and requires target policy smoothing that interacts badly with market regime changes.
- **Not a 4th "scale" branch**: Exposure direction and scale are semantically coupled ("how much long" depends on "going long"). Tavakoli (2018) requires branch independence for the dueling decomposition to remain valid.

### Why Not More

Going beyond 9 (e.g., 13 levels at ~15% steps or 21 at 10%) causes:
- Sample efficiency degradation: exploration needs ~2x more samples per doubling of action count
- Diminishing returns: the difference between 75% and 85% exposure on ES futures is negligible
- Q-value resolution: with 101 atoms and v_range=60, each atom covers 1.2 units. More actions = smaller per-action Q-value differences = harder for C51 to resolve

## Impact Analysis

### C51 Distributional RL

- Branch atom tensor: `9 * 101 = 909` floats (was `5 * 101 = 505`). Per batch element.
- Total branch atoms: `(9+3+3) * 101 = 1515` (was `(5+3+3) * 101 = 1111`). ~36% increase.
- Shared memory in C51 kernel: `MAX_BRANCH_SIZE * NUM_ATOMS = 9 * 101 = 909 floats = 3636 bytes`. Under 4KB — no occupancy impact on H100 (164KB shmem/SM).
- VRAM: negligible increase (~1.5MB for batch=1024).

### Q-Gap Conviction Filter

- Flat index changes from 2 to 4 (center of 9-element array).
- All hardcoded `flat_idx = 2` must become `flat_idx = b0_size / 2`.
- Filter logic unchanged: `if (Q_best - Q_flat < threshold) → flat`.
- With more intermediate levels, the filter becomes more useful: the model can express partial conviction (25% long) without triggering the full-position threshold.

### Training Sample Efficiency

- Branching DQN explores 9 exposure actions independently (not 81 combinations).
- Going from 5 → 9 increases exploration requirement by ~80% for the exposure branch.
- Estimated convergence slowdown: **10-20% more epochs** (not 80%).
- Count-bonus exploration (UCB) already tracks per-branch visitation and auto-adapts.

### Network Architecture

- Exposure advantage head: output dim changes from `5 * num_atoms` to `9 * num_atoms`.
- Weight count increase: `adv_h * 9 * 101 = 116,364` (was `adv_h * 5 * 101 = 64,640`). +51K params. Negligible.
- No hidden layer changes needed — the shared trunk and value head are unaffected.

## Kernel Changes

### `experience_kernels.cu`

**`exposure_idx_to_fraction()`** — rewrite from switch-case to formula:

```cuda
__device__ __forceinline__ float exposure_idx_to_fraction(int idx) {
    /* 9 levels: -1.0, -0.75, -0.50, -0.25, 0.0, +0.25, +0.50, +0.75, +1.0 */
    return -1.0f + (float)idx * 0.25f;
}
```

**Q-gap filter flat_idx** — generalize from hardcoded 2:

```cuda
/* Flat is always the center of the exposure array */
int flat_idx = b0_size / 2;  /* 9/2 = 4 */
```

### `epsilon_greedy_kernel.cu`

All three kernels: replace hardcoded `5` and `4` with runtime `b0_size`:

```cuda
/* Was: exposure = (int)(gpu_random(&rng) * 5.0f); if (exposure >= 5) exposure = 4; */
exposure = (int)(gpu_random(&rng) * (float)b0_size);
if (exposure >= b0_size) exposure = b0_size - 1;
```

### `common_device_functions.cuh`

```cuda
#define DQN_NUM_ACTIONS 9          /* was 5 */
#define DQN_TOTAL_ACTIONS 81       /* was 45 (9 × 3 × 3) */
#define BRANCH_EXPOSURE_ACTIONS 9  /* was 5 */
```

### `c51_loss_kernel.cu`

```cuda
#define MAX_BRANCH_SIZE 9          /* was 5 — shared memory sizing */
```

## Rust Changes

### `ml-core/src/common/action.rs`

Add 4 new `ExposureLevel` variants: `Short75`, `Short25`, `Long25`, `Long75`.
Update `from_index()`, `target_exposure()`, `to_trading_action()`.
All hardcoded `45` → `81`.

### `ml/src/trainers/dqn/monitoring.rs`

Arrays from `[_; 5]` to `[_; 9]`. Add exposure names.

### `ml/src/trainers/dqn/trainer/metrics.rs`

`DIRECTION_LUT: [f32; 9] = [-1.0, -0.75, -0.5, -0.25, 0.0, 0.25, 0.5, 0.75, 1.0]`

### `ml/src/cuda_pipeline/gpu_experience_collector.rs`

`branch_sizes: [9, 3, 3]` (was `[5, 3, 3]`).

## TOML Configuration

```toml
[branching]
branch_0_size = 9    # was 5
branch_1_size = 3    # unchanged
branch_2_size = 3    # unchanged
```

## What Stays the Same

- **Order type branch (3 actions)**: LimitMaker / IoC / Market — unchanged.
- **Urgency branch (3 actions)**: Patient / Normal / Aggressive — unchanged.
- **Factored action formula**: `exposure * 9 + order * 3 + urgency` — the stride `3 × 3 = 9` is coincidentally unchanged! The divisor in `from_index()` stays 9.
- **Branching DQN architecture**: shared trunk + 3 independent heads. Only the exposure head output dim changes.
- **Replay buffer**: stores factored action index (u32) — range changes from 0-44 to 0-80, no structural change.
- **Hyperopt search space**: branch sizes are not searched (fixed). No new dimensions.
- **Backtest evaluator**: already parameterized via `DqnBacktestConfig.branch_0_size`.

## Migration / Backward Compatibility

- **Old checkpoints are incompatible** — the exposure head has different weight dimensions. This is expected. No backward compatibility shim needed.
- **Emergency safe defaults**: keep `use_branching: false` with 5-action fallback for safety. Or update to 9.
- **Tests**: all `45` → `81` assertions, all `[_; 5]` → `[_; 9]` arrays.

## Testing Strategy

1. **Unit test**: verify `exposure_idx_to_fraction()` produces correct values for 0-8
2. **Roundtrip test**: `FactoredAction::from_index(i).to_index() == i` for 0-80
3. **CUDA kernel test**: verify PTX compilation with `BRANCH_0_SIZE=9`
4. **Smoke test**: 3 epochs with 9-action branching on ES.FUT — loss decreases, Q-values non-zero
5. **A/B test on H100**: compare hyperopt results with 5 vs 9 exposure levels (same reward function)

## Implementation Sequence

1. Core types (`action.rs`) — add 4 variants, update all methods
2. CUDA header (`common_device_functions.cuh`) — update macros
3. CUDA kernels — `exposure_idx_to_fraction()` formula, flat_idx generalization
4. Monitoring arrays — expand from 5 to 9
5. Config / TOML — `branch_0_size = 9`
6. Tests — update all assertions
7. Compile + smoke test locally
8. H100 validation
