foxhunt/docs/superpowers/specs/2026-03-23-9-exposure-action-space-design.md

Expanded Exposure Action Space: 5 → 9 Levels — Design Spec

Problem

The current Branching DQN has 5 exposure levels: {-100%, -50%, 0%, +50%, +100%}. This is too coarse — the model can only choose between "full position" and "half position", with no ability to scale in/out gradually. With 50 features (42 market + 8 OFI microstructure) and C51 distributional RL, the model has rich state information but blunt position sizing tools.

A trader with conviction in a trend would scale in gradually (25% → 50% → 75%) rather than jumping from flat to 50%. The 5-level discretization forces all-or-nothing decisions that increase transaction costs and drawdown.

Solution

Expand the exposure branch from 5 to 9 discrete levels with 25% steps:

Index:   0      1      2      3      4     5     6     7     8
Label: S100   S75    S50    S25   Flat  L25   L50   L75  L100
Value: -1.00  -0.75  -0.50  -0.25  0.00  0.25  0.50  0.75  1.00

Total factored action space: 9 × 3 × 3 = 81 (was 5 × 3 × 3 = 45). Branching DQN outputs: 9 + 3 + 3 = 15 Q-values (was 5 + 3 + 3 = 11).

Why 9 Levels

• 25% step size: Each step = ~0.5 ES contracts at max_position=2. This is the minimum meaningful increment for single-instrument futures on 1-minute bars.
• Not 11 (20% steps): Produces fractional contract counts (0.4 at max_position=2). Since ES contracts are indivisible, this precision is wasted.
• Not continuous (DDPG/SAC): Continuous action spaces are unstable with non-stationary financial data. Actor-critic training doubles parameters and requires target policy smoothing that interacts badly with market regime changes.
• Not a 4th "scale" branch: Exposure direction and scale are semantically coupled ("how much long" depends on "going long"). Tavakoli (2018) requires branch independence for the dueling decomposition to remain valid.

Why Not More

Going beyond 9 (e.g., 13 levels at ~15% steps or 21 at 10%) causes:

• Sample efficiency degradation: exploration needs ~2x more samples per doubling of action count
• Diminishing returns: the difference between 75% and 85% exposure on ES futures is negligible
• Q-value resolution: with 101 atoms and v_range=60, each atom covers 1.2 units. More actions = smaller per-action Q-value differences = harder for C51 to resolve

Impact Analysis

C51 Distributional RL

• Branch atom tensor: 9 * 101 = 909 floats (was 5 * 101 = 505). Per batch element.
• Total branch atoms: (9+3+3) * 101 = 1515 (was (5+3+3) * 101 = 1111). ~36% increase.
• Shared memory in C51 kernel: MAX_BRANCH_SIZE * NUM_ATOMS = 9 * 101 = 909 floats = 3636 bytes. Under 4KB — no occupancy impact on H100 (164KB shmem/SM).
• VRAM: negligible increase (~1.5MB for batch=1024).

Q-Gap Conviction Filter

• Flat index changes from 2 to 4 (center of 9-element array).
• All hardcoded flat_idx = 2 must become flat_idx = b0_size / 2.
• Filter logic unchanged: if (Q_best - Q_flat < threshold) → flat.
• With more intermediate levels, the filter becomes more useful: the model can express partial conviction (25% long) without triggering the full-position threshold.

Training Sample Efficiency

• Branching DQN explores 9 exposure actions independently (not 81 combinations).
• Going from 5 → 9 increases exploration requirement by ~80% for the exposure branch.
• Estimated convergence slowdown: 10-20% more epochs (not 80%).
• Count-bonus exploration (UCB) already tracks per-branch visitation and auto-adapts.

Network Architecture

• Exposure advantage head: output dim changes from 5 * num_atoms to 9 * num_atoms.
• Weight count increase: adv_h * 9 * 101 = 116,364 (was adv_h * 5 * 101 = 64,640). +51K params. Negligible.
• No hidden layer changes needed — the shared trunk and value head are unaffected.

Kernel Changes

experience_kernels.cu

exposure_idx_to_fraction() — rewrite from switch-case to formula:

__device__ __forceinline__ float exposure_idx_to_fraction(int idx) {
    /* 9 levels: -1.0, -0.75, -0.50, -0.25, 0.0, +0.25, +0.50, +0.75, +1.0 */
    return -1.0f + (float)idx * 0.25f;
}

Q-gap filter flat_idx — generalize from hardcoded 2:

/* Flat is always the center of the exposure array */
int flat_idx = b0_size / 2;  /* 9/2 = 4 */

epsilon_greedy_kernel.cu

All three kernels: replace hardcoded 5 and 4 with runtime b0_size:

/* Was: exposure = (int)(gpu_random(&rng) * 5.0f); if (exposure >= 5) exposure = 4; */
exposure = (int)(gpu_random(&rng) * (float)b0_size);
if (exposure >= b0_size) exposure = b0_size - 1;

common_device_functions.cuh

#define DQN_NUM_ACTIONS 9          /* was 5 */
#define DQN_TOTAL_ACTIONS 81       /* was 45 (9 × 3 × 3) */
#define BRANCH_EXPOSURE_ACTIONS 9  /* was 5 */

c51_loss_kernel.cu

#define MAX_BRANCH_SIZE 9          /* was 5 — shared memory sizing */

Rust Changes

ml-core/src/common/action.rs

Add 4 new ExposureLevel variants: Short75, Short25, Long25, Long75. Update from_index(), target_exposure(), to_trading_action(). All hardcoded 45 → 81.

ml/src/trainers/dqn/monitoring.rs

Arrays from [_; 5] to [_; 9]. Add exposure names.

ml/src/trainers/dqn/trainer/metrics.rs

DIRECTION_LUT: [f32; 9] = [-1.0, -0.75, -0.5, -0.25, 0.0, 0.25, 0.5, 0.75, 1.0]

ml/src/cuda_pipeline/gpu_experience_collector.rs

branch_sizes: [9, 3, 3] (was [5, 3, 3]).

TOML Configuration

[branching]
branch_0_size = 9    # was 5
branch_1_size = 3    # unchanged
branch_2_size = 3    # unchanged

What Stays the Same

• Order type branch (3 actions): LimitMaker / IoC / Market — unchanged.
• Urgency branch (3 actions): Patient / Normal / Aggressive — unchanged.
• Factored action formula: exposure * 9 + order * 3 + urgency — the stride 3 × 3 = 9 is coincidentally unchanged! The divisor in from_index() stays 9.
• Branching DQN architecture: shared trunk + 3 independent heads. Only the exposure head output dim changes.
• Replay buffer: stores factored action index (u32) — range changes from 0-44 to 0-80, no structural change.
• Hyperopt search space: branch sizes are not searched (fixed). No new dimensions.
• Backtest evaluator: already parameterized via DqnBacktestConfig.branch_0_size.

Migration / Backward Compatibility

• Old checkpoints are incompatible — the exposure head has different weight dimensions. This is expected. No backward compatibility shim needed.
• Emergency safe defaults: keep use_branching: false with 5-action fallback for safety. Or update to 9.
• Tests: all 45 → 81 assertions, all [_; 5] → [_; 9] arrays.

Testing Strategy

1. Unit test: verify exposure_idx_to_fraction() produces correct values for 0-8
2. Roundtrip test: FactoredAction::from_index(i).to_index() == i for 0-80
3. CUDA kernel test: verify PTX compilation with BRANCH_0_SIZE=9
4. Smoke test: 3 epochs with 9-action branching on ES.FUT — loss decreases, Q-values non-zero
5. A/B test on H100: compare hyperopt results with 5 vs 9 exposure levels (same reward function)

Implementation Sequence

1. Core types (action.rs) — add 4 variants, update all methods
2. CUDA header (common_device_functions.cuh) — update macros
3. CUDA kernels — exposure_idx_to_fraction() formula, flat_idx generalization
4. Monitoring arrays — expand from 5 to 9
5. Config / TOML — branch_0_size = 9
6. Tests — update all assertions
7. Compile + smoke test locally
8. H100 validation