From 739dd986bb79e0392b091b3df6a8b378e1182f90 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Mon, 23 Mar 2026 08:45:18 +0100 Subject: [PATCH] =?UTF-8?q?spec:=209-exposure=20action=20space=20design=20?= =?UTF-8?q?=E2=80=94=20expand=20from=205=20to=209=20levels?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 25% step increments: {-100%, -75%, -50%, -25%, 0%, +25%, +50%, +75%, +100%} Total factored actions: 81 (was 45). Branching Q-values: 15 (was 11). Implementation after reward v2 H100 validation. Co-Authored-By: Claude Opus 4.6 (1M context) --- ...26-03-23-9-exposure-action-space-design.md | 171 ++++++++++++++++++ 1 file changed, 171 insertions(+) create mode 100644 docs/superpowers/specs/2026-03-23-9-exposure-action-space-design.md diff --git a/docs/superpowers/specs/2026-03-23-9-exposure-action-space-design.md b/docs/superpowers/specs/2026-03-23-9-exposure-action-space-design.md new file mode 100644 index 000000000..938ca7183 --- /dev/null +++ b/docs/superpowers/specs/2026-03-23-9-exposure-action-space-design.md @@ -0,0 +1,171 @@ +# Expanded Exposure Action Space: 5 → 9 Levels — Design Spec + +## Problem + +The current Branching DQN has 5 exposure levels: {-100%, -50%, 0%, +50%, +100%}. This is too coarse — the model can only choose between "full position" and "half position", with no ability to scale in/out gradually. With 50 features (42 market + 8 OFI microstructure) and C51 distributional RL, the model has rich state information but blunt position sizing tools. + +A trader with conviction in a trend would scale in gradually (25% → 50% → 75%) rather than jumping from flat to 50%. The 5-level discretization forces all-or-nothing decisions that increase transaction costs and drawdown. + +## Solution + +Expand the exposure branch from 5 to 9 discrete levels with 25% steps: + +``` +Index: 0 1 2 3 4 5 6 7 8 +Label: S100 S75 S50 S25 Flat L25 L50 L75 L100 +Value: -1.00 -0.75 -0.50 -0.25 0.00 0.25 0.50 0.75 1.00 +``` + +Total factored action space: 9 × 3 × 3 = 81 (was 5 × 3 × 3 = 45). +Branching DQN outputs: 9 + 3 + 3 = 15 Q-values (was 5 + 3 + 3 = 11). + +### Why 9 Levels + +- **25% step size**: Each step = ~0.5 ES contracts at max_position=2. This is the minimum meaningful increment for single-instrument futures on 1-minute bars. +- **Not 11 (20% steps)**: Produces fractional contract counts (0.4 at max_position=2). Since ES contracts are indivisible, this precision is wasted. +- **Not continuous (DDPG/SAC)**: Continuous action spaces are unstable with non-stationary financial data. Actor-critic training doubles parameters and requires target policy smoothing that interacts badly with market regime changes. +- **Not a 4th "scale" branch**: Exposure direction and scale are semantically coupled ("how much long" depends on "going long"). Tavakoli (2018) requires branch independence for the dueling decomposition to remain valid. + +### Why Not More + +Going beyond 9 (e.g., 13 levels at ~15% steps or 21 at 10%) causes: +- Sample efficiency degradation: exploration needs ~2x more samples per doubling of action count +- Diminishing returns: the difference between 75% and 85% exposure on ES futures is negligible +- Q-value resolution: with 101 atoms and v_range=60, each atom covers 1.2 units. More actions = smaller per-action Q-value differences = harder for C51 to resolve + +## Impact Analysis + +### C51 Distributional RL + +- Branch atom tensor: `9 * 101 = 909` floats (was `5 * 101 = 505`). Per batch element. +- Total branch atoms: `(9+3+3) * 101 = 1515` (was `(5+3+3) * 101 = 1111`). ~36% increase. +- Shared memory in C51 kernel: `MAX_BRANCH_SIZE * NUM_ATOMS = 9 * 101 = 909 floats = 3636 bytes`. Under 4KB — no occupancy impact on H100 (164KB shmem/SM). +- VRAM: negligible increase (~1.5MB for batch=1024). + +### Q-Gap Conviction Filter + +- Flat index changes from 2 to 4 (center of 9-element array). +- All hardcoded `flat_idx = 2` must become `flat_idx = b0_size / 2`. +- Filter logic unchanged: `if (Q_best - Q_flat < threshold) → flat`. +- With more intermediate levels, the filter becomes more useful: the model can express partial conviction (25% long) without triggering the full-position threshold. + +### Training Sample Efficiency + +- Branching DQN explores 9 exposure actions independently (not 81 combinations). +- Going from 5 → 9 increases exploration requirement by ~80% for the exposure branch. +- Estimated convergence slowdown: **10-20% more epochs** (not 80%). +- Count-bonus exploration (UCB) already tracks per-branch visitation and auto-adapts. + +### Network Architecture + +- Exposure advantage head: output dim changes from `5 * num_atoms` to `9 * num_atoms`. +- Weight count increase: `adv_h * 9 * 101 = 116,364` (was `adv_h * 5 * 101 = 64,640`). +51K params. Negligible. +- No hidden layer changes needed — the shared trunk and value head are unaffected. + +## Kernel Changes + +### `experience_kernels.cu` + +**`exposure_idx_to_fraction()`** — rewrite from switch-case to formula: + +```cuda +__device__ __forceinline__ float exposure_idx_to_fraction(int idx) { + /* 9 levels: -1.0, -0.75, -0.50, -0.25, 0.0, +0.25, +0.50, +0.75, +1.0 */ + return -1.0f + (float)idx * 0.25f; +} +``` + +**Q-gap filter flat_idx** — generalize from hardcoded 2: + +```cuda +/* Flat is always the center of the exposure array */ +int flat_idx = b0_size / 2; /* 9/2 = 4 */ +``` + +### `epsilon_greedy_kernel.cu` + +All three kernels: replace hardcoded `5` and `4` with runtime `b0_size`: + +```cuda +/* Was: exposure = (int)(gpu_random(&rng) * 5.0f); if (exposure >= 5) exposure = 4; */ +exposure = (int)(gpu_random(&rng) * (float)b0_size); +if (exposure >= b0_size) exposure = b0_size - 1; +``` + +### `common_device_functions.cuh` + +```cuda +#define DQN_NUM_ACTIONS 9 /* was 5 */ +#define DQN_TOTAL_ACTIONS 81 /* was 45 (9 × 3 × 3) */ +#define BRANCH_EXPOSURE_ACTIONS 9 /* was 5 */ +``` + +### `c51_loss_kernel.cu` + +```cuda +#define MAX_BRANCH_SIZE 9 /* was 5 — shared memory sizing */ +``` + +## Rust Changes + +### `ml-core/src/common/action.rs` + +Add 4 new `ExposureLevel` variants: `Short75`, `Short25`, `Long25`, `Long75`. +Update `from_index()`, `target_exposure()`, `to_trading_action()`. +All hardcoded `45` → `81`. + +### `ml/src/trainers/dqn/monitoring.rs` + +Arrays from `[_; 5]` to `[_; 9]`. Add exposure names. + +### `ml/src/trainers/dqn/trainer/metrics.rs` + +`DIRECTION_LUT: [f32; 9] = [-1.0, -0.75, -0.5, -0.25, 0.0, 0.25, 0.5, 0.75, 1.0]` + +### `ml/src/cuda_pipeline/gpu_experience_collector.rs` + +`branch_sizes: [9, 3, 3]` (was `[5, 3, 3]`). + +## TOML Configuration + +```toml +[branching] +branch_0_size = 9 # was 5 +branch_1_size = 3 # unchanged +branch_2_size = 3 # unchanged +``` + +## What Stays the Same + +- **Order type branch (3 actions)**: LimitMaker / IoC / Market — unchanged. +- **Urgency branch (3 actions)**: Patient / Normal / Aggressive — unchanged. +- **Factored action formula**: `exposure * 9 + order * 3 + urgency` — the stride `3 × 3 = 9` is coincidentally unchanged! The divisor in `from_index()` stays 9. +- **Branching DQN architecture**: shared trunk + 3 independent heads. Only the exposure head output dim changes. +- **Replay buffer**: stores factored action index (u32) — range changes from 0-44 to 0-80, no structural change. +- **Hyperopt search space**: branch sizes are not searched (fixed). No new dimensions. +- **Backtest evaluator**: already parameterized via `DqnBacktestConfig.branch_0_size`. + +## Migration / Backward Compatibility + +- **Old checkpoints are incompatible** — the exposure head has different weight dimensions. This is expected. No backward compatibility shim needed. +- **Emergency safe defaults**: keep `use_branching: false` with 5-action fallback for safety. Or update to 9. +- **Tests**: all `45` → `81` assertions, all `[_; 5]` → `[_; 9]` arrays. + +## Testing Strategy + +1. **Unit test**: verify `exposure_idx_to_fraction()` produces correct values for 0-8 +2. **Roundtrip test**: `FactoredAction::from_index(i).to_index() == i` for 0-80 +3. **CUDA kernel test**: verify PTX compilation with `BRANCH_0_SIZE=9` +4. **Smoke test**: 3 epochs with 9-action branching on ES.FUT — loss decreases, Q-values non-zero +5. **A/B test on H100**: compare hyperopt results with 5 vs 9 exposure levels (same reward function) + +## Implementation Sequence + +1. Core types (`action.rs`) — add 4 variants, update all methods +2. CUDA header (`common_device_functions.cuh`) — update macros +3. CUDA kernels — `exposure_idx_to_fraction()` formula, flat_idx generalization +4. Monitoring arrays — expand from 5 to 9 +5. Config / TOML — `branch_0_size = 9` +6. Tests — update all assertions +7. Compile + smoke test locally +8. H100 validation