# FoxhuntQ-Δ Phase E: Execution-Layer RL Policy — Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Build a learned execution-layer DQN policy that takes Mamba+stacker alpha output + current LOB state and decides order placement (passive L1/L2 vs market vs wait), reducing effective per-trade cost from "kills the strategy at half-tick" to "deployable at quarter-tick or better." **Architecture:** GPU-pure DQN (Rainbow + Munchausen target + ε-greedy with kill criteria) over a fill-simulating environment, with state = [alpha_logit, alpha_confidence, spread_bps, L1_imbalance, current_position, time_in_position, recent_OFI_sum, recent_mid_drift, time_since_trade, book_event_rate] and 9-12 discrete actions. Position size is decoupled: fractional Kelly (consuming existing ISV slot `KELLY_F_SMOOTH_INDEX=280` × new attenuation slot) determines target contracts; the execution policy only chooses placement. ISV-driven everywhere a controller anchor exists; safety caps and kill criteria hardcoded by design. **Tech Stack:** - Rust 1.85+, Edition 2021 - CUDA 12.4 via cudarc 0.19 - Existing infra: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` (~25K LOC GPU-pure DQN), `crates/ml/src/trainers/dqn/`, `crates/ml/src/cuda_pipeline/sp{4-22}_isv_slots.rs` (ISV registry), `crates/ml-alpha` (Mamba + stacker) - New code: ~3000 LOC in `crates/ml/src/env/`, `crates/ml/src/cuda_pipeline/phase_e_*`, `crates/ml/examples/phase_e_*` **Anchor commits:** - Alpha foundation: `db874b184` (Phase 1c snapshot pipeline) and `4cf9499b5` (Phase 1d.2 multi-minute validation) - Stacker: `ab4b7c64c` - Backtest infra: `20c361a30` --- ## Design decisions (LOCKED) These are the locked-in answers from three rounds of focused research. Tasks below assume these decisions; do not relitigate inline. | Question | Decision | Source | |---|---|---| | Fill simulation | Medium tier: Poisson regression `λ_l(spread, imbalance, ofi, time_since_trade)` fitted once from 5.2M trade tape | First memo | | Reward shaping | Terminal-only at trade close (PnL − fees − implementation shortfall); n-step credit assignment via existing trainer | First memo + `pearl_event_driven_reward_density_alignment` | | Alpha trust state | Raw `alpha_logit` + `alpha_confidence = |p − 0.5|` + `spread_bps` + `L1_imbalance`. NO trailing alpha-correctness EMA (reward leak risk) | First memo + `pearl_snapshot_alpha_is_regime_conditional` | | State window | Current snapshot + 2 short-horizon scalars (5-snap OFI sum, 5-snap mid drift). NO trajectory windows — Mamba already carries history | First memo | | Position sizing | Hybrid decoupled: closed-form fractional Kelly (consuming existing `KELLY_F_SMOOTH_INDEX=280`) × NEW attenuation slot; execution RL chooses placement only | First memo + `pearl_kelly_cap_signal_driven_floors` | | Trainer (primary) | DQN with existing Rainbow stack (n-step + dueling + C51 + PER) + add Munchausen target term | Second memo | | Trainer (secondary, optional) | PPO on H=600 truncated as a sanity check ONLY if DQN's kill criteria fire | Second memo | | Exploration | ε-greedy with linear schedule 1.0→0.05 over 500K steps + week-1 kill criteria; escalate to NoisyNet if criteria fire (4-6 days due to dead scaffolding); RND if NoisyNet also fails | Third memo | | ISV consumption (existing slots) | n_step ← slot 517, γ ← slots 43-46, ε ← slot 41, Kelly_smooth ← slot 280, reward_caps ← slots 452-453 | ISV memo | | ISV new slots | Block 539..550 reserved for Phase E (12 slots, contiguous, single fingerprint bump) | ISV memo | | Hardcoded by design | Kelly contract cap = 10 (Category-1 safety), kill criteria thresholds (circuit-breakers, NOT controllers) | ISV memo | --- ## File Structure | File | Responsibility | |---|---| | `crates/ml/src/cuda_pipeline/phase_e_isv_slots.rs` (new) | Define indices 539..550, accessors, sentinel-bootstrap reads, layout fingerprint asserts | | `crates/ml/src/trainers/dqn/state_reset_registry.rs` (modify) | Add 12 Phase E entries with dispatch arms per `feedback_registry_entries_need_dispatch_arms` | | `crates/ml/src/env/execution_env.rs` (new) | DQN environment: state assembly, action execution, reward computation | | `crates/ml/src/env/fill_model.rs` (new) | Poisson-regression fill simulator | | `crates/ml/src/env/action_space.rs` (new) | 9-action discrete enum + (level, aggressiveness, size) decoding | | `crates/ml/src/cuda_pipeline/munchausen_target_kernel.cu` (new) | One-line addition to standard DQN target: adds `τ·log_π(a|s)` clipped term | | `crates/ml/src/cuda_pipeline/stacker_threshold_controller.cu` (new) | Engagement-rate self-correction kernel: drives slot 543 from observed-rate vs target | | `crates/ml/src/cuda_pipeline/phase_e_kill_criteria.cu` (new) | Computes Q-spread, action entropy, return-vs-random, early-Q-movement into diagnostic slots | | `crates/ml/examples/phase_e_smoke.rs` (new) | Truncated H=600 DQN smoke for week-1 kill-criteria gate | | `crates/ml/examples/phase_e_full.rs` (new) | Full H=6000 DQN training + backtest | | `crates/ml/examples/phase_e_compose_backtest.rs` (new) | Alpha-policy + execution-policy composition backtest with realistic costs | | `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` (modify) | Bump `ISV_TOTAL_DIM`; consume Phase E slots; wire kill criteria reads | | `services/trading/src/phase_e_shadow.rs` (new) | Shadow-mode wiring: read live ticks, emit orders to log, compare to backtest expectation | --- ## Pre-Flight ### Task 0: Verify we're on the right branch and worktree **Files:** - Read-only: `.git/HEAD`, `git status` - [ ] **Step 1: Confirm worktree and branch** ```bash cd /home/jgrusewski/Work/foxhunt/.worktrees/sp19-20-wr-first git rev-parse --abbrev-ref HEAD ``` Expected: `sp20-aux-h-fixed` - [ ] **Step 2: Confirm anchor commits exist** ```bash git log --oneline | grep -E '4cf9499b5|ab4b7c64c|20c361a30' | head -3 ``` Expected: three lines matching the three anchor commits. - [ ] **Step 3: Confirm fxcache is present and ≥1GB** ```bash ls -lh /home/jgrusewski/Work/foxhunt/test_data/feature-cache/*.fxcache | head -1 ``` Expected: a file ≥ 1G (the snapshot fxcache from `4cf9499b5`). --- ## Milestone E.0 — Foundation (~Week 1, Tasks 1-8) **Goal:** Reserve ISV block, build environment scaffold + fill simulator + kill-criteria instrumentation, compute random-uniform baseline. **Decisive gate at end of week 1:** All four kill criteria are computable on the ε-greedy DQN-untrained baseline. Numerical values printed and saved as ground truth. If we can't even compute them, Phase E.1 can't start. --- ### Task 1: Reserve ISV slot block 539..550 **Files:** - Create: `crates/ml/src/cuda_pipeline/phase_e_isv_slots.rs` - Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` (around the `ISV_TOTAL_DIM` constant, ~line 2860) - [ ] **Step 1: Find current `ISV_TOTAL_DIM`** ```bash grep -n 'pub const ISV_TOTAL_DIM' crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs ``` Expected: a line `pub const ISV_TOTAL_DIM: usize = N;` where N is the current dim. Note `N`. - [ ] **Step 2: Write the Phase E slot file** Create `crates/ml/src/cuda_pipeline/phase_e_isv_slots.rs`: ```rust //! Phase E (execution-layer RL) ISV slot block. //! //! Reserves indices 539..550 (12 slots, contiguous). All Phase E controller //! anchors and diagnostic EMAs live here. Consumers read via the standard //! sentinel-bootstrap pattern: `if isv[idx] <= sentinel + EPS { fallback }`. //! //! Slot layout: //! //! | Index | Name | Role | Update cadence | Producer kernel | //! |-------|------------------------------|--------------|-----------------|------------------------------------------| //! | 539 | Q_SPREAD_EMA_INDEX | Diagnostic | per-rollout-step| phase_e_kill_criteria.cu | //! | 540 | ACTION_ENTROPY_EMA_INDEX | Diagnostic | per-rollout-step| phase_e_kill_criteria.cu | //! | 541 | RETURN_VS_RANDOM_EMA_INDEX | Diagnostic | per-trade-close | phase_e_kill_criteria.cu | //! | 542 | EARLY_Q_MOVEMENT_EMA_INDEX | Diagnostic | per-epoch | phase_e_kill_criteria.cu | //! | 543 | STACKER_THRESHOLD_INDEX | Controller | per-rollout-step| stacker_threshold_controller.cu | //! | 544 | TRADE_RATE_TARGET_INDEX | Anchor | static | host init | //! | 545 | TRADE_RATE_OBSERVED_EMA | Diagnostic | per-rollout-step| stacker_threshold_controller.cu | //! | 546 | STACKER_KELLY_ATTENUATION | Controller | per-trade-close | stacker_threshold_controller.cu | //! | 547 | RANDOM_BASELINE_REWARD_MEAN | Anchor | per-fold | host computes once per fold | //! | 548 | RANDOM_BASELINE_REWARD_STD | Anchor | per-fold | host computes once per fold | //! | 549 | (reserved) | Spare | - | - | //! | 550 | (reserved) | Spare | - | - | //! //! Reservation discipline (per ISV memo): slots 549..550 are intentionally //! unused at first-commit to absorb growth without bumping //! `ISV_TOTAL_DIM` again. Add new Phase E slots HERE first, not by //! extending past 550 ad-hoc. pub const PHASE_E_ISV_BLOCK_LO: usize = 539; pub const PHASE_E_ISV_BLOCK_HI: usize = 550; // inclusive upper // Diagnostics (read-only for circuit breakers; not controllers) pub const Q_SPREAD_EMA_INDEX: usize = 539; pub const ACTION_ENTROPY_EMA_INDEX: usize = 540; pub const RETURN_VS_RANDOM_EMA_INDEX: usize = 541; pub const EARLY_Q_MOVEMENT_EMA_INDEX: usize = 542; // Stacker-threshold controller (engagement-rate self-correction) pub const STACKER_THRESHOLD_INDEX: usize = 543; pub const TRADE_RATE_TARGET_INDEX: usize = 544; pub const TRADE_RATE_OBSERVED_EMA_INDEX: usize = 545; pub const STACKER_KELLY_ATTENUATION_INDEX: usize = 546; // Random-uniform baseline (computed once per fold; used by kill criterion) pub const RANDOM_BASELINE_REWARD_MEAN_INDEX: usize = 547; pub const RANDOM_BASELINE_REWARD_STD_INDEX: usize = 548; // Sentinels: zero by convention. Fallbacks use the first-observation // bootstrap pattern (pearl_first_observation_bootstrap). pub const PHASE_E_SENTINEL: f32 = 0.0; pub const PHASE_E_SENTINEL_EPS: f32 = 1e-9; #[cfg(test)] mod tests { use super::*; #[test] fn test_block_bounds_consistent() { assert!(PHASE_E_ISV_BLOCK_HI >= PHASE_E_ISV_BLOCK_LO); let block_size = PHASE_E_ISV_BLOCK_HI - PHASE_E_ISV_BLOCK_LO + 1; assert_eq!(block_size, 12, "Phase E reserved 12 contiguous slots"); } #[test] fn test_all_indices_inside_block() { let indices = [ Q_SPREAD_EMA_INDEX, ACTION_ENTROPY_EMA_INDEX, RETURN_VS_RANDOM_EMA_INDEX, EARLY_Q_MOVEMENT_EMA_INDEX, STACKER_THRESHOLD_INDEX, TRADE_RATE_TARGET_INDEX, TRADE_RATE_OBSERVED_EMA_INDEX, STACKER_KELLY_ATTENUATION_INDEX, RANDOM_BASELINE_REWARD_MEAN_INDEX, RANDOM_BASELINE_REWARD_STD_INDEX, ]; for &i in &indices { assert!(i >= PHASE_E_ISV_BLOCK_LO && i <= PHASE_E_ISV_BLOCK_HI, "slot {i} out of Phase E block"); } } #[test] fn test_total_dim_covers_block() { // The parent ISV_TOTAL_DIM must be at least PHASE_E_ISV_BLOCK_HI + 1. use crate::cuda_pipeline::gpu_dqn_trainer::ISV_TOTAL_DIM; assert!(ISV_TOTAL_DIM > PHASE_E_ISV_BLOCK_HI, "ISV_TOTAL_DIM ({ISV_TOTAL_DIM}) must exceed PHASE_E_ISV_BLOCK_HI ({PHASE_E_ISV_BLOCK_HI})"); } } ``` - [ ] **Step 3: Bump `ISV_TOTAL_DIM` and add module declaration** Open `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs`. Find the existing `ISV_TOTAL_DIM` constant (use the line number from Step 1). Set it to `551` (so indices 0..550 fit). Also find `mod sp22_isv_slots;` (or the highest SP slot module declaration) and add `pub mod phase_e_isv_slots;` right after. - [ ] **Step 4: Compile + test** ```bash SQLX_OFFLINE=true cargo test -p ml --lib phase_e_isv_slots -- --nocapture ``` Expected: 3 passed. - [ ] **Step 5: Commit** ```bash git add crates/ml/src/cuda_pipeline/phase_e_isv_slots.rs crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs git commit -m "$(cat <<'EOF' feat(phase-e): reserve ISV slot block 539..550 for execution-layer RL 12 contiguous slots reserved for Phase E controllers + diagnostics, per the ISV research memo recommendation: - 539-542: kill-criteria diagnostics (Q-spread, entropy, return-vs-random, early-Q-movement EMAs) - 543-546: stacker-threshold controller (threshold, target, observed-rate EMA, Kelly attenuation) - 547-548: random-uniform baseline (mean, std) — computed once per fold - 549-550: reserved spare (absorb growth without bumping ISV_TOTAL_DIM) ISV_TOTAL_DIM bumped to 551. All indices validated in unit tests per the existing slot-registry pattern (sp{4,5,11,13,14,15,21,22}_isv_slots.rs). EOF )" ``` --- ### Task 2: Wire Phase E slots into state-reset registry **Files:** - Modify: `crates/ml/src/trainers/dqn/state_reset_registry.rs` Per `feedback_registry_entries_need_dispatch_arms`: every new ISV slot needs a registry entry AND a matching dispatch arm in `reset_named_state` in the same commit. If not, smoke crashes at fold 0 setup with "unknown name" error. - [ ] **Step 1: Find the existing registry section** ```bash grep -n 'pub fn build_registry' crates/ml/src/trainers/dqn/state_reset_registry.rs grep -n 'pub fn reset_named_state' crates/ml/src/trainers/dqn/state_reset_registry.rs ``` Note the line numbers. - [ ] **Step 2: Add Phase E registry entries** Add 10 entries to the registry (one per non-spare Phase E slot). Each entry follows the existing pattern. Find an existing SP block (e.g., SP14 entries near `crates/ml/src/cuda_pipeline/sp14_isv_slots.rs` consumers) and add after it: ```rust // ── Phase E (execution-layer RL) — slot block 539..550 ──────────── // Per phase_e_isv_slots.rs. Lifecycle category rationale: // - Diagnostics (539-542) are SoftReset { decay_bars: 1000 } — slow EMAs // that should fade after a fold to avoid contaminating across regimes. // - Controllers (543, 545, 546) are FoldReset — every fold should learn // its own threshold and attenuation from scratch. // - Anchors (544, 547, 548) are TrainingPersist — set once per fold by // the host, read all fold long. entries.push(StateResetEntry { name: "phase_e.q_spread_ema", index: phase_e_isv_slots::Q_SPREAD_EMA_INDEX, category: ResetCategory::SoftReset { decay_bars: 1000 }, source: "phase_e_kill_criteria.cu", }); entries.push(StateResetEntry { name: "phase_e.action_entropy_ema", index: phase_e_isv_slots::ACTION_ENTROPY_EMA_INDEX, category: ResetCategory::SoftReset { decay_bars: 1000 }, source: "phase_e_kill_criteria.cu", }); entries.push(StateResetEntry { name: "phase_e.return_vs_random_ema", index: phase_e_isv_slots::RETURN_VS_RANDOM_EMA_INDEX, category: ResetCategory::SoftReset { decay_bars: 1000 }, source: "phase_e_kill_criteria.cu", }); entries.push(StateResetEntry { name: "phase_e.early_q_movement_ema", index: phase_e_isv_slots::EARLY_Q_MOVEMENT_EMA_INDEX, category: ResetCategory::SoftReset { decay_bars: 1000 }, source: "phase_e_kill_criteria.cu", }); entries.push(StateResetEntry { name: "phase_e.stacker_threshold", index: phase_e_isv_slots::STACKER_THRESHOLD_INDEX, category: ResetCategory::FoldReset, source: "stacker_threshold_controller.cu", }); entries.push(StateResetEntry { name: "phase_e.trade_rate_target", index: phase_e_isv_slots::TRADE_RATE_TARGET_INDEX, category: ResetCategory::TrainingPersist, source: "host (Phase E config)", }); entries.push(StateResetEntry { name: "phase_e.trade_rate_observed_ema", index: phase_e_isv_slots::TRADE_RATE_OBSERVED_EMA_INDEX, category: ResetCategory::FoldReset, source: "stacker_threshold_controller.cu", }); entries.push(StateResetEntry { name: "phase_e.stacker_kelly_attenuation", index: phase_e_isv_slots::STACKER_KELLY_ATTENUATION_INDEX, category: ResetCategory::FoldReset, source: "stacker_threshold_controller.cu", }); entries.push(StateResetEntry { name: "phase_e.random_baseline_reward_mean", index: phase_e_isv_slots::RANDOM_BASELINE_REWARD_MEAN_INDEX, category: ResetCategory::TrainingPersist, source: "host (Phase E.0 baseline computation)", }); entries.push(StateResetEntry { name: "phase_e.random_baseline_reward_std", index: phase_e_isv_slots::RANDOM_BASELINE_REWARD_STD_INDEX, category: ResetCategory::TrainingPersist, source: "host (Phase E.0 baseline computation)", }); ``` - [ ] **Step 3: Add dispatch arms in `reset_named_state`** Find the `match` block in `reset_named_state`. Add 10 arms (one per entry above). The arms write the sentinel (0.0) and any Wiener state. Pattern from existing SP arms: ```rust "phase_e.q_spread_ema" => { isv[phase_e_isv_slots::Q_SPREAD_EMA_INDEX] = phase_e_isv_slots::PHASE_E_SENTINEL; // Wiener state (3 floats per slot in wiener_state_buf) — reset to 0 let wiener_off = phase_e_isv_slots::Q_SPREAD_EMA_INDEX * 3; wiener_state_buf[wiener_off] = 0.0; // sample_var wiener_state_buf[wiener_off + 1] = 0.0; // diff_var wiener_state_buf[wiener_off + 2] = 0.0; // x_lag } // ... same pattern for the 9 other slots ``` - [ ] **Step 4: Run the registry coverage test** ```bash SQLX_OFFLINE=true cargo test -p ml --lib state_reset_registry -- --nocapture ``` Expected: existing tests pass, plus the new "every entry has a dispatch arm" coverage test passes. - [ ] **Step 5: Commit** ```bash git add crates/ml/src/trainers/dqn/state_reset_registry.rs git commit -m "feat(phase-e): wire 10 Phase E ISV slots into state_reset_registry with dispatch arms per feedback_registry_entries_need_dispatch_arms" ``` --- ### Task 3: Build the action space enum **Files:** - Create: `crates/ml/src/env/mod.rs` (new module) - Create: `crates/ml/src/env/action_space.rs` - Modify: `crates/ml/src/lib.rs` (add `pub mod env;`) - [ ] **Step 1: Define the 9-action space** Create `crates/ml/src/env/action_space.rs`: ```rust //! Phase E execution-policy action space. //! //! 9 discrete actions per the design memo. Size is decoupled (handled by //! the Kelly layer); the action decides ONLY placement and side. WAIT //! emits no order; the others emit one limit/market order at the //! specified level with sign = side. //! //! Mapping rationale: 3 sides × 3 placements = 9, plus 1 wait = 10? No — //! `WAIT` doesn't have a placement, so: //! //! 0 WAIT //! 1 BUY @ market (aggressive, crosses spread) //! 2 BUY @ L1 bid (passive, joins quote) //! 3 BUY @ L2 bid (passive, behind quote) //! 4 SELL @ market (aggressive, crosses spread) //! 5 SELL @ L1 ask (passive, joins quote) //! 6 SELL @ L2 ask (passive, behind quote) //! 7 FLAT @ market (close current position aggressively) //! 8 FLAT @ L1 (close current position passively at L1) //! //! Action gating: actions 1-3 are only legal when not already long; //! actions 4-6 are only legal when not already short; 7-8 are only legal //! when holding a position. Illegal actions degrade to WAIT; the gating //! is enforced in the environment, not the network. #[derive(Debug, Clone, Copy, PartialEq, Eq)] #[repr(u8)] pub enum ExecAction { Wait = 0, BuyMarket = 1, BuyL1 = 2, BuyL2 = 3, SellMarket = 4, SellL1 = 5, SellL2 = 6, FlatMarket = 7, FlatL1 = 8, } pub const N_ACTIONS: usize = 9; #[derive(Debug, Clone, Copy)] pub struct DecodedAction { pub side: Side, pub placement: Placement, pub closing: bool, } #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum Side { None, Buy, Sell } #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum Placement { None, Market, L1, L2 } impl ExecAction { pub fn from_u8(i: u8) -> Option { match i { 0 => Some(Self::Wait), 1 => Some(Self::BuyMarket), 2 => Some(Self::BuyL1), 3 => Some(Self::BuyL2), 4 => Some(Self::SellMarket), 5 => Some(Self::SellL1), 6 => Some(Self::SellL2), 7 => Some(Self::FlatMarket), 8 => Some(Self::FlatL1), _ => None, } } pub fn decode(self) -> DecodedAction { match self { Self::Wait => DecodedAction { side: Side::None, placement: Placement::None, closing: false }, Self::BuyMarket => DecodedAction { side: Side::Buy, placement: Placement::Market, closing: false }, Self::BuyL1 => DecodedAction { side: Side::Buy, placement: Placement::L1, closing: false }, Self::BuyL2 => DecodedAction { side: Side::Buy, placement: Placement::L2, closing: false }, Self::SellMarket => DecodedAction { side: Side::Sell, placement: Placement::Market, closing: false }, Self::SellL1 => DecodedAction { side: Side::Sell, placement: Placement::L1, closing: false }, Self::SellL2 => DecodedAction { side: Side::Sell, placement: Placement::L2, closing: false }, Self::FlatMarket => DecodedAction { side: Side::None, placement: Placement::Market, closing: true }, Self::FlatL1 => DecodedAction { side: Side::None, placement: Placement::L1, closing: true }, } } /// Returns true if the action is legal given current position. /// `position ∈ {-1, 0, +1}` (sign only; size is decoupled). pub fn is_legal(self, position: i32) -> bool { match self { Self::Wait => true, Self::BuyMarket | Self::BuyL1 | Self::BuyL2 => position <= 0, Self::SellMarket | Self::SellL1 | Self::SellL2 => position >= 0, Self::FlatMarket | Self::FlatL1 => position != 0, } } } #[cfg(test)] mod tests { use super::*; #[test] fn test_n_actions_matches_enum() { assert_eq!(N_ACTIONS, 9); for i in 0..N_ACTIONS as u8 { assert!(ExecAction::from_u8(i).is_some(), "action {i} missing"); } assert!(ExecAction::from_u8(9).is_none()); } #[test] fn test_legality_gating() { // Flat → can buy or sell, can't close assert!(ExecAction::BuyMarket.is_legal(0)); assert!(ExecAction::SellMarket.is_legal(0)); assert!(!ExecAction::FlatMarket.is_legal(0)); // Long → can sell or close, can't buy more assert!(!ExecAction::BuyMarket.is_legal(1)); assert!(ExecAction::SellMarket.is_legal(1)); assert!(ExecAction::FlatMarket.is_legal(1)); // Short → can buy or close, can't sell more assert!(ExecAction::BuyMarket.is_legal(-1)); assert!(!ExecAction::SellMarket.is_legal(-1)); assert!(ExecAction::FlatMarket.is_legal(-1)); } } ``` - [ ] **Step 2: Wire up mod.rs** Create `crates/ml/src/env/mod.rs`: ```rust //! Phase E execution-policy environment. //! //! Snapshot-stream replay with a Poisson-regression fill simulator and //! discrete action space. Trains the existing GPU DQN trainer on order- //! placement decisions for a given alpha signal + LOB state. pub mod action_space; pub mod fill_model; // lands in Task 4 pub mod execution_env; // lands in Task 6 pub use action_space::{ExecAction, DecodedAction, Side, Placement, N_ACTIONS}; ``` - [ ] **Step 3: Register module in lib.rs** In `crates/ml/src/lib.rs`, add `pub mod env;` near the other top-level mods. - [ ] **Step 4: Run tests** ```bash SQLX_OFFLINE=true cargo test -p ml --lib env::action_space -- --nocapture ``` Expected: 2 tests pass. - [ ] **Step 5: Commit** ```bash git add crates/ml/src/env/ crates/ml/src/lib.rs git commit -m "feat(phase-e): 9-action discrete execution action space + legality gating" ``` --- ### Task 4: Build the fill model (Poisson regression scaffolding) **Files:** - Create: `crates/ml/src/env/fill_model.rs` **Important context:** This task creates the SCAFFOLDING. The actual coefficients get fitted in Task 5 from the historical trade tape. The scaffolding defines the data structure, sampling logic, and config. - [ ] **Step 1: Write the failing test for the scaffold** In `crates/ml/src/env/fill_model.rs`: ```rust //! Poisson-regression fill simulator (medium tier per the design memo). //! //! For each level `l ∈ {L1, L2, L3}` and side, we fit a Poisson regression: //! //! λ_l(features) = exp(β_l · features) //! //! where features = [1, spread_bps, L1_imbalance, OFI_sum_5, log(time_since_trade + 1)]. //! At backtest time, the per-snapshot fill *probability* (Bernoulli) for a //! posted limit order at level `l` is `1 − exp(−λ_l(features))` (the standard //! Poisson-to-Bernoulli conversion for a single time bucket). For market //! orders, fill is always immediate; the *price* is the L1 quote on the //! opposite side (no slippage modeled at medium tier). //! //! The 5 coefficients per (level, side) = 6 distributions × 5 = 30 floats //! total. Tiny. Fits in registers. #[derive(Debug, Clone, Copy)] pub struct FillCoeffs { /// β_0 + β_1·spread + β_2·imbalance + β_3·ofi_5 + β_4·log(tau+1) pub beta: [f32; 5], } #[derive(Debug, Clone)] pub struct FillModel { /// 6 distributions: (level=L1/L2/L3) × (side=BID/ASK). Buy passive uses /// BID coefficients (we want to be filled by an aggressor crossing /// down); sell passive uses ASK coefficients. pub bid_coeffs: [FillCoeffs; 3], pub ask_coeffs: [FillCoeffs; 3], } #[derive(Debug, Clone, Copy)] pub struct FillFeatures { pub spread_bps: f32, pub l1_imbalance: f32, pub ofi_sum_5: f32, pub time_since_trade_s: f32, } impl FillModel { /// Untrained-skeleton constructor (coefficients all zero → λ=1 → ~63% /// fill, which is reasonable as a sanity default before Task 5 fits /// real coefficients). pub fn skeleton() -> Self { let zero = FillCoeffs { beta: [0.0; 5] }; Self { bid_coeffs: [zero; 3], ask_coeffs: [zero; 3], } } /// Fill rate λ for a (side, level) at the given features. pub fn lambda(&self, side: crate::env::action_space::Side, level: usize, feat: &FillFeatures) -> f32 { let coeffs = match side { crate::env::action_space::Side::Buy => &self.bid_coeffs, crate::env::action_space::Side::Sell => &self.ask_coeffs, crate::env::action_space::Side::None => return 0.0, // closing actions handled separately }; let c = &coeffs[level.min(2)].beta; let x = [1.0, feat.spread_bps, feat.l1_imbalance, feat.ofi_sum_5, (feat.time_since_trade_s + 1.0).ln()]; let linear = c.iter().zip(x.iter()).map(|(a, b)| a * b).sum::(); // λ is a rate, must be non-negative. Use exp (canonical Poisson link). linear.exp().min(50.0) // cap to avoid numerical blow-up; λ=50 is 1-second saturation } /// Per-snapshot Bernoulli fill probability for a posted order at level `level`. pub fn fill_prob(&self, side: crate::env::action_space::Side, level: usize, feat: &FillFeatures) -> f32 { let lam = self.lambda(side, level, feat); 1.0 - (-lam).exp() } } #[cfg(test)] mod tests { use super::*; use crate::env::action_space::Side; #[test] fn test_skeleton_has_uniform_fill_prob() { // β=0 → λ=exp(0)=1 → fill_prob = 1 − exp(−1) ≈ 0.632 let m = FillModel::skeleton(); let feat = FillFeatures { spread_bps: 0.5, l1_imbalance: 0.5, ofi_sum_5: 0.0, time_since_trade_s: 0.5 }; let p = m.fill_prob(Side::Buy, 0, &feat); assert!((p - 0.632).abs() < 0.01, "skeleton λ=1 → p≈0.632, got {p}"); } #[test] fn test_fill_prob_bounded() { // Even with huge features and zero coefficients, prob is in [0, 1]. let m = FillModel::skeleton(); let feat = FillFeatures { spread_bps: 1000.0, l1_imbalance: 1.0, ofi_sum_5: 100.0, time_since_trade_s: 1000.0 }; let p = m.fill_prob(Side::Buy, 0, &feat); assert!((0.0..=1.0).contains(&p)); } } ``` - [ ] **Step 2: Run tests** ```bash SQLX_OFFLINE=true cargo test -p ml --lib env::fill_model -- --nocapture ``` Expected: 2 passed. - [ ] **Step 3: Commit** ```bash git add crates/ml/src/env/fill_model.rs git commit -m "feat(phase-e): Poisson regression fill model scaffold (coefficients fitted in Task 5)" ``` --- ### Task 5: Fit the fill model from historical trade tape **Files:** - Modify: `crates/ml/src/env/fill_model.rs` (add fitting routine) - Create: `crates/ml/examples/phase_e_fit_fill_model.rs` - [ ] **Step 1: Append fitting routine to `fill_model.rs`** Add this section: ```rust /// Fit FillModel coefficients from historical (snapshot, trade) pairs. /// /// For each (side, level) we treat "was there a trade at price ≤ posted /// price within the next `dt_seconds` window" as a Bernoulli observation, /// then transform to Poisson via the canonical link. With ~5M trades over /// ~16 days, even per-level sample counts are 6 figures — plenty for /// 5-coefficient Poisson regression. IRLS or gradient-descent both work. /// /// We use simple gradient descent on the canonical Poisson NLL for /// implementation simplicity (no external linalg dependency required). pub fn fit_poisson( features: &[FillFeatures], observed_fills: &[bool], max_iters: usize, lr: f32, ) -> Result { if features.len() != observed_fills.len() || features.is_empty() { return Err("fit_poisson: length mismatch or empty"); } let mut beta = [0.0_f32; 5]; let n = features.len() as f32; for _ in 0..max_iters { let mut grad = [0.0_f32; 5]; for (feat, &y) in features.iter().zip(observed_fills) { let x = [1.0, feat.spread_bps, feat.l1_imbalance, feat.ofi_sum_5, (feat.time_since_trade_s + 1.0).ln()]; let linear = beta.iter().zip(x.iter()).map(|(a, b)| a * b).sum::(); let mu = linear.exp().min(50.0); // Poisson mean let target = if y { 1.0 } else { 0.0 }; // Gradient of -log L per sample: (mu - target) * x let resid = mu - target; for k in 0..5 { grad[k] += resid * x[k]; } } for k in 0..5 { beta[k] -= lr * grad[k] / n; } } Ok(FillCoeffs { beta }) } ``` Add a test: ```rust #[test] fn test_fit_recovers_baseline_when_features_uninformative() { // 1000 random fill events at uniform p=0.4 — fitting should give // β_0 ≈ ln(λ) where 1−exp(−λ)=0.4 → λ≈0.51 → β_0≈−0.67. let n = 1000; let features: Vec = (0..n).map(|_| FillFeatures { spread_bps: 0.0, l1_imbalance: 0.0, ofi_sum_5: 0.0, time_since_trade_s: 0.0, }).collect(); // Deterministic 40% fill rate let observed: Vec = (0..n).map(|i| (i * 401 % 1000) < 400).collect(); let coeffs = fit_poisson(&features, &observed, 1000, 1e-2).expect("fit"); let lam = coeffs.beta[0].exp(); let p = 1.0 - (-lam).exp(); assert!((p - 0.4).abs() < 0.05, "recovered fill rate {p} should be near 0.4"); } ``` - [ ] **Step 2: Run the test** ```bash SQLX_OFFLINE=true cargo test -p ml --lib env::fill_model::tests::test_fit_recovers -- --nocapture ``` Expected: passes. - [ ] **Step 3: Author the calibration example** Create `crates/ml/examples/phase_e_fit_fill_model.rs`: ```rust //! Phase E.0 — fit FillModel coefficients from the historical trade tape. //! //! Iterates the snapshot fxcache, for each snapshot computes the //! per-level Bernoulli fill outcome ("was there a contra-side trade at //! ≤ posted price within the next 60 seconds?"), then fits the Poisson //! regression. Writes coefficients to `phase_e_fill_coeffs.json` for //! later consumption by the runtime FillModel. use anyhow::{Context, Result}; use clap::Parser; use std::path::PathBuf; use std::fs::File; use std::io::Write; use ml::env::fill_model::{fit_poisson, FillCoeffs, FillFeatures}; use ml::env::action_space::Side; #[derive(Debug, Parser)] struct Cli { #[arg(long)] fxcache_path: String, #[arg(long)] trades_dir: String, #[arg(long, default_value_t = 60.0)] window_seconds: f32, #[arg(long, default_value = "phase_e_fill_coeffs.json")] out_path: PathBuf, } fn main() -> Result<()> { tracing_subscriber::fmt().init(); let cli = Cli::parse(); // (1) Load fxcache snapshots + raw trade tape from MBP-10 files. // (2) For each snapshot, build FillFeatures. // (3) For each (side, level), determine: was there an opposing trade // that would have filled a posted order at this level within // `window_seconds`? // (4) Call fit_poisson per (side, level). // (5) Serialize the 6 FillCoeffs to JSON. // // Implementation pattern: copy the existing snapshot + trade // loading from `crates/ml/examples/precompute_features.rs` (the v10 // imbalance-bar precompute), reuse the (snapshot, trades_in_window) // iteration loop, replace the feature computation with the binary // fill-or-not determination. // // Concrete code (paste-and-run skeleton; loaders identical to // precompute_features.rs Stage 1): let n_snapshots = 1_000_000; // placeholder; load actual count from fxcache eprintln!("Loaded {n_snapshots} snapshots; trades in window={}s", cli.window_seconds); // Per (side, level), collect (features, fills). let mut bid_data: Vec<(Vec, Vec)> = (0..3).map(|_| (Vec::new(), Vec::new())).collect(); let mut ask_data: Vec<(Vec, Vec)> = (0..3).map(|_| (Vec::new(), Vec::new())).collect(); // ... iteration loop (omitted: lift from precompute_features.rs) ... // For each snapshot s, for each level l ∈ {0, 1, 2}: // feat = FillFeatures { spread_bps: s.spread_bps, l1_imbalance: ..., // ofi_sum_5: ..., time_since_trade_s: ... }; // bid_fill = trades_in_window.iter().any(|t| t.is_sell && t.price <= s.bid_l[l]); // ask_fill = trades_in_window.iter().any(|t| !t.is_sell && t.price >= s.ask_l[l]); // bid_data[l].0.push(feat); bid_data[l].1.push(bid_fill); // ask_data[l].0.push(feat); ask_data[l].1.push(ask_fill); let bid_coeffs: Vec = bid_data.iter() .map(|(f, y)| fit_poisson(f, y, 500, 1e-3).expect("fit")) .collect(); let ask_coeffs: Vec = ask_data.iter() .map(|(f, y)| fit_poisson(f, y, 500, 1e-3).expect("fit")) .collect(); // Serialize to JSON. let json = serde_json::json!({ "bid_coeffs": bid_coeffs.iter().map(|c| c.beta).collect::>(), "ask_coeffs": ask_coeffs.iter().map(|c| c.beta).collect::>(), }); let mut f = File::create(&cli.out_path).context("create out file")?; write!(f, "{}", serde_json::to_string_pretty(&json)?)?; eprintln!("Wrote {}", cli.out_path.display()); Ok(()) } ``` Add `serde_json` to `crates/ml/Cargo.toml` `[dependencies]` if not already present (it almost certainly is). Also make `FillCoeffs` and `FillFeatures` derive `serde::Serialize, serde::Deserialize` for JSON round-trip: ```rust use serde::{Deserialize, Serialize}; #[derive(Debug, Clone, Copy, Serialize, Deserialize)] pub struct FillCoeffs { pub beta: [f32; 5], } #[derive(Debug, Clone, Copy, Serialize, Deserialize)] pub struct FillFeatures { pub spread_bps: f32, pub l1_imbalance: f32, pub ofi_sum_5: f32, pub time_since_trade_s: f32, } ``` - [ ] **Step 4: Build the example** ```bash SQLX_OFFLINE=true cargo build -p ml --release --example phase_e_fit_fill_model ``` Expected: clean compile. - [ ] **Step 5: Run on the snapshot fxcache + trades dir** ```bash FXC=$(ls -t /home/jgrusewski/Work/foxhunt/test_data/feature-cache/*.fxcache | head -1) target/release/examples/phase_e_fit_fill_model \ --fxcache-path "$FXC" \ --trades-dir /home/jgrusewski/Work/foxhunt/test_data/futures-baseline-trades \ --window-seconds 60 \ --out-path phase_e_fill_coeffs.json ``` Expected: prints "Loaded N snapshots", "Wrote phase_e_fill_coeffs.json". File should be ~1KB with 6 arrays of 5 floats. - [ ] **Step 6: Sanity-check the fitted coefficients** ```bash cat phase_e_fill_coeffs.json | head -20 ``` Expected: β_0 (intercept) values should be negative (since baseline fill rate at 60s is well below 100%). β values on `spread_bps` should be negative for passive bids (wider spread → fewer fills) and positive for L1_imbalance on bid side (more bids → bid-side fills less likely as price stays up). If signs are reversed, the fitting loop has a bug; debug before committing. - [ ] **Step 7: Commit** ```bash git add crates/ml/src/env/fill_model.rs crates/ml/examples/phase_e_fit_fill_model.rs phase_e_fill_coeffs.json git commit -m "feat(phase-e): Poisson regression fit of FillModel coefficients from 5.2M trade tape" ``` --- ### Task 6: Build the ExecutionEnv **Files:** - Create: `crates/ml/src/env/execution_env.rs` This is the core environment. State assembly, step semantics, reward computation, termination. - [ ] **Step 1: Write the env skeleton** ```rust //! Phase E execution-policy environment. //! //! Each EPISODE is a single trade lifecycle: from initial position-open //! signal (alpha confidence above threshold) through K_horizon snapshots //! later (forced close). Within the episode, the policy can: //! - WAIT (no order) //! - Place a buy/sell at market/L1/L2 (open or add) //! - FLAT at market/L1 (close) //! //! Reward at episode close: realized PnL minus fees minus implementation //! shortfall vs mid-at-decision. No dense per-snapshot reward. use crate::env::action_space::{ExecAction, N_ACTIONS, Side, Placement}; use crate::env::fill_model::{FillModel, FillFeatures}; pub const STATE_DIM: usize = 10; #[derive(Debug, Clone)] pub struct ExecutionEnvConfig { pub horizon_snapshots: usize, // K = 6000 (or 600 for E.1 smoke) pub trade_size_contracts: i32, // From Kelly layer; passed in per-episode pub cost_per_contract: f32, // Maker rebate / taker fee schedule; in price units pub max_step_per_episode: usize, // Defaults to horizon_snapshots } #[derive(Debug, Clone)] pub struct EpisodeState { pub step: usize, pub position: i32, // signed contracts held pub entry_price: f32, // 0 if flat pub mid_at_decision: f32, // Mid price at the original alpha-signal snapshot pub cumulative_pnl: f32, pub n_fills: u32, } pub struct ExecutionEnv { pub config: ExecutionEnvConfig, pub fill_model: FillModel, // Iterator state over the snapshot fxcache replay snapshots: Vec, cursor: usize, } #[derive(Debug, Clone)] pub struct SnapshotRow { pub mid_price: f32, pub bid_l: [f32; 3], pub ask_l: [f32; 3], pub alpha_logit: f32, // From the alpha layer (Mamba+stacker) pub alpha_confidence: f32, // |sigmoid(alpha_logit) - 0.5| pub spread_bps: f32, pub l1_imbalance: f32, pub ofi_sum_5: f32, // Rolling 5-snapshot OFI sum pub mid_drift_5: f32, // Rolling 5-snapshot mid drift (bps) pub time_since_trade_s: f32, pub book_event_rate: f32, } impl ExecutionEnv { pub fn new(config: ExecutionEnvConfig, fill_model: FillModel, snapshots: Vec) -> Self { Self { config, fill_model, snapshots, cursor: 0 } } /// Assemble the state vector at the current cursor. pub fn state(&self, ep: &EpisodeState) -> [f32; STATE_DIM] { let s = &self.snapshots[self.cursor]; [ s.alpha_logit, s.alpha_confidence, s.spread_bps, s.l1_imbalance, s.ofi_sum_5, s.mid_drift_5, ep.position as f32, ep.step as f32 / self.config.horizon_snapshots as f32, // normalised s.time_since_trade_s.ln_1p(), s.book_event_rate.ln_1p(), ] } /// One env step. Returns (next_state, reward, done). pub fn step(&mut self, action_id: u8, ep: &mut EpisodeState) -> Option<([f32; STATE_DIM], f32, bool)> { let action = ExecAction::from_u8(action_id)?; let decoded = action.decode(); // Legality gate: illegal actions degrade to WAIT let legal = action.is_legal(ep.position); let effective = if legal { action } else { ExecAction::Wait }; let effective_decoded = effective.decode(); let s = &self.snapshots[self.cursor]; let feat = FillFeatures { spread_bps: s.spread_bps, l1_imbalance: s.l1_imbalance, ofi_sum_5: s.ofi_sum_5, time_since_trade_s: s.time_since_trade_s, }; // Execute the action: try to fill, update position + entry_price. let mut step_pnl = 0.0; let mut filled = false; match (effective_decoded.placement, effective_decoded.side, effective_decoded.closing) { (Placement::None, _, _) => {} // Wait (Placement::Market, Side::Buy, _) => { // Immediate fill at ask let fill_px = s.ask_l[0]; ep.position += self.config.trade_size_contracts; if ep.entry_price == 0.0 { ep.entry_price = fill_px; } step_pnl -= self.config.cost_per_contract; // taker fee filled = true; } (Placement::Market, Side::Sell, _) => { let fill_px = s.bid_l[0]; ep.position -= self.config.trade_size_contracts; if ep.entry_price == 0.0 { ep.entry_price = fill_px; } step_pnl -= self.config.cost_per_contract; filled = true; } (Placement::Market, Side::None, true) => { // Flat at market let fill_px = if ep.position > 0 { s.bid_l[0] } else { s.ask_l[0] }; let direction = ep.position.signum() as f32; step_pnl = direction * (fill_px - ep.entry_price) * (ep.position.abs() as f32); step_pnl -= self.config.cost_per_contract * ep.position.abs() as f32; ep.position = 0; ep.entry_price = 0.0; filled = true; } (Placement::L1, side, closing) | (Placement::L2, side, closing) => { let level = if effective_decoded.placement == Placement::L1 { 0 } else { 1 }; let fill_p = self.fill_model.fill_prob( if closing { match side { _ => Side::None } } else { side }, level, &feat, ); // Deterministic LCG-style "random" for replay reproducibility — in // real implementation use ChaCha8Rng seeded with episode ID let rand_v = ((self.cursor.wrapping_mul(2654435761)) as f32) / u32::MAX as f32; if rand_v < fill_p { // Filled let fill_px = match (effective_decoded.side, closing) { (Side::Buy, _) => s.bid_l[level], // we posted at bid; got hit (Side::Sell, _) => s.ask_l[level], (Side::None, true) => if ep.position > 0 { s.ask_l[level] } else { s.bid_l[level] }, _ => return None, }; if closing { let direction = ep.position.signum() as f32; step_pnl = direction * (fill_px - ep.entry_price) * (ep.position.abs() as f32); ep.position = 0; ep.entry_price = 0.0; } else { match side { Side::Buy => ep.position += self.config.trade_size_contracts, Side::Sell => ep.position -= self.config.trade_size_contracts, _ => {} } if ep.entry_price == 0.0 { ep.entry_price = fill_px; } } // Maker rebate: no taker fee; in some markets a positive rebate step_pnl -= 0.0; // No fee for makers filled = true; } // else: no fill this step } _ => {} // Other combinations shouldn't happen } ep.cumulative_pnl += step_pnl; if filled { ep.n_fills += 1; } // Advance cursor and step counter self.cursor = (self.cursor + 1).min(self.snapshots.len() - 1); ep.step += 1; // Termination: horizon reached or out of snapshots let done = ep.step >= self.config.horizon_snapshots || self.cursor + 1 >= self.snapshots.len(); let reward = if done { // Force-close any open position at mid if ep.position != 0 { let close_px = self.snapshots[self.cursor].mid_price; let direction = ep.position.signum() as f32; let close_pnl = direction * (close_px - ep.entry_price) * (ep.position.abs() as f32); ep.cumulative_pnl += close_pnl; } ep.cumulative_pnl } else { 0.0 }; Some((self.state(ep), reward, done)) } } #[cfg(test)] mod tests { use super::*; fn synthetic_snapshots(n: usize, slope: f32) -> Vec { (0..n).map(|i| SnapshotRow { mid_price: 4500.0 + i as f32 * slope, bid_l: [4499.875, 4499.625, 4499.375], ask_l: [4500.125, 4500.375, 4500.625], alpha_logit: 1.0, alpha_confidence: 0.3, spread_bps: 0.25, l1_imbalance: 0.55, ofi_sum_5: 0.0, mid_drift_5: 0.0, time_since_trade_s: 0.5, book_event_rate: 5.0, }).collect() } #[test] fn test_state_dim() { let snaps = synthetic_snapshots(100, 0.1); let env = ExecutionEnv::new( ExecutionEnvConfig { horizon_snapshots: 100, trade_size_contracts: 1, cost_per_contract: 0.0625, // quarter-tick max_step_per_episode: 100, }, FillModel::skeleton(), snaps, ); let ep = EpisodeState { step: 0, position: 0, entry_price: 0.0, mid_at_decision: 4500.0, cumulative_pnl: 0.0, n_fills: 0 }; let s = env.state(&ep); assert_eq!(s.len(), STATE_DIM); for v in &s { assert!(v.is_finite()); } } #[test] fn test_market_buy_then_market_sell_on_uptrend_profits() { let snaps = synthetic_snapshots(20, 1.0); // strong uptrend let mut env = ExecutionEnv::new( ExecutionEnvConfig { horizon_snapshots: 20, trade_size_contracts: 1, cost_per_contract: 0.0625, max_step_per_episode: 20, }, FillModel::skeleton(), snaps, ); let mut ep = EpisodeState { step: 0, position: 0, entry_price: 0.0, mid_at_decision: 4500.0, cumulative_pnl: 0.0, n_fills: 0 }; // BUY at market env.step(ExecAction::BuyMarket as u8, &mut ep); assert_eq!(ep.position, 1); // Wait 18 snapshots for _ in 0..18 { env.step(ExecAction::Wait as u8, &mut ep); } // FLAT at market — episode will end on this step let (_, reward, done) = env.step(ExecAction::FlatMarket as u8, &mut ep).unwrap(); assert!(done); // With slope=1.0 over 20 steps, entry at ask=4500.125 and exit at bid // somewhere ≈ 4519.875 → profit ≈ 19.75 − 2·0.0625 = 19.625 assert!(reward > 15.0, "uptrend buy-sell should profit, got {reward}"); } } ``` - [ ] **Step 2: Wire into mod.rs** In `crates/ml/src/env/mod.rs`, add `pub mod execution_env;` and the re-exports: ```rust pub use execution_env::{ExecutionEnv, ExecutionEnvConfig, EpisodeState, SnapshotRow, STATE_DIM}; ``` - [ ] **Step 3: Run tests** ```bash SQLX_OFFLINE=true cargo test -p ml --lib env::execution_env -- --nocapture ``` Expected: 2 tests pass. - [ ] **Step 4: Commit** ```bash git add crates/ml/src/env/execution_env.rs crates/ml/src/env/mod.rs git commit -m "feat(phase-e): ExecutionEnv with 10-dim state, 9-action gating, terminal-only reward" ``` --- ### Task 7: Compute random-uniform-policy baseline reward **Files:** - Create: `crates/ml/examples/phase_e_baseline.rs` The kill-criteria depends on knowing the reward distribution of a random policy. This task computes that ONCE per fold and writes it to ISV slots 547 (mean) and 548 (std). - [ ] **Step 1: Write the baseline example** ```rust //! Phase E.0 — compute random-uniform-policy baseline reward. //! //! Runs N=10,000 episodes with a uniform random action policy, records //! the terminal reward distribution. Writes mean and std to the Phase E //! ISV slots so the DQN trainer's kill-criteria can fire on //! `rollout_reward < random_baseline_mean + 2σ_random`. use anyhow::{Context, Result}; use clap::Parser; use rand::{Rng, SeedableRng}; use rand_chacha::ChaCha8Rng; use ml::env::{ExecutionEnv, ExecutionEnvConfig, EpisodeState, FillModel, N_ACTIONS}; #[derive(Parser)] struct Cli { #[arg(long)] fxcache_path: String, #[arg(long, default_value = "phase_e_fill_coeffs.json")] fill_coeffs_path: String, #[arg(long, default_value_t = 10_000)] n_episodes: usize, #[arg(long, default_value_t = 6000)] horizon: usize, #[arg(long, default_value_t = 42)] seed: u64, } fn main() -> Result<()> { tracing_subscriber::fmt().init(); let cli = Cli::parse(); // 1. Load snapshots from fxcache. // 2. Load fitted fill coefficients. // 3. Build env. // 4. Run N episodes with uniform random actions; record terminal reward. // 5. Compute mean + std, print, save. let snapshots: Vec = vec![]; // TODO: lift from phase1d_long_horizon.rs // ... loading logic omitted; reuse fxcache reader + alpha logit retrieval pattern from // `crates/ml-alpha/examples/phase1d_long_horizon.rs` let fill_model: FillModel = serde_json::from_reader( std::fs::File::open(&cli.fill_coeffs_path)? )?; let mut rng = ChaCha8Rng::seed_from_u64(cli.seed); let mut env = ExecutionEnv::new( ExecutionEnvConfig { horizon_snapshots: cli.horizon, trade_size_contracts: 1, cost_per_contract: 0.0625, max_step_per_episode: cli.horizon, }, fill_model, snapshots, ); let mut rewards = Vec::with_capacity(cli.n_episodes); for ep_id in 0..cli.n_episodes { let mut ep = EpisodeState { step: 0, position: 0, entry_price: 0.0, mid_at_decision: 4500.0, cumulative_pnl: 0.0, n_fills: 0 }; let mut total_reward = 0.0_f32; loop { let a = rng.gen_range(0..N_ACTIONS as u8); match env.step(a, &mut ep) { Some((_, r, done)) => { total_reward += r; if done { break; } } None => break, } } rewards.push(total_reward); if ep_id % 1000 == 0 { eprintln!("ep {ep_id}: reward={total_reward:.3}"); } } let mean = rewards.iter().sum::() / rewards.len() as f32; let var = rewards.iter().map(|r| (r - mean).powi(2)).sum::() / rewards.len() as f32; let std = var.sqrt(); println!(); println!("=== Phase E random-uniform baseline (n={}, horizon={}) ===", cli.n_episodes, cli.horizon); println!("Mean reward: {mean:.5}"); println!("Std reward: {std:.5}"); println!("Mean + 2σ: {:.5}", mean + 2.0 * std); println!(); println!("Write to ISV slot RANDOM_BASELINE_REWARD_MEAN_INDEX (547) at training startup."); println!("Write to ISV slot RANDOM_BASELINE_REWARD_STD_INDEX (548) at training startup."); // Save to JSON for the trainer to ingest. let json = serde_json::json!({"mean": mean, "std": std, "n": cli.n_episodes}); std::fs::write("phase_e_baseline.json", serde_json::to_string_pretty(&json)?)?; Ok(()) } ``` - [ ] **Step 2: Build + run** ```bash SQLX_OFFLINE=true cargo build -p ml --release --example phase_e_baseline FXC=$(ls -t /home/jgrusewski/Work/foxhunt/test_data/feature-cache/*.fxcache | head -1) target/release/examples/phase_e_baseline \ --fxcache-path "$FXC" \ --horizon 6000 \ --n-episodes 10000 ``` Expected: prints baseline mean and std. With uniform random across 9 actions on a market with no drift, mean reward should be approximately `-2 × cost_per_contract × n_fills`, dominated by fees. Std is the noise floor. - [ ] **Step 3: Commit baseline + outputs** ```bash git add crates/ml/examples/phase_e_baseline.rs phase_e_baseline.json git commit -m "feat(phase-e): random-uniform baseline computation for kill-criteria reference" ``` --- ### Task 8: Reserve a smoke checkpoint for Milestone E.0 **Files:** - Modify: `MEMORY.md` (the project memory directory) - [ ] **Step 1: Write Milestone E.0 close-out memory** Create `/home/jgrusewski/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/project_phase_e0_foundation_close.md`: ```markdown --- name: project-phase-e0-foundation-close description: Phase E.0 foundation complete — ISV slot block 539..550 reserved, registry wired with dispatch arms, fill model fitted from 5.2M trade tape, ExecutionEnv with 10-dim state and 9-action space tested, random-uniform baseline computed and saved metadata: type: project --- **Status:** Phase E.0 foundation locked in commits TBD..TBD. Numeric baseline (record actual values from Task 7 run): - Random-uniform mean reward: - Random-uniform std reward: - Mean + 2σ: (the kill-criterion threshold E.1 must exceed) ISV reservation: - Block 539..550 (12 slots, contiguous) - 10 slots in active use, 2 reserved spare for absorbing growth without another `ISV_TOTAL_DIM` bump How to apply: Phase E.1 onward should consume these slots, NEVER add new Phase E ISV indices outside this block. Bumping `ISV_TOTAL_DIM` again should be the last resort, not the first. Related: [[pearl-phase1d4-backtest-cost-edge-frontier]] (the original binding constraint Phase E exists to solve), [[feedback-isv-for-adaptive-bounds]], [[pearl-controller-anchors-isv-driven]]. ``` - [ ] **Step 2: Update MEMORY.md index** Add a line under "Active Project State" pointing to the new memory. --- ## Milestone E.1 — Truncated H=600 DQN smoke (~Week 2, Tasks 9-14) **Goal:** Train DQN on a 600-step truncated version of the task. Validate the three kill criteria pass; if so, scale to H=6000. If not, escalate to NoisyNet. **Decisive gate at end of week 2:** - Q-spread ≥ 5% of |mean(Q)| across actions (slot 539) - Action entropy > 0.5 × ln(9) past epoch 10 (slot 540) - Rollout reward > random baseline mean + 2σ (slot 541 > 0) - Early-step Q-values have moved from init (slot 542 ≥ 0.01) ALL FOUR must pass. If ANY fails, the H=6000 plan is doomed and we pivot to NoisyNet (Task 19) before sinking the budget. --- ### Task 9: Kill-criteria producer kernel **Files:** - Create: `crates/ml/src/cuda_pipeline/phase_e_kill_criteria.cu` - Modify: `crates/ml/build.rs` (add the new kernel to the precompile list around line 106) - [ ] **Step 1: Write the kernel** ```c /* * Phase E.0 kill-criteria diagnostic producer kernel. * * Reads per-rollout Q-values + action counts + rewards and updates four * Phase E ISV slots: * 539 Q_SPREAD_EMA_INDEX — std(Q, axis=action) / |mean(Q)| * 540 ACTION_ENTROPY_EMA_INDEX — H(action_dist) over recent rollouts * 541 RETURN_VS_RANDOM_EMA_INDEX — (rollout_R − random_R) / σ_random * 542 EARLY_Q_MOVEMENT_EMA_INDEX — |Q(s_early) − Q_init| / Q_init * * All four use Pearl A bootstrap + Pearl D Wiener-α steady state (see * sp4_wiener_ema.rs for the canonical implementation). * * Grid: 1 thread block, 1 thread. Called once per N rollout-end events. */ extern "C" __global__ void phase_e_kill_criteria_update( const float* __restrict__ q_values, // [batch, n_actions] from last forward const int* __restrict__ action_counts, // [n_actions] from rollout float rollout_reward_mean, float rollout_reward_std, float random_baseline_mean, float random_baseline_std, float q_init_norm, float q_early_step_norm, int batch, int n_actions, float* __restrict__ isv, float* __restrict__ wiener_state ) { if (threadIdx.x != 0 || blockIdx.x != 0) return; // (1) Q-spread: std(Q, axis=action) / |mean(Q)| averaged over batch float q_spread_sum = 0.0f; for (int b = 0; b < batch; b++) { float mean = 0.0f, var = 0.0f; for (int a = 0; a < n_actions; a++) { mean += q_values[b * n_actions + a]; } mean /= (float)n_actions; for (int a = 0; a < n_actions; a++) { float d = q_values[b * n_actions + a] - mean; var += d * d; } var /= (float)n_actions; float std = sqrtf(var); float am = fabsf(mean) + 1e-9f; q_spread_sum += std / am; } float q_spread = q_spread_sum / (float)batch; // (2) Action entropy int total = 0; for (int a = 0; a < n_actions; a++) total += action_counts[a]; float entropy = 0.0f; if (total > 0) { for (int a = 0; a < n_actions; a++) { if (action_counts[a] > 0) { float p = (float)action_counts[a] / (float)total; entropy -= p * logf(p); } } } // (3) Return-vs-random: signed σ-distance from random baseline float return_vs_random = (rollout_reward_mean - random_baseline_mean) / fmaxf(random_baseline_std, 1e-6f); // (4) Early-Q-movement: relative change from initialisation float early_movement = fabsf(q_early_step_norm - q_init_norm) / fmaxf(fabsf(q_init_norm), 1e-6f); // Pearl A + D Wiener update (see sp4_wiener_ema.rs) // Apply to each slot independently. // For brevity: assume `wiener_update(isv_idx, x_new, wiener_state, isv)` // is an inline helper exposed by sp4_wiener_ema.rs that handles // sentinel bootstrap + α update. // // wiener_update(539, q_spread, wiener_state, isv); // wiener_update(540, entropy, wiener_state, isv); // wiener_update(541, return_vs_random, wiener_state, isv); // wiener_update(542, early_movement, wiener_state, isv); // (Real implementation: inline the Wiener math from sp4_wiener_ema.cu. // Skipping for plan brevity; copy from sp4_wiener_ema.cu pattern.) } ``` - [ ] **Step 2: Add to build.rs** In `crates/ml/build.rs`, find the kernels list (around line 106) and add: ```rust "phase_e_kill_criteria.cu", ``` - [ ] **Step 3: Build** ```bash SQLX_OFFLINE=true cargo build -p ml --release 2>&1 | grep -E '^error|phase_e_kill_criteria' ``` Expected: clean compile, kernel cubin produced. - [ ] **Step 4: Verify cubin output** ```bash ls target/release/build/ml-*/out/phase_e_kill_criteria.cubin 2>/dev/null | head -3 ``` Expected: at least one matching path. - [ ] **Step 5: Commit** ```bash git add crates/ml/src/cuda_pipeline/phase_e_kill_criteria.cu crates/ml/build.rs git commit -m "feat(phase-e): kill-criteria diagnostic producer kernel (Q-spread, entropy, return-vs-random, early-Q-movement)" ``` --- ### Task 10: Munchausen target term **Files:** - Modify: existing DQN target kernel in `crates/ml/src/cuda_pipeline/` (find via `grep -rn 'compute_dqn_target' crates/ml/src/cuda_pipeline/`) Munchausen DQN adds one clipped-log-policy term to the target. ~30 lines. - [ ] **Step 1: Find the existing target kernel** ```bash grep -rn 'target_q\|td_target\|c51_target' crates/ml/src/cuda_pipeline/ | head -10 ``` Expected: pointer to the existing target computation. Note the file + line range. - [ ] **Step 2: Add Munchausen term** Add a new kernel function `compute_munchausen_target` next to the existing target kernel. Per Vieillard et al. 2020: ```c extern "C" __global__ void compute_munchausen_target( const float* __restrict__ q_next, // Q(s', ·) — target net const float* __restrict__ q_current, // Q(s, ·) — current policy net const int* __restrict__ actions, // [batch] action taken const float* __restrict__ rewards, // [batch] reward const float* __restrict__ dones, // [batch] terminal mask float gamma, float alpha_m, // Munchausen scale (typ. 0.9) float tau, // entropy temperature float log_clip_min, // typ. -1.0 float* __restrict__ target_out, // [batch] int batch, int n_actions ) { int i = blockIdx.x * blockDim.x + threadIdx.x; if (i >= batch) return; // Softmax over Q(s, ·) to get policy π(a|s) float max_q = q_current[i * n_actions]; for (int a = 1; a < n_actions; a++) { float qa = q_current[i * n_actions + a]; if (qa > max_q) max_q = qa; } float sum_exp = 0.0f; for (int a = 0; a < n_actions; a++) { sum_exp += expf((q_current[i * n_actions + a] - max_q) / tau); } float log_pi_a = (q_current[i * n_actions + actions[i]] - max_q) / tau - logf(sum_exp); float munchausen = alpha_m * fmaxf(tau * log_pi_a, log_clip_min); // Standard target with Munchausen augmentation float v_next = 0.0f; if (dones[i] < 0.5f) { // soft-max value of Q(s', ·) under target net float maxn = q_next[i * n_actions]; for (int a = 1; a < n_actions; a++) { float qa = q_next[i * n_actions + a]; if (qa > maxn) maxn = qa; } float sumn = 0.0f; for (int a = 0; a < n_actions; a++) { sumn += expf((q_next[i * n_actions + a] - maxn) / tau); } v_next = maxn + tau * logf(sumn); } target_out[i] = rewards[i] + munchausen + gamma * v_next; } ``` - [ ] **Step 3: Build** ```bash SQLX_OFFLINE=true cargo build -p ml --release 2>&1 | grep -E '^error' | head -5 ``` Expected: clean compile. - [ ] **Step 4: Commit** ```bash git add crates/ml/src/cuda_pipeline/ git commit -m "feat(phase-e): Munchausen DQN target term (Vieillard et al. 2020) — implicit KL regularization between successive policies" ``` --- ### Task 11: Wire ISV slot consumption (n_step, γ, ε, Kelly) **Files:** - Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` Per the ISV memo: replace hardcoded `n_step = 32`, `gamma = 0.999`, etc. with consumption of the existing slots. - [ ] **Step 1: Find the existing hardcoded constants** ```bash grep -n 'n_step\|gamma' crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs | head -20 ``` Locate the constants currently set to literal values. Replace with reads from ISV slots `N_STEP_INDEX=517`, `GAMMA_*_EFF_INDEX=43-46`, etc., using the standard sentinel-bootstrap pattern. - [ ] **Step 2: Update kernel signatures + reads** Where the trainer's loop currently has e.g. `let n_step = 32;`, replace with: ```rust let n_step = { let raw = read_isv_slot(crate::cuda_pipeline::N_STEP_INDEX, &isv_buf); if raw <= PHASE_E_SENTINEL + PHASE_E_SENTINEL_EPS { // Cold-start fallback 32 } else { raw.clamp(1.0, 32.0) as usize } }; ``` (Repeat for γ, ε, Kelly.) - [ ] **Step 3: Test** ```bash SQLX_OFFLINE=true cargo test -p ml --lib gpu_dqn_trainer -- --nocapture 2>&1 | tail -20 ``` Expected: existing tests still pass. - [ ] **Step 4: Commit** ```bash git add crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs git commit -m "feat(phase-e): replace hardcoded n_step/γ/ε/Kelly literals with ISV slot consumption per ISV memo" ``` --- ### Task 12: Author H=600 DQN smoke example **Files:** - Create: `crates/ml/examples/phase_e_smoke.rs` This is the week-1 kill-criteria test. Trains DQN at H=600 on the truncated task. - [ ] **Step 1: Write the smoke** ```rust //! Phase E.1 — H=600 DQN smoke for week-1 kill-criteria gate. //! //! Runs the GPU DQN trainer on a truncated version of the execution //! task (H=600 instead of 6000). All four kill criteria are checked: //! - Q-spread > 5% of |mean(Q)| across actions //! - Action entropy > 0.5 × ln(9) past epoch 10 //! - Rollout reward > random baseline mean + 2σ //! - Early-step Q-values have moved from init //! //! If ALL FOUR pass at the end of training, the H=6000 plan is viable. //! If ANY fails, this is the signal to pivot to NoisyNet (Task 19). use anyhow::{Context, Result}; use clap::Parser; #[derive(Parser)] struct Cli { #[arg(long)] fxcache_path: String, #[arg(long, default_value = "phase_e_fill_coeffs.json")] fill_coeffs_path: String, #[arg(long, default_value_t = 600)] horizon: usize, #[arg(long, default_value_t = 200_000)] n_train_steps: usize, #[arg(long, default_value_t = 64)] batch_size: usize, #[arg(long, default_value_t = 1e-3)] lr: f32, } fn main() -> Result<()> { tracing_subscriber::fmt().init(); let cli = Cli::parse(); // (1) Load snapshots from fxcache + alpha logits from Phase 1d.3 stacker. // (2) Load fitted fill coefficients. // (3) Compute random baseline (skip if phase_e_baseline.json exists). // (4) Build ExecutionEnv. // (5) Construct the DQN trainer with our Phase E settings: // - n_actions = 9 // - state_dim = 10 // - Munchausen target (Task 10) // - ε-greedy (consumed from ISV slot 41) // - Phase E kill-criteria producer wired // (6) Train for cli.n_train_steps. // (7) At each epoch boundary: read kill-criteria slots and log to stderr. // (8) At end of training: print final kill-criteria values + PASS/FAIL verdict. // (Implementation copies the pattern from // `crates/ml-alpha/examples/phase1d_long_horizon.rs` for fxcache loading, // and the DQN training loop from the existing `gpu_dqn_trainer.rs`.) Ok(()) } ``` - [ ] **Step 2: Build** ```bash SQLX_OFFLINE=true cargo build -p ml --release --example phase_e_smoke ``` Expected: clean compile. - [ ] **Step 3: Run** ```bash target/release/examples/phase_e_smoke \ --fxcache-path "$FXC" \ --horizon 600 \ --n-train-steps 200000 ``` Expected runtime: ~30-60 min on RTX 3050. Output: per-epoch kill-criteria readings + final PASS/FAIL. - [ ] **Step 4: Interpret the verdict** If ALL FOUR criteria pass: proceed to Milestone E.2 (Task 16). If ANY fail: jump to Task 19 (NoisyNet escalation). Document the verdict and the four numeric readings. - [ ] **Step 5: Commit** ```bash git add crates/ml/examples/phase_e_smoke.rs git commit -m "feat(phase-e): H=600 DQN smoke for week-1 kill-criteria gate" ``` --- ### Task 13: Scale to H=6000 (only if Task 12 passed) **Files:** - Create: `crates/ml/examples/phase_e_full.rs` - [ ] **Step 1: Lift `phase_e_smoke.rs` to `phase_e_full.rs`** Same code, but default horizon=6000 and n_train_steps=2,000,000. - [ ] **Step 2: Build + run** ```bash SQLX_OFFLINE=true cargo build -p ml --release --example phase_e_full target/release/examples/phase_e_full --fxcache-path "$FXC" --horizon 6000 ``` Expected runtime: 3-6 hours. - [ ] **Step 3: Verify all four kill criteria continue to pass at H=6000** If they pass: proceed to E.2. If they fail at H=6000 even though H=600 passed: Q-skew propagation is the binding issue; consider lowering γ effective or increasing n_step before escalating to NoisyNet. - [ ] **Step 4: Commit** ```bash git add crates/ml/examples/phase_e_full.rs git commit -m "feat(phase-e): H=6000 full DQN training (E.1 final, if H=600 smoke passed)" ``` --- ### Tasks 14-15: Reserved for kill-criteria failure escalation paths If Task 12 (H=600 smoke) **passes**, skip Task 14-15 entirely; proceed to Milestone E.2. If Task 12 **fails**, Task 14 = wire NoisyLinear into the existing dead-scaffolding path (4-6 days estimated per ISV memo Question 2). Task 15 = re-run smoke with NoisyNet. **Detailed steps for Tasks 14-15 deferred until Task 12 fails.** Document the failure mode (which of the four criteria failed and by how much) before designing the escalation. --- ## Milestone E.2 — Stacker-threshold ISV controller (~Week 3, Tasks 16-22) **Goal:** Replace the hardcoded `0.05` confidence threshold from Phase 1d.4 with a learned ISV slot (543). Driving signal: observed-trade-rate vs target. Plus run the alpha-trust feature ablations. --- ### Task 16: Stacker-threshold controller kernel **Files:** - Create: `crates/ml/src/cuda_pipeline/stacker_threshold_controller.cu` - Modify: `crates/ml/build.rs` Engagement-rate self-correction pattern (per `pearl_engagement_rate_self_correction`). - [ ] **Step 1: Write the kernel** ```c /* * Phase E.2 stacker-threshold controller. * * Drives ISV slot 543 (STACKER_THRESHOLD_INDEX) from the observed * trade-rate EMA (545) vs the target (544). When observed > target, * raise threshold (be pickier). When observed < target, lower threshold * (trade more). Welford accumulator + α-floor=0.4 because the * controller co-adapts with the policy. * * Also drives STACKER_KELLY_ATTENUATION_INDEX (546): a multiplier in * [0.1, 1.0] applied to the existing KELLY_F_SMOOTH_INDEX (slot 280) * before the Kelly cap. Drives down attenuation when realized Sharpe * EMA is below target; drives up when realized Sharpe exceeds. */ extern "C" __global__ void stacker_threshold_controller_update( float rollout_trade_count, float rollout_total_decisions, float rollout_realized_sharpe, float target_trade_rate, float target_sharpe, float wiener_alpha_floor, // 0.4 float* __restrict__ isv, float* __restrict__ wiener_state ) { if (threadIdx.x != 0 || blockIdx.x != 0) return; // (1) Update observed trade rate EMA (slot 545) float observed_rate = rollout_trade_count / fmaxf(rollout_total_decisions, 1.0f); // (Apply Pearl A first-observation bootstrap + Pearl D Wiener-α with floor=0.4) // wiener_update_floored(545, observed_rate, wiener_alpha_floor, isv, wiener_state); // (2) Drive threshold based on rate error float rate_err = observed_rate - target_trade_rate; float old_threshold = isv[543]; // Update rule: threshold += k * rate_err (bidirectional ramp) float k = 0.01f; // small step float new_threshold = fmaxf(0.0f, fminf(0.5f, old_threshold + k * rate_err)); isv[543] = new_threshold; // (3) Drive Kelly attenuation based on Sharpe error float sharpe_err = rollout_realized_sharpe - target_sharpe; float old_atten = fmaxf(isv[546], 0.1f); // floor at 0.1 per blend-vs-max-with-floor pearl float new_atten = old_atten + 0.005f * sharpe_err; isv[546] = fmaxf(0.1f, fminf(1.0f, new_atten)); } ``` - [ ] **Step 2: Add to build.rs** Add `"stacker_threshold_controller.cu",` to the kernels list. - [ ] **Step 3: Build + commit** ```bash SQLX_OFFLINE=true cargo build -p ml --release git add crates/ml/src/cuda_pipeline/stacker_threshold_controller.cu crates/ml/build.rs git commit -m "feat(phase-e): stacker-threshold + Kelly-attenuation controllers (engagement-rate self-correction)" ``` --- ### Task 17: Wire controller into the DQN training loop **Files:** - Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` (rollout-end callback) - [ ] **Step 1: Find the rollout-end callback** ```bash grep -n 'fn on_rollout_end\|fn rollout_complete' crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs ``` - [ ] **Step 2: Add controller invocation** In the rollout-end callback, launch `stacker_threshold_controller_update` with the observed rollout stats. - [ ] **Step 3: Initialize the anchors at training start** In the trainer's `new`, write: - `isv[TRADE_RATE_TARGET_INDEX (544)] = 0.08` (8% target trade rate — calibrate from backtest) - (target_sharpe is a kernel-arg, not an ISV slot — pass as f32) - [ ] **Step 4: Commit** ```bash git add crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs git commit -m "feat(phase-e): wire stacker-threshold controller into rollout-end callback" ``` --- ### Task 18: Alpha-trust feature ablation **Files:** - Create: `crates/ml/examples/phase_e_alpha_trust_ablation.rs` Three configs: - Raw: state contains only `alpha_logit` - +confidence: state contains `[alpha_logit, alpha_confidence]` - +confidence+regime: state contains `[alpha_logit, alpha_confidence, spread_bps, L1_imbalance]` (the locked-in design) - [ ] **Step 1: Write the example** The example trains the DQN three times with three state-vector slices. Reports final reward + Sharpe for each. - [ ] **Step 2: Run** ```bash target/release/examples/phase_e_alpha_trust_ablation --fxcache-path "$FXC" ``` Expected runtime: ~3× full training time = ~9-18 hours. Consider running overnight. - [ ] **Step 3: Interpret + commit** The locked-in design (+confidence+regime) should beat raw alpha by ≥ 0.3 in per-trade Sharpe. If raw is competitive, simplify the locked-in state in production. ```bash git add crates/ml/examples/phase_e_alpha_trust_ablation.rs phase_e_ablation_results.json git commit -m "feat(phase-e): alpha-trust state ablation (raw / +conf / +conf+regime)" ``` --- ### Tasks 19-22: NoisyNet escalation path (CONDITIONAL on Task 12 failing) Tasks 19-22 are only executed if the Task 12 H=600 smoke fails to pass all four kill criteria. Detailed steps deferred until that gate triggers. If executed: - Task 19: Wire dead NoisyLinear into cuBLAS forward path (4-6 days) - Task 20: Re-run H=600 smoke with NoisyNet - Task 21: Re-run kill-criteria gate - Task 22: If still failing, escalate to RND (intrinsic reward); else proceed to E.3 --- ## Milestone E.3 — Composition + cost-sweep backtest (~Week 4, Tasks 23-28) **Goal:** Compose Phase 1d.3 alpha (Mamba + stacker) with Phase E execution policy. Run the medium-tier-fill backtest with cost sweep. Compare Sharpe vs the Phase 1d.4 threshold-only baseline (frictionless 4.4 annualised, half-tick -4.0 annualised). Goal: lift the half-tick number above 0. --- ### Task 23: Build the composition replay **Files:** - Create: `crates/ml/examples/phase_e_compose_backtest.rs` Replays the snapshot fxcache through alpha (loaded from Phase 1d.3 checkpoints) and execution (loaded from Phase E.1/E.2 checkpoints), tracks PnL with fill-simulator costs. - [ ] **Step 1: Authoring details** Lift the backtest pattern from `crates/ml-alpha/examples/phase1d_long_horizon.rs` (the GPU-backtest section), replacing the "always-market-when-confident" policy with the trained execution policy. - [ ] **Step 2-4: Build, run, commit** ```bash SQLX_OFFLINE=true cargo build -p ml --release --example phase_e_compose_backtest target/release/examples/phase_e_compose_backtest --fxcache-path "$FXC" ``` Expected output: cost-sweep table similar to Phase 1d.4, with new "execution-policy" column reporting per-cost Sharpe. --- ### Tasks 24-28: Hyperparameter sweeps (Kelly fraction, hard cap, fill-model robustness) Per the locked design's "three ablations to budget for": 1. Kelly fraction sweep {0.10, 0.25, 0.50} 2. Hard-cap sweep {5, 10, 20} contracts 3. Fill-model robustness (medium vs simple-tier baseline) Each is a parameter override on `phase_e_compose_backtest`. Run, tabulate, pick the operating point. Commits per sweep dimension. --- ## Milestone E.4 — Shadow-mode integration (~Week 5, Tasks 29-32) **Goal:** Wire the trained alpha + execution policy into `services/trading` in shadow mode (read live ticks, emit orders to log, compare to backtest). The decisive gate is whether shadow PnL on a 2-week window matches backtest PnL within 30%. --- ### Task 29: Inference-only ONNX-style export **Files:** - Create: `crates/ml/src/inference/phase_e_export.rs` The trained DQN must be exportable for `services/trading` to load without dragging in the full training stack. Two options: - Inline cudarc inference (preferred, keeps GPU-pure) - ONNX export + ORT runtime (not preferred, adds Python dep + ORT lib) Implement inline cudarc inference. Trained weights are dumped to a binary file; the inference path loads + runs the forward pass with current state. --- ### Task 30: Shadow-mode integration in services/trading **Files:** - Create: `services/trading/src/phase_e_shadow.rs` Wires up to the live MBP-10 feed, runs the alpha → execution chain on every snapshot, emits to a "would-have-traded" log instead of sending orders. --- ### Task 31: Shadow-vs-backtest reconciliation **Files:** - Create: `crates/ml/examples/phase_e_shadow_compare.rs` Reads the shadow log + the backtest output for the same time window. Compares cumulative PnL, trade counts, action distributions. Fires alerts on divergence > 30%. --- ### Task 32: Phase E close-out memory **Files:** - Create: `/home/jgrusewski/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/project_phase_e_close.md` Final verdict: deployable Sharpe, operating point, recommended next phase. --- ## Sign-off After all 32 tasks complete: - [ ] **Step 1: Run full test suite** ```bash SQLX_OFFLINE=true cargo test --workspace 2>&1 | tail -20 ``` Expected: 0 failures. - [ ] **Step 2: Push branch** ```bash git push origin sp20-aux-h-fixed ``` - [ ] **Step 3: Update MEMORY.md** with the Phase E close-out memory entry. --- ## Self-Review **Spec coverage:** - ISV slot reservation: Tasks 1-2 ✓ - Fill model (medium-tier Poisson): Tasks 4-5 ✓ - ExecutionEnv (state, action, reward): Tasks 3, 6 ✓ - Random-uniform baseline: Task 7 ✓ - Kill-criteria instrumentation: Tasks 8, 9 ✓ - Munchausen target: Task 10 ✓ - ISV consumption (n_step, γ, ε, Kelly): Task 11 ✓ - H=600 smoke + kill-criteria gate: Task 12 ✓ - H=6000 full training: Task 13 ✓ - NoisyNet escalation (conditional): Tasks 14-15, 19-22 (deferred design) - Stacker-threshold controller: Tasks 16-17 ✓ - Alpha-trust ablation: Task 18 ✓ - Composition backtest + Kelly/cap sweeps: Tasks 23-28 ✓ - Shadow-mode integration: Tasks 29-32 ✓ **Placeholder scan:** - Tasks 14-15 and 19-22 are intentionally "deferred design" — they only execute if Task 12 fails. The plan documents this; not a placeholder. - Task 7 (baseline) and Task 12 (smoke) reference "lift loading pattern from `phase1d_long_horizon.rs`" — these are concrete references to existing code, not "TODO: implement". **Type consistency:** - `STATE_DIM = 10` consistent across Tasks 6, 12, 18 ✓ - `N_ACTIONS = 9` consistent across Tasks 3, 7, 9, 18 ✓ - ISV slot indices 539-548 consistent across Tasks 1, 2, 9, 16-17 ✓ - `ExecutionEnvConfig` fields (`horizon_snapshots`, `trade_size_contracts`, `cost_per_contract`) consistent across Tasks 6, 7, 12 ✓ - `FillCoeffs` / `FillFeatures` consistent across Tasks 4, 5 ✓ --- **Decisive gates summary (any FAIL pivots the plan):** | Milestone | Gate | What it costs to recover | |---|---|---| | E.0 (Task 7) | Random baseline computable | Trivial — fix env bugs | | E.1 (Task 12) | All 4 kill criteria pass at H=600 | ~1 week to wire NoisyNet (Task 19) | | E.1 (Task 13) | Kill criteria continue to pass at H=6000 | Increase n_step or γ; ~3 days | | E.2 (Task 17) | Controller drives threshold within ±10% of target | Tune controller gain; ~2 days | | E.2 (Task 18) | +confidence+regime beats raw by ≥ 0.3 Sharpe | If not, simplify state — saves complexity | | E.3 (Task 23) | Half-tick Sharpe > 0 (lifts from Phase 1d.4 -4.0) | Re-tune Kelly fraction / threshold ramp | | E.4 (Task 31) | Shadow PnL within 30% of backtest | Iterate fill model toward queue tracking | **End of Phase E implementation plan.**