diff --git a/docs/superpowers/specs/2026-05-16-ml-alpha-cfc-ppo-design.md b/docs/superpowers/specs/2026-05-16-ml-alpha-cfc-ppo-design.md new file mode 100644 index 000000000..362212370 --- /dev/null +++ b/docs/superpowers/specs/2026-05-16-ml-alpha-cfc-ppo-design.md @@ -0,0 +1,635 @@ +# ml-alpha — Greenfield CfC + PPO Trading System Design + +**Date**: 2026-05-16 +**Status**: Approved (all 7 sections + ISV-driven auto-tuning addition) +**Supersedes**: `2026-05-16-alpha-ppo-trainer.md` (the earlier DQN-port plan, abandoned per user direction to rebuild from first principles) + +## TL;DR + +A clean greenfield rebuild of `ml-alpha` as a full closed-loop alpha trading system for ES.FUT on IBKR with $35k starting capital. Three components: (1) a Rust model library that owns a CfC perception trunk + 5 multi-horizon alpha heads + a CfC actor-critic policy, (2) an offline training binary, (3) an `AlphaPpoStrategy` integration in the existing `trading_agent_service` for live runtime. Everything GPU-resident on the hot path. Online learning with EWC anchoring + replay buffer (80% offline / 20% live) + kill-switch on σ-band metric deviations. Hyperparameters that vary during training are ISV-driven (signal-driven controllers, no magic numbers). Reuse the existing IBKR adapter, data ingestion, execution engine, and risk manager. + +## Why this design exists + +The session that led here started with the DQN smoke baseline (`alpha_baseline.rs`) running at per-bar trade frequency, then discovered via stride sweep that **the framing was the bug**: decision cadence at per-bar level was coin-flipping on noise. Switching to `decision_stride=200` with scaled training flipped 3-fold mean Sharpe at quarter-tick cost from **-4.29 (per-bar) to +1.78 (stride=200 + scaled)**, with variance collapsing from ±8.8 to ±1.15. + +That validated the alpha signal at IBKR-realistic costs but exposed that `alpha_baseline.rs` was a smoke trainer — not the right vehicle for production. The first attempt to port the production DQN trainer (`gpu_dqn_trainer.rs`, 1.9 MB) was abandoned because (a) DQN's per-bar Bellman target is structurally wrong for minute-horizon alpha, (b) we have no validation that the existing DQN architecture fits this problem, (c) porting DQN means inheriting DQN-shaped assumptions. User direction: rebuild from first principles, no backward-compat, no transitional layers — "consistent in development without endless gates or backward compats." + +## Architecture summary (one paragraph) + +A snapshot-level CfC perception trunk (Closed-form Continuous-time recurrent network, ~128 cells, each with its own learnable time constant) reads Databento MBP-10 snapshots and emits 5 multi-horizon direction probabilities (P(up at h) for h ∈ {30, 100, 300, 1000, 6000} snapshots forward). A small CfC actor-critic policy reads a 20-dimensional state (5 alpha probs + 8-dim perception projection + position state + spread / vol / time features) and outputs a target position categorical over {-10, ..., 0, ..., +10}. The policy fires every `decision_stride=50` snapshots (~1 minute). Trades happen when `target ≠ previous_target`. Reward per decision step is dense per-segment P&L minus IBKR commission minus optional vol penalty. The model is fully GPU-resident on the hot path. Offline training is two-phase: (A) supervised multi-horizon BCE on the perception, with a hard validation gate requiring CfC AUC ≥ existing Mamba2 baseline before continuing, and (B) PPO on the frozen perception with chronological walk-forward CV over 6 folds spanning 9 quarters of historical data. Live runtime reuses `data_acquisition_service` (Databento streaming already built), `trading_agent_service` (where a new `AlphaPpoStrategy` plugs in), `trading_service::execution_engine` + `risk_manager`, and `crates/trading_engine/src/brokers/interactive_brokers.rs` (the IBKR adapter, 1177 LOC, already exists). Online updates run every 4 hours or 10K live decisions, applying PPO with an EWC term anchored to the offline-trained baseline, drawing 80% from a frozen offline replay buffer + 20% from live trajectories, with a kill-switch that rolls back weights if any of {loss, mean reward, hit-rate, KL divergence, entropy} deviates outside σ-band thresholds. + +--- + +## Section 1 — System scope and integration + +### Component boundaries + +| | Where it lives | Role | Approx new LOC | +|---|---|---|---| +| `ml-alpha` Rust library | `crates/ml-alpha/` (gut existing files, rebuild clean) | CfC perception + heads + CfC policy + EWC anchor + Adam state + replay buffer types + training kernels | ~3000 | +| `alpha_train` binary | `crates/ml-alpha/examples/alpha_train.rs` | Offline training entry: Phase A (perception pretrain), then Phase B (PPO) per fold | ~300 | +| `AlphaPpoStrategy` | `services/trading_agent_service/src/strategies/alpha_ppo.rs` (NEW) | Live runtime adapter: owns `AlphaModel` + `ReplayBuffer`, called per snapshot, emits target signals | ~250 | + +### What gets reused (no new work) + +| Component | Existing path | +|---|---| +| Databento streaming | `crates/data/src/providers/databento_streaming.rs` | +| Live MBP-10 ingestion | `services/data_acquisition_service/` | +| Order execution + routing | `services/trading_service/src/core/execution_engine.rs`, `broker_routing.rs`, `order_manager.rs`, `position_manager.rs` | +| Risk manager (limits, kill-switch hook) | `services/trading_service/src/core/risk_manager.rs` | +| Broker gateway | `services/broker_gateway_service/` | +| IBKR adapter (1177 LOC, `ibapi` v10.19+) | `crates/trading_engine/src/brokers/interactive_brokers.rs` | + +### Public API of the model library + +```rust +// crates/ml-alpha/src/lib.rs (public surface — total ~5 types, ~10 methods) + +pub struct AlphaModel { /* CfC weights + heads + policy + EWC anchor + Adam state, all GPU-resident */ } + +impl AlphaModel { + pub fn load(weights_path: &Path) -> Result; + pub fn save(&self, weights_path: &Path) -> Result<()>; + + // Hot path — called per snapshot during live. Target latency p99 ≤ 500μs. + pub fn perception_forward(&mut self, snapshot_pinned: &MappedPinnedSnapshot) -> AlphaProbsDevicePtr; + + // Decision point — called every decision_stride snapshots. + pub fn policy_action(&self, state_gpu: &AlphaStateGpu) -> (Action, LogProb, Value); + + // Slow path — periodic online update; ~30s wall on L40S. + pub fn update(&mut self, batch: &MiniBatch) -> UpdateStatus; +} + +pub struct ReplayBuffer { /* offline + live trajectories, GPU-resident */ } + +impl ReplayBuffer { + pub fn record_step(&mut self, transition: Transition); + pub fn complete_last_segment_reward(&mut self, reward: f32); + pub fn sample_mixed(&self, n: usize, offline_frac: f32) -> MiniBatch; + pub fn load_offline_from(&mut self, path: &Path) -> Result<()>; +} + +pub enum UpdateStatus { + Applied { metrics: TrainingMetrics }, + RolledBack { reason: String, deviating_metric: String, sigma: f32 }, +} + +pub struct AlphaState { /* 20-dim vector, constructed by trading_agent */ } + +#[repr(i8)] +pub enum Action { /* target_position ∈ {-10, ..., +10} */ } +``` + +GPU constraint: weights, hidden states, replay buffer, Adam state, and EWC anchor all live on the GPU. CPU only touches: snapshot bytes from Databento socket (via mapped-pinned), the chosen Action enum (~1 byte), the UpdateStatus enum, and serialized weights at save/load. No CPU shadow buffers. No GPU→CPU roundtrips on the hot path. Mirrors the foxhunt rules: `feedback_cpu_is_read_only.md`, `feedback_no_htod_htoh_only_mapped_pinned.md`, `feedback_no_atomicadd.md`, `feedback_no_nvrtc.md`. + +--- + +## Section 2 — Model architecture + +### Snapshot feature vector (input to CfC perception, ~32 dims) + +Per MBP-10 snapshot, build a fixed-shape feature vector for the CfC trunk. Built on GPU once when the snapshot lands (via mapped-pinned), reused across all downstream calls. + +| Index | Field | Source | Notes | +|---|---|---|---| +| 0 | mid log-return since prev snapshot | (mid_t − mid_{t-1}) / mid_{t-1}, then log | Direction signal | +| 1 | L1 spread in ticks | (best_ask − best_bid) / tick_size | Liquidity signal | +| 2–6 | Bid-side per-level size (L1..L5) | Normalized log-size | Top-of-book depth | +| 7–11 | Ask-side per-level size (L1..L5) | Normalized log-size | Top-of-book depth | +| 12–16 | Per-level OFI (L1..L5, signed) | Standard OFI from `ml_features::ofi_calculator` | Order flow imbalance | +| 17 | Trade arrival count since prev snapshot | Integer count, log-scaled | Trade-flow proxy | +| 18 | Trade signed volume since prev snapshot | Sum(buy_size) − Sum(sell_size), normalized | Trade-flow direction | +| 19 | Snapshot inter-arrival time (Δt in ms) | From MBP-10 timestamp | Already used by CfC for time-constant integration; included here for the input network too | +| 20–31 | Reserved for v2 (Hawkes intensity, microprice deviation, multi-level imbalances, etc.) | Initially zero | Architectural headroom | + +L5 depth chosen (not full L10) to keep the input dim modest. L6-L10 are rarely informative for ES.FUT given liquidity is concentrated in L1-L3. Reserved slots 20-31 allow extension without retraining if v2 features prove necessary. + +### Perception trunk (CfC, snapshot-level, stateful) + +Closed-form Continuous-time recurrent network (Hasani et al. 2022). The practical trainable variant of LTC — replaces ODE solve with a closed-form recurrence, ~50× faster than naive LTC, preserves the "each neuron has its own time-constant" property. + +- 128 CfC cells in the perception trunk +- Each cell has its own learnable time constant `τ_i`. Time-constant diversity gives multi-horizon natively: some cells specialize to fast dynamics, others to slow. +- Forward equation per snapshot: + ``` + h_new_i = h_old_i · exp(-Δt / τ_i) + + (1 − exp(-Δt / τ_i)) · tanh(W_in_i · snap_features + W_rec_i · h_old + b_i) + ``` + where `Δt` is the actual nanoseconds-since-last-snapshot from MBP-10 timestamps. Irregular timestamps are native — no aggregation or padding to fixed bar intervals. +- Hidden state `h ∈ ℝ^128` lives on device, persists across snapshots. During training: one state per parallel env. During live: single state, persists across calls (re-initialized only at startup or after explicit reset). + +### Multi-horizon alpha heads (5 thin linear projections) + +For each `h ∈ {30, 100, 300, 1000, 6000}` snapshots forward: a `128 → 1` linear layer + sigmoid produces P(up at h). Trained jointly with the trunk via multi-horizon BCE loss. + +### Projection layer (perception → policy state) + +A learned `W_proj ∈ ℝ^(8×128)` linear layer projects the perception hidden state to an 8-dim summary that flows into the policy state. Trained jointly with perception (so the projection learns what the policy needs beyond just the explicit alpha probabilities). + +### Policy network (CfC actor-critic) + +| Component | Architecture | +|---|---| +| Actor | 20-dim state → CfC(64) → linear → 21 logits (target position ∈ {-10, ..., +10}) | +| Critic | 20-dim state → CfC(64) → linear → 1 scalar (V(s)) | + +CfC time-constants in the policy let the policy "remember" recent context across decision steps. Useful because the policy fires every `decision_stride=50` snapshots, and the gap between calls is meaningful state. + +### EWC anchor + +- Stores baseline weights `θ_anchor` (copy of model weights after offline training) +- Stores diagonal Fisher information `F_i = E_val[(∂L_PPO / ∂θ_i)²]` (one scalar per parameter) +- Online updates add a penalty term: `λ_ewc · Σ_i F_i · (θ_current_i − θ_anchor_i)²` +- `θ_anchor` is frozen — only updated by explicit operator action + +### Parameter count + +| Component | Params | +|---|---| +| CfC perception trunk (128 cells, ~20 input dims) | ~20K | +| Multi-horizon heads (5 × 128 → 1) | ~700 | +| Projection (128 → 8) | ~1K | +| CfC actor (20 → 64 → 21) | ~6K | +| CfC critic (20 → 64 → 1) | ~5K | +| EWC anchor + Fisher diagonal | 2× weight count | +| AdamW optimizer state | 2× weight count | +| **Total trainable weights** | **~35K** | +| **Total device memory (weights + state + optimizer)** | **~1 MB** | + +Inference is microseconds per snapshot. Training is minutes per fold even on consumer GPU. + +--- + +## Section 3 — Action space, reward, state + +### Action space + +Single categorical head, target_position ∈ {-10, -9, ..., 0, ..., +9, +10} (21 outputs). + +- Architecture is fixed at `max_train = 10` (21 outputs always). +- Live inference takes runtime parameter `max_live ≤ max_train`. Logits outside `[-max_live, +max_live]` are set to `-inf` on-device before softmax; sample or argmax operates on the remaining live range. Zero retraining cost to dial down at deployment. +- Trade-event semantics are derived: `trade_happens = (target_t ≠ target_{t-1})`. + +### Reward (dense per-segment, all units in PRICE UNITS, not dollars) + +``` +r_t = ΔPnL_segment_t ← P&L over the decision_stride window (50 snapshots) + − C_trade · |target_t − target_{t-1}| ← IBKR commission per contract changed + − C_vol · max(0, |segment_return_t| − r_threshold) ← vol penalty (default coef 0) +``` + +All quantities in **price units of the underlying (ES.FUT points)**, not dollars. Conversion: $ amount = price units × $50/point × contracts. Keeping the reward in price units makes it position-size invariant: a 1-point move is the same reward regardless of how many contracts are held; the policy already accounts for `target` magnitude. Dollar P&L is computed only for risk monitoring (daily loss limit, margin) — never in the reward function. + +`ΔPnL_segment_t` formal definition: +``` +ΔPnL_segment_t = target_t · (mid_close_of_segment − mid_open_of_segment) +``` +where `mid_open_of_segment` is the mid-price at the snapshot at which the segment started (the previous decision), and `mid_close_of_segment` is the mid at the current decision snapshot. + +Constants: +- `C_trade = 0.034` price units per contract (IBKR ES: $0.85/side × 2 = $1.70 RT ÷ $50/point = 0.034) +- `C_vol = 0.0` initially (vol penalty disabled; ISV-driven controller can ramp up if vol becomes a problem — see Section 7) +- `r_threshold = 0.25` (1 tick — first tick treated as expected noise) + +### State vector (20 floats, GPU-resident) + +| Index | Field | Source | Normalization | +|---|---|---|---| +| 0–4 | 5 multi-horizon alpha probs (h=30/100/300/1000/6000) | Perception heads (sigmoid) | [0, 1] | +| 5–12 | 8-dim projection of perception hidden | Learned `W_proj: 128 → 8` | ~unit-variance from layer-norm | +| 13 | Current target position | Strategy state | `current / max_train` ∈ [-1, +1] | +| 14 | Unrealized P&L (in ticks since entry) | Strategy state | clipped to [-20, +20], scaled by /20 | +| 15 | Time-since-entry (decision steps) | Strategy state | clipped to [0, 100], scaled by /100 | +| 16, 17 | Minute-of-day sin, cos | Wall clock | [-1, +1] | +| 18 | Spread quantile (rolling 500-bar) | Live computation | [0, 1] | +| 19 | Recent realized vol (rolling 100-bar log-return std) | Live computation | scaled by historical 95th percentile | + +--- + +## Section 3.5 — Capital envelope and IBKR broker constraints + +### Capital math + +Initial capital: **$35,000**. ES.FUT contract value ≈ $50/point × ~5500 points ≈ $275k notional per contract. + +IBKR margin requirements (subject to broker discretion): +- Day-trading initial margin: ~$500-1,500/contract +- Overnight initial margin: ~$13,200-15,000/contract (exchange minimum ~$11,500 plus IBKR cushion) +- Maintenance margin (intraday): ~$1,000-1,500/contract + +Conservative `max_live` ramp plan: + +| Phase | `max_live` | Notional exposure | Trigger | +|---|---|---|---| +| Initial deployment | **1 contract** | ~7.8× leverage | Day-1 | +| After 100+ trades with positive realized Sharpe | 2 | ~16× leverage | Manual config bump | +| After 500+ trades, ≥3-month positive track record | 3 | ~23× leverage | Manual config bump | +| Beyond 3 | — | — | Requires more capital or better margin tier | + +The model is always trained with `max_train = 10` (network output dim). The live cap grows by manual config bump only — never automatically. + +### IBKR quirks (the "tricks" to design around) + +| # | Quirk | Required design response | +|---|---|---| +| 1 | Pattern Day Trader rule: ≥4 day-trades in 5 biz days requires ≥$25k equity | Monitor day-trade count; halt strategy if equity drops near $25k threshold to avoid disqualification | +| 2 | Margin can be raised mid-session by IBKR risk dept | Pre-trade `whatif` margin check via IBKR API before submit; reject if margin insufficient | +| 3 | Liquidation if margin breach: IBKR force-closes at market | Maintain ≥40% margin cushion; if margin utilization > 60%, force flat *before* IBKR does | +| 4 | Order rejection without warning | Robust order-status state machine; treat rejection as halt-and-alert; manual ack to resume | +| 5 | Contract rollover: ES quarterly expiry (Mar/Jun/Sep/Dec) | Auto-detect rollover via front-month volume; flat positions in expiring contract 5 days before expiry; open in next month | +| 6 | Market halts (LULDs, circuit breakers) | Subscribe to TWS halt status; force-cancel pending orders; log + alert if positioned | +| 7 | API rate limits (>50 messages/sec) | Bound order submission rate; coalesce target changes within decision_stride | +| 8 | Commission tier (Pro vs Lite, Smart vs direct routing) | Use Smart Routing for ES; pin `$0.85/side` as assumed commission | +| 9 | Symbol drift on rollover (ESM6 → ESU6) | Symbol-lookup layer; never hardcode contract month | +| 10 | Daily loss circuit by IBKR | Match internal threshold; trigger flat earlier than IBKR would | + +### Pre-deployment IBKR audit tasks + +The existing `crates/trading_engine/src/brokers/interactive_brokers.rs` (1177 LOC, `ibapi` crate) needs an audit before go-live: +1. Verify it handles ES futures specifically (the adapter is built but may be stock/option focused) +2. Confirm `whatif` pre-trade margin check is exposed +3. Confirm rollover detection and symbol-update logic exists +4. Confirm halt-status subscription +5. Audit rejection-code catalog coverage +6. Check PDT counter integration with `risk_manager` + +--- + +## Section 4 — Training pipeline + +### Phase A: Perception pretrain (supervised, multi-horizon BCE) + +**Data path:** +- Source: predecoded MBP-10 sidecars from the PVC (already built earlier this session). 61M snapshots / 9Q. +- Loader: zero-copy mmap of sidecars → mapped-pinned device staging buffer → CfC reads device memory directly. No CPU shadow. +- Sample shape: sequences of `seq_len = 64` consecutive snapshots, anchored at random valid positions. +- Labels: multi-horizon labels at `{30, 100, 300, 1000, 6000}` snapshots forward, computed once during data pipeline init. + +**Training step (per minibatch):** +1. Sample `batch_size = 256` sequences. Upload bar indices to device via mapped-pinned. +2. CfC perception forward over `seq_len = 64`, producing 128-dim hidden + 8-dim projection at every position. +3. Multi-horizon heads emit 5 logits per position → sigmoid. +4. Loss = `(1/5) · Σ_h BCE(σ(logit_h), label_h)` averaged over valid positions. +5. Backward → AdamW step. + +**Defaults:** +- Optimizer: AdamW, lr=3e-3, weight_decay=1e-2 +- Epochs: 5 over the training window +- Time-constant init: log-uniform over [10ms, 1000s] + +### Validation gate: CfC must meet Mamba2 + +After Phase A on the first 2Q (Q1-Q2 2024): +- Measure val AUC at each of the 5 horizons on Q2 (held out from training) +- Reference: Mamba2 stacker baseline AUC ≈ 0.66-0.71 at K=6000 from prior session runs +- **Gate**: CfC AUC ≥ Mamba2 AUC at every horizon (within 0.01 tolerance). If not met, pause and investigate; do not proceed to Phase B. + +If the gate fails, the revert path is clean: the perception layer is the only component that depends on CfC. A revert to Mamba2 trunk (existing code, validated) leaves the policy, env, reward, online update, and runtime integration untouched. + +### Phase B: PPO policy training on frozen perception + +**Data path:** +- Perception is frozen; its forward outputs (5 alpha probs + 8-dim projection) are computed once per snapshot and cached in a device tensor. +- Iterated in temporal order — we're running an env, not sampling. + +**Rollout collection (on-device):** +- `n_par = 64` parallel envs. Each env maintains (current_position, current_pnl, time_since_entry, last_trade_pos) on device. +- Step loop: every snapshot, perception forward (frozen). Every `decision_stride = 50` snapshots, build AlphaState, policy forward, sample action, compute reward over the 50-snapshot segment, store transition. +- After `rollout_length = 2048` decisions per env: stop collecting, run PPO updates. + +**PPO update:** +1. Compute GAE: γ=0.99, λ=0.95 (both fixed) +2. K=4 PPO epochs, minibatches of 256 decisions +3. Per minibatch: PPO clip loss + value MSE + entropy bonus + EWC term; AdamW step +4. KL early-stop within K epochs if `KL(current || rollout) > 0.02` + +### Walk-forward CV (chronological, 6 folds, expanding window) + +| Fold | Train | Val (select within-fold hyperparams) | Test (held-out) | +|---|---|---|---| +| 1 | Q1-Q4 2024 | Q1 2025 | Q2 2025 | +| 2 | Q1 2024 - Q1 2025 | Q2 2025 | Q3 2025 | +| 3 | Q1 2024 - Q2 2025 | Q3 2025 | Q4 2025 | +| 4 | Q1 2024 - Q3 2025 | Q4 2025 | Q1 2026 | +| 5 | Q1 2024 - Q4 2025 | Q1 2026 | (no test — future) | +| 6 | All 9Q | (final calibration) | (live) | + +Fold 4's test (Q1 2026) is true forward OOS. Reported test Sharpe is the mean of folds 1-4 test segments, with std as the uncertainty band. + +--- + +## Section 5 — Live runtime integration + +### Data flow at runtime + +``` +data_acquisition_service trading_agent_service trading_service + Databento stream ──→ snapshot event ──→ AlphaPpoStrategy: ──→ emit target signal + 1. mapped-pinned snapshot ──→ risk_manager (limits, kill-switch, PDT, margin) + 2. model.perception_forward (GPU) ──→ execution_engine + 3. every 50 snaps: model.policy ──→ broker_routing → IBKR adapter + 4. build Action ↓ + 5. buffer.record_step broker_gateway_service + ↓ + IBKR TWS / Gateway +``` + +### AlphaPpoStrategy lifecycle (sketch) + +```rust +pub struct AlphaPpoStrategy { + model: AlphaModel, // CfC + policy + EWC anchor + Adam, all GPU + buffer: ReplayBuffer, // offline + live trajectories, GPU + snap_pinned: MappedPinnedSnapshot, // CPU→GPU staging for one snapshot + decision_counter: usize, + last_target: i8, + last_state_gpu: AlphaStateGpu, + config: AlphaConfig, // max_live, decision_stride, update_cadence, ... + metrics: AlphaMetrics, + update_clock: UpdateClock, +} + +impl Strategy for AlphaPpoStrategy { + fn on_snapshot(&mut self, snap: &Mbp10Snapshot) -> Option { + self.snap_pinned.write_volatile(snap); + self.model.perception_forward(&self.snap_pinned); + self.decision_counter += 1; + if self.decision_counter % self.config.decision_stride != 0 { return None; } + + self.build_state_gpu(snap); + let (action, log_prob, value) = self.model.policy_action(&self.last_state_gpu); + let target = action.clamp_to(self.config.max_live); + let signal = (target != self.last_target).then(|| Signal::SetTargetPosition { contracts: target }); + + self.buffer.record_step(Transition { state: self.last_state_gpu.clone(), action: target, log_prob, value }); + self.last_target = target; + + if self.update_clock.tick() { + let batch = self.buffer.sample_mixed(8192, /* offline_frac */ 0.8); + match self.model.update(&batch) { + UpdateStatus::Applied { metrics } => self.metrics.update_applied(metrics), + UpdateStatus::RolledBack { reason, .. } => { + self.metrics.update_rolled_back(&reason); + self.config.online_updates_enabled = false; // auto-disable; manual ack required + tracing::error!(reason, "online update rolled back; auto-updates DISABLED"); + } + } + } + signal + } + + fn on_fill(&mut self, fill: &Fill) { + self.update_pnl_and_position(fill); + self.buffer.complete_last_segment_reward(self.pnl_delta(fill)); + } +} +``` + +### Persistence and restart semantics + +| Element | Persistence | Restart behavior | +|---|---|---| +| Model weights | Atomic write to `/data/ml-alpha/weights-.bin` on every successful `update()` | Load latest at startup | +| Perception hidden state | Not persisted (v1) | Cold-start: zeros; emit flat signal for first 60s while CfC saturates | +| Replay buffer | Live trajectories written to QuestDB; offline trajectories loaded from PVC | Rebuild: load offline + replay live from QuestDB since last weight snapshot | +| Position state | Owned by `trading_service::position_manager` | Reconcile with broker on startup | +| EWC anchor + Fisher matrix | Saved alongside weights | Loaded with weights | + +### Observability + +- Per snapshot: latency p50/p99 (Prometheus histogram) +- Per decision: action sampled, alpha probs at each horizon (QuestDB row) +- Per fill: realized P&L, position change, commission (QuestDB row) +- Per update: training metrics, kill-switch margin (Prometheus) +- Alerts: kill-switch fired, daily loss limit hit, margin breach, IBKR rejection + +--- + +## Section 6 — Online learning mechanics + +### EWC anchor — what gets frozen, what gets learned + +After Phase B finishes on the training window: +1. Take trained weights `θ_anchor` and freeze them as reference. +2. Compute diagonal Fisher information `F_i = E_val[(∂L_PPO / ∂θ_i)²]` per parameter. +3. Persist `(θ_anchor, F)` to disk alongside live weights. + +During online updates: +``` +loss = loss_PPO + λ_ewc · Σ_i F_i · (θ_current_i − θ_anchor_i)² +``` +`λ_ewc` is the conservatism dial — initial value 1e3 but driven by a controller during operation (see Section 7). + +### Replay buffer composition + +| Component | Source | Fraction | Lifecycle | +|---|---|---|---| +| Offline trajectories | Saved during Phase B (last ~3 training quarters, ~50K decisions) | 80% (controller-tunable) | Loaded once at startup; GPU-resident; never updated | +| Live trajectories | Collected by `AlphaPpoStrategy::on_snapshot` | 20% (controller-tunable) | Ring buffer of ~10K most recent live decisions; mirror written to QuestDB | + +Each minibatch: 8192 transitions = 6553 offline + 1638 live (at default 80/20). + +### Update cadence + +Whichever fires first: +- Time-based: every `update_interval_secs = 14_400` (4 hours) +- Volume-based: every `update_interval_transitions = 10_000` live decisions + +The update lasts ~30s on L40S; `policy_action` calls block momentarily during atomic weights swap (~10ms typical). + +### Atomic weights swap with kill-switch + +```rust +fn update(&mut self, batch: &MiniBatch) -> UpdateStatus { + let snapshot_path = self.write_weights_snapshot()?; // versioned snapshot first + + let candidate_weights = self.weights.clone_gpu(); // GPU-side copy + let metrics = self.run_ppo_epochs_on(&candidate_weights, batch); + + match self.kill_switch.check(&metrics) { + KillSwitchVerdict::Ok => { + std::mem::swap(&mut self.weights, &mut candidate_weights); + self.kill_switch.record(metrics.clone()); + UpdateStatus::Applied { metrics } + } + KillSwitchVerdict::Reject { reason, deviating_metric, sigma } => { + // candidate_weights dropped — live weights untouched + UpdateStatus::RolledBack { reason, deviating_metric, sigma } + } + } +} +``` + +### Kill-switch criteria + +Rolling baseline tracked from last 20 successful updates. + +| Metric | Reject condition | Why | +|---|---|---| +| `loss_ppo` | > baseline_mean + 3σ | Training divergence | +| `mean_reward_per_decision` | < baseline_mean − 2σ | Strategy degrading | +| `hit_rate` | absolute < 0.45 | Below empirical floor | +| `kl_divergence(candidate \|\| anchor)` | > 0.1 | Policy drifting; EWC has failed | +| `entropy` | < 0.1 nats | Policy collapse (deterministic) | + +If any fires: weights stay at pre-update state, `online_updates_enabled = false` in config, operator alert via Prometheus + log. Manual ack required to re-enable. + +### Snapshot retention + +Every successful update creates `weights--.bin` (~150 KB). Retain rolling last 100. Rollback CLI: `alpha-control rollback --to `. + +The anchor `(θ_anchor, F)` is FROZEN automatically — only updated by explicit operator command after a multi-month track record. + +--- + +## Section 7 — Testing strategy, ISV-driven hyperparameters, risks + +### Testing tiers (in order of cost / fidelity) + +| Tier | What | When | Pass criterion | +|---|---|---|---| +| Unit tests | CfC cell, EWC loss, GAE, PPO clip, reward formula, action masking, replay buffer | Continuously | All pass | +| Bit-equiv | CfC forward CPU vs GPU on small inputs | Per commit | < 1e-5 relative error | +| Smoke training | Phase A perception on 1Q subset | Before each merge | val AUC > 0.55 at K=6000 | +| **CfC vs Mamba2 gate** | Phase A perception on 2Q | Before Phase B | CfC AUC ≥ Mamba2 AUC at every horizon | +| Walk-forward Sharpe | 6-fold CV with PPO on 9Q | Before paper trading | Mean test Sharpe > 0 at IBKR commission; std < mean | +| **Paper trading** | Live MBP-10 + simulated execution (no broker orders); 5 trading days | Before live capital | Live P&L tracks backtest ±15%; no kill-switch fires; latency p99 < 500μs | +| Live with $1k cap | Real money, `max_live=1`, manual kill armed, 5-day window | Before scaling | Realized Sharpe > 0; zero IBKR rejections; rollover clean if it occurs | + +### ISV-driven hyperparameters (per `pearl_controller_anchors_isv_driven.md`) + +Most "hyperparameters" are controller outputs, not config constants. The ml-alpha trainer gets its own ISV bus (~32 slots). + +#### Controller-driven (signal-driven adaptation) + +| Hyperparam | Controller signal | Update rule | +|---|---|---| +| `entropy_coef` | Entropy deficit = (target_entropy − current_entropy), target = 70% of uniform | `entropy_coef ← entropy_coef · exp(η · deficit)` | +| `lr_policy` | Loss-slope EMA + grad-norm stability | Wiener-α blend toward target_loss_slope = -ε per epoch; floor at 1e-5 | +| `λ_ewc` | Weight drift magnitude `‖θ − θ_anchor‖²` vs target | Raise λ if drift > 1.2× target; lower if < 0.5× target | +| `clip_eps` (PPO) | KL divergence trajectory; target KL = 0.01 | Raise clip if KL too low (under-update); lower if too high (over-update) | +| `value_coef` | Ratio of value loss to policy loss; target = 1:5 | Adjust value_coef to match target ratio | +| Replay offline/live mix | Reward gap = `live_reward_mean − offline_reward_mean` | Increase live fraction if gap > Nσ (regime shift signal); decrease if gap < 0 | +| `C_vol` (vol penalty) | Variance of returns vs target | Raise coef if return std > target × 1.5; keep at 0 otherwise | + +All controllers use the standard foxhunt pattern: Wiener-α optimal blending + permanent floor (per `pearl_wiener_alpha_floor_for_nonstationary.md` and `pearl_blend_formulas_must_have_permanent_floor.md`). + +#### Fixed parameters (genuinely architectural) + +| Parameter | Value | Why constant | +|---|---|---| +| `decision_stride` | 50 snapshots | Architecture choice; changing requires retraining | +| `seq_len` (perception) | 64 snapshots | Architecture choice | +| `n_horizons` | 5 (30/100/300/1000/6000) | Architecture choice | +| `max_train` | 10 contracts | Architecture choice (network output dim) | +| `n_par` (parallel envs) | 64 | Hardware fit | +| `batch_size` (perception) | 256 sequences | Hardware fit | +| `batch_size` (PPO update) | 8192 | Hardware fit | +| GAE γ | 0.99 | Standard; controller would risk subtle bugs | +| GAE λ | 0.95 | Standard | +| K (PPO epochs) | 4 | KL early-stop within K catches over-update | +| Kill-switch σ thresholds | 3σ loss, 2σ reward | Safety floor — operator-controlled, not signal-driven | + +#### ISV slot allocation (32 slots) + +``` +ml-alpha ISV bus +───────────────────────────────── + 0..5 Perception output: 5 alpha probs at h={30,100,300,1000,6000} + 5..13 Perception hidden projection (8 dims) + + 13..18 Training diagnostics: + 13: loss_ppo EMA + 14: kl_divergence (vs anchor) + 15: entropy_current + 16: grad_norm EMA + 17: weight_drift |θ − θ_anchor|² + + 18..24 Controller outputs (effective hyperparams): + 18: lr_policy_eff + 19: lr_perception_eff + 20: entropy_coef_eff + 21: clip_eps_eff + 22: value_coef_eff + 23: λ_ewc_eff + + 24..28 Replay buffer state: + 24: mean_live_reward EMA + 25: mean_offline_reward EMA + 26: live_offline_mix_eff (0..1) + 27: buffer_fill_fraction + + 28..32 Operational / live: + 28: current_pos_eff + 29: realized_pnl_session_eff + 30: kill_switch_margin (σ-distance from rolling baseline) + 31: rolling_baseline_idx +``` + +### Known risks + +| Risk | Severity | Mitigation | +|---|---|---| +| CfC fails to match Mamba2 | High | Validation gate stops Phase B; perception layer is replaceable in isolation; revert to Mamba2 if needed | +| PPO entropy collapse | Medium | Kill-switch fires on entropy < 0.1; entropy_coef controller raises automatically | +| Replay buffer staleness | Medium | 80% offline data is from fixed historical window; if regime shifts dramatically the offline anchor is obsolete. Manual remix or new offline trajectory generation needed periodically | +| IBKR margin spike on FOMC | Medium | Pre-trade `whatif` rejects oversized orders; daily loss limit catches systemic damage | +| Snapshot CfC compute slower than 500μs | Medium | Profile early; if perception forward > 100μs per snapshot may need TensorRT export for live (not training) | +| Decision-stride / horizon mismatch | Low | Decision_stride=50 << longest horizon 6000; plenty of decision points. Verified this session | +| EWC λ mis-tuning | Low | Now controller-driven, not hardcoded | +| Contract rollover bugs | Low (but high impact) | Existing trading_engine handles symbol changes; audit before go-live | + +### What this design does NOT include + +- HFT / sub-millisecond latency (target ≤500μs per snapshot, not nanoseconds) +- Cross-asset universe expansion (ES.FUT only) +- Hedging logic (outright ES only) +- Options strategies +- Hyperparameter sweep over architecture (the architecture itself — number of CfC cells, horizons, decision_stride — is fixed) +- Auto-rollback of `θ_anchor` (anchor moves only by explicit operator action) +- Time-of-day filter (overnight trading allowed; operator-controlled risk) + +--- + +## Implementation order + +1. **IBKR adapter audit** (1-2 days) — verify ES futures, `whatif`, rollover, halt, PDT counter +2. **ml-alpha library scaffold + ISV slot allocation** (1 day) +3. **CfC perception forward + multi-horizon heads + Phase A trainer** (3-4 days) +4. **Phase A run + CfC-vs-Mamba2 validation gate on 2Q** (1 day cluster) — GATE +5. **CfC policy (actor-critic) + reward + state construction** (2-3 days) +6. **PPO loss + GAE + replay buffer + EWC term + Phase B trainer** (3-4 days) +7. **ISV controllers (entropy, lr, λ_ewc, clip_eps, value_coef, replay mix, C_vol)** (2-3 days) +8. **Online update + kill-switch + atomic swap** (2 days) +9. **AlphaPpoStrategy integration in trading_agent_service** (2-3 days) +10. **Walk-forward CV (6 folds, full 9Q) on cluster** (1-2 days, mostly cluster wall) +11. **Paper trading on live MBP-10 + simulated execution (5 day window)** +12. **Live deploy with `max_live=1` (5 day window)** +13. **Ramp `max_live` per the table in Section 3.5** + +Estimated end-to-end timeline to live: **~3-4 weeks of focused work**, including paper trading. + +## Appendix A — Memory entries this design embodies + +- `feedback_cpu_is_read_only.md` — no CPU compute, GPU is sole arithmetic engine +- `feedback_no_htod_htoh_only_mapped_pinned.md` — snapshot bytes via mapped-pinned, no HtoD +- `feedback_no_atomicadd.md` — block tree-reduce only (relevant for the multi-horizon BCE + EWC term kernels) +- `feedback_no_nvrtc.md` — pre-compiled cubins for any CUDA kernels +- `pearl_controller_anchors_isv_driven.md` — all tunables ISV-controlled +- `feedback_isv_for_adaptive_bounds.md` — bounds live in ISV +- `feedback_adaptive_not_tuned.md` — fixes are signal-driven, not tuned constants +- `pearl_wiener_alpha_floor_for_nonstationary.md` — floor of 0.4 on Wiener-α in non-stationary loops +- `pearl_blend_formulas_must_have_permanent_floor.md` — max-with-floor, not blend +- `pearl_first_observation_bootstrap.md` — EMAs bootstrap from sentinel = 0; first observation replaces directly +- `pearl_bar_abstraction_can_be_alpha_ceiling.md` — bar aggregation can cap alpha; snapshot-level chosen for this reason +- `pearl_state_amplifies_short_horizon_into_long_horizon.md` — stateful encoder (Mamba2/CfC) gives multi-minute alpha from snapshot input +- `pearl_phase1d4_backtest_cost_edge_frontier.md` — frictionless Sharpe ~4.4 baseline; break-even quarter-tick; this session pushed to ~1-tick at stride=200 + +## Appendix B — Today's session findings that motivated this design + +- Per-bar trading = coin-flip on noise. Decision-stride 200 with scaled training flipped Fold A Sharpe at quarter-tick cost from -3.7 to +2.52; 3-fold mean from -4.29 to +0.78; with 16× scaled training to +1.78. The signal exists at minute-horizon; the framing of "decide every bar" was the bug. +- Variance across folds collapsed from ±8.8 (per-bar) to ±0.80 (stride=200) — removing noise-mining made the alpha consistent. +- IBKR commission at $1.70 RT translates to 0.034 price units — well below the new break-even of ~1 tick. Economic deployment is plausible. +- `alpha_baseline.rs` was a smoke trainer; the production DQN trainer (`gpu_dqn_trainer.rs`) is engineered for a different problem regime (per-bar Bellman, branched discrete actions); porting it would inherit the wrong shape. Greenfield rebuild with PPO + multi-horizon perception + CfC is the right move.