# Layer C: Regime Gate — Trade Only In Alpha-Dense Book States

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task.

**Goal:** Lift trade WR from current ~22% baseline by gating trade entry on book-state regime (spread quintile). Per `pearl_snapshot_alpha_is_regime_conditional`, alpha concentrates in ~20% of book states; spread-Q4 hits 75% accuracy unconditionally. Layer C does NOT depend on aux supervision — directly attacks WR by filtering out the 80% of states where the model has no edge.

**Architecture:** Two-task pipeline:
1. **C1 — NumPy investigation** (no compute) — empirically determine which spread quintile(s) have net-of-cost positive expected PnL on the actual ES data the cluster trains on. Decides the gate configuration.
2. **C2 — Decision policy gate** — implement spread-quintile gate in `crates/ml-backtesting/cuda/decision_policy.cu` using ISV-driven trailing-window percentiles (P-square estimator).
3. **C3 — Argo Smoke validation** — backtest with gate active using existing checkpoint; compare outcome_by_entry_conv to baselines.

**Tech Stack:** Python+NumPy for C1, Rust+CUDA for C2, Argo+L40S for C3.

## Pre-conditions

- Branch: `ml-alpha-phase-a` HEAD `1764cc394`
- F1 (dir_acc metric fix) RETAINED — empirically helpful
- F2 (mid_price NaN) REVERTED — wrong hypothesis
- Aux supervision infrastructure (CB1-CB5) LANDED but not consumed by decision policy
- D1 signed-conviction EMA fix (commit `0384f7674`) IS in baseline — handles part of anti-calibration

## Strategic rationale

Why Layer C over more aux iterations:
- **F4 synthesis verdict**: 70% implementation bugs / 30% architectural ceiling. Aux supervision continues to oscillate near chance regardless of label formulation.
- **Layer C is INDEPENDENT** of aux supervision. Gates on book state, not predicted profitability. Different mechanism entirely.
- **Empirical support exists**: `pearl_snapshot_alpha_is_regime_conditional` shows spread-Q4 = 75% accuracy directly from data analysis on ES MBP-10.
- **Layer C is COMPATIBLE with aux**: future aux work can layer on top of the regime gate (gate→aux predicts→size).

## Sub-decisions (resolved with defaults)

| # | Decision | Default | Rationale |
|---|---|---|---|
| 1 | Regime feature | Spread quintile | Per pearl: spread is the strongest single regime-conditional signal in ES MBP-10 |
| 2 | Quintile estimator | P-square (single-pass, O(1) per event) | Per E3+plan: avoids reservoir recompute overhead |
| 3 | Trailing window for quintile thresholds | EMA-tracked p20/p40/p60/p80 with α=2/(N+1), N=10000 events | Long enough to be stable across session boundaries, short enough to track regime shifts |
| 4 | Allowed quintile set | TBD by C1 — likely Q4 only OR Q1+Q4 union | Decision depends on cost-adjusted PnL per quintile |
| 5 | Gate scope | Per-batch (each parallel sim has own regime state) | Necessary for batched backtest correctness |
| 6 | Gate config plumbing | New `allowed_quintile_mask: u8` (5 bits) in policy config | Bitset more flexible than single quintile id |

## Task breakdown

### Task C1 — Per-regime cost-adjusted PnL analysis (NumPy)

**Files:**
- Create: `/tmp/layer_c1_quintile_pnl.py` (analysis script, not committed)

**Cost:** ~2 hours wallclock.

- [ ] **Step 1: Decode ES MBP-10 records on local data**

```python
# Reuse local file /home/jgrusewski/Work/foxhunt/test_data/futures-baseline-mbp10/ES.FUT/ES.FUT_2024-Q1.dbn.zst
# Decode ~2M records to span multiple sessions (avoid the gap-mismatch we saw in Step 2 of diagnosis plan)
```

- [ ] **Step 2: Compute spread at each snapshot**

```python
spread = ask_px[0] - bid_px[0]  # filter sentinels (price > 1e7)
```

- [ ] **Step 3: Compute spread quintiles via streaming p20/p40/p60/p80**

Use simple percentile-of-trailing-N-window for the NumPy version (P-square comes later in C2). Window = 10000 events.

- [ ] **Step 4: Per (quintile, horizon), compute expected directionally-optimal PnL net of cost**

For each (snapshot t, K, quintile q):
```
delta = mid[t+K] - mid[t]
optimal_dir = sign(delta)
gross_pnl = |delta|
net_pnl = gross_pnl - cost  # take if positive, else zero (assume policy is smart enough not to trade)
```

Aggregate `mean(net_pnl)` per (quintile, K). Positive means "this regime has alpha after cost".

- [ ] **Step 5: Decide gate set**

Output `allowed_quintile_mask: u8`. Default decision rule:
- Include quintile q if `mean(net_pnl) > 0` for ANY horizon
- Exclude middle quintiles where alpha is below noise

- [ ] **Step 6: Sanity check + write findings**

If Q4 has the largest cost-adjusted PnL (per pearl), confirm and use `0b10000 = 0x10`.
If Q1+Q4 both positive, use `0b10001 = 0x11`.

### Task C2 — Decision policy regime gate implementation

**Files:**
- Modify: `crates/ml-backtesting/cuda/decision_policy.cu` (~80 LOC for gate + P-square + per-batch state)
- Modify: `crates/ml-backtesting/src/sim/mod.rs` (allocate regime state buffers + `allowed_quintile_mask` upload path)
- Modify: `crates/ml-backtesting/src/policy/mod.rs` (Strategy config field `regime_gate_quintile_mask: u8`)
- Modify: `crates/ml-backtesting/src/harness.rs` (per-quintile trade counter logging)

**Cost:** ~8 hours.

- [ ] **Step 1: Add per-batch P-square state**

```cuda
// In LobSim state struct:
struct RegimeState {
    float p_estimates[4];  // p20, p40, p60, p80
    float p_heights[5];    // marker positions for P-square
    int n_seen;            // step count for warmup
};
```

- [ ] **Step 2: P-square quantile update**

Standard Jain-Chlamtac P² algorithm. Per event:
```cuda
__device__ void p_square_update(RegimeState* st, float new_value) {
    if (st->n_seen < 5) { st->p_heights[st->n_seen] = new_value; ... }
    // Else: increment markers, adjust positions, parabolic adjust outer markers
}
```

- [ ] **Step 3: Spread quintile classification at decision time**

```cuda
const float s = book.ask_px[0] - book.bid_px[0];
RegimeState* st = &regime_state[b];
p_square_update(st, s);
const int q = (st->n_seen < 100) ? 4 : (
    (s < st->p_estimates[0]) ? 0
  : (s < st->p_estimates[1]) ? 1
  : (s < st->p_estimates[2]) ? 2
  : (s < st->p_estimates[3]) ? 3 : 4
);
```

Warmup: first 100 events use Q4 (most permissive) to avoid blocking trading before quintiles stabilize.

- [ ] **Step 4: Gate logic**

```cuda
const bool allowed = (allowed_quintile_mask & (1u << q)) != 0;
if (!allowed) {
    target_lots = 0;
    regime_trades_filtered[b][q] += 1u;  // diagnostic counter
    return;
}
regime_trades_executed[b][q] += 1u;
```

- [ ] **Step 5: Rust wiring**

- Strategy config: `pub regime_gate_quintile_mask: u8` (default = 0xFF = all allowed = no-op gate)
- Sim: allocate `regime_state_d`, `regime_trades_filtered_d`, `regime_trades_executed_d`
- Harness: log per-quintile counters at end of run

- [ ] **Step 6: GPU oracle tests**

- `regime_p_square_converges_to_true_quintiles` — on a known-distribution stream, verify P-square estimates match true p20/p40/p60/p80 within 5%
- `regime_gate_filters_out_disallowed_quintiles` — with mask = 0x10 (Q4 only), verify ~80% of trade attempts are filtered

### Task C3 — Argo Smoke 2 with regime gate

**Files:**
- Modify: `config/ml/sweep_smoke_perhoriz_cfc.yaml` (add `regime_gate_quintile_mask: 0x10` or whatever C1 decided)

**Cost:** ~1 hour wallclock.

- [ ] **Step 1: Push + apply workflow template (if needed)**

- [ ] **Step 2: Dispatch sweep**

```bash
./scripts/argo-lob-sweep.sh --grid config/ml/sweep_smoke_perhoriz_cfc.yaml --branch ml-alpha-phase-a
```

- [ ] **Step 3: Analyze outcome_by_entry_conv**

Compare to baselines:
- D1 baseline (`lob-backtest-sweep-tcdxp`): WR 22%, anti-calibrated
- Layer C result: WR target ≥35% with similar or higher mean PnL

**Pass gate:** WR at top-decile conv buckets ≥ 40% AND mean PnL closer to zero than D1 baseline.
**Fail gate:** trade count collapses >95% (gate too restrictive) OR WR doesn't improve.

### Task C4 — Pearls

- [ ] Write pearl_regime_gate_essential_for_efficient_markets
- [ ] Write pearl_cost_adjusted_per_regime_pnl_decides_gate_set
- [ ] Document the F1/F2 aux supervision falsification chain

## Validation gates

| Gate | Layer | Pass condition |
|---|---|---|
| Library compile | C2 | `cargo check --workspace --all-targets` clean |
| GPU oracle tests | C2 | regime_p_square + regime_gate tests pass |
| Trade count | C3 | drops 50-90% (expected — only 1-2 quintiles allowed) |
| WR improvement | C3 | WR ≥ 40% in top conv buckets (baseline 22%) |
| Net PnL | C3 | mean PnL > -50% of baseline (much closer to zero) |

## Risks

| Risk | Mitigation |
|---|---|
| Q4 has highest spread → highest entry/exit cost | C1's net-of-cost computation explicitly handles this. If Q4 net < Q1 net, gate to Q1. |
| P-square slow to converge on first 100 events | Warmup permits all trades (Q4 fallback). Trades during warmup are a small fraction of total. |
| Trade count drops 99% — gate too tight | Fallback: relax to Q1∪Q4 union. C1 outputs the union mask if needed. |
| Per-batch regime state breaks captured CUDA graph | P-square is per-event scalar update — no host branches, no pointer changes. Compatible with graph capture per `pearl_no_host_branches_in_captured_graph`. |
| Regime feature drift across sessions | EMA window N=10000 events handles intra-session drift. Daily regime shifts ARE captured (boundary spreads inflate Q5 temporarily). |

## Estimated combined timeline

- Task C1 (NumPy): ~2 hours
- Task C2 (CUDA + Rust): ~8 hours
- Task C3 (Argo): ~1 hour wallclock
- Task C4 (Pearls): ~1 hour
- **Total: ~12 hours**

## Out-of-scope

- **Aux supervision wiring into decision policy** — that was B7; deferred indefinitely per the F4 falsification verdict
- **CB6 per-horizon Huber kernel split** — deferred
- **Layer C2 regime gate based on imbalance or OFI** — start with spread per pearl; revisit if WR target not hit
- **Multiple-feature gates** — could compose later (spread AND volatility AND time-of-day)

## Execution

After plan saved: dispatch task-by-task via subagent-driven-development.

C1 is independent and can run first. C2 starts immediately after C1's gate mask decision lands. C3 validates the combined work.
