# Alpha PPO Trainer — Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Build a new alpha-native RL trainer in `ml-alpha` that replaces the per-bar discrete-action DQN smoke approach (`alpha_baseline`) with a PPO actor-critic over multi-horizon alpha heads on a frozen Mamba2 perception trunk. Designed from the ground up for **multi-bar, multi-horizon trading** with continuous position sizing, decision-stride as a first-class concept, and IBKR-realistic passive-execution economics.

**Architecture (3-layer):**
1. **Perception** — Mamba2 trunk + multi-horizon alpha heads predicting `P(up | horizon=h)` for `h ∈ {30, 100, 1000, 6000}` bars. Trained supervised on labelled fxcache data. Frozen after pretrain.
2. **Decision policy** — small PPO actor-critic. Inputs: multi-horizon alpha probs + market state + position state. Outputs: direction categorical, hold-horizon categorical, position-size Gaussian, value scalar.
3. **Risk / execution** — Kelly-cap position sizing, max-drawdown gate, passive-limit posting via existing `ExecutionEnv`.

**Tech stack:** Rust 1.85+, cudarc CUDA, cuBLAS, Mamba2 SSM (reuse from `ml-alpha::mamba2_block`), PPO actor-critic on top, GAE for advantage estimation, multi-horizon supervised loss for perception.

**Rationale for PPO over DQN:** The existing fused-CUDA DQN trainer (`gpu_dqn_trainer.rs`, 1.9 MB) is sophisticated but was engineered for a different problem regime — per-bar discrete actions with distributional Q and synthetic reward shaping. We have no validation that this fits minute-horizon alpha with continuous position sizing. PPO addresses several DQN failure modes empirically observed today: bootstrapping noise at per-bar level (today's coin-flip problem), no native continuous action space (we want continuous position size), exploration via ε-greedy/Thompson clunky in trading. PPO's GAE + policy-gradient avoids these by design.

---

## File Structure

Following Path B (new binary in `ml-alpha`, library dep on `ml` only for kernels we genuinely need to share):

```
crates/ml-alpha/
├── src/
│   ├── lib.rs                  (modify: expose new modules)
│   ├── mamba2_block.rs         (existing, reuse)
│   ├── fxcache_reader.rs       (existing, reuse)
│   ├── multi_horizon_labels.rs (existing, extend to multi-h labels)
│   ├── perception.rs           (NEW: Mamba2 trunk + multi-horizon alpha heads)
│   ├── policy.rs               (NEW: actor-critic network heads)
│   ├── ppo.rs                  (NEW: PPO loss, GAE, clip)
│   ├── alpha_env.rs            (NEW: decision-stride env + segment reward)
│   ├── alpha_state.rs          (NEW: 55-dim state construction)
│   ├── walk_forward.rs         (NEW: native multi-fold driver)
│   ├── gae.rs                  (NEW: GAE advantage estimation)
│   └── kernels/                (NEW: subdir for PPO CUDA kernels)
│       ├── ppo_loss.cu
│       ├── gae.cu
│       └── policy_forward.cu
├── examples/
│   ├── alpha_train_perception.rs (NEW: pretrain multi-horizon alpha heads)
│   └── alpha_ppo_train.rs        (NEW: PPO training on frozen perception)
└── tests/
    └── (unit tests in src/, integration tests under tests/)
```

Reuse from `ml` (library deps, no modifications):
- `ml::env::action_space::N_ACTIONS` (just the constant, may not need with continuous size)
- `ml::env::execution_env::{ExecutionEnv, ExecutionEnvConfig}` (passive-fill env)
- `ml::env::loaders::load_fill_model_from_json` (alpha_fill_coeffs.json reader)
- `ml::cuda_pipeline::mapped_pinned::*` (pinned-memory helpers, for zero-copy CPU↔GPU)
- `ml::cuda_pipeline::alpha_kernels::*` (Mamba2 forward kernels, already validated)
- `ml::fxcache::*` (fxcache loader)

We do **not** depend on:
- `ml::trainers::dqn::*` (DQN-specific, not reusable for PPO)
- `ml::cuda_pipeline::gpu_dqn_trainer::*` (DQN-specific kernels)
- `ml::cuda_pipeline::fused_training::*` (DQN-specific)

---

## Phase 0: Pre-work (current 9-fold result + plan baseline)

- [ ] **Step 1: Wait for the running `alpha-cv-6jlm5` workflow to complete.** Need the 9-fold stride=200 + scaled-training result on full 9Q before committing to a baseline. Captures the **target Sharpe to beat** with the new PPO trainer.

Run: `argo get alpha-cv-6jlm5 -n foxhunt` until status = Succeeded.
Then: aggregate the 9 fold JSONs into a single summary table.
Expected: mean Sharpe at quarter-tick cost, std across 9 folds, break-even cost.

- [ ] **Step 2: Record baseline numbers in a memory file**

Create `/home/jgrusewski/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/project_alpha_dqn_baseline_9fold.md`:
- 9-fold mean Sharpe at `cost ∈ {0, ¼, ½, 1, 2}` ticks
- 9-fold std at each cost point
- Break-even cost
- Date + commit SHA
- Note: this is the DQN baseline that PPO must beat

This memory entry becomes the empirical bar PPO needs to clear.

---

## Phase 1: Multi-horizon labels + perception layer

Extend the existing single-horizon stacker into a multi-horizon perception model.

### Task 1.1: Multi-horizon label generation

**Files:**
- Modify: `crates/ml-alpha/src/multi_horizon_labels.rs`
- Test: `crates/ml-alpha/src/multi_horizon_labels.rs` (inline `#[cfg(test)]`)

- [ ] **Step 1: Write the failing test**

```rust
#[test]
fn multi_horizon_labels_match_known_prices() {
    // price series with known up moves at h=30, 100, 1000
    let prices: Vec<f32> = (0..2000).map(|i| {
        let base = 100.0;
        let bump_30 = if i < 30 { 0.0 } else { 0.5 };
        let bump_100 = if i < 100 { 0.0 } else { 1.0 };
        let bump_1000 = if i < 1000 { 0.0 } else { 2.0 };
        base + bump_30 + bump_100 + bump_1000
    }).collect();
    let labels = generate_multi_horizon_labels(&prices, &[30, 100, 1000]);
    // At bar 0, label[0] should be up at all 3 horizons (price moves up)
    assert_eq!(labels.labels[0][0], 1.0, "h=30 label at bar 0");
    assert_eq!(labels.labels[0][1], 1.0, "h=100 label at bar 0");
    assert_eq!(labels.labels[0][2], 1.0, "h=1000 label at bar 0");
    // Bar valid_indices should drop the last `max(horizons)` bars
    assert_eq!(labels.valid_indices.last().copied(), Some(prices.len() - 1001));
}
```

- [ ] **Step 2: Run test to verify it fails**

```bash
SQLX_OFFLINE=true cargo test -p ml-alpha --lib multi_horizon_labels_match_known_prices -- --nocapture
```
Expected: FAIL with "function not defined" or "wrong signature"

- [ ] **Step 3: Implement `generate_multi_horizon_labels`**

```rust
pub struct MultiHorizonLabels {
    /// labels[bar_idx] = [label@h0, label@h1, ...]
    pub labels: Vec<Vec<f32>>,
    /// Bar indices where ALL horizons have valid forward windows
    pub valid_indices: Vec<usize>,
    pub horizons: Vec<usize>,
    pub n_dropped_edge: usize,
    pub n_dropped_invalid: usize,
}

pub fn generate_multi_horizon_labels(
    prices: &[f32],
    horizons: &[usize],
) -> MultiHorizonLabels {
    let max_h = horizons.iter().copied().max().unwrap_or(0);
    let n = prices.len();
    let mut labels = Vec::with_capacity(n);
    let mut valid_indices = Vec::with_capacity(n);
    let mut n_dropped_edge = 0;
    let mut n_dropped_invalid = 0;
    for i in 0..n {
        if i + max_h >= n {
            n_dropped_edge += 1;
            continue;
        }
        let p0 = prices[i];
        if !p0.is_finite() || p0 <= 0.0 {
            n_dropped_invalid += 1;
            continue;
        }
        let row: Vec<f32> = horizons
            .iter()
            .map(|&h| if prices[i + h] > p0 { 1.0 } else { 0.0 })
            .collect();
        labels.push(row);
        valid_indices.push(i);
    }
    MultiHorizonLabels {
        labels,
        valid_indices,
        horizons: horizons.to_vec(),
        n_dropped_edge,
        n_dropped_invalid,
    }
}
```

- [ ] **Step 4: Run test to verify it passes**

```bash
SQLX_OFFLINE=true cargo test -p ml-alpha --lib multi_horizon_labels_match_known_prices
```
Expected: PASS

- [ ] **Step 5: Commit**

```bash
git add crates/ml-alpha/src/multi_horizon_labels.rs
git commit -m "feat(alpha): multi-horizon label generation"
```

### Task 1.2: Multi-horizon alpha head module

**Files:**
- Create: `crates/ml-alpha/src/perception.rs`
- Test: inline `#[cfg(test)]`

- [ ] **Step 1: Write the failing test for forward shape**

```rust
#[test]
fn multi_horizon_head_emits_one_logit_per_horizon() {
    let stream = CudaContext::new(0).unwrap().default_stream();
    let head = MultiHorizonHead::new(&stream, 64, &[30, 100, 1000]).unwrap();
    let batch = 8;
    let hidden = GpuTensor::zeros(&[batch, 64], &stream).unwrap();
    let out = head.forward(&hidden).unwrap();
    assert_eq!(out.shape(), &[batch, 3]);
}
```

- [ ] **Step 2: Run test**

```bash
SQLX_OFFLINE=true CUDA_COMPUTE_CAP=86 cargo test -p ml-alpha --lib multi_horizon_head_emits_one_logit_per_horizon
```
Expected: FAIL — type not defined.

- [ ] **Step 3: Implement `MultiHorizonHead`**

```rust
/// One linear layer per horizon, sharing the upstream hidden state.
/// Output: [batch, n_horizons] of pre-sigmoid logits.
pub struct MultiHorizonHead {
    pub horizons: Vec<usize>,
    weights: GpuTensor, // [hidden_dim, n_horizons]
    bias: GpuTensor,    // [n_horizons]
}

impl MultiHorizonHead {
    pub fn new(stream: &Arc<CudaStream>, hidden_dim: usize, horizons: &[usize]) -> Result<Self> {
        let n_h = horizons.len();
        // Xavier init: std = sqrt(2 / (fan_in + fan_out)) for tanh-ish nonlinearity
        // upstream; use small std for logit-output layer to start near 0.5 prob.
        let std = (2.0_f32 / (hidden_dim as f32 + n_h as f32)).sqrt() * 0.1;
        let mut rng = rand::thread_rng();
        let w_cpu: Vec<f32> = (0..hidden_dim * n_h)
            .map(|_| rng.sample(rand_distr::Normal::new(0.0, std).unwrap()))
            .collect();
        let weights = GpuTensor::from_slice(&[hidden_dim, n_h], &w_cpu, stream)?;
        let bias = GpuTensor::zeros(&[n_h], stream)?;
        Ok(Self { horizons: horizons.to_vec(), weights, bias })
    }

    /// Forward: y = hidden @ weights + bias.
    /// Returns pre-sigmoid logits [batch, n_horizons].
    pub fn forward(&self, hidden: &GpuTensor) -> Result<GpuTensor> {
        // Reuse the matmul kernel from mamba2_block or ml-alpha::mlp
        // (whichever is exposed); fall back to cuBLAS sgemm if needed.
        crate::mlp::linear_forward(hidden, &self.weights, &self.bias)
    }
}
```

- [ ] **Step 4: Run test**

```bash
SQLX_OFFLINE=true CUDA_COMPUTE_CAP=86 cargo test -p ml-alpha --lib multi_horizon_head_emits_one_logit_per_horizon
```
Expected: PASS

- [ ] **Step 5: Commit**

```bash
git add crates/ml-alpha/src/perception.rs
git commit -m "feat(alpha): multi-horizon alpha head — one logit per horizon"
```

### Task 1.3: Perception entry binary `alpha_train_perception`

Replaces the single-horizon `alpha_train_stacker`. Trains Mamba2 trunk + multi-horizon heads jointly, saves the trunk weights + per-bar multi-horizon logit cache.

**Files:**
- Create: `crates/ml-alpha/examples/alpha_train_perception.rs`

(Boilerplate: copy from `alpha_train_stacker.rs`, swap out single-horizon loss for multi-horizon BCE, persist `perception_weights.bin` + `multi_horizon_logits_cache.bin`)

- [ ] **Step 1: Scaffold the CLI (steal from `alpha_train_stacker.rs`)**

```rust
// crates/ml-alpha/examples/alpha_train_perception.rs (NEW)
use anyhow::Result;
use clap::Parser;

#[derive(Parser, Debug)]
#[command(name = "alpha_train_perception")]
struct Cli {
    #[arg(long)] fxcache_path: String,
    #[arg(long, value_delimiter = ',', default_value = "30,100,1000,6000")]
    horizons: Vec<usize>,
    #[arg(long, default_value_t = 32)] seq_len: usize,
    #[arg(long, default_value_t = 64)] hidden_dim: usize,
    #[arg(long, default_value_t = 16)] state_dim: usize,
    #[arg(long, default_value_t = 5)]  epochs: usize,
    #[arg(long, default_value_t = 128)] batch_size: usize,
    #[arg(long, default_value_t = 3e-3)] lr: f32,
    #[arg(long, default_value_t = 0.8)] train_frac: f32,
    #[arg(long)] perception_out: String,    // trunk + heads weights
    #[arg(long)] logits_cache_out: String,  // per-bar multi-horizon logits
    #[arg(long)] max_rows: Option<usize>,
}

fn main() -> Result<()> {
    // (1) load fxcache, extract prices + alpha_features
    // (2) generate multi-horizon labels for each h
    // (3) train Mamba2 trunk + MultiHorizonHead with per-horizon BCE
    // (4) snapshot pass: emit logits[bar, h] for all bars to logits_cache_out
    // (5) serialize trunk + head weights to perception_out
    todo!("implement using alpha_train_stacker.rs as template")
}
```

- [ ] **Step 2: Implement the training loop** — mirrors alpha_train_stacker but with multi-horizon BCE loss = `mean_h(BCE(sigmoid(logit_h), label_h))`.

- [ ] **Step 3: Implement the logits-cache export** — same structure as `alpha_logits_cache.bin` but with `n_horizons` floats per bar instead of one.

```
binary format:
  u32 LE: magic 0x504D4848 ("MHLP")
  u32 LE: version (1)
  u32 LE: n_bars
  u32 LE: n_horizons
  f32 LE × n_horizons: horizon values
  f32 LE × n_bars × n_horizons: logits (pre-sigmoid)
```

- [ ] **Step 4: Smoke test on 2Q fxcache locally**

```bash
CUDA_COMPUTE_CAP=86 SQLX_OFFLINE=true cargo build --release -p ml-alpha --example alpha_train_perception
./target/release/examples/alpha_train_perception \
  --fxcache-path /tmp/foxhunt-profile/out/1e36bcab*.fxcache \
  --horizons 30,100,1000,6000 \
  --epochs 5 \
  --perception-out /tmp/perception_2q.bin \
  --logits-cache-out /tmp/multi_horizon_logits_2q.bin
```
Expected: file `/tmp/multi_horizon_logits_2q.bin` ≈ 4 × `alpha_logits_cache_2q.bin` size (~48 MB).

- [ ] **Step 5: Commit**

```bash
git add crates/ml-alpha/examples/alpha_train_perception.rs
git commit -m "feat(alpha): multi-horizon perception trainer + logits cache export"
```

---

## Phase 2: PPO trainer scaffolding

### Task 2.1: Alpha state (55-dim)

**Files:**
- Create: `crates/ml-alpha/src/alpha_state.rs`

- [ ] **Step 1: Test that state construction is deterministic**

```rust
#[test]
fn alpha_state_dim_55() {
    let fx = mock_fxcache_record();              // OHLC + 38 market features
    let alpha = [0.1_f32; 4];                    // 4-horizon alpha probs
    let mamba_hidden = [0.5_f32; 6];             // 6-dim Mamba2 hidden
    let portfolio = [0.0_f32; 2];                // position, P&L (no spread/regime here yet)
    let state = AlphaState::from_parts(&fx, &alpha, &mamba_hidden, &portfolio);
    assert_eq!(state.to_vec().len(), 55);
}
```

- [ ] **Step 2: Implement `AlphaState`**

```rust
pub const ALPHA_STATE_DIM: usize = 55;
// 4 OHLC log returns + 38 market features = 42 from fxcache
// + 4 multi-horizon alpha probs
// + 6 Mamba2 hidden state
// + 3 portfolio (position, unrealized P&L, time-since-entry)
// = 55

pub struct AlphaState {
    pub vec: [f32; ALPHA_STATE_DIM],
}

impl AlphaState {
    pub fn from_parts(
        fxcache_features: &[f32],         // 42 elements
        multi_horizon_alpha: &[f32],      // 4 elements (sigmoid(logits))
        mamba_hidden: &[f32],             // 6 elements
        portfolio: &[f32],                // 3 elements
    ) -> Self {
        assert_eq!(fxcache_features.len(), 42);
        assert_eq!(multi_horizon_alpha.len(), 4);
        assert_eq!(mamba_hidden.len(), 6);
        assert_eq!(portfolio.len(), 3);
        let mut vec = [0_f32; ALPHA_STATE_DIM];
        vec[0..42].copy_from_slice(fxcache_features);
        vec[42..46].copy_from_slice(multi_horizon_alpha);
        vec[46..52].copy_from_slice(mamba_hidden);
        vec[52..55].copy_from_slice(portfolio);
        Self { vec }
    }

    pub fn to_vec(&self) -> &[f32] {
        &self.vec
    }
}
```

- [ ] **Step 3: Run test + commit**

```bash
SQLX_OFFLINE=true cargo test -p ml-alpha --lib alpha_state_dim_55
git add crates/ml-alpha/src/alpha_state.rs
git commit -m "feat(alpha): 55-dim AlphaState (production features + multi-h alpha + mamba + portfolio)"
```

### Task 2.2: Decision-stride env

**Files:**
- Create: `crates/ml-alpha/src/alpha_env.rs`

The env wraps `ml::env::execution_env::ExecutionEnv` and exposes a stride-aware decision interface: `decide_at(bar_idx)` returns `Some(state)` only at decision points; calling `step(action)` advances `decision_stride` bars at once and returns the segment reward.

- [ ] **Step 1: Test that stride correctly gates action emission**

```rust
#[test]
fn env_stride_yields_decisions_at_boundary_only() {
    let env = build_test_env(/* decision_stride */ 5, /* horizon */ 100);
    let mut decisions = 0;
    for bar in 0..100 {
        if env.is_decision_point(bar) { decisions += 1; }
    }
    assert_eq!(decisions, 100 / 5);
}
```

- [ ] **Step 2: Implement `AlphaExecutionEnv`** — wraps `ExecutionEnv`, adds `decision_stride`, segment-PnL reward aggregation.

```rust
pub struct AlphaExecutionEnv {
    inner: ml::env::execution_env::ExecutionEnv,
    pub decision_stride: usize,
    pub bar_cursor: usize,
}

impl AlphaExecutionEnv {
    pub fn is_decision_point(&self, bar: usize) -> bool {
        bar % self.decision_stride == 0
    }

    /// Emit one decision: take action, advance `decision_stride` bars,
    /// return (next_state, segment_reward, done).
    pub fn step_segment(
        &mut self,
        action: PolicyAction,
        portfolio: &mut PortfolioState,
    ) -> Option<(AlphaState, f32, bool)> {
        let mut reward_sum = 0_f32;
        let target_bar = self.bar_cursor + self.decision_stride;
        // Apply the action ONCE at the start of the segment; hold through.
        self.apply_action(action, portfolio)?;
        while self.bar_cursor < target_bar {
            let (_, bar_r, done) = self.inner.step(0 /* wait */, portfolio)?;
            reward_sum += bar_r;
            self.bar_cursor += 1;
            if done { return Some((self.build_state(portfolio), reward_sum, true)); }
        }
        Some((self.build_state(portfolio), reward_sum, false))
    }
    // ...
}
```

- [ ] **Step 3: Test segment reward aggregates correctly**

```rust
#[test]
fn segment_reward_sums_intra_segment_pnl() {
    let mut env = build_test_env(5, 20);
    let mut port = PortfolioState::default();
    let (_, r, _) = env.step_segment(PolicyAction::Long(1.0, Horizon::H30), &mut port).unwrap();
    // r should equal sum of bar-level rewards over 5 bars; check magnitude.
    assert!(r.abs() < 10.0, "segment reward in plausible range");
}
```

- [ ] **Step 4: Commit**

```bash
git add crates/ml-alpha/src/alpha_env.rs
git commit -m "feat(alpha): decision-stride env with segment-reward aggregation"
```

### Task 2.3: Policy network — actor-critic heads

**Files:**
- Create: `crates/ml-alpha/src/policy.rs`

Multi-head policy: 3 categorical (direction / horizon / size-bin) + 1 scalar (value). PPO needs both action sampling and log-probability computation.

- [ ] **Step 1: Define `PolicyAction` enum and `Policy` struct**

```rust
#[derive(Debug, Clone, Copy)]
pub enum Direction { Long, Short, Flat }

#[derive(Debug, Clone, Copy)]
pub enum Horizon { H30, H100, H1k, H6k }

/// 8 discrete size bins (logarithmic): 0, 1, 2, 4, 8, 16, 32, 64 % of capital.
/// Discrete to avoid continuous-action PPO complexity; can swap to Gaussian
/// later if size resolution matters.
#[derive(Debug, Clone, Copy)]
pub struct SizeBin(pub u8);

#[derive(Debug, Clone, Copy)]
pub struct PolicyAction {
    pub direction: Direction,
    pub horizon: Horizon,
    pub size: SizeBin,
}

pub struct PolicyOutput {
    pub direction_logits: [f32; 3],
    pub horizon_logits:   [f32; 4],
    pub size_logits:      [f32; 8],
    pub value:            f32,
}
```

- [ ] **Step 2: Implement the network forward**

Shared trunk (MLP, 2 hidden layers @ 128) → 3 logit heads + value head. Use existing `ml_alpha::mlp::MlpModel` as the trunk; add 4 head linears.

- [ ] **Step 3: Implement action sampling + log-prob**

```rust
impl Policy {
    /// Sample an action from the distribution; return action + log-prob.
    pub fn sample(&self, state: &AlphaState, rng: &mut impl Rng) -> (PolicyAction, f32) {
        let out = self.forward(state);
        let dir = sample_categorical(&out.direction_logits, rng);
        let hor = sample_categorical(&out.horizon_logits, rng);
        let siz = sample_categorical(&out.size_logits, rng);
        let log_p = log_softmax(&out.direction_logits)[dir]
                  + log_softmax(&out.horizon_logits)[hor]
                  + log_softmax(&out.size_logits)[siz];
        (PolicyAction { direction: dir.into(), horizon: hor.into(), size: SizeBin(siz as u8) },
         log_p)
    }
}
```

- [ ] **Step 4: Test that sampled actions are within the action space**

- [ ] **Step 5: Commit**

```bash
git add crates/ml-alpha/src/policy.rs
git commit -m "feat(alpha): actor-critic policy with 3-head categorical + value"
```

### Task 2.4: GAE advantage estimation

**Files:**
- Create: `crates/ml-alpha/src/gae.rs`

Standard Generalised Advantage Estimation (Schulman 2016). Single function: given (rewards, values, dones), produce advantages + returns.

- [ ] **Step 1: Test on a known trajectory**

```rust
#[test]
fn gae_matches_paper_formula_simple_case() {
    let rewards = vec![1.0, 0.0, 0.0, 1.0];
    let values  = vec![0.5, 0.5, 0.5, 0.5, 0.0];  // V(s_4) = 0 since done
    let dones   = vec![false, false, false, true];
    let gamma = 0.99_f32;
    let lambda = 0.95_f32;
    let (adv, ret) = compute_gae(&rewards, &values, &dones, gamma, lambda);
    // hand-compute: delta_t = r_t + gamma*V(s_{t+1})*(1-done_t) - V(s_t)
    // advantage_t = delta_t + gamma*lambda*advantage_{t+1}*(1-done_t)
    // ...
    assert!((adv[3] - 0.5).abs() < 1e-6);
}
```

- [ ] **Step 2: Implement `compute_gae`**

- [ ] **Step 3: Run test + commit**

### Task 2.5: PPO loss

**Files:**
- Create: `crates/ml-alpha/src/ppo.rs`

The canonical PPO clip objective + value MSE + entropy regulariser.

- [ ] **Step 1: Test clip behaviour**

- [ ] **Step 2: Implement `ppo_loss(old_log_probs, new_log_probs, advantages, returns, values, clip_eps, value_coef, entropy_coef)`**

```rust
pub struct PpoLossResult {
    pub policy_loss:  f32,
    pub value_loss:   f32,
    pub entropy_bonus: f32,
    pub total: f32,
    pub n_clipped: usize,
}

pub fn ppo_loss(
    old_log_probs: &[f32],
    new_log_probs: &[f32],
    advantages: &[f32],
    returns: &[f32],
    values: &[f32],
    entropies: &[f32],
    clip_eps: f32,
    value_coef: f32,
    entropy_coef: f32,
) -> PpoLossResult {
    let mut policy_loss = 0_f32;
    let mut value_loss = 0_f32;
    let mut entropy = 0_f32;
    let mut clipped = 0;
    for i in 0..old_log_probs.len() {
        let ratio = (new_log_probs[i] - old_log_probs[i]).exp();
        let unclipped = ratio * advantages[i];
        let clipped_ratio = ratio.clamp(1.0 - clip_eps, 1.0 + clip_eps);
        let clipped_val = clipped_ratio * advantages[i];
        if clipped_ratio != ratio { clipped += 1; }
        // PPO maximises advantage * ratio, so minimise -min(unclipped, clipped)
        policy_loss -= unclipped.min(clipped_val);
        let td = returns[i] - values[i];
        value_loss += td * td;
        entropy += entropies[i];
    }
    let n = old_log_probs.len() as f32;
    let policy_loss = policy_loss / n;
    let value_loss = value_loss / n;
    let entropy_bonus = entropy / n;
    let total = policy_loss + value_coef * value_loss - entropy_coef * entropy_bonus;
    PpoLossResult { policy_loss, value_loss, entropy_bonus, total, n_clipped: clipped }
}
```

- [ ] **Step 3: Run test + commit**

---

## Phase 3: Training loop

### Task 3.1: Rollout collection

Collect a buffer of (state, action, log_prob, reward, value, done) tuples over N decision steps, using the current policy.

**Files:**
- Modify: `crates/ml-alpha/examples/alpha_ppo_train.rs` (or new helper module)

- [ ] **Step 1: Implement `collect_rollouts(env, policy, n_steps)` → trajectory buffer**

- [ ] **Step 2: Test that the rollout length matches and terminal handling is correct**

### Task 3.2: PPO update step

- [ ] **Step 1: Given a rollout buffer, run K epochs of minibatch PPO updates on (state, action, advantage, return, old_log_prob).**

```python
for epoch in 0..ppo_epochs:
    shuffle minibatches
    for minibatch in batches:
        recompute new_log_prob, value, entropy via policy.forward
        loss = ppo_loss(old, new, adv, ret, val, ent, ...)
        backward + Adam step (clip grad norm)
```

- [ ] **Step 2: Verify that ppo loss decreases over the same rollout** (sanity check: many gradient steps on the same data should reduce loss).

### Task 3.3: Main training driver `alpha_ppo_train`

**Files:**
- Create: `crates/ml-alpha/examples/alpha_ppo_train.rs`

CLI similar to `alpha_train_stacker`:

```
--fxcache-path ...
--perception ...                # pretrained perception (frozen)
--horizons 30,100,1000,6000
--decision-stride 200
--n-rollouts 8000               # total rollout steps
--rollout-length 200            # decisions per rollout
--ppo-epochs 4
--clip-eps 0.2
--gamma 0.99
--lambda 0.95
--value-coef 0.5
--entropy-coef 0.01
--lr 3e-4
--train-frac 0.6
--max-snapshots 1900000
--data-start-offset 0
--out-path /tmp/ppo_fold_A.json
```

- [ ] **Step 1: Scaffold the CLI**

- [ ] **Step 2: Implement the outer loop** (load fxcache → load frozen perception → init policy → loop: collect rollouts, run PPO updates, log Sharpe on held-out segment).

- [ ] **Step 3: Build + run on Fold A 2Q locally; confirm it converges (loss drops, Sharpe positive on eval segment).**

```bash
CUDA_COMPUTE_CAP=86 SQLX_OFFLINE=true cargo build --release -p ml-alpha --example alpha_ppo_train
./target/release/examples/alpha_ppo_train \
  --fxcache-path /tmp/foxhunt-profile/out/1e36bcab*.fxcache \
  --perception /tmp/perception_2q.bin \
  --decision-stride 200 \
  --n-rollouts 8000 \
  --out-path /tmp/ppo_fold_A.json
```
Expected: eval Sharpe at quarter-tick cost ≥ +1.0 (must clear the DQN baseline meaningfully to justify the rebuild).

---

## Phase 4: GPU acceleration

The Phase 3 driver runs CPU-only PPO (Rust + cudarc for matmul). For full training scale, we need GPU-resident PPO update.

### Task 4.1: Policy forward kernel

**Files:**
- Create: `crates/ml-alpha/src/kernels/policy_forward.cu`

Fused MLP forward: trunk → 3 logit heads + value. Single kernel launch per minibatch.

- [ ] **Step 1: Write CUDA C source**
- [ ] **Step 2: Add NVRTC build hook in `build.rs`**
- [ ] **Step 3: Wire into `Policy::forward_gpu` (replaces the CPU forward)**
- [ ] **Step 4: Verify bit-equivalence against CPU forward on small batch**

### Task 4.2: PPO loss + backward kernel

- [ ] Fused kernel: given (new_log_probs, old_log_probs, advantages, returns, values, entropies) compute total loss + per-parameter gradients in one pass.
- [ ] Verify bit-equivalence on small batch.

### Task 4.3: GAE on GPU

- [ ] Sequential dependency makes parallelism hard; standard approach is reverse-prefix-scan. Single block thread-cooperative scan since rollout sizes are small (<10K).

---

## Phase 5: Walk-forward CV + cluster integration

### Task 5.1: Native walk-forward driver

**Files:**
- Create: `crates/ml-alpha/src/walk_forward.rs`

Iterate over fold offsets, train PPO from scratch on each fold, backtest on held-out portion.

- [ ] **Step 1: Implement `walk_forward(fxcache, perception, fold_offsets, fold_window, ppo_config)` → Vec<FoldResult>**
- [ ] **Step 2: Test on 3 folds locally; compare to DQN baseline**

### Task 5.2: New Argo workflow `alpha-ppo`

Replaces `alpha-cv-template.yaml`. One pod runs the full 9-fold pipeline natively.

**Files:**
- Create: `infra/k8s/argo/alpha-ppo-template.yaml`
- Create: `scripts/argo-alpha-ppo.sh`

- [ ] **Step 1: Adapt `alpha-cv-template.yaml`** — replace ensure-binary BINARIES list, replace stacker-train with perception-train, replace 9× alpha_baseline folds with single `alpha_ppo_train --walk-forward 9`.

- [ ] **Step 2: Apply WorkflowTemplate + commit + push**

```bash
kubectl -n foxhunt apply -f infra/k8s/argo/alpha-ppo-template.yaml
git add infra/k8s/argo/alpha-ppo-template.yaml scripts/argo-alpha-ppo.sh
git commit -m "infra(argo): alpha-ppo workflow for PPO 9-fold training"
git push
```

- [ ] **Step 3: Submit + monitor**

```bash
./scripts/argo-alpha-ppo.sh --branch main
```

Expected wall: ~90-120 min on L40S (perception 15 min + 9 folds × 10-12 min).

---

## Phase 6: Validation gate

Before declaring the new trainer ready, must clear the **DQN 9-fold baseline** captured in Phase 0 by a meaningful margin.

Acceptance criteria:
- [ ] Mean Sharpe at quarter-tick cost ≥ baseline + 0.5 (one std).
- [ ] Standard deviation across 9 folds ≤ baseline std (no regime overfitting).
- [ ] Break-even cost ≥ baseline (no economic regression).
- [ ] No fold worse than baseline worst fold by more than 1 Sharpe (robustness check).

If criteria are not met:
- [ ] Investigate failure modes. Common: PPO entropy collapse (policy gets too confident → no exploration), value-function lag (advantage misestimation), perception over/under-fit. Each has a standard remediation.

If criteria are met:
- [ ] Update memory `project_alpha_ppo_baseline.md` with the new bar.
- [ ] Document the architectural choice in a `pearl_*.md` entry (memory writes happen via Write tool with the auto-memory format).

---

## Self-Review

**Spec coverage:** Plan covers perception (multi-horizon labels + heads + pretraining binary), policy network (3-head categorical + value), env (decision-stride + segment reward), PPO loss + GAE, training loop, GPU acceleration, walk-forward, cluster workflow, validation gate. The user's three answered design decisions (Path B / branched-as-categorical / multi-horizon blend / hybrid 55-dim state) are all baked in.

**Placeholder scan:** Three `todo!()` markers in code snippets are illustrative; they get replaced when the task is executed (the body of `alpha_train_perception::main` follows the alpha_train_stacker template). No "TBD" / "later" markers.

**Type consistency:** `AlphaState`, `PolicyAction`, `Direction`, `Horizon`, `SizeBin`, `PolicyOutput`, `PpoLossResult`, `MultiHorizonLabels`, `MultiHorizonHead`, `AlphaExecutionEnv` defined once, referenced consistently. State dimension 55 = 42 + 4 + 6 + 3 verified in `alpha_state.rs` test.

**Ambiguity callout:** The size space (8 discrete bins vs Gaussian continuous) is a soft decision in the plan. Phase 3 implements discrete bins for PPO simplicity; if size-resolution matters empirically, swap to Gaussian (10–20 LOC change in `policy.rs`). Noted in Task 2.3.

**Total scope:** ~3 weeks of focused work for Phases 1–5; Phase 6 is the validation gate. Phase 0 baseline capture is blocking on the current `alpha-cv-6jlm5` cluster run.
