# Eval-Summary Trade Aggregation Fix — Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:subagent-driven-development. Steps use checkbox `- [ ]` syntax.

**Spec:** `docs/superpowers/specs/2026-05-31-eval-summary-trade-aggregation-design.md`

**Goal:** `eval_summary.json` reflects ALL `b_size` accounts' eval-phase-only trades, with no train-phase contamination. Currently it computes metrics on backtest 0's last 1024 records (a mix of train + eval trades).

**Architecture:** Add two methods to `LobSimCuda` (read per-backtest counts + read all backtests' records). Replace the broken slicing block in `alpha_rl_train.rs`. Bump `TRADE_LOG_CAP` from 1024 → 4096 for cluster-scale headroom.

**Tech Stack:** Rust, cudarc.

**Branch:** `ml-alpha-eval-summary-fix` (off `ml-alpha-regime-observer`)

**Independent of Phase A** — touches different files (lobsim + eval slicing block, not the train-loop json!{}). Can run in parallel via separate subagent dispatch but must NOT branch off the in-flight `ml-alpha-checkpoints-eval-diag` branch (would conflict on alpha_rl_train.rs around the eval phase). Use a separate base.

---

## File structure

| Path | Action |
|------|--------|
| `crates/ml-backtesting/src/lob/mod.rs` | Modify — bump `TRADE_LOG_CAP` from 1024 to 4096 |
| `crates/ml-backtesting/src/sim/mod.rs` | Modify — add 2 new public methods on `LobSimCuda` |
| `crates/ml-alpha/examples/alpha_rl_train.rs` | Modify — replace slicing block with per-account aggregation |
| `crates/ml-backtesting/tests/eval_aggregation_smoke.rs` | Create — unit test for the new methods |

---

## E.1 — Lobsim API: `read_per_backtest_trade_counts`

**Files:**
- Modify: `crates/ml-backtesting/src/sim/mod.rs`

- [ ] **Step 1: Locate current `read_trade_records` for reference**

```bash
grep -nE "pub fn read_trade_records|pub fn read_total_trade_count" crates/ml-backtesting/src/sim/mod.rs
```

Expected: `read_trade_records` at ~line 1487 and `read_total_trade_count` somewhere nearby.

- [ ] **Step 2: Add `read_per_backtest_trade_counts`**

```rust
// crates/ml-backtesting/src/sim/mod.rs — in impl LobSimCuda { ... }
// Place immediately AFTER read_total_trade_count or read_trade_records.

/// Read per-backtest cumulative trade counters. Returns a Vec of length
/// `n_backtests` where entry `i` is the total number of closed-trade
/// records ever pushed to backtest `i`'s ring (may exceed TRADE_LOG_CAP).
///
/// Used by alpha_rl_train to snapshot the eval-phase boundary per
/// account: `head_before_per_b[i] = read_per_backtest_trade_counts()[i]`
/// at eval start, then post-eval `head_after_per_b[i] - head_before_per_b[i]`
/// is the true eval-phase trade count for backtest `i`.
pub fn read_per_backtest_trade_counts(&self) -> Result<Vec<u32>> {
    let mut heads = vec![0u32; self.n_backtests];
    self.stream.synchronize()
        .context("sync before reading trade_log_head_d")?;
    self.stream.memcpy_dtoh(&self.trade_log_head_d, heads.as_mut_slice())?;
    Ok(heads)
}
```

- [ ] **Step 3: cargo check**

```bash
SQLX_OFFLINE=true cargo check -p ml-backtesting
```

Expected: clean.

- [ ] **Step 4: Commit**

```bash
git add crates/ml-backtesting/src/sim/mod.rs
git commit -m "feat(lobsim): read_per_backtest_trade_counts (E.1)"
```

---

## E.2 — Lobsim API: `read_trade_records_all`

**Files:**
- Modify: `crates/ml-backtesting/src/sim/mod.rs`

- [ ] **Step 1: Add `read_trade_records_all`**

```rust
// crates/ml-backtesting/src/sim/mod.rs — immediately after read_per_backtest_trade_counts

/// Read all backtests' trade record rings. Returns Vec<Vec<TradeRecord>>
/// of length `n_backtests`; entry `i` is the most recent up-to-
/// TRADE_LOG_CAP records for backtest `i` (oldest dropped on wrap).
///
/// Single DtoH of the full trade_log_d tensor + per-backtest head
/// values, split into per-account slices. Cost: O(b_size × cap × 40 B)
/// — at b=1024, cap=4096 this is 167 MB transferred once.
///
/// Callers compose this with `read_per_backtest_trade_counts()` to
/// determine each account's slice into eval-phase-only records.
pub fn read_trade_records_all(
    &self,
) -> Result<Vec<Vec<crate::order::TradeRecord>>> {
    self.stream.synchronize()
        .context("sync before reading trade logs")?;

    let mut heads = vec![0u32; self.n_backtests];
    self.stream.memcpy_dtoh(&self.trade_log_head_d, heads.as_mut_slice())?;

    let rec_bytes = crate::lob::TRADE_RECORD_BYTES;
    let cap = crate::lob::TRADE_LOG_CAP;
    let mut raw = vec![0u8; self.n_backtests * cap * rec_bytes];
    self.stream.memcpy_dtoh(&self.trade_log_d, raw.as_mut_slice())?;

    let mut out = Vec::with_capacity(self.n_backtests);
    for b in 0..self.n_backtests {
        let head = heads[b] as usize;
        let n_to_read = head.min(cap);
        let off = b * cap * rec_bytes;
        let mut records = Vec::with_capacity(n_to_read);
        for i in 0..n_to_read {
            let rec_off = off + i * rec_bytes;
            let r: crate::order::TradeRecord =
                bytemuck::pod_read_unaligned(&raw[rec_off..rec_off + rec_bytes]);
            records.push(r);
        }
        out.push(records);
    }
    Ok(out)
}
```

- [ ] **Step 2: cargo check**

```bash
SQLX_OFFLINE=true cargo check -p ml-backtesting
```

Expected: clean.

- [ ] **Step 3: Commit**

```bash
git add crates/ml-backtesting/src/sim/mod.rs
git commit -m "feat(lobsim): read_trade_records_all (E.2)"
```

---

## E.3 — `TRADE_LOG_CAP` bump

**Files:**
- Modify: `crates/ml-backtesting/src/lob/mod.rs`

- [ ] **Step 1: Verify the constant is referenced nowhere as hardcoded `1024`**

```bash
grep -rn "1024" crates/ml-backtesting/src/ | grep -iE "trade_log|TradeLog" | head -10
```

Expected: only `pub const TRADE_LOG_CAP: usize = 1024;` itself. Any other hits are flagged for review.

- [ ] **Step 2: Bump the constant**

```rust
// crates/ml-backtesting/src/lob/mod.rs:20
// BEFORE:
pub const TRADE_LOG_CAP: usize = 1024;
// AFTER:
pub const TRADE_LOG_CAP: usize = 4096;  // 2026-05-31: bumped from 1024 to prevent
                                        // eval-phase trade ring wrap at b=1024 cluster scale
                                        // (typical eval trades/account ≈ 342, peak ~1000;
                                        // 4096 gives 4× headroom). Memory: b × cap × 40 B
                                        // = 1024 × 4096 × 40 = 167 MB on GPU (was 41 MB).
                                        // See spec 2026-05-31-eval-summary-trade-aggregation.
```

- [ ] **Step 3: cargo build (to catch any test asserting the old value)**

```bash
SQLX_OFFLINE=true cargo build -p ml-backtesting
SQLX_OFFLINE=true cargo build -p ml-alpha
```

Expected: clean.

- [ ] **Step 4: Run existing lobsim tests to verify no regression**

```bash
SQLX_OFFLINE=true cargo test -p ml-backtesting --lib 2>&1 | tail -10
```

Expected: all tests pass (or only-failing tests are known-failing for unrelated reasons).

- [ ] **Step 5: Commit**

```bash
git add crates/ml-backtesting/src/lob/mod.rs
git commit -m "feat(lobsim): bump TRADE_LOG_CAP 1024 → 4096 for eval headroom (E.3)"
```

---

## E.4 — alpha_rl_train.rs: replace slicing block with aggregation

**Files:**
- Modify: `crates/ml-alpha/examples/alpha_rl_train.rs`

This is the core fix. Replace the existing broken block with per-account aggregation that correctly isolates eval-phase trades.

- [ ] **Step 1: Locate the existing block**

```bash
grep -nE "head_before_eval|read_trade_records\(0\)|trade log wrapped" crates/ml-alpha/examples/alpha_rl_train.rs
```

Expected: the block lives roughly at lines 743 (pre-eval snapshot) and 1402-1413 (post-eval slicing).

- [ ] **Step 2: Replace pre-eval snapshot (around line 743)**

```rust
// BEFORE:
let head_before_eval = sim
    .read_total_trade_count()
    .context("read trade count pre-eval")?;
eprintln!(
    "── eval phase: {} steps on {} held-out files (trade-record checkpoint head={}) ──",
    cli.n_eval_steps, eval_files.len(), head_before_eval
);

// AFTER:
let head_before_per_b = sim
    .read_per_backtest_trade_counts()
    .context("snapshot per-backtest head pre-eval")?;
let head_before_min = head_before_per_b.iter().min().copied().unwrap_or(0);
let head_before_max = head_before_per_b.iter().max().copied().unwrap_or(0);
let head_before_sum: u64 = head_before_per_b.iter().map(|&h| h as u64).sum();
eprintln!(
    "── eval phase: {} steps on {} held-out files; pre-eval head per b_size={} accounts: \
     min={} max={} sum={} ──",
    cli.n_eval_steps, eval_files.len(), head_before_per_b.len(),
    head_before_min, head_before_max, head_before_sum
);
```

- [ ] **Step 3: Replace the post-eval slicing block (around lines 1402-1413)**

```rust
// BEFORE:
let all_records = sim
    .read_trade_records(0)
    .context("read trade records post-eval")?;
let head_before_usize = head_before_eval as usize;
let eval_records: Vec<_> = if all_records.len() > head_before_usize {
    all_records[head_before_usize..].to_vec()
} else {
    eprintln!(
        "warning: trade log wrapped — head_before={} but only {} records readable",
        head_before_usize, all_records.len()
    );
    all_records
};

// AFTER:
let all_per_b = sim
    .read_trade_records_all()
    .context("read all backtests' trade records post-eval")?;
let head_after_per_b = sim
    .read_per_backtest_trade_counts()
    .context("snapshot per-backtest head post-eval")?;

let cap = ml_backtesting::lob::TRADE_LOG_CAP as u32;
let mut eval_records: Vec<ml_backtesting::order::TradeRecord> = Vec::new();
let mut n_eval_trades_dropped: u64 = 0;
let mut n_pre_eval_wrapped: u64 = 0;
let mut n_eval_trades_seen: u64 = 0;

for (b, records) in all_per_b.iter().enumerate() {
    let head_before = head_before_per_b[b];
    let head_after = head_after_per_b[b];
    let eval_count_total = head_after.saturating_sub(head_before);
    n_eval_trades_seen += eval_count_total as u64;

    // The ring's contents cover cumulative-stream indices
    // [max(0, head_after - cap), head_after).
    let ring_start_stream_idx = head_after.saturating_sub(cap);

    if head_before < ring_start_stream_idx {
        // Some pre-eval trades wrapped out before eval — informational.
        n_pre_eval_wrapped += (ring_start_stream_idx - head_before) as u64;
    }

    if eval_count_total > cap {
        // Some eval trades wrapped during eval.
        n_eval_trades_dropped += (eval_count_total - cap) as u64;
    }

    // Slice the ring to eval-only.
    // ring index of first eval trade = head_before − ring_start_stream_idx
    // (clamped at 0 if all pre-eval already wrapped out).
    let eval_start_in_ring = head_before
        .saturating_sub(ring_start_stream_idx) as usize;
    if eval_start_in_ring < records.len() {
        eval_records.extend_from_slice(&records[eval_start_in_ring..]);
    }
}

if n_eval_trades_dropped > 0 {
    eprintln!(
        "warning: {} eval trades wrapped out across {} accounts (TRADE_LOG_CAP={}); \
         summary based on {} captured eval trades",
        n_eval_trades_dropped, all_per_b.len(), cap, eval_records.len()
    );
}
if n_pre_eval_wrapped > 0 {
    eprintln!(
        "info: {} pre-eval trades had already wrapped before eval phase \
         (no effect on eval summary)",
        n_pre_eval_wrapped
    );
}
eprintln!(
    "eval phase trade accounting: n_eval_trades_seen={} n_captured={} n_dropped={}",
    n_eval_trades_seen, eval_records.len(), n_eval_trades_dropped
);
```

- [ ] **Step 4: Augment `eval_summary.json` with the new fields**

```rust
// The existing code is:
let eval_summary = ml_backtesting::artifacts::compute_summary(&eval_records, &pnl_curve);
// ... serde_json::to_writer_pretty writes eval_summary directly to file.

// Replace the writer block (alpha_rl_train.rs around line 1440) with:
let aggregated_summary = serde_json::json!({
    // Original compute_summary fields:
    "n_trades":          eval_summary.n_trades,
    "total_pnl_usd":     eval_summary.total_pnl_usd,
    "profit_factor":     eval_summary.profit_factor,
    "sharpe_ann":        eval_summary.sharpe_ann,
    "max_drawdown_usd":  eval_summary.max_drawdown_usd,
    "win_rate":          eval_summary.win_rate,
    // New aggregation-aware fields:
    "n_eval_trades_seen":          n_eval_trades_seen,
    "n_eval_trades_dropped":       n_eval_trades_dropped,
    "n_pre_eval_trades_wrapped":   n_pre_eval_wrapped,
    "b_size":                      cli.n_backtests,
});

let eval_summary_path = cli.out.join("eval_summary.json");
let f = std::fs::File::create(&eval_summary_path)
    .with_context(|| format!("create {}", eval_summary_path.display()))?;
serde_json::to_writer_pretty(f, &aggregated_summary)
    .with_context(|| format!("write {}", eval_summary_path.display()))?;
eprintln!("eval summary written: {}", eval_summary_path.display());

// Update the eprintln summary line to include new fields:
eprintln!(
    "eval summary: n_trades={} pnl_usd={:.2} pf={:.3} sharpe_ann={:.3} \
     max_dd_usd={:.2} win_rate={:.3} | seen={} dropped={} b={}",
    eval_summary.n_trades, eval_summary.total_pnl_usd, eval_summary.profit_factor,
    eval_summary.sharpe_ann, eval_summary.max_drawdown_usd, eval_summary.win_rate,
    n_eval_trades_seen, n_eval_trades_dropped, cli.n_backtests
);
```

- [ ] **Step 5: cargo check + build**

```bash
SQLX_OFFLINE=true cargo build --release --example alpha_rl_train -p ml-alpha
```

Expected: clean (warning-free for the eval block).

- [ ] **Step 6: Commit**

```bash
git add crates/ml-alpha/examples/alpha_rl_train.rs
git commit -m "feat(rl): eval_summary aggregates across all b_size accounts (E.4)"
```

---

## E.5 — Local validation

**Files:**
- (No new files; runs existing alpha_rl_train binary)

- [ ] **Step 1: Run b=16, 1k train + 250 eval smoke**

```bash
mkdir -p /tmp/eval-agg-test
rm -rf /tmp/eval-agg-test/*
SQLX_OFFLINE=true target/release/examples/alpha_rl_train \
  --n-steps 1000 --n-eval-steps 250 \
  --fold-idx 1 --n-folds 3 \
  --mbp10-data-dir test_data/futures-baseline/ES.FUT \
  --predecoded-dir test_data/futures-baseline/ES.FUT \
  --out /tmp/eval-agg-test \
  --instrument-mode all \
  --n-backtests 16 \
  --log-every 100 \
  --seed 16962 \
  > /tmp/eval-agg-test/stderr.log 2>&1
echo "exit=$?"
```

- [ ] **Step 2: Inspect the new pre-eval boundary log line**

```bash
grep -E "pre-eval head per b_size" /tmp/eval-agg-test/stderr.log
```

Expected: a line like `── eval phase: 250 steps on 3 held-out files; pre-eval head per b_size=16 accounts: min=X max=Y sum=Z ──`. `sum` should be roughly the previous run's `head=600` order of magnitude (depending on policy behavior).

- [ ] **Step 3: Inspect the eval summary**

```bash
cat /tmp/eval-agg-test/eval_summary.json | jq .
grep "eval summary:" /tmp/eval-agg-test/stderr.log
```

Expected JSON shape:
```json
{
  "n_trades": <number, expect > 50 (was 58 single-account); roughly 16 × eval_avg_per_account ≈ 200-400>,
  "total_pnl_usd": <real number>,
  "profit_factor": <real number>,
  "sharpe_ann": <real number>,
  "max_drawdown_usd": <real number>,
  "win_rate": <real number>,
  "n_eval_trades_seen": <number, ≥ n_trades>,
  "n_eval_trades_dropped": 0,
  "n_pre_eval_trades_wrapped": 0,
  "b_size": 16
}
```

Assert:
- `n_trades > 50` (previous run was 58 single-account; aggregated should be 4–10× higher)
- `n_eval_trades_dropped == 0` (TRADE_LOG_CAP=4096 ≫ per-account count at b=16)
- `n_trades <= n_eval_trades_seen` (captured ≤ seen)

- [ ] **Step 4: Compare to previous (pre-fix) run for sanity**

```bash
# Previous (pre-fix, from earlier session):
#   n_trades=58 pnl_usd=-17287.56 pf=0.778 sharpe=-2.518 max_dd=23975.09 wr=0.414
# Current expectation: n_trades much larger, wr should be reasonable
#   (0.30-0.50 range typical for this seed), pnl scales with account count.
diff <(jq -S . /tmp/eval-agg-test/eval_summary.json) <(jq -S . /tmp/eval-math-test/eval_summary.json 2>/dev/null) || true
```

The diff is for informational comparison; expected to show n_trades / pnl change significantly.

- [ ] **Step 5: Commit any inline tweaks discovered during validation**

If the smoke surfaces a small issue (e.g., the `eval_start_in_ring` clamp needs adjusting), make a small follow-up commit:

```bash
git diff
# ... fix ...
git add crates/ml-alpha/examples/alpha_rl_train.rs
git commit -m "fix(rl): <specific tweak> from local validation (E.5)"
```

---

## E.6 — Cluster validation

- [ ] **Step 1: Push**

```bash
git push -u origin ml-alpha-eval-summary-fix
```

- [ ] **Step 2: Submit cluster run (same config as v11 for direct comparison)**

```bash
scripts/argo-alpha-rl.sh \
  --sha $(git rev-parse --short HEAD) \
  --branch ml-alpha-eval-summary-fix \
  --gpu-pool ci-training-l40s \
  --n-steps 20000 --n-backtests 1024 --seed 16962 \
  --instrument-mode front-month \
  --fold-idx 1 --n-folds 3 --n-eval-steps 5000
```

Note: this run uses the v11 cluster config exactly so eval_summary metrics are directly comparable.

- [ ] **Step 3: Monitor**

While running: every 5–10 min, sample the trainer pod for progress:
```bash
kubectl logs -n foxhunt alpha-rl-<wf> -c main --tail=5
```

- [ ] **Step 4: After completion, verify eval_summary.json on PVC**

```bash
# Spin up cleanup pod (PVC unlocked post-completion)
kubectl run e6-investigator -n foxhunt --image=alpine:latest --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"main","image":"alpine:latest","command":["sh","-c","apk add --no-cache jq; sleep 600"],"volumeMounts":[{"name":"feature-cache","mountPath":"/feature-cache"}]}],"volumes":[{"name":"feature-cache","persistentVolumeClaim":{"claimName":"feature-cache-pvc"}}]}}'
sleep 15
kubectl exec -n foxhunt e6-investigator -- cat /feature-cache/alpha-rl-runs/<sha>/fold1/eval_summary.json
```

Expected:
- `n_trades` in the **hundreds of thousands** (was 1024 in v11)
- `n_eval_trades_seen` > 300k
- `n_eval_trades_dropped` == 0 (TRADE_LOG_CAP=4096 sufficient)
- `b_size` == 1024
- `total_pnl_usd`, `profit_factor`, `max_drawdown_usd`, `win_rate` are now **interpretable** values across the whole eval phase

- [ ] **Step 5: Cleanup investigator pod**

```bash
kubectl delete pod e6-investigator -n foxhunt --wait=false
```

- [ ] **Step 6: Save a pearl documenting the bug class + fix**

```markdown
# /home/jgrusewski/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/pearl_eval_summary_aggregation_bug.md
---
name: pearl-eval-summary-aggregation-bug
description: eval_summary.json was computed on 1024 single-backtest records mixing train+eval trades, not aggregated across b_size; v10/v11 fold-1 comparison was meaningless.
metadata:
  type: pearl
---
[body documenting the 3 flaws + fix + cluster-vs-local validation numbers]
```

- [ ] **Step 7: Update MEMORY.md index**

Add a one-line entry under "Patterns / Pearls" linking to the new pearl.

- [ ] **Step 8: Commit pearl + index update**

```bash
# (pearl files live outside the repo; the MEMORY.md index update is also outside)
# No commit needed in the repo for the pearl.
```

---

## Self-review checklist

- [ ] E.1 and E.2 lobsim methods both call `self.stream.synchronize()` before DtoH
- [ ] E.3 bump didn't break any existing test that hardcodes 1024
- [ ] E.4 eval slice handles three edge cases:
  - All-eval-fit: `head_after - head_before ≤ cap` → no drop
  - Eval-wraps: `head_after - head_before > cap` → drop counted
  - Pre-eval-wraps: `head_before < head_after - cap` → some pre-eval already gone, doesn't affect eval slice
- [ ] No `unwrap()` in new code
- [ ] Eval summary keys preserved (n_trades, total_pnl_usd, profit_factor, sharpe_ann, max_drawdown_usd, win_rate) for downstream consumers
- [ ] New fields are additive

## Done means

- E.1–E.5 implemented + committed
- Local smoke shows `n_trades > 50` at b=16 (was 58 single-account; expect 200-400 aggregated)
- Cluster run produces `eval_summary.json` with `n_trades` in hundreds of thousands and `n_eval_trades_dropped == 0`
- Pearl saved + MEMORY.md updated
- Branch ready to merge into `ml-alpha-regime-observer`
