foxhunt/docs/superpowers/plans/2026-05-31-eval-summary-trade-aggregation.md

Eval-Summary Trade Aggregation Fix — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: superpowers:subagent-driven-development. Steps use checkbox - [ ] syntax.

Spec: docs/superpowers/specs/2026-05-31-eval-summary-trade-aggregation-design.md

Goal: eval_summary.json reflects ALL b_size accounts' eval-phase-only trades, with no train-phase contamination. Currently it computes metrics on backtest 0's last 1024 records (a mix of train + eval trades).

Architecture: Add two methods to LobSimCuda (read per-backtest counts + read all backtests' records). Replace the broken slicing block in alpha_rl_train.rs. Bump TRADE_LOG_CAP from 1024 → 4096 for cluster-scale headroom.

Tech Stack: Rust, cudarc.

Branch: ml-alpha-eval-summary-fix (off ml-alpha-regime-observer)

Independent of Phase A — touches different files (lobsim + eval slicing block, not the train-loop json!{}). Can run in parallel via separate subagent dispatch but must NOT branch off the in-flight ml-alpha-checkpoints-eval-diag branch (would conflict on alpha_rl_train.rs around the eval phase). Use a separate base.

---

File structure

Path	Action
crates/ml-backtesting/src/lob/mod.rs	Modify — bump TRADE_LOG_CAP from 1024 to 4096
crates/ml-backtesting/src/sim/mod.rs	Modify — add 2 new public methods on LobSimCuda
crates/ml-alpha/examples/alpha_rl_train.rs	Modify — replace slicing block with per-account aggregation
crates/ml-backtesting/tests/eval_aggregation_smoke.rs	Create — unit test for the new methods

---

E.1 — Lobsim API: read_per_backtest_trade_counts

Files:

• Modify: crates/ml-backtesting/src/sim/mod.rs

• Step 1: Locate current read_trade_records for reference

grep -nE "pub fn read_trade_records|pub fn read_total_trade_count" crates/ml-backtesting/src/sim/mod.rs

Expected: read_trade_records at ~line 1487 and read_total_trade_count somewhere nearby.

• Step 2: Add read_per_backtest_trade_counts

// crates/ml-backtesting/src/sim/mod.rs — in impl LobSimCuda { ... }
// Place immediately AFTER read_total_trade_count or read_trade_records.

/// Read per-backtest cumulative trade counters. Returns a Vec of length
/// `n_backtests` where entry `i` is the total number of closed-trade
/// records ever pushed to backtest `i`'s ring (may exceed TRADE_LOG_CAP).
///
/// Used by alpha_rl_train to snapshot the eval-phase boundary per
/// account: `head_before_per_b[i] = read_per_backtest_trade_counts()[i]`
/// at eval start, then post-eval `head_after_per_b[i] - head_before_per_b[i]`
/// is the true eval-phase trade count for backtest `i`.
pub fn read_per_backtest_trade_counts(&self) -> Result<Vec<u32>> {
    let mut heads = vec![0u32; self.n_backtests];
    self.stream.synchronize()
        .context("sync before reading trade_log_head_d")?;
    self.stream.memcpy_dtoh(&self.trade_log_head_d, heads.as_mut_slice())?;
    Ok(heads)
}

• Step 3: cargo check

SQLX_OFFLINE=true cargo check -p ml-backtesting

Expected: clean.

• Step 4: Commit

git add crates/ml-backtesting/src/sim/mod.rs
git commit -m "feat(lobsim): read_per_backtest_trade_counts (E.1)"

---

E.2 — Lobsim API: read_trade_records_all

Files:

• Modify: crates/ml-backtesting/src/sim/mod.rs

• Step 1: Add read_trade_records_all

// crates/ml-backtesting/src/sim/mod.rs — immediately after read_per_backtest_trade_counts

/// Read all backtests' trade record rings. Returns Vec<Vec<TradeRecord>>
/// of length `n_backtests`; entry `i` is the most recent up-to-
/// TRADE_LOG_CAP records for backtest `i` (oldest dropped on wrap).
///
/// Single DtoH of the full trade_log_d tensor + per-backtest head
/// values, split into per-account slices. Cost: O(b_size × cap × 40 B)
/// — at b=1024, cap=4096 this is 167 MB transferred once.
///
/// Callers compose this with `read_per_backtest_trade_counts()` to
/// determine each account's slice into eval-phase-only records.
pub fn read_trade_records_all(
    &self,
) -> Result<Vec<Vec<crate::order::TradeRecord>>> {
    self.stream.synchronize()
        .context("sync before reading trade logs")?;

    let mut heads = vec![0u32; self.n_backtests];
    self.stream.memcpy_dtoh(&self.trade_log_head_d, heads.as_mut_slice())?;

    let rec_bytes = crate::lob::TRADE_RECORD_BYTES;
    let cap = crate::lob::TRADE_LOG_CAP;
    let mut raw = vec![0u8; self.n_backtests * cap * rec_bytes];
    self.stream.memcpy_dtoh(&self.trade_log_d, raw.as_mut_slice())?;

    let mut out = Vec::with_capacity(self.n_backtests);
    for b in 0..self.n_backtests {
        let head = heads[b] as usize;
        let n_to_read = head.min(cap);
        let off = b * cap * rec_bytes;
        let mut records = Vec::with_capacity(n_to_read);
        for i in 0..n_to_read {
            let rec_off = off + i * rec_bytes;
            let r: crate::order::TradeRecord =
                bytemuck::pod_read_unaligned(&raw[rec_off..rec_off + rec_bytes]);
            records.push(r);
        }
        out.push(records);
    }
    Ok(out)
}

• Step 2: cargo check

SQLX_OFFLINE=true cargo check -p ml-backtesting

Expected: clean.

• Step 3: Commit

git add crates/ml-backtesting/src/sim/mod.rs
git commit -m "feat(lobsim): read_trade_records_all (E.2)"

---

E.3 — TRADE_LOG_CAP bump

Files:

• Modify: crates/ml-backtesting/src/lob/mod.rs

• Step 1: Verify the constant is referenced nowhere as hardcoded 1024

grep -rn "1024" crates/ml-backtesting/src/ | grep -iE "trade_log|TradeLog" | head -10

Expected: only pub const TRADE_LOG_CAP: usize = 1024; itself. Any other hits are flagged for review.

• Step 2: Bump the constant

// crates/ml-backtesting/src/lob/mod.rs:20
// BEFORE:
pub const TRADE_LOG_CAP: usize = 1024;
// AFTER:
pub const TRADE_LOG_CAP: usize = 4096;  // 2026-05-31: bumped from 1024 to prevent
                                        // eval-phase trade ring wrap at b=1024 cluster scale
                                        // (typical eval trades/account ≈ 342, peak ~1000;
                                        // 4096 gives 4× headroom). Memory: b × cap × 40 B
                                        // = 1024 × 4096 × 40 = 167 MB on GPU (was 41 MB).
                                        // See spec 2026-05-31-eval-summary-trade-aggregation.

• Step 3: cargo build (to catch any test asserting the old value)

SQLX_OFFLINE=true cargo build -p ml-backtesting
SQLX_OFFLINE=true cargo build -p ml-alpha

Expected: clean.

• Step 4: Run existing lobsim tests to verify no regression

SQLX_OFFLINE=true cargo test -p ml-backtesting --lib 2>&1 | tail -10

Expected: all tests pass (or only-failing tests are known-failing for unrelated reasons).

• Step 5: Commit

git add crates/ml-backtesting/src/lob/mod.rs
git commit -m "feat(lobsim): bump TRADE_LOG_CAP 1024 → 4096 for eval headroom (E.3)"

---

E.4 — alpha_rl_train.rs: replace slicing block with aggregation

Files:

• Modify: crates/ml-alpha/examples/alpha_rl_train.rs

This is the core fix. Replace the existing broken block with per-account aggregation that correctly isolates eval-phase trades.

• Step 1: Locate the existing block

grep -nE "head_before_eval|read_trade_records\(0\)|trade log wrapped" crates/ml-alpha/examples/alpha_rl_train.rs

Expected: the block lives roughly at lines 743 (pre-eval snapshot) and 1402-1413 (post-eval slicing).

• Step 2: Replace pre-eval snapshot (around line 743)

// BEFORE:
let head_before_eval = sim
    .read_total_trade_count()
    .context("read trade count pre-eval")?;
eprintln!(
    "── eval phase: {} steps on {} held-out files (trade-record checkpoint head={}) ──",
    cli.n_eval_steps, eval_files.len(), head_before_eval
);

// AFTER:
let head_before_per_b = sim
    .read_per_backtest_trade_counts()
    .context("snapshot per-backtest head pre-eval")?;
let head_before_min = head_before_per_b.iter().min().copied().unwrap_or(0);
let head_before_max = head_before_per_b.iter().max().copied().unwrap_or(0);
let head_before_sum: u64 = head_before_per_b.iter().map(|&h| h as u64).sum();
eprintln!(
    "── eval phase: {} steps on {} held-out files; pre-eval head per b_size={} accounts: \
     min={} max={} sum={} ──",
    cli.n_eval_steps, eval_files.len(), head_before_per_b.len(),
    head_before_min, head_before_max, head_before_sum
);

• Step 3: Replace the post-eval slicing block (around lines 1402-1413)

// BEFORE:
let all_records = sim
    .read_trade_records(0)
    .context("read trade records post-eval")?;
let head_before_usize = head_before_eval as usize;
let eval_records: Vec<_> = if all_records.len() > head_before_usize {
    all_records[head_before_usize..].to_vec()
} else {
    eprintln!(
        "warning: trade log wrapped — head_before={} but only {} records readable",
        head_before_usize, all_records.len()
    );
    all_records
};

// AFTER:
let all_per_b = sim
    .read_trade_records_all()
    .context("read all backtests' trade records post-eval")?;
let head_after_per_b = sim
    .read_per_backtest_trade_counts()
    .context("snapshot per-backtest head post-eval")?;

let cap = ml_backtesting::lob::TRADE_LOG_CAP as u32;
let mut eval_records: Vec<ml_backtesting::order::TradeRecord> = Vec::new();
let mut n_eval_trades_dropped: u64 = 0;
let mut n_pre_eval_wrapped: u64 = 0;
let mut n_eval_trades_seen: u64 = 0;

for (b, records) in all_per_b.iter().enumerate() {
    let head_before = head_before_per_b[b];
    let head_after = head_after_per_b[b];
    let eval_count_total = head_after.saturating_sub(head_before);
    n_eval_trades_seen += eval_count_total as u64;

    // The ring's contents cover cumulative-stream indices
    // [max(0, head_after - cap), head_after).
    let ring_start_stream_idx = head_after.saturating_sub(cap);

    if head_before < ring_start_stream_idx {
        // Some pre-eval trades wrapped out before eval — informational.
        n_pre_eval_wrapped += (ring_start_stream_idx - head_before) as u64;
    }

    if eval_count_total > cap {
        // Some eval trades wrapped during eval.
        n_eval_trades_dropped += (eval_count_total - cap) as u64;
    }

    // Slice the ring to eval-only.
    // ring index of first eval trade = head_before − ring_start_stream_idx
    // (clamped at 0 if all pre-eval already wrapped out).
    let eval_start_in_ring = head_before
        .saturating_sub(ring_start_stream_idx) as usize;
    if eval_start_in_ring < records.len() {
        eval_records.extend_from_slice(&records[eval_start_in_ring..]);
    }
}

if n_eval_trades_dropped > 0 {
    eprintln!(
        "warning: {} eval trades wrapped out across {} accounts (TRADE_LOG_CAP={}); \
         summary based on {} captured eval trades",
        n_eval_trades_dropped, all_per_b.len(), cap, eval_records.len()
    );
}
if n_pre_eval_wrapped > 0 {
    eprintln!(
        "info: {} pre-eval trades had already wrapped before eval phase \
         (no effect on eval summary)",
        n_pre_eval_wrapped
    );
}
eprintln!(
    "eval phase trade accounting: n_eval_trades_seen={} n_captured={} n_dropped={}",
    n_eval_trades_seen, eval_records.len(), n_eval_trades_dropped
);

• Step 4: Augment eval_summary.json with the new fields

// The existing code is:
let eval_summary = ml_backtesting::artifacts::compute_summary(&eval_records, &pnl_curve);
// ... serde_json::to_writer_pretty writes eval_summary directly to file.

// Replace the writer block (alpha_rl_train.rs around line 1440) with:
let aggregated_summary = serde_json::json!({
    // Original compute_summary fields:
    "n_trades":          eval_summary.n_trades,
    "total_pnl_usd":     eval_summary.total_pnl_usd,
    "profit_factor":     eval_summary.profit_factor,
    "sharpe_ann":        eval_summary.sharpe_ann,
    "max_drawdown_usd":  eval_summary.max_drawdown_usd,
    "win_rate":          eval_summary.win_rate,
    // New aggregation-aware fields:
    "n_eval_trades_seen":          n_eval_trades_seen,
    "n_eval_trades_dropped":       n_eval_trades_dropped,
    "n_pre_eval_trades_wrapped":   n_pre_eval_wrapped,
    "b_size":                      cli.n_backtests,
});

let eval_summary_path = cli.out.join("eval_summary.json");
let f = std::fs::File::create(&eval_summary_path)
    .with_context(|| format!("create {}", eval_summary_path.display()))?;
serde_json::to_writer_pretty(f, &aggregated_summary)
    .with_context(|| format!("write {}", eval_summary_path.display()))?;
eprintln!("eval summary written: {}", eval_summary_path.display());

// Update the eprintln summary line to include new fields:
eprintln!(
    "eval summary: n_trades={} pnl_usd={:.2} pf={:.3} sharpe_ann={:.3} \
     max_dd_usd={:.2} win_rate={:.3} | seen={} dropped={} b={}",
    eval_summary.n_trades, eval_summary.total_pnl_usd, eval_summary.profit_factor,
    eval_summary.sharpe_ann, eval_summary.max_drawdown_usd, eval_summary.win_rate,
    n_eval_trades_seen, n_eval_trades_dropped, cli.n_backtests
);

• Step 5: cargo check + build

SQLX_OFFLINE=true cargo build --release --example alpha_rl_train -p ml-alpha

Expected: clean (warning-free for the eval block).

• Step 6: Commit

git add crates/ml-alpha/examples/alpha_rl_train.rs
git commit -m "feat(rl): eval_summary aggregates across all b_size accounts (E.4)"

---

E.5 — Local validation

Files:

• (No new files; runs existing alpha_rl_train binary)

• Step 1: Run b=16, 1k train + 250 eval smoke

mkdir -p /tmp/eval-agg-test
rm -rf /tmp/eval-agg-test/*
SQLX_OFFLINE=true target/release/examples/alpha_rl_train \
  --n-steps 1000 --n-eval-steps 250 \
  --fold-idx 1 --n-folds 3 \
  --mbp10-data-dir test_data/futures-baseline/ES.FUT \
  --predecoded-dir test_data/futures-baseline/ES.FUT \
  --out /tmp/eval-agg-test \
  --instrument-mode all \
  --n-backtests 16 \
  --log-every 100 \
  --seed 16962 \
  > /tmp/eval-agg-test/stderr.log 2>&1
echo "exit=$?"

• Step 2: Inspect the new pre-eval boundary log line

grep -E "pre-eval head per b_size" /tmp/eval-agg-test/stderr.log

Expected: a line like ── eval phase: 250 steps on 3 held-out files; pre-eval head per b_size=16 accounts: min=X max=Y sum=Z ──. sum should be roughly the previous run's head=600 order of magnitude (depending on policy behavior).

• Step 3: Inspect the eval summary

cat /tmp/eval-agg-test/eval_summary.json | jq .
grep "eval summary:" /tmp/eval-agg-test/stderr.log

Expected JSON shape:

{
  "n_trades": <number, expect > 50 (was 58 single-account); roughly 16 × eval_avg_per_account ≈ 200-400>,
  "total_pnl_usd": <real number>,
  "profit_factor": <real number>,
  "sharpe_ann": <real number>,
  "max_drawdown_usd": <real number>,
  "win_rate": <real number>,
  "n_eval_trades_seen": <number, ≥ n_trades>,
  "n_eval_trades_dropped": 0,
  "n_pre_eval_trades_wrapped": 0,
  "b_size": 16
}

Assert:

• n_trades > 50 (previous run was 58 single-account; aggregated should be 4–10× higher)

• n_eval_trades_dropped == 0 (TRADE_LOG_CAP=4096 ≫ per-account count at b=16)

• n_trades <= n_eval_trades_seen (captured ≤ seen)

• Step 4: Compare to previous (pre-fix) run for sanity

# Previous (pre-fix, from earlier session):
#   n_trades=58 pnl_usd=-17287.56 pf=0.778 sharpe=-2.518 max_dd=23975.09 wr=0.414
# Current expectation: n_trades much larger, wr should be reasonable
#   (0.30-0.50 range typical for this seed), pnl scales with account count.
diff <(jq -S . /tmp/eval-agg-test/eval_summary.json) <(jq -S . /tmp/eval-math-test/eval_summary.json 2>/dev/null) || true

The diff is for informational comparison; expected to show n_trades / pnl change significantly.

• Step 5: Commit any inline tweaks discovered during validation

If the smoke surfaces a small issue (e.g., the eval_start_in_ring clamp needs adjusting), make a small follow-up commit:

git diff
# ... fix ...
git add crates/ml-alpha/examples/alpha_rl_train.rs
git commit -m "fix(rl): <specific tweak> from local validation (E.5)"

---

E.6 — Cluster validation

• Step 1: Push

git push -u origin ml-alpha-eval-summary-fix

• Step 2: Submit cluster run (same config as v11 for direct comparison)

scripts/argo-alpha-rl.sh \
  --sha $(git rev-parse --short HEAD) \
  --branch ml-alpha-eval-summary-fix \
  --gpu-pool ci-training-l40s \
  --n-steps 20000 --n-backtests 1024 --seed 16962 \
  --instrument-mode front-month \
  --fold-idx 1 --n-folds 3 --n-eval-steps 5000

Note: this run uses the v11 cluster config exactly so eval_summary metrics are directly comparable.

• Step 3: Monitor

While running: every 5–10 min, sample the trainer pod for progress:

kubectl logs -n foxhunt alpha-rl-<wf> -c main --tail=5

• Step 4: After completion, verify eval_summary.json on PVC

# Spin up cleanup pod (PVC unlocked post-completion)
kubectl run e6-investigator -n foxhunt --image=alpine:latest --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"main","image":"alpine:latest","command":["sh","-c","apk add --no-cache jq; sleep 600"],"volumeMounts":[{"name":"feature-cache","mountPath":"/feature-cache"}]}],"volumes":[{"name":"feature-cache","persistentVolumeClaim":{"claimName":"feature-cache-pvc"}}]}}'
sleep 15
kubectl exec -n foxhunt e6-investigator -- cat /feature-cache/alpha-rl-runs/<sha>/fold1/eval_summary.json

Expected:

• n_trades in the hundreds of thousands (was 1024 in v11)

• n_eval_trades_seen > 300k

• n_eval_trades_dropped == 0 (TRADE_LOG_CAP=4096 sufficient)

• b_size == 1024

• total_pnl_usd, profit_factor, max_drawdown_usd, win_rate are now interpretable values across the whole eval phase

• Step 5: Cleanup investigator pod

kubectl delete pod e6-investigator -n foxhunt --wait=false

• Step 6: Save a pearl documenting the bug class + fix

# /home/jgrusewski/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/pearl_eval_summary_aggregation_bug.md
---
name: pearl-eval-summary-aggregation-bug
description: eval_summary.json was computed on 1024 single-backtest records mixing train+eval trades, not aggregated across b_size; v10/v11 fold-1 comparison was meaningless.
metadata:
  type: pearl
---
[body documenting the 3 flaws + fix + cluster-vs-local validation numbers]

• Step 7: Update MEMORY.md index

Add a one-line entry under "Patterns / Pearls" linking to the new pearl.

• Step 8: Commit pearl + index update

# (pearl files live outside the repo; the MEMORY.md index update is also outside)
# No commit needed in the repo for the pearl.

---

Self-review checklist

• E.1 and E.2 lobsim methods both call self.stream.synchronize() before DtoH
• E.3 bump didn't break any existing test that hardcodes 1024
• E.4 eval slice handles three edge cases:
  • All-eval-fit: head_after - head_before ≤ cap → no drop
  • Eval-wraps: head_after - head_before > cap → drop counted
  • Pre-eval-wraps: head_before < head_after - cap → some pre-eval already gone, doesn't affect eval slice
• No unwrap() in new code
• Eval summary keys preserved (n_trades, total_pnl_usd, profit_factor, sharpe_ann, max_drawdown_usd, win_rate) for downstream consumers
• New fields are additive

Done means

• E.1–E.5 implemented + committed
• Local smoke shows n_trades > 50 at b=16 (was 58 single-account; expect 200-400 aggregated)
• Cluster run produces eval_summary.json with n_trades in hundreds of thousands and n_eval_trades_dropped == 0
• Pearl saved + MEMORY.md updated
• Branch ready to merge into ml-alpha-regime-observer