779c03b9d7
Spec: docs/superpowers/specs/2026-06-02-trainer-rollout-buffer-gae.md Plan: docs/superpowers/plans/2026-06-02-trainer-rollout-buffer-gae-implementation.md §Phase 1B-B Builds on commit5cd2f8703(1B-A foundation). Wires the RolloutBuffer into the per-step training loop via a new RolloutCollection helper, gated behind FOXHUNT_USE_ROLLOUT=1 env flag. Mode 0 (default unset) is bit-equal to HEAD; Mode 1 (flag=1) snapshots per-step data into the rollout buffer and invokes GAE every T_rollout steps. This phase is a STRUCTURAL validation, not an empirical pnl test. Phase 1B-C will replace the per-step training in Mode 1 with multi- epoch PPO over the rollout buffer. New components: * crates/ml-alpha/cuda/rollout_pack.cu (42 LOC): Single-thread-per-batch deterministic f32 -> u8 packer for dones. Used because RolloutBuffer.dones_d is u8 (per spec for memory) but the trainer's dones_d is f32 (per existing pipeline). * crates/ml-alpha/src/trainer/rollout_collection.rs (375 LOC): RolloutCollection helper with three methods: - new(ctx): loads rollout_pack cubin - snapshot_step(trainer, buf, t, b_size, t_max): DtoD scatter via cuMemcpy2DAsync_v2 for rewards/v_t/actions/log_pi/h_t at offset b*t_max+t in the [B × T] buffer; rollout_pack kernel for dones - copy_bootstrap_v(trainer, buf, b_size): DtoD V(s_T) from trainer.v_pred_tp1_d into buf.v_t_bootstrap_d at rollout end All transfers use mapped-pinned / DtoD only — no raw memcpy_dtoh on regular slices (`feedback_no_htod_htoh_only_mapped_pinned`). * crates/ml-alpha/examples/alpha_rl_train.rs (+137 LOC at lines 59-66, 410-466, 591-666): - FOXHUNT_USE_ROLLOUT env-flag parse at startup - Conditional RolloutBuffer + RolloutCollection allocation - Per-step snapshot_step() after step_with_lobsim_gpu - Every T_rollout steps: copy_bootstrap_v + compute_gae + stderr log `[rollout] step=N GAE complete (T=…, γ=…, λ=0.95): adv_mean=… adv_abs_mean=… returns_mean=…` Adaptations from the dispatch (documented inline): 1. T_rollout fallback to 32 when slot 758 is unbootstrapped. Slot 758 (RL_PPO_ROLLOUT_HORIZON_INDEX) was allocated in 1B-A but no kernel bootstraps it — `read_isv_host(758)` returns 0.0. Binary uses `.clamp(32, 1024)`, flooring to T=32 for the smoke. Phase 1B-C will add the trainer bootstrap entry (or a controller kernel) to make T_rollout=256 the production default. 2. Mode 1 still runs per-step training as a side effect; the rollout buffer is populated as a parallel snapshot. The "stop training in Mode 1" separation would require a ~1500 LOC refactor of step_with_lobsim_gpu (single-step/training-coupled per pearl_foxhunt_trainer_is_genuinely_single_step). The dispatch explicitly authorized this "skeleton with correct semantics" adaptation — the load-bearing 1B-B gate is "collection + GAE pipeline works structurally", which is verified. Phase 1B-C will replace the per-step training with multi-epoch PPO over the rollout buffer. Validation (all gates PASS): * cargo build --release --example alpha_rl_train -p ml-alpha: exit 0 (~60s) * cargo test rollout_buffer_invariants --release: 5/5 PASS (no regression) * Mode 0 (flag unset, default): ./scripts/determinism-check.sh --quick exit 0 - DETERMINISTIC: all checksums.* leaves match across all 200 rows - pipeline output bit-equal to HEAD5cd2f8703baseline * Mode 1 (FOXHUNT_USE_ROLLOUT=1): smoke completes in 2m 53s - 500 train + 100 eval steps, exit 0, completed_clean=true - 16 GAE windows logged (T=32 each) - adv_abs_mean range [4.4e-3, 1.92e-1] — always > 1e-3 sanity gate - eval_summary written: total_pnl_usd=-$101,487.70, n_trades=301, wr=0.346 * Cross-source consistency BOTH modes within $50 (Mode 0: $0.00 diff; Mode 1: $0.01 diff). Phase 0 contract preserved. * Pre-commit hook PASS (0 memcpy_dtoh raw violations, 0 atomicAdd). Next: Phase 1B-C will (a) bootstrap slot 758 to production T_rollout=256, (b) replace Mode 1's per-step training with multi-epoch PPO over the rollout buffer using the existing PPO surrogate + V regression kernels operating on minibatches of [B × T] flattened to [B*T]. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1483 lines
72 KiB
Rust
1483 lines
72 KiB
Rust
// The per-step diag `json!{...}` block now spans ≥ 130 fields after the
|
||
// adaptive-clamp additions, exceeding serde_json's default macro recursion
|
||
// budget of 128. Bumping at the crate root keeps the expansion within limits.
|
||
#![recursion_limit = "256"]
|
||
|
||
//! Integrated RL trainer CLI (DQN + PPO on shared Mamba2 -> CfC encoder).
|
||
//!
|
||
//! Drives `IntegratedTrainer::step_with_lobsim` against a real `LobSimCuda`
|
||
//! backed by MBP-10 windows loaded via `MultiHorizonLoader::next_sequence_pair`
|
||
//! (R2) for true `(s_t, s_{t+1})` adjacency. The trainer combines:
|
||
//! * R3: GPU-resident EMA + advantage/return kernels.
|
||
//! * R4: GPU Thompson + Double-DQN argmax + log π_old kernels.
|
||
//! * R5: 7 RL controllers + target-net soft update.
|
||
//! * R6: GPU-pure env step (extract_realized_pnl_delta, apply_reward_scale,
|
||
//! actions_to_market_targets).
|
||
//! * R7a/R7b: host-lift of every per-step orchestration loop.
|
||
//! * R7c-data: true `h_{t+1}` / `V(s_{t+1})` / Bellman target on `h_{t+1}`.
|
||
//! * R7d: PER push/sample + off-policy DQN with stop-grad on encoder.
|
||
//!
|
||
//! Emits `alpha_rl_train_summary.json` on exit. Gate G8 (NaN abort): if any
|
||
//! per-head loss is non-finite at any step, the binary exits with code 2 so
|
||
//! the cluster smoke harness can terminate the workflow on first divergence.
|
||
//!
|
||
//! Usage:
|
||
//! alpha_rl_train \
|
||
//! --mbp10-data-dir /data/futures-baseline-mbp10/ES.FUT \
|
||
//! --predecoded-dir /feature-cache/predecoded \
|
||
//! --out /feature-cache/alpha-rl-runs/<sha> \
|
||
//! --n-steps 1000 \
|
||
//! --seed 16962 \
|
||
//! --instrument-mode front-month \
|
||
//! --gpu-idx 0
|
||
//!
|
||
//! Plan-doc reference: docs/superpowers/plans/2026-05-23-integrated-rl-trainer-rebuild.md
|
||
//! Section R8 (CLI binary + Argo template + dispatcher).
|
||
|
||
use anyhow::{Context, Result};
|
||
use clap::Parser;
|
||
use data::providers::databento::dbn_parser::InstrumentFilter;
|
||
// Use the canonical Action enum size, NOT a literal. Caught 2026-05-24
|
||
// during SP20 P4 dogfood: a `0..9` range survived the N_ACTIONS=9→11
|
||
// bump and silently dropped HalfFlat samples (a9, a10 showed 0% in
|
||
// diag despite P_MIN=0.02 floor guaranteeing 2% each). Per the
|
||
// emerging meta-pattern: any constant that mirrors a kernel-side
|
||
// structural dimension MUST reference the Rust const, not duplicate
|
||
// the literal value.
|
||
use ml_alpha::rl::common::N_ACTIONS;
|
||
use ml_alpha::data::gpu_dataset::{GpuDataLoader, GpuDataset};
|
||
use ml_alpha::data::loader::{
|
||
discover_mbp10_files_sorted, MultiHorizonLoader, MultiHorizonLoaderConfig,
|
||
DEFAULT_OUTCOME_LABEL_COST_ES,
|
||
};
|
||
use ml_alpha::heads::HORIZONS;
|
||
// ISV slot imports — only the slots still read DIRECTLY in this file.
|
||
// Phase A of the 2026-05-31 checkpoints+eval-diag plan moved all ~140
|
||
// diag-block ISV reads behind `IntegratedTrainer::build_diag_value`, so
|
||
// the previous wall of imports collapsed to the four slots still in
|
||
// the per-N-step stderr ticker.
|
||
use ml_alpha::rl::isv_slots::{
|
||
RL_CONF_GATE_FIRED_COUNT_INDEX, RL_FRD_GATE_FIRED_COUNT_INDEX, RL_GAMMA_INDEX,
|
||
RL_HEAT_CAP_FIRED_COUNT_INDEX, RL_PER_ALPHA_INDEX, RL_PPO_CLIP_INDEX,
|
||
RL_PPO_ROLLOUT_HORIZON_INDEX, RL_PYRAMID_ADD_COUNT_INDEX, RL_REWARD_SCALE_INDEX,
|
||
};
|
||
use ml_alpha::heads::HIDDEN_DIM as ROLLOUT_HIDDEN_DIM;
|
||
use ml_alpha::trainer::diag_staging::DiagStaging;
|
||
use ml_alpha::trainer::integrated::{
|
||
read_slice_d_into, DiagInputs, IntegratedStepStats, IntegratedTrainer, IntegratedTrainerConfig,
|
||
};
|
||
use ml_alpha::trainer::perception::PerceptionTrainerConfig;
|
||
use ml_alpha::trainer::rollout_buffer::RolloutBuffer;
|
||
use ml_alpha::trainer::rollout_collection::RolloutCollection;
|
||
use ml_backtesting::sim::LobSimCuda;
|
||
use ml_core::device::MlDevice;
|
||
use serde::Serialize;
|
||
use std::io::{BufWriter, Write};
|
||
use std::path::PathBuf;
|
||
use std::process;
|
||
|
||
#[derive(Parser)]
|
||
#[command(name = "alpha_rl_train")]
|
||
struct Cli {
|
||
/// Directory containing MBP-10 .dbn.zst files. Per
|
||
/// `feedback_mbp10_mandatory`, this argument is required — no
|
||
/// synthetic-data fallback in the production training path.
|
||
#[arg(long)]
|
||
mbp10_data_dir: PathBuf,
|
||
|
||
/// Directory holding predecoded sidecar caches (created on miss).
|
||
#[arg(long)]
|
||
predecoded_dir: PathBuf,
|
||
|
||
/// Output directory for summary JSON + per-step log artefacts.
|
||
#[arg(long)]
|
||
out: PathBuf,
|
||
|
||
/// Total `step_with_lobsim` calls to run. The validation smoke
|
||
/// (audit prerequisite) ships with 1000; production scale-up runs
|
||
/// 50k+. Each call processes one `(s_t, s_{t+1})` sequence pair
|
||
/// across `n_backtests` parallel LobSim instances.
|
||
#[arg(long, default_value_t = 1000)]
|
||
n_steps: usize,
|
||
|
||
/// Encoder sequence length (snapshots per step). Must match
|
||
/// `PerceptionTrainerConfig::seq_len`. Default 32 matches the
|
||
/// validated single-scale `1:32` window from alpha_train.
|
||
#[arg(long, default_value_t = 32)]
|
||
seq_len: usize,
|
||
|
||
/// Batch dimension — number of parallel LobSim backtests run in
|
||
/// lockstep per step. Larger batches amortise the encoder forward
|
||
/// cost across more trajectories but push GPU memory pressure
|
||
/// linearly. Default 16 — fixes the b_size=1 signal starvation
|
||
/// per `pearl_b_size_1_signal_starvation_blocks_q_learning`
|
||
/// (Q stayed at uniform baseline ln(21)=3.04 across all b_size=1
|
||
/// smokes regardless of controller tuning; 16 parallel
|
||
/// transitions per Adam step give 16× the gradient signal). Was
|
||
/// 1 (validation-smoke default); production sweep can push to
|
||
/// 32-64 on L40S 48GB.
|
||
#[arg(long, default_value_t = 16)]
|
||
n_backtests: usize,
|
||
|
||
/// PER buffer capacity. At b_size=16, capacity/16 = unique steps
|
||
/// retained. 32768/16 = 2048 steps of replay depth.
|
||
#[arg(long, default_value_t = 32768)]
|
||
per_capacity: usize,
|
||
|
||
/// Random seed pinning RNG + Xavier init + ISV controller bootstrap
|
||
/// state. Per `pearl_scoped_init_seed_for_reproducibility`.
|
||
#[arg(long, default_value_t = 16962)]
|
||
seed: u64,
|
||
|
||
/// MBP-10 instrument filter — `all`, `front-month`, or `id=<N>`.
|
||
/// `front-month` is the canonical setting for ES (auto-detects the
|
||
/// dominant contract per file, handles quarterly roll).
|
||
#[arg(long, default_value = "front-month", value_parser = parse_instrument_mode)]
|
||
instrument_mode: InstrumentFilter,
|
||
|
||
/// CUDA device index. Single-GPU dev (RTX 3050 Ti) uses 0; cluster
|
||
/// L40S/H100 also 0 (one GPU per pod per the alpha-rl template).
|
||
#[arg(long, default_value_t = 0)]
|
||
gpu_idx: usize,
|
||
|
||
/// Round-trip trade cost in price units for the D-style outcome
|
||
/// labels (consumed by the perception aux head, not by the RL
|
||
/// path directly — but the loader needs it). Default 0.5 = 2 ticks
|
||
/// × $0.25/tick for ES.
|
||
#[arg(long, default_value_t = DEFAULT_OUTCOME_LABEL_COST_ES)]
|
||
outcome_label_cost: f32,
|
||
|
||
/// How often (in steps) to flush a progress line to stderr. The
|
||
/// summary JSON is written ONCE at the end; this controls in-flight
|
||
/// visibility only.
|
||
#[arg(long, default_value_t = 100)]
|
||
log_every: usize,
|
||
|
||
/// Walk-forward fold index for the multi-fold G8 gate (per
|
||
/// `pearl_single_window_oos_is_not_oos` — a single window is NOT
|
||
/// out-of-sample). Slices the MBP-10 file list into K equal-sized
|
||
/// blocks; fold k trains on blocks [0..k+1) and evaluates on block
|
||
/// [k+1] (purged walk-forward). At `fold_idx=0` the first block
|
||
/// is the train window, second is eval. At `fold_idx=K-2` the
|
||
/// trainer sees almost all the data and evaluates on the final
|
||
/// block. The dispatcher (`scripts/argo-alpha-rl.sh --n-folds K`)
|
||
/// fans out K-1 workflows, one per fold; the aggregator computes
|
||
/// mean ± SD of the per-fold profit_factor.
|
||
///
|
||
/// Default 0 + n-folds=1 = single-window mode (no eval split).
|
||
#[arg(long, default_value_t = 0)]
|
||
fold_idx: usize,
|
||
|
||
/// Total walk-forward fold count. Set with `--fold-idx` for the
|
||
/// multi-fold G8 gate. n_folds=1 disables the train/eval split
|
||
/// (single-window mode — only meaningful for smoke validation).
|
||
/// n_folds≥3 enables a real evaluation: train on first
|
||
/// `fold_idx + 1` of `n_folds` data blocks, evaluate on the next.
|
||
#[arg(long, default_value_t = 1)]
|
||
n_folds: usize,
|
||
|
||
/// Eval-phase step count. After the train phase completes its
|
||
/// `--n-steps`, the eval phase runs `--n-eval-steps` additional
|
||
/// steps on the held-out fold using the same trainer
|
||
/// (acknowledged minor in-eval learning contamination at b_size=1
|
||
/// — pure eval mode is a follow-up). LobSim trade records are
|
||
/// drained at the END so the summary reflects eval-phase trades
|
||
/// only (train-phase trades are flushed when the eval phase
|
||
/// starts).
|
||
///
|
||
/// 0 disables the eval phase entirely (single-window training
|
||
/// only — the summary's profit_factor will reflect the full run
|
||
/// including train).
|
||
#[arg(long, default_value_t = 0)]
|
||
n_eval_steps: usize,
|
||
|
||
/// Per-step diagnostic JSONL path. One record per step capturing:
|
||
/// * step number, wall time
|
||
/// * loss components (l_bce/q/pi/v/aux/total) + λs (5)
|
||
/// * all 7 RL controller outputs (γ τ ε coef n_roll per_α scale)
|
||
/// * all 5 per-head learning rates (lr_bce/q/pi/v/aux)
|
||
/// * all 7 EMA inputs the controllers consume
|
||
/// * replay buffer length
|
||
/// * per-step reward + done aggregates (DtoH b_size floats per stat)
|
||
/// * per-step action histogram (counts per action class)
|
||
///
|
||
/// Default: `<out>/diag.jsonl`. Pass `--diag-jsonl /dev/null` (or
|
||
/// any unwritable path that errors at open) to disable. The dump
|
||
/// happens AFTER each `step_with_lobsim` so it sees the freshly-
|
||
/// adapted ISV values. Writes are line-buffered + flushed every
|
||
/// `log_every` steps so a `tail -f` shows in-flight progress.
|
||
///
|
||
/// Critically: this provides the per-step diagnostic visibility
|
||
/// the cluster smoke needs to detect anomalies early. Without it,
|
||
/// the smoke is "blind" — only the final summary is observable,
|
||
/// and a controller drifting off-anchor mid-run is invisible until
|
||
/// it produces a non-finite loss (G8 NaN abort).
|
||
#[arg(long)]
|
||
diag_jsonl: Option<PathBuf>,
|
||
|
||
/// Per-step diagnostic JSONL path for the EVAL phase (Phase A of
|
||
/// the 2026-05-31 checkpoints+eval-diag plan). Same 643-leaf
|
||
/// schema (642 scalars + 1 boolean) as `--diag-jsonl` — produced by the SAME builder
|
||
/// (`IntegratedTrainer::build_diag_value`), guaranteeing parity
|
||
/// per `feedback_single_source_of_truth_no_duplicates`.
|
||
///
|
||
/// Motivation: cluster run alpha-rl-8ll7j completed with
|
||
/// +$61,513 final pnl but max_dd -$444,512 during the eval
|
||
/// phase — and the eval phase emitted ZERO per-step diag, so
|
||
/// the drawdown trajectory was invisible. This file unblocks
|
||
/// that diagnostic surface.
|
||
///
|
||
/// Default: `<out>/eval_diag.jsonl`. Only written when the eval
|
||
/// phase runs (`--n-eval-steps > 0` AND walk-forward enabled).
|
||
#[arg(long)]
|
||
eval_diag_jsonl: Option<PathBuf>,
|
||
}
|
||
|
||
fn parse_instrument_mode(s: &str) -> Result<InstrumentFilter, String> {
|
||
let trimmed = s.trim();
|
||
if trimmed.eq_ignore_ascii_case("all") {
|
||
return Ok(InstrumentFilter::All);
|
||
}
|
||
if trimmed.eq_ignore_ascii_case("front-month")
|
||
|| trimmed.eq_ignore_ascii_case("front_month")
|
||
{
|
||
return Ok(InstrumentFilter::FrontMonth);
|
||
}
|
||
if let Some(num) = trimmed.strip_prefix("id=") {
|
||
let id: u32 = num.parse().map_err(|e| {
|
||
format!("instrument-mode: expected `id=<u32>` but failed to parse '{num}': {e}")
|
||
})?;
|
||
return Ok(InstrumentFilter::Id(id));
|
||
}
|
||
Err(format!(
|
||
"instrument-mode: expected `all`, `front-month`, or `id=<N>`; got '{s}'"
|
||
))
|
||
}
|
||
|
||
#[derive(Serialize, Default)]
|
||
struct AlphaRlTrainSummary {
|
||
n_steps_planned: usize,
|
||
n_steps_completed: usize,
|
||
seq_len: usize,
|
||
n_backtests: usize,
|
||
per_capacity: usize,
|
||
seed: u64,
|
||
|
||
/// True iff `n_steps_completed == n_steps_planned` AND no NaN
|
||
/// abort fired. False if early-terminated for any reason (NaN,
|
||
/// loader EOF, panic — though panics also exit nonzero).
|
||
completed_clean: bool,
|
||
|
||
/// Final-step loss components (raw, not λ-weighted). All NaN if
|
||
/// trainer never returned a finite step.
|
||
final_l_bce: f32,
|
||
final_l_q: f32,
|
||
final_l_pi: f32,
|
||
final_l_v: f32,
|
||
final_l_aux: f32,
|
||
final_l_total: f32,
|
||
|
||
/// Replay buffer state at exit.
|
||
final_replay_len: usize,
|
||
|
||
/// Step at which the first NaN was observed, if any. -1 means
|
||
/// no NaN observed.
|
||
nan_abort_step: i64,
|
||
}
|
||
|
||
fn main() -> Result<()> {
|
||
tracing_subscriber::fmt()
|
||
.with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
|
||
.init();
|
||
|
||
let cli = Cli::parse();
|
||
|
||
std::fs::create_dir_all(&cli.out)
|
||
.with_context(|| format!("mkdir -p {}", cli.out.display()))?;
|
||
std::fs::create_dir_all(&cli.predecoded_dir)
|
||
.with_context(|| format!("mkdir -p {}", cli.predecoded_dir.display()))?;
|
||
|
||
let dev = MlDevice::cuda(cli.gpu_idx)
|
||
.with_context(|| format!("MlDevice::cuda({})", cli.gpu_idx))?;
|
||
eprintln!("CUDA device {} initialised", cli.gpu_idx);
|
||
|
||
// Loader: paired (s_t, s_{t+1}) sequences. inference_only=false so
|
||
// next_sequence_pair is available (the labels it preloads are
|
||
// unused by step_with_lobsim, but the API requires the label
|
||
// precomputation pass per `feedback_no_partial_refactor`).
|
||
let all_files = discover_mbp10_files_sorted(&cli.mbp10_data_dir)
|
||
.with_context(|| format!("discover_mbp10_files_sorted({})", cli.mbp10_data_dir.display()))?;
|
||
anyhow::ensure!(
|
||
!all_files.is_empty(),
|
||
"no MBP-10 .dbn.zst files under {}",
|
||
cli.mbp10_data_dir.display()
|
||
);
|
||
eprintln!("loader: {} MBP-10 files total", all_files.len());
|
||
|
||
// Walk-forward fold split per `pearl_single_window_oos_is_not_oos`.
|
||
// Slice the file list into `n_folds` equal blocks. fold k uses
|
||
// blocks [0..=k] for training and block [k+1] for evaluation. With
|
||
// n_folds=1 (default) train_files = all_files and eval_files is
|
||
// empty (single-window mode, no eval phase).
|
||
let (train_files, eval_files) = if cli.n_folds <= 1 || cli.n_eval_steps == 0 {
|
||
eprintln!("walk-forward: disabled (n_folds={}, n_eval_steps={})", cli.n_folds, cli.n_eval_steps);
|
||
(all_files, Vec::new())
|
||
} else {
|
||
anyhow::ensure!(
|
||
cli.fold_idx + 1 < cli.n_folds,
|
||
"fold_idx {} requires fold_idx + 1 < n_folds ({}); the last block must be the eval window",
|
||
cli.fold_idx,
|
||
cli.n_folds
|
||
);
|
||
let n_files = all_files.len();
|
||
let block_size = n_files / cli.n_folds;
|
||
anyhow::ensure!(
|
||
block_size >= 1,
|
||
"not enough MBP-10 files ({}) to split into n_folds={} (need ≥ n_folds files)",
|
||
n_files,
|
||
cli.n_folds
|
||
);
|
||
// Train: blocks 0..=fold_idx, Eval: block fold_idx+1.
|
||
let train_end = (cli.fold_idx + 1) * block_size;
|
||
let eval_end = (cli.fold_idx + 2) * block_size;
|
||
let train: Vec<_> = all_files[..train_end].to_vec();
|
||
let eval: Vec<_> = all_files[train_end..eval_end.min(n_files)].to_vec();
|
||
eprintln!(
|
||
"walk-forward fold {}/{}: train={} files ({}..{}), eval={} files ({}..{})",
|
||
cli.fold_idx, cli.n_folds, train.len(), 0, train_end,
|
||
eval.len(), train_end, eval_end.min(n_files)
|
||
);
|
||
(train, eval)
|
||
};
|
||
|
||
let multi_resolution: ml_alpha::data::aggregation::MultiResolutionConfig =
|
||
format!("1:{}", cli.seq_len)
|
||
.parse()
|
||
.with_context(|| format!("multi-resolution `1:{}`", cli.seq_len))?;
|
||
anyhow::ensure!(
|
||
multi_resolution.total_positions() == cli.seq_len,
|
||
"internal: single-scale 1:{} should produce total_positions={} but got {}",
|
||
cli.seq_len,
|
||
cli.seq_len,
|
||
multi_resolution.total_positions()
|
||
);
|
||
|
||
let loader_cfg = MultiHorizonLoaderConfig {
|
||
files: train_files,
|
||
predecoded_dir: cli.predecoded_dir.clone(),
|
||
multi_resolution: multi_resolution.clone(),
|
||
horizons: HORIZONS,
|
||
// Soft cap above n_steps × n_backtests × 2 so next_sequence_pair
|
||
// never returns Ok(None) mid-run. Each step samples n_backtests
|
||
// pairs (one per batch slot) to build the B×K snapshot tensor
|
||
// that forward_encoder consumes — per-batch market diversity is
|
||
// what gives the gradient-variance reduction promised by
|
||
// `pearl_b_size_1_signal_starvation_blocks_q_learning`.
|
||
n_max_sequences: cli.n_steps
|
||
.saturating_mul(cli.n_backtests.max(1))
|
||
.saturating_mul(2)
|
||
.max(64),
|
||
seed: cli.seed,
|
||
inference_only: false,
|
||
outcome_label_cost: cli.outcome_label_cost,
|
||
instrument_filter: cli.instrument_mode,
|
||
frd_horizon_ticks: ml_alpha::rl::common::FRD_HORIZON_TICKS,
|
||
frd_bucket_range_sigma: ml_alpha::rl::common::FRD_BUCKET_RANGE_SIGMA,
|
||
};
|
||
let loader = MultiHorizonLoader::new(&loader_cfg).context("MultiHorizonLoader::new")?;
|
||
|
||
// ── GPU-resident dataset upload. ────────────────────────────────
|
||
// Upload ALL pre-converted snapshots + FRD labels to GPU memory as
|
||
// a single contiguous device buffer. Per-step data loading becomes
|
||
// pure GPU kernel work (sample + gather) — zero CPU data work.
|
||
let gpu_dataset: GpuDataset = loader
|
||
.upload_to_gpu(&dev, cli.n_backtests.max(1), cli.seed)
|
||
.context("loader.upload_to_gpu")?;
|
||
let mut gpu_data_loader = GpuDataLoader::new(&dev, cli.n_backtests.max(1))
|
||
.context("GpuDataLoader::new")?;
|
||
eprintln!(
|
||
"GPU dataset uploaded: {} snapshots, {} files, {:.1} GB",
|
||
gpu_dataset.total_snapshots,
|
||
gpu_dataset.n_files,
|
||
(gpu_dataset.total_snapshots * 216) as f64 / 1e9,
|
||
);
|
||
|
||
// Trainer + LobSim.
|
||
let perception_cfg = PerceptionTrainerConfig {
|
||
seq_len: cli.seq_len,
|
||
n_batch: cli.n_backtests,
|
||
..PerceptionTrainerConfig::default()
|
||
};
|
||
let trainer_cfg = IntegratedTrainerConfig {
|
||
perception: perception_cfg,
|
||
dqn_seed: cli.seed.wrapping_add(0xDA),
|
||
ppo_seed: cli.seed.wrapping_add(0xBE),
|
||
per_capacity: cli.per_capacity,
|
||
per_seed: cli.seed.wrapping_add(0x9E37_79B9),
|
||
};
|
||
let mut trainer =
|
||
IntegratedTrainer::new(&dev, trainer_cfg).context("IntegratedTrainer::new")?;
|
||
let mut sim = LobSimCuda::new(cli.n_backtests, &dev).context("LobSimCuda::new")?;
|
||
eprintln!(
|
||
"trainer + LobSimCuda initialised: seq_len={} n_backtests={} per_capacity={}",
|
||
cli.seq_len, cli.n_backtests, cli.per_capacity
|
||
);
|
||
|
||
// ── Phase 1B-B rollout collection gate (FOXHUNT_USE_ROLLOUT). ────
|
||
//
|
||
// Spec: docs/superpowers/specs/2026-06-02-trainer-rollout-buffer-gae.md §1.2
|
||
// Plan: docs/superpowers/plans/2026-06-02-trainer-rollout-buffer-gae-implementation.md §Phase 1B-B
|
||
//
|
||
// TEMPORARY VALIDATION FLAG — per `feedback_no_feature_flags`,
|
||
// permanent feature flags are forbidden. This env toggle is the
|
||
// same pattern as `FOXHUNT_DETERMINISTIC` (Phase 2.3 determinism
|
||
// foundation). REMOVED in Phase 1B-E once rollout-buffer + GAE is
|
||
// the default path.
|
||
//
|
||
// Gate semantics:
|
||
// * `FOXHUNT_USE_ROLLOUT=0` (default, or unset): legacy path
|
||
// unchanged. RolloutBuffer NOT allocated. Bit-equal to HEAD.
|
||
// * `FOXHUNT_USE_ROLLOUT=1`: per-step snapshot into RolloutBuffer
|
||
// after each `step_with_lobsim_gpu`. Every T_rollout steps run
|
||
// GAE backward sweep + log advantage / returns stats to stderr.
|
||
// Training pipeline runs UNCHANGED (1B-B is structural validation
|
||
// of collection + GAE wiring, not training replacement — that's
|
||
// 1B-C).
|
||
let use_rollout: bool = std::env::var("FOXHUNT_USE_ROLLOUT")
|
||
.ok()
|
||
.and_then(|v| v.parse::<u8>().ok())
|
||
.map(|n| n > 0)
|
||
.unwrap_or(false);
|
||
|
||
let t_rollout_default: usize = {
|
||
let v = trainer.read_isv_host(RL_PPO_ROLLOUT_HORIZON_INDEX) as usize;
|
||
v.clamp(32, 1024)
|
||
};
|
||
|
||
let (mut rollout_buf, rollout_collection): (Option<RolloutBuffer>, Option<RolloutCollection>) =
|
||
if use_rollout {
|
||
let ctx = dev
|
||
.cuda_context()
|
||
.context("FOXHUNT_USE_ROLLOUT=1: dev.cuda_context()")?;
|
||
let stream = trainer.stream.clone();
|
||
let buf = RolloutBuffer::new(
|
||
ctx,
|
||
&stream,
|
||
cli.n_backtests,
|
||
t_rollout_default,
|
||
ROLLOUT_HIDDEN_DIM,
|
||
)
|
||
.context("FOXHUNT_USE_ROLLOUT=1: RolloutBuffer::new")?;
|
||
let coll = RolloutCollection::new(ctx)
|
||
.context("FOXHUNT_USE_ROLLOUT=1: RolloutCollection::new")?;
|
||
eprintln!(
|
||
"[rollout] FOXHUNT_USE_ROLLOUT=1 → RolloutBuffer + GAE active \
|
||
(B={} T={} HIDDEN_DIM={}, γ from ISV[{}], λ=0.95)",
|
||
cli.n_backtests,
|
||
t_rollout_default,
|
||
ROLLOUT_HIDDEN_DIM,
|
||
RL_GAMMA_INDEX,
|
||
);
|
||
(Some(buf), Some(coll))
|
||
} else {
|
||
eprintln!(
|
||
"[rollout] FOXHUNT_USE_ROLLOUT unset/=0 → legacy per-step path \
|
||
(RolloutBuffer not allocated; bit-equal to HEAD)"
|
||
);
|
||
(None, None)
|
||
};
|
||
|
||
let mut summary = AlphaRlTrainSummary {
|
||
n_steps_planned: cli.n_steps,
|
||
seq_len: cli.seq_len,
|
||
n_backtests: cli.n_backtests,
|
||
per_capacity: cli.per_capacity,
|
||
seed: cli.seed,
|
||
nan_abort_step: -1,
|
||
..Default::default()
|
||
};
|
||
let mut last_stats: Option<IntegratedStepStats> = None;
|
||
|
||
// ── Per-step JSONL diag writer. ──────────────────────────────────
|
||
// Default path: `<out>/diag.jsonl`. The smoke run's failure-mode
|
||
// catalogue (NaN abort, controller drift, replay stagnation, reward
|
||
// explosion) is detected via post-hoc inspection of this file —
|
||
// without it, the run is blind beyond the eprintln every-N-step
|
||
// ticks.
|
||
let diag_path = cli
|
||
.diag_jsonl
|
||
.clone()
|
||
.unwrap_or_else(|| cli.out.join("diag.jsonl"));
|
||
let diag_file = std::fs::File::create(&diag_path)
|
||
.with_context(|| format!("create diag jsonl {}", diag_path.display()))?;
|
||
let mut diag = BufWriter::new(diag_file);
|
||
eprintln!("per-step diag JSONL: {}", diag_path.display());
|
||
|
||
// Async non-blocking diagnostic staging: separate CUDA stream +
|
||
// double-buffered mapped-pinned memory. ALL per-step device reads
|
||
// (ISV, rewards, dones, actions, raw_rewards, trade_duration,
|
||
// outcome_ema, position_lots, pyramid_count, unit state x4,
|
||
// close_unit_index, frd_logits) use async DtoD copies — zero
|
||
// stream syncs on the training stream. The host reads the PREVIOUS
|
||
// step's buffer while current step copies run on the diag stream.
|
||
let mut diag_staging = DiagStaging::new(&dev, cli.n_backtests)
|
||
.context("DiagStaging::new")?;
|
||
|
||
// Per-step scratch (replaced each step via mapped-pinned helpers).
|
||
|
||
// ── Training loop. ───────────────────────────────────────────────
|
||
// Windowed action distribution — EMA-smoothed across recent steps
|
||
// so action_entropy is meaningful at b_size=1 (per-step act_hist
|
||
// is one-hot, H=0; windowed dist captures actual policy variety).
|
||
// α = 1/1000 → half-life ≈ 690 steps. Tracks the recent ~1k steps'
|
||
// action distribution; entropy of normalised windowed_act_hist is
|
||
// a real exploration signal.
|
||
// SP20 P4: action histogram width MUST track the Action enum
|
||
// size. Always reference the const (NOT a literal) — see
|
||
// import-site comment for the dogfood-caught literal-drift bug.
|
||
let mut windowed_act_hist: [f32; N_ACTIONS] = [0.0; N_ACTIONS];
|
||
const WINDOWED_ACT_ALPHA: f32 = 1.0 / 1000.0;
|
||
|
||
// Phase 0 reward-redesign (2026-06-02): SHAPED-reward cumulative (raw_rewards
|
||
// sum at close events). NOT in USD — pts × lots × Phase-5 shaping. The
|
||
// legacy field name (`realized_pnl_cum_usd`) is gone; this is the gradient-
|
||
// signal cumulative, useful for measuring reward-policy alignment via the
|
||
// Pearson(rewards.sum, Δpnl) signal but never to be confused with USD pnl.
|
||
let mut shaped_reward_close_event_cum: f64 = 0.0;
|
||
// Phase 0 reward-redesign (2026-06-02): authoritative USD pnl cumulative
|
||
// sourced from `pnl_track_step`'s `pnl_step_close_usd_d` buffer (same
|
||
// realised_pnl_usd_fp/100 arithmetic as `eval_summary.total_pnl_usd`).
|
||
// Cross-source consistent with the eval_summary aggregator within $0.01.
|
||
let mut realized_pnl_usd_cum: f64 = 0.0;
|
||
// Phase 0 reward-redesign (2026-06-02): direct-readback scratch buffer
|
||
// for the per-step `pnl_step_close_usd_d` view. `read_slice_d_into`
|
||
// runs a mapped-pinned DtoD + stream-sync on the trainer's main stream
|
||
// — same stream as `pnl_track_step`'s write, so CUDA stream ordering
|
||
// makes the read consistent without any explicit cross-stream barrier.
|
||
// Allocated once outside the loop and re-filled in-place each step
|
||
// (no per-step allocation cost).
|
||
let mut pnl_step_close_usd_host: Vec<f32> = vec![0.0_f32; cli.n_backtests];
|
||
let mut win_count: u64 = 0;
|
||
let mut total_trades: u64 = 0;
|
||
let mut hold_time_sum: f64 = 0.0;
|
||
let mut trail_fired_total: u64 = 0;
|
||
let mut trail_tighten_total: u64 = 0;
|
||
let mut trail_loosen_total: u64 = 0;
|
||
let mut pyramid_added_total: u64 = 0;
|
||
let mut partial_flat_total: u64 = 0;
|
||
let mut partial_flat_long_total: u64 = 0;
|
||
let mut partial_flat_short_total: u64 = 0;
|
||
let mut conf_gate_total: u64 = 0;
|
||
let mut frd_gate_total: u64 = 0;
|
||
let mut heat_cap_total: u64 = 0;
|
||
|
||
let t_start = std::time::Instant::now();
|
||
// Phase 1B-B: rollout write-cursor (counts steps within the CURRENT
|
||
// T_rollout window; resets to 0 after each GAE invocation). Only
|
||
// advances when `use_rollout` is true.
|
||
let mut rollout_t: usize = 0;
|
||
for step in 0..cli.n_steps {
|
||
// GPU-resident data loading: sample_and_gather + gather_next +
|
||
// gather_current + gather_frd_labels all run as GPU kernels
|
||
// inside step_with_lobsim_gpu. Zero CPU data work per step.
|
||
let stats = trainer
|
||
.step_with_lobsim_gpu(&mut gpu_data_loader, &gpu_dataset, &mut sim)
|
||
.with_context(|| format!("step_with_lobsim_gpu at step {step}"))?;
|
||
|
||
// ── Phase 1B-B rollout snapshot + GAE (gated). ───────────────
|
||
//
|
||
// Stream-ordered after step_with_lobsim_gpu's writes (same
|
||
// trainer.stream). The snapshot is DtoD-only (cuMemcpy2DAsync +
|
||
// pack kernel) — zero host roundtrip, zero atomic ops.
|
||
//
|
||
// Every T_rollout steps: invoke GAE backward sweep, log the
|
||
// advantage / returns summary stats to stderr, then reset the
|
||
// write cursor and `current_t`. This is STRUCTURAL VALIDATION
|
||
// per the 1B-B dispatch — the resulting advantages_d / returns_d
|
||
// are not yet consumed by training (1B-C wires that in).
|
||
if let (Some(buf), Some(coll)) = (rollout_buf.as_mut(), rollout_collection.as_ref()) {
|
||
coll.snapshot_step(
|
||
&mut trainer,
|
||
buf,
|
||
rollout_t,
|
||
cli.n_backtests,
|
||
t_rollout_default,
|
||
)
|
||
.with_context(|| {
|
||
format!("rollout snapshot_step at step={step} rollout_t={rollout_t}")
|
||
})?;
|
||
buf.current_t = rollout_t + 1;
|
||
rollout_t += 1;
|
||
|
||
if rollout_t >= t_rollout_default {
|
||
// Bootstrap V(s_T) from the last step's v_pred_tp1_d
|
||
// (V on h_{t+1}), then run GAE backward sweep.
|
||
coll.copy_bootstrap_v(&mut trainer, buf, cli.n_backtests)
|
||
.context("rollout copy_bootstrap_v")?;
|
||
let gamma_isv = trainer.read_isv_host(RL_GAMMA_INDEX);
|
||
// λ = 0.95 hardcoded for 1B-B (1B-C allocates a slot).
|
||
buf.compute_gae(&trainer.stream, gamma_isv, 0.95_f32)
|
||
.context("rollout compute_gae")?;
|
||
|
||
// Diagnostic: read advantages / returns means to verify
|
||
// the buffer was populated non-trivially. Uses
|
||
// `read_slice_d_into` (mapped-pinned, stream-ordered
|
||
// sync per `feedback_no_htod_htoh_only_mapped_pinned`).
|
||
let bt = cli.n_backtests * t_rollout_default;
|
||
let mut adv_host: Vec<f32> = vec![0.0; bt];
|
||
let mut ret_host: Vec<f32> = vec![0.0; bt];
|
||
read_slice_d_into(&trainer.stream, &buf.advantages_d, &mut adv_host)
|
||
.context("rollout read advantages_d")?;
|
||
read_slice_d_into(&trainer.stream, &buf.returns_d, &mut ret_host)
|
||
.context("rollout read returns_d")?;
|
||
let adv_mean: f64 =
|
||
adv_host.iter().map(|&x| x as f64).sum::<f64>() / bt as f64;
|
||
let adv_abs_mean: f64 =
|
||
adv_host.iter().map(|&x| (x as f64).abs()).sum::<f64>() / bt as f64;
|
||
let ret_mean: f64 =
|
||
ret_host.iter().map(|&x| x as f64).sum::<f64>() / bt as f64;
|
||
eprintln!(
|
||
"[rollout] step={step} GAE complete (T={t_rollout_default}, γ={gamma_isv:.4}, λ=0.95): \
|
||
adv_mean={adv_mean:.4e} adv_abs_mean={adv_abs_mean:.4e} returns_mean={ret_mean:.4e}"
|
||
);
|
||
|
||
rollout_t = 0;
|
||
buf.reset();
|
||
}
|
||
}
|
||
|
||
// Gate G8: NaN abort. Per `feedback_stop_on_anomaly` +
|
||
// `feedback_kill_runs_on_anomaly_quickly` the cluster smoke
|
||
// tail-watcher kills the workflow on non-zero exit; here we
|
||
// signal that with exit code 2 (distinct from clean exit 0
|
||
// and panic-derived exit codes).
|
||
let non_finite = !stats.l_total.is_finite()
|
||
|| !stats.l_bce.is_finite()
|
||
|| !stats.l_q.is_finite()
|
||
|| !stats.l_pi.is_finite()
|
||
|| !stats.l_v.is_finite()
|
||
|| !stats.l_aux.is_finite();
|
||
if non_finite {
|
||
eprintln!(
|
||
"G8 NaN ABORT at step {step}: l_bce={} l_q={} l_pi={} l_v={} l_aux={} l_total={}",
|
||
stats.l_bce, stats.l_q, stats.l_pi, stats.l_v, stats.l_aux, stats.l_total
|
||
);
|
||
summary.n_steps_completed = step;
|
||
summary.nan_abort_step = step as i64;
|
||
summary.completed_clean = false;
|
||
write_summary(&cli.out, &summary)?;
|
||
process::exit(2);
|
||
}
|
||
|
||
// Event-sync the diag snapshot and swap buffers. Previous step's
|
||
// async DtoD copies complete here. Placed AFTER step_with_lobsim
|
||
// so the diag copies have had the entire training step to finish —
|
||
// the event sync is instant (~0 us). On step 0 this is a no-op
|
||
// (event sync on a freshly-created event returns immediately).
|
||
diag_staging.sync_and_swap().context("diag sync_and_swap")?;
|
||
|
||
// Launch async DtoD copies of ALL diag buffers into the current
|
||
// staging buffer. Non-blocking on the training stream — the copies
|
||
// run on diag_staging's separate stream.
|
||
diag_staging
|
||
.snapshot_async(
|
||
trainer.isv_dev_ptr,
|
||
trainer.rewards_d.raw_ptr(),
|
||
trainer.dones_d.raw_ptr(),
|
||
trainer.actions_d.raw_ptr(),
|
||
trainer.raw_rewards_d.raw_ptr(),
|
||
trainer.trade_duration_emit_d.raw_ptr(),
|
||
trainer.outcome_ema_d.raw_ptr(),
|
||
trainer.prev_position_lots_d.raw_ptr(),
|
||
trainer.pyramid_units_count_d.raw_ptr(),
|
||
trainer.unit_entry_price_d.raw_ptr(),
|
||
trainer.unit_entry_step_d.raw_ptr(),
|
||
trainer.unit_lots_d.raw_ptr(),
|
||
trainer.unit_trail_distance_d.raw_ptr(),
|
||
trainer.close_unit_index_d.raw_ptr(),
|
||
trainer.frd_logits_d.raw_ptr(),
|
||
)
|
||
.context("diag snapshot_async")?;
|
||
|
||
// Phase 0 reward-redesign (2026-06-02): direct mapped-pinned readback
|
||
// of `pnl_step_close_usd_d` from the LobSim. `pnl_track_step` ran
|
||
// earlier in this step on the trainer's main stream; `read_slice_d_into`
|
||
// queues a DtoD + stream-sync on the same stream, so CUDA stream
|
||
// ordering guarantees the read sees the kernel's writes — no
|
||
// cross-stream sync gap. Routing through `diag_staging` was found
|
||
// to break same-seed bit-equality (the diag stream's DtoD raced
|
||
// the main stream's `raw_memset_d8_zero + pnl_track_step write`);
|
||
// a direct same-stream read avoids the staging entirely. Mega-graph
|
||
// compatible: this read runs AFTER the per-step pipeline finishes,
|
||
// so it stays outside the captured graph (one extra DtoD per step
|
||
// post-graph-replay).
|
||
read_slice_d_into(
|
||
&trainer.stream,
|
||
sim.pnl_step_close_usd_d(),
|
||
&mut pnl_step_close_usd_host,
|
||
)
|
||
.context("read pnl_step_close_usd_d (train)")?;
|
||
|
||
// ── Per-step diag dump (async staging reads). ────────────────
|
||
// ALL diag reads come from the DiagStaging double-buffer — zero
|
||
// GPU stalls on the training stream. On step 0 these report
|
||
// zeros (no prior snapshot); from step 1+ they report step N-1's
|
||
// data (one-step delay, acceptable for diagnostics).
|
||
let rewards_host = diag_staging.read_rewards();
|
||
let dones_host = diag_staging.read_dones();
|
||
let actions_raw = diag_staging.read_actions_raw();
|
||
let actions_host: Vec<i32> = actions_raw
|
||
.iter()
|
||
.map(|f| f.to_bits() as i32)
|
||
.collect();
|
||
let raw_rewards_host = diag_staging.read_raw_rewards();
|
||
let trade_duration_host = diag_staging.read_trade_duration();
|
||
let outcome_ema_host = diag_staging.read_outcome_ema();
|
||
let position_lots_host = diag_staging.read_position_lots();
|
||
let pyramid_count_host = diag_staging.read_pyramid_count();
|
||
let unit_entry_price_host = diag_staging.read_unit_entry_price();
|
||
let unit_entry_step_host = diag_staging.read_unit_entry_step();
|
||
let unit_lots_host = diag_staging.read_unit_lots();
|
||
let unit_trail_host = diag_staging.read_unit_trail();
|
||
let close_unit_index_host = diag_staging.read_close_unit_index();
|
||
|
||
// Per-step action histogram (counts) — still computed here because
|
||
// the windowed EMA and several per-step counters (tighten/loosen/
|
||
// half-flat) read directly from it. `build_diag_value` recomputes
|
||
// its own act_hist internally — the two are bit-equivalent.
|
||
let mut act_hist = [0u32; N_ACTIONS];
|
||
for &a in &actions_host {
|
||
if (a as usize) < N_ACTIONS {
|
||
act_hist[a as usize] += 1;
|
||
}
|
||
}
|
||
// EMA-smoothed windowed action distribution (cross-step state).
|
||
// Mutated here so the next step inherits the updated EMA;
|
||
// build_diag_value reads the post-update value via `windowed_act_hist`
|
||
// in DiagInputs.
|
||
let total_actions: u32 = act_hist.iter().sum();
|
||
if total_actions > 0 {
|
||
for i in 0..N_ACTIONS {
|
||
let p_step = (act_hist[i] as f32) / (total_actions as f32);
|
||
windowed_act_hist[i] = (1.0 - WINDOWED_ACT_ALPHA)
|
||
* windowed_act_hist[i]
|
||
+ WINDOWED_ACT_ALPHA * p_step;
|
||
}
|
||
}
|
||
let reward_sum: f32 = rewards_host.iter().sum();
|
||
let done_count: u32 = dones_host.iter().map(|&d| if d > 0.5 { 1 } else { 0 }).sum();
|
||
|
||
// Per-trade cumulative stats (surfer validation) — mutate the
|
||
// train-loop running counters (shaped_reward_close_event_cum /
|
||
// total_trades / win_count / hold_time_sum). build_diag_value receives
|
||
// the post-update values via DiagInputs.
|
||
//
|
||
// Phase 0 reward-redesign (2026-06-02): authoritative per-batch USD pnl
|
||
// delta sourced from `pnl_step_close_usd_d` via the direct-readback
|
||
// above. The dones/raw_rewards reads are still previous-step (diag
|
||
// staging double-buffer); the pnl read is CURRENT step. The cumulative
|
||
// total is direction-agnostic — running sum converges to
|
||
// eval_summary.total_pnl_usd within fp/100 precision regardless of
|
||
// alignment. The per-batch win attribution below (`if dones[b] > 0.5
|
||
// { win_count += pnl[b] > 0 }`) is best-effort diagnostic; the
|
||
// authoritative win count comes from the trade_records aggregation
|
||
// at the end of eval. The one-step misalignment between dones (prev)
|
||
// and pnl (curr) is statistically negligible over thousands of trades.
|
||
let pnl_step_usd_delta: f64 = pnl_step_close_usd_host.iter().map(|&v| v as f64).sum();
|
||
realized_pnl_usd_cum += pnl_step_usd_delta;
|
||
for b in 0..cli.n_backtests {
|
||
if dones_host[b] > 0.5 {
|
||
// Phase 0 reward-redesign: this is the SHAPED reward (pts × lots
|
||
// × Phase-5 shaping), NOT USD. Direct sum of raw_rewards exposes
|
||
// the gradient signal magnitude separate from authoritative pnl.
|
||
shaped_reward_close_event_cum += raw_rewards_host[b] as f64;
|
||
total_trades += 1;
|
||
if pnl_step_close_usd_host[b] > 0.0 { win_count += 1; }
|
||
hold_time_sum += trade_duration_host[b] as f64;
|
||
}
|
||
}
|
||
let win_rate = if total_trades > 0 { win_count as f64 / total_trades as f64 } else { 0.0 };
|
||
|
||
// Snapshot the ISV slots we read on the host side. Determinism
|
||
// Phase 1 (2026-06-02) turned `build_diag_value` into `&mut self`
|
||
// (needed to launch the checksum kernels), so the `isv` slice
|
||
// can no longer co-exist with the trainer mutable borrow.
|
||
// Copying the eight specific values we use is cheaper than
|
||
// re-grabbing the slice twice and keeps the diff local.
|
||
let isv_trail_fired_step = trainer.read_isv_host(RL_CONF_GATE_FIRED_COUNT_INDEX) as u64;
|
||
let isv_pyramid_add_step = trainer.read_isv_host(RL_PYRAMID_ADD_COUNT_INDEX) as u64;
|
||
let isv_conf_gate_step = trainer.read_isv_host(RL_CONF_GATE_FIRED_COUNT_INDEX) as u64;
|
||
let isv_frd_gate_step = trainer.read_isv_host(RL_FRD_GATE_FIRED_COUNT_INDEX) as u64;
|
||
let isv_heat_cap_step = trainer.read_isv_host(RL_HEAT_CAP_FIRED_COUNT_INDEX) as u64;
|
||
let isv_gamma = trainer.read_isv_host(RL_GAMMA_INDEX);
|
||
let isv_ppo_clip = trainer.read_isv_host(RL_PPO_CLIP_INDEX);
|
||
let isv_per_alpha = trainer.read_isv_host(RL_PER_ALPHA_INDEX);
|
||
let isv_reward_scale = trainer.read_isv_host(RL_REWARD_SCALE_INDEX);
|
||
|
||
// Per-step counters derived from action histogram + ISV diag slots.
|
||
let trail_fired_step = isv_trail_fired_step; // reusing — trail stop uses actions override
|
||
let tighten_step = act_hist[ml_alpha::rl::common::Action::TrailTighten as usize] as u64;
|
||
let loosen_step = act_hist[ml_alpha::rl::common::Action::TrailLoosen as usize] as u64;
|
||
let pyramid_add_step = isv_pyramid_add_step;
|
||
let half_flat_long_step = act_hist[ml_alpha::rl::common::Action::HalfFlatLong as usize] as u64;
|
||
let half_flat_short_step = act_hist[ml_alpha::rl::common::Action::HalfFlatShort as usize] as u64;
|
||
let partial_flat_step = half_flat_long_step + half_flat_short_step;
|
||
let conf_gate_step = isv_conf_gate_step;
|
||
let frd_gate_step = isv_frd_gate_step;
|
||
let heat_cap_step = isv_heat_cap_step;
|
||
|
||
trail_fired_total += trail_fired_step;
|
||
trail_tighten_total += tighten_step;
|
||
trail_loosen_total += loosen_step;
|
||
pyramid_added_total += pyramid_add_step;
|
||
partial_flat_total += partial_flat_step;
|
||
partial_flat_long_total += half_flat_long_step;
|
||
partial_flat_short_total += half_flat_short_step;
|
||
conf_gate_total += conf_gate_step;
|
||
frd_gate_total += frd_gate_step;
|
||
heat_cap_total += heat_cap_step;
|
||
|
||
// Build the per-step diag record via the single-source-of-truth
|
||
// builder on the trainer (Phase A of the 2026-05-31
|
||
// checkpoints+eval-diag plan). Same builder is invoked from the
|
||
// eval phase below, guaranteeing schema parity per
|
||
// `feedback_single_source_of_truth_no_duplicates`.
|
||
let frd_logits_host = diag_staging.read_frd_logits();
|
||
let diag_inputs = DiagInputs {
|
||
b_size: cli.n_backtests,
|
||
stats: &stats,
|
||
rewards: rewards_host,
|
||
dones: dones_host,
|
||
actions: &actions_host,
|
||
raw_rewards: raw_rewards_host,
|
||
trade_duration: trade_duration_host,
|
||
outcome_ema: outcome_ema_host,
|
||
position_lots: &position_lots_host,
|
||
pyramid_count: &pyramid_count_host,
|
||
unit_entry_price: unit_entry_price_host,
|
||
unit_entry_step: &unit_entry_step_host,
|
||
unit_lots: &unit_lots_host,
|
||
unit_trail: unit_trail_host,
|
||
close_unit_index: &close_unit_index_host,
|
||
frd_logits: frd_logits_host,
|
||
shaped_reward_close_event_cum,
|
||
realized_pnl_usd_delta: pnl_step_usd_delta,
|
||
realized_pnl_usd_cum,
|
||
total_trades,
|
||
win_count,
|
||
hold_time_sum,
|
||
trail_fired_total,
|
||
trail_tighten_total,
|
||
trail_loosen_total,
|
||
pyramid_added_total,
|
||
partial_flat_total,
|
||
partial_flat_long_total,
|
||
partial_flat_short_total,
|
||
conf_gate_total,
|
||
frd_gate_total,
|
||
heat_cap_total,
|
||
windowed_act_hist: &windowed_act_hist,
|
||
};
|
||
let record = trainer.build_diag_value(
|
||
step as u64,
|
||
t_start.elapsed().as_secs_f32(),
|
||
&diag_inputs,
|
||
).context("build_diag_value (train)")?;
|
||
writeln!(diag, "{}", record).context("diag: writeln jsonl")?;
|
||
|
||
if step % cli.log_every == 0 || step == cli.n_steps - 1 {
|
||
diag.flush().context("diag: flush")?;
|
||
let elapsed = t_start.elapsed().as_secs_f32();
|
||
let sps = if elapsed > 0.0 { (step + 1) as f32 / elapsed } else { 0.0 };
|
||
eprintln!(
|
||
"step {:>6}/{}: l_q={:.4} l_pi={:.4} l_v={:.4} l_total={:.4} \
|
||
γ={:.4} ε={:.4} per_α={:.4} scale={:.4} \
|
||
replay={} dones={} rew_sum={:.3} sps={:.0} pnl=${:.0} wr={:.2} elapsed={:.1}s",
|
||
step,
|
||
cli.n_steps,
|
||
stats.l_q,
|
||
stats.l_pi,
|
||
stats.l_v,
|
||
stats.l_total,
|
||
isv_gamma,
|
||
isv_ppo_clip,
|
||
isv_per_alpha,
|
||
isv_reward_scale,
|
||
trainer.gpu_replay.capacity.min(step + 1),
|
||
done_count,
|
||
reward_sum,
|
||
sps,
|
||
// Phase 0 reward-redesign (2026-06-02): authoritative USD pnl
|
||
// (same source as eval_summary.total_pnl_usd). Was previously
|
||
// pnl_cum_usd which un-scaled clamped rewards — not USD.
|
||
realized_pnl_usd_cum,
|
||
win_rate,
|
||
elapsed,
|
||
);
|
||
}
|
||
last_stats = Some(stats);
|
||
summary.n_steps_completed = step + 1;
|
||
}
|
||
diag.flush().context("diag: final flush")?;
|
||
|
||
// ── Eval phase (walk-forward G8). ────────────────────────────────
|
||
// After the train phase, run n_eval_steps additional step_with_lobsim
|
||
// calls on the held-out eval files. Trade records accumulated by
|
||
// LobSim during eval are isolated from train-phase records via the
|
||
// `head_before_eval` checkpoint — eval records are sliced as
|
||
// `all_records[head_before_eval..]` post-eval.
|
||
//
|
||
// Note: this is NOT pure-eval mode — step_with_lobsim still runs
|
||
// its learning machinery (Adam steps, PER updates). At b_size=1
|
||
// the per-step learning effect is small relative to the policy
|
||
// already accumulated in train; the eval profit_factor approximates
|
||
// the train-end policy's OOS performance. True pure-eval (forward
|
||
// only, no backward) is a follow-up architectural change.
|
||
if cli.n_eval_steps > 0 && !eval_files.is_empty() {
|
||
// Adaptive risk management (spec 2026-05-30): reset Layer 1
|
||
// session-level state on train→eval transition. Clears session
|
||
// pnl, DD-triggered flag, consec-loss counter, cooldown counter.
|
||
// Kelly + inventory EMAs intentionally preserved.
|
||
trainer
|
||
.reset_session_state()
|
||
.context("reset_session_state at train→eval boundary")?;
|
||
|
||
let head_before_per_b = sim
|
||
.read_per_backtest_trade_counts()
|
||
.context("snapshot per-backtest trade-count pre-eval")?;
|
||
let head_before_min = head_before_per_b.iter().min().copied().unwrap_or(0);
|
||
let head_before_max = head_before_per_b.iter().max().copied().unwrap_or(0);
|
||
let head_before_sum: u64 =
|
||
head_before_per_b.iter().map(|&h| h as u64).sum();
|
||
eprintln!(
|
||
"── eval phase: {} steps on {} held-out files; pre-eval head per b_size={} \
|
||
accounts: min={} max={} sum={} ──",
|
||
cli.n_eval_steps,
|
||
eval_files.len(),
|
||
head_before_per_b.len(),
|
||
head_before_min,
|
||
head_before_max,
|
||
head_before_sum
|
||
);
|
||
|
||
let eval_loader_cfg = MultiHorizonLoaderConfig {
|
||
files: eval_files.clone(),
|
||
predecoded_dir: cli.predecoded_dir.clone(),
|
||
multi_resolution,
|
||
horizons: HORIZONS,
|
||
n_max_sequences: cli.n_eval_steps
|
||
.saturating_mul(cli.n_backtests.max(1))
|
||
.saturating_mul(2)
|
||
.max(64),
|
||
// Different seed for eval so sequence sampling is independent
|
||
// of the train seed (deterministic given the eval seed).
|
||
seed: cli.seed.wrapping_add(0xE7AE),
|
||
inference_only: false,
|
||
outcome_label_cost: cli.outcome_label_cost,
|
||
instrument_filter: cli.instrument_mode,
|
||
frd_horizon_ticks: ml_alpha::rl::common::FRD_HORIZON_TICKS,
|
||
frd_bucket_range_sigma: ml_alpha::rl::common::FRD_BUCKET_RANGE_SIGMA,
|
||
};
|
||
let eval_loader = MultiHorizonLoader::new(&eval_loader_cfg)
|
||
.context("MultiHorizonLoader::new (eval)")?;
|
||
|
||
// Upload eval dataset to GPU (separate from train dataset).
|
||
let eval_gpu_dataset = eval_loader
|
||
.upload_to_gpu(&dev, cli.n_backtests.max(1), cli.seed.wrapping_add(0xE7AE))
|
||
.context("eval loader.upload_to_gpu")?;
|
||
let mut eval_gpu_loader = GpuDataLoader::new(&dev, cli.n_backtests.max(1))
|
||
.context("GpuDataLoader::new (eval)")?;
|
||
|
||
// ── Per-step eval-phase diag writer (Phase A of the 2026-05-31
|
||
// checkpoints+eval-diag plan). Same 643-leaf schema (642 scalars
|
||
// + 1 boolean) as train's diag.jsonl — emitted by the SAME builder.
|
||
// The eval-phase
|
||
// trajectory was previously invisible (alpha-rl-8ll7j: +$61k
|
||
// pnl, -$444k drawdown, zero per-step records).
|
||
let eval_diag_path = cli
|
||
.eval_diag_jsonl
|
||
.clone()
|
||
.unwrap_or_else(|| cli.out.join("eval_diag.jsonl"));
|
||
let eval_diag_file = std::fs::File::create(&eval_diag_path)
|
||
.with_context(|| format!("create eval diag jsonl {}", eval_diag_path.display()))?;
|
||
let mut eval_diag = BufWriter::new(eval_diag_file);
|
||
eprintln!("per-step eval diag JSONL: {}", eval_diag_path.display());
|
||
|
||
// ── Eval-phase running counters (independent of train counters)
|
||
// ───────────────────────────────────────────────────────────────
|
||
// reset_session_state above clears device-side Layer 1 state but
|
||
// does NOT zero the host-side counters maintained by THIS loop;
|
||
// they reflect the eval window only (matches eval_summary.json's
|
||
// post-checkpoint trade slicing).
|
||
let mut eval_windowed_act_hist: [f32; N_ACTIONS] = [0.0; N_ACTIONS];
|
||
// Phase 0 reward-redesign (2026-06-02): mirror of train-loop counters.
|
||
// `shaped_reward_close_event_cum` is sum of raw_rewards at close (pts
|
||
// × lots × Phase-5 shaping, NOT USD). `realized_pnl_usd_cum` is
|
||
// authoritative USD from `pnl_step_close_usd_d` (same source as
|
||
// eval_summary.total_pnl_usd), reset to 0 at the train→eval boundary
|
||
// so the last eval row matches eval_summary.total_pnl_usd within $50.
|
||
let mut eval_shaped_reward_close_event_cum: f64 = 0.0;
|
||
let mut eval_realized_pnl_usd_cum: f64 = 0.0;
|
||
let mut eval_win_count: u64 = 0;
|
||
let mut eval_total_trades: u64 = 0;
|
||
let mut eval_hold_time_sum: f64 = 0.0;
|
||
let mut eval_trail_fired_total: u64 = 0;
|
||
let mut eval_trail_tighten_total: u64 = 0;
|
||
let mut eval_trail_loosen_total: u64 = 0;
|
||
let mut eval_pyramid_added_total: u64 = 0;
|
||
let mut eval_partial_flat_total: u64 = 0;
|
||
let mut eval_partial_flat_long_total: u64 = 0;
|
||
let mut eval_partial_flat_short_total: u64 = 0;
|
||
let mut eval_conf_gate_total: u64 = 0;
|
||
let mut eval_frd_gate_total: u64 = 0;
|
||
let mut eval_heat_cap_total: u64 = 0;
|
||
|
||
for eval_step in 0..cli.n_eval_steps {
|
||
let stats = trainer
|
||
.step_with_lobsim_gpu(&mut eval_gpu_loader, &eval_gpu_dataset, &mut sim)
|
||
.with_context(|| format!("step_with_lobsim_gpu eval step {eval_step}"))?;
|
||
if !stats.l_total.is_finite() {
|
||
eprintln!("G8 NaN ABORT during eval at step {eval_step}");
|
||
process::exit(2);
|
||
}
|
||
|
||
// Mirror train-phase diag pipeline: sync prior staging copies,
|
||
// then enqueue async DtoD copies of THIS step's device buffers.
|
||
// Per `feedback_no_partial_refactor` the eval path uses the
|
||
// SAME staging machinery as train (separate diag stream, no
|
||
// sync on the training stream, one-step latency).
|
||
diag_staging.sync_and_swap().context("eval diag sync_and_swap")?;
|
||
diag_staging
|
||
.snapshot_async(
|
||
trainer.isv_dev_ptr,
|
||
trainer.rewards_d.raw_ptr(),
|
||
trainer.dones_d.raw_ptr(),
|
||
trainer.actions_d.raw_ptr(),
|
||
trainer.raw_rewards_d.raw_ptr(),
|
||
trainer.trade_duration_emit_d.raw_ptr(),
|
||
trainer.outcome_ema_d.raw_ptr(),
|
||
trainer.prev_position_lots_d.raw_ptr(),
|
||
trainer.pyramid_units_count_d.raw_ptr(),
|
||
trainer.unit_entry_price_d.raw_ptr(),
|
||
trainer.unit_entry_step_d.raw_ptr(),
|
||
trainer.unit_lots_d.raw_ptr(),
|
||
trainer.unit_trail_distance_d.raw_ptr(),
|
||
trainer.close_unit_index_d.raw_ptr(),
|
||
trainer.frd_logits_d.raw_ptr(),
|
||
)
|
||
.context("eval diag snapshot_async")?;
|
||
|
||
// Phase 0 reward-redesign (2026-06-02): direct mapped-pinned
|
||
// readback of `pnl_step_close_usd_d`. Same rationale as train
|
||
// loop (no staging routing — avoid the cross-stream race).
|
||
// Mirrors the buffer reused from train; resets to current eval
|
||
// step's per-batch USD pnl delta. Eval step 0 still receives a
|
||
// valid current-step read (NOT a previous-step warmup like the
|
||
// staging path) — but the eval-step==0 accumulator skip below
|
||
// preserves the train→eval boundary semantics: the prior
|
||
// train-loop's leftover close events would otherwise leak.
|
||
read_slice_d_into(
|
||
&trainer.stream,
|
||
sim.pnl_step_close_usd_d(),
|
||
&mut pnl_step_close_usd_host,
|
||
)
|
||
.context("read pnl_step_close_usd_d (eval)")?;
|
||
|
||
// Host-side reads from the (just-swapped) staging buffer.
|
||
let rewards_host = diag_staging.read_rewards();
|
||
let dones_host = diag_staging.read_dones();
|
||
let actions_raw = diag_staging.read_actions_raw();
|
||
let actions_host: Vec<i32> = actions_raw
|
||
.iter()
|
||
.map(|f| f.to_bits() as i32)
|
||
.collect();
|
||
let raw_rewards_host = diag_staging.read_raw_rewards();
|
||
let trade_duration_host = diag_staging.read_trade_duration();
|
||
let outcome_ema_host = diag_staging.read_outcome_ema();
|
||
let position_lots_host = diag_staging.read_position_lots();
|
||
let pyramid_count_host = diag_staging.read_pyramid_count();
|
||
let unit_entry_price_host = diag_staging.read_unit_entry_price();
|
||
let unit_entry_step_host = diag_staging.read_unit_entry_step();
|
||
let unit_lots_host = diag_staging.read_unit_lots();
|
||
let unit_trail_host = diag_staging.read_unit_trail();
|
||
let close_unit_index_host = diag_staging.read_close_unit_index();
|
||
let frd_logits_host = diag_staging.read_frd_logits();
|
||
|
||
// Per-step action histogram (mirrors train pipeline — needed
|
||
// for windowed-EMA + per-counter updates below).
|
||
let mut act_hist = [0u32; N_ACTIONS];
|
||
for &a in &actions_host {
|
||
if (a as usize) < N_ACTIONS {
|
||
act_hist[a as usize] += 1;
|
||
}
|
||
}
|
||
let total_actions: u32 = act_hist.iter().sum();
|
||
if total_actions > 0 {
|
||
for i in 0..N_ACTIONS {
|
||
let p_step = (act_hist[i] as f32) / (total_actions as f32);
|
||
eval_windowed_act_hist[i] = (1.0 - WINDOWED_ACT_ALPHA)
|
||
* eval_windowed_act_hist[i]
|
||
+ WINDOWED_ACT_ALPHA * p_step;
|
||
}
|
||
}
|
||
|
||
// Phase 0 reward-redesign (2026-06-02): authoritative USD pnl
|
||
// delta sourced from the direct-readback above (NOT staging).
|
||
// Each call to `pnl_track_step` zeros the buffer then writes only
|
||
// THIS step's per-batch close pnl, so `pnl_step_close_usd_host`
|
||
// reflects exactly eval_step's close events — no train→eval
|
||
// leakage (the prior staging route needed an eval_step==0 skip
|
||
// because the one-step delay surfaced the LAST train step's
|
||
// closes on eval step 0; direct readback eliminates that path).
|
||
//
|
||
// Note: `dones_host` / `raw_rewards_host` are still PREVIOUS-step
|
||
// (diag staging), so the per-batch `if dones[b] > 0.5 { ... }`
|
||
// win attribution is misaligned by one step against the
|
||
// current-step pnl. That misalignment is statistically
|
||
// negligible over `n_eval_steps × b_size` close events; the
|
||
// authoritative win count is recomputed from `trade_records`
|
||
// at the end of eval anyway.
|
||
let eval_pnl_step_usd_delta: f64 =
|
||
pnl_step_close_usd_host.iter().map(|&v| v as f64).sum();
|
||
eval_realized_pnl_usd_cum += eval_pnl_step_usd_delta;
|
||
for b in 0..cli.n_backtests {
|
||
if dones_host[b] > 0.5 {
|
||
// SHAPED reward (NOT USD) — gradient signal cumulative.
|
||
eval_shaped_reward_close_event_cum += raw_rewards_host[b] as f64;
|
||
eval_total_trades += 1;
|
||
if pnl_step_close_usd_host[b] > 0.0 { eval_win_count += 1; }
|
||
eval_hold_time_sum += trade_duration_host[b] as f64;
|
||
}
|
||
}
|
||
|
||
let isv = trainer.isv_host_slice();
|
||
// Per-step ISV-derived counters — same indices as train loop.
|
||
let trail_fired_step = isv[RL_CONF_GATE_FIRED_COUNT_INDEX] as u64;
|
||
let tighten_step = act_hist[ml_alpha::rl::common::Action::TrailTighten as usize] as u64;
|
||
let loosen_step = act_hist[ml_alpha::rl::common::Action::TrailLoosen as usize] as u64;
|
||
let pyramid_add_step = isv[RL_PYRAMID_ADD_COUNT_INDEX] as u64;
|
||
let half_flat_long_step = act_hist[ml_alpha::rl::common::Action::HalfFlatLong as usize] as u64;
|
||
let half_flat_short_step = act_hist[ml_alpha::rl::common::Action::HalfFlatShort as usize] as u64;
|
||
let partial_flat_step = half_flat_long_step + half_flat_short_step;
|
||
let conf_gate_step = isv[RL_CONF_GATE_FIRED_COUNT_INDEX] as u64;
|
||
let frd_gate_step = isv[RL_FRD_GATE_FIRED_COUNT_INDEX] as u64;
|
||
let heat_cap_step = isv[RL_HEAT_CAP_FIRED_COUNT_INDEX] as u64;
|
||
|
||
eval_trail_fired_total += trail_fired_step;
|
||
eval_trail_tighten_total += tighten_step;
|
||
eval_trail_loosen_total += loosen_step;
|
||
eval_pyramid_added_total += pyramid_add_step;
|
||
eval_partial_flat_total += partial_flat_step;
|
||
eval_partial_flat_long_total += half_flat_long_step;
|
||
eval_partial_flat_short_total += half_flat_short_step;
|
||
eval_conf_gate_total += conf_gate_step;
|
||
eval_frd_gate_total += frd_gate_step;
|
||
eval_heat_cap_total += heat_cap_step;
|
||
|
||
// Build inputs and emit the per-step record. Same builder as
|
||
// train → schema parity is guaranteed by construction.
|
||
let diag_inputs = DiagInputs {
|
||
b_size: cli.n_backtests,
|
||
stats: &stats,
|
||
rewards: rewards_host,
|
||
dones: dones_host,
|
||
actions: &actions_host,
|
||
raw_rewards: raw_rewards_host,
|
||
trade_duration: trade_duration_host,
|
||
outcome_ema: outcome_ema_host,
|
||
position_lots: &position_lots_host,
|
||
pyramid_count: &pyramid_count_host,
|
||
unit_entry_price: unit_entry_price_host,
|
||
unit_entry_step: &unit_entry_step_host,
|
||
unit_lots: &unit_lots_host,
|
||
unit_trail: unit_trail_host,
|
||
close_unit_index: &close_unit_index_host,
|
||
frd_logits: frd_logits_host,
|
||
shaped_reward_close_event_cum: eval_shaped_reward_close_event_cum,
|
||
realized_pnl_usd_delta: eval_pnl_step_usd_delta,
|
||
realized_pnl_usd_cum: eval_realized_pnl_usd_cum,
|
||
total_trades: eval_total_trades,
|
||
win_count: eval_win_count,
|
||
hold_time_sum: eval_hold_time_sum,
|
||
trail_fired_total: eval_trail_fired_total,
|
||
trail_tighten_total: eval_trail_tighten_total,
|
||
trail_loosen_total: eval_trail_loosen_total,
|
||
pyramid_added_total: eval_pyramid_added_total,
|
||
partial_flat_total: eval_partial_flat_total,
|
||
partial_flat_long_total: eval_partial_flat_long_total,
|
||
partial_flat_short_total: eval_partial_flat_short_total,
|
||
conf_gate_total: eval_conf_gate_total,
|
||
frd_gate_total: eval_frd_gate_total,
|
||
heat_cap_total: eval_heat_cap_total,
|
||
windowed_act_hist: &eval_windowed_act_hist,
|
||
};
|
||
// Eval-phase records use a separate step namespace: continue
|
||
// the train-phase step index (`cli.n_steps + eval_step`) so
|
||
// post-hoc tooling can concatenate train + eval JSONL and
|
||
// get a monotone step axis.
|
||
let record = trainer.build_diag_value(
|
||
(cli.n_steps + eval_step) as u64,
|
||
t_start.elapsed().as_secs_f32(),
|
||
&diag_inputs,
|
||
).context("build_diag_value (eval)")?;
|
||
writeln!(eval_diag, "{}", record).context("eval diag: writeln jsonl")?;
|
||
|
||
if eval_step % cli.log_every == 0 || eval_step == cli.n_eval_steps - 1 {
|
||
eval_diag.flush().context("eval diag: flush")?;
|
||
eprintln!(
|
||
"eval {:>5}/{}: l_total={:.4} elapsed={:.1}s",
|
||
eval_step, cli.n_eval_steps, stats.l_total,
|
||
t_start.elapsed().as_secs_f32()
|
||
);
|
||
}
|
||
}
|
||
// Phase 0 reward-redesign (2026-06-02): drain the eval staging
|
||
// pipeline and EMIT A FINAL DRAIN ROW. The other diag streams
|
||
// (dones / raw_rewards / actions / etc.) still have one-step
|
||
// double-buffer latency, so the LAST eval step's per-batch state
|
||
// is in the swapped-out half; this drain row surfaces it. The
|
||
// `realized_pnl_usd_*` fields are sourced from the direct-readback
|
||
// path which already captured the last eval step's pnl in the
|
||
// regular loop above — so the drain row's pnl delta is 0 and the
|
||
// cum carries forward unchanged. Cross-source consistency check
|
||
// (`signal_consistency` in tier1_5_verdict.py) reads THIS final
|
||
// row's `realized_pnl_usd_cum`.
|
||
diag_staging.sync_and_swap().context("eval drain sync_and_swap")?;
|
||
let drain_dones = diag_staging.read_dones();
|
||
let drain_raw_rewards = diag_staging.read_raw_rewards();
|
||
let drain_actions_raw = diag_staging.read_actions_raw();
|
||
let drain_actions: Vec<i32> = drain_actions_raw.iter().map(|f| f.to_bits() as i32).collect();
|
||
let drain_trade_duration = diag_staging.read_trade_duration();
|
||
let drain_outcome_ema = diag_staging.read_outcome_ema();
|
||
let drain_position_lots = diag_staging.read_position_lots();
|
||
let drain_pyramid_count = diag_staging.read_pyramid_count();
|
||
let drain_unit_entry_price = diag_staging.read_unit_entry_price();
|
||
let drain_unit_entry_step = diag_staging.read_unit_entry_step();
|
||
let drain_unit_lots = diag_staging.read_unit_lots();
|
||
let drain_unit_trail = diag_staging.read_unit_trail();
|
||
let drain_close_unit_index = diag_staging.read_close_unit_index();
|
||
let drain_frd_logits = diag_staging.read_frd_logits();
|
||
let drain_rewards = diag_staging.read_rewards();
|
||
// No additional pnl accumulated on the drain row — direct readback
|
||
// already captured the last eval step's pnl in the regular loop.
|
||
let drain_delta: f64 = 0.0;
|
||
for b in 0..cli.n_backtests {
|
||
if drain_dones[b] > 0.5 {
|
||
eval_shaped_reward_close_event_cum += drain_raw_rewards[b] as f64;
|
||
eval_total_trades += 1;
|
||
eval_hold_time_sum += drain_trade_duration[b] as f64;
|
||
}
|
||
}
|
||
// Emit the drain row using the same builder for schema parity.
|
||
// `stats` reused from the last eval loop iteration; non-close-event
|
||
// fields (ISV, loss components) reflect post-last-train-step state.
|
||
let drain_diag_inputs = DiagInputs {
|
||
b_size: cli.n_backtests,
|
||
stats: last_stats.as_ref().expect("at least one eval step ran"),
|
||
rewards: drain_rewards,
|
||
dones: drain_dones,
|
||
actions: &drain_actions,
|
||
raw_rewards: drain_raw_rewards,
|
||
trade_duration: drain_trade_duration,
|
||
outcome_ema: drain_outcome_ema,
|
||
position_lots: &drain_position_lots,
|
||
pyramid_count: &drain_pyramid_count,
|
||
unit_entry_price: drain_unit_entry_price,
|
||
unit_entry_step: &drain_unit_entry_step,
|
||
unit_lots: &drain_unit_lots,
|
||
unit_trail: drain_unit_trail,
|
||
close_unit_index: &drain_close_unit_index,
|
||
frd_logits: drain_frd_logits,
|
||
shaped_reward_close_event_cum: eval_shaped_reward_close_event_cum,
|
||
realized_pnl_usd_delta: drain_delta,
|
||
realized_pnl_usd_cum: eval_realized_pnl_usd_cum,
|
||
total_trades: eval_total_trades,
|
||
win_count: eval_win_count,
|
||
hold_time_sum: eval_hold_time_sum,
|
||
trail_fired_total: eval_trail_fired_total,
|
||
trail_tighten_total: eval_trail_tighten_total,
|
||
trail_loosen_total: eval_trail_loosen_total,
|
||
pyramid_added_total: eval_pyramid_added_total,
|
||
partial_flat_total: eval_partial_flat_total,
|
||
partial_flat_long_total: eval_partial_flat_long_total,
|
||
partial_flat_short_total: eval_partial_flat_short_total,
|
||
conf_gate_total: eval_conf_gate_total,
|
||
frd_gate_total: eval_frd_gate_total,
|
||
heat_cap_total: eval_heat_cap_total,
|
||
windowed_act_hist: &eval_windowed_act_hist,
|
||
};
|
||
let drain_step = (cli.n_steps + cli.n_eval_steps) as u64;
|
||
let drain_record = trainer.build_diag_value(
|
||
drain_step,
|
||
t_start.elapsed().as_secs_f32(),
|
||
&drain_diag_inputs,
|
||
).context("build_diag_value (eval drain)")?;
|
||
writeln!(eval_diag, "{}", drain_record).context("eval diag drain: writeln jsonl")?;
|
||
eval_diag.flush().context("eval diag: final flush after drain")?;
|
||
eprintln!(
|
||
"eval drain row emitted at step {drain_step}: delta=${drain_delta:.2} → \
|
||
realized_pnl_usd_cum=${eval_realized_pnl_usd_cum:.2}",
|
||
);
|
||
|
||
// Aggregate eval-phase trades across ALL b_size accounts, with
|
||
// correct per-account head_before slicing. Replaces the previous
|
||
// single-account read of backtest 0 + aggregate-vs-per-account
|
||
// scale mismatch (see spec
|
||
// docs/superpowers/specs/2026-05-31-eval-summary-trade-aggregation-design.md
|
||
// for the root-cause analysis).
|
||
let all_per_b = sim
|
||
.read_trade_records_all()
|
||
.context("read all backtests' trade records post-eval")?;
|
||
let head_after_per_b = sim
|
||
.read_per_backtest_trade_counts()
|
||
.context("snapshot per-backtest trade-count post-eval")?;
|
||
|
||
let cap_u32 = ml_backtesting::lob::TRADE_LOG_CAP as u32;
|
||
let mut eval_records: Vec<ml_backtesting::order::TradeRecord> = Vec::new();
|
||
let mut n_eval_trades_seen: u64 = 0;
|
||
let mut n_eval_trades_dropped: u64 = 0;
|
||
let mut n_pre_eval_wrapped: u64 = 0;
|
||
|
||
for (b, records) in all_per_b.iter().enumerate() {
|
||
let head_before = head_before_per_b[b];
|
||
let head_after = head_after_per_b[b];
|
||
let eval_count_total = head_after.saturating_sub(head_before);
|
||
n_eval_trades_seen += eval_count_total as u64;
|
||
|
||
// The ring's contents cover cumulative-stream indices
|
||
// [max(0, head_after - cap), head_after).
|
||
let ring_start_stream_idx = head_after.saturating_sub(cap_u32);
|
||
|
||
if head_before < ring_start_stream_idx {
|
||
// Some pre-eval trades had wrapped out before eval started —
|
||
// diagnostic only, doesn't affect eval slicing.
|
||
n_pre_eval_wrapped += (ring_start_stream_idx - head_before) as u64;
|
||
}
|
||
|
||
if eval_count_total > cap_u32 {
|
||
// Eval-phase trades wrapped (lost). Should be 0 with
|
||
// TRADE_LOG_CAP=4096 at typical cluster scale.
|
||
n_eval_trades_dropped += (eval_count_total - cap_u32) as u64;
|
||
}
|
||
|
||
// Slice the ring contents to eval-only.
|
||
// ring index of first eval trade = head_before − ring_start_stream_idx
|
||
// (clamped at 0 if all pre-eval already wrapped out).
|
||
let eval_start_in_ring =
|
||
head_before.saturating_sub(ring_start_stream_idx) as usize;
|
||
if eval_start_in_ring < records.len() {
|
||
eval_records.extend_from_slice(&records[eval_start_in_ring..]);
|
||
}
|
||
}
|
||
|
||
if n_eval_trades_dropped > 0 {
|
||
eprintln!(
|
||
"warning: {} eval trades wrapped out across {} accounts \
|
||
(TRADE_LOG_CAP={}); summary based on {} captured eval trades",
|
||
n_eval_trades_dropped,
|
||
all_per_b.len(),
|
||
cap_u32,
|
||
eval_records.len()
|
||
);
|
||
}
|
||
if n_pre_eval_wrapped > 0 {
|
||
eprintln!(
|
||
"info: {} pre-eval trades had already wrapped before eval phase \
|
||
(no effect on eval summary)",
|
||
n_pre_eval_wrapped
|
||
);
|
||
}
|
||
eprintln!(
|
||
"eval phase trade accounting: n_eval_trades_seen={} n_captured={} \
|
||
n_dropped={} b_size={}",
|
||
n_eval_trades_seen,
|
||
eval_records.len(),
|
||
n_eval_trades_dropped,
|
||
all_per_b.len()
|
||
);
|
||
|
||
// Synthesise a pnl_curve from per-trade cumulative PnL for
|
||
// compute_summary's max_drawdown calc.
|
||
let mut pnl_curve = Vec::with_capacity(eval_records.len());
|
||
let mut cum = 0.0f32;
|
||
for r in &eval_records {
|
||
cum += (r.realised_pnl_usd_fp as f32) / 100.0;
|
||
pnl_curve.push(cum);
|
||
}
|
||
let eval_summary =
|
||
ml_backtesting::artifacts::compute_summary(&eval_records, &pnl_curve);
|
||
|
||
eprintln!(
|
||
"eval summary: n_trades={} pnl_usd={:.2} pf={:.3} sharpe_ann={:.3} \
|
||
max_dd_usd={:.2} win_rate={:.3} | seen={} dropped={} b={}",
|
||
eval_summary.n_trades,
|
||
eval_summary.total_pnl_usd,
|
||
eval_summary.profit_factor,
|
||
eval_summary.sharpe_ann,
|
||
eval_summary.max_drawdown_usd,
|
||
eval_summary.win_rate,
|
||
n_eval_trades_seen,
|
||
n_eval_trades_dropped,
|
||
all_per_b.len(),
|
||
);
|
||
|
||
// Write eval_summary.json with the existing compute_summary fields
|
||
// PLUS four new aggregation-aware fields:
|
||
// n_eval_trades_seen — true total cumulative dones across b_size
|
||
// n_eval_trades_dropped — eval trades lost to ring wrap (0 at typical scale)
|
||
// n_pre_eval_trades_wrapped — pre-eval trades wrapped before eval (diagnostic)
|
||
// b_size — context for downstream interpretation
|
||
let aggregated_summary = serde_json::json!({
|
||
"n_trades": eval_summary.n_trades,
|
||
"total_pnl_usd": eval_summary.total_pnl_usd,
|
||
"profit_factor": eval_summary.profit_factor,
|
||
"sharpe_ann": eval_summary.sharpe_ann,
|
||
"max_drawdown_usd": eval_summary.max_drawdown_usd,
|
||
"win_rate": eval_summary.win_rate,
|
||
"n_eval_trades_seen": n_eval_trades_seen,
|
||
"n_eval_trades_dropped": n_eval_trades_dropped,
|
||
"n_pre_eval_trades_wrapped": n_pre_eval_wrapped,
|
||
"b_size": cli.n_backtests,
|
||
});
|
||
|
||
let eval_summary_path = cli.out.join("eval_summary.json");
|
||
let f = std::fs::File::create(&eval_summary_path)
|
||
.with_context(|| format!("create {}", eval_summary_path.display()))?;
|
||
serde_json::to_writer_pretty(f, &aggregated_summary)
|
||
.with_context(|| format!("write {}", eval_summary_path.display()))?;
|
||
eprintln!("eval summary written: {}", eval_summary_path.display());
|
||
}
|
||
|
||
// ── Compose + emit summary. ──────────────────────────────────────
|
||
if let Some(s) = last_stats {
|
||
summary.final_l_bce = s.l_bce;
|
||
summary.final_l_q = s.l_q;
|
||
summary.final_l_pi = s.l_pi;
|
||
summary.final_l_v = s.l_v;
|
||
summary.final_l_aux = s.l_aux;
|
||
summary.final_l_total = s.l_total;
|
||
}
|
||
summary.final_replay_len = trainer.gpu_replay.capacity.min(cli.n_steps);
|
||
summary.completed_clean = summary.n_steps_completed == summary.n_steps_planned;
|
||
write_summary(&cli.out, &summary)?;
|
||
|
||
eprintln!(
|
||
"alpha_rl_train complete: {} / {} steps in {:.1}s (replay len = {})",
|
||
summary.n_steps_completed,
|
||
summary.n_steps_planned,
|
||
t_start.elapsed().as_secs_f32(),
|
||
summary.final_replay_len
|
||
);
|
||
Ok(())
|
||
}
|
||
|
||
fn write_summary(out_dir: &std::path::Path, summary: &AlphaRlTrainSummary) -> Result<()> {
|
||
let path = out_dir.join("alpha_rl_train_summary.json");
|
||
let f = std::fs::File::create(&path)
|
||
.with_context(|| format!("create {}", path.display()))?;
|
||
serde_json::to_writer_pretty(f, summary)
|
||
.with_context(|| format!("write {}", path.display()))?;
|
||
eprintln!("summary written: {}", path.display());
|
||
Ok(())
|
||
}
|