The diag's `trading.*_pnl_*_usd` fields were systematically mislabeled, making
the eval verdict signal untrustworthy. Three quantities shared the `_usd`
suffix but were not in USD:
- `eval_summary.total_pnl_usd` (USD ground truth, $50/pt ES applied)
- `trading.realized_pnl_cum_usd` (cumulative SHAPED reward at close events;
pts × lots × Phase-5 shaping; NOT USD despite the suffix)
- `trading.pnl_cum_usd` (mathematically nonsense — reads rewards_d after
apply_reward_scale AND rl_popart_normalize whiten it in-place; dividing
by current_scale un-does only the first transform)
At baseline mid-smoke (b=128, 2000+500, seed=42): eval_summary.total_pnl_usd
= -$4,457,625 while diag.trading.realized_pnl_cum_usd = +$469k and
diag.trading.pnl_cum_usd = +$12.9k. 346x ratio, sign-opposite. Same
mechanism as `pearl_grwwh_eval_catastrophic_collapse` ($234M cluster
discrepancy). Every Pearson(reward, Δpnl_usd) computation against these
fields was shaped-vs-shaped tautology, not pnl alignment.
This commit:
DELETE `trading.pnl_cum_usd` (mathematically nonsense post-popart).
RENAME `trading.realized_pnl_cum_usd` -> `trading.shaped_reward_close_event_cum`.
Truthful name; the underlying values are post-Phase-5 shaped reward
summed at close events, useful for gradient-signal diagnostics but never
to be confused with USD pnl.
ADD `trading.realized_pnl_usd_delta` and `trading.realized_pnl_usd_cum` in
diag.jsonl / eval_diag.jsonl. Source: new per-batch float buffer
pnl_step_close_usd_d on LobSimCuda, zeroed via raw_memset_d8_zero before
each pnl_track_step launch, written by pnl_track.cu's close branch as
realised_pnl_usd_fp / 100.0 (same arithmetic as eval_summary's
TradeRecord aggregator, applying ES $50/pt). Single-writer per block —
no atomicAdd per feedback_no_atomicadd.
Direct readback via read_slice_d_into<f32> from sim.pnl_step_close_usd_d()
after step_with_lobsim_gpu returns (main stream already synchronized via
self.stream.synchronize() at pnl_track_step exit). NOT routed through
diag_staging double-buffer — a prior implementation tried this and
broke determinism (cross-stream race between main-stream
raw_memset_d8_zero + pnl_track_step write and diag-stream
raw_memcpy_dtod_async read). Direct same-stream read avoids the race
entirely and stays mega-graph compatible (the readback runs AFTER
per-step pipeline finishes, outside any captured CUDA graph).
Cumulative tracked Rust-side in alpha_rl_train.rs; reset at train->eval
boundary so realized_pnl_usd_cum tracks the eval window only.
scripts/tier1_5_verdict.py upgrade:
- signal_reward_alignment reads trading.realized_pnl_usd_cum
- signal_eval_pnl falls back to realized_pnl_usd_cum
- signal_consistency upgraded from WARN-at-5% to KILL-at-$50 with a new
consistency_kill_usd threshold key; cross-source disagreement is now
a hard kill not a warning
tests/eval_diag_emission.rs: bumped EXPECTED_LEAVES 711 -> 712 (-1 deleted,
-0 renamed, +2 added); added invariant that cum == running_sum(delta) and
regression check that legacy pnl_cum_usd / realized_pnl_cum_usd fields are
absent from the schema; allowed n_eval_steps + 1 row count for the
drain-row pattern.
Falsification gate (load-bearing, must hold every smoke from this commit
forward): `diag.last_eval_row.trading.realized_pnl_usd_cum` matches
`eval_summary.total_pnl_usd` within $50.
Validation (mid-smoke b=128, 2000 train + 500 eval, seed=42, RTX 3050):
- cargo build --release --example alpha_rl_train -p ml-alpha: exit 0
- cargo test -p ml-alpha --test eval_diag_emission: PASS (712 leaves)
- local-mid-smoke.sh: exit 0, drain row at step 2500
- cross-source consistency: eval_summary=-$4,457,625.00 vs
diag.realized_pnl_usd_cum=-$4,457,625.18 -> delta=$0.18 (exactly fp/100
f32 rounding noise; well within $50 tolerance)
- ./scripts/determinism-check.sh --quick: PASS — all checksums.* leaves
match across all 200 rows (rel-tol 1e-5, abs-tol 1e-7)
- Pre-commit hook on staged diff: PASS
Memory pearls created in this session document the diagnostic chain:
- pearl_diag_pnl_fields_are_shaped_reward_not_usd (the mislabeling root)
- pearl_reward_signal_anti_aligned_with_pnl ADDENDUM 2026-06-02b
(correction to the 2026-06-02 ADDENDUM — the "+0.93 Pearson at HEAD"
finding was a measurement artifact because both sides were in shaped
pts x lots units, not USD)
- pearl_advantage_kernel_is_done_gated_td_not_gae (related signal-density
gap — Phase 1.5 follow-up)
- pearl_phase5_term_4_is_almost_potential (Ng-Harada-Russell analysis of
Phase 5 shaping terms — Phase 1 follow-up)
Unlocks: TRUE Pearson(rewards.sum, Δpnl_usd) can now be measured per-step
at HEAD. The eval-collapse investigation (next phases: Ng-Harada-Russell
shaping cleanup + GAE upgrade) is now falsifiable with bit-deterministic
verdicts grounded in authoritative USD pnl.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
cfaa420bd4