Catalogue every numeric field in the HEALTH_DIAG family of log lines
(7 emit sites across training_loop.rs + metrics.rs) and classify each
source as GPU-already / CPU-bound / Mixed / Host-state / ISV-already.
Identifies the dominant cost contributors driving the observed
~70 s/epoch HEALTH_DIAG overhead on L40S after the eval async-split
(commit f815f7239) — they are all CPU-side reductions over multi-million
element per-sample buffers fetched via memcpy_dtoh:
- update_q_mag_means_cached (~786 KB DtoH + B×total_actions loop)
- var_scale_epoch_mean (N-element DtoH + conditional avg)
- trail_fire_and_hold_per_mag (4× N-element DtoH + per-mag loop)
- per_magnitude_winrate_and_variance (4× DtoH + sumsq loop)
- reward_contrib_fractions (6× large-buffer DtoH + 6 passes)
- read_eval_action_distribution_* (4 sites; CPU iter on pinned)
- branch_noisy_sigma_mean (NoisyNet param DtoH per branch)
Aggregate per-emit DtoH ≈ 50-150 MB depending on alloc_episodes ×
alloc_timesteps; fully consistent with the 70 s/epoch observation.
All inputs are already device-resident, so the migration is "stop
reducing on the host" plus a small kernel family writing into a single
mapped-pinned HealthDiagSnapshot struct.
Phase 0 deliverable per the dispatching prompt; subsequent commits
(struct + mapped wrapper, kernel family, CPU emit rewrite, audit-doc
updates) follow the phased plan documented at the bottom of the file
and gate-review on this inventory before touching code.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
333ea71844