diff --git a/docs/health_diag_inventory.md b/docs/health_diag_inventory.md new file mode 100644 index 000000000..b8dfb96c5 --- /dev/null +++ b/docs/health_diag_inventory.md @@ -0,0 +1,223 @@ +# HEALTH_DIAG Inventory — Phase 0 (read-only) + +**Purpose.** Catalogue every numeric value emitted in the HEALTH_DIAG family of log lines, classified by where the compute lives today, ahead of the greenfield port to a single GPU-resident `HealthDiagSnapshot`. + +**Scope.** All `tracing::info!("HEALTH_DIAG[...]: ...", …)` emissions in `crates/ml/src/`. Target: every numeric placeholder in those format strings. + +**Why.** L40S observed ~70 s/epoch attributed to HEALTH_DIAG even after `f815f7239` (`launch_validation_loss` / `consume_validation_loss` split). Async eval moved the host wait off the critical path; it did NOT move the *compute* — the per-epoch HEALTH_DIAG emit still triggers a wave of `memcpy_dtoh` reads of multi-million-element per-sample buffers followed by Rust-side reductions (counting, summation, variance, histograms). + +**Architectural rules** (re-stated for cross-reference; see prompt for full list): +- `feedback_no_htod_htoh_only_mapped_pinned.md` — mapped-pinned is the ONLY CPU↔GPU path. +- `feedback_no_atomicadd.md` — no `atomicAdd`; shmem tree reductions only. +- `feedback_adaptive_not_tuned.md` — block/grid sizes derive from input dims. +- `feedback_no_partial_refactor.md` — every consumer of the new struct migrates in the same commit. +- `feedback_wire_everything_up.md` — every metric in the snapshot is emitted; every metric currently emitted has a slot. + +## Classification key + +- **GPU-already** — value is produced by an existing CUDA kernel that writes via mapped-pinned (`cuMemHostAlloc(DEVICEMAP|PORTABLE)`) or to an ISV slot. CPU just reads. **Migration: rename the source slot into the new `HealthDiagSnapshot` struct field; same kernel writes; same data.** +- **CPU-bound** — value is computed on the CPU after one or more `memcpy_dtoh` reads. Often involves `Vec::with_capacity(N)` followed by per-element loops on N=1M+ buffers. **Migration: write a new CUDA reduction kernel; delete the CPU path entirely.** +- **Mixed** — kernel produces a per-batch buffer (no reduction) and CPU does a final reduction in Rust. **Migration: extend the kernel with a fused reduction tail OR add a small dedicated reducer kernel; delete the CPU loop.** +- **Host-state** — value is a Rust-side state machine output (e.g. `last_anti_mult` after the anti-LR controller decided something). No GPU compute involved; remains CPU. **Migration: read into a host-side scratchpad and write into the snapshot via `host_ptr` (allocator-free since the snapshot is a fixed-size struct).** +- **ISV-already** — already lives in the ISV signal bus (`isv_signals_dev_ptr`); CPU reads via `read_isv_signal_at`. **Migration: copy ISV slot into snapshot in a small ISV-mirror kernel, OR keep the existing host read but write into the snapshot's host_ptr from CPU after the read; no new GPU compute needed.** + +## Format-string anchor lines + +All sites surveyed: +1. `training_loop.rs:3229` — main giant `HEALTH_DIAG[{epoch}]: …` (the elephant; ~50 metric placeholders). +2. `training_loop.rs:3394` — `HEALTH_DIAG[{epoch}]: reward_split […]` (6 floats). +3. `training_loop.rs:3430` — `HEALTH_DIAG[{epoch}]: aux […]` (4 floats). +4. `training_loop.rs:3456` — `HEALTH_DIAG[{epoch}]: aux_moe […]` (8 utils + ent + λ_eff = 10 floats). +5. `metrics.rs:837` — `HEALTH_DIAG[{epoch}]: val […]` (16 floats). +6. `metrics.rs:890` — `HEALTH_DIAG[{epoch}]: val_dir_dist […]` (4 floats). +7. `metrics.rs:908` — `HEALTH_DIAG[{epoch}]: val_picked_dir_dist […]` (4 floats). +8. `training_loop.rs:900` — `HEALTH_DIAG[{epoch}]: TERMINATED_BY_REGRESSION …` (one-shot fault-path emission; not part of the per-epoch hot path). + +`TERMINATED_BY_REGRESSION` (#8) fires once at end-of-run on regression and is excluded from the snapshot port — it's a rare fault-path string carrying already-resolved values. + +## Inventory table — line `training_loop.rs:3229` (main HEALTH_DIAG) + +| Group | Metric | Source path | Class | Lines | Cost-per-epoch | Notes | +|---|---|---|---|---|---|---| +| top | `health` (`health_value`) | `self.learning_health.value()` (CPU computation over EMA components) | Host-state | training_loop.rs (computed earlier in `process_epoch_boundary`) | trivial | Pure scalar arithmetic; aggregator of the 7 component fields below. | +| components | `q_gap` | `self.learning_health.components.q_gap_norm` | Host-state | mod.rs / learning_health.rs | trivial | Cached EMA component, scalar. | +| components | `q_var` | `self.learning_health.components.q_var_norm` | Host-state | mod.rs / learning_health.rs | trivial | Cached EMA component. | +| components | `atoms` | `self.learning_health.components.atom_util_norm` | Host-state | mod.rs / learning_health.rs | trivial | Cached EMA component. | +| components | `grad_stable` | `self.learning_health.components.grad_stable` | Host-state | mod.rs / learning_health.rs | trivial | Cached EMA component. | +| components | `ens_agree` | `self.learning_health.components.ens_agree` | Host-state | mod.rs / learning_health.rs | trivial | Cached EMA component. | +| components | `grad_cos` | `self.learning_health.components.grad_consistency_norm` | Host-state | mod.rs / learning_health.rs | trivial | Cached EMA component. | +| components | `spectral` | `self.learning_health.components.spectral_gap_norm` | Host-state | mod.rs / learning_health.rs | trivial | Cached EMA component. | +| effective | `cql_alpha` | `self.last_cql_alpha_eff` | Host-state | training_loop.rs | trivial | Set by adaptive controller earlier in epoch. | +| effective | `iqn_budget` | `self.last_iqn_budget_eff` | Host-state | training_loop.rs | trivial | Same. | +| effective | `cql_budget` | `self.last_cql_budget_eff` | Host-state | training_loop.rs | trivial | Same. | +| effective | `c51_budget` | `self.last_c51_budget_eff` | Host-state | training_loop.rs | trivial | Same. | +| effective | `tau` | `self.last_tau_eff` | Host-state | training_loop.rs | trivial | Same. | +| effective | `sarsa_tau` | `self.last_sarsa_tau_factor` | Host-state | training_loop.rs | trivial | Same. | +| effective | `gamma` | `self.last_gamma_eff` | Host-state | training_loop.rs | trivial | Same. | +| effective | `cf_ratio` | `self.last_cf_ratio_eff` | Host-state | training_loop.rs | trivial | Set by `collector.set_learning_health()`. | +| novels | `distill` (on/off bool) | `self.last_distill_active` | Host-state | training_loop.rs | trivial | Set by D-mechanism logic earlier in epoch. | +| novels | `barrier` | `self.last_barrier_loss` | Host-state | training_loop.rs | trivial | Same. | +| novels | `plasticity` (ready/cooldown bool) | `self.last_plasticity_ready` | Host-state | training_loop.rs | trivial | Same. | +| novels | `ib` | `self.last_ib_penalty` | Host-state | training_loop.rs ~2410 | trivial | Pure scalar arithmetic over EMA. | +| novels | `ensemble_collapse` | `self.last_ensemble_collapse_score` | Host-state | training_loop.rs ~2424 | trivial | Smoothstep over `last_ens_disagreement`. | +| novels | `contrarian` (on/off bool) | `self.last_contrarian_active` | Host-state | training_loop.rs ~2480 | trivial | State-machine output. | +| novels | `meta_q_pred` | `self.last_meta_q_pred` | Host-state | training_loop.rs ~2531 | trivial | `self.meta_q.predict(input)` — small CPU MLP. Stays on CPU; meta-Q is a tiny network. | +| diag | `sharpe_ema` | `self.training_sharpe_ema` | Host-state | training_loop.rs | trivial | Updated from train pnl. | +| diag | `action_entropy` | `self.last_action_entropy` | Mixed | training_loop.rs ~2790 | small CPU loop over 12-bin counts | Currently CPU computes Shannon entropy normalised over `summary.action_counts[12]`. **Move to GPU**: extend the existing `monitoring_kernel`'s reducer to also emit normalised entropy. | +| gems | `g12_predictive` | `fused.read_gem_g12_loss()` | GPU-already | gpu_dqn_trainer.rs ~13169 | trivial | Already pinned. | +| mag | `q_full / q_half / q_quarter` | `fused.q_magnitude_bucket_means()` after `update_q_mag_means_cached` | CPU-bound (HEAVY) | gpu_dqn_trainer.rs:4437 | **786 KB DtoH/epoch + B×total_actions CPU loop** (B=16384, total_actions=12 → 196,608 floats summed). | Hot reduction. **Migrate**: small kernel reducing q_out_buf cols [b0..b0+b1] across batch into 3 floats. | +| mag | `var_scale` (var_scale_mean) | `collector.var_scale_epoch_mean()` | CPU-bound (HEAVY) | gpu_experience_collector.rs:2784 | DtoH ~N×4 bytes (N = alloc_episodes × alloc_timesteps, ~512K-2M); CPU loop with conditional avg. | **Migrate**: GPU reduction kernel summing only `v > 0` slots + count, returns mean. | +| mag | `kelly_f` (kelly_f_mean) | `self.trade_stats_history.last()` arithmetic | Host-state | training_loop.rs ~2753 | trivial | Pure CPU formula on 6-float `TradeStats`. | +| mag | `avg_win_ratio` | same as above | Host-state | training_loop.rs ~2753 | trivial | Pure CPU formula. | +| mag | `grad_ratio_mag_dir` | `grad_ratio_mag_dir_cached` | GPU-already | populated by reduction kernel into pinned struct | trivial | Cached at top of `process_epoch_boundary`. | +| mag | `dist_q / dist_h / dist_f` | `monitor.action_counts[..]` summed in Rust over 12 entries | Mixed | training_loop.rs ~2839 | trivial CPU sum of 12 ints + division | Counts come from GPU via `monitoring_summary` (already pinned). The CPU sum is 12 ints — cheap, but **part of the cascade we're collapsing**. **Migrate**: GPU monitoring kernel emits dist_q/h/f directly. | +| grad_split_bwd | `iqn cql c51 ens` | `fused.grad_mag_*_ratio()` after `refresh_grad_component_norms()` | GPU-already | gpu_dqn_trainer.rs (in-graph reduction kernel writes pinned slots) | trivial | Read pinned. | +| grad_split_aux | `distill rec pred cql_sx c51_bs` | same family of `grad_mag_*_ratio()` | GPU-already | same | trivial | Read pinned. | +| grad_trunk | 9 floats | `fused.grad_trunk_norms_by_component()` (`[f32; 9]`) | GPU-already | same | trivial | Read pinned. | +| grad_abs | `dir mag` | `grad_dir_abs_cached`, `grad_mag_abs_cached` | GPU-already | populated by same reduction kernel | trivial | Read pinned. | +| trail | `fire_q/h/f hold_q/h/f` | `collector.trail_fire_and_hold_per_mag()` | CPU-bound (HEAVY) | gpu_experience_collector.rs:2063 | **4× DtoH** of N-element buffers (N = alloc_episodes × alloc_timesteps); CPU loop with branch + accumulate per-mag. | **Migrate**: 3-block kernel (one per mag) with shmem reductions over the 4 input buffers; write 6 floats (3 rates + 3 hold means). | +| mag_stats | `wr_q/h/f var_q/h/f` | `collector.per_magnitude_winrate_and_variance()` | CPU-bound (HEAVY) | gpu_experience_collector.rs:2682 | **4× DtoH** of N-element buffers; CPU loop computing Welford-style mean+sumsq per-mag. | **Migrate**: 3-block kernel; shmem reduction; emit 6 floats. | +| noisy | `vsn_mag vsn_dir` | ISV slots `VSN_MAG_EMA_INDEX`, `VSN_DIR_EMA_INDEX` | ISV-already | gpu_dqn_trainer.rs (constants); `attention_focus_ema` kernel writes | trivial | Read via `read_isv_signal_at`. | +| noisy | `sigma_mag sigma_dir` | `agent.branch_noisy_sigma_mean(0/1)` | CPU-bound | branched DQN model; reads NoisyNet sigma params | medium (param DtoH per branch) | NoisyNets σ mean across fc+out layers. **Migrate**: dedicated kernel summing |w| over the relevant param slices into 2 floats. | +| noisy | `drift_mag drift_dir` | ISV slots `TARGET_DRIFT_MAG_EMA_INDEX`, `TARGET_DRIFT_DIR_EMA_INDEX` | ISV-already | `target_drift_ema` kernel writes | trivial | Read via `read_isv_signal_at`. | +| eval_dist | `eq eh ef` | `self.last_eval_magnitude_dist[0..3]` (computed at val time) | GPU-already (pinned read) → CPU loop | metrics.rs ~862 calls evaluator's `read_eval_action_distribution_per_magnitude` | medium (CPU loop over actions_history_pinned, ~64K-256K i32s) | Currently CPU iterates the mapped-pinned actions buffer counting `(a/9) % 3` per non-negative entry. **Migrate**: small GPU histogram kernel reading the device pointer of the same buffer; write 3 floats. | +| reward_contrib | `popart cf trail_r micro` | `self.last_reward_contrib` (4 floats); slot 0 = `prev_reward_contrib_popart_var` derived; slots 1..4 from `collector.reward_contrib_fractions()` | CPU-bound (HEAVY for slots 1..4) | gpu_experience_collector.rs:2163 | **6× DtoH** (cf_flip, slot_live, trail, traded, micro, total) of large per-sample buffers + multi-pass CPU summation. | **Migrate**: single fused reduction kernel over the 6 buffers; emit 4 floats. PopArt slot stays host-state (it's already a small scalar derived from `read_popart_variance`). | +| controller | `anti_lr tau gamma clip cql cost` (6 bools), `fire_frac` (f32) | host-side detection: `fire_lr` etc. | Host-state | training_loop.rs ~2902-2955 | trivial | Pure CPU comparisons against `prev_controller_values`. | +| explore | `ent_mag ent_dir` | `shannon_entropy_normalized(&[dist_q, dist_h, dist_f])` and same over `[dir0..dir3]` | Mixed | training_loop.rs ~2858 | trivial CPU loop over 3-4 elements | Source values come from `monitor.action_counts[12]` (GPU). **Migrate**: have the new GPU monitoring extension emit ent_mag and ent_dir directly. | +| explore | `sigma_mean` | `(sigma_mag + sigma_dir) * 0.5` | Host-state | training_loop.rs ~2640 | trivial | Trivial after sigma_mag/dir migrate. | + +### Hidden-cost lines that materially affect total HEALTH_DIAG cost + +These are computed *for* HEALTH_DIAG but happen in upstream methods invoked just before the emit: + +| Site | Cost | Migration | +|---|---|---| +| `update_q_mag_means_cached` (gpu_dqn_trainer.rs:4437) | 786 KB DtoH + B×total_actions Rust loop | **DELETE the CPU path; replace with kernel.** | +| `var_scale_epoch_mean` (gpu_experience_collector.rs:2784) | N×4-byte DtoH; conditional CPU avg | Replace with kernel. | +| `trail_fire_and_hold_per_mag` (gpu_experience_collector.rs:2063) | 4× N-element DtoH; per-mag CPU loop | Replace with kernel. | +| `per_magnitude_winrate_and_variance` (gpu_experience_collector.rs:2682) | 4× N-element DtoH; per-mag CPU sum + sum_sq | Replace with kernel. | +| `reward_contrib_fractions` (gpu_experience_collector.rs:2163) | 6× large-buffer DtoH; multi-pass CPU sums | Replace with kernel. | +| `read_eval_action_distribution_per_magnitude` (gpu_backtest_evaluator.rs:1255) | actions_history_pinned read (mapped, but CPU iterates ~64-256K entries) | Replace with kernel reading the dev pointer of the same mapped buffer; CPU just reads the resulting 3 floats from the snapshot. | +| `read_eval_action_distribution_per_direction` (gpu_backtest_evaluator.rs:1339) | same buffer, different histogram | Same kernel, additional 4-float output. | +| `read_eval_intent_magnitude_distribution` (gpu_backtest_evaluator.rs:1295) | intent_mag_pinned iter | Replace with kernel; 3-float output. | +| `read_chunked_actions_direction_distribution` (gpu_backtest_evaluator.rs:1384) | picked_action_history_pinned iter | Replace with kernel; 4-float output. | + +## Inventory table — line `training_loop.rs:3394` (`reward_split`) + +| Metric | Source | Class | Notes | +|---|---|---|---| +| `popart` | ISV `REWARD_POPART_EMA_INDEX` | ISV-already | `read_isv_signal_at`. | +| `cf` | ISV `REWARD_CF_EMA_INDEX` | ISV-already | Same. | +| `trail` | ISV `REWARD_TRAIL_EMA_INDEX` | ISV-already | Same. | +| `micro` | ISV `REWARD_MICRO_EMA_INDEX` | ISV-already | Same. | +| `opp_cost` | ISV `REWARD_OPP_COST_EMA_INDEX` | ISV-already | Same. | +| `bonus` | ISV `REWARD_BONUS_EMA_INDEX` | ISV-already | Same. | + +All 6 already on GPU; ISV-already. Snapshot fields = mirror copies. + +## Inventory table — line `training_loop.rs:3430` (`aux`) + +| Metric | Source | Class | Notes | +|---|---|---|---| +| `next_bar_mse` | ISV `AUX_NEXT_BAR_MSE_EMA_INDEX` (113) | ISV-already | `read_isv_signal_at`. | +| `regime_ce` | ISV `AUX_REGIME_CE_EMA_INDEX` (114) | ISV-already | Same. | +| `w` (aux_weight) | `trainer.aux_weight()` | Host-state | Plain scalar field on the trainer. | +| `label_scale` | ISV `AUX_LABEL_SCALE_EMA_INDEX` (117) | ISV-already | Same. | + +## Inventory table — line `training_loop.rs:3456` (`aux_moe`) + +| Metric | Source | Class | Notes | +|---|---|---|---| +| `util[0..8]` | ISV `MOE_EXPERT_UTIL_EMA_BASE..+8` | ISV-already | 8 `read_isv_signal_at`. | +| `ent` | ISV `MOE_GATE_ENTROPY_EMA_INDEX` | ISV-already | Same. | +| `λ_eff` | ISV `MOE_LAMBDA_EFF_INDEX` | ISV-already | Same. | + +## Inventory table — line `metrics.rs:837` (`val`) + +All 16 fields originate from `evaluator.consume_metrics_after_event()`, which already reads from `metrics_buf` (mapped-pinned, populated by `backtest_metrics_kernel`). The CPU does a small post-process to derive `trades_per_bar`, `active_frac`, `dir_entropy` — pure arithmetic over scalar floats already in the metrics struct. + +| Metric | Source | Class | Notes | +|---|---|---|---| +| `sharpe` | `m.sharpe` | GPU-already | metrics_buf scalar. | +| `sortino` | `m.sortino` | GPU-already | Same. | +| `win_rate` | `m.win_rate` | GPU-already | Same. | +| `max_drawdown` | `m.max_drawdown` | GPU-already | Same. | +| `trade_count` | `m.total_trades` | GPU-already | Same. | +| `calmar` | `m.calmar` | GPU-already | Same. | +| `omega_ratio` | `m.omega_ratio` | GPU-already | Same. | +| `total_pnl` | `m.total_pnl` | GPU-already | Same. | +| `var_95` | `m.var_95` | GPU-already | Same. | +| `cvar_95` | `m.cvar_95` | GPU-already | Same. | +| `trades_per_bar` | derived (3 buckets ÷ window) | Host-state (trivial) | Three scalar f32 divides. | +| `active_frac` | derived | Host-state (trivial) | Same. | +| `dir_entropy` | derived (3-prob Shannon entropy) | Host-state (trivial) | Per-3-bucket entropy. **Could move to kernel** to land directly in `metrics_buf`; trivial enough to either keep or migrate. | +| `sharpe_annualised` | `m.sharpe` (alias) | GPU-already | Aggregator alias. | +| `profit_factor` | `m.omega_ratio` (alias) | GPU-already | Aggregator alias. | +| `window_bars` | `buy + sell + hold` | Host-state (trivial) | One add. | + +**Migration note for `val [...]`:** the metrics_buf is *already* mapped-pinned, so the val emit's CPU work today is 3 scalar adds + 3 entropy multiplies. The big perf win for HEALTH_DIAG is NOT in `val`; it's in the main line's per-sample buffer reductions. We should still fold `val`'s fields into `HealthDiagSnapshot` for layout uniformity (one struct, allocator-free formatting), but the work to extend the `backtest_metrics_kernel` to directly emit `dir_entropy` etc. is optional — it can copy from `metrics_buf` into the snapshot at emit time. + +## Inventory table — line `metrics.rs:890` (`val_dir_dist`) + +| Metric | Source | Class | Notes | +|---|---|---|---| +| `short hold long flat` (4 floats) | `read_eval_action_distribution_per_direction` (gpu_backtest_evaluator.rs:1339) | Mixed | CPU iterates `actions_history_pinned` counting per direction. **Migrate**: kernel histogram. | + +## Inventory table — line `metrics.rs:908` (`val_picked_dir_dist`) + +| Metric | Source | Class | Notes | +|---|---|---|---| +| `short hold long flat` (4 floats) | `read_chunked_actions_direction_distribution` (gpu_backtest_evaluator.rs:1384) | Mixed | CPU iterates `picked_action_history_pinned`. Same pattern as above. | + +## Aggregate cost summary (per-epoch, batch 16384) + +Counting only the CPU-bound + Mixed sites whose work is dominated by `Vec` alloc + DtoH + per-element loops: + +| Site | DtoH bytes | CPU loop work | Notes | +|---|---|---|---| +| `update_q_mag_means_cached` | ~786 KB (B × total_actions × 4) | 196 608 float-adds + 3 div | One epoch; ✗ should be a kernel. | +| `var_scale_epoch_mean` | ~N × 4 bytes (≈ 2-8 MB at full scale) | N ifs + adds | ✗. | +| `trail_fire_and_hold_per_mag` | 4× N-element DtoH (~8-32 MB) | 4N ifs + accumulate | ✗. | +| `per_magnitude_winrate_and_variance` | 4× N-element DtoH (same scale) | 4N ifs + sumsq | ✗. | +| `reward_contrib_fractions` | 6× large DtoH (≈ 24-100 MB) | 6N passes | ✗ — biggest site. | +| `read_eval_action_distribution_*` | reads mapped-pinned (no DtoH but CPU iterates) | up to 256K i32 iter × 4 (3 mag + 4 dir + 3 intent + 4 picked) | ✗ — CPU iter eats into the 70 s budget. | +| `branch_noisy_sigma_mean(0)` and `(1)` | NoisyNet param DtoH | small but per-branch | ✗ at the margins. | + +Total estimated DtoH per HEALTH_DIAG emit: **~50-150 MB depending on N (alloc_episodes × alloc_timesteps)**. At PCIe Gen4 x16 (~20 GB/s sustained) that's ~3-7 ms of pure DtoH plus the host loop overhead at 1-2 ns per element on N up to 4M, which gets us into the 30-100 ms range per emit just for these readback paths. Multiplied by other epoch-boundary work (eval consumption, controller fires, optimizer maintenance) and synchronous host syncs, the 70 s/epoch number is consistent with the per-sample-buffer reductions being the dominant cost. **They are also the easiest to eliminate** because every per-sample buffer is already device-resident — we just stop reducing on the host. + +## What stays CPU after the port + +Per the prompt's explicit allowance ("CPU is now strictly a string formatter + log emitter"), these remain CPU: + +1. **Format-string assembly** — `String::with_capacity(2048)` once per emit, fixed cap. No `Vec` on hot path. +2. **Host-state mirrors** — fields tagged `Host-state` above (controllers, mechanism flags, EMA scalars on the LearningHealth aggregator). These are already small fields on the trainer struct; we read them with no reduction. Their inclusion in `HealthDiagSnapshot` is "host writes the field via `host_ptr` immediately before format" — allocator-free, no GPU compute, trivially fast. +3. **Meta-Q predictor** (`self.meta_q.predict`) — tiny CPU MLP; staying CPU is a deliberate architectural choice, not a HEALTH_DIAG concern. + +Everything in **Mixed** and **CPU-bound (HEAVY)** rows is in scope for the kernel port and the corresponding CPU code is deleted in the same commit chain. + +## Snapshot field list (proposed; finalised in Phase 1 commit) + +Field order matches the format-string emit order so log determinism is preserved across the port. Counts: 7 components + 8 effective + 7 novels + 2 diag + 1 gems + 10 mag + 4 grad_split_bwd + 5 grad_split_aux + 9 grad_trunk + 2 grad_abs + 6 trail + 6 mag_stats + 6 noisy + 3 eval_dist + 4 reward_contrib + 6 fire_bools + 1 fire_frac + 3 explore + 16 val + 4 val_dir_dist + 4 val_picked_dir_dist + 6 reward_split + 4 aux + 10 aux_moe + 12 monitor.action_counts (for direction-bin computation if we keep that path adjacent) ≈ **~150 f32 fields**. + +The booleans become `u8` slots (0/1) so the struct stays POD with no padding surprises across CPU/GPU. The struct sits in a single mapped-pinned allocation of ~600-800 bytes — well under one cache line × 16, easy to fence + read. + +## Phased commit plan (Phase 1+ — to be authored) + +1. **Commit 1 — this inventory doc** (Phase 0). Gate review here. +2. **Commit 2 — `MappedHealthDiagSnapshot` wrapper + `HealthDiagSnapshot` struct in `mapped_pinned.rs`.** No kernel yet, no emit changes. +3. **Commit 3 — single `health_diag_kernel.cu`** (or small fused family) reading device pointers of existing buffers + ISV bus. One block per metric category; shmem tree reductions; `__threadfence_system()` writes. No `atomicAdd`. Wire into the per-epoch graph alongside other producer kernels. +4. **Commit 4 — rewrite emit path in `metrics.rs` and `training_loop.rs`.** Delete every Vec allocator on the HEALTH_DIAG path. Delete `reward_contrib_fractions`, `trail_fire_and_hold_per_mag`, `per_magnitude_winrate_and_variance`, `var_scale_epoch_mean`, `update_q_mag_means_cached`, `read_eval_action_distribution_per_magnitude`, `read_eval_action_distribution_per_direction`, `read_eval_intent_magnitude_distribution`, `read_chunked_actions_direction_distribution`, `branch_noisy_sigma_mean` (or fold into kernel call) — *all of them* in this commit. Per `feedback_no_partial_refactor.md`, every consumer migrates simultaneously. +5. **Commit 5 — audit doc updates** (`docs/dqn-wire-up-audit.md` Invariant 7 entries; `docs/dqn-gpu-hot-path-audit.md` rows for the new pinned alloc and any new DtoD if needed). Layout fingerprint bump if ISV slots added. + +## Open questions for Phase 1 design review + +1. **Single kernel vs small fused family?** A single mega-kernel reading every input buffer and emitting all ~150 fields is monolithic but minimises launch overhead (one kernel, one fence). A 5-7 kernel family lets us keep block configs adapted to each input's shape (per-sample buffers want N-thread reductions; ISV mirrors are scalar copies that want 1 thread). Recommended: family of 4-5 kernels run sequentially in the captured graph — `health_diag_per_sample_reduce`, `health_diag_q_mag_reduce`, `health_diag_eval_histogram`, `health_diag_isv_mirror`, `health_diag_finalise` (the last one runs `__threadfence_system()` once over the whole struct). + +2. **Where does the kernel run in the captured graph?** End of the per-step training graph would update the snapshot every step (waste). End of `process_epoch_boundary`'s producer launches (next to `launch_h_s2_rms_ema`, `launch_vsn_mask_ema`, `launch_iqn_quantile_ema`, `launch_aux_heads_loss_ema`) is the right cadence — once per epoch, after all upstream producers have written. + +3. **booleans + u8 in `#[repr(C)]`** — confirm padding behaviour. If padding leaks bytes between `[u8; 6]` (the controller fires) and the next `f32`, the kernel must explicitly write through padding-aware offsets; safer to lift `u8` to `u32` (4-byte aligned) at a tiny ~24-byte cost. Pinned allocs round to page anyway. + +4. **`val [...]` block — extend `backtest_metrics_kernel` or copy from `metrics_buf` into the snapshot?** The cheaper path is the latter: `metrics_buf` is already mapped-pinned and the consumer reads it once already. We can have a small `health_diag_isv_mirror` kernel pull from `metrics_buf.dev_ptr` into the snapshot's val_* slots — avoids touching the existing eval graph. This keeps the eval contract stable. + +5. **What about `HEALTH_DIAG[...]: TERMINATED_BY_REGRESSION ...`?** Fault-path; values are already known scalars. Keep CPU. Out of scope. +