foxhunt/docs/health_diag_inventory.md

# HEALTH_DIAG Inventory — Phase 0 (read-only)

**Purpose.** Catalogue every numeric value emitted in the HEALTH_DIAG family of log lines, classified by where the compute lives today, ahead of the greenfield port to a single GPU-resident `HealthDiagSnapshot`.

**Scope.** All `tracing::info!("HEALTH_DIAG[...]: ...", …)` emissions in `crates/ml/src/`. Target: every numeric placeholder in those format strings.

**Why.** L40S observed ~70 s/epoch attributed to HEALTH_DIAG even after `f815f7239` (`launch_validation_loss` / `consume_validation_loss` split). Async eval moved the host wait off the critical path; it did NOT move the *compute* — the per-epoch HEALTH_DIAG emit still triggers a wave of `memcpy_dtoh` reads of multi-million-element per-sample buffers followed by Rust-side reductions (counting, summation, variance, histograms).

**Architectural rules** (re-stated for cross-reference; see prompt for full list):
- `feedback_no_htod_htoh_only_mapped_pinned.md` — mapped-pinned is the ONLY CPU↔GPU path.
- `feedback_no_atomicadd.md` — no `atomicAdd`; shmem tree reductions only.
- `feedback_adaptive_not_tuned.md` — block/grid sizes derive from input dims.
- `feedback_no_partial_refactor.md` — every consumer of the new struct migrates in the same commit.
- `feedback_wire_everything_up.md` — every metric in the snapshot is emitted; every metric currently emitted has a slot.

## Classification key

- **GPU-already** — value is produced by an existing CUDA kernel that writes via mapped-pinned (`cuMemHostAlloc(DEVICEMAP|PORTABLE)`) or to an ISV slot. CPU just reads. **Migration: rename the source slot into the new `HealthDiagSnapshot` struct field; same kernel writes; same data.**
- **CPU-bound** — value is computed on the CPU after one or more `memcpy_dtoh` reads. Often involves `Vec::with_capacity(N)` followed by per-element loops on N=1M+ buffers. **Migration: write a new CUDA reduction kernel; delete the CPU path entirely.**
- **Mixed** — kernel produces a per-batch buffer (no reduction) and CPU does a final reduction in Rust. **Migration: extend the kernel with a fused reduction tail OR add a small dedicated reducer kernel; delete the CPU loop.**
- **Host-state** — value is a Rust-side state machine output (e.g. `last_anti_mult` after the anti-LR controller decided something). No GPU compute involved; remains CPU. **Migration: read into a host-side scratchpad and write into the snapshot via `host_ptr` (allocator-free since the snapshot is a fixed-size struct).**
- **ISV-already** — already lives in the ISV signal bus (`isv_signals_dev_ptr`); CPU reads via `read_isv_signal_at`. **Migration: copy ISV slot into snapshot in a small ISV-mirror kernel, OR keep the existing host read but write into the snapshot's host_ptr from CPU after the read; no new GPU compute needed.**

## Format-string anchor lines

All sites surveyed:
1. `training_loop.rs:3229` — main giant `HEALTH_DIAG[{epoch}]: …` (the elephant; ~50 metric placeholders).
2. `training_loop.rs:3394` — `HEALTH_DIAG[{epoch}]: reward_split […]` (6 floats).
3. `training_loop.rs:3430` — `HEALTH_DIAG[{epoch}]: aux […]` (4 floats).
4. `training_loop.rs:3456` — `HEALTH_DIAG[{epoch}]: aux_moe […]` (8 utils + ent + λ_eff = 10 floats).
5. `metrics.rs:837` — `HEALTH_DIAG[{epoch}]: val […]` (16 floats).
6. `metrics.rs:890` — `HEALTH_DIAG[{epoch}]: val_dir_dist […]` (4 floats).
7. `metrics.rs:908` — `HEALTH_DIAG[{epoch}]: val_picked_dir_dist […]` (4 floats).
8. `training_loop.rs:900` — `HEALTH_DIAG[{epoch}]: TERMINATED_BY_REGRESSION …` (one-shot fault-path emission; not part of the per-epoch hot path).

`TERMINATED_BY_REGRESSION` (#8) fires once at end-of-run on regression and is excluded from the snapshot port — it's a rare fault-path string carrying already-resolved values.

## Inventory table — line `training_loop.rs:3229` (main HEALTH_DIAG)

| Group | Metric | Source path | Class | Lines | Cost-per-epoch | Notes |
|---|---|---|---|---|---|---|
| top | `health` (`health_value`) | `self.learning_health.value()` (CPU computation over EMA components) | Host-state | training_loop.rs (computed earlier in `process_epoch_boundary`) | trivial | Pure scalar arithmetic; aggregator of the 7 component fields below. |
| components | `q_gap` | `self.learning_health.components.q_gap_norm` | Host-state | mod.rs / learning_health.rs | trivial | Cached EMA component, scalar. |
| components | `q_var` | `self.learning_health.components.q_var_norm` | Host-state | mod.rs / learning_health.rs | trivial | Cached EMA component. |
| components | `atoms` | `self.learning_health.components.atom_util_norm` | Host-state | mod.rs / learning_health.rs | trivial | Cached EMA component. |
| components | `grad_stable` | `self.learning_health.components.grad_stable` | Host-state | mod.rs / learning_health.rs | trivial | Cached EMA component. |
| components | `ens_agree` | `self.learning_health.components.ens_agree` | Host-state | mod.rs / learning_health.rs | trivial | Cached EMA component. |
| components | `grad_cos` | `self.learning_health.components.grad_consistency_norm` | Host-state | mod.rs / learning_health.rs | trivial | Cached EMA component. |
| components | `spectral` | `self.learning_health.components.spectral_gap_norm` | Host-state | mod.rs / learning_health.rs | trivial | Cached EMA component. |
| effective | `cql_alpha` | `self.last_cql_alpha_eff` | Host-state | training_loop.rs | trivial | Set by adaptive controller earlier in epoch. |
| effective | `iqn_budget` | `self.last_iqn_budget_eff` | Host-state | training_loop.rs | trivial | Same. |
| effective | `cql_budget` | `self.last_cql_budget_eff` | Host-state | training_loop.rs | trivial | Same. |
| effective | `c51_budget` | `self.last_c51_budget_eff` | Host-state | training_loop.rs | trivial | Same. |
| effective | `tau` | `self.last_tau_eff` | Host-state | training_loop.rs | trivial | Same. |
| effective | `sarsa_tau` | `self.last_sarsa_tau_factor` | Host-state | training_loop.rs | trivial | Same. |
| effective | `gamma` | `self.last_gamma_eff` | Host-state | training_loop.rs | trivial | Same. |
| effective | `cf_ratio` | `self.last_cf_ratio_eff` | Host-state | training_loop.rs | trivial | Set by `collector.set_learning_health()`. |
| novels | `distill` (on/off bool) | `self.last_distill_active` | Host-state | training_loop.rs | trivial | Set by D-mechanism logic earlier in epoch. |
| novels | `barrier` | `self.last_barrier_loss` | Host-state | training_loop.rs | trivial | Same. |
| novels | `plasticity` (ready/cooldown bool) | `self.last_plasticity_ready` | Host-state | training_loop.rs | trivial | Same. |
| novels | `ib` | `self.last_ib_penalty` | Host-state | training_loop.rs ~2410 | trivial | Pure scalar arithmetic over EMA. |
| novels | `ensemble_collapse` | `self.last_ensemble_collapse_score` | Host-state | training_loop.rs ~2424 | trivial | Smoothstep over `last_ens_disagreement`. |
| novels | `contrarian` (on/off bool) | `self.last_contrarian_active` | Host-state | training_loop.rs ~2480 | trivial | State-machine output. |
| novels | `meta_q_pred` | `self.last_meta_q_pred` | Host-state | training_loop.rs ~2531 | trivial | `self.meta_q.predict(input)` — small CPU MLP. Stays on CPU; meta-Q is a tiny network. |
| diag | `sharpe_ema` | `self.training_sharpe_ema` | Host-state | training_loop.rs | trivial | Updated from train pnl. |
| diag | `action_entropy` | `self.last_action_entropy` | Mixed | training_loop.rs ~2790 | small CPU loop over 12-bin counts | Currently CPU computes Shannon entropy normalised over `summary.action_counts[12]`. **Move to GPU**: extend the existing `monitoring_kernel`'s reducer to also emit normalised entropy. |
| gems | `g12_predictive` | `fused.read_gem_g12_loss()` | GPU-already | gpu_dqn_trainer.rs ~13169 | trivial | Already pinned. |
| mag | `q_full / q_half / q_quarter` | `fused.q_magnitude_bucket_means()` after `update_q_mag_means_cached` | CPU-bound (HEAVY) | gpu_dqn_trainer.rs:4437 | **786 KB DtoH/epoch + B×total_actions CPU loop** (B=16384, total_actions=12 → 196,608 floats summed). | Hot reduction. **Migrate**: small kernel reducing q_out_buf cols [b0..b0+b1] across batch into 3 floats. |
| mag | `var_scale` (var_scale_mean) | `collector.var_scale_epoch_mean()` | CPU-bound (HEAVY) | gpu_experience_collector.rs:2784 | DtoH ~N×4 bytes (N = alloc_episodes × alloc_timesteps, ~512K-2M); CPU loop with conditional avg. | **Migrate**: GPU reduction kernel summing only `v > 0` slots + count, returns mean. |
| mag | `kelly_f` (kelly_f_mean) | `self.trade_stats_history.last()` arithmetic | Host-state | training_loop.rs ~2753 | trivial | Pure CPU formula on 6-float `TradeStats`. |
| mag | `avg_win_ratio` | same as above | Host-state | training_loop.rs ~2753 | trivial | Pure CPU formula. |
| mag | `grad_ratio_mag_dir` | `grad_ratio_mag_dir_cached` | GPU-already | populated by reduction kernel into pinned struct | trivial | Cached at top of `process_epoch_boundary`. |
| mag | `dist_q / dist_h / dist_f` | `monitor.action_counts[..]` summed in Rust over 12 entries | Mixed | training_loop.rs ~2839 | trivial CPU sum of 12 ints + division | Counts come from GPU via `monitoring_summary` (already pinned). The CPU sum is 12 ints — cheap, but **part of the cascade we're collapsing**. **Migrate**: GPU monitoring kernel emits dist_q/h/f directly. |
| grad_split_bwd | `iqn cql c51 ens` | `fused.grad_mag_*_ratio()` after `refresh_grad_component_norms()` | GPU-already | gpu_dqn_trainer.rs (in-graph reduction kernel writes pinned slots) | trivial | Read pinned. |
| grad_split_aux | `distill rec pred cql_sx c51_bs` | same family of `grad_mag_*_ratio()` | GPU-already | same | trivial | Read pinned. |
| grad_trunk | 9 floats | `fused.grad_trunk_norms_by_component()` (`[f32; 9]`) | GPU-already | same | trivial | Read pinned. |
| grad_abs | `dir mag` | `grad_dir_abs_cached`, `grad_mag_abs_cached` | GPU-already | populated by same reduction kernel | trivial | Read pinned. |
| trail | `fire_q/h/f hold_q/h/f` | `collector.trail_fire_and_hold_per_mag()` | CPU-bound (HEAVY) | gpu_experience_collector.rs:2063 | **4× DtoH** of N-element buffers (N = alloc_episodes × alloc_timesteps); CPU loop with branch + accumulate per-mag. | **Migrate**: 3-block kernel (one per mag) with shmem reductions over the 4 input buffers; write 6 floats (3 rates + 3 hold means). |
| mag_stats | `wr_q/h/f var_q/h/f` | `collector.per_magnitude_winrate_and_variance()` | CPU-bound (HEAVY) | gpu_experience_collector.rs:2682 | **4× DtoH** of N-element buffers; CPU loop computing Welford-style mean+sumsq per-mag. | **Migrate**: 3-block kernel; shmem reduction; emit 6 floats. |
| noisy | `vsn_mag vsn_dir` | ISV slots `VSN_MAG_EMA_INDEX`, `VSN_DIR_EMA_INDEX` | ISV-already | gpu_dqn_trainer.rs (constants); `attention_focus_ema` kernel writes | trivial | Read via `read_isv_signal_at`. |
| noisy | `sigma_mag sigma_dir` | `agent.branch_noisy_sigma_mean(0/1)` | CPU-bound | branched DQN model; reads NoisyNet sigma params | medium (param DtoH per branch) | NoisyNets σ mean across fc+out layers. **Migrate**: dedicated kernel summing |w| over the relevant param slices into 2 floats. |
| noisy | `drift_mag drift_dir` | ISV slots `TARGET_DRIFT_MAG_EMA_INDEX`, `TARGET_DRIFT_DIR_EMA_INDEX` | ISV-already | `target_drift_ema` kernel writes | trivial | Read via `read_isv_signal_at`. |
| eval_dist | `eq eh ef` | `self.last_eval_magnitude_dist[0..3]` (computed at val time) | GPU-already (pinned read) → CPU loop | metrics.rs ~862 calls evaluator's `read_eval_action_distribution_per_magnitude` | medium (CPU loop over actions_history_pinned, ~64K-256K i32s) | Currently CPU iterates the mapped-pinned actions buffer counting `(a/9) % 3` per non-negative entry. **Migrate**: small GPU histogram kernel reading the device pointer of the same buffer; write 3 floats. |
| reward_contrib | `popart cf trail_r micro` | `self.last_reward_contrib` (4 floats); slot 0 = `prev_reward_contrib_popart_var` derived; slots 1..4 from `collector.reward_contrib_fractions()` | CPU-bound (HEAVY for slots 1..4) | gpu_experience_collector.rs:2163 | **6× DtoH** (cf_flip, slot_live, trail, traded, micro, total) of large per-sample buffers + multi-pass CPU summation. | **Migrate**: single fused reduction kernel over the 6 buffers; emit 4 floats. PopArt slot stays host-state (it's already a small scalar derived from `read_popart_variance`). |
| controller | `anti_lr tau gamma clip cql cost` (6 bools), `fire_frac` (f32) | host-side detection: `fire_lr` etc. | Host-state | training_loop.rs ~2902-2955 | trivial | Pure CPU comparisons against `prev_controller_values`. |
| explore | `ent_mag ent_dir` | `shannon_entropy_normalized(&[dist_q, dist_h, dist_f])` and same over `[dir0..dir3]` | Mixed | training_loop.rs ~2858 | trivial CPU loop over 3-4 elements | Source values come from `monitor.action_counts[12]` (GPU). **Migrate**: have the new GPU monitoring extension emit ent_mag and ent_dir directly. |
| explore | `sigma_mean` | `(sigma_mag + sigma_dir) * 0.5` | Host-state | training_loop.rs ~2640 | trivial | Trivial after sigma_mag/dir migrate. |

### Hidden-cost lines that materially affect total HEALTH_DIAG cost

These are computed *for* HEALTH_DIAG but happen in upstream methods invoked just before the emit:

| Site | Cost | Migration |
|---|---|---|
| `update_q_mag_means_cached` (gpu_dqn_trainer.rs:4437) | 786 KB DtoH + B×total_actions Rust loop | **DELETE the CPU path; replace with kernel.** |
| `var_scale_epoch_mean` (gpu_experience_collector.rs:2784) | N×4-byte DtoH; conditional CPU avg | Replace with kernel. |
| `trail_fire_and_hold_per_mag` (gpu_experience_collector.rs:2063) | 4× N-element DtoH; per-mag CPU loop | Replace with kernel. |
| `per_magnitude_winrate_and_variance` (gpu_experience_collector.rs:2682) | 4× N-element DtoH; per-mag CPU sum + sum_sq | Replace with kernel. |
| `reward_contrib_fractions` (gpu_experience_collector.rs:2163) | 6× large-buffer DtoH; multi-pass CPU sums | Replace with kernel. |
| `read_eval_action_distribution_per_magnitude` (gpu_backtest_evaluator.rs:1255) | actions_history_pinned read (mapped, but CPU iterates ~64-256K entries) | Replace with kernel reading the dev pointer of the same mapped buffer; CPU just reads the resulting 3 floats from the snapshot. |
| `read_eval_action_distribution_per_direction` (gpu_backtest_evaluator.rs:1339) | same buffer, different histogram | Same kernel, additional 4-float output. |
| `read_eval_intent_magnitude_distribution` (gpu_backtest_evaluator.rs:1295) | intent_mag_pinned iter | Replace with kernel; 3-float output. |
| `read_chunked_actions_direction_distribution` (gpu_backtest_evaluator.rs:1384) | picked_action_history_pinned iter | Replace with kernel; 4-float output. |

## Inventory table — line `training_loop.rs:3394` (`reward_split`)

| Metric | Source | Class | Notes |
|---|---|---|---|
| `popart` | ISV `REWARD_POPART_EMA_INDEX` | ISV-already | `read_isv_signal_at`. |
| `cf` | ISV `REWARD_CF_EMA_INDEX` | ISV-already | Same. |
| `trail` | ISV `REWARD_TRAIL_EMA_INDEX` | ISV-already | Same. |
| `micro` | ISV `REWARD_MICRO_EMA_INDEX` | ISV-already | Same. |
| `opp_cost` | ISV `REWARD_OPP_COST_EMA_INDEX` | ISV-already | Same. |
| `bonus` | ISV `REWARD_BONUS_EMA_INDEX` | ISV-already | Same. |

All 6 already on GPU; ISV-already. Snapshot fields = mirror copies.

## Inventory table — line `training_loop.rs:3430` (`aux`)

| Metric | Source | Class | Notes |
|---|---|---|---|
| `next_bar_mse` | ISV `AUX_NEXT_BAR_MSE_EMA_INDEX` (113) | ISV-already | `read_isv_signal_at`. |
| `regime_ce` | ISV `AUX_REGIME_CE_EMA_INDEX` (114) | ISV-already | Same. |
| `w` (aux_weight) | `trainer.aux_weight()` | Host-state | Plain scalar field on the trainer. |
| `label_scale` | ISV `AUX_LABEL_SCALE_EMA_INDEX` (117) | ISV-already | Same. |

## Inventory table — line `training_loop.rs:3456` (`aux_moe`)

| Metric | Source | Class | Notes |
|---|---|---|---|
| `util[0..8]` | ISV `MOE_EXPERT_UTIL_EMA_BASE..+8` | ISV-already | 8 `read_isv_signal_at`. |
| `ent` | ISV `MOE_GATE_ENTROPY_EMA_INDEX` | ISV-already | Same. |
| `λ_eff` | ISV `MOE_LAMBDA_EFF_INDEX` | ISV-already | Same. |

## Inventory table — line `metrics.rs:837` (`val`)

All 16 fields originate from `evaluator.consume_metrics_after_event()`, which already reads from `metrics_buf` (mapped-pinned, populated by `backtest_metrics_kernel`). The CPU does a small post-process to derive `trades_per_bar`, `active_frac`, `dir_entropy` — pure arithmetic over scalar floats already in the metrics struct.

| Metric | Source | Class | Notes |
|---|---|---|---|
| `sharpe` | `m.sharpe` | GPU-already | metrics_buf scalar. |
| `sortino` | `m.sortino` | GPU-already | Same. |
| `win_rate` | `m.win_rate` | GPU-already | Same. |
| `max_drawdown` | `m.max_drawdown` | GPU-already | Same. |
| `trade_count` | `m.total_trades` | GPU-already | Same. |
| `calmar` | `m.calmar` | GPU-already | Same. |
| `omega_ratio` | `m.omega_ratio` | GPU-already | Same. |
| `total_pnl` | `m.total_pnl` | GPU-already | Same. |
| `var_95` | `m.var_95` | GPU-already | Same. |
| `cvar_95` | `m.cvar_95` | GPU-already | Same. |
| `trades_per_bar` | derived (3 buckets ÷ window) | Host-state (trivial) | Three scalar f32 divides. |
| `active_frac` | derived | Host-state (trivial) | Same. |
| `dir_entropy` | derived (3-prob Shannon entropy) | Host-state (trivial) | Per-3-bucket entropy. **Could move to kernel** to land directly in `metrics_buf`; trivial enough to either keep or migrate. |
| `sharpe_annualised` | `m.sharpe` (alias) | GPU-already | Aggregator alias. |
| `profit_factor` | `m.omega_ratio` (alias) | GPU-already | Aggregator alias. |
| `window_bars` | `buy + sell + hold` | Host-state (trivial) | One add. |

**Migration note for `val [...]`:** the metrics_buf is *already* mapped-pinned, so the val emit's CPU work today is 3 scalar adds + 3 entropy multiplies. The big perf win for HEALTH_DIAG is NOT in `val`; it's in the main line's per-sample buffer reductions. We should still fold `val`'s fields into `HealthDiagSnapshot` for layout uniformity (one struct, allocator-free formatting), but the work to extend the `backtest_metrics_kernel` to directly emit `dir_entropy` etc. is optional — it can copy from `metrics_buf` into the snapshot at emit time.

## Inventory table — line `metrics.rs:890` (`val_dir_dist`)

| Metric | Source | Class | Notes |
|---|---|---|---|
| `short hold long flat` (4 floats) | `read_eval_action_distribution_per_direction` (gpu_backtest_evaluator.rs:1339) | Mixed | CPU iterates `actions_history_pinned` counting per direction. **Migrate**: kernel histogram. |

## Inventory table — line `metrics.rs:908` (`val_picked_dir_dist`)

| Metric | Source | Class | Notes |
|---|---|---|---|
| `short hold long flat` (4 floats) | `read_chunked_actions_direction_distribution` (gpu_backtest_evaluator.rs:1384) | Mixed | CPU iterates `picked_action_history_pinned`. Same pattern as above. |

## Aggregate cost summary (per-epoch, batch 16384)

Counting only the CPU-bound + Mixed sites whose work is dominated by `Vec` alloc + DtoH + per-element loops:

| Site | DtoH bytes | CPU loop work | Notes |
|---|---|---|---|
| `update_q_mag_means_cached` | ~786 KB (B × total_actions × 4) | 196 608 float-adds + 3 div | One epoch; ✗ should be a kernel. |
| `var_scale_epoch_mean` | ~N × 4 bytes (≈ 2-8 MB at full scale) | N ifs + adds | ✗. |
| `trail_fire_and_hold_per_mag` | 4× N-element DtoH (~8-32 MB) | 4N ifs + accumulate | ✗. |
| `per_magnitude_winrate_and_variance` | 4× N-element DtoH (same scale) | 4N ifs + sumsq | ✗. |
| `reward_contrib_fractions` | 6× large DtoH (≈ 24-100 MB) | 6N passes | ✗ — biggest site. |
| `read_eval_action_distribution_*` | reads mapped-pinned (no DtoH but CPU iterates) | up to 256K i32 iter × 4 (3 mag + 4 dir + 3 intent + 4 picked) | ✗ — CPU iter eats into the 70 s budget. |
| `branch_noisy_sigma_mean(0)` and `(1)` | NoisyNet param DtoH | small but per-branch | ✗ at the margins. |

Total estimated DtoH per HEALTH_DIAG emit: **~50-150 MB depending on N (alloc_episodes × alloc_timesteps)**. At PCIe Gen4 x16 (~20 GB/s sustained) that's ~3-7 ms of pure DtoH plus the host loop overhead at 1-2 ns per element on N up to 4M, which gets us into the 30-100 ms range per emit just for these readback paths. Multiplied by other epoch-boundary work (eval consumption, controller fires, optimizer maintenance) and synchronous host syncs, the 70 s/epoch number is consistent with the per-sample-buffer reductions being the dominant cost. **They are also the easiest to eliminate** because every per-sample buffer is already device-resident — we just stop reducing on the host.

## What stays CPU after the port

Per the prompt's explicit allowance ("CPU is now strictly a string formatter + log emitter"), these remain CPU:

1. **Format-string assembly** — `String::with_capacity(2048)` once per emit, fixed cap. No `Vec` on hot path.
2. **Host-state mirrors** — fields tagged `Host-state` above (controllers, mechanism flags, EMA scalars on the LearningHealth aggregator). These are already small fields on the trainer struct; we read them with no reduction. Their inclusion in `HealthDiagSnapshot` is "host writes the field via `host_ptr` immediately before format" — allocator-free, no GPU compute, trivially fast.
3. **Meta-Q predictor** (`self.meta_q.predict`) — tiny CPU MLP; staying CPU is a deliberate architectural choice, not a HEALTH_DIAG concern.

Everything in **Mixed** and **CPU-bound (HEAVY)** rows is in scope for the kernel port and the corresponding CPU code is deleted in the same commit chain.

## Snapshot field list (proposed; finalised in Phase 1 commit)

Field order matches the format-string emit order so log determinism is preserved across the port. Counts: 7 components + 8 effective + 7 novels + 2 diag + 1 gems + 10 mag + 4 grad_split_bwd + 5 grad_split_aux + 9 grad_trunk + 2 grad_abs + 6 trail + 6 mag_stats + 6 noisy + 3 eval_dist + 4 reward_contrib + 6 fire_bools + 1 fire_frac + 3 explore + 16 val + 4 val_dir_dist + 4 val_picked_dir_dist + 6 reward_split + 4 aux + 10 aux_moe + 12 monitor.action_counts (for direction-bin computation if we keep that path adjacent) ≈ **~150 f32 fields**.

The booleans become `u8` slots (0/1) so the struct stays POD with no padding surprises across CPU/GPU. The struct sits in a single mapped-pinned allocation of ~600-800 bytes — well under one cache line × 16, easy to fence + read.

## Phased commit plan (revised 2026-04-28 — kernel-by-kernel + parallel-shadow validation)

The original Phase-1-design recommendation of a 5-kernel family with sequential
landings has been adopted. Per `feedback_no_quickfixes.md`, **landing all 5
kernels in one commit risks silent numerical mismatches** that downstream
parsers (`aggregate-multi-seed-metrics.py`, smoke-test summarisers) would
mis-parse. Each kernel lands in its own commit with parallel-shadow
validation; only after all 5 are bit-identical (mod float-reordering tolerance
1e-5 relative) does Phase 4 delete the CPU paths in one shot per
`feedback_no_partial_refactor.md`.

| Phase | Commit                                          | Status     | Files touched                                                              |
|-------|-------------------------------------------------|------------|----------------------------------------------------------------------------|
| 0     | This inventory doc                              | LANDED     | `docs/health_diag_inventory.md`                                            |
| 1     | `HealthDiagSnapshot` + `MappedHealthDiagSnapshot` | LANDED   | `crates/ml/src/cuda_pipeline/health_diag.rs`                               |
| 2A    | `health_diag_isv_mirror` kernel                 | **LANDED** | `health_diag_kernel.cu` + `gpu_health_diag.rs` (NEW) + `build.rs` + trainer wiring |
| 2B    | `health_diag_eval_histogram` kernel             | Pending    | `health_diag_kernel.cu` (append) + launcher                                |
| 2C    | `health_diag_q_mag_reduce` kernel               | Pending    | `health_diag_kernel.cu` (append) + launcher (most complex; masking-heavy)  |
| 2D    | `health_diag_per_sample_reduce` kernel          | Pending    | `health_diag_kernel.cu` (append) + launcher                                |
| 2E    | `health_diag_finalise` kernel                   | Pending    | `health_diag_kernel.cu` (append) + launcher (single `__threadfence_system()`) |
| 3     | Wire kernel chain into captured graph           | Pending    | `training_loop.rs` (call sites alongside `launch_h_s2_rms_ema` etc.)       |
| 4     | Rewrite emit path + delete CPU code             | Pending    | `training_loop.rs`, `metrics.rs`, `gpu_experience_collector.rs`, `gpu_backtest_evaluator.rs`, `gpu_dqn_trainer.rs`, agent — see "Functions to delete" below |
| 5     | Final polish + audit doc updates                | Pending    | `docs/dqn-gpu-hot-path-audit.md`, `docs/dqn-wire-up-audit.md`              |

### Phase 2A — what landed

* New file `crates/ml/src/cuda_pipeline/health_diag_kernel.cu`. Single
  `health_diag_isv_mirror` kernel, single-block single-thread, ~22 ISV reads
  + ~22 snapshot writes. WORD-index table inline at top of `.cu` matches the
  Rust `HealthDiagSnapshot` field order; static_assert pins `WORD_TOTAL = 147`
  in lockstep with `snapshot_size_is_stable` test.
* New file `crates/ml/src/cuda_pipeline/gpu_health_diag.rs`. `GpuHealthDiag`
  struct holds the cubin module, the `MappedHealthDiagSnapshot`, and the
  `health_diag_isv_mirror` `CudaFunction` handle. Phases 2B-2E append further
  handles to the same struct. Public method `launch_isv_mirror`
  takes the ISV indices by-value so the kernel never embeds them.
* `build.rs` registers `health_diag_kernel.cu` in `kernels_with_common`.
* `mod.rs` declares `pub mod gpu_health_diag;`.
* `gpu_dqn_trainer.rs`:
  * imports `super::gpu_health_diag::GpuHealthDiag`;
  * adds `health_diag: Option<GpuHealthDiag>` field;
  * constructor calls `GpuHealthDiag::new(&stream)?` after the existing
    cubin loads;
  * adds `pub fn launch_health_diag_isv_mirror(&self) -> Result<(), MLError>`
    that pulls every ISV index from the trainer's compile-time constants
    (single source of truth);
  * adds `pub fn health_diag_snapshot(&self) -> Option<&MappedHealthDiagSnapshot>`
    accessor for downstream consumers.

**Snapshot fields populated by Phase 2A:** `noisy_vsn_mag`, `noisy_vsn_dir`,
`noisy_drift_mag`, `noisy_drift_dir`, `aux_next_bar_mse`, `aux_regime_ce`,
`aux_label_scale`, `aux_moe_util[0..8]`, `aux_moe_ent`, `aux_moe_lambda_eff`,
`reward_split_popart`, `reward_split_cf`, `reward_split_trail`,
`reward_split_micro`, `reward_split_opp_cost`, `reward_split_bonus`. **22 of 147
fields.** The remaining 125 are populated by Phases 2B-2D (histograms +
per-sample reductions + grad-trunk mirror).

**No production callers of `launch_health_diag_isv_mirror` yet.** Wire-up into
`training_loop.rs`'s producer family lands in Phase 3 alongside the parallel-
shadow assertion. Producer-only by design — the orchestrator is built as
additive scaffolding the same way `gpu_aux_heads.rs`'s `AuxHeadsForwardOps`
landed in Plan 4 Task 6 Commit A (handles loaded, struct in place, no
production caller yet; Commit B wires it in).

### Functions to delete in Phase 4 (per Phase 0 inventory)

Tracked here so the deletion commit can be checked against the source of
truth (this list) rather than against a freshly-greppable signal:

* `gpu_experience_collector::var_scale_epoch_mean`
* `gpu_experience_collector::trail_fire_and_hold_per_mag`
* `gpu_experience_collector::per_magnitude_winrate_and_variance`
* `gpu_experience_collector::reward_contrib_fractions`
* `gpu_experience_collector::launch_reward_component_ema_inplace` —
  KEEP (still produces ISV slots that other consumers read; only the
  HEALTH_DIAG-side CPU readback is in scope).
* (additional `gpu_experience_collector` aux readers identified per the
  inventory's "6 collector methods" line — final list pinned in Phase 4
  PR description after kernel 2D's masking semantics resolve which legacy
  methods become dead code.)
* `gpu_backtest_evaluator::read_eval_action_distribution_per_magnitude`
* `gpu_backtest_evaluator::read_eval_action_distribution_per_direction`
* `gpu_backtest_evaluator::read_eval_intent_magnitude_distribution`
* `gpu_backtest_evaluator::read_chunked_actions_direction_distribution`
* `gpu_dqn_trainer::update_q_mag_means_cached` (the 786 KB DtoH).
* `branch_noisy_sigma_mean` (agent-side method).

## Open questions for Phase 1 design review

1. **Single kernel vs small fused family?** A single mega-kernel reading every input buffer and emitting all ~150 fields is monolithic but minimises launch overhead (one kernel, one fence). A 5-7 kernel family lets us keep block configs adapted to each input's shape (per-sample buffers want N-thread reductions; ISV mirrors are scalar copies that want 1 thread). Recommended: family of 4-5 kernels run sequentially in the captured graph — `health_diag_per_sample_reduce`, `health_diag_q_mag_reduce`, `health_diag_eval_histogram`, `health_diag_isv_mirror`, `health_diag_finalise` (the last one runs `__threadfence_system()` once over the whole struct).

2. **Where does the kernel run in the captured graph?** End of the per-step training graph would update the snapshot every step (waste). End of `process_epoch_boundary`'s producer launches (next to `launch_h_s2_rms_ema`, `launch_vsn_mask_ema`, `launch_iqn_quantile_ema`, `launch_aux_heads_loss_ema`) is the right cadence — once per epoch, after all upstream producers have written.

3. **booleans + u8 in `#[repr(C)]`** — confirm padding behaviour. If padding leaks bytes between `[u8; 6]` (the controller fires) and the next `f32`, the kernel must explicitly write through padding-aware offsets; safer to lift `u8` to `u32` (4-byte aligned) at a tiny ~24-byte cost. Pinned allocs round to page anyway.

4. **`val [...]` block — extend `backtest_metrics_kernel` or copy from `metrics_buf` into the snapshot?** The cheaper path is the latter: `metrics_buf` is already mapped-pinned and the consumer reads it once already. We can have a small `health_diag_isv_mirror` kernel pull from `metrics_buf.dev_ptr` into the snapshot's val_* slots — avoids touching the existing eval graph. This keeps the eval contract stable.

5. **What about `HEALTH_DIAG[...]: TERMINATED_BY_REGRESSION ...`?** Fault-path; values are already known scalars. Keep CPU. Out of scope.