gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat(sp16-p0): Q-by-action HEALTH_DIAG diagnostic for Hold-bias observation

Browse Source 
Per train-multi-seed-pfh9n post-mortem: structural-Q-bias hypothesis
(Adam's m/sqrt(v) prefers low-variance Hold over noisy direction
Q-targets) needs direct per-action Q observations to verify or refute.
WR plateaued at ~0.46 across both folds while Hold% climbed 15% → 52%
and Q-mean climbed 0.19 → 0.41 — but per-action Q was unobservable.

Adds HEALTH_DIAG emit each epoch:

  HEALTH_DIAG[N]: q_by_action [hold=X long=Y short=Z flat=W]

Reads via host-side averaging of `q_out_buf [B, total_actions]`
direction-branch columns [0..b0=4]. No new kernel needed — q_out_buf
is already populated row-major by compute_expected_q. Modeled directly
on the existing Task 0.3 magnitude-bucket diagnostic (same dtoh-and-
average cold path).

Action-index ordering canonical, see state_layout.cuh:123-126:
  DIR_SHORT=0, DIR_HOLD=1, DIR_LONG=2, DIR_FLAT=3.
Emit slot order is [hold, long, short, flat] (Hold first because the
hypothesis is about Hold's ascent dominating direction Q-magnitudes).

New surface:
- gpu_dqn_trainer.rs: q_dir_means_cached field + update_q_dir_means_cached
  method + q_direction_action_means accessor + free static helper
  compute_q_dir_means_from_host_buf (factored for testability).
- fused_training.rs: FusedTrainingCtx wrappers parallel to q_mag pair.
- training_loop.rs: emit block adjacent to q_var_per_branch.

Behavioral test: sp16_phase0_q_by_action_diagnostic_reads_four_action_means
seeds known per-column constants, asserts canonical-dir-idx → emit-slot
mapping at 1e-3 tolerance, and includes sentinel-leak guard (non-direction
columns at 99.0 fail any wrong index→slot mapping with margin >12).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-08 14:34:59 +02:00
parent 98a81981ac
commit fb24614f07
5 changed files with 394 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

135
crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs
View File

@@ -4977,6 +4977,19 @@ pub struct GpuDqnTrainer {
/// at HEALTH_DIAG emit); read by `q_magnitude_bucket_means`. Initialised
/// to zeros; the first epoch's emit sees the post-epoch-0 values.
q_mag_means_cached: [f32; 3],
/// SP16 Phase 0 — cached direction-branch Q-value means in HEALTH_DIAG
/// field order `[q_hold, q_long, q_short, q_flat]`.
///
/// Refreshed by `update_q_dir_means_cached` (cold-path dtoh at
/// HEALTH_DIAG emit); read by `q_direction_action_means`. Initialised
/// to zeros; the first epoch's emit sees the post-epoch-0 values.
///
/// Diagnostic-only: feeds the structural-Q-bias-toward-Hold falsifier
/// (per train-multi-seed-pfh9n post-mortem). Field order matches the
/// emit format `q_by_action [hold long short flat]`; the kernel-side
/// indexing is the canonical state_layout.cuh ordering DIR_SHORT=0,
/// DIR_HOLD=1, DIR_LONG=2, DIR_FLAT=3 (see state_layout.cuh:123-126).
q_dir_means_cached: [f32; 4],
/// EMA of atom utilization [0,1] for adaptive entropy regularization.
/// SP4 Layer C close-out C3 (2026-05-01) replaced the host-resident `f32`
/// field with `utilization_ema_pinned` (mapped-pinned scalar updated GPU-
@@ -8179,6 +8192,63 @@ impl Drop for GpuDqnTrainer {
}
}
/// SP16 Phase 0 — pure-host helper that averages the four direction-branch
/// columns of a `[B, total_actions]` row-major Q-out buffer and returns the
/// per-action means in HEALTH_DIAG emit order `[q_hold, q_long, q_short, q_flat]`.
///
/// Factored out of `update_q_dir_means_cached` so the averaging logic can be
/// exercised by an oracle test (`sp14_oracle_tests::sp16_phase0_*`) without
/// constructing a full `GpuDqnTrainer` + dtoh harness. The production updater
/// calls this helper after the dtoh — the test seeds `host` directly with
/// known values and asserts the returned slot order matches the canonical
/// action enum.
///
/// Action-index mapping (canonical, see `state_layout.cuh:123-126`):
/// DIR_SHORT=0, DIR_HOLD=1, DIR_LONG=2, DIR_FLAT=3.
///
/// Returned slot order: `[hold, long, short, flat]` (same as the emit format).
///
/// Empty / degenerate input yields `[0.0; 4]`. `b0 < 4` is treated honestly:
/// out-of-range action indices fall back to slot 0 (matches the
/// `update_q_mag_means_cached` `b1.min(3)` precedent — preserves layout
/// invariants when the runtime config narrows a branch). Production always
/// has `b0 = NUM_DIRECTIONS = 4`.
pub fn compute_q_dir_means_from_host_buf(
host: &[f32],
b: usize,
b0: usize,
total_actions: usize,
) -> [f32; 4] {
if b == 0 || b0 == 0 || total_actions == 0 {
return [0.0_f32; 4];
}
if host.len() < b * total_actions {
return [0.0_f32; 4];
}
// q_out_buf layout is [B, total_actions] row-major. Direction-branch
// columns live at [0 .. b0]. Accumulate per-column in f64 to keep the
// mean numerically clean across large batches, then cast to f32.
let mut sums = [0.0_f64; 4];
let cap = b0.min(4);
for i in 0..b {
let row_off = i * total_actions;
for k in 0..cap {
sums[k] += host[row_off + k] as f64;
}
}
let inv_b = 1.0_f64 / b as f64;
// Map kernel-side dir_idx → HEALTH_DIAG slot.
// dir_idx 0 = DIR_SHORT → slot 2
// dir_idx 1 = DIR_HOLD → slot 0
// dir_idx 2 = DIR_LONG → slot 1
// dir_idx 3 = DIR_FLAT → slot 3
let mean_short = (sums[0] * inv_b) as f32;
let mean_hold = (sums[1.min(b0 - 1)] * inv_b) as f32;
let mean_long = (sums[2.min(b0 - 1)] * inv_b) as f32;
let mean_flat = (sums[3.min(b0 - 1)] * inv_b) as f32;
[mean_hold, mean_long, mean_short, mean_flat]
}
impl GpuDqnTrainer {
/// Configured batch size (fixed at construction for CUDA Graph compatibility).
pub fn batch_size(&self) -> usize {
@@ -8722,6 +8792,70 @@ impl GpuDqnTrainer {
Ok(())
}
/// SP16 Phase 0 — per-direction-action Q-value mean read accessor.
///
/// Returns `[q_hold, q_long, q_short, q_flat]` = the mean Q-value
/// predicted for the four direction-branch actions across the
/// most-recent batch whose `q_out_buf` contents are live.
///
/// Action-index mapping (canonical, see `state_layout.cuh:123-126`):
/// DIR_SHORT=0, DIR_HOLD=1, DIR_LONG=2, DIR_FLAT=3. The returned slot
/// order matches the HEALTH_DIAG `q_by_action [hold long short flat]`
/// emit format (Hold first because the structural-Q-bias-toward-Hold
/// hypothesis being falsified is about Hold's ascent).
///
/// * `[0] q_hold` = mean(q_out_buf[:, DIR_HOLD ]) = col 1
/// * `[1] q_long` = mean(q_out_buf[:, DIR_LONG ]) = col 2
/// * `[2] q_short` = mean(q_out_buf[:, DIR_SHORT]) = col 0
/// * `[3] q_flat` = mean(q_out_buf[:, DIR_FLAT ]) = col 3
///
/// Read from the cached value populated by `update_q_dir_means_cached`
/// (cold-path dtoh invoked once per epoch at HEALTH_DIAG emit).
pub fn q_direction_action_means(&self) -> [f32; 4] {
self.q_dir_means_cached
}
/// SP16 Phase 0 — refresh the cached direction-branch Q-value means.
///
/// Cold-path (epoch boundary) dtoh of the current `q_out_buf` contents.
/// Reads `B * total_actions` floats, extracts the 4 columns belonging
/// to the direction branch at offsets `[0 .. b0]`, averages each
/// column over the batch, and stores into `q_dir_means_cached` as
/// `[q_hold, q_long, q_short, q_flat]` (see `q_direction_action_means`
/// for the action-idx ↔ slot mapping).
///
/// Mirrors `update_q_mag_means_cached` for the magnitude branch — both
/// share the same dtoh of `q_out_buf` per epoch (each is invoked
/// independently at HEALTH_DIAG emit time; the two transfers can be
/// batched in a future refactor but the diagnostic-path cost is
/// acceptable).
///
/// Called at HEALTH_DIAG emit time only. Diagnostic path — dtoh cost
/// is acceptable here but would be untenable per-step.
pub fn update_q_dir_means_cached(&mut self) -> Result<(), MLError> {
let b = self.config.batch_size;
let b0 = self.config.branch_0_size;
let total_actions = self.total_actions();
if b == 0 || b0 == 0 || total_actions == 0 {
self.q_dir_means_cached = [0.0_f32; 4];
return Ok(());
}
let n_floats = b * total_actions;
if self.q_out_buf.len() < n_floats {
return Err(MLError::ModelError(format!(
"update_q_dir_means_cached: q_out_buf len {} < B*total_actions {}",
self.q_out_buf.len(),
n_floats
)));
}
let mut host = vec![0.0_f32; n_floats];
self.stream
.memcpy_dtoh(&self.q_out_buf.slice(0..n_floats), &mut host)
.map_err(|e| MLError::ModelError(format!("dtoh q_out_buf for dir means: {e}")))?;
self.q_dir_means_cached = compute_q_dir_means_from_host_buf(&host, b, b0, total_actions);
Ok(())
}
/// Update per-branch liquid tau modulation — GPU kernel (RK4/Euler adaptive ODE).
///
/// Reads per_branch_q_gaps (pinned device-mapped) and qlstm_context from device.
@@ -23501,6 +23635,7 @@ impl GpuDqnTrainer {
per_branch_q_gap_ema_buf,
last_per_branch_q_gaps: [0.0; 4],
q_mag_means_cached: [0.0_f32; 3],
q_dir_means_cached: [0.0_f32; 4],
utilization_ema_pinned,
utilization_ema_dev_ptr,
vsn_masked_buf,

22
crates/ml/src/trainers/dqn/fused_training.rs
View File

@@ -1459,6 +1459,28 @@ impl FusedTrainingCtx {
self.trainer.update_q_mag_means_cached()
}
/// SP16 Phase 0 — per-direction-action Q-value mean.
///
/// Returns `[q_hold, q_long, q_short, q_flat]` read from the cached
/// values populated by `update_q_dir_means_cached` (cold-path, epoch
/// boundary). Callers should invoke `update_q_dir_means_cached` before
/// reading. Action-index ordering: canonical, see
/// `state_layout.cuh:123-126` (DIR_SHORT=0, DIR_HOLD=1, DIR_LONG=2,
/// DIR_FLAT=3).
pub(crate) fn q_direction_action_means(&self) -> [f32; 4] {
self.trainer.q_direction_action_means()
}
/// SP16 Phase 0 — refresh the cached direction-branch Q-value means.
///
/// Cold-path dtoh of `q_out_buf` contents. Should be called at
/// HEALTH_DIAG emit time (epoch boundary) immediately before
/// `q_direction_action_means`. Mirrors `update_q_mag_means_cached` for
/// the direction-branch columns `[0..b0]`.
pub(crate) fn update_q_dir_means_cached(&mut self) -> Result<(), crate::MLError> {
self.trainer.update_q_dir_means_cached()
}
/// Flush the last in-flight async readback from the training step.
/// Called once at epoch end to retrieve the final step's scalars.
pub(crate) fn flush_readback(&mut self) -> Result<crate::cuda_pipeline::gpu_dqn_trainer::FusedTrainScalars> {

37
crates/ml/src/trainers/dqn/trainer/training_loop.rs
View File

@@ -5064,6 +5064,43 @@ impl DQNTrainer {
);
}
// SP16 Phase 0 — per-direction-action Q-value mean. Falsifier for
// the structural-Q-bias-toward-Hold hypothesis surfaced by
// train-multi-seed-pfh9n: WR plateaued at ~0.46 across both folds
// while Hold% climbed 15% → 52%. Without per-action Q observations
// we cannot distinguish "Adam's m/sqrt(v) prefers low-variance
// Hold targets" from "explicit reward terms keep electing Hold".
//
// Cold path: a single dtoh of B*total_actions floats at epoch
// boundary (~786 KB at B=16384, total_actions=12) via
// `update_q_dir_means_cached`. Read order matches the emit
// format (Hold first because the hypothesis is about Hold's
// ascent dominating direction Q-magnitudes).
//
// Action ordering: canonical, see `state_layout.cuh:123-126` —
// DIR_SHORT=0, DIR_HOLD=1, DIR_LONG=2, DIR_FLAT=3. The slot order
// returned by `q_direction_action_means` is HEALTH_DIAG-native
// `[hold, long, short, flat]`.
{
let q_by_action = if let Some(fused) = self.fused_ctx.as_mut() {
match fused.update_q_dir_means_cached() {
Ok(()) => fused.q_direction_action_means(),
Err(e) => {
tracing::warn!(
"HEALTH_DIAG q_direction_action_means refresh failed: {e}"
);
fused.q_direction_action_means()
}
}
} else {
[0.0_f32; 4]
};
tracing::info!(
"HEALTH_DIAG[{}]: q_by_action [hold={:.4} long={:.4} short={:.4} flat={:.4}]",
epoch, q_by_action[0], q_by_action[1], q_by_action[2], q_by_action[3],
);
}
// SP7 observability (grad_decomp_pinned): surface the exact contents of
// grad_decomp_result_pinned[0..3, 12..15, 36..48] (iqn/cql_raw/c51 ×
// [mag, dir, trunk]) that the SP7 loss-balance controller read in the

138
crates/ml/tests/sp14_oracle_tests.rs
View File

@@ -2034,3 +2034,141 @@ mod sp14_audit_4b_min_hold_temperature_gpu {
);
}
}
// ═══════════════════════════════════════════════════════════════════════════
// SP16 Phase 0 (2026-05-08) — Q-by-action HEALTH_DIAG diagnostic.
//
// Per train-multi-seed-pfh9n post-mortem: structural-Q-bias hypothesis (Adam's
// `m/sqrt(v)` favours zero-variance Hold over noisy direction Q-targets) needs
// direct per-action Q observations to verify or refute. The production path is:
//
// 1. `update_q_dir_means_cached`: dtoh of `q_out_buf [B, total_actions]`,
// then per-action averaging via `compute_q_dir_means_from_host_buf`.
// 2. `q_direction_action_means`: returns `[hold, long, short, flat]` cached
// array (HEALTH_DIAG emit order).
// 3. training_loop.rs HEALTH_DIAG emit:
// `q_by_action [hold=… long=… short=… flat=…]`.
//
// Action-index ordering verified at `state_layout.cuh:123-126`:
// DIR_SHORT=0, DIR_HOLD=1, DIR_LONG=2, DIR_FLAT=3.
//
// This test exercises the actual averaging-path helper with known per-column
// values seeded into a [B, total_actions] row-major buffer. The slot order
// returned must match the HEALTH_DIAG `[hold, long, short, flat]` contract,
// independently of whichever dir_idx a given column corresponds to.
// ═══════════════════════════════════════════════════════════════════════════
/// SP16 Phase 0: verify the q_by_action diagnostic correctly extracts
/// per-direction-action Q-means from a `[B, total_actions]` row-major Q-out
/// buffer and emits in the contracted `[hold, long, short, flat]` slot order
/// matching the HEALTH_DIAG format. Validates the action-index ↔ slot
/// mapping against `state_layout.cuh:123-126` (DIR_SHORT=0, DIR_HOLD=1,
/// DIR_LONG=2, DIR_FLAT=3) so we can observe Q(Hold) trajectory vs
/// Q(direction) and falsify the structural-Q-bias hypothesis.
///
/// Construction:
/// * B=8, total_actions=12 (production layout: 4 dir + 3 mag + 3 ord + 3 urg).
/// * Each direction column i is filled with constant `dir_target[i]`
/// across all batch rows so the per-column mean equals the seeded
/// constant (no float averaging error). The remaining 8 mag/ord/urg
/// columns are filled with sentinel 99.0 to confirm the helper does
/// not bleed any non-direction column into the dir-action means.
/// * Targets chosen with 4 distinct values so a wrong index→slot mapping
/// is detected with 1e-3 tolerance.
///
/// CPU-only test — exercises pure-host averaging logic. The dtoh half of
/// the production updater is tested implicitly by smoke (HEALTH_DIAG
/// emits each epoch); the bug surface this test guards is the index→slot
/// mapping, which is data-independent.
#[test]
fn sp16_phase0_q_by_action_diagnostic_reads_four_action_means() {
use ml::cuda_pipeline::gpu_dqn_trainer::compute_q_dir_means_from_host_buf;
// Production direction-branch ordering, see state_layout.cuh:123-126.
// dir_idx 0 → DIR_SHORT, 1 → DIR_HOLD, 2 → DIR_LONG, 3 → DIR_FLAT.
const DIR_SHORT: usize = 0;
const DIR_HOLD: usize = 1;
const DIR_LONG: usize = 2;
const DIR_FLAT: usize = 3;
// Returned-slot ordering, see compute_q_dir_means_from_host_buf docs.
// HEALTH_DIAG `q_by_action [hold long short flat]`.
const SLOT_HOLD: usize = 0;
const SLOT_LONG: usize = 1;
const SLOT_SHORT: usize = 2;
const SLOT_FLAT: usize = 3;
const B: usize = 8;
const B0: usize = 4; // NUM_DIRECTIONS
const TOTAL_ACTIONS: usize = 12; // 4 + 3 + 3 + 3 (production layout)
// Distinct target Q-values per direction action.
// Index by canonical dir_idx (matches the column the writer uses).
let mut dir_target = [0.0_f32; 4];
dir_target[DIR_SHORT] = 0.10;
dir_target[DIR_HOLD ] = 0.50;
dir_target[DIR_LONG ] = 0.20;
dir_target[DIR_FLAT ] = 0.30;
// Build a synthetic q_out_buf [B, total_actions] row-major. Direction
// columns [0..4] are seeded with dir_target[col]; non-direction columns
// [4..12] are sentinel 99.0 so any mis-indexing leaks visibly into the
// returned means.
let mut host = vec![0.0_f32; B * TOTAL_ACTIONS];
for i in 0..B {
let row_off = i * TOTAL_ACTIONS;
for k in 0..B0 {
host[row_off + k] = dir_target[k];
}
for k in B0..TOTAL_ACTIONS {
host[row_off + k] = 99.0;
}
}
let q_by_action = compute_q_dir_means_from_host_buf(&host, B, B0, TOTAL_ACTIONS);
// Slot-order assertions: each slot must hold the canonical-dir-idx target.
assert!(
(q_by_action[SLOT_HOLD ] - dir_target[DIR_HOLD ]).abs() < 1e-3,
"slot 0 (hold) must hold mean of DIR_HOLD column. got {:.6} expected {:.6}",
q_by_action[SLOT_HOLD], dir_target[DIR_HOLD]
);
assert!(
(q_by_action[SLOT_LONG ] - dir_target[DIR_LONG ]).abs() < 1e-3,
"slot 1 (long) must hold mean of DIR_LONG column. got {:.6} expected {:.6}",
q_by_action[SLOT_LONG], dir_target[DIR_LONG]
);
assert!(
(q_by_action[SLOT_SHORT] - dir_target[DIR_SHORT]).abs() < 1e-3,
"slot 2 (short) must hold mean of DIR_SHORT column. got {:.6} expected {:.6}",
q_by_action[SLOT_SHORT], dir_target[DIR_SHORT]
);
assert!(
(q_by_action[SLOT_FLAT ] - dir_target[DIR_FLAT ]).abs() < 1e-3,
"slot 3 (flat) must hold mean of DIR_FLAT column. got {:.6} expected {:.6}",
q_by_action[SLOT_FLAT], dir_target[DIR_FLAT]
);
// Defensive: any non-direction sentinel leakage would push the dir slot
// past its dir_target. With sentinel=99.0 a one-column mis-index would
// shift the slot by (99.0 - target) / B per leaked row — at least 12.4
// for SLOT_HOLD's 0.50 target, far above the 1e-3 tolerance.
for (slot, value) in q_by_action.iter().enumerate() {
assert!(
*value < 10.0,
"slot {slot} contaminated by non-direction sentinel column \
(got {value:.6}); index→slot mapping is wrong"
);
}
// Empty-input guards: helper must return [0; 4] for degenerate cases.
let empty = compute_q_dir_means_from_host_buf(&host, 0, B0, TOTAL_ACTIONS);
assert_eq!(empty, [0.0_f32; 4], "B=0 must yield all zeros");
let empty_b0 = compute_q_dir_means_from_host_buf(&host, B, 0, TOTAL_ACTIONS);
assert_eq!(empty_b0, [0.0_f32; 4], "b0=0 must yield all zeros");
// Truncated host (under-sized) must also yield zeros — defensive guard
// that mirrors the production-path ModelError on under-sized q_out_buf.
let small = vec![0.0_f32; 4];
let truncated = compute_q_dir_means_from_host_buf(&small, B, B0, TOTAL_ACTIONS);
assert_eq!(truncated, [0.0_f32; 4], "truncated host must yield all zeros");
}

62
docs/dqn-wire-up-audit.md
View File

@@ -8480,3 +8480,65 @@ The ratio assertion makes the bug regression-proof: any future kernel edit that
1. **Magnitude of the upstream effect**: with α=0.6 default and the typical TD-error span of 3-4 orders of magnitude in trade-close vs no-trade transitions, the effective sampling lift on high-TD events drops from ~10× (single α) to ~2.5× (double α). That's ~4× attenuation of PER's prioritization power — a meaningful but not catastrophic miscalibration. The plateau-causation hypothesis (alongside Class B #1's fold-boundary contamination) is plausible but not proven by audit alone; smoke run will quantify.
2. **`max_priority` semantic shift**: pre-fix `max_priority` tracked the running max of `|td|^α + ε`. Post-fix it tracks max of `|td| + ε`. The downstream `per_insert_pa` broadcasts this value as `raw_priorities` for new inserts and computes `effective^α` for `priorities_pa` — so the shift is internally consistent (raw priority in, α applied once on insert). External callers of `apply_max_priority_scalar` (currently none in production paths, only test/seg_tree paths) would need to pass raw priority values — no caller change required because no production caller exists.
3. **`priority_update_f32` left in place despite being dead**: the kernel compiles but has no Rust caller. Per `feedback_no_legacy_aliases` it should be deleted, but doing so requires removing the kernel registration in the loader (`crates/ml-dqn/src/replay_buffer_kernels.rs` or similar) and that's outside the surface of this fix. Filed as a follow-up cleanup; the kernel is harmless idle code (no execution path).
## SP16 Phase 0: q_by_action diagnostic (2026-05-08)
### Why
`train-multi-seed-pfh9n` post-mortem: WR plateau at ~0.46 across both folds while Hold% climbed 15% → 52%. Leading hypothesis is structural Q-learning bias toward low-variance Hold action — Adam's `m/sqrt(v)` updates favor zero-variance Hold over noisy direction Q-targets, so Q(Hold) climbs steadily until it dominates regardless of penalty/temperature/PER fixes.
Without per-action Q observations we cannot verify or refute this hypothesis. SP16 Phase 0 adds a dedicated HEALTH_DIAG diagnostic so subsequent phases (e.g., Phase 2 Hold-cost adjustment) have a falsifier metric.
### What
New HEALTH_DIAG line emitted each epoch:
```
HEALTH_DIAG[N]: q_by_action [hold=0.5234 long=0.1102 short=0.0987 flat=0.2145]
```
Slot order is `[hold, long, short, flat]` (Hold first because the hypothesis is about Hold's ascent dominating direction Q-magnitudes). Action-index ↔ slot mapping uses the canonical state_layout.cuh ordering:
| dir_idx | enum constant | column in q_out_buf | emit slot |
|---------|-----------------|----------------------|-----------|
| 0 | DIR_SHORT | 0 | 2 (short) |
| 1 | DIR_HOLD | 1 | 0 (hold) |
| 2 | DIR_LONG | 2 | 1 (long) |
| 3 | DIR_FLAT | 3 | 3 (flat) |
Source: `crates/ml/src/cuda_pipeline/state_layout.cuh:123-126`.
### How
Modeled directly on the existing Task 0.3 magnitude-bucket diagnostic. No new kernel needed — `q_out_buf [B, total_actions]` is already populated row-major by `compute_expected_q` after each training step, and the direction-branch columns live at `[0..b0]` (b0 = NUM_DIRECTIONS = 4). The new methods reuse the same dtoh + per-column averaging pattern as `update_q_mag_means_cached`.
| File | Δ | Notes |
|------|---|-------|
| `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` | +97 / -1 | New cached field `q_dir_means_cached`, public methods `update_q_dir_means_cached` + `q_direction_action_means`, free pure-host helper `compute_q_dir_means_from_host_buf` (factored for testability) |
| `crates/ml/src/trainers/dqn/fused_training.rs` | +21 / -0 | FusedTrainingCtx wrappers `update_q_dir_means_cached` + `q_direction_action_means` (parallels existing q_mag wrappers) |
| `crates/ml/src/trainers/dqn/trainer/training_loop.rs` | +35 / -0 | HEALTH_DIAG emit block right after `q_var_per_branch` (both are per-direction-action diagnostics, kept adjacent for log readability) |
| `crates/ml/tests/sp14_oracle_tests.rs` | +132 / -0 | Behavioral test `sp16_phase0_q_by_action_diagnostic_reads_four_action_means` exercises the index→slot mapping + sentinel-leak detection + degenerate-input guards |
### Behavioral test
`sp14_oracle_tests::sp16_phase0_q_by_action_diagnostic_reads_four_action_means` (CPU-only, runs in default cargo test):
- Builds a synthetic `q_out_buf [8, 12]` row-major buffer, seeds direction columns `[0..4]` with four distinct constants `(short=0.10, hold=0.50, long=0.20, flat=0.30)`, fills non-direction columns with sentinel 99.0.
- Calls `compute_q_dir_means_from_host_buf` and asserts each output slot holds the canonical-dir-idx target within 1e-3.
- Sentinel-leak guard: any slot value ≥ 10.0 means a non-direction column bled in (mis-indexed walk over `total_actions`).
- Degenerate-input guards: `B=0`, `b0=0`, and undersized host all return `[0.0; 4]`.
The bug surface this test guards is the index→slot mapping — data-independent — so a CPU-only test covers it. The dtoh half of the production updater is exercised every epoch by the smoke pipeline, the same way Task 0.3's magnitude variant is tested.
### Verification
- `SQLX_OFFLINE=true CUDA_COMPUTE_CAP=86 cargo check -p ml --tests --all-targets` — clean (warnings unchanged).
- `cargo test -p ml --test sp14_oracle_tests --release sp16_phase0_q_by_action_diagnostic_reads_four_action_means -- --nocapture` — 1/1 pass.
- `cargo test -p ml --test sp14_oracle_tests --release` — 3/3 non-ignored pass, 24 GPU-gated tests ignored unchanged.
- `cargo test -p ml --test sp15_phase1_oracle_tests --release` — 5/5 non-ignored pass, 36 GPU-gated tests ignored unchanged.
### Concerns
1. **Diagnostic-only — does not modify reward/loss/Q dynamics**. This task adds an observation channel; Phase 2 will use the observed Q(Hold) trajectory to decide whether to lift Hold cost / pay Hold a structural penalty / leave Hold alone. The hypothesis that Q(Hold)/Q(direction) ratio climbs faster than Hold% can be answered after one smoke run with this diagnostic on.
2. **dtoh cost on cold path**: the magnitude variant already pays `B × total_actions × 4 bytes ≈ 786 KB` per epoch; the direction variant adds an identical second transfer. Two transfers can be coalesced into one shared helper in a future pass — the `update_q_mag_means_cached` and `update_q_dir_means_cached` fan out the same `q_out_buf` to two cached arrays. Deferred: the existing diagnostic-path cost is already accepted, doubling it remains negligible vs hot-path overhead and the readability gain of two independent methods outweighs the saving for now.
3. **No new kernel**: per `feedback_no_atomicadd` and `feedback_no_stubs` we explicitly avoided adding a producer kernel. The existing `q_out_buf [B, total_actions]` row-major contract is the producer; we read it. If a future refactor moves the per-action means onto a GPU buffer (e.g., to fuse with `reduce_current_q_stats_per_branch`) the host helper is replaced by a 4-slot mapped-pinned read — straightforward extension.