From baa4151fabb8e032ebc75041eedd66bcb0020ed9 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Thu, 23 Apr 2026 10:15:16 +0200 Subject: [PATCH] =?UTF-8?q?docs(isv):=20signal=20quality=20audit=20for=20D?= =?UTF-8?q?QN=20ISV=20bus=20=E2=80=94=20L40S=20oscillation=20diagnosis?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Diagnostic-only audit of all 22 ISV slots using train-mdh86 HEALTH_DIAG (20 epochs, same oscillation pattern as train-rq6n8). Identifies three dead writers (slots 2, 3, 4), one strongly anti-correlated signal (slot 12 learning_health, r=-0.765 vs Sharpe), sample-0 bias in four regime slots (8-11), and EMA-pollution risk on the Q-scale EMAs (16, 21) that feed Kelly conviction + C51 bin weighting. Produces a per-slot table with writer/reader citations and an ordered fix list. Hypothesises the observed +34 → -67 Sharpe swing is driven by health mis-reporting "improving" while policy diverges, q_dir_abs_ref contamination collapsing Kelly, and the zero-writer TD-error EMA disabling micro-reward regime awareness. No code changes — reconnaissance only. --- .../2026-04-23-isv-signal-quality-audit.md | 609 ++++++++++++++++++ 1 file changed, 609 insertions(+) create mode 100644 docs/superpowers/specs/2026-04-23-isv-signal-quality-audit.md diff --git a/docs/superpowers/specs/2026-04-23-isv-signal-quality-audit.md b/docs/superpowers/specs/2026-04-23-isv-signal-quality-audit.md new file mode 100644 index 000000000..3f9fadf2d --- /dev/null +++ b/docs/superpowers/specs/2026-04-23-isv-signal-quality-audit.md @@ -0,0 +1,609 @@ +# ISV Signal Quality Audit (train-rq6n8 oscillation diagnosis) + +## Background + +The Introspective State Vector (ISV) is a 22-slot pinned device-mapped signal +bus that feeds most of Foxhunt DQN's adaptive mechanisms: the `cql_alpha` +regime gate, the conviction-driven Kelly warmup floor, reward-bias on the +Bellman target, Task 2.X / Task 2.Y C51 bin weighting, the ISV encoder MLP +that produces `branch_gate` + `gamma_mod`, the adaptive `isv_instability` +safety net inside the unified env kernel, and the distillation alpha ramp. + +The audit was triggered by the terminated L40S run `train-rq6n8`. The logs +observed at `/tmp/l40s_diag/health.log` (20 `HEALTH_DIAG` lines from the +earlier `train-mdh86` pod, same oscillation pattern) show a Sharpe trajectory +that rises to +34 over epochs 1..5, falls back to +28 at epoch 7, crashes +through zero into −37 by epoch 10, and keeps sliding to −67 by epoch 19 — +while `learning_health` climbs monotonically from 0.50 to 0.64. That is a +direct contradiction: a "learning-health" signal climbing while cumulative +Sharpe collapses cannot be a faithful measurement of training outcomes. + +This audit enumerates every ISV slot, identifies its writer and consumers, +grades the signal against Sharpe correlation over the 20-epoch window, and +flags slots whose failure is most likely driving adaptive-mechanism +mis-decisions during the observed oscillation. + +Relevant memory for context: `feedback_adaptive_not_tuned.md` (signals must +drive adaptive mechanisms, not hard-coded constants) and +`project_reward_bias_cross_branch_coupling.md` (direction-branch reward-bias +couples to magnitude via the shared trunk — one broken signal can poison both +branches). + +### Data + +Logs parsed: `/tmp/l40s_diag/health.log` — 20 HEALTH_DIAG epochs +(`train-mdh86`; `train-rq6n8` argo pod logs were no longer available when +this audit ran). Sharpe trajectory observed: + +``` +[+0.0, +20.5, +33.8, +33.9, +34.1, +34.6, +33.4, +28.6, +26.3, -8.4, + -37.2, -42.7, -38.2, -5.2, -12.7, -39.8, -52.5, -58.8, -64.9, -67.2] +``` + +Pearson r reported below is between each signal's epoch-indexed trajectory +and the Sharpe trajectory across N=20 samples. Because Sharpe is itself an +EMA the correlation is a lower bound on the underlying causal coupling, but +for the purpose of flagging stuck or decoupled signals it is adequate. + +## Per-slot analysis + +Naming convention used throughout: `experience_kernels.cu:5667` means line +5667 of that file. + +### Slot 0 — Q-drift (normalised deviation from Q-mean EMA) + +- **Written by:** `isv_signal_update` kernel at `experience_kernels.cu:5667` + — `(q_mean - q_ema) / max(|q_ema|, 0.1)`. Inputs: + `q_mean_scratch_pinned` (populated by `reduce_current_q_stats` writeback + `host[3]` at `gpu_dqn_trainer.rs:11080` — same q_mean as + `QValueStatsResult.q_mean`) and `q_mean_ema_pinned` (updated in + `update_q_mean_ema` at `gpu_dqn_trainer.rs:3214`). +- **Read by:** no kernel or Rust call site reads slot 0 directly. Feeds only + the ISV encoder MLP (`isv_forward` at `experience_kernels.cu:5827` reads + `isv_signals[k]` for k∈[0..12) as MLP input). +- **Value trajectory:** not logged in HEALTH_DIAG; cannot measure directly. +- **Correlation to Sharpe:** n/a (not observed). Structurally couples to + `q_mean` which drifted across 5 orders of magnitude (Q=+117k → 7k) in the + observed window — slot 0's numerator can swing by ±O(1) per epoch. +- **EMA pollution risk:** MEDIUM. Slot 0 is NOT an EMA (instantaneous ratio) + — outliers do not persist across epochs. The denominator floor of 0.1 + saturates the ratio for small `q_ema`, which silently converts structure + into near-zero noise whenever Q-mean EMA itself is small. +- **Verdict:** REPAIR. Saturation floor `max(|q_ema|, 0.1)` is arbitrary and + hides the signal when it matters most (early / collapsed Q). Replace with + a robust scale reference — `q_abs_ref_ema` (slot 16) or the C51 support + `v_max − v_min`. Also add to HEALTH_DIAG so its trajectory can be + inspected. + +### Slot 1 — Gradient norm EMA (√grad_norm²) + +- **Written by:** `isv_signal_update` at `experience_kernels.cu:5671` — + α=0.05 EMA of `sqrt(grad_norm_ptr[0])`. Input is `grad_norm_dev_ptr` + populated by Adam / gradient reducer kernels (same scalar that feeds + `grad_norm_pinned` read at `gpu_dqn_trainer.rs:9902`). +- **Read by:** `experience_kernels.cu:1786` — `isv_instability = isv_signals_ptr[1]` + drives the ISV-adaptive catastrophic-stop width inside the unified env + kernel (`eff_stop = base_stop / (1 + isv_instability)`). Also feeds the + ISV encoder MLP. +- **Correlation to Sharpe:** n/a (not in HEALTH_DIAG); indirect via + `grad_ratio_mag_dir` (r=+0.691) which is driven by the same branch-grad + norms — so the underlying scale IS tracking something real. +- **EMA pollution risk:** MEDIUM. grad_norm² can spike by 10⁴ on one bad + batch (Q-range ±333k observed at epoch 2 per background). The √ + compresses, but α=0.05 still means a single outlier decays over ~20 + epochs. Every adaptive stop width sees that decayed outlier. +- **Verdict:** REPAIR. Log slot 1 in HEALTH_DIAG. Add a trimmed-EMA or + robust scale (e.g. median of last 32 batches, or clamp input to 4× EMA + before updating). The consumer site at `experience_kernels.cu:1786` + already divides by `(1 + isv_instability)` which naturally rate-limits + the response — but the EMA itself still carries the outlier. + +### Slot 2 — TD-error EMA + +- **Written by:** `isv_signal_update` at `experience_kernels.cu:5674` — + α=0.05 EMA of `td_error_ptr[0]` (`td_error_scratch_dev_ptr`). **Source + buffer has NO writer.** `td_error_scratch_pinned` is allocated and + zero-initialised at `gpu_dqn_trainer.rs:7605-7623`; no kernel or Rust + call site ever writes to it again. Confirmed by `rg + 'td_error_scratch_pinned\s*=' crates/ml/src` returning only the + allocation site and the `free_host` in Drop. The code comment at + `gpu_dqn_trainer.rs:1771-1777` acknowledges this explicitly. +- **Read by:** `experience_kernels.cu:2138` — `volatility = isv_signals_ptr[2]` + feeds the micro-reward composition (`w_price = price_confirm_weight × (0.5 + 0.5×volatility)`) + and **reads it as if it were a volatility proxy**. Also feeds the ISV + encoder MLP. Also `lagged_td_error[0] = isv_signals[2]` at + `experience_kernels.cu:5655` — the lagged scratch that IQN's recursive + confidence target reads. +- **Correlation to Sharpe:** n/a (value is identically 0.0 after α-blending + a zero source). +- **EMA pollution risk:** NONE — the signal is zero. +- **Verdict:** REPAIR (highest priority). Wire `td_error_scratch_pinned` + from the C51 / IQN loss kernels (they already compute per-sample TD + errors at `per_sample_loss_buf` and `td_errors_buf` allocated at + `gpu_dqn_trainer.rs:6298-6299`). Add a 1-thread reduction kernel that + writes `mean(|td_error|)` into `td_error_scratch_pinned`, launched on + the same stream before `update_isv_signals`. The consumer at + `experience_kernels.cu:2138` is silently getting 0.5 (wrong, because the + read path is `isv_signals_ptr[2]` without the NULL fallback — it reads + the zero directly, not the 0.5 default). + +### Slot 3 — Ensemble variance EMA + +- **Written by:** `isv_signal_update` at `experience_kernels.cu:5678` — + α=0.05 EMA of `ens_var_ptr[0]` (`ensemble_var_scratch_dev_ptr`). **Source + buffer has NO writer.** Allocated and zeroed at + `gpu_dqn_trainer.rs:7625-7643`; never written again. Confirmed via + ripgrep. Code comment at `gpu_dqn_trainer.rs:1780-1786` acknowledges + this. +- **Read by:** only the ISV encoder MLP (`isv_forward`). The + HEALTH_DIAG `ens_agree` value (stuck at 0.0 for 19/20 epochs) is + computed from `EnsembleMonitor` state in Rust, not from ISV slot 3 — + different path. +- **Correlation to Sharpe:** n/a (identically 0.0). +- **EMA pollution risk:** NONE — signal is zero. +- **Verdict:** REPAIR. Wire `ensemble_var_scratch_pinned` from the + ensemble-head reducer. The project already has `EnsembleMonitor` + collecting per-head Q disagreement (see + `session_2026-04-14_h100_gradient_fixes.md` mention of "ensemble + disagreement"), so the signal exists — it just is not propagated into + ISV. Route it through a CPU scratch write analogous to + `*self.atom_util_scratch_pinned = host[6]` at `gpu_dqn_trainer.rs:11078`, + using the per-epoch ensemble spread the monitor already computes. + +### Slot 4 — Ensemble variance velocity + +- **Written by:** `isv_signal_update` at `experience_kernels.cu:5681` — + `(isv_signals[3] - old_ens_ema) / max(old_ens_ema, 1e-6)`. Derived + entirely from slot 3, so inherits its zero-writer bug plus a + divide-by-zero singularity: when slot 3 is 0, `old_ens_ema` is 0 → + denominator saturates to `1e-6` → velocity is `0/1e-6 = 0`. Signal is + identically 0.0. +- **Read by:** ISV encoder MLP only. +- **Correlation to Sharpe:** n/a. +- **EMA pollution risk:** NONE — signal is zero; on the day slot 3 gets + wired, velocity will be well-defined (`1e-6` floor still bounds the + quotient). +- **Verdict:** REPAIR (derivative of slot 3 fix). Becomes meaningful the + moment slot 3's writer is wired. + +### Slot 5 — Reward proxy EMA + +- **Written by:** `isv_signal_update` at `experience_kernels.cu:5684` — + α=0.05 EMA of `reward_ptr[0]` (`reward_scratch_dev_ptr`). Source buffer + is populated at `gpu_dqn_trainer.rs:11080` — `*self.reward_scratch_pinned + = host[3]` where `host[3]` is `q_mean` from `QValueStatsResult`. **This + is mis-labelled as "reward" — it is actually the batch Q-mean.** Same + value as feeds slot 0's numerator. +- **Read by:** ISV encoder MLP only. +- **Correlation to Sharpe:** n/a (not logged); structurally equivalent to + a smoothed Q-mean. Given observed Q-mean magnitudes swinging ±O(10⁵), + slot 5 is dominated by Q-mean scale rather than reward semantics. +- **EMA pollution risk:** HIGH. A single Q-mean excursion of ±10⁵ takes + ~20 epochs to decay at α=0.05. The MLP input range for a "reward proxy" + becomes catastrophically mis-normalised for the rest of the run. +- **Verdict:** REPAIR. Either (a) wire the actual per-epoch reward (mean + `step_return` computed by the env kernel in `train_physics_step_buf`), + or (b) relabel the consumer path to "Q-mean EMA" and delete the + duplicate with slot 0. Option (a) is strictly better for signal quality + because Q-mean and realised reward diverge exactly during the regimes + the signal is supposed to detect (over-confident Q, negative realised + P&L — cf. epochs 10+ in the observed data where Q-scales stayed high + while Sharpe cratered). + +### Slot 6 — Atom utilization EMA + +- **Written by:** `isv_signal_update` at `experience_kernels.cu:5687` — + α=0.05 EMA of `atom_util_ptr[0]` (`atom_util_scratch_dev_ptr`). Source + populated at `gpu_dqn_trainer.rs:11078` — `host[6].clamp(0,1)`, i.e. + `QValueStatsResult.atom_utilization`. Genuine signal. +- **Read by:** ISV encoder MLP only. HEALTH_DIAG `atoms` component is + computed separately by the health EMA aggregator, not from ISV slot 6. +- **Correlation to Sharpe:** the HEALTH_DIAG `atoms` proxy (observed, + r=−0.165, range [0,1], 6 unique values) is weakly anti-correlated. Slot + 6 itself tracks the same underlying quantity but smoothed with different + α than the Rust-side health aggregator uses, so the numerical value + differs but the semantic is identical. +- **EMA pollution risk:** LOW — input already clamped to [0,1]. +- **Verdict:** KEEP. Expose slot 6 in HEALTH_DIAG to confirm it matches + the Rust-side `atom_util` component within rounding (sanity check). + +### Slot 7 — Loss EMA + +- **Written by:** `isv_signal_update` at `experience_kernels.cu:5690` — + α=0.05 EMA of `loss_ptr[0]` (`total_loss_dev_ptr`). Source is the + zero-copy pinned scalar populated by the C51/IQN loss reduction kernel + (`launch_loss_reduce(self.total_loss_dev_ptr)` at + `gpu_dqn_trainer.rs:10411`). Genuine. +- **Read by:** ISV encoder MLP only. +- **Correlation to Sharpe:** n/a (not logged). Indirect observation: the + `grad_abs[mag]` values in HEALTH_DIAG ranged from 1.47e4 to 6.86e4 over + the window — loss magnitude has the same order of swing. Slot 7 will + reflect that. +- **EMA pollution risk:** HIGH. Same mechanism as slot 5 — loss outliers + of ±10⁴ with α=0.05 leave their fingerprint for ~20 epochs. The MLP + receives a noisy-and-pollution-carrying loss signal. +- **Verdict:** REPAIR. Apply a robust scaling: either clamp input to + e.g. `4× current EMA`, or normalise by `grad_norm EMA` (slot 1) so the + MLP input is a loss/grad ratio rather than a raw loss magnitude. A + trimmed mean on the source side (loss reduction kernel) is also + reasonable — drop top/bottom 1% per batch before reducing. + +### Slot 8 — Regime velocity (ADX EMA scratch) + +- **Written by:** `isv_signal_update` at `experience_kernels.cu:5704` + (α=0.05 EMA of `states_ptr[0 × state_dim + 40]` = sample 0's ADX) when + `states_ptr != NULL && state_dim > 41`; else `0.0f` at + `experience_kernels.cu:5718`. +- **Read by:** `experience_kernels.cu:5698` — `prev_adx = isv_signals[8]` + feeds the local `adx_delta` computation for slot 10's velocity. Feeds + ISV encoder MLP. +- **Correlation to Sharpe:** n/a (not logged). +- **EMA pollution risk:** LOW. ADX is bounded in [0,100] by construction, + α=0.05 is appropriate for a regime indicator. +- **Design flaw:** Only sample 0's ADX is used. For a batch of B=16384 this + throws away 99.994% of the regime information and makes the signal + noisy at the batch-shuffling level. The states buffer is batch-ordered + and shuffled every epoch, so sample 0 is effectively random. +- **Verdict:** REPAIR. Compute batch-mean ADX across the B rows (one + reduction kernel call, microseconds at typical B). Same change applies + structurally to the CUSUM read at slot 10. + +### Slot 9 — Regime disagreement (|ADX − CUSUM| normalised) + +- **Written by:** `isv_signal_update` at `experience_kernels.cu:5709` + (using sample 0 only, same batch-element bias as slot 8). +- **Read by:** ISV encoder MLP only. +- **Correlation to Sharpe:** n/a. +- **EMA pollution risk:** NONE — instantaneous (not an EMA). +- **Verdict:** REPAIR. Switch to batch-mean ADX / CUSUM (shared fix with + slot 8 / 10). + +### Slot 10 — Regime transition EMA (velocity EMA) + +- **Written by:** `isv_signal_update` at `experience_kernels.cu:5712` + (α=0.05 EMA of `regime_vel = adx_delta + cusum_delta`, both + sample-0-only). +- **Read by:** `experience_kernels.cu:5700` (own re-read for delta), ISV + encoder MLP. +- **Correlation to Sharpe:** n/a. +- **EMA pollution risk:** LOW (bounded input). +- **Verdict:** REPAIR (batch-mean input, shared fix). + +### Slot 11 — Regime stability `1 − sigmoid(5 × regime_vel)` + +- **Written by:** `isv_signal_update` at `experience_kernels.cu:5715`, + fallback 0.5 at `experience_kernels.cu:5721`. +- **Read by:** HEAVY consumer — this is the second-most-read ISV slot + after slot 12: + - `experience_kernels.cu:618` (regime-shift check vs ps[29]) + - `experience_kernels.cu:1750` (entry regime_stability captured into + ps[29] for drift detection) + - `experience_kernels.cu:1787` (`regime_stab` for max-hold-length + inside the adaptive catastrophic-stop net) + - `experience_kernels.cu:2137` (`regime_stab` for micro-reward + composition: `w_book = book_aggression_weight × (0.5 + 0.5×regime_stab)` + and `w_hold = hold_quality_weight × regime_stab`) + - `gpu_dqn_trainer.rs:5552` (`cql_alpha_eff = base × (1 − regime_stability) × health`) + - `gpu_dqn_trainer.rs:9150, 9209` (stability readers for exposure and + conviction-safety) + - `mamba2_temporal_kernel.cu:125,182` (Mamba2 temporal update gating) +- **Correlation to Sharpe:** n/a directly (not logged); proxies through + `cql_alpha_eff` which is stuck at 0.0000 for all 20 epochs — meaning + `(1 − regime_stability) × health ≈ 0`, implying `regime_stability ≈ 1` + throughout. Combined with the sample-0-only bias this is a strong + indicator that the sigmoid is saturating (regime_vel consistently + small or the sample happens to be stationary). +- **EMA pollution risk:** LOW — sigmoid bounds output to (0,1). +- **Verdict:** REPAIR. The slot is heavily consumed but fed from sample + 0 only — fix together with slots 8/9/10. Also: re-audit whether the + `cql_alpha = base × (1 − stab) × health` formula is correct — if + regime_stability saturates at 1 (as the observed data suggests) then + CQL is effectively always off, which is a training-stability bug in + itself. Document the stability saturation explicitly. + +### Slot 12 — Learning health (`LEARNING_HEALTH_INDEX`) + +- **Written by:** NOT `isv_signal_update`. Written host-side by + `fused.write_isv_signal_at(LEARNING_HEALTH_INDEX, health_value)` at + `training_loop.rs:1965-1968`. `health_value` comes from + `self.learning_health.update(&raw)` at `training_loop.rs:1954`, driven + by the health EMA aggregator with 7 components (q_gap, q_var, + atom_util, grad_norm, ens_disagreement, grad_consistency, spectral_gap). +- **Read by:** HEAVIEST ISV consumer in the codebase: + - `c51_loss_kernel.cu:229, 1167, 1321` — C51 label-smoothing + reward + bias `health` damping + - `c51_grad_kernel.cu:106, 152` — bin-weight derivation + - `backtest_env_kernel.cu:209, 453` — eval-mode adaptive behaviour + - `dqn_utility_kernels.cu:255` — distillation alpha + - `experience_kernels.cu:1669` — `health_k_train` in the unified env + (drives Kelly warmup scale through `target_exposure`) + - `gpu_dqn_trainer.rs:5551, 9208, 9222, 12827` — CQL alpha gate + + several exposure/safety fallbacks +- **Value trajectory:** 0.50 → 0.50 → 0.50 → 0.52 → 0.55 → 0.57 → 0.58 + → 0.57 → 0.57 → 0.58 → 0.58 → 0.60 → 0.60 → 0.61 → 0.62 → 0.62 → 0.63 + → 0.64 → 0.64 → 0.64. Monotonic rise while Sharpe goes from +34 to −67. +- **Correlation to Sharpe:** r=**−0.765** (strongly ANTI-correlated). + Range [0.50, 0.64], 10 unique values. This is the smoking gun — + `health` is SUPPOSED to rise when training is learning well, and it does + rise, but the rise coincides with a profitability collapse. +- **EMA pollution risk:** HIGH (by construction). Health monotonically + integrates q_gap=1.0 (observed stuck at 1.0 for 19/20 epochs — see + "Smoking guns" below) and q_var=1.0 (same) and spectral=0.98 (near-1 + throughout). The Rust-side EMA absorbs these near-saturated components + into a monotone climb that cannot reverse on bad epochs. +- **Verdict:** REPAIR (highest priority along with slot 2). Diagnosis: + health's components q_gap and q_var are step-functions stuck at 1.0 + for 19/20 epochs, so the aggregated health is dominated by whichever + non-saturated components exist. The component most likely to be the + actual outcome-coupling signal is `ens_agree` (stuck at 0.0 for 19/20 + — inverse behaviour) and `grad_stable` (stuck at 0.0 for 20/20 — dead + signal). Specific fixes (do NOT change the health formula itself per + audit scope non-goals — instead fix the components): + 1. `grad_stable` is permanently 0.0 — root-cause the `HealthEma` + `grad_stable` computation (training_loop.rs health_ema.update). + 2. `q_gap` / `q_var` saturate to 1.0 — these are normalized to + [0,1] via a divisor that is probably stale EMAs; investigate the + normaliser in `HealthEma`. The observed q_full / q_quarter + spread of ±10⁵ is the numerator, and the divisor has clearly + exceeded whatever floor exists — so the ratio clips to 1. + 3. `ens_disagreement` behaviour (stuck at 0 after one non-zero + epoch) hints at ensemble monitor losing state across epochs. + +### Slot 13-15 — Per-magnitude Q-mean EMAs (Quarter/Half/Full) + +- **Written by:** `isv_signal_update` at `experience_kernels.cu:5749-5756` + — α=0.05 EMA from `q_mag_means_ptr[k]`, populated by + `launch_q_mag_bin_means_reduce` at `gpu_dqn_trainer.rs:3239-3266` which + reads `q_out_buf`. +- **Read by:** + - `c51_loss_kernel.cu:365-368, 505` (reward-bias baselines for magnitude) + - `c51_grad_kernel.cu:92-94, 170` (adaptive bin weighting on magnitude) + - `gpu_dqn_trainer.rs:9169-9171` (magnitude preference exposure) +- **Value trajectory (from HEALTH_DIAG `mag q_quarter/q_half/q_full`):** + - q_full: [+118k, +259, +8.2k, +45k, +34k, +13k, +6.6k, +51k, +16k, +470, + +2.7k, +49k, +8.3k, +144, +341, +7.3k, +3.5k, +559, +744, +6.7k] + - q_half: [−80k, −189, −6.8k, −63k, −20k, −730, +95, +13k, +5.5k, +108, + +1.6k, +10k, +3.3k, +74, +197, +3.5k, +1.7k, +247, +339, +2.4k] + - q_quarter: [−88k, −200, −6.9k, −79k, −35k, −12k, −6.3k, −36k, −12k, + −275, −1.8k, −28k, −4.8k, −81, −185, −4.3k, −1.9k, −340, −425, −4.1k] +- **Correlation to Sharpe:** q_full r=+0.257 (decoupled), q_half r=−0.284 + (decoupled), q_quarter r=−0.375 (weakly anti-correlated). +- **EMA pollution risk:** VERY HIGH. Slot 13's first observation was + Q_quarter=−88k. With α=0.05, after one epoch the EMA is 0.95×−88k + + 0.05×new; a single such excursion dominates the EMA for ~20 epochs + regardless of what the policy actually learns next. This is consistent + with the user's background observation that "ep 2 Q-range=[-333k, +333k] + poisoning Kelly cap for 20 epochs". +- **Verdict:** REPAIR. Two fixes required: + 1. Clamp the source `q_mag_means_ptr[k]` input to a rolling robust + scale before feeding the EMA — e.g. `clamp(x, −4×|EMA|, +4×|EMA|)`. + Bounds outlier influence to 4× current mass without losing + directional information. + 2. Skip the EMA update when `learning_health < 0.55` or when batch + `ens_disagreement > 2σ_EMA` — effectively freezes the EMA during + unstable epochs so collapse-fraction computations in C51 kernels + see a stable historical baseline, not a corrupted one. + +### Slot 16 — Magnitude |Q|-scale reference EMA + +- **Written by:** `isv_signal_update` at `experience_kernels.cu:5758-5760` + — α=0.05 EMA of `max(q_abs_ref_ptr[0], 0)`. +- **Read by:** + - `c51_loss_kernel.cu:214, 368, 505` (normaliser for magnitude + collapse-fraction) + - `c51_grad_kernel.cu:95, 170` (bin weight denominator) + - `gpu_dqn_trainer.rs:9172` (exposure readback) +- **Correlation to Sharpe:** n/a (not logged directly; implicitly the + denominator of the `collapse_frac` and `bin_weight` formulas). +- **EMA pollution risk:** VERY HIGH. Same α=0.05 plus `fmaxf(x, 0)` means + a single Q-scale spike of 10⁵ takes ~20 epochs to normalize back down. + The divisor in `bin_weight` formulas stays artificially inflated → + `bin_weight → 1.0` → the adaptive mechanism self-disables for the rest + of the run. This is worse than if the signal were merely noisy; it is + silently permissive. +- **Verdict:** REPAIR. Add `(x ≤ 4×EMA ? x : 4×EMA)` input clipping to + prevent single-epoch spikes from driving the denominator. Alternative: + switch to a rolling max over last 64 batches (harder against outliers + than α-EMA) — but needs a scratch ring. + +### Slots 17-20 — Per-direction Q-mean EMAs (Short/Hold/Long/Flat) + +- **Written by:** `isv_signal_update` at `experience_kernels.cu:5786-5793` + — α=0.05 EMA from `q_dir_means_ptr[k]`, populated by + `launch_q_dir_bin_means_reduce` at `gpu_dqn_trainer.rs:3281-3310`. +- **Read by:** + - `c51_loss_kernel.cu:210-213, 453-456` (direction reward-bias — + `q_hold`, `q_flat`, `q_tradable`) + - `c51_grad_kernel.cu:137-140` (direction bin weighting) + - `gpu_dqn_trainer.rs:9193-9196` (direction preference exposure) +- **Correlation to Sharpe:** n/a (not logged directly); consumed through + `reward_bias` derivation which affects Bellman target. Per + `project_reward_bias_cross_branch_coupling.md`, a broken Q_dir EMA + leaks into the magnitude branch via the shared trunk. +- **EMA pollution risk:** SAME AS SLOT 13-15 — very high. Direction + Q-means scale with Q-magnitude, and the observed run had direction Q + magnitudes in the same ±10⁵ regime. +- **Verdict:** REPAIR. Same two-step fix as slots 13-15: outlier clamp + + health-gated update skip. This slot is especially load-bearing because + it feeds the `q_tradable` path in reward-bias, and a corrupted + q_tradable directly flips the `deficit = target_q − q[a0]` sign + (potentially training the network in the wrong direction for up to 20 + epochs before the EMA recovers). + +### Slot 21 — Direction |Q|-scale reference EMA (`Q_DIR_ABS_REF_INDEX`) + +- **Written by:** `isv_signal_update` at `experience_kernels.cu:5794-5796` + — α=0.05 EMA of `max(q_dir_abs_ref_ptr[0], 0)`. +- **Read by:** + - `experience_kernels.cu:1189` — CRITICAL: conviction denominator in + `action_select`-style kernel (`conviction = q_range / q_dir_abs_ref`). + Conviction then feeds the adaptive Kelly warmup floor. + - `c51_loss_kernel.cu:215, 457` (lead_scale composite for reward-bias) + - `c51_grad_kernel.cu:141` (same) + - `gpu_dqn_trainer.rs:9197` (readback) + - `trade_physics.cuh:254` documents the conviction dependency + explicitly. +- **Correlation to Sharpe:** n/a directly. INDIRECT ALARM: user + background explicitly flagged "q_dir_abs_ref EMA getting contaminated + by a single excursion poisoning Kelly cap for 20 epochs" — this slot is + the denominator of `conviction`, so a ±10⁵ excursion in direction Q + pushes conviction → ~0 for ~20 epochs → Kelly warmup floor collapses → + position sizes collapse right when the policy genuinely diverges. +- **EMA pollution risk:** VERY HIGH — highest-impact consumer (Kelly) + + worst outlier sensitivity + slowest decay. Single-epoch contamination + reaches far beyond the poison epoch. +- **Verdict:** REPAIR (priority #2 after slot 12). Same clamp/health-gate + approach as slot 16. Additionally, the fallback path at + `experience_kernels.cu:1188-1193` already does `denom = (q_dir_abs_ref + > 1e-6) ? q_dir_abs_ref : q_range` — the `1e-6` floor is too permissive; + increase it to something structural like `0.1 × (v_max − v_min)` + (matching the `lead_scale` floor already used in + `c51_loss_kernel.cu:141`). This gives the Kelly path a robust fallback + when the EMA is contaminated. + +## Summary + +| Slot | Name | Consumers (R=read, A=adaptive) | r vs Sharpe | Verdict | +|------|------|--------------------------------|-------------|---------| +| 0 | Q-drift | R: ISV MLP | unobserved | REPAIR (saturation floor) | +| 1 | grad_norm EMA | A: adaptive stop width | unobs. (proxy r=+0.69) | REPAIR (outlier clamp) | +| 2 | **TD-error EMA** | A: micro-reward `volatility`, IQN lagged target | **NONE (stuck 0)** | **REPAIR (no writer)** | +| 3 | **Ensemble var EMA** | R: ISV MLP | **NONE (stuck 0)** | **REPAIR (no writer)** | +| 4 | Ens var velocity | R: ISV MLP | NONE (stuck 0) | REPAIR (derivative of #3) | +| 5 | "Reward" proxy EMA | R: ISV MLP | unobs. | REPAIR (mislabel; use real reward) | +| 6 | Atom util EMA | R: ISV MLP | r=−0.17 | KEEP (expose in logs) | +| 7 | Loss EMA | R: ISV MLP | unobs. | REPAIR (outlier clamp or ratio) | +| 8 | Regime velocity | R: slot 10 chain | unobs. | REPAIR (sample-0 bias) | +| 9 | Regime disagreement | R: ISV MLP | unobs. | REPAIR (sample-0 bias) | +| 10 | Regime transition EMA | R: slot 8/self | unobs. | REPAIR (sample-0 bias) | +| 11 | Regime stability | A: 7 heavy consumers incl. CQL alpha, hold width, micro-reward | implied saturated at 1 | REPAIR (sample-0 bias) | +| 12 | **Learning health** | A: ~10 consumers (C51, CQL, env, distill, Kelly) | **r=−0.765 (ANTI)** | **REPAIR (fix components)** | +| 13-15 | Mag Q-mean EMAs | A: C51 mag bin weights, reward-bias | direct r≈±0.3 | REPAIR (outlier clamp + health-gate) | +| 16 | Mag |Q|-scale ref | A: C51 bin-weight denominator | unobs. | REPAIR (outlier clamp) | +| 17-20 | Dir Q-mean EMAs | A: C51 dir reward-bias, bin weights | indirect | REPAIR (outlier clamp + health-gate) | +| 21 | **Dir |Q|-scale ref** | A: **Kelly conviction denominator** + C51 bias | **unobs. but high-impact** | **REPAIR (clamp + raise fallback floor)** | + +Bolded rows are the highest-severity failures: dead writers (slots 2, 3, 4), +the anti-correlated health signal (slot 12), and the Kelly-conviction +contamination risk (slot 21). + +Also documented in HEALTH_DIAG but not formally part of the ISV bus — all +stuck-signal diagnostics: + +| Signal (not an ISV slot) | Stuck value | Epochs | +|---|---|---| +| `q_gap` (HealthEma component) | 1.00 | 19/20 | +| `q_var` (HealthEma component) | 1.00 | 19/20 | +| `grad_stable` (HealthEma component) | 0.00 | **20/20** | +| `ens_agree` (HealthEma component) | 0.00 | 19/20 | +| `ensemble_collapse` (novel detector) | 0.00 | 19/20 | + +## Proposed fixes (priority order) + +1. **Wire `td_error_scratch_pinned` (slot 2) from the loss kernel.** The + pinned scratch is already allocated, the kernel ABI already accepts it, + and per-sample TD errors are already computed (`td_errors_buf` at + `gpu_dqn_trainer.rs:6299`). Add one reduction kernel call to write + `mean(|td_error|)` into the scratch before `update_isv_signals`. + Downstream payoff: the micro-reward volatility weight + (`experience_kernels.cu:2138`) and the IQN recursive confidence target + (`experience_kernels.cu:5655`) both stop reading zero. +2. **Fix `learning_health` component saturation (slot 12).** Three + sub-issues, each independently fixable in `trainers/dqn/trainer/*`: + a. Root-cause `grad_stable` = 0.0 permanently — inspect + `HealthEma::update_grad_consistency_vec` path (invoked at + `training_loop.rs:1938`) and verify its divisor/floor. + b. q_gap / q_var saturating at 1.0 — their normaliser in + `HealthEma::update` is probably using an EMA divisor that lags the + ±10⁵ Q-scale explosion. Replace with a robust scale (median of + recent Q-range, or use slot 16 / slot 21 as normalizers). + c. `ens_agree` decay — after initial transient it sticks at 0, which + suggests the Rust-side monitor stops receiving updates or the + normaliser zeros out. + Health's bad composition is the single most damaging ISV failure — its + r=−0.765 means every consumer (C51 bin weights, reward-bias damping, + CQL gate, Kelly safety, distillation alpha) is being steered in the + wrong direction. +3. **Outlier clamp on EMA inputs for slots 13-21.** Pre-clamp + `q_mag_means_ptr[k]`, `q_abs_ref_ptr[0]`, `q_dir_means_ptr[k]`, + `q_dir_abs_ref_ptr[0]` to `[−4×prev_ema, +4×prev_ema]` (or 4× for the + unsigned |Q|-refs) before the α-blend in `isv_signal_update`. This is + a 4-line patch inside that kernel and bounds a single bad epoch's + pollution to ≤5× its own magnitude rather than ~20 epochs of leakage. +4. **Wire `ensemble_var_scratch_pinned` (slot 3).** The ensemble monitor + already computes per-head Q spread; route the per-epoch value into + the pinned scratch the same way slot 6 (atom_util) is wired at + `gpu_dqn_trainer.rs:11078`. Slot 4 (velocity) becomes meaningful + automatically. +5. **Fix the sample-0 bias in slots 8-11.** Replace the `states_ptr[0 × + state_dim + 40]` read with a one-thread batch-mean reduction over all B + rows. The regime signals then reflect batch regime rather than whichever + single row the shuffle happened to place first. +6. **Raise the conviction fallback floor at `experience_kernels.cu:1191`.** + Change the epsilon floor from `1e-6` to `0.1 × (v_max − v_min)` — + matches the architectural floor already used by the `lead_scale` reward- + bias computation (`c51_loss_kernel.cu:141`) and gives the Kelly path a + meaningful denominator when slot 21 is polluted. +7. **Relabel or repurpose slot 5.** It is reading `q_mean` but named + `reward_scratch`; downstream MLP interpretation is wrong. Either feed + the actual mean step-return (from `train_physics_step_buf`) or drop the + slot and reuse it for a signal the ISV genuinely lacks (e.g. realised + position P&L velocity, or win-rate EMA). +8. **Add HEALTH_DIAG exposure for slots 0-7 and 13-21.** The diagnostic + only shows slot 12 + derived mag_Q values directly. The remaining + slots are invisible at runtime — making future regressions hard to + detect. Adding them is a few lines in `training_loop.rs` where + HEALTH_DIAG is emitted. + +## Hypothesis on the oscillation + +The +34 → −67 Sharpe collapse is most plausibly caused by three ISV signals +simultaneously giving mis-information during epochs 5-10: + +**Primary driver — slot 12 (`learning_health`):** r=−0.765 against Sharpe. +This signal directly damps reward-bias (`c51_loss_kernel.cu:229`: `reward_bias +× (1 − health)`), modulates CQL alpha (`gpu_dqn_trainer.rs:5555`: `cql_alpha +× (1 − regime_stab) × health`), and sets the distillation alpha and Kelly +warmup floor. Its climb from 0.50 to 0.64 over the observed window +progressively: +- DAMPS the reward-bias correction (the mechanism designed to correct the + exact collapse regime being observed) — at health=0.5 the bias is fully + applied; at health=0.64 it's damped 28% more. +- Keeps `cql_alpha ≈ 0` regardless of training quality (multiplying by an + apparently-saturated `regime_stab` that makes `(1 − stab) ≈ 0`). +- Inflates `target_exposure` through the Kelly path — training progressively + ramps position size exposure as health rises, which is exactly wrong when + the policy is collapsing. + +**Secondary driver — slot 21 (`q_dir_abs_ref_ema`):** user background flagged +contamination from a single Q-excursion. In the observed data, direction +Q-scales swung through ±10⁵ at epochs 0, 3, 7, 11 — each one re-poisons +slot 21 for the next ~20 epochs. This pushes conviction → ~0 (denominator +saturation), which triggers the Kelly warmup floor collapse path. Positions +shrink, but gradient updates continue training against the contaminated +Q-values — a classic positive-feedback loop. + +**Tertiary driver — slot 2 (`td_error EMA`) being identically 0:** the +micro-reward composition kernel at `experience_kernels.cu:2138` reads +`volatility = isv_signals[2]` and uses it to scale `w_price` and +indirectly `w_book`. Reading 0 means `w_price = 0.5 × weight` and `w_book += 0.5 × weight` regardless of actual TD-error regime — the micro-reward +becomes regime-unaware exactly when the regime shifts. This is not the +largest quantitative driver but it removes the last line of defense +against the primary mechanism failures above. + +Together: health mis-reports "improving" when the policy is diverging, +q_dir_abs_ref gets polluted then stays polluted, and the micro-reward +cannot respond to the divergence because its regime signal is dead. The +adaptive mechanisms designed to self-correct such collapses are all +driven by these three signals, so the corrective pressure is exactly +inverted relative to actual outcome. The Sharpe collapse is not caused +by a broken policy — it is caused by adaptive-mechanism mis-steering +driven by broken ISV measurement. + +The specific epoch where Sharpe tips (epoch 9: +26.3 → −8.4 → −37.2 → +−42.7) is consistent with the outlier at epoch 7 (q_full=51k) finally +contaminating both Q-scale EMAs past recovery point while health keeps +rising due to its saturated components. This is a deterministic +signature: fix slots 2, 3, 12, and 21 and the oscillation should not +reappear.