docs(isv): signal quality audit for DQN ISV bus — L40S oscillation diagnosis
Diagnostic-only audit of all 22 ISV slots using train-mdh86 HEALTH_DIAG (20 epochs, same oscillation pattern as train-rq6n8). Identifies three dead writers (slots 2, 3, 4), one strongly anti-correlated signal (slot 12 learning_health, r=-0.765 vs Sharpe), sample-0 bias in four regime slots (8-11), and EMA-pollution risk on the Q-scale EMAs (16, 21) that feed Kelly conviction + C51 bin weighting. Produces a per-slot table with writer/reader citations and an ordered fix list. Hypothesises the observed +34 → -67 Sharpe swing is driven by health mis-reporting "improving" while policy diverges, q_dir_abs_ref contamination collapsing Kelly, and the zero-writer TD-error EMA disabling micro-reward regime awareness. No code changes — reconnaissance only.
This commit is contained in:
jgrusewski
609
docs/superpowers/specs/2026-04-23-isv-signal-quality-audit.md
Normal file
609
docs/superpowers/specs/2026-04-23-isv-signal-quality-audit.md
Normal file
@@ -0,0 +1,609 @@
|
||||
# ISV Signal Quality Audit (train-rq6n8 oscillation diagnosis)
|
||||
|
||||
## Background
|
||||
|
||||
The Introspective State Vector (ISV) is a 22-slot pinned device-mapped signal
|
||||
bus that feeds most of Foxhunt DQN's adaptive mechanisms: the `cql_alpha`
|
||||
regime gate, the conviction-driven Kelly warmup floor, reward-bias on the
|
||||
Bellman target, Task 2.X / Task 2.Y C51 bin weighting, the ISV encoder MLP
|
||||
that produces `branch_gate` + `gamma_mod`, the adaptive `isv_instability`
|
||||
safety net inside the unified env kernel, and the distillation alpha ramp.
|
||||
|
||||
The audit was triggered by the terminated L40S run `train-rq6n8`. The logs
|
||||
observed at `/tmp/l40s_diag/health.log` (20 `HEALTH_DIAG` lines from the
|
||||
earlier `train-mdh86` pod, same oscillation pattern) show a Sharpe trajectory
|
||||
that rises to +34 over epochs 1..5, falls back to +28 at epoch 7, crashes
|
||||
through zero into −37 by epoch 10, and keeps sliding to −67 by epoch 19 —
|
||||
while `learning_health` climbs monotonically from 0.50 to 0.64. That is a
|
||||
direct contradiction: a "learning-health" signal climbing while cumulative
|
||||
Sharpe collapses cannot be a faithful measurement of training outcomes.
|
||||
|
||||
This audit enumerates every ISV slot, identifies its writer and consumers,
|
||||
grades the signal against Sharpe correlation over the 20-epoch window, and
|
||||
flags slots whose failure is most likely driving adaptive-mechanism
|
||||
mis-decisions during the observed oscillation.
|
||||
|
||||
Relevant memory for context: `feedback_adaptive_not_tuned.md` (signals must
|
||||
drive adaptive mechanisms, not hard-coded constants) and
|
||||
`project_reward_bias_cross_branch_coupling.md` (direction-branch reward-bias
|
||||
couples to magnitude via the shared trunk — one broken signal can poison both
|
||||
branches).
|
||||
|
||||
### Data
|
||||
|
||||
Logs parsed: `/tmp/l40s_diag/health.log` — 20 HEALTH_DIAG epochs
|
||||
(`train-mdh86`; `train-rq6n8` argo pod logs were no longer available when
|
||||
this audit ran). Sharpe trajectory observed:
|
||||
|
||||
```
|
||||
[+0.0, +20.5, +33.8, +33.9, +34.1, +34.6, +33.4, +28.6, +26.3, -8.4,
|
||||
-37.2, -42.7, -38.2, -5.2, -12.7, -39.8, -52.5, -58.8, -64.9, -67.2]
|
||||
```
|
||||
|
||||
Pearson r reported below is between each signal's epoch-indexed trajectory
|
||||
and the Sharpe trajectory across N=20 samples. Because Sharpe is itself an
|
||||
EMA the correlation is a lower bound on the underlying causal coupling, but
|
||||
for the purpose of flagging stuck or decoupled signals it is adequate.
|
||||
|
||||
## Per-slot analysis
|
||||
|
||||
Naming convention used throughout: `experience_kernels.cu:5667` means line
|
||||
5667 of that file.
|
||||
|
||||
### Slot 0 — Q-drift (normalised deviation from Q-mean EMA)
|
||||
|
||||
- **Written by:** `isv_signal_update` kernel at `experience_kernels.cu:5667`
|
||||
— `(q_mean - q_ema) / max(|q_ema|, 0.1)`. Inputs:
|
||||
`q_mean_scratch_pinned` (populated by `reduce_current_q_stats` writeback
|
||||
`host[3]` at `gpu_dqn_trainer.rs:11080` — same q_mean as
|
||||
`QValueStatsResult.q_mean`) and `q_mean_ema_pinned` (updated in
|
||||
`update_q_mean_ema` at `gpu_dqn_trainer.rs:3214`).
|
||||
- **Read by:** no kernel or Rust call site reads slot 0 directly. Feeds only
|
||||
the ISV encoder MLP (`isv_forward` at `experience_kernels.cu:5827` reads
|
||||
`isv_signals[k]` for k∈[0..12) as MLP input).
|
||||
- **Value trajectory:** not logged in HEALTH_DIAG; cannot measure directly.
|
||||
- **Correlation to Sharpe:** n/a (not observed). Structurally couples to
|
||||
`q_mean` which drifted across 5 orders of magnitude (Q=+117k → 7k) in the
|
||||
observed window — slot 0's numerator can swing by ±O(1) per epoch.
|
||||
- **EMA pollution risk:** MEDIUM. Slot 0 is NOT an EMA (instantaneous ratio)
|
||||
— outliers do not persist across epochs. The denominator floor of 0.1
|
||||
saturates the ratio for small `q_ema`, which silently converts structure
|
||||
into near-zero noise whenever Q-mean EMA itself is small.
|
||||
- **Verdict:** REPAIR. Saturation floor `max(|q_ema|, 0.1)` is arbitrary and
|
||||
hides the signal when it matters most (early / collapsed Q). Replace with
|
||||
a robust scale reference — `q_abs_ref_ema` (slot 16) or the C51 support
|
||||
`v_max − v_min`. Also add to HEALTH_DIAG so its trajectory can be
|
||||
inspected.
|
||||
|
||||
### Slot 1 — Gradient norm EMA (√grad_norm²)
|
||||
|
||||
- **Written by:** `isv_signal_update` at `experience_kernels.cu:5671` —
|
||||
α=0.05 EMA of `sqrt(grad_norm_ptr[0])`. Input is `grad_norm_dev_ptr`
|
||||
populated by Adam / gradient reducer kernels (same scalar that feeds
|
||||
`grad_norm_pinned` read at `gpu_dqn_trainer.rs:9902`).
|
||||
- **Read by:** `experience_kernels.cu:1786` — `isv_instability = isv_signals_ptr[1]`
|
||||
drives the ISV-adaptive catastrophic-stop width inside the unified env
|
||||
kernel (`eff_stop = base_stop / (1 + isv_instability)`). Also feeds the
|
||||
ISV encoder MLP.
|
||||
- **Correlation to Sharpe:** n/a (not in HEALTH_DIAG); indirect via
|
||||
`grad_ratio_mag_dir` (r=+0.691) which is driven by the same branch-grad
|
||||
norms — so the underlying scale IS tracking something real.
|
||||
- **EMA pollution risk:** MEDIUM. grad_norm² can spike by 10⁴ on one bad
|
||||
batch (Q-range ±333k observed at epoch 2 per background). The √
|
||||
compresses, but α=0.05 still means a single outlier decays over ~20
|
||||
epochs. Every adaptive stop width sees that decayed outlier.
|
||||
- **Verdict:** REPAIR. Log slot 1 in HEALTH_DIAG. Add a trimmed-EMA or
|
||||
robust scale (e.g. median of last 32 batches, or clamp input to 4× EMA
|
||||
before updating). The consumer site at `experience_kernels.cu:1786`
|
||||
already divides by `(1 + isv_instability)` which naturally rate-limits
|
||||
the response — but the EMA itself still carries the outlier.
|
||||
|
||||
### Slot 2 — TD-error EMA
|
||||
|
||||
- **Written by:** `isv_signal_update` at `experience_kernels.cu:5674` —
|
||||
α=0.05 EMA of `td_error_ptr[0]` (`td_error_scratch_dev_ptr`). **Source
|
||||
buffer has NO writer.** `td_error_scratch_pinned` is allocated and
|
||||
zero-initialised at `gpu_dqn_trainer.rs:7605-7623`; no kernel or Rust
|
||||
call site ever writes to it again. Confirmed by `rg
|
||||
'td_error_scratch_pinned\s*=' crates/ml/src` returning only the
|
||||
allocation site and the `free_host` in Drop. The code comment at
|
||||
`gpu_dqn_trainer.rs:1771-1777` acknowledges this explicitly.
|
||||
- **Read by:** `experience_kernels.cu:2138` — `volatility = isv_signals_ptr[2]`
|
||||
feeds the micro-reward composition (`w_price = price_confirm_weight × (0.5 + 0.5×volatility)`)
|
||||
and **reads it as if it were a volatility proxy**. Also feeds the ISV
|
||||
encoder MLP. Also `lagged_td_error[0] = isv_signals[2]` at
|
||||
`experience_kernels.cu:5655` — the lagged scratch that IQN's recursive
|
||||
confidence target reads.
|
||||
- **Correlation to Sharpe:** n/a (value is identically 0.0 after α-blending
|
||||
a zero source).
|
||||
- **EMA pollution risk:** NONE — the signal is zero.
|
||||
- **Verdict:** REPAIR (highest priority). Wire `td_error_scratch_pinned`
|
||||
from the C51 / IQN loss kernels (they already compute per-sample TD
|
||||
errors at `per_sample_loss_buf` and `td_errors_buf` allocated at
|
||||
`gpu_dqn_trainer.rs:6298-6299`). Add a 1-thread reduction kernel that
|
||||
writes `mean(|td_error|)` into `td_error_scratch_pinned`, launched on
|
||||
the same stream before `update_isv_signals`. The consumer at
|
||||
`experience_kernels.cu:2138` is silently getting 0.5 (wrong, because the
|
||||
read path is `isv_signals_ptr[2]` without the NULL fallback — it reads
|
||||
the zero directly, not the 0.5 default).
|
||||
|
||||
### Slot 3 — Ensemble variance EMA
|
||||
|
||||
- **Written by:** `isv_signal_update` at `experience_kernels.cu:5678` —
|
||||
α=0.05 EMA of `ens_var_ptr[0]` (`ensemble_var_scratch_dev_ptr`). **Source
|
||||
buffer has NO writer.** Allocated and zeroed at
|
||||
`gpu_dqn_trainer.rs:7625-7643`; never written again. Confirmed via
|
||||
ripgrep. Code comment at `gpu_dqn_trainer.rs:1780-1786` acknowledges
|
||||
this.
|
||||
- **Read by:** only the ISV encoder MLP (`isv_forward`). The
|
||||
HEALTH_DIAG `ens_agree` value (stuck at 0.0 for 19/20 epochs) is
|
||||
computed from `EnsembleMonitor` state in Rust, not from ISV slot 3 —
|
||||
different path.
|
||||
- **Correlation to Sharpe:** n/a (identically 0.0).
|
||||
- **EMA pollution risk:** NONE — signal is zero.
|
||||
- **Verdict:** REPAIR. Wire `ensemble_var_scratch_pinned` from the
|
||||
ensemble-head reducer. The project already has `EnsembleMonitor`
|
||||
collecting per-head Q disagreement (see
|
||||
`session_2026-04-14_h100_gradient_fixes.md` mention of "ensemble
|
||||
disagreement"), so the signal exists — it just is not propagated into
|
||||
ISV. Route it through a CPU scratch write analogous to
|
||||
`*self.atom_util_scratch_pinned = host[6]` at `gpu_dqn_trainer.rs:11078`,
|
||||
using the per-epoch ensemble spread the monitor already computes.
|
||||
|
||||
### Slot 4 — Ensemble variance velocity
|
||||
|
||||
- **Written by:** `isv_signal_update` at `experience_kernels.cu:5681` —
|
||||
`(isv_signals[3] - old_ens_ema) / max(old_ens_ema, 1e-6)`. Derived
|
||||
entirely from slot 3, so inherits its zero-writer bug plus a
|
||||
divide-by-zero singularity: when slot 3 is 0, `old_ens_ema` is 0 →
|
||||
denominator saturates to `1e-6` → velocity is `0/1e-6 = 0`. Signal is
|
||||
identically 0.0.
|
||||
- **Read by:** ISV encoder MLP only.
|
||||
- **Correlation to Sharpe:** n/a.
|
||||
- **EMA pollution risk:** NONE — signal is zero; on the day slot 3 gets
|
||||
wired, velocity will be well-defined (`1e-6` floor still bounds the
|
||||
quotient).
|
||||
- **Verdict:** REPAIR (derivative of slot 3 fix). Becomes meaningful the
|
||||
moment slot 3's writer is wired.
|
||||
|
||||
### Slot 5 — Reward proxy EMA
|
||||
|
||||
- **Written by:** `isv_signal_update` at `experience_kernels.cu:5684` —
|
||||
α=0.05 EMA of `reward_ptr[0]` (`reward_scratch_dev_ptr`). Source buffer
|
||||
is populated at `gpu_dqn_trainer.rs:11080` — `*self.reward_scratch_pinned
|
||||
= host[3]` where `host[3]` is `q_mean` from `QValueStatsResult`. **This
|
||||
is mis-labelled as "reward" — it is actually the batch Q-mean.** Same
|
||||
value as feeds slot 0's numerator.
|
||||
- **Read by:** ISV encoder MLP only.
|
||||
- **Correlation to Sharpe:** n/a (not logged); structurally equivalent to
|
||||
a smoothed Q-mean. Given observed Q-mean magnitudes swinging ±O(10⁵),
|
||||
slot 5 is dominated by Q-mean scale rather than reward semantics.
|
||||
- **EMA pollution risk:** HIGH. A single Q-mean excursion of ±10⁵ takes
|
||||
~20 epochs to decay at α=0.05. The MLP input range for a "reward proxy"
|
||||
becomes catastrophically mis-normalised for the rest of the run.
|
||||
- **Verdict:** REPAIR. Either (a) wire the actual per-epoch reward (mean
|
||||
`step_return` computed by the env kernel in `train_physics_step_buf`),
|
||||
or (b) relabel the consumer path to "Q-mean EMA" and delete the
|
||||
duplicate with slot 0. Option (a) is strictly better for signal quality
|
||||
because Q-mean and realised reward diverge exactly during the regimes
|
||||
the signal is supposed to detect (over-confident Q, negative realised
|
||||
P&L — cf. epochs 10+ in the observed data where Q-scales stayed high
|
||||
while Sharpe cratered).
|
||||
|
||||
### Slot 6 — Atom utilization EMA
|
||||
|
||||
- **Written by:** `isv_signal_update` at `experience_kernels.cu:5687` —
|
||||
α=0.05 EMA of `atom_util_ptr[0]` (`atom_util_scratch_dev_ptr`). Source
|
||||
populated at `gpu_dqn_trainer.rs:11078` — `host[6].clamp(0,1)`, i.e.
|
||||
`QValueStatsResult.atom_utilization`. Genuine signal.
|
||||
- **Read by:** ISV encoder MLP only. HEALTH_DIAG `atoms` component is
|
||||
computed separately by the health EMA aggregator, not from ISV slot 6.
|
||||
- **Correlation to Sharpe:** the HEALTH_DIAG `atoms` proxy (observed,
|
||||
r=−0.165, range [0,1], 6 unique values) is weakly anti-correlated. Slot
|
||||
6 itself tracks the same underlying quantity but smoothed with different
|
||||
α than the Rust-side health aggregator uses, so the numerical value
|
||||
differs but the semantic is identical.
|
||||
- **EMA pollution risk:** LOW — input already clamped to [0,1].
|
||||
- **Verdict:** KEEP. Expose slot 6 in HEALTH_DIAG to confirm it matches
|
||||
the Rust-side `atom_util` component within rounding (sanity check).
|
||||
|
||||
### Slot 7 — Loss EMA
|
||||
|
||||
- **Written by:** `isv_signal_update` at `experience_kernels.cu:5690` —
|
||||
α=0.05 EMA of `loss_ptr[0]` (`total_loss_dev_ptr`). Source is the
|
||||
zero-copy pinned scalar populated by the C51/IQN loss reduction kernel
|
||||
(`launch_loss_reduce(self.total_loss_dev_ptr)` at
|
||||
`gpu_dqn_trainer.rs:10411`). Genuine.
|
||||
- **Read by:** ISV encoder MLP only.
|
||||
- **Correlation to Sharpe:** n/a (not logged). Indirect observation: the
|
||||
`grad_abs[mag]` values in HEALTH_DIAG ranged from 1.47e4 to 6.86e4 over
|
||||
the window — loss magnitude has the same order of swing. Slot 7 will
|
||||
reflect that.
|
||||
- **EMA pollution risk:** HIGH. Same mechanism as slot 5 — loss outliers
|
||||
of ±10⁴ with α=0.05 leave their fingerprint for ~20 epochs. The MLP
|
||||
receives a noisy-and-pollution-carrying loss signal.
|
||||
- **Verdict:** REPAIR. Apply a robust scaling: either clamp input to
|
||||
e.g. `4× current EMA`, or normalise by `grad_norm EMA` (slot 1) so the
|
||||
MLP input is a loss/grad ratio rather than a raw loss magnitude. A
|
||||
trimmed mean on the source side (loss reduction kernel) is also
|
||||
reasonable — drop top/bottom 1% per batch before reducing.
|
||||
|
||||
### Slot 8 — Regime velocity (ADX EMA scratch)
|
||||
|
||||
- **Written by:** `isv_signal_update` at `experience_kernels.cu:5704`
|
||||
(α=0.05 EMA of `states_ptr[0 × state_dim + 40]` = sample 0's ADX) when
|
||||
`states_ptr != NULL && state_dim > 41`; else `0.0f` at
|
||||
`experience_kernels.cu:5718`.
|
||||
- **Read by:** `experience_kernels.cu:5698` — `prev_adx = isv_signals[8]`
|
||||
feeds the local `adx_delta` computation for slot 10's velocity. Feeds
|
||||
ISV encoder MLP.
|
||||
- **Correlation to Sharpe:** n/a (not logged).
|
||||
- **EMA pollution risk:** LOW. ADX is bounded in [0,100] by construction,
|
||||
α=0.05 is appropriate for a regime indicator.
|
||||
- **Design flaw:** Only sample 0's ADX is used. For a batch of B=16384 this
|
||||
throws away 99.994% of the regime information and makes the signal
|
||||
noisy at the batch-shuffling level. The states buffer is batch-ordered
|
||||
and shuffled every epoch, so sample 0 is effectively random.
|
||||
- **Verdict:** REPAIR. Compute batch-mean ADX across the B rows (one
|
||||
reduction kernel call, microseconds at typical B). Same change applies
|
||||
structurally to the CUSUM read at slot 10.
|
||||
|
||||
### Slot 9 — Regime disagreement (|ADX − CUSUM| normalised)
|
||||
|
||||
- **Written by:** `isv_signal_update` at `experience_kernels.cu:5709`
|
||||
(using sample 0 only, same batch-element bias as slot 8).
|
||||
- **Read by:** ISV encoder MLP only.
|
||||
- **Correlation to Sharpe:** n/a.
|
||||
- **EMA pollution risk:** NONE — instantaneous (not an EMA).
|
||||
- **Verdict:** REPAIR. Switch to batch-mean ADX / CUSUM (shared fix with
|
||||
slot 8 / 10).
|
||||
|
||||
### Slot 10 — Regime transition EMA (velocity EMA)
|
||||
|
||||
- **Written by:** `isv_signal_update` at `experience_kernels.cu:5712`
|
||||
(α=0.05 EMA of `regime_vel = adx_delta + cusum_delta`, both
|
||||
sample-0-only).
|
||||
- **Read by:** `experience_kernels.cu:5700` (own re-read for delta), ISV
|
||||
encoder MLP.
|
||||
- **Correlation to Sharpe:** n/a.
|
||||
- **EMA pollution risk:** LOW (bounded input).
|
||||
- **Verdict:** REPAIR (batch-mean input, shared fix).
|
||||
|
||||
### Slot 11 — Regime stability `1 − sigmoid(5 × regime_vel)`
|
||||
|
||||
- **Written by:** `isv_signal_update` at `experience_kernels.cu:5715`,
|
||||
fallback 0.5 at `experience_kernels.cu:5721`.
|
||||
- **Read by:** HEAVY consumer — this is the second-most-read ISV slot
|
||||
after slot 12:
|
||||
- `experience_kernels.cu:618` (regime-shift check vs ps[29])
|
||||
- `experience_kernels.cu:1750` (entry regime_stability captured into
|
||||
ps[29] for drift detection)
|
||||
- `experience_kernels.cu:1787` (`regime_stab` for max-hold-length
|
||||
inside the adaptive catastrophic-stop net)
|
||||
- `experience_kernels.cu:2137` (`regime_stab` for micro-reward
|
||||
composition: `w_book = book_aggression_weight × (0.5 + 0.5×regime_stab)`
|
||||
and `w_hold = hold_quality_weight × regime_stab`)
|
||||
- `gpu_dqn_trainer.rs:5552` (`cql_alpha_eff = base × (1 − regime_stability) × health`)
|
||||
- `gpu_dqn_trainer.rs:9150, 9209` (stability readers for exposure and
|
||||
conviction-safety)
|
||||
- `mamba2_temporal_kernel.cu:125,182` (Mamba2 temporal update gating)
|
||||
- **Correlation to Sharpe:** n/a directly (not logged); proxies through
|
||||
`cql_alpha_eff` which is stuck at 0.0000 for all 20 epochs — meaning
|
||||
`(1 − regime_stability) × health ≈ 0`, implying `regime_stability ≈ 1`
|
||||
throughout. Combined with the sample-0-only bias this is a strong
|
||||
indicator that the sigmoid is saturating (regime_vel consistently
|
||||
small or the sample happens to be stationary).
|
||||
- **EMA pollution risk:** LOW — sigmoid bounds output to (0,1).
|
||||
- **Verdict:** REPAIR. The slot is heavily consumed but fed from sample
|
||||
0 only — fix together with slots 8/9/10. Also: re-audit whether the
|
||||
`cql_alpha = base × (1 − stab) × health` formula is correct — if
|
||||
regime_stability saturates at 1 (as the observed data suggests) then
|
||||
CQL is effectively always off, which is a training-stability bug in
|
||||
itself. Document the stability saturation explicitly.
|
||||
|
||||
### Slot 12 — Learning health (`LEARNING_HEALTH_INDEX`)
|
||||
|
||||
- **Written by:** NOT `isv_signal_update`. Written host-side by
|
||||
`fused.write_isv_signal_at(LEARNING_HEALTH_INDEX, health_value)` at
|
||||
`training_loop.rs:1965-1968`. `health_value` comes from
|
||||
`self.learning_health.update(&raw)` at `training_loop.rs:1954`, driven
|
||||
by the health EMA aggregator with 7 components (q_gap, q_var,
|
||||
atom_util, grad_norm, ens_disagreement, grad_consistency, spectral_gap).
|
||||
- **Read by:** HEAVIEST ISV consumer in the codebase:
|
||||
- `c51_loss_kernel.cu:229, 1167, 1321` — C51 label-smoothing + reward
|
||||
bias `health` damping
|
||||
- `c51_grad_kernel.cu:106, 152` — bin-weight derivation
|
||||
- `backtest_env_kernel.cu:209, 453` — eval-mode adaptive behaviour
|
||||
- `dqn_utility_kernels.cu:255` — distillation alpha
|
||||
- `experience_kernels.cu:1669` — `health_k_train` in the unified env
|
||||
(drives Kelly warmup scale through `target_exposure`)
|
||||
- `gpu_dqn_trainer.rs:5551, 9208, 9222, 12827` — CQL alpha gate +
|
||||
several exposure/safety fallbacks
|
||||
- **Value trajectory:** 0.50 → 0.50 → 0.50 → 0.52 → 0.55 → 0.57 → 0.58
|
||||
→ 0.57 → 0.57 → 0.58 → 0.58 → 0.60 → 0.60 → 0.61 → 0.62 → 0.62 → 0.63
|
||||
→ 0.64 → 0.64 → 0.64. Monotonic rise while Sharpe goes from +34 to −67.
|
||||
- **Correlation to Sharpe:** r=**−0.765** (strongly ANTI-correlated).
|
||||
Range [0.50, 0.64], 10 unique values. This is the smoking gun —
|
||||
`health` is SUPPOSED to rise when training is learning well, and it does
|
||||
rise, but the rise coincides with a profitability collapse.
|
||||
- **EMA pollution risk:** HIGH (by construction). Health monotonically
|
||||
integrates q_gap=1.0 (observed stuck at 1.0 for 19/20 epochs — see
|
||||
"Smoking guns" below) and q_var=1.0 (same) and spectral=0.98 (near-1
|
||||
throughout). The Rust-side EMA absorbs these near-saturated components
|
||||
into a monotone climb that cannot reverse on bad epochs.
|
||||
- **Verdict:** REPAIR (highest priority along with slot 2). Diagnosis:
|
||||
health's components q_gap and q_var are step-functions stuck at 1.0
|
||||
for 19/20 epochs, so the aggregated health is dominated by whichever
|
||||
non-saturated components exist. The component most likely to be the
|
||||
actual outcome-coupling signal is `ens_agree` (stuck at 0.0 for 19/20
|
||||
— inverse behaviour) and `grad_stable` (stuck at 0.0 for 20/20 — dead
|
||||
signal). Specific fixes (do NOT change the health formula itself per
|
||||
audit scope non-goals — instead fix the components):
|
||||
1. `grad_stable` is permanently 0.0 — root-cause the `HealthEma`
|
||||
`grad_stable` computation (training_loop.rs health_ema.update).
|
||||
2. `q_gap` / `q_var` saturate to 1.0 — these are normalized to
|
||||
[0,1] via a divisor that is probably stale EMAs; investigate the
|
||||
normaliser in `HealthEma`. The observed q_full / q_quarter
|
||||
spread of ±10⁵ is the numerator, and the divisor has clearly
|
||||
exceeded whatever floor exists — so the ratio clips to 1.
|
||||
3. `ens_disagreement` behaviour (stuck at 0 after one non-zero
|
||||
epoch) hints at ensemble monitor losing state across epochs.
|
||||
|
||||
### Slot 13-15 — Per-magnitude Q-mean EMAs (Quarter/Half/Full)
|
||||
|
||||
- **Written by:** `isv_signal_update` at `experience_kernels.cu:5749-5756`
|
||||
— α=0.05 EMA from `q_mag_means_ptr[k]`, populated by
|
||||
`launch_q_mag_bin_means_reduce` at `gpu_dqn_trainer.rs:3239-3266` which
|
||||
reads `q_out_buf`.
|
||||
- **Read by:**
|
||||
- `c51_loss_kernel.cu:365-368, 505` (reward-bias baselines for magnitude)
|
||||
- `c51_grad_kernel.cu:92-94, 170` (adaptive bin weighting on magnitude)
|
||||
- `gpu_dqn_trainer.rs:9169-9171` (magnitude preference exposure)
|
||||
- **Value trajectory (from HEALTH_DIAG `mag q_quarter/q_half/q_full`):**
|
||||
- q_full: [+118k, +259, +8.2k, +45k, +34k, +13k, +6.6k, +51k, +16k, +470,
|
||||
+2.7k, +49k, +8.3k, +144, +341, +7.3k, +3.5k, +559, +744, +6.7k]
|
||||
- q_half: [−80k, −189, −6.8k, −63k, −20k, −730, +95, +13k, +5.5k, +108,
|
||||
+1.6k, +10k, +3.3k, +74, +197, +3.5k, +1.7k, +247, +339, +2.4k]
|
||||
- q_quarter: [−88k, −200, −6.9k, −79k, −35k, −12k, −6.3k, −36k, −12k,
|
||||
−275, −1.8k, −28k, −4.8k, −81, −185, −4.3k, −1.9k, −340, −425, −4.1k]
|
||||
- **Correlation to Sharpe:** q_full r=+0.257 (decoupled), q_half r=−0.284
|
||||
(decoupled), q_quarter r=−0.375 (weakly anti-correlated).
|
||||
- **EMA pollution risk:** VERY HIGH. Slot 13's first observation was
|
||||
Q_quarter=−88k. With α=0.05, after one epoch the EMA is 0.95×−88k +
|
||||
0.05×new; a single such excursion dominates the EMA for ~20 epochs
|
||||
regardless of what the policy actually learns next. This is consistent
|
||||
with the user's background observation that "ep 2 Q-range=[-333k, +333k]
|
||||
poisoning Kelly cap for 20 epochs".
|
||||
- **Verdict:** REPAIR. Two fixes required:
|
||||
1. Clamp the source `q_mag_means_ptr[k]` input to a rolling robust
|
||||
scale before feeding the EMA — e.g. `clamp(x, −4×|EMA|, +4×|EMA|)`.
|
||||
Bounds outlier influence to 4× current mass without losing
|
||||
directional information.
|
||||
2. Skip the EMA update when `learning_health < 0.55` or when batch
|
||||
`ens_disagreement > 2σ_EMA` — effectively freezes the EMA during
|
||||
unstable epochs so collapse-fraction computations in C51 kernels
|
||||
see a stable historical baseline, not a corrupted one.
|
||||
|
||||
### Slot 16 — Magnitude |Q|-scale reference EMA
|
||||
|
||||
- **Written by:** `isv_signal_update` at `experience_kernels.cu:5758-5760`
|
||||
— α=0.05 EMA of `max(q_abs_ref_ptr[0], 0)`.
|
||||
- **Read by:**
|
||||
- `c51_loss_kernel.cu:214, 368, 505` (normaliser for magnitude
|
||||
collapse-fraction)
|
||||
- `c51_grad_kernel.cu:95, 170` (bin weight denominator)
|
||||
- `gpu_dqn_trainer.rs:9172` (exposure readback)
|
||||
- **Correlation to Sharpe:** n/a (not logged directly; implicitly the
|
||||
denominator of the `collapse_frac` and `bin_weight` formulas).
|
||||
- **EMA pollution risk:** VERY HIGH. Same α=0.05 plus `fmaxf(x, 0)` means
|
||||
a single Q-scale spike of 10⁵ takes ~20 epochs to normalize back down.
|
||||
The divisor in `bin_weight` formulas stays artificially inflated →
|
||||
`bin_weight → 1.0` → the adaptive mechanism self-disables for the rest
|
||||
of the run. This is worse than if the signal were merely noisy; it is
|
||||
silently permissive.
|
||||
- **Verdict:** REPAIR. Add `(x ≤ 4×EMA ? x : 4×EMA)` input clipping to
|
||||
prevent single-epoch spikes from driving the denominator. Alternative:
|
||||
switch to a rolling max over last 64 batches (harder against outliers
|
||||
than α-EMA) — but needs a scratch ring.
|
||||
|
||||
### Slots 17-20 — Per-direction Q-mean EMAs (Short/Hold/Long/Flat)
|
||||
|
||||
- **Written by:** `isv_signal_update` at `experience_kernels.cu:5786-5793`
|
||||
— α=0.05 EMA from `q_dir_means_ptr[k]`, populated by
|
||||
`launch_q_dir_bin_means_reduce` at `gpu_dqn_trainer.rs:3281-3310`.
|
||||
- **Read by:**
|
||||
- `c51_loss_kernel.cu:210-213, 453-456` (direction reward-bias —
|
||||
`q_hold`, `q_flat`, `q_tradable`)
|
||||
- `c51_grad_kernel.cu:137-140` (direction bin weighting)
|
||||
- `gpu_dqn_trainer.rs:9193-9196` (direction preference exposure)
|
||||
- **Correlation to Sharpe:** n/a (not logged directly); consumed through
|
||||
`reward_bias` derivation which affects Bellman target. Per
|
||||
`project_reward_bias_cross_branch_coupling.md`, a broken Q_dir EMA
|
||||
leaks into the magnitude branch via the shared trunk.
|
||||
- **EMA pollution risk:** SAME AS SLOT 13-15 — very high. Direction
|
||||
Q-means scale with Q-magnitude, and the observed run had direction Q
|
||||
magnitudes in the same ±10⁵ regime.
|
||||
- **Verdict:** REPAIR. Same two-step fix as slots 13-15: outlier clamp +
|
||||
health-gated update skip. This slot is especially load-bearing because
|
||||
it feeds the `q_tradable` path in reward-bias, and a corrupted
|
||||
q_tradable directly flips the `deficit = target_q − q[a0]` sign
|
||||
(potentially training the network in the wrong direction for up to 20
|
||||
epochs before the EMA recovers).
|
||||
|
||||
### Slot 21 — Direction |Q|-scale reference EMA (`Q_DIR_ABS_REF_INDEX`)
|
||||
|
||||
- **Written by:** `isv_signal_update` at `experience_kernels.cu:5794-5796`
|
||||
— α=0.05 EMA of `max(q_dir_abs_ref_ptr[0], 0)`.
|
||||
- **Read by:**
|
||||
- `experience_kernels.cu:1189` — CRITICAL: conviction denominator in
|
||||
`action_select`-style kernel (`conviction = q_range / q_dir_abs_ref`).
|
||||
Conviction then feeds the adaptive Kelly warmup floor.
|
||||
- `c51_loss_kernel.cu:215, 457` (lead_scale composite for reward-bias)
|
||||
- `c51_grad_kernel.cu:141` (same)
|
||||
- `gpu_dqn_trainer.rs:9197` (readback)
|
||||
- `trade_physics.cuh:254` documents the conviction dependency
|
||||
explicitly.
|
||||
- **Correlation to Sharpe:** n/a directly. INDIRECT ALARM: user
|
||||
background explicitly flagged "q_dir_abs_ref EMA getting contaminated
|
||||
by a single excursion poisoning Kelly cap for 20 epochs" — this slot is
|
||||
the denominator of `conviction`, so a ±10⁵ excursion in direction Q
|
||||
pushes conviction → ~0 for ~20 epochs → Kelly warmup floor collapses →
|
||||
position sizes collapse right when the policy genuinely diverges.
|
||||
- **EMA pollution risk:** VERY HIGH — highest-impact consumer (Kelly) +
|
||||
worst outlier sensitivity + slowest decay. Single-epoch contamination
|
||||
reaches far beyond the poison epoch.
|
||||
- **Verdict:** REPAIR (priority #2 after slot 12). Same clamp/health-gate
|
||||
approach as slot 16. Additionally, the fallback path at
|
||||
`experience_kernels.cu:1188-1193` already does `denom = (q_dir_abs_ref
|
||||
> 1e-6) ? q_dir_abs_ref : q_range` — the `1e-6` floor is too permissive;
|
||||
increase it to something structural like `0.1 × (v_max − v_min)`
|
||||
(matching the `lead_scale` floor already used in
|
||||
`c51_loss_kernel.cu:141`). This gives the Kelly path a robust fallback
|
||||
when the EMA is contaminated.
|
||||
|
||||
## Summary
|
||||
|
||||
| Slot | Name | Consumers (R=read, A=adaptive) | r vs Sharpe | Verdict |
|
||||
|------|------|--------------------------------|-------------|---------|
|
||||
| 0 | Q-drift | R: ISV MLP | unobserved | REPAIR (saturation floor) |
|
||||
| 1 | grad_norm EMA | A: adaptive stop width | unobs. (proxy r=+0.69) | REPAIR (outlier clamp) |
|
||||
| 2 | **TD-error EMA** | A: micro-reward `volatility`, IQN lagged target | **NONE (stuck 0)** | **REPAIR (no writer)** |
|
||||
| 3 | **Ensemble var EMA** | R: ISV MLP | **NONE (stuck 0)** | **REPAIR (no writer)** |
|
||||
| 4 | Ens var velocity | R: ISV MLP | NONE (stuck 0) | REPAIR (derivative of #3) |
|
||||
| 5 | "Reward" proxy EMA | R: ISV MLP | unobs. | REPAIR (mislabel; use real reward) |
|
||||
| 6 | Atom util EMA | R: ISV MLP | r=−0.17 | KEEP (expose in logs) |
|
||||
| 7 | Loss EMA | R: ISV MLP | unobs. | REPAIR (outlier clamp or ratio) |
|
||||
| 8 | Regime velocity | R: slot 10 chain | unobs. | REPAIR (sample-0 bias) |
|
||||
| 9 | Regime disagreement | R: ISV MLP | unobs. | REPAIR (sample-0 bias) |
|
||||
| 10 | Regime transition EMA | R: slot 8/self | unobs. | REPAIR (sample-0 bias) |
|
||||
| 11 | Regime stability | A: 7 heavy consumers incl. CQL alpha, hold width, micro-reward | implied saturated at 1 | REPAIR (sample-0 bias) |
|
||||
| 12 | **Learning health** | A: ~10 consumers (C51, CQL, env, distill, Kelly) | **r=−0.765 (ANTI)** | **REPAIR (fix components)** |
|
||||
| 13-15 | Mag Q-mean EMAs | A: C51 mag bin weights, reward-bias | direct r≈±0.3 | REPAIR (outlier clamp + health-gate) |
|
||||
| 16 | Mag |Q|-scale ref | A: C51 bin-weight denominator | unobs. | REPAIR (outlier clamp) |
|
||||
| 17-20 | Dir Q-mean EMAs | A: C51 dir reward-bias, bin weights | indirect | REPAIR (outlier clamp + health-gate) |
|
||||
| 21 | **Dir |Q|-scale ref** | A: **Kelly conviction denominator** + C51 bias | **unobs. but high-impact** | **REPAIR (clamp + raise fallback floor)** |
|
||||
|
||||
Bolded rows are the highest-severity failures: dead writers (slots 2, 3, 4),
|
||||
the anti-correlated health signal (slot 12), and the Kelly-conviction
|
||||
contamination risk (slot 21).
|
||||
|
||||
Also documented in HEALTH_DIAG but not formally part of the ISV bus — all
|
||||
stuck-signal diagnostics:
|
||||
|
||||
| Signal (not an ISV slot) | Stuck value | Epochs |
|
||||
|---|---|---|
|
||||
| `q_gap` (HealthEma component) | 1.00 | 19/20 |
|
||||
| `q_var` (HealthEma component) | 1.00 | 19/20 |
|
||||
| `grad_stable` (HealthEma component) | 0.00 | **20/20** |
|
||||
| `ens_agree` (HealthEma component) | 0.00 | 19/20 |
|
||||
| `ensemble_collapse` (novel detector) | 0.00 | 19/20 |
|
||||
|
||||
## Proposed fixes (priority order)
|
||||
|
||||
1. **Wire `td_error_scratch_pinned` (slot 2) from the loss kernel.** The
|
||||
pinned scratch is already allocated, the kernel ABI already accepts it,
|
||||
and per-sample TD errors are already computed (`td_errors_buf` at
|
||||
`gpu_dqn_trainer.rs:6299`). Add one reduction kernel call to write
|
||||
`mean(|td_error|)` into the scratch before `update_isv_signals`.
|
||||
Downstream payoff: the micro-reward volatility weight
|
||||
(`experience_kernels.cu:2138`) and the IQN recursive confidence target
|
||||
(`experience_kernels.cu:5655`) both stop reading zero.
|
||||
2. **Fix `learning_health` component saturation (slot 12).** Three
|
||||
sub-issues, each independently fixable in `trainers/dqn/trainer/*`:
|
||||
a. Root-cause `grad_stable` = 0.0 permanently — inspect
|
||||
`HealthEma::update_grad_consistency_vec` path (invoked at
|
||||
`training_loop.rs:1938`) and verify its divisor/floor.
|
||||
b. q_gap / q_var saturating at 1.0 — their normaliser in
|
||||
`HealthEma::update` is probably using an EMA divisor that lags the
|
||||
±10⁵ Q-scale explosion. Replace with a robust scale (median of
|
||||
recent Q-range, or use slot 16 / slot 21 as normalizers).
|
||||
c. `ens_agree` decay — after initial transient it sticks at 0, which
|
||||
suggests the Rust-side monitor stops receiving updates or the
|
||||
normaliser zeros out.
|
||||
Health's bad composition is the single most damaging ISV failure — its
|
||||
r=−0.765 means every consumer (C51 bin weights, reward-bias damping,
|
||||
CQL gate, Kelly safety, distillation alpha) is being steered in the
|
||||
wrong direction.
|
||||
3. **Outlier clamp on EMA inputs for slots 13-21.** Pre-clamp
|
||||
`q_mag_means_ptr[k]`, `q_abs_ref_ptr[0]`, `q_dir_means_ptr[k]`,
|
||||
`q_dir_abs_ref_ptr[0]` to `[−4×prev_ema, +4×prev_ema]` (or 4× for the
|
||||
unsigned |Q|-refs) before the α-blend in `isv_signal_update`. This is
|
||||
a 4-line patch inside that kernel and bounds a single bad epoch's
|
||||
pollution to ≤5× its own magnitude rather than ~20 epochs of leakage.
|
||||
4. **Wire `ensemble_var_scratch_pinned` (slot 3).** The ensemble monitor
|
||||
already computes per-head Q spread; route the per-epoch value into
|
||||
the pinned scratch the same way slot 6 (atom_util) is wired at
|
||||
`gpu_dqn_trainer.rs:11078`. Slot 4 (velocity) becomes meaningful
|
||||
automatically.
|
||||
5. **Fix the sample-0 bias in slots 8-11.** Replace the `states_ptr[0 ×
|
||||
state_dim + 40]` read with a one-thread batch-mean reduction over all B
|
||||
rows. The regime signals then reflect batch regime rather than whichever
|
||||
single row the shuffle happened to place first.
|
||||
6. **Raise the conviction fallback floor at `experience_kernels.cu:1191`.**
|
||||
Change the epsilon floor from `1e-6` to `0.1 × (v_max − v_min)` —
|
||||
matches the architectural floor already used by the `lead_scale` reward-
|
||||
bias computation (`c51_loss_kernel.cu:141`) and gives the Kelly path a
|
||||
meaningful denominator when slot 21 is polluted.
|
||||
7. **Relabel or repurpose slot 5.** It is reading `q_mean` but named
|
||||
`reward_scratch`; downstream MLP interpretation is wrong. Either feed
|
||||
the actual mean step-return (from `train_physics_step_buf`) or drop the
|
||||
slot and reuse it for a signal the ISV genuinely lacks (e.g. realised
|
||||
position P&L velocity, or win-rate EMA).
|
||||
8. **Add HEALTH_DIAG exposure for slots 0-7 and 13-21.** The diagnostic
|
||||
only shows slot 12 + derived mag_Q values directly. The remaining
|
||||
slots are invisible at runtime — making future regressions hard to
|
||||
detect. Adding them is a few lines in `training_loop.rs` where
|
||||
HEALTH_DIAG is emitted.
|
||||
|
||||
## Hypothesis on the oscillation
|
||||
|
||||
The +34 → −67 Sharpe collapse is most plausibly caused by three ISV signals
|
||||
simultaneously giving mis-information during epochs 5-10:
|
||||
|
||||
**Primary driver — slot 12 (`learning_health`):** r=−0.765 against Sharpe.
|
||||
This signal directly damps reward-bias (`c51_loss_kernel.cu:229`: `reward_bias
|
||||
× (1 − health)`), modulates CQL alpha (`gpu_dqn_trainer.rs:5555`: `cql_alpha
|
||||
× (1 − regime_stab) × health`), and sets the distillation alpha and Kelly
|
||||
warmup floor. Its climb from 0.50 to 0.64 over the observed window
|
||||
progressively:
|
||||
- DAMPS the reward-bias correction (the mechanism designed to correct the
|
||||
exact collapse regime being observed) — at health=0.5 the bias is fully
|
||||
applied; at health=0.64 it's damped 28% more.
|
||||
- Keeps `cql_alpha ≈ 0` regardless of training quality (multiplying by an
|
||||
apparently-saturated `regime_stab` that makes `(1 − stab) ≈ 0`).
|
||||
- Inflates `target_exposure` through the Kelly path — training progressively
|
||||
ramps position size exposure as health rises, which is exactly wrong when
|
||||
the policy is collapsing.
|
||||
|
||||
**Secondary driver — slot 21 (`q_dir_abs_ref_ema`):** user background flagged
|
||||
contamination from a single Q-excursion. In the observed data, direction
|
||||
Q-scales swung through ±10⁵ at epochs 0, 3, 7, 11 — each one re-poisons
|
||||
slot 21 for the next ~20 epochs. This pushes conviction → ~0 (denominator
|
||||
saturation), which triggers the Kelly warmup floor collapse path. Positions
|
||||
shrink, but gradient updates continue training against the contaminated
|
||||
Q-values — a classic positive-feedback loop.
|
||||
|
||||
**Tertiary driver — slot 2 (`td_error EMA`) being identically 0:** the
|
||||
micro-reward composition kernel at `experience_kernels.cu:2138` reads
|
||||
`volatility = isv_signals[2]` and uses it to scale `w_price` and
|
||||
indirectly `w_book`. Reading 0 means `w_price = 0.5 × weight` and `w_book
|
||||
= 0.5 × weight` regardless of actual TD-error regime — the micro-reward
|
||||
becomes regime-unaware exactly when the regime shifts. This is not the
|
||||
largest quantitative driver but it removes the last line of defense
|
||||
against the primary mechanism failures above.
|
||||
|
||||
Together: health mis-reports "improving" when the policy is diverging,
|
||||
q_dir_abs_ref gets polluted then stays polluted, and the micro-reward
|
||||
cannot respond to the divergence because its regime signal is dead. The
|
||||
adaptive mechanisms designed to self-correct such collapses are all
|
||||
driven by these three signals, so the corrective pressure is exactly
|
||||
inverted relative to actual outcome. The Sharpe collapse is not caused
|
||||
by a broken policy — it is caused by adaptive-mechanism mis-steering
|
||||
driven by broken ISV measurement.
|
||||
|
||||
The specific epoch where Sharpe tips (epoch 9: +26.3 → −8.4 → −37.2 →
|
||||
−42.7) is consistent with the outlier at epoch 7 (q_full=51k) finally
|
||||
contaminating both Q-scale EMAs past recovery point while health keeps
|
||||
rising due to its saturated components. This is a deterministic
|
||||
signature: fix slots 2, 3, 12, and 21 and the oscillation should not
|
||||
reappear.
|
||||
Reference in New Issue
Block a user