First end-to-end SP4 producer. Kernel reads denoise_target_q_buf,
computes p99(|target_q|) via sp4_histogram_p99<256>, writes step_obs
to producer_step_scratch_buf[0] with __threadfence_system. Launcher
syncs, applies Pearls A+D via pearls_ad_update (zero-copy mapped-pinned
reads of ISV[TARGET_Q_BOUND_INDEX=131] + wiener_state_buf[(131-base)*3]),
writes new x_mean back to ISV + state back to wiener_state. Cold-path
launch (no captured graph in this task; Task A10 may move to captured).
Producer-step-scratch slot 0 reserved for TARGET_Q_BOUND (stable layout
documented in launcher comment for Tasks A6-A11 to extend).
GPU unit test verifies kernel writes step_p99 ≈ p99(|N(0,1)|) within
5% rel_err on 4096 Box-Muller samples, then exercises Pearl A's
sentinel branch (sentinel ISV + zero Wiener state → first-observation
replacement). Helper asserts non-target scratch slots stay zero.
No consumer wired yet — Mech 1's clamp still uses 10 × Q_ABS_REF.max(1.0).
Behavior unchanged. cargo check --lib --tests clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Header-only `__device__` function `sp4_histogram_p99<BLOCK_SIZE>(buf, count)`
returning p99(|buf|) via three-pass single-block algorithm:
Pass 1: block-wide max-reduce → step_max (0.0 short-circuit on degenerate)
Pass 2: linear-spaced [0, step_max] binning into 256 bins via per-warp
tiles (no atomicAdd; lane-collisions cost <0.012% precision)
Pass 3: cumulative-from-top → p99 = bin upper-edge
Returns 0.0 for degenerate step_max=0 (caller skips ISV update). Linear
spacing chosen over log because SP4 producers care about resolution at
the top of the distribution — top bin width = step_max/256 ≈ 0.39%, well
within the 1% quantile precision budget.
Wrapper kernel `sp4_histogram_p99_test_kernel` exposes the device fn for
Rust testing; mapped-pinned scalar output with __threadfence_system() so
host read_volatile sees the result (no memcpy_dtoh). Build.rs
registration mirrors thompson_test_kernel.cu pattern + adds
rerun-if-changed for the .cuh header.
Unit test: 4096 deterministic |N(0,1)| Box-Muller samples, sorted
ground-truth p99 ≈ 2.576 z-score one-tailed, asserts rel_err < 5%
against device output. GPU-gated (#[ignore]). Local L40S run:
true_p99=2.59758, computed_p99=2.59858, rel_err=0.039% — passes.
No producer wired yet — header is library code included only by the
test wrapper; SP4 Tasks A5-A9 add the magnitude-bound producer kernels
that #include "sp4_histogram_p99.cuh" directly. Behaviour unchanged.
cargo check --lib --tests clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single source-of-truth implementation of:
- Pearl A (first-observation bootstrap): sentinel-detect at fold reset,
replace x_mean directly with first observation when prev_x_mean=0 AND
state.x_lag=0. Bypass Pearl D's Wiener math.
- Pearl D (Wiener-optimal adaptive α): for t≥1, α* = diff_var /
(diff_var + sample_var + ε_div); variances tracked at uniform meta-α.
6 unit tests: Pearl A sentinel replacement; Pearl D anchors at
stationary signal; Pearl D tracks step-change; Pearl D does NOT subsume
Pearl A at t=0 (mathematical correctness check from spec self-review);
meta-constants are structural; Pearl A only fires when both x_mean and
x_lag are zero (does not re-fire post-Pearl-D).
ALPHA_META = 1e-3 (structural — single uniform meta-rate, no per-signal
tuning). EPS_DIV = 1e-8 (Adam-ε numerical category). EPS_CLAMP_FLOOR =
1.0 (consumer cold-start floor).
No consumers wired yet — helper is library code unused by the producer
pipeline. Behavior unchanged. cargo check clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Layer A additive: three mapped-pinned buffers added to GpuDqnTrainer.
- wiener_state_buf (141 floats) — Pearl D state (47 producers × 3 floats:
sample_var, diff_var, x_lag). Per `feedback_no_htod_htoh_only_mapped_pinned`.
- clamp_engage_per_block_buf (2048 ints) — Pearl C engagement counters
(8 param-groups × 256 max blocks per Adam launch).
- producer_step_scratch_buf (47 floats) — per-producer per-step
step_observation output. Host applies Pearls A+D to map step_obs to
ISV bound slot via pearls_ad_update (Task A3, pending).
All three zero-initialized at construction (Pearl A sentinels). Reset
registry entries follow in Task A12.
No consumers wired yet — buffers are reserved but unread. Behavior
unchanged. cargo check clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 important + 1 optional findings from code quality review:
1. layout_fingerprint_seed() now lists all 40 SP4 slots and bumps
`ISV_TOTAL_DIM=131` -> `ISV_TOTAL_DIM=171`. Without this, the fail-fast
checkpoint-load guard would not detect the SP4 layout extension —
a binary built against new code would falsely compare-equal to old
checkpoints' fingerprints.
2. Added `SP4_BRANCH_COUNT=4` constant and `debug_assert!(branch <
SP4_BRANCH_COUNT)` in `atom_pos_bound`. Test extended to verify
atom_pos_bound max does not alias into WEIGHT_BOUND family.
3. Refreshed stale content in docs/isv-slots.md: ISV_TOTAL_DIM 96->171,
fingerprint location [37..39)/[47..49) -> [115..117), table entries
[94]/[95] -> [115]/[116].
4. Added `debug_assert!(group < SP4_PARAM_GROUP_COUNT)` to weight_bound,
adam_m_bound, adam_v_bound, wd_rate accessors (symmetric with the
atom_pos_bound change).
cargo check clean, cargo test slot_layout_is_contiguous_and_total_40 passes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PEARL A — First-observation bootstrap: eliminates Xavier-derived
formulas (2.33 z-score, √2 std, √(2/K_in)) from the bootstrap section.
Sentinel ISV[X]=0 at fold reset; producer step 0 detects sentinel,
replaces directly with step_observation. Subsequent steps EMA-blend.
Consumer cold-start safety via .max(1.0) numerical floor only (Adam-ε
category, not magnitude). Truly zero magnitude constants in bootstrap.
PEARL B — Fused per-param-group statistics oracle: producer count
36 → 14. Per-group fused kernel reads (params, grads, adam_m, adam_v)
once and computes WEIGHT_BOUND, ADAM_M_BOUND, ADAM_V_BOUND, WD_RATE
in one multi-pass operation. Trunk's oracle adds Pass E for L1_LAMBDA
gradient-direction entropy. 4× memory bandwidth reduction. Cleaner
conceptual unit (per-param-group bounds = one oracle).
PEARL C — Engagement-rate self-correction: detects post-clamp
feedback-loop saturation. For in-kernel clamps, theoretical engagement
rate = 1% (top 1% by p99 definition). Producer-side rate-deficit EMA
detects sustained deviation; force-bumps bound to step_max when
detected. Resolves the "in-kernel feedback loop accepted" limitation
from first draft. Per-Adam-kernel block-shared-memory engagement
counter (no atomicAdd), block-wide reduce, host-side rate-deficit EMA.
PEARL D — Wiener-optimal adaptive α: replaces all hardcoded EMA rates
across 14 new producers AND 7 existing pre-SP4 producers. Per-step:
α* = diff_var / (diff_var + sample_var + ε_num). Theoretically optimal
under Wiener-filter analysis; subsumes Pearl A as t=0 edge case
(both vars=0 → α=1 → first-observation replacement). On stationary
signals: α→0 (smooth). On non-stationary: α→1 (track). Eliminates
the recursion problem (α controlled by signal stats, not another α).
ε_num = 1e-8 (Adam-ε numerical category). 3 floats of state per
producer. 36+ hardcoded α values eliminated codebase-wide.
Limitations section restructured: 6 of 8 first-draft limitations
RESOLVED by pearls (in-kernel feedback loop, smoke time-budget,
producer plumbing, stale-bound, 8 carved-out items, magnitude bootstrap
formulas). Remaining 6 limitations are genuinely irreducible (F0
launch-scheduling variance, Layer B atomic flip risk, novel-pearl
field-validation, equilibrium-formula non-stationarity, Pearl C
counter-state cost, Pearl D state cost).
SP4 now closes 100% of the magnitude/regularization/EMA-rate surface.
No hardcoded scalars remain in the entire bound/clamp/EMA chain.
Producer count: 14. ISV slot count: 36. Unit tests: 36.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Folds two new producer signals into SP4 scope, eliminating the earlier
"carved out" exception:
WEIGHT DECAY (per param-group, 7 new ISV slots):
λ = |w·g| / max(||w||², ε)
Derived from equilibrium analysis of d/dt(||w||²) = 2(w·g) - 2λ||w||².
The equilibrium gradient-projection-onto-weight-direction divided by
weight norm. Same theoretical-derivation category as Adam β values.
EMA half-life α=0.005 (~140 steps). Bootstrap 1.0.
L1 LAMBDA (trunk only, 1 new ISV slot — NOVEL PEARL):
λ = (mean(|g|) / mean(|w|)) × D
where D = (log K - H_observed) / log K is gradient-direction entropy deficit
and H_observed = -Σ p[i]·log p[i], p[i] = ||g[:,i]|| / Σ ||g[:,j]||
L1 regularization-strength derives from gradient-direction entropy
deficit across input features. When gradient is uniform across features
(D≈0): network hasn't differentiated, λ=0 (no pruning). When gradient
concentrates on few features (D≈1): network has identified what matters,
λ ramps up to prune the rest. Self-curriculum — L1 strength tracks the
emergence of feature differentiation.
This extends pearl_adaptive_moe_lambda (regularization strength = EMA-
tracked deficit of regularized quantity) to feature-redundancy domain.
Pearl-name candidate (post-validation): pearl_signal_driven_regularisation_strength.
Bootstrap λ=0 means cold-start = no L1 pruning; ramp-up only after
gradient differentiates. Worst-case behavior is "L1 disabled" — graceful.
Total ISV slot count: 28 → 36. Total producer kernels: 28 → 36.
Effort estimate: 3000-4000 → 3500-4500 LOC, 1-1.5 → 1.5-2 weeks.
Acknowledged limitations updated: removed item #8 (carve-out) since
no carve-outs remain. Added items for L1 pearl novelty (untested) and
weight decay equilibrium-formula non-stationarity. Both have graceful
worst-case behavior and explicit validation criteria (#10 and #11) to
detect anomalies.
SP4 now closes 100% of the magnitude/regularization surface — no
hardcoded scalars remain in the entire chain. AdamW config fields for
weight_decay and l1_lambda removed from HyperParams to prevent
accidental hardcoding regression.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Second-pass self-review found a new critical issue: under "diagnostic =
clamp engagement", post-clamp diagnostics (Mech 9 weights, Mech 5 Adam
m/v slots 36-43, slots 44-45) NEVER fire — because post-clamp |v| ≤
bound by construction, so producer-side comparison always returns false.
Fix: split diagnostic implementation by clamp location.
- Buffer-based clamps (Mech 1, 2, 10): diagnostic stays in producer
(reads pre-clamp buffer, fires when step_max > bound).
- In-kernel clamps (Mech 6, 9): diagnostic lives INSIDE the Adam
kernel at the clamp step. Each Adam kernel takes a `diag_slot` arg
alongside `weight_clamp_max_abs`; on clamp engagement, writes
`nan_flags_buf[diag_slot] = 1`. Idempotent per-thread store (all
threads writing 1, race-free per existing convention). No atomicAdd.
Without this fix, slots 36-45 would be dead diagnostics under SP4 —
detecting nothing, providing no signal. With this fix, every diagnostic
slot fires on its corresponding clamp engagement regardless of where
the clamp lives.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Self-review found multiple problems with the first draft. This revision
addresses all 8 critical issues:
1. P² parallelization claim was WRONG — P² is sequential.
Replaced with dynamic-range histogram (3-pass single-block kernel:
max-reduce → log-spaced bin → cumulative-from-top to find p99).
256 bins → ~0.4% quantile precision; numerical-precision derivation
in same theoretical-constant category as floating-point precision.
2. Bootstrap values had wrong magnitudes — used Xavier σ instead of
p99 of max-element. Recomputed: WEIGHT_BOUND[group] = 2.33 ×
√(2/K_in[group]); H_S2_BOUND = 2.33 × √2 ≈ 3.3. All bootstraps
now from theoretical p99 under Xavier-init, not std.
3. EMA half-life of 700 steps (α=0.001) didn't converge in 5-epoch
smoke. Revised α=0.005 (~140 steps) for weight/Adam producers —
reaches ~99% convergence within one fold's training. Smoke now
validates steady-state behavior, not just bootstrap.
4. Weight decay, L1 lambda, CLIP_MULTIPLIER were listed in scope but
undesignable in producer-consumer pattern. CARVED OUT explicitly:
weight decay + L1 λ → separate research-spec; CLIP_MULTIPLIER and
MIN_CLIP subsumed by SP4's GRAD_CLIP_BOUND slot.
5. F0 ≥ 45 acceptance criterion was uncertain. Revised to F0 ≥ 37.5
(matches the 1e30-effectively-unclamped diagnostic smoke). The
~8-point F0 variance from launch-scheduling-shift is independent
of clamp value; SP4 cannot guarantee F0=45 even with ideal design.
6. Per-param-group p99 plumbing concretized: each producer takes
(offset, length) launch args; main DQN params buffer sliced into
trunk/value/branch using existing param_sizes layout knowledge.
7. Pre/post-clamp feedback loop EXPLICIT: producer runs BEFORE
consumer for buffer-based clamps (h_s2, target_q, atom_pos —
in captured graph immediately before clamp). Producer runs AFTER
for in-kernel clamps (weights, Adam m/v — feedback loop accepted
with documented soft-anchor dynamics). No more hand-waving.
8. Unit test strategy: per-producer kernel test with synthetic
Gaussian input → known p99 ≈ 2.33 → assert |computed - 2.33|<5%.
28 tests total. Catches algorithm bugs before L40S smoke.
Effort estimate revised UP: 3500-4500 LOC (from 2-3000), 1.5-2.5 weeks
(from 1-2). 28 producer kernels + 28 unit tests is more plumbing than
first draft assumed.
Self-review limitations section now explicit about: in-kernel feedback
loop accepted, smoke time-budget marginally validates Adam EMAs, F0
may not return to 45, Layer B atomic-flip risk, plumbing density,
stale-by-one-step bound, histogram-precision is design choice not
tuning, carved-out items remain hardcoded post-SP4.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comprehensive design replacing every hardcoded magnitude multiplier in
the SP3 mechanism stack (Mechs 1, 2, 5, 6, 9, 10) plus pre-SP3 mechs in
the same magnitude-control surface, per feedback_isv_for_adaptive_bounds
and feedback_adaptive_not_tuned.
Core principle: the BOUND lives in an ISV slot, computed by a producer
kernel as p99 EMA of observed signal magnitude. Consumer reads the slot
and clamps directly — no multiplier between ISV read and clamp. Cold-
start ε from theoretical-init bootstrap (Xavier, etc.) — same theoretical-
constant category as Adam β values, not tuning knobs.
Architecture:
- 28 new ISV slots (7 base bounds × per-param-group split where appropriate)
- Per-signal P² (Jain-Chlamtac) quantile producer kernels
- Diagnostic = clamp engagement (sticky flag from producer's max-comparison)
- Migration in 3 layers: additive infra → atomic consumer flip → smoke
Out of scope: theoretical/structural constants (Adam β, Xavier formula,
attention 1/√d, hidden_dim, num_atoms). EMA rates stay as documented
statistical-design parameters (half-life ≈ observation time-window).
Estimated: ~2-3000 LOC across 3 layers, 1-2 weeks, 1 L40S smoke.
Awaiting user spec review before writing implementation plan.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mech 10 with 100× multiplier (commit 55576d047) restored F1 to +81.16
and F2 to +80.55, both above baseline 55.87 — F1 cascade definitively
fixed. But F0 dropped to 21.09 (vs v2's 45.47). HEALTH_DIAG shows
Q-values O(0.01) and noisy-σ at 0.032 in F0 epoch 1 — natural h_s2
magnitudes should be tiny (~0.1-5), so 100×max(isv,1)=100 floor
shouldn't bind. Yet F0 plateaus at epoch-1's 21.09 with no improvement
in epochs 2-5 — same value as smoke-test-d25vq's over-clipped F0.
Two hypotheses to distinguish:
1. Clamp value is over-tightening h_s2 in F0 (despite the math).
2. The new launch in submit_forward_ops_main shifts captured-graph
kernel scheduling, changing TF32 sgemm accumulator order, which
pushes F0 into a different deterministic convergence basin.
Investigation: raise multiplier to 1e30. With 1e30 × max(isv,1) =
1e30 always, launch_clamp_finite_f32 degrades to a pure NaN/Inf
sanitizer — `isfinite(v) ? v : 0.0f` — leaves finite values
untouched (|v| < 1e30 always). The launch still happens
(scheduling preserved), value-clamping effectively removed.
Decision criteria for the smoke at this commit:
- F0 returns to ~45 + F1 still healthy → 100× was over-clamp; ship a
middle multiplier (1e3 or 1e4) that protects F1 without binding F0
- F0 returns to ~45 + F1 NaNs again → clamp value matters for F1;
search for a multiplier that protects F1 without over-tightening F0
- F0 stays at ~21 → kernel-scheduling artifact, not Mech 10 multiplier;
F0=21 is launch-presence-induced basin shift independent of clamp
value, and the right SP3 close-out is to keep Mech 10 at 100× and
document F0 RNG variance as residual
Temporary diagnostic commit; final SP3 close-out reverts to the proper
multiplier informed by the smoke result.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Real root cause of the F1 step-1000 NaN, identified by smoke-test-2xrxk
diagnostic. With Mech 9 clamping weights at 100×Q_ABS_REF=5000, F0/F2
trained clean (weights stay at natural scale ~0.5) but F1's regime
shift drove weights toward the clamp ceiling. With weights pinned at
5000 and K=256 inner-dim, forward GEMM accumulators compound by
~80000× per layer; trunk depth ≥6 (encoder + VSN + GRN-blocks +
bottleneck) hits f32_max ≈ 3.4e38 in the chain.
Slot 12 (save_h_s2) firing was the smoking gun — h_s2 itself goes
non-finite mid-forward. Slots 39+43+48 (IQN-side) stayed clean,
disproving the close-out v2 hypothesis (IQN partial refactor) and
confirming the surface that wasn't protected: ACTIVATIONS.
Mech 1+2 clamp targets+atoms. Mech 9 clamps weights. Nothing clamped
the trunk's intermediate hidden state. Mech 10 closes that gap by
clamping h_s2 at 100 × H_S2_RMS_EMA.max(1.0) immediately after the
trunk + temporal pipeline finalises it, before any downstream consumer
reads it (branches, IQN, value head, replay save).
Implementation:
- Reuses existing launch_clamp_finite_f32 (dqn_utility_kernels.cu:462)
which does isfinite(v) ? clamp(v) : 0.0f — NaN→0, Inf→0, finite
clamped. No new kernel.
- ISV-driven via H_S2_RMS_EMA_INDEX=96 (already consumed by mag_concat
adaptive scale per P4.T2c.3c.6 — no new slot).
- ε on multiplier per SP1 pearl: isv[96].max(1.0). Cold-start RMS
near 0 floors the bound at 100×1=100, not at 0.
- Inline NaN check (slot 12) preserved BEFORE the clamp per SP1
diagnostic-ordering pearl — sentinel sees the original Inf/NaN
before sanitization replaces it with 0.
- Insertion site: end of submit_forward_ops_main 1c temporal-pipeline
block (after mamba2_step + apply_regime_dropout — the last writers
to save_h_s2), immediately before launch_curiosity_inference and the
loss path. Captured in the same forward child graph.
New diagnostic slot 49 = h_s2_max_abs at threshold 1e3 × H_S2_RMS_EMA.max(1.0).
Mirrors Mech 5 family pattern (clamp at 100×, diagnostic at 1000×) —
sentinel that NEVER fires under normal Mech 10 operation. If it does
fire, the activation clamp is disabled or H_S2_RMS_EMA itself drifted.
Buffer bumps: nan_flags_buf 49→50, metadata 25→26, fused-kernel block
count 25→26. Kernel signature gains second ISV-derived float param
h_s2_rms_ema_eff (distinct from q_abs_ref_eff — h_s2 lives in
activation space, not Q space; don't conflate the two ISV bases).
Both name tables in training_loop.rs (halt_nan + halt_grad_collapse)
extended to 50 entries.
No new ISV slots, no new kernels, no graph-topology surprise (one
extra launch in captured graph). cargo check clean. Audit docs
docs/dqn-wire-up-audit.md (Mech 10 entry prepended) and
docs/dqn-backward-nan-audit.md (Mech 10 row in mechanism table + slot
49 row in slot-allocation table + ISV bindings note + full Mech 10
section) updated per Invariant 7.
Validation: deferred to one L40S smoke. Expected: F1 trains past
step 1000; slots 36-43 + slot 49 stay quiet.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The SP3 close-out v2 commit 1bcc70392 added the slot-48 "iqn_weight_max"
name to the halt_nan path's name table at training_loop.rs:2057 but missed
the symmetric halt_grad_collapse path's table at line 2143. With
nan_flags_buf now sized 49, the halt_grad_collapse formatter would
out-of-bounds index when slot 48 fires under that path.
One-line additive fix: append the same `"iqn_weight_max"` entry (with the
same SP3 close-out v2 comment) so both name tables match the buffer size.
Wire-up audit updated to document the symmetry requirement.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Real fix for the F1 step-3540 NaN. Commit 8956c2fb7 wired Mech 9
(post-Adam |p_val| clamp) into ONE Adam kernel (dqn_adam_update_kernel)
out of FIVE in the codebase. The slot-26 GEMM (apply_iqn_trunk_gradient
output) computes `grad_iqn @ W_iqn^T`; W_iqn lives in IQN's separate
param buffer, updated by iqn_adam_kernel — never reached by Mech 9.
Slots 44-45 only cover trunk + heads, leaving IQN weights without a
diagnostic. The chain: IQN weights drift via iqn_adam_kernel → finite
gradient × non-finite weight → slot-26 NaN. Same partial-refactor class
as Mech 4's missing GpuAttention reset (caught by B5 audit), corrected
the same way: every consumer of the contract migrates together.
Extended Mech 9 to all four remaining Adam kernels:
- iqn_adam_kernel (iqn_dual_head_kernel.cu)
- iql_adam_kernel (iql_value_kernel.cu)
- attn_adam_kernel (attention_backward_kernel.cu — used by GpuAttention + GpuTlob)
- curiosity_adam_step (curiosity_training_kernel.cu)
Same `if (weight_clamp_max_abs > 0.0f) p = fminf(fmaxf(p, -bound), bound)`
pattern. Same `100 × Q_ABS_REF.max(1.0)` ISV-driven bound. Each launch
site computes the bound host-side and passes it as the trailing kernel
arg, mirroring the existing dqn_adam_update_kernel pattern. DT launch
keeps the 0.0 disable (offline-RL, outside SP3 scope). Curiosity reads
ISV from FusedTrainingCtx in training_loop and threads through the
collector wrapper (collector doesn't own ISV).
Added IQN-weight diagnostic slot 48:
- nan_flags_buf 48 → 49
- Fused-kernel block count 24 → 25; new branch for absolute slot 48
(relative slot 24): threshold = 1e3 × q_abs_ref_eff (matches slot 44-45)
- Metadata buffers (nan_check_buf_ptrs/_lens) populate slot 48 with
IQN online_params pointer + length (new public accessors on GpuIqnHead)
- name table (halt_grad_collapse): 49 entries with "iqn_weight_max"
- Audit doc: slot 48 row in Mech 5 table; Mech 9 row updated to span
all 5 Adam kernels
No new ISV slots. No graph-topology change beyond the +1 block in the
already-fused NaN check. cargo check -p ml --lib clean (12 pre-existing
warnings, none new).
Validation: deferred to one L40S smoke at this HEAD. Expected: F1
trains past step 3720 (Mech-6-only ceiling) and slots 36-43 + new
slot 48 stay quiet.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace `<this commit>` placeholders with the actual close-out SHA
8956c2fb7 in the audit-doc mechanism table. Also append decision_transformer.rs
to the Mech 9 file-list cell — the implementer correctly wired the
DT Adam launch with the new kernel arg (passing 0.0 to disable, since
DT is offline-RL training outside SP3 scope and has no ISV bus access);
the file-list now reflects all three touched paths.
Doc-only fix-up. No code change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Coordinated close-out of SP3 Q-learning numerical-stability per
feedback_no_partial_refactor.
REVERT Mech 8 (slow_ema fold-boundary reset). smoke-test-rxhjh on b8a7ac6f7
showed F1 NaN at step 2300 — WORSE than the 3720 ceiling with Mech 6 alone.
Persistent slow_ema across folds was providing unintentional F1 protection
(anchored at F0's smaller grad scale → tighter Mech 6 upper_bound during F1
ramp-up). Resetting loosened the clip and accelerated Adam saturation.
Removed: reset_grad_norm_slow_ema() method + reset_for_fold call site.
Kept: grad_norm_slow_ema_pinned field (consumed by Mech 6).
Kept: Mech 6 multiplier at 100× (already restored from the 5× experiment
in the Mech 8 commit; correct steady-state value).
ADD Mech 9 (post-Adam ISV-driven weight clamp). Root cause being targeted:
cuBLAS sgemm f32 accumulator overflow at slots 26 (iqn_trunk_m) + 32
(bn_d_concat_buf) at F1 step ~3540 with INPUTS clean (slots 27, 35
unflagged). The matmul output saturates because the WEIGHT matrices have
drifted to extreme-but-finite magnitudes via Adam updates over thousands
of steps. Mech 1 (gradient clamp) and Mech 6 (grad-norm clip) bound
gradients, not parameters — neither prevents this drift.
Mech 9 clamps |p_val| ≤ 100 × Q_ABS_REF.max(1.0) inside dqn_adam_update_kernel
after the L1 proximal step. ISV-driven (slot 16, ε on multiplier per SP1
pearl). 1 OOM below slot 44-45 diagnostic threshold (1e3 × Q_ABS_REF) so
the regression sentinel retains 10× firing headroom — symmetric with
Mech 1's clamp/diagnostic ratio.
Implementation:
- dqn_adam_update_kernel: new trailing arg `weight_clamp_max_abs`.
Clamp `p_val = fminf(fmaxf(p_val, -bound), bound)` after L1 step.
- All Adam launch sites in gpu_dqn_trainer.rs: compute bound from
read_isv_signal_at(Q_ABS_REF_INDEX).max(1.0) × 100.0 host-side, pass
as final arg.
- No new ISV slots. No new kernels. No graph topology change.
Validation: deferred to one L40S smoke at the new HEAD. Expected:
- F1 trains past step 3720 (Mech-6-only ceiling) — Mech 9 closes SP3.
- F1 still NaNs at similar steps — close SP3 honestly with documented
residual pointing at SP4/SP5.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User-identified root cause: grad_norm_slow_ema (α=0.001, half-life
~693 steps) persists across fold boundaries while every other
distribution-tracking signal (Adam m/v, target nets, atom positions)
gets reset. Mech 6's upper_bound formula (100 × slow_ema × isv) was
anchored to the WRONG scale during F1 ramp-up, driving the
non-monotonic multiplier-tuning dance across smokes:
- smoke-test-fxvkk (mult=100×): F0=44, F1 NaN @ 3720
- smoke-test-ftdjz (mult=100×, +Mech7): F0=38, F1 collapse @ 2040
- smoke-test-d25vq (mult=5×): F0=21, F1 collapse @ 2820
The multiplier was searching the wrong dimension — the anchor itself
was stale.
Three coordinated changes (per feedback_no_partial_refactor):
1. RESTORE Mech 6 multiplier 5× → 100×. The original 100× was correct
for steady-state; the F1 saturation was driven by anchor staleness,
not multiplier looseness. Tightening it harmed F0 (over-clip during
ramp-up when slow_ema lagged grad_norm_ema).
2. ADD reset_grad_norm_slow_ema method on GpuDqnTrainer. Zeroes the
mapped-pinned scalar. First step of new fold builds up slow_ema
fresh, with MIN_CLIP=1.0 floor active during the brief transient.
3. WIRE Mech 8 call in fused_training.rs::reset_for_fold, alongside
Mech 4's existing Adam resets. Mech 6's anchor now aligns with
the new fold's grad scale from step 1 — same philosophy as Mech 3
(target net hard-sync) and Mech 4 (Adam EMA reset).
Net SP3 design: Mech 6 stays at 100× multiplier (broad, principled
headroom), Mech 8 keeps the anchor honest. The pair is more robust
than either change alone:
- Without Mech 8: anchor is stale, multiplier tuning has no winning
setting (5× hurts F0 ramp, 100× lets F1 saturate at slot 36).
- With Mech 8: anchor is fresh per fold; 100× multiplier provides
legitimate per-step headroom over CURRENT fold's grad norm.
Mech 7 stays reverted (per-element clip was misdiagnosis — over-clipped
legitimate gradient outliers without addressing the saturation root
cause).
F0 risk: low — F0 starts with slow_ema=0 anyway (cold start), so Mech 8
is a no-op on F0. Only changes F1+F2 fold-boundary behavior.
F1+F2 expectation: Mech 6's upper_bound now scales with the CURRENT
fold's grad norm, providing legitimate ~10× headroom per step without
allowing Adam EMA saturation. Slot 36-42 should stay quiet.
Smoke smoke-test-ftdjz (commit d9a4d98a3, Mech 6 + Mech 7) regressed
both F0 (44 -> 38.82, per-element cap clipped legitimate outliers) and
F1 (grad-collapse at step 2040 vs 3720 with Mech 6 alone). Slots 36-42
STILL fired — Mech 7's per-element clip didn't prevent Adam EMA
saturation, just slowed training to grad-collapse.
Re-analysis: Mech 6's 100x multiplier on the upper-bound formula was
mismatched with the slot 36 threshold ratio. Per-element gradient max
<= adaptive_clip = 100 x slow_ema x isv ~= 200. Adam m_X steady-state
reaches 200, exceeding slot 36 threshold of 100*isv = 100. Mech 6 was
doing its job but the bound was wider than the diagnostic threshold.
Two coordinated changes (per feedback_no_partial_refactor):
1. REVERT Mech 7 (per-element clip in dqn_adam_update_kernel). The
per-element approach was misdiagnosis — clipping post-global-clip
gradients tighter than legitimate per-element variance harms F0
training without addressing Adam saturation root cause. Kernel
returns to its post-Mech-6 state (blob 546feee48).
2. TIGHTEN Mech 6's upper-bound multiplier from 100x to 5x. Standard
DL practice (5-10x steady-state grad norm). Per-element gradient
max becomes <= 5 x slow_ema x isv ~= 10, well below slot 36
threshold of 100. Adam m_X EMA stays bounded <= 10 — slots 36-42
should not fire.
Why 5x and not 10x: slot 36 threshold is 100 x isv. 5x slow_ema
(~=10 absolute) leaves 10x headroom against the threshold, providing
robust margin. 10x slow_ema would be 20 absolute, only 5x margin —
risk of fluctuations triggering slot 36.
Why not tighter (e.g., 2x): per-step gradient norms can legitimately
spike to 5x slow_ema in normal training (e.g., gradient resumption
after warmup); tighter bounds would over-clip.
F0 risk: low — F0 typical adaptive_clip values are bounded by Mech 6's
upper anchor only when the EMA-driven clip exceeds 5x slow_ema, which
is rare in steady F0 training. Cap should be a no-op for F0.
F1 risk: prevents the saturation pathway diagnosed by smoke
smoke-test-ftdjz. Validates by running the next smoke at this commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Smoke smoke-test-fxvkk (commit 48c25d999, Mech 6 anchored clip) F1-NaN'd
at step 3720 with slots 36-38, 40-42 still firing — Adam EMAs saturated
despite Mech 6 capping the global clip threshold.
Diagnosis: Mech 6 bounds the AGGREGATE L2 gradient norm, but Adam m/v
EMAs are PER-ELEMENT. A gradient with one large element (e.g.,
element_X = 100, rest small) has L2 norm ≈ 100, passing a clip
threshold of 1000 untouched. Adam m_X EMA accumulates the large
element; β1=0.9 steady-state gives m_X ≈ 1000, exceeding slot 36
threshold (100 × isv).
Fix: in dqn_adam_update_kernel, after the global L2 clip is applied,
clip EACH gradient element to ±per_element_cap where:
per_element_cap = 10 × sqrt(adaptive_clip / total_params)
Rationale:
- sqrt(adaptive_clip / N) is the average per-element contribution to
the L2 norm budget
- 10× allows legitimate per-element deviations up to 10× average
- Scales with adaptive_clip — when global clip is doing its job
(Mech 6), per_element_cap is tight enough to prevent saturation
- When global clip is loose (post-fold warmup), per_element_cap
scales with it — never tighter than the global clip's intent
Together with Mech 6, prevents Adam saturation at both the aggregate
(L2 norm) and per-element levels. Closes the residual pathology after
Mech 6's partial 3060→3720 improvement.
Adds an upper bound to update_adaptive_clip's new_clip formula to
prevent clip-drift pathology: consecutive elevated samples were
ratcheting the EMA-driven clip threshold upward without bound,
eventually rendering clipping a no-op against in-distribution drift.
Smoke smoke-test-5rqzs (commit b9edccfc1) F1-NaN'd at step 3060 with
Mech 5 diagnostic slots 36-38, 40-42 firing — DQN main Adam m + v
EMAs saturated. Diagnostic confirmed Adam saturation as the root cause.
Diagnostic chain identified: grad_norm_ema ratchets up via repeat-
sample compounding (winsorizer caps single-sample but not consecutive
elevated samples) -> clip threshold > actual grad_norm -> clip no-op
-> Adam m/v poisoned -> saturate at slot 36-42 thresholds -> cuBLAS
overflow at slots 26+32 -> loss=34.68 grad=inf.
Fix: cap new_clip by grad_norm_slow_ema * 100 * isv[Q_ABS_REF=16].max(1).
- grad_norm_slow_ema (existing alpha=0.001 slow EMA): anchors against
legitimate steady-state grad norm
- 100x: headroom for per-step deviations
- isv[Q_ABS_REF].max(1): adaptive scale per
feedback_isv_for_adaptive_bounds; epsilon on multiplier per SP1 pearl
- .max(MIN_CLIP): cold-start epsilon-floor
Standard DL practice — bounds the runaway EMA clip-drift while
preserving adaptive behavior. F0 risk: low — slow_ema * 100 is
broad headroom; F0-typical clip thresholds are well below this
cap. F1+F2 benefit by preventing the ratchet pathology.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces rsv36-rsv47 placeholders with the SP3 Mech 5 diagnostic
slot names. Both name table sites (halt_nan + halt_grad_collapse)
updated identically per shared-contract migration principle
(feedback_no_partial_refactor).
Slot names mirror the audit doc table:
- 36-39: Adam m max-abs (trunk/value/branch/IQN)
- 40-43: Adam v max-abs (trunk/value/branch/IQN)
- 44-45: Weight max-abs (trunk/heads)
- 46: target_q post-clip
- 47: atom_span_max
When the fused kernel sets a slot bit, the readback log line names
the buffer that exceeded its ISV-derived threshold — providing
direct observability for SP3 mechanism effectiveness.
Extends dqn_nan_check_fused_f32_kernel to handle 24 slots (12 NaN-only +
12 ISV-threshold). Per-slot thresholds computed inline in the kernel
from a single q_abs_ref_eff arg (no HtoD per step; no separate
thresholds buffer needed). Slot index → threshold mapping:
- 12-15 (slots 36-39): Adam m ≥ 100 × q_abs_ref_eff
- 16-19 (slots 40-43): Adam v ≥ 1e6 × q_abs_ref_eff²
- 20-21 (slots 44-45): Weight max ≥ 1e3 × q_abs_ref_eff
- 22 (slot 46): target_q ≥ 95 × q_abs_ref_eff (= 9.5 × max_abs_target_q)
- 23 (slot 47): atom span ≥ 190 × q_abs_ref_eff (= 9.5 × max_atom_abs × 2)
nan_check_buf_ptrs/lens resized 12→24 entries (mapped-pinned host
write at construction). populate_nan_check_meta extended with 4 new
args for IQN Adam m/v ptr+len (Option<u64>/Option<usize> — null when
IQN inactive).
Grid: 12 → 24 blocks. Single launch covers all 24 backward-path
diagnostic slots. Slot 31 (deferred) + slots 33-35 (inline elsewhere)
+ slots with null IQN entries no-op via the kernel's null-pointer
guard.
q_abs_ref_eff = max(isv[Q_ABS_REF=16], 1.0) — ε on multiplier per SP1
pearl; cold-start ISV (~0) gives q_abs_ref_eff = 1, so all thresholds
floor at their ε-floor multiplier × 1.
SP3 Mech 5 closes the diagnostic instrumentation loop — slots 36-47
fire when their threshold is exceeded, providing observability for
SP3's other 4 mechanisms' effectiveness. Fail-safe: if SP3 fix doesn't
fully resolve F1 NaN, slot 36-47 firing pattern guides the next
iteration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Audit of all Adam optimizer state in crates/ml/src/ against the existing
reset_adam_state calls at the fold-boundary in
fused_training.rs::reset_for_fold:
- DQN main (line 971) — covers m_buf/v_buf, IQN-trunk, q_attn, sel,
denoise, mamba2, ofi_embed, PopArt, Q-div EMA via
GpuDqnTrainer::reset_adam_state. Also covers VSN params (in main
params buffer slots) and aux next_bar/regime heads (slots 119-126).
- IQN head (line 1011)
- TLOB (line 1026)
- IQL-high (line 1031)
- IQL-low (line 1036)
Missing wire identified and added:
- GpuAttention (4-head feature attention on h_s2): owns a separate
(attn_m, attn_v, attn_adam_step) tuple in gpu_attention.rs, called
every training step via attn.adam_step at fused_training.rs:1962/1995,
but was never zeroed at fold boundary. Same pathology as IQN/TLOB/IQL
— fold N momentum oversizes fold N+1's first-epoch SDP + output
projection updates, compounding through the trunk gradient.
Implementation:
- Add GpuAttention::reset_adam_state mirroring the gpu_iqn_head and
gpu_tlob pattern (memset_zeros m/v + zero step counter + write 0
through pinned t_pinned for next adam_step launch).
- Wire it under `if let Some(ref mut attn) = self.gpu_attention` in
reset_for_fold, immediately after the IQL-low reset, with the same
warn-on-failure / info-on-success log pattern.
- Append SP3 Task B5 entry to docs/dqn-wire-up-audit.md documenting
the audit + fix (Invariant 7 contract).
Out-of-scope optimizers (documented in audit, not changed):
- DecisionTransformer: scoped within DT pretrain block, dropped at end
of pretrain (no fold leak possible).
- GpuCuriosityTrainer: owned by GpuExperienceCollector, external to
FusedTrainingCtx. Reset belongs at the collector layer if needed.
- MetaQNetwork: host-side observability MLP for collapse prediction
(does not feed LearningHealth or trunk gradients).
- GpuMoeHead: forward-only, no Adam state of its own.
Comprehensive coverage now per feedback_no_partial_refactor — every
in-context Adam state buffer zeros at fold transition. SP3 Mech 4
closes the "Adam EMA accumulation drives weight pathology" pathway.
Clamps C51 atom positions during all dynamic write paths to
±10 × ISV[Q_ABS_REF=16].max(1.0) via inline fminf(fmaxf(...)):
- atoms_update_kernel.cu — active GPU-driven shared atom grid
(all 4 branches via grid.x = branch_id)
- iql_value_kernel.cu::iql_compute_per_sample_support — per-sample
[v_min, v_max, delta_z] tile (delta_z recomputed from clamped span)
- experience_kernels.cu::adaptive_atom_positions — legacy single-branch
entry kept in lockstep per feedback_no_partial_refactor
Same ISV bound as Mech 1 (target_q clip) — atoms and target_q share
the magnitude scale. ε on the multiplier (`isv.max(1.0)`) per the SP1
ε-floor pearl; bound ≥ 10 even with cold-start ISV.
Closes the second of two Q-target inflation pathways: with Mech 1
capping the projection target and Mech 2 capping the atom support,
the C51 expected_q (mean of atom × prob) is bounded by
±10 × ISV[16].max(1.0) regardless of probability mass distribution.
Prevents the Adam EMA saturation → weight pathology → cuBLAS overflow
chain.
Reuses isv_signals already in scope at all three kernels — no new
kernel arg, no new launch site, no new ISV slot, no new buffer,
no new kernel.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Clips denoise_target_q_buf to ±10 × ISV[Q_ABS_REF=16].max(1.0) after
compute_denoise_target_q. Reuses dqn_clamp_finite_f32_kernel from SP1
(no new kernel). Single-point clip covers all downstream consumers
(C51 / IQN / MSE).
ε on multiplier per SP1 pearl; bound at least 10 even with cold-start
ISV. F0 paper-review: F0-typical |target_q| ≤ 10; with Q_ABS_REF EMA
≈ 1-5 at F0, max_abs = 10-50 — F0 guard no-op. F1 inflation
(thousands+) gets clamped, breaking the Q-target inflation pathway
that drives Adam EMA saturation → weight pathology → cuBLAS overflow.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds pub(crate) accessor methods on GpuDqnTrainer and pub accessors
on GpuIqnHead exposing device pointers for the SP3 Mech 5 threshold
checks at slots 36-47:
GpuDqnTrainer (pub(crate)):
36: trunk_adam_m_ptr/_len — m_buf trunk slice (tensors [0..13))
37: value_adam_m_ptr/_len — m_buf value-head slice (tensors [13..17))
38: branch_adam_m_ptr/_len — m_buf branch-head slice (tensors [17..33))
40: trunk_adam_v_ptr/_len — v_buf trunk slice
41: value_adam_v_ptr/_len — v_buf value-head slice
42: branch_adam_v_ptr/_len — v_buf branch-head slice
44: trunk_params_ptr/_len — params_buf trunk slice
45: heads_params_ptr/_len — params_buf value+branch concat
46: target_q_ptr/_len — denoise_target_q_buf [B, 12]
47: atom_positions_ptr/_len — atom_positions_buf [4, num_atoms]
GpuIqnHead (pub):
39: adam_m_ptr/_len — IQN Adam first-moment buffer
43: adam_v_ptr/_len — IQN Adam second-moment buffer
DQN Adam state is UNIFIED in m_buf/v_buf (single TOTAL_PARAMS-sized
buffer covering trunk + heads at the same offsets as params_buf).
Slot 36/37/38 (and 40/41/42) accessors return pointers into the same
buffer at offsets computed via padded_byte_offset over the existing
compute_param_sizes layout. The kernel discriminates by slot index
for threshold checks. IQN Adam state lives separately on GpuIqnHead.
Trunk/heads param-buf split (slots 44/45) uses the same offset helper
— no new buffers, no new ISV slots, no DtoD/HtoD copies. Each ptr
accessor returns self.<field>.raw_ptr() (or +offset for slices); each
len accessor returns self.<field>.len() or computed from
compute_param_sizes.
Audit doc updated (docs/dqn-wire-up-audit.md SP3 Task B1 entry).
Used by populate_nan_check_meta_v2 (Task B6) to feed the fused
kernel's threshold-check entries when extended to 24 slots. Unused
yet — wired in B6.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
run_nan_checks_post_backward now fires a single fused kernel launch
covering slots 24-35 in 12 blocks. Reduces graph-capture overhead from
8 launches to 1 per step. Slots 33-35 inline checks at backward
orchestration sites (gpu_dqn_trainer.rs:18580/:18601/:6939) unchanged
— multi-point localization preserved.
Method signature simplified — IQN pointers no longer per-step args.
populate_nan_check_meta (called once at construction in fused_training.rs)
baked Option<u64> nullity into the metadata buffer entries; the fused
kernel's null-pointer guard handles slots 27/28 when IQN inactive
without per-step branching at the Rust caller.
Both call sites in fused_training.rs (ungraphed + graph-captured paths)
updated together per feedback_no_partial_refactor.
This is the F0 regression fix — Phase B's per-step kernel-launch
overhead was the F0 cause across 3 SP1 smokes (F0 ~ 35 vs baseline
55.87). Gate 1 smoke (Task A6) validates F0 >= 53.08.
Replaces A2's CudaSlice<u64>/<i32> field types with MappedU64Buffer/
MappedI32Buffer per feedback_no_htod_htoh_only_mapped_pinned. Mapped-
pinned eliminates the HtoD copy entirely — the kernel reads via the
device-mapped pointer (cuMemHostGetDevicePointer_v2) while the trainer
writes through the same mapped pages on the host side.
Adds populate_nan_check_meta on GpuDqnTrainer (one-shot construction-
time write of 12 (ptr, len) tuples for slots 24-35). Slot 31 deferred
(null entry); slots 27/28 nullable on Option<u64> (None when IQN
inactive); slots 33-35 null (inline checks fire separately at backward
orchestration phases — kept individual for entry-point localization).
Adds launch_nan_check_fused_f32 (per-step kernel launch wrapper with
grid_dim=12, block_dim=256, base_flag_idx=24). Registers
dqn_nan_check_fused_f32_kernel in compile_training_kernels (tuple
43→44, info log 38→39 utility kernels) — same module as the per-buffer
dqn_nan_check_f32 to share the captured replay group.
Constructor-time wire-up lands in FusedTrainingCtx::new after gpu_iqn
construction (gpu_iqn is owned by FusedTrainingCtx, not GpuDqnTrainer
— mirrors the same Option<u64> arg pattern used by
apply_iqn_trunk_gradient and run_nan_checks_post_backward).
Wrapper unused yet — call-site replacement (8 individual check_nan_f32
calls in run_nan_checks_post_backward → single fused launch) lands in
A4.
Audit doc updated (Invariant 7).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds 12-entry u64 ptr table + 12-entry i32 len table on GpuDqnTrainer
for the fused NaN-check kernel. Allocated as zeros; populated once
in next commit (slot pointers stable across trainer lifetime).
Slot 31 (deferred ensemble) entry will be 0 — kernel skips that block.
Two separate buffers chosen over packed-stride layout for direct
ABI match with the kernel signature (const float* const* buf_ptrs,
const int* buf_lens). Avoids stride-alignment concerns.
Unused yet — populate + wrapper + call-site replacement in next commits.
SP1 (F1 NaN root-cause investigation) closes at commit ab2133463 on
plan-c-phase-2-thompson with the surgical-fix-scope deliverables. Three
L40S validation smokes (smoke-test-xvzgk, dr2bn, ckgv8) characterize
both what SP1 CAN address (cold-start clamp pathology) and what it
CANNOT (structural Adam weight pathology — SP3 scope).
What SP1 produced
-----------------
1. Permanent diagnostic infrastructure verified working:
- nan_flags_buf 24→48 with backward-path coverage at slots 24-35.
- run_nan_checks_post_backward + 3 inline checks during backward
orchestration (slots 33/34/35 multi-point bw_d_h_s2 snapshots).
- Sentinel correctly fires on cuBLAS GEMM accumulator overflow.
2. F1 NaN entry kernels pinpointed (slots 26 + 32):
- apply_iqn_trunk_gradient cuBLAS sgemm (slot 26 = iqn_trunk_m)
- Bottleneck Linear backward sgemm (slot 32 = bn_d_concat_buf)
- IQN-internal buffers (slots 27, 28, 35) stayed CLEAN — the IQN
backward chain is NOT the seed; the cuBLAS consumer of clean
inputs produces NaN via accumulator overflow.
3. Cold-start clamp pathology fixed (ε floor formula):
- Original `(1e6 × isv).max(1e3)` clipped F1-startup gradients to
±1e3 when fold-boundary ISV reset put H_S2_RMS_EMA ≈ 0.
- New `1e6 × isv.max(1.0)` guarantees max_abs ≥ 1e6 in all states.
- Validated: F1 NaN moved from step 240 → 3540 (15× later).
What is SP3 territory (NOT fixed in SP1)
-----------------------------------------
F1 NaN at step 3540 has same `[6, 12, 26, 32]` signature with clean
inputs (slots 27, 35). The cuBLAS GEMM produces NaN/Inf because the
WEIGHTS being multiplied have accumulated NaN/Inf via Adam updates
over 3540 training steps. Surgical clamps on gradient inputs cannot
prevent this; root cause is Q-target inflation drives Adam EMA
saturation drives weight pathology drives GEMM overflow.
SP3 scope per spec:
- Target-Q clipping or pessimistic ensemble
- Atom-position growth bounds (C51)
- Conservative target sync at fold boundary
- Adam EMA reset frequency for fold transitions
What is SP2 territory (NOT fixed in SP1)
-----------------------------------------
F0 regression: 35.30 (Phase B+C+ε-fix) vs 55.87 baseline. Three
consecutive smokes show consistent F0 ≈ 35 across post-Phase-B
commits, indicating Phase B instrumentation has real F0 cost.
SP2 framework codification opportunity: consolidate the 11
per-step check_nan_f32 launches into a single fused reduce
kernel scanning all 12 backward-path slots in parallel.
Expected: F0 returns to ~55 baseline, regression sentinel preserved.
Pearls for memory
-----------------
1. ε-floor placement: when an ISV-driven adaptive bound has a cold-start
safety floor, the ε goes on the ISV MULTIPLIER (`isv.max(1.0)`),
not the BOUND (`bound.max(1e3)`). Putting ε on the bound clips
legitimate signal at cold-start; putting it on the multiplier
preserves the wide guard-band intent.
2. NaN diagnostic ordering: when adding sanitization (clamp,
isfinite-or-zero) to a buffer that's also NaN-checked by a
post-backward sentinel, the inline NaN check MUST fire BEFORE
the sanitization. Otherwise the sentinel sees zero'd buffers
and silently dies.
References
----------
- Spec: docs/superpowers/specs/2026-04-29-numerical-stability-investigation-design.md
- Plan: docs/superpowers/plans/2026-04-29-numerical-stability-sp1-f1-nan-root-cause.md
- Audit (durable for SP2/SP3): docs/dqn-backward-nan-audit.md
- Memory entry: ~/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/project_sp1_f1_nan_root_cause_resolved.md
Smoke smoke-test-dr2bn (commit 19b008e1c) F1-NaN'd at step 240 — earlier
than pre-fix smoke smoke-test-xvzgk (step 890). The fix made things
worse, indicating the ε floor `(1e6 × isv).max(1e3)` is actively
destabilizing F1 startup.
Diagnosis: at fold boundary, ISV[H_S2_RMS_EMA_INDEX=96] and
ISV[Q_DIR_ABS_REF_INDEX=21] reset to 0 (per StateResetRegistry).
Formula `(1e6 × 0).max(1e3) = 1e3` makes max_abs aggressively narrow.
F1 startup gradients can have natural magnitudes > 1e3 (post-fold
Bellman-target shift); clamping them to ±1e3 destabilizes Adam EMAs,
which then drive cuBLAS GEMM accumulators into pathological inputs
that overflow → slot 26 + 32 NaN.
New formula: `1e6 × isv.max(1.0)` guarantees max_abs ≥ 1e6 regardless
of ISV state:
- ISV = 0 → max_abs = 1e6 × max(0, 1.0) = 1e6
- ISV = 0.5 → max_abs = 1e6 × max(0.5, 1.0) = 1e6
- ISV = 2.0 → max_abs = 1e6 × max(2.0, 1.0) = 2e6
- ISV = 100 → max_abs = 1e8
F0 no-op intent preserved (F0 inputs ≪ 1e6 in all states; ISV[96]≈1.0
for converged F0 → max_abs = 1e6, well above F0-typical |iqn_d_h_s2|
≤ ~10²). F1 startup gradients ≤ 1e6 are un-clipped.
The 1.0 ε floor is on the ISV multiplier (Invariant 1 carve-out per
`feedback_isv_for_adaptive_bounds`), not on the bound itself — bound
is still ISV-driven when ISV is meaningful (≥ 1.0).
Two edit sites:
- gpu_dqn_trainer.rs:6967 (apply_iqn_trunk_gradient, slot 26 path)
- gpu_dqn_trainer.rs:18631 (launch_cublas_backward_to, slot 32 path)
F0 regression to 35.24 still unsolved — separate investigation thread
within SP1 (no deferral; Phase B instrumentation timing impact suspected).
Smoke smoke-test-dr2bn (commit 19b008e1c) failed all 7 SP1 pass criteria.
F1 first-fire signature at step 240: flagged=[6=grad_buf, 12=save_h_s2,
26=iqn_trunk_m, 32=bn_d_concat_buf] — IDENTICAL to pre-fix smoke-xvzgk
but at step 240 vs 890. The surgical fix at 19b008e1c made F1 NaN
EARLIER, not prevented it.
Diagnosis: the ε floor `(1e6 * isv).max(1e3)` clips legitimate F1
startup gradients to ±1e3 when fold-boundary ISV reset puts ISV[96]
or ISV[21] near 0. Clipped gradients destabilize Adam EMAs → cuBLAS
GEMM accumulator overflow → slot 26 + 32 NaN.
F0 regression unchanged across Phase C (35.24 vs pre-fix 34.55).
Confirms the F0 regression is a Phase B instrumentation side effect,
not Phase C. Separate investigation thread within SP1.
Next iteration: Task 6 fix-up #2 changes ε floor to `(1e6 * isv.max(1.0))`
guaranteeing max_abs ≥ 1e6 regardless of ISV state. Removes cold-start
clamp pathology while preserving F0 paper-review intent.
Quality-review follow-ups to commit 97f1d25f5:
1. CRITICAL: preserve regression sentinel for slots 24, 25, 26, 32.
The post-clamps in apply_iqn_trunk_gradient + launch_cublas_backward_to
ran BEFORE run_nan_checks_post_backward, silently zeroing the buffers
the diagnostic was supposed to observe. Fix: inline check_nan_f32 calls
IMMEDIATELY BEFORE each launch_clamp_finite_f32 — same pattern as the
existing slot 35 check at line 6949. Now the diagnostic fires on NaN
entry; the clamp then sanitizes (preventing propagation but preserving
the flag — flags are sticky/OR'd in dqn_nan_check_f32, never cleared).
2. CRITICAL: correct F0 paper-review line reference + reasoning. The
commit body cited "5.0 norm-clip at line 16747" — that's a memset, not
a clip. Real norm-clip lives at lines 16827-16868. Also tightened the
L2-vs-per-element reasoning: L2 norm <= 5.0 worst-case bounds per-element
|x| <= 5.0 (single-element edge), still several orders of magnitude
below the 1e6 max_abs guard.
3. Inline comment at line 6951 mislabeled the clamp target as 'slot 27
input' — actually clamps the slot 35 buffer (bw_d_h_s2 post-DtoD). Fixed.
4. Articulated pre-clamp rationale: defense-in-depth regression protection
for input buffers (slots 24, 25, 35) against future pathologies that
could create extreme-but-finite inputs. The F1 ep2 NaN was outputs
overflowing finite inputs, but the same fix template covers both
failure modes per feedback_no_partial_refactor.
5. Renamed clamp_finite_f32_kernel -> dqn_clamp_finite_f32_kernel for
consistency with sibling utility kernels (dqn_nan_check_f32,
dqn_zero_kernel, dqn_grad_norm_kernel).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Patches two cuBLAS GEMM backward operations identified by SP1 Phase B
smoke smoke-test-xvzgk (commit f139a63ee) as the F1 ep2 NaN sources:
1. apply_iqn_trunk_gradient (gpu_dqn_trainer.rs:6885+):
- Pre-call: clamp bw_d_h_s2 (the IQN DtoD-overwrite target that feeds
the GRN trunk encoder's cuBLAS Linear_a/Linear_b dW/dX/dB GEMMs;
symmetric with slot 27 source-side check) to ±1e6×ISV[96].
- Post-call: sanitize iqn_trunk_m (slot 26 output) — isfinite-or-zero
+ magnitude clamp; defence-in-depth.
2. launch_cublas_backward_to main backward path:
- Pre-call: clamp d_value_logits_buf + d_adv_logits_buf (slots 24/25
inputs that feed bw.backward_full's dueling/branch backward GEMMs)
to ±1e6×ISV[21].
- Post-call: sanitize bn_d_concat_buf (slot 32 output, gated on
bottleneck_dim > 0).
Both fixes use the new clamp_finite_f32_kernel utility (CUDA — replaces
NaN/Inf with 0 + clamps finite values to ±max_abs) + launch_clamp_finite_f32
(Rust wrapper). Kernel lives in dqn_utility_kernels.cu (already in
build.rs); loaded into the same module as the NaN check kernels in
compile_training_kernels — both call sites land inside captured children
(forward_child for slot 32, aux_child for slot 26) and use only stream-
bound launch_builder, so the kernel is graph-safe.
ISV-driven max_abs bound = 1e6 × isv[<slot>] with 1e3 ε floor for
uninitialized state (Invariant 1 carve-out per
feedback_isv_for_adaptive_bounds). Wide guard band — F0 inputs sit
several orders of magnitude below the threshold (F0 ISV[96] ≈ 1.0
LayerNorm RMS, F0 ISV[21] ≈ 1.0 Q-value scale; F0 |iqn_d_h_s2| ≤ ~10²,
F0 |d_value_logits| ≤ ~10³ post the existing 5.0 norm-clip at
line 16747), so guards are no-ops on F0; F1 overflows trigger clamp.
F0 paper-review pre-smoke confirmed.
Combined RELATED commit per feedback_no_partial_refactor — both kernels
share the same unsafe-pattern (cuBLAS GEMM extreme-intermediate-product
overflow) + the same fix template, so they ship together.
Phase B slots 24-35 remain as permanent diagnostic infrastructure;
will catch any future regression of this NaN class.
SP1 Phase D validation smoke pending.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires per-step NaN checks on backward-path kernel outputs. Coverage
per audit per-slot accessor table (docs/dqn-backward-nan-audit.md
:530-548).
GpuDqnTrainer::run_nan_checks_post_backward (NEW) — fire-once-at-end:
- 24 d_value_logits_buf (post-c51_grad value gradient)
- 25 d_adv_logits_buf (post-c51_grad branch advantage)
- 26 iqn_trunk_m (apply_iqn_trunk_gradient cuBLAS bwd output)
- 27 iqn_d_h_s2_buf (IQN backward dh_s2; arg from FusedTrainingCtx)
- 28 d_branch_logits_buf (IQN production backward; arg from caller)
- 29 cql_d_value_logits (CQL gradient output)
- 30 aux_dh_s2_nb_buf (Aux next-bar backward dh_s2)
- 32 bn_d_concat_buf (Bottleneck Linear backward dy)
Inline checks during backward orchestration:
- 33 bw_d_h_s2 post-main (in launch_cublas_backward_to, after
backward_full + branch concat accum,
BEFORE aux_heads_backward SAXPY)
- 34 bw_d_h_s2 post-aux (in launch_cublas_backward_to, after
aux_heads_backward SAXPY)
- 35 bw_d_h_s2 post-iqn (in apply_iqn_trunk_gradient, after
graph_safe_copy_f32 DtoD overwrite,
BEFORE encoder_backward_chain consumes)
Sequential 33→34→35 fire pattern localises the NaN entry point:
- 33 alone fires → main backward chain (c51 + MSE + branch concat)
- 34 fires after 33-clean → aux SAXPY (aux_heads_backward)
- 35 fires after 34-clean → IQN DtoD or per-sample IQN backward
Slot 31 (ensemble_d_logits_buf) cleanly deferred per Task 2 commit
387335e2b — owner is FusedDqnTraining (different struct); will be
un-deferred when ensemble Phase B saxpy guards are verified.
IQN pointers (slots 27, 28) passed as Option<u64> from FusedTrainingCtx
because GpuDqnTrainer does NOT own GpuIqnHead (Task 3 deviation
finding, audit lines 535-536). None case (when iqn_lambda == 0.0
and gpu_iqn = None) cleanly skips slots 27/28 — semantically honest
"buffer doesn't exist this run" rather than false-clean signal.
Pattern matches apply_iqn_trunk_gradient(iqn_d_h_s2_ptr: u64, ...)
at gpu_dqn_trainer.rs:6843.
Call sites (atomic — feedback_no_partial_refactor):
- fused_training.rs:1519 ungraphed step path
- fused_training.rs:2324 capture_training_graph closure (post_aux child)
- gpu_dqn_trainer.rs::launch_cublas_backward_to (slots 33, 34 — captured
in forward child via submit_forward_ops_main)
- gpu_dqn_trainer.rs::apply_iqn_trunk_gradient (slot 35 — captured in
aux child via submit_aux_ops)
Each check uses existing check_nan_f32(buf_ptr, len, flag_idx) —
single-block GPU reduce, no atomicAdd, no DtoD/HtoD/HtoH copies, no
per-step DtoH. Lengths use CudaSlice::len() where possible (auto-syncs
with allocator padding); inline arithmetic for slot 28 since
d_branch_logits_buf.len() is private to GpuIqnHead. Flags accumulate
within fold; reset at fold boundary via reset_nan_flags(). Readback
flow (commit d1808df14) consumes them in BOTH halt_nan and
halt_grad_collapse paths via name tables annotated in commit 387335e2b.
Permanent diagnostic infrastructure — stays as production-grade
regression sentinel after the surgical fix lands.
Adds 5 new pub(crate) accessor methods exposing backward-path buffer
device pointers for Task 4's per-step NaN checks (slots 24, 25, 28,
29, 30 per audit per-slot table at docs/dqn-backward-nan-audit.md
:530-548):
GpuDqnTrainer (4 new):
- d_value_logits_buf_ptr (slot 24 — post-c51_grad value gradient)
- d_adv_logits_buf_ptr (slot 25 — post-c51_grad branch advantage)
- cql_d_value_logits_ptr (slot 29 — CQL gradient output)
- aux_dh_s2_nb_buf_ptr (slot 30 — aux next-bar backward dh_s2)
GpuIqnHead (1 new):
- d_branch_logits_buf_ptr (slot 28 — production IQN backward output,
iqn_quantile_huber_loss)
Slot 27 (iqn_d_h_s2_buf) reuses existing GpuIqnHead::d_h_s2_raw_ptr()
at gpu_iqn_head.rs:1660 — no new method per feedback_no_legacy_aliases:
the existing accessor is already public and sufficient; renaming +
chasing the single call site adds churn without value.
Slots 26, 32, 33-35 reuse pre-existing handles:
- 26: self.ptrs.iqn_trunk_m
- 32: self.bn_d_concat_buf() (existing, returns &CudaSlice<f32>)
- 33-35: self.ptrs.bw_d_h_s2 (3 different Task 4 call sites)
Slot 31 (ensemble_d_logits_buf) deferred per Task 2 commit 387335e2b
(cross-struct on FusedDqnTraining).
DEVIATION FROM PLAN: the plan called for "delegate accessors on
GpuDqnTrainer for slots 27/28" — structurally invalid because
GpuDqnTrainer does NOT own GpuIqnHead. The IQN head is owned by
FusedTrainingCtx (fused_training.rs:289) alongside the trainer at
line 234. The audit's per-slot accessor table (lines 535-536) is
correct: accessors land on GpuIqnHead. Task 4's
run_nan_checks_post_backward will receive IQN pointers as u64
arguments from the FusedTrainingCtx call site — same pattern already
in use at gpu_dqn_trainer.rs:6843
(apply_iqn_trunk_gradient(&mut self, iqn_d_h_s2_ptr: u64, ...)).
NO new scratch buffer added — the plan's bw_d_h_s2_pre_saxpy scratch
+ DtoD-copy approach was superseded by the audit's 3-call-site
reformulation. Slots 33/34/35 are post-main / post-aux / post-iqn
snapshots of the same bw_d_h_s2 (one buffer, three Task 4 invocations).
Pattern follows commit e9096c7be's GRN-block accessors (concise
pub(crate) fn name_ptr(&self) -> u64 with doc-comment referencing
slot number + audit doc + buffer semantics). Additive — no behavioral
change; new accessors consumed by Task 4's NaN check call sites.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Quality-review follow-ups to commit 53bc0bc50 (nan_flags_buf 24->48):
1. Update run_nan_checks_post_forward docstring (gpu_dqn_trainer.rs:14842-14873):
replace 24-slot map with 48-slot range summary + audit-doc pointer.
Was actively misleading after the 24->48 expansion; partial-refactor
residue per feedback_no_partial_refactor.
2. Update '[24] system' comment in training_loop.rs (around line 2044):
reflect the 48-slot post-expansion state (slots 0-23 fwd, 24-35 bwd,
36-47 reserved). Also fix stale '0..11' tracing message to '0..47'.
3. Slot 31 (ensemble_d_logits_buf) annotation: flag DEFERRED + owner
on FusedDqnTraining (different struct than 24-30, 32-35). Prevents
Task 4 from blanket-launching check_nan_f32 on slot 31's null
accessor.
4. Both name-table header comments now reference the future
run_nan_checks_post_backward method (Task 4) plus the audit's
per-slot table — pre-empts contract drift when Task 4 lands a
3-way name-table dependency.
Audit-doc entry appended to docs/dqn-wire-up-audit.md SP1 Phase B
section. No behavioral change. Both name tables remain byte-identical.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Expands the NaN flag buffer from 24 to 48 slots to make room for
backward-path NaN checks (slots 24-35 per audit doc per-slot table)
plus 12 reserved headroom slots (36-47) for SP2 framework + SP3
observer hooks.
Touches:
- gpu_dqn_trainer.rs: alloc size 24→48 (line ~11178); read_nan_flags
signature [i32; 24]→[i32; 48] (line ~14982); field docstring updated
(line ~2658) to reflect 48-slot layout
- fused_training.rs: pub(crate) read_nan_flags signature [i32; 24]→[i32; 48]
- training_loop.rs: BOTH name table sites (halt_nan + halt_grad_collapse
block from commit d1808df14) updated to 48 entries
Slot names use audit allocation (docs/dqn-backward-nan-audit.md per-slot
table), which supersedes the plan's placeholder names per the audit's
plan-supersession note. Slots 24-35 cover production buffers:
d_value_logits_buf, d_adv_logits_buf, iqn_trunk_m, iqn_d_h_s2_ptr,
d_branch_logits_buf, cql_d_value_logits, aux_dh_s2_nb_buf,
ensemble_d_logits_buf, bn_d_concat_buf, bw_d_h_s2 (×3 call sites).
Slots 36-47 are rsv36-rsv47 (headroom).
No behavioral change — new slots stay at zero until Task 4 wires the
kernel-output NaN checks. Buffer size reviewable by SP2.
