foxhunt/docs/superpowers/notes/2026-06-02-determinism-phase2.5-rl-loop-investigation.md

Determinism Phase 2.5 — RL-loop state investigation

Date: 2026-06-02 Dispatch: Phase 2.5 follow-up to Phase 2.4's mamba2 falsification. Outcome: ALL 6 candidate groups (CfC carry, PER state, action RNG, LOB sim, ISV controllers, reward shaping) falsified at step 2 dump. The residual non-determinism lives in step 2's BACKWARD PIPELINE, with identical forward activations but divergent backward outputs. Per dispatch STOP rule: no fix applied — culprit is OUTSIDE the 6 candidate groups. Phase 2.6 dispatch needed.

---

Recap (Phase 2.4 → Phase 2.5)

Phase 2.4 falsified mamba2 / cuBLAS / trunk backward — 52 buffers EQUAL at step 1's dump (post-step-1 state). At step 2's dump the mamba2 L1/L2 weights diverged but the upstream RL-loop state had not been instrumented. Phase 2.5's dispatch asked: which of the 6 not-yet-dumped RL-loop candidates is the culprit?

The 6 candidate groups:

• A. CfC recurrent cross-step carry (attn_context_d, h_t_d)
• B. PER replay state (priority_tree_d, sample_prng_d, sample_indices_d)
• C. Action sampling RNG (prng_state_d, iqn_prng_state_d)
• D. LOB simulator (pyramid_units_count_d, unit_*_d, prev_realized_pnl_d, prev_position_lots_d, steps_since_done_d, trade_duration_emit_d, close_unit_index_d, unit_prev_pos_lots_d, unit_initial_r_d, unit_trail_distance_d)
• E. ISV controllers (full mapped-pinned ISV array — 757 floats)
• F. Reward shaping outputs (rewards_d, raw_rewards_d, outcome_ema_d, reward_abs_d, dones_d, actions_d, next_actions_d, log_pi_old_d, advantages_d, returns_d,
  • CMDP + edge-decay per-batch: session_pnl, consec_loss, cooldown_remaining, ph_mu/_count/_m/_mmin/_stat)

A and B were already dumped by dump_mamba2_state_for_debug (attn_context, h_t) and dump_per_state_for_debug (priority_tree, sample_prng, sample_indices). Phase 2.5 added C/D/E/F via a new sibling method IntegratedTrainer::dump_rl_state_for_debug (env-gated by FOXHUNT_DETERMINISM_DEBUG_RL=1), plus a matching scripts/compare-rl-state.py.

Task 1 — extended dump infrastructure

Added IntegratedTrainer::dump_rl_state_for_debug(step) invoked from build_diag_value for step ∈ {0,1,2,3}. Captures 35 new device buffers via DtoH copies on self.stream, plus the full mapped-pinned ISV array (757 f32, read directly host-side). Files written as raw little-endian binaries to $FOXHUNT_DEBUG_DUMP_DIR.

scripts/compare-rl-state.py reads run_a/ vs run_b/ dumps and emits per-group verdict (EQUAL/DIVERGE) with first-divergent index + magnitude. Naming convention matches compare-mamba2-state.py and compare-per-state.py exactly.

scripts/determinism-check.sh gained a new --debug-dump-rl flag that sets FOXHUNT_DETERMINISM_DEBUG_RL=1 for both runs and invokes compare-rl-state.py after the runs finish. Composes with the existing --debug-dump and --debug-dump-mamba2 flags — running all three in one go gives the full picture (PER + mamba2 + RL state).

Task 3 — run + verdict

Triple-dump invocation:

FOXHUNT_DETERMINISTIC=1 ./scripts/determinism-check.sh \
  --quick --debug-dump --debug-dump-mamba2 --debug-dump-rl

Per-group verdict at step ∈ {0, 1, 2}

Group	Buffers	Step 0	Step 1	Step 2
A — CfC carry / h_t	attn_context, h_t	EQUAL	EQUAL	EQUAL
B — PER state	prng, tree, indices	EQUAL	EQUAL	EQUAL
C — Action / IQN RNG	action_prng_state, iqn_prng_state	EQUAL	EQUAL	EQUAL
D — LOB sim	13 buffers (units, pos, steps...)	EQUAL	EQUAL	EQUAL
E — ISV controllers	full 757-float ISV	EQUAL	EQUAL	EQUAL
F — Reward shaping	18 buffers (rewards, advantages, actions, ...)	EQUAL	EQUAL	EQUAL
mamba2 forward activations	vsn_out, mamba2_l{1,2}_h_enriched_seq, ln_{a,b}_out, attn_context, h_t	EQUAL	EQUAL	EQUAL
mamba2 / trunk weights	18 mamba2 + 6 trunk	EQUAL	EQUAL	DIVERGE
mamba2 / trunk gradients	13 mamba2 bwd outputs + 6 trunk grads	EQUAL	EQUAL	DIVERGE

Step 3 (consequence-of-step-2 divergence)

Once step 2's weights diverge, step 3's forward (using divergent weights) produces divergent activations, divergent log_pi_old at step 3 dump, and propagates to ISV at step 3 dump (slot 407 = q_pi_agree EMA, slot 419 = KL EMA, slot 421 = adv_var_ratio EMA — all derived from step-3's logits).

Decisive evidence — step 2 dump

=== step 2 ===
  vsn_out:                          EQUAL  (229376 elements)
  mamba2_l1_h_enriched_seq:         EQUAL  (524288 elements)
  ln_a_out:                         EQUAL  (524288 elements)
  mamba2_l2_h_enriched_seq:         EQUAL  (524288 elements)
  ln_b_out:                         EQUAL  (524288 elements)
  attn_context:                     EQUAL  (16384 elements)
  h_t:                              EQUAL  (16384 elements)

  mamba2_l1_w_in:                   DIVERGE  Δ ~ 1.6e-6
  mamba2_l1_b_in:                   DIVERGE  Δ ~ 7.6e-7
  ... (all 18 mamba2 weights diverge)
  vsn_w:                            DIVERGE  Δ ~ 1.2e-6
  attn_q:                           DIVERGE  Δ ~ 9.0e-7
  cfc_w_in / cfc_w_rec / ln_*_gain: DIVERGE

  mamba2_l1_d_w_c_per_sample:       DIVERGE  Δ ~ 1.6e-7
  mamba2_l1_dw_c_reduced:           DIVERGE  Δ ~ 9.1e-6
  mamba2_l1_d_h_s2:                 DIVERGE  Δ ~ 2.9e-7
  mamba2_l1_d_a_per_channel:        DIVERGE  Δ ~ 3.4e-10
  mamba2_l1_d_b_per_channel:        DIVERGE  Δ ~ 7.1e-9
  mamba2_l1_d_a_proj_2d:            DIVERGE  Δ ~ 2.0e-8
  mamba2_l1_d_b_proj_2d:            DIVERGE  Δ ~ 1.2e-6
  mamba2_l1_dw_in:                  DIVERGE  Δ ~ 3.6e-6
  mamba2_l1_dw_a:                   DIVERGE  Δ ~ 4.5e-6
  mamba2_l1_dw_b:                   DIVERGE  Δ ~ 2.4e-5
  mamba2_l1_db_in / db_a / db_b:    DIVERGE  Δ up to 1.7e-4
  mamba2_l1_d_x / d_x_from_in:      DIVERGE  Δ ~ 5.5e-7
  grad_vsn_w:                       DIVERGE  Δ ~ 2.0e-5
  grad_attn_q:                      DIVERGE  Δ ~ 8.6e-6
  grad_cfc_w_in / w_rec:            DIVERGE
  grad_ln_a_gain / ln_b_gain:       DIVERGE

  action_prng_state:                EQUAL  (128 elements)
  iqn_prng_state:                   EQUAL  (128 elements)
  pyramid_units_count:              EQUAL  (128 elements)
  unit_{entry_step,entry_price,lots,active,initial_r,trail_distance,prev_pos_lots}: EQUAL
  close_unit_index:                 EQUAL
  prev_realized_pnl:                EQUAL
  prev_position_lots:               EQUAL
  steps_since_done:                 EQUAL
  trade_duration_emit:              EQUAL
  isv_full:                         EQUAL  (757 elements)
  rewards / raw_rewards / outcome_ema / reward_abs: EQUAL
  dones / actions / next_actions:   EQUAL
  log_pi_old:                       EQUAL  (128 elements)
  advantages:                       EQUAL  (128 elements)
  returns:                          EQUAL  (128 elements)
  session_pnl / consec_loss:        EQUAL
  cooldown_remaining:               EQUAL
  ph_mu / ph_count / ph_m / ph_mmin / ph_stat: EQUAL

Eval consequence

Same seed, same FOXHUNT_DETERMINISTIC=1 PEDANTIC:

Run	eval pnl_usd	max_drawdown_usd	trades	win_rate	sharpe
A	+$187,087.50	$191,412.50	355	0.327	+1.63
B	-$261,425.00	$363,750.00	430	0.291	-1.83

Eval pnl differs by ~$450k between same-seed runs. Same root cause as Phase 2.4's $184k drift — the tiny step-2 weight divergence cascades through 247 RL steps × 50 eval steps × non-linear LOB sim dynamics into a trajectory-level chasm.

Interpretation — bug is in step 2's BACKWARD KERNEL CHAIN

The step 2 evidence is decisive:

1. Forward at step 2 is bit-deterministic (weights at step 2 start are EQUAL per Phase 2.4 step 1 dump; all forward activations match bit-exactly at step 2 dump).
2. All loss-input scalars to backward are bit-deterministic at step 2 dump:
   • log_pi_old, advantages, returns (saved during step 2 forward)
   • actions, next_actions, rewards, dones (from RL env)
   • PER sample_indices, priority_tree, prng (PER state EQUAL)
   • action RNG state, IQN RNG state (prng_state, iqn_prng_state)
   • LOB sim state (positions, pyramid, units, balance)
   • ISV controller state (popart, kelly, wr_ema, edge-decay PH — all 757 slots EQUAL)
3. Yet step 2's BACKWARD OUTPUTS diverge: every reduced gradient (grad_vsn_w, grad_attn_q, grad_cfc_*, grad_ln_*), every mamba2 backward scratch (d_w_c_per_sample, d_h_s2, d_a_per_channel, d_b_per_channel), every cuBLAS GEMM output in the mamba2 backward (dw_in, dw_a, dw_b).

For backward to differ with identical forward + identical loss-inputs, some kernel in the gradient-emission chain must be non-deterministic. Phase 2.3 already applied CUBLAS_PEDANTIC_MATH to all 3 cuBLAS sites (mamba2_block, dqn, iqn) so PEDANTIC GEMMs are NOT the cause.

Candidates NOT yet investigated (Phase 2.6 scope)

The actual culprit lives in one of:

1. PER replay sample during step 2 K-loop — at k_iter=0 the sample produces sampled_h_t_d / sampled_actions_d / sampled_rewards_d / sampled_log_pi_old_d / sampled_dones_d. These were NOT dumped this dispatch (they're overwritten each k_iter — the dump captures only the LAST k_iter's output).
2. Head backward kernels (Q / π / V / IQN / Dueling-Q) writing grad_h_t — any non-deterministic accumulation across batch + atom
   • tau dimensions. NOT dumped this dispatch.
3. grad_h_accumulate combining per-head grad_h's into the encoder accumulator (with λ scaling). If it has a non-deterministic parallel-add path... NOT dumped this dispatch.
4. Smoothness backward running upstream of Loop 1 in dispatch_train_step (heads' BCE / aux portion). NOT dumped.
5. Aux head + trunk backward — separate K-loop. NOT dumped.
6. Encoder backward K-loop reductions — reduce_axis0 on per-batch param-grad scratches (cfc_w_in/_rec, attn_q, vsn_w, LN_a/b gain/bias). The K-loop accumulates += per-K then reduces over B. If reduce_axis0 has a parallel reduction with non-deterministic ordering, it would explain why ALL trunk grads diverge simultaneously. NOT dumped this dispatch.

The strongest candidate by pattern-match (Phase 2's PER tree-rebuild fix prototype) is reduce_axis0 or a head backward kernel's batch reduction. Multiple grads diverging SIMULTANEOUSLY (not cascading through one upstream source) is most easily explained by ONE shared reducer that all of them go through.

Phase 2.5 STOPS per dispatch instruction

Per dispatch: "If Task 3 finds the culprit is something OTHER than the 6 candidates, STOP — Phase 2.6 dispatch territory." Applies here: the bug is in the gradient-emission chain, not in any of the 6 RL-loop candidate groups. Phase 2.6 must dump:

• sampled_h_t_d, sampled_h_tp1_d, sampled_actions_d, sampled_rewards_d, sampled_dones_d, sampled_log_pi_old_d — captured BEFORE the K-loop's heads forward, with one dump per k_iter (not just last).
• Head outputs: q_logits_d, pi_logits_d, v_pred_d, iqn_q_values_d, dueling_q_composed_d (immediately after head forward, BEFORE backward).
• Pre-reduce per-batch param-grad scratches: vsn_grad_w_scratch_d, cfc_grad_w_in_scratch_d, attn_grad_q_scratch_d, ln_*_grad_*_scratch_d (catches non-determinism IN reduce_axis0).
• grad_h_t per-head outputs and grad_h_t_combined_d (head-grad accumulator before encoder backward).

If pre-reduce per-batch scratches match but post-reduce grads diverge → reduce_axis0 is the culprit (Phase 2 PER-rebuild fix pattern applies). If pre-reduce already diverges → individual head backward kernel is non-deterministic.

Files touched (Phase 2.5, uncommitted)

Path	Change	LOC
crates/ml-alpha/src/trainer/integrated.rs	Added dump_rl_state_for_debug method (env-gated by FOXHUNT_DETERMINISM_DEBUG_RL=1) + invocation in build_diag_value	+290
scripts/determinism-check.sh	Added --debug-dump-rl flag, composes with existing PER/mamba2 dump flags	+25
scripts/compare-rl-state.py	New: bytewise compare for 35 RL-loop buffers + full ISV, grouped by candidate class (C/D/E/F) with verdict logic	+266 (new file)

No .cu files added/changed. No commit. Phase 2.6 dispatch should reuse the dump infrastructure and add the missing 4 buffer classes listed above.

Discipline checklist

• No new files in repo root
• Scripts in scripts/, docs in docs/superpowers/notes/
• No .cu files added/changed (bug is NOT in any CUDA kernel we've identified yet)
• Per feedback_no_atomicadd.md: not relevant
• Per feedback_no_nvrtc.md: not relevant
• Per feedback_cpu_is_read_only.md: dump method does DtoH only; mapped-pinned ISV is read-only on host
• Per feedback_no_stubs.md: dump wired end-to-end via build_diag_value; no-op when env var unset
• Per feedback_local_smoke_before_cluster.md: validated on RTX 3050, no cluster submit
• Per feedback_systematic_debugging + dispatch STOP rule: no fix applied — diagnosis only, culprit outside 6 candidates
• Per feedback_no_quickfixes.md: did NOT apply a tolerance-mask / f64-accumulator fix despite the temptation
• No commit — left uncommitted for human review

Reproduction

# Build with extended RL-state dump
SQLX_OFFLINE=true cargo build --release --example alpha_rl_train -p ml-alpha

# Triple-dump: PER + mamba2 + RL state
FOXHUNT_DETERMINISTIC=1 ./scripts/determinism-check.sh --quick \
  --debug-dump --debug-dump-mamba2 --debug-dump-rl

# Expected output:
#   [debug-dump]        all 3 PER buffers EQUAL through step 2
#   [debug-dump-mamba2] step 1 ALL 52 buffers EQUAL
#                       step 2 ALL forward activations EQUAL
#                       step 2 ALL weights + grads + backward outputs DIVERGE
#   [debug-dump-rl]     ALL 35 RL-loop buffers + 757-float ISV EQUAL through step 2
#   FIRST_DIVERGENCE: step=2 leaf=checksums.{attn_q|vsn_w|mamba2_grad_w_c_l1}

# Per-leaf interpretation: see scripts/compare-rl-state.py output's
# "Per-group culprit table" and "Next-step interpretation" hints.