# Determinism Phase 2.5 — RL-loop state investigation

**Date**: 2026-06-02
**Dispatch**: Phase 2.5 follow-up to Phase 2.4's mamba2 falsification.
**Outcome**: ALL 6 candidate groups (CfC carry, PER state, action RNG, LOB
sim, ISV controllers, reward shaping) **falsified at step 2 dump**. The
residual non-determinism lives in **step 2's BACKWARD PIPELINE**, with
identical forward activations but divergent backward outputs.
**Per dispatch STOP rule**: no fix applied — culprit is OUTSIDE the 6
candidate groups. Phase 2.6 dispatch needed.

---

## Recap (Phase 2.4 → Phase 2.5)

Phase 2.4 falsified mamba2 / cuBLAS / trunk backward — 52 buffers EQUAL
at step 1's dump (post-step-1 state). At step 2's dump the mamba2 L1/L2
weights diverged but the upstream RL-loop state had not been
instrumented. Phase 2.5's dispatch asked: which of the 6 not-yet-dumped
RL-loop candidates is the culprit?

The 6 candidate groups:
- **A**. CfC recurrent cross-step carry (`attn_context_d`, `h_t_d`)
- **B**. PER replay state (`priority_tree_d`, `sample_prng_d`, `sample_indices_d`)
- **C**. Action sampling RNG (`prng_state_d`, `iqn_prng_state_d`)
- **D**. LOB simulator (`pyramid_units_count_d`, `unit_*_d`,
  `prev_realized_pnl_d`, `prev_position_lots_d`, `steps_since_done_d`,
  `trade_duration_emit_d`, `close_unit_index_d`, `unit_prev_pos_lots_d`,
  `unit_initial_r_d`, `unit_trail_distance_d`)
- **E**. ISV controllers (full mapped-pinned ISV array — 757 floats)
- **F**. Reward shaping outputs (`rewards_d`, `raw_rewards_d`,
  `outcome_ema_d`, `reward_abs_d`, `dones_d`, `actions_d`,
  `next_actions_d`, `log_pi_old_d`, `advantages_d`, `returns_d`,
  + CMDP + edge-decay per-batch: `session_pnl`, `consec_loss`,
  `cooldown_remaining`, `ph_mu/_count/_m/_mmin/_stat`)

A and B were already dumped by `dump_mamba2_state_for_debug` (attn_context,
h_t) and `dump_per_state_for_debug` (priority_tree, sample_prng,
sample_indices). Phase 2.5 added C/D/E/F via a new sibling method
`IntegratedTrainer::dump_rl_state_for_debug` (env-gated by
`FOXHUNT_DETERMINISM_DEBUG_RL=1`), plus a matching `scripts/compare-rl-state.py`.

## Task 1 — extended dump infrastructure

Added `IntegratedTrainer::dump_rl_state_for_debug(step)` invoked from
`build_diag_value` for `step ∈ {0,1,2,3}`. Captures 35 new device
buffers via DtoH copies on `self.stream`, plus the full mapped-pinned
ISV array (757 f32, read directly host-side). Files written as raw
little-endian binaries to `$FOXHUNT_DEBUG_DUMP_DIR`.

`scripts/compare-rl-state.py` reads run_a/ vs run_b/ dumps and emits
per-group verdict (EQUAL/DIVERGE) with first-divergent index +
magnitude. Naming convention matches `compare-mamba2-state.py` and
`compare-per-state.py` exactly.

`scripts/determinism-check.sh` gained a new `--debug-dump-rl` flag that
sets `FOXHUNT_DETERMINISM_DEBUG_RL=1` for both runs and invokes
`compare-rl-state.py` after the runs finish. Composes with the existing
`--debug-dump` and `--debug-dump-mamba2` flags — running all three in one
go gives the full picture (PER + mamba2 + RL state).

## Task 3 — run + verdict

Triple-dump invocation:
```bash
FOXHUNT_DETERMINISTIC=1 ./scripts/determinism-check.sh \
  --quick --debug-dump --debug-dump-mamba2 --debug-dump-rl
```

### Per-group verdict at step ∈ {0, 1, 2}

| Group | Buffers | Step 0 | Step 1 | Step 2 |
|-------|---------|:------:|:------:|:------:|
| A — CfC carry / h_t | `attn_context`, `h_t` | EQUAL | EQUAL | EQUAL |
| B — PER state | `prng`, `tree`, `indices` | EQUAL | EQUAL | EQUAL |
| C — Action / IQN RNG | `action_prng_state`, `iqn_prng_state` | EQUAL | EQUAL | EQUAL |
| D — LOB sim | 13 buffers (units, pos, steps...) | EQUAL | EQUAL | EQUAL |
| E — ISV controllers | full 757-float ISV | EQUAL | EQUAL | EQUAL |
| F — Reward shaping | 18 buffers (rewards, advantages, actions, ...) | EQUAL | EQUAL | EQUAL |
| **mamba2 forward activations** | `vsn_out`, `mamba2_l{1,2}_h_enriched_seq`, `ln_{a,b}_out`, `attn_context`, `h_t` | EQUAL | EQUAL | **EQUAL** |
| **mamba2 / trunk weights** | 18 mamba2 + 6 trunk | EQUAL | EQUAL | **DIVERGE** |
| **mamba2 / trunk gradients** | 13 mamba2 bwd outputs + 6 trunk grads | EQUAL | EQUAL | **DIVERGE** |

### Step 3 (consequence-of-step-2 divergence)

Once step 2's weights diverge, step 3's forward (using divergent
weights) produces divergent activations, divergent log_pi_old at step 3
dump, and propagates to ISV at step 3 dump (slot 407 = q_pi_agree EMA,
slot 419 = KL EMA, slot 421 = adv_var_ratio EMA — all derived from
step-3's logits).

### Decisive evidence — step 2 dump

```
=== step 2 ===
  vsn_out:                          EQUAL  (229376 elements)
  mamba2_l1_h_enriched_seq:         EQUAL  (524288 elements)
  ln_a_out:                         EQUAL  (524288 elements)
  mamba2_l2_h_enriched_seq:         EQUAL  (524288 elements)
  ln_b_out:                         EQUAL  (524288 elements)
  attn_context:                     EQUAL  (16384 elements)
  h_t:                              EQUAL  (16384 elements)

  mamba2_l1_w_in:                   DIVERGE  Δ ~ 1.6e-6
  mamba2_l1_b_in:                   DIVERGE  Δ ~ 7.6e-7
  ... (all 18 mamba2 weights diverge)
  vsn_w:                            DIVERGE  Δ ~ 1.2e-6
  attn_q:                           DIVERGE  Δ ~ 9.0e-7
  cfc_w_in / cfc_w_rec / ln_*_gain: DIVERGE

  mamba2_l1_d_w_c_per_sample:       DIVERGE  Δ ~ 1.6e-7
  mamba2_l1_dw_c_reduced:           DIVERGE  Δ ~ 9.1e-6
  mamba2_l1_d_h_s2:                 DIVERGE  Δ ~ 2.9e-7
  mamba2_l1_d_a_per_channel:        DIVERGE  Δ ~ 3.4e-10
  mamba2_l1_d_b_per_channel:        DIVERGE  Δ ~ 7.1e-9
  mamba2_l1_d_a_proj_2d:            DIVERGE  Δ ~ 2.0e-8
  mamba2_l1_d_b_proj_2d:            DIVERGE  Δ ~ 1.2e-6
  mamba2_l1_dw_in:                  DIVERGE  Δ ~ 3.6e-6
  mamba2_l1_dw_a:                   DIVERGE  Δ ~ 4.5e-6
  mamba2_l1_dw_b:                   DIVERGE  Δ ~ 2.4e-5
  mamba2_l1_db_in / db_a / db_b:    DIVERGE  Δ up to 1.7e-4
  mamba2_l1_d_x / d_x_from_in:      DIVERGE  Δ ~ 5.5e-7
  grad_vsn_w:                       DIVERGE  Δ ~ 2.0e-5
  grad_attn_q:                      DIVERGE  Δ ~ 8.6e-6
  grad_cfc_w_in / w_rec:            DIVERGE
  grad_ln_a_gain / ln_b_gain:       DIVERGE

  action_prng_state:                EQUAL  (128 elements)
  iqn_prng_state:                   EQUAL  (128 elements)
  pyramid_units_count:              EQUAL  (128 elements)
  unit_{entry_step,entry_price,lots,active,initial_r,trail_distance,prev_pos_lots}: EQUAL
  close_unit_index:                 EQUAL
  prev_realized_pnl:                EQUAL
  prev_position_lots:               EQUAL
  steps_since_done:                 EQUAL
  trade_duration_emit:              EQUAL
  isv_full:                         EQUAL  (757 elements)
  rewards / raw_rewards / outcome_ema / reward_abs: EQUAL
  dones / actions / next_actions:   EQUAL
  log_pi_old:                       EQUAL  (128 elements)
  advantages:                       EQUAL  (128 elements)
  returns:                          EQUAL  (128 elements)
  session_pnl / consec_loss:        EQUAL
  cooldown_remaining:               EQUAL
  ph_mu / ph_count / ph_m / ph_mmin / ph_stat: EQUAL
```

### Eval consequence

Same seed, same FOXHUNT_DETERMINISTIC=1 PEDANTIC:

| Run | eval pnl_usd | max_drawdown_usd | trades | win_rate | sharpe |
|-----|-------------:|-----------------:|-------:|---------:|-------:|
| A | +$187,087.50 | $191,412.50 | 355 | 0.327 | +1.63 |
| B | -$261,425.00 | $363,750.00 | 430 | 0.291 | -1.83 |

Eval pnl differs by ~$450k between same-seed runs. Same root cause as
Phase 2.4's $184k drift — the tiny step-2 weight divergence cascades
through 247 RL steps × 50 eval steps × non-linear LOB sim dynamics into
a trajectory-level chasm.

## Interpretation — bug is in step 2's BACKWARD KERNEL CHAIN

The step 2 evidence is decisive:

1. **Forward at step 2 is bit-deterministic** (weights at step 2 start
   are EQUAL per Phase 2.4 step 1 dump; all forward activations match
   bit-exactly at step 2 dump).
2. **All loss-input scalars to backward are bit-deterministic** at
   step 2 dump:
   - `log_pi_old`, `advantages`, `returns` (saved during step 2 forward)
   - `actions`, `next_actions`, `rewards`, `dones` (from RL env)
   - PER `sample_indices`, `priority_tree`, `prng` (PER state EQUAL)
   - action RNG state, IQN RNG state (`prng_state`, `iqn_prng_state`)
   - LOB sim state (positions, pyramid, units, balance)
   - ISV controller state (popart, kelly, wr_ema, edge-decay PH —
     all 757 slots EQUAL)
3. **Yet step 2's BACKWARD OUTPUTS diverge**: every reduced gradient
   (`grad_vsn_w`, `grad_attn_q`, `grad_cfc_*`, `grad_ln_*`), every
   mamba2 backward scratch (`d_w_c_per_sample`, `d_h_s2`,
   `d_a_per_channel`, `d_b_per_channel`), every cuBLAS GEMM output in
   the mamba2 backward (`dw_in`, `dw_a`, `dw_b`).

For backward to differ with identical forward + identical loss-inputs,
some kernel in the gradient-emission chain must be non-deterministic.
Phase 2.3 already applied `CUBLAS_PEDANTIC_MATH` to all 3 cuBLAS sites
(mamba2_block, dqn, iqn) so PEDANTIC GEMMs are NOT the cause.

## Candidates NOT yet investigated (Phase 2.6 scope)

The actual culprit lives in one of:

1. **PER replay sample during step 2 K-loop** — at k_iter=0 the sample
   produces `sampled_h_t_d` / `sampled_actions_d` / `sampled_rewards_d`
   / `sampled_log_pi_old_d` / `sampled_dones_d`. **These were NOT
   dumped this dispatch** (they're overwritten each k_iter — the dump
   captures only the LAST k_iter's output).
2. **Head backward kernels** (Q / π / V / IQN / Dueling-Q) writing
   `grad_h_t` — any non-deterministic accumulation across batch + atom
   + tau dimensions. NOT dumped this dispatch.
3. **`grad_h_accumulate`** combining per-head grad_h's into the encoder
   accumulator (with λ scaling). If it has a non-deterministic
   parallel-add path... NOT dumped this dispatch.
4. **Smoothness backward** running upstream of Loop 1 in
   `dispatch_train_step` (heads' BCE / aux portion). NOT dumped.
5. **Aux head + trunk backward** — separate K-loop. NOT dumped.
6. **Encoder backward K-loop reductions** — `reduce_axis0` on
   per-batch param-grad scratches (cfc_w_in/_rec, attn_q, vsn_w, LN_a/b
   gain/bias). The K-loop accumulates `+=` per-K then reduces over B.
   If reduce_axis0 has a parallel reduction with non-deterministic
   ordering, it would explain why ALL trunk grads diverge simultaneously.
   NOT dumped this dispatch.

The strongest candidate by pattern-match (Phase 2's PER tree-rebuild
fix prototype) is **reduce_axis0** or a head backward kernel's batch
reduction. Multiple grads diverging SIMULTANEOUSLY (not cascading
through one upstream source) is most easily explained by ONE shared
reducer that all of them go through.

## Phase 2.5 STOPS per dispatch instruction

Per dispatch: *"If Task 3 finds the culprit is something OTHER than the
6 candidates, STOP — Phase 2.6 dispatch territory."* Applies here: the
bug is in the gradient-emission chain, not in any of the 6 RL-loop
candidate groups. Phase 2.6 must dump:

- `sampled_h_t_d`, `sampled_h_tp1_d`, `sampled_actions_d`,
  `sampled_rewards_d`, `sampled_dones_d`, `sampled_log_pi_old_d` —
  captured BEFORE the K-loop's heads forward, with one dump per
  k_iter (not just last).
- Head outputs: `q_logits_d`, `pi_logits_d`, `v_pred_d`,
  `iqn_q_values_d`, `dueling_q_composed_d` (immediately after head
  forward, BEFORE backward).
- Pre-reduce per-batch param-grad scratches: `vsn_grad_w_scratch_d`,
  `cfc_grad_w_in_scratch_d`, `attn_grad_q_scratch_d`,
  `ln_*_grad_*_scratch_d` (catches non-determinism IN reduce_axis0).
- `grad_h_t` per-head outputs and `grad_h_t_combined_d` (head-grad
  accumulator before encoder backward).

If pre-reduce per-batch scratches match but post-reduce grads diverge
→ reduce_axis0 is the culprit (Phase 2 PER-rebuild fix pattern applies).
If pre-reduce already diverges → individual head backward kernel is
non-deterministic.

## Files touched (Phase 2.5, uncommitted)

| Path | Change | LOC |
|------|--------|-----|
| `crates/ml-alpha/src/trainer/integrated.rs` | Added `dump_rl_state_for_debug` method (env-gated by `FOXHUNT_DETERMINISM_DEBUG_RL=1`) + invocation in `build_diag_value` | +290 |
| `scripts/determinism-check.sh` | Added `--debug-dump-rl` flag, composes with existing PER/mamba2 dump flags | +25 |
| `scripts/compare-rl-state.py` | New: bytewise compare for 35 RL-loop buffers + full ISV, grouped by candidate class (C/D/E/F) with verdict logic | +266 (new file) |

No `.cu` files added/changed. No commit. Phase 2.6 dispatch should
reuse the dump infrastructure and add the missing 4 buffer classes
listed above.

## Discipline checklist

- [x] No new files in repo root
- [x] Scripts in `scripts/`, docs in `docs/superpowers/notes/`
- [x] No `.cu` files added/changed (bug is NOT in any CUDA kernel
      we've identified yet)
- [x] Per `feedback_no_atomicadd.md`: not relevant
- [x] Per `feedback_no_nvrtc.md`: not relevant
- [x] Per `feedback_cpu_is_read_only.md`: dump method does DtoH only;
      mapped-pinned ISV is read-only on host
- [x] Per `feedback_no_stubs.md`: dump wired end-to-end via
      `build_diag_value`; no-op when env var unset
- [x] Per `feedback_local_smoke_before_cluster.md`: validated on
      RTX 3050, no cluster submit
- [x] Per `feedback_systematic_debugging` + dispatch STOP rule: no
      fix applied — diagnosis only, culprit outside 6 candidates
- [x] Per `feedback_no_quickfixes.md`: did NOT apply a tolerance-mask
      / f64-accumulator fix despite the temptation
- [x] No commit — left uncommitted for human review

## Reproduction

```bash
# Build with extended RL-state dump
SQLX_OFFLINE=true cargo build --release --example alpha_rl_train -p ml-alpha

# Triple-dump: PER + mamba2 + RL state
FOXHUNT_DETERMINISTIC=1 ./scripts/determinism-check.sh --quick \
  --debug-dump --debug-dump-mamba2 --debug-dump-rl

# Expected output:
#   [debug-dump]        all 3 PER buffers EQUAL through step 2
#   [debug-dump-mamba2] step 1 ALL 52 buffers EQUAL
#                       step 2 ALL forward activations EQUAL
#                       step 2 ALL weights + grads + backward outputs DIVERGE
#   [debug-dump-rl]     ALL 35 RL-loop buffers + 757-float ISV EQUAL through step 2
#   FIRST_DIVERGENCE: step=2 leaf=checksums.{attn_q|vsn_w|mamba2_grad_w_c_l1}

# Per-leaf interpretation: see scripts/compare-rl-state.py output's
# "Per-group culprit table" and "Next-step interpretation" hints.
```
