Session 2026-06-05: investigation-first falsification gated the F4 oracle
build (7-14 days) and walked back through the reward/edge question to a
code-verified root cause — the MBP-10 decoder corrupts the inside quote
(dbn_parser.rs:903 'for lvl in 1..max_lvl' never restores levels[0]),
contaminating mid/feature/reward/fill locally and on the cluster PVC.
- notes/2026-06-05-edge-investigation-decision-record.md: full 8-verdict arc
(incl. 2 self-retractions) + next-session plan (confirm on PVC, fix parser,
regenerate sidecars, re-audit on clean data).
- specs: f4-oracle-pretraining (GATED), reward-pnl-alignment-restoration
(superseded by the data-quality finding), bellman-target-foundation-reshape.
- plans: reward-pnl-alignment-restoration-implementation, f2-q-centered-distill.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two same-seed runs now produce bit-equal eval_summary.json, alpha_rl_train_summary.json,
and diag.jsonl (modulo wall-clock elapsed_s). The 5-phase falsification chain landed:
Phase 2 PER tree-rebuild: __threadfence is NOT a grid-wide barrier; multiple blocks
raced across sum-tree levels. Fix: Grid=(1) Block=(1024) + __syncthreads
in rl_per_tree_rebuild.cu.
Phase 2.3 cuBLAS GEMM_DFALT + TF32 default-math allowed split-K non-deterministic
accumulation at 3 sites. New crates/ml-alpha/src/cublas_determinism.rs
applies CUBLAS_PEDANTIC_MATH via FOXHUNT_DETERMINISTIC env toggle
(0=TF32 prod, 1=PEDANTIC dev default, 2=DEFAULT_MATH control).
Phase 2.6 Two bugs surfaced sequentially in the backward kernel chain:
(1) rl_iqn_tau_cos_features had a multi-block r/w race on prng_state[batch]
— all N_TAU=32 blocks read seed; only tau_idx==0 wrote back; no
inter-block barrier. Fix: split into READ-ONLY rl_iqn_tau_cos_features
+ new sibling rl_iqn_advance_prng_state launched on same stream
(kernel-launch ordering = grid-wide barrier).
(2) OutcomeHead::new called near_zero_xavier without scoped_init_seed,
falling back to time+thread-id RNG. Stayed dormant until first done
event activated non-sentinel labels and divergent weights flowed via
grad_h_t_outcome into encoder gradient. Fix: add seed param + install
scoped_init_seed(dqn_seed.wrapping_add(0x0CE0)) guard.
Validation (./scripts/determinism-check.sh --quick, RTX 3050, b=128, 200+50 steps):
- All 200 rows of checksums.* leaves match (rel-tol 1e-5, abs-tol 1e-7)
- eval_summary.json, alpha_rl_train_summary.json byte-equal between runs
- diag.jsonl byte-equal modulo elapsed_s
- Eval pnl identical run-A vs run-B at seed 42
Pre-fix baseline (Phase 2.5 measurement): same-seed eval pnl spread $450k
($187k vs -$261k). Post-fix: $0 spread.
Speed cost: ~1.5ms/step amortised; ~10-15% slower than TF32 production
(PEDANTIC tax — acceptable in dev, toggle to FOXHUNT_DETERMINISTIC=0 for prod).
Mapped-pinned discipline: all 11 NEW memcpy_dtoh sites in diagnostic dump methods
+ per-step checksum readback use a new pub(crate) helper
read_slice_d_into<T: Copy>(stream, src, dst) — MappedRecordBuffer + raw
memcpy_dtod_async + raw_stream_sync + volatile read. Generic over T (f32, f64,
i32, u32, u8). Satisfies feedback_no_htod_htoh_only_mapped_pinned + hook guard.
Bundled Tier 1.5 fast-dev-cycle infrastructure (spec
docs/superpowers/specs/2026-06-02-fast-dev-cycle.md):
- scripts/local-mid-smoke.sh b=128, 2000+500, ~10min on RTX 3050
- scripts/determinism-check.sh runs mid-smoke twice, diffs checksums
- scripts/tier1_5_verdict.py behavioral kill verdict
- AdamW checkpoint save/load (crates/ml-alpha/src/trainer/optim.rs)
- IntegratedTrainer checkpoint save/load (resume from checkpoint)
- 15 Phase 1 checksum leaves in build_diag_value
- Env-gated dump methods (FOXHUNT_DETERMINISM_DEBUG_PER/MAMBA2/RL/BACKWARD)
for future divergence-chasing — never run in production
Documentation:
- docs/superpowers/specs/2026-06-02-determinism-foundation.md
- docs/superpowers/specs/2026-06-02-fast-dev-cycle.md
- docs/superpowers/plans/2026-06-02-determinism-foundation-implementation.md
- docs/superpowers/notes/2026-06-02-determinism-phase{1,2,2.2,2.5,2.6}-*.md
- Adjacent specs/plans/notes from the analytical chain that surfaced determinism
as the load-bearing blocker (eval-summary, eval-boundary, regime-observer,
multi-head policy, regime-invariance, Phase 3 IQN-complement post-mortem)
Unlocks: every controller / architecture / reward-shaping A/B from this commit
onward attributes outcome differences to the change, not random-init kernel-race
drift cascading through training x eval LOB-sim trajectories. The eval-collapse
investigation (pearl_reward_signal_anti_aligned_with_pnl, multi-head spec,
regime-invariance spec) is now testable with trustworthy verdicts.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per spec §4.1 + Q-3-seed-baseline. Both baseline Argo workflows
succeeded:
- alpha-perception-baseline-seed16963 (39min)
- alpha-perception-baseline-seed16964 (26min)
Precise AUC values deferred: train pod stdout is in MinIO archive
(xl.meta format, needs mc auth). The on-disk alpha_train_summary.json
on /feature-cache/alpha-perception-runs/f22f3f948/ corresponds to
seed 16963 (last to finish). Task 15 implementer retrieves via debug
pod mount of feature-cache PVC.
Known data point: seed 16962 = 0.7454 best_auc_h6000.
The Smoke 1 invariant gate threshold is signal-derived per
pearl_tests_must_prove_not_lock_observations — formulas captured in
the JSON for Task 15 to apply once values are retrieved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drafts `pearl_diagnostic_decomposition_before_reward_intervention` per
the plan's Task 0.5 deliverable. Captures the discipline pattern
applied in Tasks 0.1–0.2:
Before changing reward weights or Bellman target arithmetic to fight
a Q-attractor, instrument a per-action decomposition of the existing
reward components AND a trajectory observable on the Q-target
distribution. The instrumentation is observability-only and lands
atomically. One epoch of dispatch surfaces whether the suspected
pathology is real BEFORE any production-path code changes.
Pearl is DRAFT — pending the Task 0.3 (reviewer L40S 1-epoch
dispatch) + Task 0.4 (KILL CRITERION evaluation) outcomes. Promotion
to user memory + cross-reference to MEMORY.md is gated on both legs
returning PROCEED.
Cross-references: pearl_canary_input_freshness_launch_order,
pearl_first_observation_bootstrap, pearl_wiener_alpha_floor_for_
nonstationary, feedback_no_partial_refactor, feedback_wire_everything_up.
No production code changes — single doc-only commit.
Plan: docs/superpowers/plans/2026-05-08-sp18-reward-shape-hold-attractor.md
§ Phase 0 Task 0.5.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
