Files

jgrusewski 629ebd667c feat(ml-alpha): deterministic same-seed training + Tier 1.5 fast-dev-cycle

Two same-seed runs now produce bit-equal eval_summary.json, alpha_rl_train_summary.json,
and diag.jsonl (modulo wall-clock elapsed_s). The 5-phase falsification chain landed:

  Phase 2   PER tree-rebuild: __threadfence is NOT a grid-wide barrier; multiple blocks
            raced across sum-tree levels. Fix: Grid=(1) Block=(1024) + __syncthreads
            in rl_per_tree_rebuild.cu.

  Phase 2.3 cuBLAS GEMM_DFALT + TF32 default-math allowed split-K non-deterministic
            accumulation at 3 sites. New crates/ml-alpha/src/cublas_determinism.rs
            applies CUBLAS_PEDANTIC_MATH via FOXHUNT_DETERMINISTIC env toggle
            (0=TF32 prod, 1=PEDANTIC dev default, 2=DEFAULT_MATH control).

  Phase 2.6 Two bugs surfaced sequentially in the backward kernel chain:
            (1) rl_iqn_tau_cos_features had a multi-block r/w race on prng_state[batch]
                — all N_TAU=32 blocks read seed; only tau_idx==0 wrote back; no
                inter-block barrier. Fix: split into READ-ONLY rl_iqn_tau_cos_features
                + new sibling rl_iqn_advance_prng_state launched on same stream
                (kernel-launch ordering = grid-wide barrier).
            (2) OutcomeHead::new called near_zero_xavier without scoped_init_seed,
                falling back to time+thread-id RNG. Stayed dormant until first done
                event activated non-sentinel labels and divergent weights flowed via
                grad_h_t_outcome into encoder gradient. Fix: add seed param + install
                scoped_init_seed(dqn_seed.wrapping_add(0x0CE0)) guard.

Validation (./scripts/determinism-check.sh --quick, RTX 3050, b=128, 200+50 steps):
  - All 200 rows of checksums.* leaves match (rel-tol 1e-5, abs-tol 1e-7)
  - eval_summary.json, alpha_rl_train_summary.json byte-equal between runs
  - diag.jsonl byte-equal modulo elapsed_s
  - Eval pnl identical run-A vs run-B at seed 42

Pre-fix baseline (Phase 2.5 measurement): same-seed eval pnl spread $450k
($187k vs -$261k). Post-fix: $0 spread.

Speed cost: ~1.5ms/step amortised; ~10-15% slower than TF32 production
(PEDANTIC tax — acceptable in dev, toggle to FOXHUNT_DETERMINISTIC=0 for prod).

Mapped-pinned discipline: all 11 NEW memcpy_dtoh sites in diagnostic dump methods
+ per-step checksum readback use a new pub(crate) helper
read_slice_d_into<T: Copy>(stream, src, dst) — MappedRecordBuffer + raw
memcpy_dtod_async + raw_stream_sync + volatile read. Generic over T (f32, f64,
i32, u32, u8). Satisfies feedback_no_htod_htoh_only_mapped_pinned + hook guard.

Bundled Tier 1.5 fast-dev-cycle infrastructure (spec
docs/superpowers/specs/2026-06-02-fast-dev-cycle.md):
  - scripts/local-mid-smoke.sh        b=128, 2000+500, ~10min on RTX 3050
  - scripts/determinism-check.sh      runs mid-smoke twice, diffs checksums
  - scripts/tier1_5_verdict.py        behavioral kill verdict
  - AdamW checkpoint save/load (crates/ml-alpha/src/trainer/optim.rs)
  - IntegratedTrainer checkpoint save/load (resume from checkpoint)
  - 15 Phase 1 checksum leaves in build_diag_value
  - Env-gated dump methods (FOXHUNT_DETERMINISM_DEBUG_PER/MAMBA2/RL/BACKWARD)
    for future divergence-chasing — never run in production

Documentation:
  - docs/superpowers/specs/2026-06-02-determinism-foundation.md
  - docs/superpowers/specs/2026-06-02-fast-dev-cycle.md
  - docs/superpowers/plans/2026-06-02-determinism-foundation-implementation.md
  - docs/superpowers/notes/2026-06-02-determinism-phase{1,2,2.2,2.5,2.6}-*.md
  - Adjacent specs/plans/notes from the analytical chain that surfaced determinism
    as the load-bearing blocker (eval-summary, eval-boundary, regime-observer,
    multi-head policy, regime-invariance, Phase 3 IQN-complement post-mortem)

Unlocks: every controller / architecture / reward-shaping A/B from this commit
onward attributes outcome differences to the change, not random-init kernel-race
drift cascading through training x eval LOB-sim trajectories. The eval-collapse
investigation (pearl_reward_signal_anti_aligned_with_pnl, multi-head spec,
regime-invariance spec) is now testable with trustworthy verdicts.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

2026-06-02 17:56:00 +02:00

16 KiB

Raw Blame History

Fast Dev Cycle — Items 1-5 Implementation Log

Date: 2026-06-02 Branch: ml-alpha-adaptive-controller-floors Spec: docs/superpowers/specs/2026-06-02-fast-dev-cycle.md §3.1 - §3.6 (items 1-5 only) Out of scope: §3.4 async double-buffer checkpoint (item 6, separate effort) Status: Items 1-5 SHIPPED. Item 3 surfaced a critical determinism break — see §3 below.

Summary

Built a working Tier 1.5 mid-smoke funnel that gates 80%+ of bad hypotheses before they reach cluster:

test_data/futures-baseline-mid/ES.FUT/ — symlinked train+eval files (Q1 + Q2 ES MBP-10, 19 GB resolved, 4 KB on-disk symlinks)
scripts/local-mid-smoke.sh — full mid-smoke runner (5m 51s wall-clock on RTX 3050 Ti at b=128 / 2000 train + 500 eval)
scripts/tier1_5_verdict.py — behavioral kill verdict (7 signals + per-step/eval_summary consistency check, stdlib-only)
scripts/determinism-check.sh — reproducibility self-test (runs mid-smoke twice with same seed, diffs final 5 rows)
CLAUDE.md — documented under "Build & Test" with three-tier funnel table
.gitignore — added test_data/futures-baseline-mid/

Verdict tooling works end-to-end on real data — first run flagged KILL_popart.sigma_CV+eval_pnl AND surfaced a $4.5M per-step/eval_summary discrepancy ($517k vs -$4M), matching the canonical pearl_grwwh_eval_catastrophic_collapse failure mode.

CRITICAL FINDING (item 3): trainer is NOT deterministic. Two same-seed runs diverge by step 2 in grad_norm_ema.q (Δ=4.5), and by end of 200 steps:

Train pnl diverges $33,632 vs $32,225 (Δ=$1,406)
Eval pnl diverges $0 vs $364,925 (Δ=$364,925)
Eval trade count diverges 0 vs 347

Most CPU-side seeding looks correct on audit. Root cause is GPU-side non-determinism (likely cuDNN heuristics or non-deterministic CUDA reductions). Fix is out of scope for this spec.

Item 1 — §3.1 PVC data sync

Plan deviation

Spec called for "rsync 2-3 highest-value MBP-10 files to local test_data/futures-baseline-mid/ (~1.5 GB)". Actual reality:

PVC path is /training-data/futures-baseline-mbp10/ES.FUT via training-data-pvc (NOT /feature-cache/futures-baseline-mbp10 per spec — feature-cache-pvc only has predecoded sidecars)
Smallest MBP-10 file is 2024-Q1 at 4.7 GB. The "~1.5 GB total" spec estimate was wrong by 10×
User already had test_data/futures-baseline-mbp10/ES.FUT/{Q1, Q2}.dbn.zst locally (verified SHA256 match against PVC)

What shipped

Created test_data/futures-baseline-mid/ES.FUT/ containing two symlinks (zero disk cost, 19 GB resolved):
- ES.FUT_2024-Q1.dbn.zst → ../../futures-baseline-mbp10/ES.FUT/ES.FUT_2024-Q1.dbn.zst (4.8 GB train)
- ES.FUT_2024-Q2.dbn.zst → ../../futures-baseline-mbp10/ES.FUT/ES.FUT_2024-Q2.dbn.zst (14 GB eval)
SHA256 verified against PVC: 9ba3bcc468ccf148d8341cd71cec008ae2532676b0f4bb8c238b0e61c5aca625 (Q1, matches PVC)
.gitignore: added test_data/futures-baseline-mid/
CLAUDE.md: updated "Build & Test" section with tiered-validation table and the three new scripts

Walk-forward fold scheme

With 2 files and --n-folds 2 --fold-idx 0, the trainer splits:

block_size = 1 (files / folds)
train = files[0..1] = Q1
eval = files[1..2] = Q2

This gives real walk-forward semantics (no train/eval overlap), matches --mbp10-data-dir /data/futures-baseline-mbp10/ES.FUT invocation pattern from infra/k8s/argo/alpha-rl-template.yaml:181.

Item 2 — §3.2 Local mid-smoke script

What shipped

scripts/local-mid-smoke.sh (executable, 152 lines):

Argparse for --seed, --out, --n-backtests, --n-steps, --n-eval-steps, --n-folds, --fold-idx
Defaults: seed=42, b=128, 2000 train + 500 eval, fold 0 of 2
Builds via SQLX_OFFLINE=true cargo build --release --example alpha_rl_train -p ml-alpha
Invokes binary with --mbp10-data-dir test_data/futures-baseline-mid/ES.FUT --instrument-mode front-month --log-every 200 --gpu-idx 0
Pre-flight checks: directory exists, ≥ 2 MBP-10 files, follows symlinks via find -L
Wall-clock reporting + next-step hint pointing to verdict script

Measured wall-clock on RTX 3050 Ti

Configuration	Wall-clock
Default (b=128, 2000+500)	5m 51s
Sanity check (b=16, 200+50)	4m 45s (dominated by 210s eval-file load)

GPU memory used: 1.2 GB / 4 GB (b=128) — well within RTX 3050 Ti capacity per spec §2.

Path correction during shake-down

Initial draft passed --mbp10-data-dir at the parent of ES.FUT/. The discover_mbp10_files_sorted (crates/ml-alpha/src/data/loader.rs:294) is non-recursive — it expects the path to be the instrument's .dbn.zst directory directly. Fixed to match argo template (--mbp10-data-dir /data/futures-baseline-mbp10/ES.FUT).

Item 3 — §3.5 Determinism audit

CPU-side seeding audit (all clean)

Every randomness entry point in crates/ml-alpha/src/ derives from cli.seed:

Source	Seeded by	Location
Loader random-anchor RNG (`MultiHorizonLoader.rng`)	`ChaCha8Rng::seed_from_u64(cfg.seed)`	`crates/ml-alpha/src/data/loader.rs:422`
`upload_to_gpu` seed	`cli.seed` arg	`crates/ml-alpha/examples/alpha_rl_train.rs:391`
DQN trainer seed	`cli.seed.wrapping_add(0xDA)`	`crates/ml-alpha/examples/alpha_rl_train.rs:410`
PPO trainer seed	`cli.seed.wrapping_add(0xBE)`	`crates/ml-alpha/examples/alpha_rl_train.rs:411`
PER buffer seed	`cli.seed.wrapping_add(0x9E37_79B9)`	`crates/ml-alpha/examples/alpha_rl_train.rs:413`
Eval-phase loader	`cli.seed.wrapping_add(0xE7AE)`	`crates/ml-alpha/examples/alpha_rl_train.rs:786`
Device-side Thompson xorshift32	`ChaCha8Rng::seed_from_u64(cfg.dqn_seed.wrapping_add(0xC0_C0_C0_C0))`	`crates/ml-alpha/src/trainer/integrated.rs:2142`
Loader explicitly does NOT shuffle	`// (no shuffle — file order is the caller's…)`	`crates/ml-alpha/src/data/loader.rs:412`

No Python data loader exists — entirely Rust. No thread_rng() / random() / SystemTime::now() calls in ml-alpha hot path. ml-ppo and ml-core use thread_rng() but those are not on the RL training path.

Self-test result: DETERMINISM IS BROKEN

scripts/determinism-check.sh --quick (200 train + 50 eval, seed=42):

FIRST NON-TIME DIVERGENCE at row 2 step=2: 4 drifts
  grad_norm_ema.q                          A=64.80356  B=60.30021  Δ=4.50335
  risk_stack.atom_calibration.target_bot_saturation_rate
                                           A=0.14137   B=0.15253   Δ=0.01116
  isv_ema_in.td_kurtosis                   A=0.00092   B=0.00093   Δ=9.59e-06
  controller_branch.per_alpha_input        A=0.00092   B=0.00093   Δ=9.59e-06

By step 199:

654 leaves diverging
trading.realized_pnl_cum_usd: $33,631 vs $32,225 (Δ=$1,406)
confidence_gate.gated_count_total: 4826 vs 495 (~10× difference)
units.trail_distance[*] differs across most of the 128 batch slots
risk_stack.regime.tail.session_pnl_variance_ema: 261 vs 1008 (~4× difference)

Eval (50 steps) shows much larger split:

Run A: 0 trades, $0 pnl, 0 win_rate
Run B: 347 trades, $364,925 pnl, 0.343 win_rate

Inferred root cause (not fixed per spec)

CPU-side seeding is comprehensive. The drift originates GPU-side, candidates:

Non-deterministic CUDA reductions — block-tree reductions on different launch geometries can produce fp32 differences in addition order even with identical inputs. Compounding through ~200 Adam steps yields macroscopic divergence.
cuDNN heuristic auto-tune — mamba2 forward / attention kernels may run different algorithm selections on first call vs warm-cache.
Stream ordering — multiple CUDA streams (prefill_graph, postfill_graph, reward_graph, training_graph) interleave in arbitrary order at sync boundaries.

The grad_norm_ema.q divergence at step 2 (Δ=4.5 on a value of ~64) is much larger than fp32 noise (1e-5), so this is structural, not roundoff. Inspection scope is a separate spec — flagged here for the next session.

Determinism-check script behavior

scripts/determinism-check.sh:

Default 2000+500 (matches mid-smoke). --quick → 200+50 for faster diagnostic.
Diffs last 5 rows of diag.jsonl leaf-by-leaf with rel-tol=1e-5, abs-tol=1e-7
Skips elapsed_s / sps / wall-clock signals (inherently non-deterministic)
Reports top 10 drifts per row sorted by absolute magnitude
Exit 0 = deterministic, 1 = drift, 2 = run failure

Item 4 — §3.3 Behavioral kill verdict script

What shipped

scripts/tier1_5_verdict.py (executable, ~430 lines, Python 3 stdlib only — no pandas/numpy):

7 behavioral signals + 1 consistency check, each with explicit KILL/WARN/OK thresholds:

Signal	Source field	KILL	Target	Falsifies
`action_entropy`	last train row	<0.6 or >2.3	[1.20, 2.04]	policy collapse / random
`q_pi_agree_ema`	last train row	<0.3	≥0.6	Q/π decoupled
`pearson(rewards.sum, Δpnl)`	full train trace	<0.3	≥0.5	`pearl_reward_signal_anti_aligned_with_pnl`
`avg_hold_steps trend`	last 1500 train rows (linreg slope)	slope<-0.01	slope>0	surfer pattern absent
`popart.sigma CV`	full train trace	CV>0.5	CV<0.5	controller wildly oscillating
`wr_train` (Kelly EMA)	last train row	<0.15	≥0.25	random or worse
`wr_eval` (Kelly EMA)	last eval row	<0.15	≥0.20	random or worse
`eval_pnl`	eval_summary.total_pnl_usd	<-$3M	>-$1M	catastrophic

Output modes: human-readable (default), --quiet (one-line verdict), --json (machine-readable).

Exit codes: 0=OK, 1=KILL, 2=input missing/insufficient.

Sample output on real mid-smoke

Ran on /tmp/foxhunt-mid-smoke (full b=128, 2000+500, seed=42):

========================================================================
Tier 1.5 verdict: KILL_popart.sigma_CV+eval_pnl_(eval_summary.total_pnl_usd)
========================================================================
  train rows:        2000
  eval rows:         500
  eval_summary:      present

STATUS  SIGNAL                                      EXPLANATION
------  ------------------------------------------  ----------------
OK      action_entropy                              1.755 within target [1.20, 2.04]
OK      q_pi_agree_ema                              0.963 ≥ 0.6
WARN    pearson(rewards.sum, Δpnl)                  0.318 below target ≥ 0.5 (n=1999)
OK      avg_hold_steps trend                        slope +0.00264/step over last 1500 rows
KILL    popart.sigma CV                             0.810 > 0.5 — controller wildly oscillating
OK      wr_train (trading.win_rate)                 0.341 ≥ 0.25
OK      wr_eval (trading.win_rate)                  0.336 ≥ 0.2
KILL    eval_pnl (eval_summary.total_pnl_usd)       $-3,996,663 < $-3,000,000 — catastrophic
WARN    per-step vs eval_summary                    per_step=$517,914 | eval_summary=$-3,996,663 | gap=$4,514,576 (112.96%) > 5% — accounting axis discrepancy

[KILL] KILL_popart.sigma_CV+eval_pnl_(eval_summary.total_pnl_usd) — DO NOT submit cluster smoke. Fix locally first.

The verdict caught two structural problems that would have wasted a 70-min cluster submit AND surfaced the known per-step/eval_summary disagreement — full payoff matrix.

Notes on signal sourcing

wr_* prefers isv_config.kelly.win_rate_ema (cumulative EMA across all trades) but falls back to trading.win_rate (point measure of last batch) when EMA path is absent. Source is reported in the signal name.
pearson(rewards.sum, Δpnl) pairs rewards.sum (the gradient signal) with Δ(trading.realized_pnl_cum_usd) (the actual pnl direction). Per pearl_reward_signal_anti_aligned_with_pnl, this is the load-bearing signal — a sign-flipped or noisy correlation explains negative-eval hysteresis.
avg_hold_steps is computed by linear regression on (step, trading.avg_hold_steps) over the trailing 1500-step window. Robust to NaN — drops rows where the field is non-finite.

Item 5 — §3.6 Per-step / eval_summary consistency

What shipped

Folded into tier1_5_verdict.py as signal_consistency():

Reads last eval_diag.jsonl row → trading.realized_pnl_cum_usd
Reads eval_summary.json → total_pnl_usd
Computes |per_step − eval_summary| / max(|per_step|, |eval_summary|, $1)
WARN if fraction > 5% (NOT KILL — gives signal without blocking, per spec)

Sample finding on the real mid-smoke

WARN    per-step vs eval_summary  per_step=$517,914 | eval_summary=$-3,996,663 | gap=$4,514,576 (112.96%) > 5% — accounting axis discrepancy

This is the same accounting-axis bug pattern as pearl_grwwh_eval_catastrophic_collapse ($234M gap at grwwh step 4863). The per-step diag's trading.realized_pnl_cum_usd and the eval_summary's total_pnl_usd are computed from different sources (per-step running sum of raw_rewards × b_size vs eval_summary's aggregation of LobSim closed-trade records). A 112% gap means the per-step diag is materially misleading for eval-period pnl observability.

Per spec §3.6 long-term action: "fix the accounting axis discrepancy in the diag/summary code" — out of scope for items 1-5, flagged for follow-up spec.

Files added / changed

Path	Change
`test_data/futures-baseline-mid/ES.FUT/` (NEW)	2 symlinks (Q1, Q2) into existing test_data
`scripts/local-mid-smoke.sh` (NEW, 152 lines, +x)	Tier 1.5 runner
`scripts/tier1_5_verdict.py` (NEW, ~430 lines, +x)	Behavioral verdict + consistency check
`scripts/determinism-check.sh` (NEW, ~125 lines, +x)	Reproducibility self-test
`.gitignore`	Added `test_data/futures-baseline-mid/`
`CLAUDE.md`	Added "Tiered local validation" subsection under Build & Test

No CUDA / Rust / training-math changes. Pure dev infrastructure. No commits — left uncommitted for human review.

Discipline checklist

No files saved to repo root
No new *.md files except this requested impl log
Read before Edit on all touched existing files (.gitignore, CLAUDE.md)
No stubs, no TODO/FIXME/XXX markers
Symlinks chosen over duplicate-data download (per feedback_no_quickfixes — leverage existing canonical files)
PVC SHA256 verification before deciding local files are usable
Verdict signals all tied to existing memory entries (pearl_grwwh, pearl_reward_signal_anti_aligned, pearl_edge_lives_at_wave_timescale)
Per-step/eval_summary check emits WARN not KILL (per spec §3.6: "WARNING — gives signal without blocking")

Recommended next steps (for next session)

Determinism fix spec — grad_norm_ema.q diverges at step 2. Inspect block-tree reduction ordering and stream synchronization in the encoder grad-norm path. The fact that eval pnl can differ by $365k across same-seed runs makes ALL controller-iteration verdicts noisy.
Per-step/eval_summary reconciliation spec — the $4.5M gap on this small smoke is the same shape as pearl_grwwh_eval_catastrophic_collapse $234M. Audit the LobSim trade accounting vs the per-step raw_rewards axis.
Item 6 — async checkpoint — separate ~4-day effort per spec §3.4. Unblocks resume-from-baseline + saves ~58 min per cluster smoke.
Tighten Pearson threshold — current run shows 0.318 (WARN, below 0.5 target). Confirm whether this is the established baseline or whether the mid-smoke window is too short for the signal to settle.

Reproduction commands

# Setup (one-time; symlinks already in place from this session)
ls test_data/futures-baseline-mid/ES.FUT/    # should show 2 symlinks

# Run mid-smoke (5-6 min on RTX 3050 Ti)
./scripts/local-mid-smoke.sh

# Verdict
python3 scripts/tier1_5_verdict.py /tmp/foxhunt-mid-smoke

# Determinism self-test (~10 min, --quick for ~6 min)
./scripts/determinism-check.sh --quick

16 KiB Raw Blame History Unescape Escape

Fast Dev Cycle — Items 1-5 Implementation Log

Summary

Item 1 — §3.1 PVC data sync

Plan deviation

What shipped

Walk-forward fold scheme

Item 2 — §3.2 Local mid-smoke script

What shipped

Measured wall-clock on RTX 3050 Ti

Path correction during shake-down

Item 3 — §3.5 Determinism audit

CPU-side seeding audit (all clean)

Self-test result: DETERMINISM IS BROKEN

Inferred root cause (not fixed per spec)

Determinism-check script behavior

Item 4 — §3.3 Behavioral kill verdict script

What shipped

Sample output on real mid-smoke

Notes on signal sourcing

Item 5 — §3.6 Per-step / eval_summary consistency

What shipped

Sample finding on the real mid-smoke

Files added / changed

Discipline checklist

Recommended next steps (for next session)

Reproduction commands

16 KiB

Raw Blame History