629ebd667c
Two same-seed runs now produce bit-equal eval_summary.json, alpha_rl_train_summary.json,
and diag.jsonl (modulo wall-clock elapsed_s). The 5-phase falsification chain landed:
Phase 2 PER tree-rebuild: __threadfence is NOT a grid-wide barrier; multiple blocks
raced across sum-tree levels. Fix: Grid=(1) Block=(1024) + __syncthreads
in rl_per_tree_rebuild.cu.
Phase 2.3 cuBLAS GEMM_DFALT + TF32 default-math allowed split-K non-deterministic
accumulation at 3 sites. New crates/ml-alpha/src/cublas_determinism.rs
applies CUBLAS_PEDANTIC_MATH via FOXHUNT_DETERMINISTIC env toggle
(0=TF32 prod, 1=PEDANTIC dev default, 2=DEFAULT_MATH control).
Phase 2.6 Two bugs surfaced sequentially in the backward kernel chain:
(1) rl_iqn_tau_cos_features had a multi-block r/w race on prng_state[batch]
— all N_TAU=32 blocks read seed; only tau_idx==0 wrote back; no
inter-block barrier. Fix: split into READ-ONLY rl_iqn_tau_cos_features
+ new sibling rl_iqn_advance_prng_state launched on same stream
(kernel-launch ordering = grid-wide barrier).
(2) OutcomeHead::new called near_zero_xavier without scoped_init_seed,
falling back to time+thread-id RNG. Stayed dormant until first done
event activated non-sentinel labels and divergent weights flowed via
grad_h_t_outcome into encoder gradient. Fix: add seed param + install
scoped_init_seed(dqn_seed.wrapping_add(0x0CE0)) guard.
Validation (./scripts/determinism-check.sh --quick, RTX 3050, b=128, 200+50 steps):
- All 200 rows of checksums.* leaves match (rel-tol 1e-5, abs-tol 1e-7)
- eval_summary.json, alpha_rl_train_summary.json byte-equal between runs
- diag.jsonl byte-equal modulo elapsed_s
- Eval pnl identical run-A vs run-B at seed 42
Pre-fix baseline (Phase 2.5 measurement): same-seed eval pnl spread $450k
($187k vs -$261k). Post-fix: $0 spread.
Speed cost: ~1.5ms/step amortised; ~10-15% slower than TF32 production
(PEDANTIC tax — acceptable in dev, toggle to FOXHUNT_DETERMINISTIC=0 for prod).
Mapped-pinned discipline: all 11 NEW memcpy_dtoh sites in diagnostic dump methods
+ per-step checksum readback use a new pub(crate) helper
read_slice_d_into<T: Copy>(stream, src, dst) — MappedRecordBuffer + raw
memcpy_dtod_async + raw_stream_sync + volatile read. Generic over T (f32, f64,
i32, u32, u8). Satisfies feedback_no_htod_htoh_only_mapped_pinned + hook guard.
Bundled Tier 1.5 fast-dev-cycle infrastructure (spec
docs/superpowers/specs/2026-06-02-fast-dev-cycle.md):
- scripts/local-mid-smoke.sh b=128, 2000+500, ~10min on RTX 3050
- scripts/determinism-check.sh runs mid-smoke twice, diffs checksums
- scripts/tier1_5_verdict.py behavioral kill verdict
- AdamW checkpoint save/load (crates/ml-alpha/src/trainer/optim.rs)
- IntegratedTrainer checkpoint save/load (resume from checkpoint)
- 15 Phase 1 checksum leaves in build_diag_value
- Env-gated dump methods (FOXHUNT_DETERMINISM_DEBUG_PER/MAMBA2/RL/BACKWARD)
for future divergence-chasing — never run in production
Documentation:
- docs/superpowers/specs/2026-06-02-determinism-foundation.md
- docs/superpowers/specs/2026-06-02-fast-dev-cycle.md
- docs/superpowers/plans/2026-06-02-determinism-foundation-implementation.md
- docs/superpowers/notes/2026-06-02-determinism-phase{1,2,2.2,2.5,2.6}-*.md
- Adjacent specs/plans/notes from the analytical chain that surfaced determinism
as the load-bearing blocker (eval-summary, eval-boundary, regime-observer,
multi-head policy, regime-invariance, Phase 3 IQN-complement post-mortem)
Unlocks: every controller / architecture / reward-shaping A/B from this commit
onward attributes outcome differences to the change, not random-init kernel-race
drift cascading through training x eval LOB-sim trajectories. The eval-collapse
investigation (pearl_reward_signal_anti_aligned_with_pnl, multi-head spec,
regime-invariance spec) is now testable with trustworthy verdicts.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
149 lines
5.1 KiB
Bash
Executable File
149 lines
5.1 KiB
Bash
Executable File
#!/usr/bin/env bash
|
||
# Tier 1.5 local mid-smoke — RTX 3050 Ti, b=128, 2000 train + 500 eval steps.
|
||
#
|
||
# Closes the 700× dev-cycle gap between Tier 1 (6 sec local correctness) and
|
||
# Tier 2 (70 min cluster eval). Gates 80%+ of bad hypotheses before they reach
|
||
# cluster. Target wall-clock: ≤ 10 min on RTX 3050.
|
||
#
|
||
# Spec: docs/superpowers/specs/2026-06-02-fast-dev-cycle.md §3.2
|
||
# Verdict: python3 scripts/tier1_5_verdict.py <out-dir>
|
||
#
|
||
# Data: test_data/futures-baseline-mid/ES.FUT/ — symlinks to 2024-Q1 (train,
|
||
# 4.8GB) + 2024-Q2 (eval, 14GB) MBP-10 files from the canonical PVC.
|
||
# Walk-forward split: --n-folds 2 --fold-idx 0 → train=Q1, eval=Q2.
|
||
#
|
||
# Usage:
|
||
# ./scripts/local-mid-smoke.sh # default seed=42
|
||
# ./scripts/local-mid-smoke.sh --seed 123 # override seed
|
||
# ./scripts/local-mid-smoke.sh --out /tmp/foxhunt-alt # custom out dir
|
||
# ./scripts/local-mid-smoke.sh --n-backtests 64 # smaller batch
|
||
#
|
||
# Per `feedback_local_smoke_before_cluster`: run this BEFORE every cluster
|
||
# smoke when touching .cu / trait impls / buffers / N_ACTIONS / ISV slots.
|
||
|
||
set -euo pipefail
|
||
|
||
SEED=42
|
||
OUT_DIR="/tmp/foxhunt-mid-smoke"
|
||
N_BACKTESTS=128
|
||
N_STEPS=2000
|
||
N_EVAL_STEPS=500
|
||
N_FOLDS=2
|
||
FOLD_IDX=0
|
||
|
||
usage() {
|
||
cat <<EOF
|
||
Usage: $(basename "$0") [OPTIONS]
|
||
|
||
Tier 1.5 mid-smoke — b=128, 2000+500 steps, ~10 min on RTX 3050.
|
||
|
||
Options:
|
||
--seed N RNG seed (default: $SEED — reproducible)
|
||
--out DIR Output directory (default: $OUT_DIR)
|
||
--n-backtests N Batch size (default: $N_BACKTESTS; drop to 64 if OOM)
|
||
--n-steps N Train steps (default: $N_STEPS)
|
||
--n-eval-steps N Eval steps (default: $N_EVAL_STEPS)
|
||
--n-folds K Walk-forward fold count (default: $N_FOLDS)
|
||
--fold-idx K Walk-forward fold index (default: $FOLD_IDX)
|
||
-h, --help Show this help
|
||
|
||
Output:
|
||
\$OUT_DIR/diag.jsonl — per-step train diag (642 scalars/step)
|
||
\$OUT_DIR/eval_diag.jsonl — per-step eval diag
|
||
\$OUT_DIR/alpha_rl_train_summary.json — final summary
|
||
EOF
|
||
}
|
||
|
||
while [[ $# -gt 0 ]]; do
|
||
case "$1" in
|
||
--seed) SEED="$2"; shift 2 ;;
|
||
--out) OUT_DIR="$2"; shift 2 ;;
|
||
--n-backtests) N_BACKTESTS="$2"; shift 2 ;;
|
||
--n-steps) N_STEPS="$2"; shift 2 ;;
|
||
--n-eval-steps) N_EVAL_STEPS="$2"; shift 2 ;;
|
||
--n-folds) N_FOLDS="$2"; shift 2 ;;
|
||
--fold-idx) FOLD_IDX="$2"; shift 2 ;;
|
||
-h|--help) usage; exit 0 ;;
|
||
*) echo "Unknown option: $1" >&2; usage >&2; exit 2 ;;
|
||
esac
|
||
done
|
||
|
||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||
cd "$REPO_ROOT"
|
||
|
||
# --mbp10-data-dir points DIRECTLY at the instrument's .dbn.zst directory
|
||
# (matches argo-rl template line: --mbp10-data-dir /data/futures-baseline-mbp10/ES.FUT).
|
||
MBP10_DIR="$REPO_ROOT/test_data/futures-baseline-mid/ES.FUT"
|
||
PREDECODED_DIR="$REPO_ROOT/test_data/futures-baseline-mid"
|
||
|
||
# Sanity checks.
|
||
if [[ ! -d "$MBP10_DIR" ]]; then
|
||
echo "ERROR: $MBP10_DIR not found." >&2
|
||
echo "Run setup: see CLAUDE.md 'Tier 1.5 data path' or docs/superpowers/specs/2026-06-02-fast-dev-cycle.md §3.1" >&2
|
||
exit 3
|
||
fi
|
||
|
||
n_mbp_files=$(find -L "$MBP10_DIR" -maxdepth 1 -name '*.dbn.zst' 2>/dev/null | wc -l)
|
||
if [[ "$n_mbp_files" -lt 2 ]]; then
|
||
echo "ERROR: $MBP10_DIR has $n_mbp_files MBP-10 files; need ≥ 2 for walk-forward split." >&2
|
||
exit 3
|
||
fi
|
||
|
||
mkdir -p "$OUT_DIR"
|
||
|
||
echo "=== Tier 1.5 mid-smoke ==="
|
||
echo " Repo: $REPO_ROOT"
|
||
echo " Seed: $SEED"
|
||
echo " Batch: $N_BACKTESTS"
|
||
echo " Steps: $N_STEPS train + $N_EVAL_STEPS eval"
|
||
echo " Walk-forward: fold $FOLD_IDX of $N_FOLDS"
|
||
echo " MBP-10 dir: $MBP10_DIR (${n_mbp_files} files)"
|
||
echo " Out dir: $OUT_DIR"
|
||
echo
|
||
|
||
# Build (sccache + dev-release for fast iteration; --release matches cluster).
|
||
echo "[build] cargo build --release --example alpha_rl_train -p ml-alpha"
|
||
SQLX_OFFLINE=true cargo build --release --example alpha_rl_train -p ml-alpha
|
||
|
||
BINARY="$REPO_ROOT/target/release/examples/alpha_rl_train"
|
||
if [[ ! -x "$BINARY" ]]; then
|
||
echo "ERROR: binary not found at $BINARY after build." >&2
|
||
exit 4
|
||
fi
|
||
|
||
echo
|
||
echo "[run] $BINARY ..."
|
||
START_TS=$(date +%s)
|
||
|
||
# Per `feedback_no_stubs`: the binary owns its full life cycle. We just
|
||
# invoke it and capture wall time. Exit code 2 = G8 NaN abort (per binary
|
||
# docstring); we propagate it so the caller knows the run died.
|
||
"$BINARY" \
|
||
--mbp10-data-dir "$MBP10_DIR" \
|
||
--predecoded-dir "$PREDECODED_DIR" \
|
||
--out "$OUT_DIR" \
|
||
--diag-jsonl "$OUT_DIR/diag.jsonl" \
|
||
--eval-diag-jsonl "$OUT_DIR/eval_diag.jsonl" \
|
||
--n-steps "$N_STEPS" \
|
||
--n-eval-steps "$N_EVAL_STEPS" \
|
||
--n-backtests "$N_BACKTESTS" \
|
||
--n-folds "$N_FOLDS" \
|
||
--fold-idx "$FOLD_IDX" \
|
||
--seed "$SEED" \
|
||
--instrument-mode front-month \
|
||
--log-every 200 \
|
||
--gpu-idx 0
|
||
|
||
END_TS=$(date +%s)
|
||
ELAPSED=$((END_TS - START_TS))
|
||
ELAPSED_MIN=$((ELAPSED / 60))
|
||
ELAPSED_SEC=$((ELAPSED % 60))
|
||
|
||
echo
|
||
echo "=== Mid-smoke complete in ${ELAPSED_MIN}m ${ELAPSED_SEC}s ==="
|
||
echo "Outputs:"
|
||
ls -lh "$OUT_DIR" 2>/dev/null
|
||
echo
|
||
echo "Next:"
|
||
echo " python3 $REPO_ROOT/scripts/tier1_5_verdict.py $OUT_DIR"
|