Production smoke panicked at trunk.rs:mamba2_l1() during
PerceptionTrainer::from_checkpoint. Root cause: load_checkpoint
called new_random which left mamba2_stack_1/2 = None, then tried
to upload weights into None blocks.
PerceptionTrainer::new was the only caller that populated the
Mamba2 stacks (X3/X5 migrations stopped at the accessor methods
but never moved construction). Inference paths that didn't go
through PerceptionTrainer::new — exactly what fxt-backtest does
via from_checkpoint → load_checkpoint — hit the gap.
Move Mamba2Block::new + state allocation into CfcTrunk::new_random.
PerceptionTrainer::new now sets up its optimizer + scratches against
the trunk-owned blocks via existing accessors (mamba2_l1/mamba2_l2).
Extends the trunk roundtrip test to:
- assert mamba2 weight slices are non-empty (catches empty-buffer
regressions that would let bit-equivalence pass trivially)
- bit-equivalence-check Mamba2 stack 1 + stack 2 weights through
save -> load, reproducing the exact failure path that hit prod.
Verified locally on RTX 3050 Ti — full workspace cargo check passes,
test passes with non-empty weight slices.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Template was missing feature-cache PVC mount on run-cell; trained
checkpoints + predecoded data live at /feature-cache/* on the
feature-cache-pvc claim. Mount it the same way alpha-perception does.
- sweep_smoke.yaml referenced /data/* (no such mount) and a 40-char
SHA path for the checkpoint dir. alpha-perception-runs subdirs are
9-char short SHAs; data path is /mnt/training-data/* (matches mount).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Working alpha-perception-template references
`foxhunt/foxhunt-training-runtime:latest`. Our lob-backtest-sweep
template stripped the `foxhunt-` prefix and got 404 from the registry
on run-cell + aggregate steps.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CARGO_TARGET_DIR=/cargo-target redirects all build output away from
the in-tree `<crate>/target/` path, so `cp $BUILD/target/release/...`
fails after a successful compile. Mirror alpha-perception-template's
correct pattern: `cp $CARGO_TARGET_DIR/release/...`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pods labeled `component: backtest` only match the `argo-base-egress`
NetworkPolicy (DNS, 443, MinIO, Mattermost). The ensure-binary step
needs SSH-to-gitlab-shell on port 2222, which is gated by
`argo-train-workflow` (selects `component=train`).
Relabel to `train` so the same NetworkPolicy that lets alpha-perception
clone the repo also covers lob-backtest-sweep. Egress shape (compile,
GPU run, PVC writes) is identical to a training run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The template wrote ~/.ssh/config but didn't chmod it. OpenSSH refused
the file with 'Bad owner or permissions on /root/.ssh/config' and the
git clone failed with 'exit status 128' before fxt-backtest could
compile.
Same one-line fix that alpha-perception-template.yaml has had since
forever. Verified: lob-backtest-sweep-<new> ensure-binary now passes
the SSH check and starts cargo build.
Matches the alpha-perception-template (which is the known-good
template in this namespace). Without the fix, ensure-binary +
run-cell + aggregate pods all stayed Init for ~20m with
'MountVolume.SetUp failed for volume "git-ssh-key" : secret
"git-ssh-key" not found' before the workflow timed out.
The actual secret in the foxhunt namespace is named
'argo-git-ssh-key' (see kubectl get secrets -n foxhunt). Same
template misconfig as the training-data PVC name.
The template referenced PVC 'training-data' but the actual claim in
the foxhunt namespace is 'training-data-pvc' (matches the convention
used by other workflows like alpha-perception-template). Without this
fix, ensure-binary + run-cell + aggregate pods all stayed Pending
forever with 'persistentvolumeclaim "training-data" not found',
blocking every smoke / threshold-tuning / deployability sweep.
Verified: smoke sweep lob-backtest-sweep-jp48v scheduled ensure-binary
onto the ci-compile-cpu pool (autoscaler triggered scale-up 0->1
within seconds of resubmission).
argo-lob-sweep.sh's awk substitution for # __SWEEP_CELLS__ matched BOTH
the actual marker (line 110, whitespace-indented) AND the docstring
reference at the top of the template that mentions `# __SWEEP_CELLS__`
inside a sentence (line 8). Result: cell task was injected twice,
once before `apiVersion:` (breaking kubectl apply with 'invalid object'
validation error) and once at the correct DAG location.
Fix: anchor the match to lines that START with whitespace + the marker
(^[[:space:]]+# __SWEEP_CELLS__), so the docstring sentence (which has
'# The' first) no longer matches.
sweep_smoke.yaml: rewrite from the spec's idealised (axes:/windows:)
schema to the actual base:/cells: schema that fxt-backtest sweep +
argo-lob-sweep.sh parse. Hardcodes SHA 58b5ebbd3 (the production
training run); bounds smoke runtime via max_events=100000.
Verified: smoke sweep workflow lob-backtest-sweep-hxwmq submits cleanly
and starts running.
Adds `fxt-backtest verdict <sweep_dir> --threshold X --windows W1,W2,W3,W4
--training-sha SHA --spec-sha SHA --out deployability_verdict.json`
that reads per-cell summary.json files at the realistic (1 tick, 200ms)
+ stress (1.5 tick, 400ms) anchors against the pre-registered threshold,
computes median Sharpe/max_dd/Sortino/profit_factor across windows,
classifies into Pass-robust / Pass-nominal / Fail-inconclusive / Fail /
Fail-degenerate per spec §3.5, and writes the audit JSON.
Adds serde_json workspace dep to fxt-backtest's Cargo.toml (was already
a transitive dep but not declared at this layer).
End-to-end CLI for Phase 2 runtime is now: argo-train.sh → argo-lob-sweep.sh
(smoke / threshold-tuning / deployability) → fxt-backtest aggregate →
fxt-backtest verdict → commit deployability_verdict.json.
Both tests exercised CfcTrunk::capture_graph_a + perception_forward_captured
+ snapshot_hidden, which feed V1-shaped CfC weights. After X8 the trunk's
CfC was reshaped to v2 layout (cfc_n_in=HIDDEN_DIM); the V1 forward path
now feeds FEATURE_DIM input into HIDDEN_DIM-shaped CfC — runtime garbage.
The methods themselves are still in trunk.rs (marked dead-code by rustc).
A deeper cleanup pass — deleting the V1 weight fields (heads_w_d, proj_*),
V1 per-step scratches, V1 cubin function handles, and the V1 forward
methods themselves — is a follow-up commit when fresh.
Verification: ml-alpha + ml-backtesting + fxt-backtest all build clean.
BacktestHarness now owns a PerceptionTrainer (in inference role) instead
of a raw CfcTrunk. The sliding K-window of recent snapshots accumulates
in the harness; at each decision-stride boundary (and only once the
window has reached cfg.seq_len), the harness calls
trainer.forward_only(&window) and broadcasts the last K position's
per-horizon probs to the LobSim.
fxt-backtest's main.rs constructs the trainer via
PerceptionTrainer::from_checkpoint when --checkpoint is supplied (else
random init for noise baseline).
Why this shape: PerceptionTrainer's evaluate_batched already runs the
full inference chain (snap → vsn → mamba2 → ln → mamba2 → ln →
attn_pool → cfc K-loop → grn heads) correctly. Duplicating that 400-line
forward chain on CfcTrunk would double the surface area for the same
result — the trunk's role is weight-source-of-truth (achieved in X1-X9),
not kernel-launch orchestration.
End-to-end status: alpha_train emits Checkpoint files via X14 wiring;
fxt-backtest now loads those Checkpoints via from_checkpoint and drives
forward via forward_only. Phase 2 (Argo runtime: training → smoke →
threshold pre-reg → 560-cell deployability sweep → verdict) is unblocked.
Adds PerceptionTrainer::config() accessor so the harness can read seq_len.
Verification: ml-alpha + ml-backtesting + fxt-backtest all build clean.
X11 inference interface for the deployability backtester. Two new
public methods on PerceptionTrainer:
forward_only(snapshots) -> Vec<f32>
Forward-only pass over a K-snapshot window. Wraps evaluate() with
dummy zero-valued labels and discards the loss. Returns the same
[K, B, N_HORIZONS]-shaped probability output as evaluate_batched.
from_checkpoint(dev, cfg, path) -> Result<Self>
Build a PerceptionTrainer ready for inference: constructs a fresh
trainer (with random init) to wire up kernel handles + grad
buffers + scratches, then overwrites the trunk's weights from the
Checkpoint file via CfcTrunk::load_checkpoint. The optimizer +
grad buffers stay allocated — unused at inference, but allocating
them keeps the struct invariant uniform. A leaner inference-only
struct can be added later if memory matters.
Architectural note: the trunk is the source of truth for weights (X1-X9
established that). PerceptionTrainer is the kernel-launch adapter that
drives forward + backward over those weights. Inference doesn't need a
separate forward path on the trunk — the trainer's evaluate_batched
already does the full chain correctly. fxt-backtest can now construct a
PerceptionTrainer in inference role via from_checkpoint + call
forward_only per decision.
Verification: ml-alpha + ml-backtesting + fxt-backtest build clean.
ml-alpha lib tests: 33 pass.
Per project_ml_alpha_starting_capital greenfield posture: there is no V1
to differentiate from (the V1 trunk forward was dead code, no V1
checkpoint files exist in the wild). The 'v2' prefix on every identifier
was historical baggage from the migration period.
Renames:
- CheckpointV2 -> Checkpoint (also drops the version: u32 field —
bincode either deserialises a current envelope or errors; no migration
path needed)
- CheckpointVersionProbe removed (was only for V1 rejection)
- LAYER_NORM_CUBIN_V2 / VARIABLE_SELECTION_CUBIN_V2 / ATTENTION_POOL_CUBIN_V2
-> LAYER_NORM_CUBIN / VARIABLE_SELECTION_CUBIN / ATTENTION_POOL_CUBIN
- _ln_module_v2 / _vsn_module_v2 / _attn_module_v2 -> drop _v2 suffix
- smoke_load_v2_checkpoint test -> smoke_load_checkpoint
- config/ml/sweep_v2_*.yaml -> config/ml/sweep_*.yaml
- migration-era 'V2 weight skeleton' / 'V2 fields' / etc. comments
cleaned to remove the v2 prefix
Pre-existing 'v2' references in ml-backtesting CUDA files
(decision_policy.cu, pnl_track.cu) are NOT touched — those refer to
future planned 'v2' refinements (Portfolio mode, multi-fill averaging)
from the C1-C19 commits and reflect aspirational features unrelated to
this session's trunk-grows work.
Verification: ml-alpha + ml-backtesting + fxt-backtest all build clean.
perception_forward_golden bit-exact (max_diff = 0.000000).
Adds CfcConfig.n_batch (default 1) and CfcConfig.seq_len (default 32).
PerceptionTrainer's trunk construction passes its training-time
n_batch/seq_len. Default values support backtester inference (B=1, K=32).
X11 foundation: subsequent commits add intermediate buffer fields +
forward_v2 method on CfcTrunk sized from these config fields. With X11
complete, fxt-backtest can drive the v2 forward through the trunk
without instantiating a full PerceptionTrainer.
Verification: ml-alpha + ml-backtesting + fxt-backtest all build clean.
Per spec §1.1 (X11 foundation).
Adds #[ignore]d integration test that loads a CheckpointV2 file produced
by alpha_train and verifies CfcTrunk::load_checkpoint accepts it.
Activated via FOXHUNT_SMOKE_CKPT env var on a CUDA-capable host.
Also adds a compile-time witness test confirming Summary.max_drawdown_pct
field exists post-X16 (required by emit_deployability_verdict).
Note: full BacktestHarness end-to-end smoke (running one cell against
fixture MBP-10) requires the v2 forward path to live on the trunk
(X11 — deferred). This commit verifies the producer/consumer wire-up.
Per spec §3.4, §4.1 (X18).
X16: Adds Summary.max_drawdown_pct field as |max_drawdown_usd| /
STARTING_CAPITAL_USD (pinned at $35k per project_ml_alpha_starting_capital
memory — realistic ES single-contract small-account anchor). Used by
the verdict emitter as the capital-deployability hard gate (median
across windows < 20%).
X17: Adds VerdictTier enum (Pass-robust / Pass-nominal / Fail-inconclusive
/ Fail / Fail-degenerate), AnchorSpec / AnchorReport / DeployabilityVerdict
types, classify_verdict (tiered logic per spec §3.5), and
emit_deployability_verdict that reads per-cell summary.json files at
both realistic (1.0 tick, 200 ms) and stress (1.5 tick, 400 ms) anchors,
computes median Sharpe / Sortino / max_dd_pct / profit_factor across
walk-forward windows, and applies the gates.
Per spec §3.2 (X16), §3.5 (X17).
X13: Adds PerceptionTrainer::save_checkpoint as a thin delegate to
self.trunk.save_checkpoint. Inference-only serialization — grads + AdamW
state aren't included.
X14: Inside the existing auc_h6000_improved block in alpha_train.rs,
calls trainer.save_checkpoint(out_dir / 'trunk_best_h6000.bin') so the
trained trunk lands alongside alpha_train_summary.json. Extends
AlphaTrainSummary with best_h6000_ckpt_path (Option<String>) so
downstream tooling (fxt-backtest --checkpoint) can locate the file
without re-deriving the path.
After this commit, every alpha-perception Argo workflow run produces
a CheckpointV2 file at every new-best-h6000 epoch, ready for backtest
consumption.
Verification:
- ml-alpha lib tests: 34 pass
- alpha_train example builds clean (release)
Per spec §1.1 (X13+X14).
Replaces CheckpointV1 with CheckpointV2 — covers the full v2 inference
graph: VSN, Mamba2 stacks 1+2 (in/a/b/c/out weights + biases), LN_a/LN_b,
attention-pool, CfC (now v2-shaped via cfc_n_in=HIDDEN_DIM), and the
full GRN heads (10 tensors: w1/b1, w2/b2, w_gate/b_gate, w_main/b_main,
w_skip/b_skip).
save_checkpoint reads every trunk weight tensor via memcpy_dtoh and
packs into the CheckpointV2 bincode envelope. load_checkpoint peeks
the version first (CheckpointVersionProbe), rejects non-2 versions,
deserialises into CheckpointV2, validates n_in / n_hid / cfc_n_in /
mamba2_state_dim match the supplied cfg, and uploads each weight
tensor with size-checked memcpy_htod.
V1 envelopes hard-rejected — alpha_train never produced V1 files, so
no migration. The old V1-shaped roundtrip test is removed; new V2
round-trip test will land alongside the alpha_train wiring (X14).
Verification:
- perception_forward_golden: PASS (max_diff = 0.000000)
- ml-alpha lib tests: 34 pass
- fxt-backtest binary builds clean
Per spec §1.2 (X12).
PerceptionTrainer's evaluate_batched + step_batched now read forward
kernel handles from self.trunk (snap_batched_fn, vsn_fwd_fn, ln_fwd_fn,
attn_fwd_fn, step_batched_fn, heads_grn_fwd_fn, transpose_3d_fn).
Trainer's duplicate cubin/function loading stays in place for now;
deferred dead-code cleanup.
Backward kernels (vsn_bwd_fn, ln_bwd_fn, etc.) stay on trainer — they
are training-only and don't belong on the inference trunk.
Verification:
- perception_forward_golden: PASS (max_diff = 0.000000)
Per spec §2.2 (X10b).
Adds the v2 forward kernel cubins (layer_norm, variable_selection,
attention_pool) and function handles (vsn_fwd_fn, ln_fwd_fn, attn_fwd_fn,
snap_batched_fn, step_batched_fn, heads_grn_fwd_fn, transpose_3d_fn)
onto CfcTrunk. The trunk now owns every kernel handle the v2 forward
chain needs.
PerceptionTrainer still loads its own copies of these cubins (duplicate
loading) and uses its own handles in evaluate_batched / step_batched —
the trainer-side consolidation lands in X10b. This commit is the
foundation: X11 will build capture_graph_a using these trunk-owned
handles, and X12 (CheckpointV2) doesn't depend on the consolidation.
Verification:
- perception_forward_golden: PASS (max_diff = 0.000000)
- ml-alpha lib tests: 34 pass
Per spec §2.2 (X10).
Adds the 4 missing GRN head fields (heads_w1, heads_b1, heads_w2, heads_b2)
that X1's skeleton under-modeled. Trunk now owns all 10 GRN head tensors:
input projection (HIDDEN→HEAD_MID), mid→mid layer, gate/main/skip outputs.
Trainer's 10 head fields removed. Trainer init still draws heads from
its ChaCha8Rng chain at the same call position, then memcpy_htod's them
into the trunk's now-allocated slots. Forward, backward, and AdamW
access sites redirect to self.trunk.heads_*. Gradient buffers + AdamW
state stay at trainer level.
Verification:
- perception_forward_golden: PASS (max_diff = 0.000000)
- ml-alpha lib tests: 34 pass
Per spec §2.2 (X9). After this commit, the trunk owns ALL v2 inference
weights — the source of truth for downstream checkpoint serialization.
Adds CfcConfig.cfc_n_in field (default HIDDEN_DIM) so the trunk's CfC
layer is sized for v2 usage (CfC input = LN_b output = HIDDEN_DIM) rather
than V1 usage (CfC input = snap features = FEATURE_DIM). The previous
CfcConfig.n_in field stays as "raw snap feature dim" for any V1 callers
still in the tree (fxt-backtest, trunk_forward.rs, graph_a_replay.rs);
their forward paths will be cleaned up in X10/X11 when the v2 forward
graph lives natively on the trunk.
Trainer's w_in_d / w_rec_d / b_d / tau_d fields are removed. Trainer
init still draws CfC weights from its ChaCha8Rng chain at the same
call position, then memcpy_htod's them into the trunk's now-v2-shaped
slots. Forward, backward, and AdamW access sites redirect to
self.trunk.{w_in,w_rec,b,tau}_d.
Verification:
- perception_forward_golden: PASS (max_diff = 0.000000)
- ml-alpha lib tests: 34 pass
Per spec §2.2 (X8).
LN_b (formerly ln_gain_d / ln_bias_d on trainer) now lives at
self.trunk.ln_b_gain_d / ln_b_bias_d. LN_b is the LayerNorm after
Mamba2 stack 2.
Verification: golden bit-exact; ml-alpha lib green.
Per spec §2.2 (X6).
Same pattern as X3: trainer constructs mamba2_l2 + Mamba2AdamW (against
&mamba2_l2), then moves the block into trunk.mamba2_stack_2. All
forward/backward/AdamW access sites redirect to self.trunk.mamba2_l2_mut().
Gradient buffers + AdamW state remain at trainer level.
Verification: golden bit-exact; ml-alpha lib green.
Per spec §2.2 (X5).
LN_a gain/bias tensors now live at self.trunk.ln_a_gain_d / ln_a_bias_d.
Trainer-side initialization values still drive the upload, preserving
PRNG-driven init values. Gradient buffers + AdamW state remain at
trainer level.
Verification: golden bit-exact (max_diff = 0.000000); ml-alpha lib green.
Per spec 2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md §2.2 (X4).
PerceptionTrainer no longer owns the Mamba2 stack-1 block; the trunk's
mamba2_stack_1 Option is filled in after Mamba2Block::new (which still
runs at the same point in PerceptionTrainer::new so the Mamba2 weight
init order is unchanged). Mamba2AdamW was already constructed against
&mamba2 before the move, so optimizer state is preserved.
CfcTrunk gains mamba2_l1_mut() / mamba2_l1() / mamba2_l2_mut() / mamba2_l2()
accessors that unwrap the Options. Forward, backward, and AdamW step
sites in evaluate_batched / step_batched / evaluate redirect through the
new accessors.
Gradient buffers (mamba2_grads_buffers) and AdamW state (mamba2_adamw)
remain at trainer level — training-only state stays with the trainer.
Verification:
- perception_forward_golden: PASS (max_diff = 0.000000)
- ml-alpha lib tests: 34 pass
Per spec 2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md §1.1, §2.2 (X3).
EOF
)
PerceptionTrainer gains a `trunk: CfcTrunk` field constructed at the
top of `new()` (after the determinism seed guard). VSN's
weight tensors (`vsn_w_d`, `vsn_b_d`) now live on `self.trunk`; the
trainer-owned copies are removed. Forward + backward + AdamW access
sites redirect to `self.trunk.vsn_w_d` / `self.trunk.vsn_b_d`.
PRNG-state preservation: the trainer's ChaCha8Rng chain still draws
VSN values at the same call position as before, then memcpy_htod's
them into the trunk's zero-initialised VSN slots from X1. This keeps
every downstream weight (attn_q, ...) bit-identical to the pre-X2
layout — perception_forward_golden continues to pass with
max_diff = 0.000000.
Gradient buffers (`grad_vsn_w_d`, `grad_vsn_b_d`) and AdamW state
(`opt_vsn_w`, `opt_vsn_b`) remain at trainer level — they're
training-only and shouldn't move to the inference trunk.
Verification:
- perception_forward_golden: PASS (max_diff = 0.000000)
- ml-alpha lib tests: 34 pass
- alpha_train example builds clean
Per spec 2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md
§1.1, §2.2 (X2).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds zero-initialised v2 weight tensors (VSN, LN_a/b, attn_q, GRN heads)
and Option<Mamba2Block> slots for stacks 1 and 2 to CfcTrunk. Extends
CfcConfig with mamba2_state_dim (default 16, matches
PerceptionTrainerConfig::default). Imports HEAD_MID_DIM for GRN head
sizing.
No forward path changes — fields allocated, not yet read. Existing
new_random init for V1 fields (CfC + simple heads + projection) is
preserved unchanged so the trunk's forward kernels still produce the
same output.
Skeleton for X2..X9 weight-group migrations.
Verification:
- ml-alpha lib tests: 34 pass
- perception_forward_golden bit-equivalence: PASS (max_diff = 0.000000)
Per spec 2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md
§1.1, §2.2 (X1).
ml_core::cuda_autograd::init::generate_uniform (backing xavier_uniform,
kaiming_uniform, bias_uniform, near_zero_xavier) defaulted to seeding
from SystemTime::now() + thread_id, producing non-reproducible weights
across processes. Mamba2 stacks initialise via OwnedGpuLinear::xavier,
which routes through this helper — so PerceptionTrainer.evaluate output
diverged 5-30% across fresh-process runs with identical cfg.seed.
Fix: thread-local seedable RNG override. New API:
let _g = ml_core::cuda_autograd::init::scoped_init_seed(seed);
// ... all xavier/kaiming/bias/near_zero calls draw from
// StdRng::seed_from_u64(seed) chain while _g is alive ...
// _g dropped here -> restores default time-based seeding
PerceptionTrainer::new now installs the guard before any Mamba2Block
construction, so the trainer is reproducible from cfg.seed end-to-end.
CfC/VSN/heads already used explicit ChaCha8Rng::seed_from_u64 — only
Mamba2 was affected.
Production behavior unchanged when no guard is set. ml-core: 306 tests
pass, ml-alpha: 34 lib tests pass.
Regression test: crates/ml-alpha/tests/perception_forward_golden.rs
captures bit-exact PerceptionTrainer.evaluate output (loss + 160 probs
on a deterministic seed=42 fixture) into a 644-byte golden file.
Three consecutive runs now produce max_abs_diff=0; pre-fix runs varied
by 0.1-0.3 absolute on individual probs.
.gitignore: added exception for crates/ml-alpha/tests/fixtures/*.bin
so deterministic test fixtures land in repo.
Per pearl_scoped_init_seed_for_reproducibility in project memory.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Supersedes the 2026-05-19 deployability spec (commit 07d5de504). The
prior spec assumed CfcTrunk::save_checkpoint was the producer-side
wiring point — discovered at execution time that alpha_train trains via
PerceptionTrainer (full v2: VSN + Mamba2 ×2 + LN ×2 + attn-pool + CfC +
heads), not the simpler CfcTrunk. The existing LOB backtester loads
CheckpointV1 envelopes that only know about CfC weights, so there is no
producer for a checkpoint containing the full v2 model.
New scope: one bigger spec covering refactor + deployability end-to-end.
Phase 1 (X0–X19, code commits): grow CfcTrunk to own the full v2
inference graph; restructure PerceptionTrainer to wrap a trunk + add
training-only state (grads, AdamW). Discipline: bit-equivalence golden
fixture (X0) gates every refactor commit (X1–X11). CheckpointV2
envelope (X12) replaces V1. Verdict emitter (X17) reuses the tiered
classification (Pass-robust / Pass-nominal / Fail-inconclusive / Fail /
Fail-degenerate) from the superseded spec.
Phase 2 (Argo runtime): production training → smoke gate → threshold
pre-registration → 560-cell deployability sweep → verdict + memory
update.
Hard gate before Phase 2: post-refactor fold-0 smoke must reproduce
recorded 3-fold A/B numbers (best_mean_auc 0.7529, best_h6000 0.7639,
both within ±0.010 absolute) from project_ml_alpha_v2_ab_verdict
memory. Prior spec marked SUPERSEDED in its header, kept in history as
audit trail.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User revisions to design spec from this session:
1. §2.1 — split anchor into realistic (200 ms, 1 tick) and stress (400 ms,
1.5 tick). Realistic remains the hard verdict gate; stress grades the Pass
into Pass-robust vs Pass-nominal.
2. §2.2 — expand metrics from Sharpe-only to four: annualized daily Sharpe
and max-drawdown are hard gates (median across windows > 1.0 and < 20%
respectively); Sortino and profit factor are diagnostics. Per-window
summary.json schema extended.
3. §2.6 — verdict emitter rewrites to two-anchor logic, tiered output:
Pass-robust / Pass-nominal / Fail-inconclusive / Fail / Fail-degenerate.
4. §3.3 — added max-dd computation and zero-trade-window failure modes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the wiring gap between the existing real-LOB backtest system (C1–C19 on
this branch) and the v2 ml-alpha model. alpha_train.rs currently never calls
save_checkpoint, so the LOB harness/sweep/aggregate machinery has never been
pointed at a real trained model.
Design defines a single-pass falsifiable deployability gate: produce a
production checkpoint (cv-n-folds=1, cv-train-window=4 → train on 2024
quarters, val on 2025-Q1, hold out 2025-Q2..2026-Q1), pre-register one
threshold on the W0 val window, then evaluate median Sharpe across 4
held-out walk-forward quarters at the realistic Scaleway→IBKR anchor (200ms
RTT, 1-tick all-in cost). Pass iff median > 1.0; inconclusive in [0.8, 1.0]
counts as fail; smoke gate halts the full sweep on any wiring failure.
Approved through brainstorming. Awaiting user spec-review before plan
handoff.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The motivating "50% saturation hit-rate" observation came from gm67g
fold 0 (Option-2 config, commit 004b662c8 — itself a -0.003 mean_auc
regression vs ISV-σ at 410ab6b0e). Re-running the same diagnostic on
the actual production baseline (ISV-σ 3 folds: rxm5t/r57lx/x24d6,
logs retrieved from MinIO argo-logs bucket) on 215 horizon-epoch
observations:
saturated λ→AUC up: 8/13 = 61.5% median Δauc = +0.0025
non-saturated→AUC up: 96/202 = 47.5% median Δauc = -0.0012
difference: +14pp in favor of the controller working
The BCE-z-score controller is empirically correlated with the
optimization target. No evidence it's misaligned with AUC. Spec
premise falsified.
Spec retained in tree as historical record. The diagnostic
methodology itself (saturation→Δauc analysis on archived MinIO logs)
is the durable artifact and is documented in the supersede block.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The current λ controller boosts horizons by BCE-EMA z-score, which conflates
three distinct causes of high BCE — only one of which (under-trained
horizon) benefits from boosting. The other two (intrinsically harder
horizon, calibration drift) are unaffected by gradient-magnitude lifts.
Empirical motivation (gm67g fold-0, this branch's 3-fold run):
when λ_h6000 saturated at 2.0, h6000's next-epoch AUC went UP 2/4
times and DOWN 2/4 times. 50% hit rate ⇒ the BCE saturation signal
is misaligned with the optimization objective.
This spec replaces BCE-z-score with AUC-regret:
best_auc[h] = running max of per-horizon validation AUC
regret[h] = max(0, best_auc[h] - current_auc[h])
regret_max_ema = EMA of max_h regret[h]
λ[h] = clamp(1.0, 2.0, 1 + regret[h] / regret_max_ema)
Properties:
- Aligned with the objective (AUC, not BCE)
- Naturally bounded (regret ∈ [0, 1])
- "At personal best" → λ=1.0 (no wasted boost)
- Auto-saturation by design (max-regret horizon → ceiling)
- Cold-start clean (e0: best=current, regret=0, uniform λ)
- Zero hardcoded magic beyond bootstrap epsilons
Implementation surface ~250 LOC:
- Split horizon_lambda kernel: horizon_loss_ema (per-step) +
horizon_lambda (per-epoch, AUC-regret math)
- Trainer state: drop z_max_ema, add best_auc + regret_max_ema
- Per-epoch entry point: trainer.update_lambda_from_auc()
- Extend isv snapshot log line with best_auc_h* + regret_h*
Test plan:
- Local 9/9 perception_overfit
- New synthetic-AUC unit test (controller correctness)
- Cluster 5-epoch smoke + 3-fold A/B vs Option-2 baseline (004b662c8)
- Success: mean_auc lifts ≥ +0.005 AND median sat→next-AUC delta positive
Awaiting user review before plan handoff.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a single tracing::info!("step") emitted every 500 training steps:
if epoch_train_steps % 500 == 0 {
tracing::info!(epoch, step = epoch_train_steps, loss = loss, "step");
}
Consumed by /tmp/alpha_monitor.py (v2: per-epoch trajectories + ISV +
liveness) to render intra-epoch loss trajectory and detect stalls
faster than the once-per-epoch granularity allowed.
At 8000 steps/epoch and ~36s/epoch on L40S → ~16 step lines per epoch
per fold, well below the cluster log volume budget.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The C16 tick-rule swap (19986c8d9) replaced the dense `trade_count` delta
at snap_feature_assemble's slot [18] with a signed L1 tick-rule estimate.
That broke a load-bearing redundancy in Phase 1+2+3:
Phase 1+2+3 feature [17] = log1p(trade_count_delta)
Phase 1+2+3 feature [18] = signed_log1p(trade_count_delta)
= log1p(trade_count_delta) for count ≥ 0
Within a file, `cur.trade_count >= prev.trade_count` (monotonic), so the
delta is non-negative and the two features were *bit-identical* floats.
The encoder's Mamba2 W_in had two random-initialised rows projecting the
same dense signal, giving it effective 2× capacity allocation on
trade-flow.
Post-C16:
feature [17] = log1p(trade_count_delta) (unchanged)
feature [18] = signed_log1p(tick_rule_estimate) (new, uncorrelated)
Empirical (7.8M ES MBP-10 snapshots, Q1+Q2 2024 production data):
| Stat | OLD count_delta | NEW tick_rule |
|------------------------|-----------------|-----------------|
| zero rate | 10.2% | 49.1% |
| mean ± std | 4.26 ± 3.82 | -0.18 ± 10.95 |
| max |value| | 100 | 3378 |
| Pearson r vs count | 1.0000 (id) | 0.0002 |
Sign-class breakdown vs Phase 1+2+3 slot [18]:
44.3% new=0 but old≠0 (44% of true trades MISSED by tick-rule)
5.5% new≠0 but old=0 (cancel-as-trade false positives)
22.8% both positive (agree)
0.0% both negative (old never negative)
22.6% sign disagreement (old saw trades, new says "seller")
The tick-rule heuristic is a strictly different (and noisier) signal,
not a superset. Three mechanisms simultaneously regressed mean_auc:
A) lost 2× W_in capacity on count signal
B) noisier signal at slot [18]
C) extreme outliers (max 33× wider) destabilise LayerNorm at [18]
Fix (Option 2 per the diagnostic):
out[26] = signed_log1p((float) trade_count)
Restores the duplicate count-delta signal at a previously-reserved slot.
Slot [18] keeps the new tick-rule signal — the 5.5% "signal added" and
the directional info at L1 are still available. FEATURE_DIM (40) is
unchanged; LayerNorm + Mamba2 W_in dimensions are unchanged.
Both the single-snapshot kernel and the batched kernel are updated.
`snap_feature_bit_equiv::reserved_slots_are_zero` updated to assert
the new slot-26 semantics (signed_log1p of synthetic trade_count=7
= log(8) ≈ 2.079).
All 9 perception_overfit tests pass + all 9 snap_feature_bit_equiv
tests pass.
Cluster verification: single-fold smoke + 3-fold validation follow.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Empirical record at this data + this architecture:
Phase 1+2+3 (no σ in loss): mean_auc 0.7749 ± 0.024
v2 (σ + axes B/C/D/E): mean_auc 0.7541 ± 0.005
σ-only (kept σ, dropped B/C/D/E): mean_auc 0.7506 ± 0.008
Perf-fix (σ-only math): mean_auc 0.7499 ± 0.010
ISV-σ (closed-form σ + adaptive λ): mean_auc ~0.75 (2/3 folds)
Every architecture with σ-in-loss lands at 0.75. Removing σ is the
only thing that hits 0.77. That is a framework mismatch, not a tuning
problem.
Kendall+Gal+Cipolla 2018 frames σ as TASK NOISE level — damp the noisy
task, trust the clean one. Our horizons don't have different label
noise; they have different intrinsic difficulty (longer horizon = more
price-walk uncertainty = lower achievable AUC). σ-Kendall sees "high
BCE on h6000" and interprets it as "h6000 is unreliable, back off" —
precisely the opposite of what we want. h6000 is the deployment
target; damping it is a self-inflicted wound. With mean_bce ~ 0.7,
σ ≈ √0.7 ≈ 0.84 ⇒ w_h ≈ 0.71, uniformly attenuating gradient by
~30% across every horizon. λ's z-score boost (max 2×) can rebalance
relative-per-horizon but cannot recover the absolute magnitude.
Per-horizon prioritization remains via the ISV-driven λ controller
(grad scaler in heads_grn_bwd, per pearl_adam_normalizes_loss_weights).
That controller IS appropriate for our problem: it boosts hard
horizons rather than damping them, and it operates on the gradient
into the trunk rather than on the loss aggregate (Adam-cancellation
safe).
Changes to bce_loss_multi_horizon.cu (six lines):
w_h = bw (was: 0.5 * bw * exp(-2 * log_sigma_h))
d_log_sigma_h[h] = 0.0 (was: 1 - 2 * w_h * mean_bce)
total_loss += bw * mean_bce (was: + w_h * mean_bce + log_sigma_h)
σ infrastructure preserved unchanged:
- horizon_ema_and_lambda still computes log_sigma_h closed-form from
loss_ema (Kendall equilibrium) for telemetry / future label-noise
estimation use cases
- log_sigma_h kernel arg still in BCE signature (zero churn at
callsite); ignored inside
All 9 perception_overfit tests pass — including
horizon_ema_and_lambda_track_after_training which validates the
per-horizon controller end-to-end through 64 K-loop iterations of
capture/replay.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per pearl_controller_anchors_isv_driven, every controller anchor/target/
cap derives from a tracked signal, not hardcoded constants. The σ-only
revert kept Kendall σ as a free Adam-learned scalar — that violated ISV
discipline and fought Adam's m/√v normalization
(pearl_adam_normalizes_loss_weights).
Single source of truth for both per-horizon controllers:
log_sigma_h[h] ← max(log(0.5), 0.5 * log(loss_ema[h]))
Kendall equilibrium (∂L/∂log σ = 0 ⟹ σ_h² = mean_bce_h)
in closed form. Asymmetric floor at log(0.5) prevents
collapse. No Adam state, no gradient delay.
lambda[h] ← clamp(1.0, 2.0, 1.0 + Z_SCALE_ISV * z_h)
Z_SCALE_ISV = (LAMBDA_CEILING - LAMBDA_FLOOR) / z_max_ema
Adaptive scale auto-uses the full clamp envelope:
the historical-max-z horizon maps exactly to
LAMBDA_CEILING. Replaces hardcoded Z_SCALE=0.5 which
rarely engaged on real data (max observed λ ~1.04).
Both anchor on the same ISV (loss_ema). z_max_ema is a new single-scalar
EMA state tracking max |z| across horizons, with first-obs bootstrap.
Removes:
- opt_log_sigma AdamW optimizer (σ no longer learned)
- grad_log_sigma_h_d memset (BCE kernel writes; output ignored — kept
only to preserve BCE kernel signature)
Kernel signature change (horizon_ema_and_lambda):
+z_max_ema [1] (read+write EMA state)
+log_sigma_h [5] (closed-form output, overwrite)
Discipline:
- First-obs bootstrap (sentinel <= 0) per pearl_first_observation_bootstrap
- Permanent floor (max(real, floor)) per pearl_blend_formulas_must_have_permanent_floor
- Asymmetric clamp per pearl_audit_unboundedness_for_implicit_asymmetry
- Z-score normalisation per pearl_zscore_normalization_for_magnitude_asymmetric_signals
- No nvrtc, no atomicAdd, no host branches in graph capture
All 9 perception_overfit tests pass — including
horizon_ema_and_lambda_track_after_training which validates the kernel
end-to-end through 64 K-loop iterations of capture/replay.
Submit local smoke; cluster A/B vs σ-only baseline (0.7506/0.7519) and
vs Phase 1+2+3 (0.7749/0.7591) follows once the perf-only 3-fold A/B
confirms no regression at b23f8f2ef.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Post-A/B verdict (see project_ml_alpha_v2_ab_verdict.md): v2 with all
5 axes was marginally tied on h6000 (+0.0013 vs 0.7591 baseline mean,
fails the +0.01 win threshold) and slightly below on mean_auc
(−0.0208 vs 0.7749 baseline mean, within 1σ) at ~5× the wall-time
cost. Per `feedback_v7_gem_methodology` (measure before delete or
wire), the architecture has been measured — it doesn't earn its
compute cost. This commit reverts the axes that didn't lift:
- axis B (L2 anchor + Wiener-α controller) — DROPPED
- axis C (horizon-token attention pool) — DROPPED
- axis D (regime-MoE gate + experts) — DROPPED
- axis E (inverted cross-variate attn) — DROPPED
- axis A (Kendall σ-weighted BCE) — KEPT
Files deleted (kernels, host bindings, numgrad tests, trainer state):
- cuda/{horizon_token_attention_pool, inverted_attention_pool,
inv_pooled_merge, regime_moe_gate, anchor_l2,
horizon_mean_collapse}.cu
- src/{horizon_token_attention_pool, inverted_attention_pool,
inv_pooled_merge, regime_moe_gate, anchor_l2,
horizon_mean_collapse}.rs
- src/trainer/{multi_horizon_attention, anchor_controller}.rs
- tests/{horizon_token_attention_pool_numgrad,
inverted_attention_pool_numgrad,
regime_moe_gate_numgrad,
anchor_l2_numgrad}.rs
Files restored (from V1 commit 41292303d):
- cuda/attention_pool.cu — legacy single-Q attention pool kernel
- src/trainer/perception.rs — pre-MHA trainer state with the
legacy `attn_*` plumbing intact.
Files modified:
- bce_loss_multi_horizon.cu stays σ-aware (kept the V7 work; it
has the kernel function name preserved from V1).
- perception.rs: ADD `log_sigma_h_d [N_HORIZONS]`,
`grad_log_sigma_h_d [N_HORIZONS]`, `opt_log_sigma` AdamW
directly on PerceptionTrainer (no MHA bundle). BCE callsites in
`step_batched` (training) and `evaluate_batched` thread the σ
args. Grad scratch zeroed each step before the BCE launch.
`opt_log_sigma.step` lives in section 9 alongside the other
AdamW updates.
NET DIFF: 23 files, 442 insertions, 3160 deletions (~2700-line
cleanup).
LOCAL VERIFICATION (RTX 3050 sm_86, --test-threads=1):
- ml-alpha builds clean (cuda feature)
- bce_grad_finite_diff 4/4 PASS (BCE still works through σ-kernel)
- perception_overfit 9/9 PASS (full trainer pipeline, loss-shrinks
tests still green)
NEXT: cluster smoke + 3-fold A/B vs task #200 baseline. Expected
wall-time ≈ baseline 17 s/epoch (we're back to baseline architecture
plus 5 scalar Kendall σ params + 1 tiny AdamW). Expected lift on
mean_auc: modest — Kendall σ rebalances per-horizon contributions
based on observed BCE EMA, which may help horizons with intrinsically
higher noise floors.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CRITICAL ARCHITECTURAL FIX discovered while planning fused kernel:
The previous Stage 2 `add_inv_broadcast` helper in
`multi_horizon_attention.rs` was a STUB that returned Ok(()) without
doing anything. Practical consequences:
- Forward: `inv_pooled_d` (output of inverted_attention_pool.forward)
was never added into `ctx_h_d`. Downstream MoE + heads never saw
the inverted-attention signal. Axis E contributed ZERO to the
forward output and the loss.
- Backward: `inv_pool.backward` was being fed `grad_ctx_mean` as
its "upstream gradient", but that's the gradient at the CHAIN
TERMINUS — not the gradient w.r.t. inv_pooled_d (which is zero
by construction since inv_pooled wasn't in the loss). The bwd
was injecting incorrect noise into `grad_ln_out`.
Net: paying inverted_attention compute for no gain, plus polluting
ln_b's gradient. Two perf rewrite rounds earlier today showed no
wall-time movement precisely because the slow path was wired into
training while the optimized one was dead.
FIX:
(a) New kernel `cuda/inv_pooled_merge.cu`:
fwd: ctx_h[b, h, d] += inv_pooled[b, d] (broadcast over h)
bwd: grad_inv_pooled[b, d] = Σ_h grad_ctx_h[b, h, d]
Tiny — single block-per-batch, no syncthreads, coalesced reads.
(b) Host binding `src/inv_pooled_merge.rs` (InvPooledMerge).
(c) `MultiHorizonAttention` adds:
- `merge: InvPooledMerge` field.
- `grad_inv_pooled_d [B, H]` buffer for the real upstream of inv_pool.bwd.
(d) `MultiHorizonAttention.forward` now calls `merge.forward(...)`
between inv_pool.forward and moe.forward. Axis E is now actually
in the model's forward output.
(e) `MultiHorizonAttention.backward` now calls `merge.backward` after
moe.bwd writes `grad_ctx_h_d`, producing `grad_inv_pooled_d`.
`inv_pool.backward` consumes the REAL upstream gradient
(`grad_inv_pooled_d`) instead of the prior `grad_ctx_mean` fake.
(f) Old stub `add_inv_broadcast` deleted.
CORRECTNESS:
- perception_overfit 9/9 PASS after the wiring. Loss still shrinks
on the constant-signal test (0.67 → -0.99 over 250 steps), now
with axis E actually contributing.
- All numgrad kernel tests still pass (kernels themselves unchanged
in this commit; only the wiring).
PERF IMPACT (expected):
- +2 tiny launches per step (merge fwd + bwd). Negligible.
- The inverted-attention compute that was previously dead now
actually feeds the loss → same wall-time, but it's earning the
cost. This unblocks meaningful axis-E perf measurement on next
smoke.
NEXT: re-run cluster smoke to confirm wall-time + verify axis E
gradient flow is healthy. After that, consider full MHA forward
fusion as a follow-up.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two perf optimizations bundled:
(1) MoE backward scratch compaction
Old: `grad_w_scratch_d [B, N_H, N_E, H, H]` = 1.3 MB per step (B=1)
New: `grad_w_scratch_d [B, N_H, H, H]` = 320 KB per step
4× memory reduction. Since each batch's top-1 router selects ONE
expert, only that expert's slot was ever non-zero in the prior
layout — the N_E axis was entirely wasteful.
The shape change required:
- Updated `regime_moe_gate_bwd` to write the compact layout.
- New `regime_moe_gate_scatter` kernel scatters per-(b, h)
rank-1 contributions into `grad_experts_W[e]` / `grad_experts_b[e]`
based on `top_e[b]`. Grid (N_E, H, ceil(H/32)) × block (32) —
one warp per (e, d_out, d_in_chunk). 65536 → 16384 grid cells
(4× fewer blocks dispatched).
- Dropped the previously-naive 65536-block `reduce_axis0` for
`grad_experts_w` from `perception.rs` (the scatter kernel
produces the final per-expert grad directly).
- `tests/regime_moe_gate_numgrad.rs` reads `grad_experts_w` from
the scatter output instead of host-side reducing the 5D scratch.
(2) inverted_attention bwd loop interchange
Phase 2's tight loop:
for k:
for j:
ds_myh_j = d_scores[my_h, j] // doesn't depend on k!
ds_j_myh = d_scores[j, my_h] // doesn't depend on k!
...
Hoisted d_scores reads out of the K-loop into J-outer with
per-thread `q_arr[K_MAX]` / `k_arr[K_MAX]` register accumulators.
Net: 32× fewer DRAM reads of d_scores per thread per bwd.
CORRECTNESS:
- regime_moe_gate numgrad PASSES (1/1, 11 numgrad checks).
- inverted_attention numgrad PASSES (1/1, 6 numgrad checks).
- perception_overfit 9/9 PASS — including loss-shrinks tests.
NEXT: re-run cluster smoke to measure the new wall-time vs the 17 s
baseline. Prior smoke at a263cd544 was 11.26 s for 1000 steps; this
commit's smoke will reveal whether the MoE scratch compaction + loop
interchange land us closer to the 30 s/epoch gate.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The cluster smoke at 9170d24fe showed ~88 s/epoch projected for the
full 8000-step epoch (vs the 17 s baseline) — a 5× regression that
fails the spec §5 wall-time gate. Diagnosis: the inverted-attention
kernel's hot loops recomputed `mean_k_X_inv[j] = (1/K) Σ_k X_inv[j, k]`
per-thread, per-j, every pass:
- fwd pool: 128 × 32 reads per thread = 4096 extra ops × 128 threads
= ~0.5 M wasted ops per fwd
- bwd phase 1: 128 × 32 × 2 passes per thread = ~1.0 M wasted ops
per bwd
At 1000 steps/epoch this alone adds ~1.5 s of pointless compute, and
the cumulative effect across fwd + bwd + DRAM round-trips for the
score tensor was the main contributor to the 5× regression.
REWRITE:
1. `mean_k_X_inv[H]` (0.5 KB) cached ONCE in shared memory at kernel
entry. Each thread h does its OWN k-trajectory load + sum in
parallel during the x_inv staging, so no extra cost vs the prior
x_inv-only stage.
2. Forward now does:
- Pass 1: compute max(score) only — no DRAM writes.
- Pass 2: compute exp(score - max) → write to attn_out (scratch),
accumulate sum locally.
- Pass 3: single sweep over j — divide attn_out by sum (in place),
accumulate pool += attn · mean_k[j]. ← uses cached mean_k.
Eliminates the post-softmax recompute of mean_k that the prior
version did 128× per thread.
3. Backward `d_scores` computation now uses cached mean_k (saves 4096
ops/thread). Also: `dot = pooled[my_h]` is now computed once at
the start of phase 1 from attn × mean_k (one pass over j) instead
of being implicit in the per-j d_attn computation.
CORRECTNESS:
- inverted_attention_pool numgrad PASSES 6 random-position checks
within 5e-2 rel / 5e-3 abs.
- perception_overfit 9/9 tests PASS — including
stacked_trainer_loss_shrinks_on_constant_signal (loss 0.67 → -0.99
over 250 steps).
SMEM FOOTPRINT:
- fwd: x_inv[H · K] + mean_k[H] = 16 KB + 0.5 KB
- bwd: x_inv[H · K] + mean_k[H] + dp[H] = 16 KB + 1 KB
Both well under the 48 KB sm_86 dynamic-shared default; no
cuFuncSetAttribute opt-in needed.
Next: re-run cluster smoke to measure the new wall-time. Expected to
land ≤ 30 s/epoch per spec §5 wall-time gate.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
