First L40S run (alpha-perception-s6hqv, commit 586d1e782) trained flat
at chance: train_loss=0.6953, val_loss=0.6943, AUCs all near 0.50
across 5 epochs and 40k gradient steps. Synthetic overfit on the same
trainer hit 0.59→0.19 in 250 steps, so wiring was sound.
The signal-killer was feature scale: raw size deltas were ±100, dt_ms
could exceed 1e4, and log-returns sat at ~1e-4. Mamba2's W_in
projection saturates on those extremes, gradient bleeds out.
Now in the kernel:
out[0] = (mid - prev_mid) / tick_size [tick-return]
out[12..17]= sgn(d) * log1p(|d|) [signed-log OFI]
out[18] = sgn(v) * log1p(|v|) [signed-log vol]
out[19] = log1p(max(dt_ms, 0)) [log dt]
No tuned constants — tick_size is a market quantity; log1p and
signed-log are monotone bounded transforms. Bit-equiv tests updated
to assert the new closed-form expressions (still GPU-only, no CPU
oracle).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The MultiHorizonLoader was calling load_or_predecode_mbp10 on every
next_sequence() call, deserializing millions of MBP-10 snapshots per
sequence. With 8000 sequences and 9 files this gave ~50+ hour
training time on what should be IO-trivial work.
Now keep one file cached (LoadedFile { snapshots, labels_full }),
yield ceil(n_max_sequences / n_files) sequences from it before
advancing. Per-horizon labels are computed once per file load and
sliced cheaply per anchor. With 8000/9: ~890 loads → 9 loads.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
alpha_train --seed is clap-typed as u64, and clap's default u64
parser only accepts decimal digits. Previous default "0x4242"
hit "invalid digit found in string" at startup.
Replace with decimal equivalent 16962 (= 0x4242) in both the
submission script default and the workflow template default.
Long-term: could add a custom clap value_parser that accepts
hex/dec/oct prefixes, but for now decimal-only matches the
foxhunt convention in other CLIs (alpha_baseline, etc.).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The training-data PVC actually has its MBP-10 .dbn.zst files at
/data/futures-baseline-mbp10/ES.FUT (9 files for ES futures), NOT at
/data/futures-baseline/mbp10. The latter path doesn't exist; the
prev run's bash check fired exit 1 with "MBP-10 data directory not
found".
The PVC root layout (from a probe pod):
/data/bin/ <- compiled binaries by SHA
/data/feature-cache/ <- predecoded sidecar cache
/data/futures-baseline/{ES,NQ,ZN,6E}.FUT/ <- legacy multi-asset
/data/futures-baseline-mbp10/ES.FUT/ <- ES MBP-10 (this is what we want)
/data/futures-baseline-trades/ES.FUT/ <- ES trades
/data/futures-baseline-1s/ES.FUT/ <- 1-second OHLCV
/data/trained-models/ <- model checkpoints
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The CI pipeline (.gitlab-ci.yml stage `build-foxhunt-training-runtime`)
builds and pushes the image as foxhunt-training-runtime:latest (with
the foxhunt- prefix). Other consumers in the repo agree:
- infra/k8s/training/image-prepuller.yaml
- infra/k8s/training/job-template.yaml
The alpha-perception template (and the older alpha-cv template) used
training-runtime:latest without the prefix — broken since the image
never existed at that path. Kubelet kept hitting ErrImagePull /
ImagePullBackOff with "not found".
This was the next blocker after the CPU oversizing fix. Stopping the
in-flight workflow and resubmitting on the corrected template.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a tiny alpine pod (check-cache) that runs first on the platform
pool (no autoscaler delay, ~3 sec end-to-end) and probes the
training-data PVC for /data/bin/$SHA/alpha_train. Outputs:
- sha: short SHA used for binary cache keying
- cache: "hit" or "miss"
ensure-binary now has `when: cache == miss` — when the binary is
already cached for the current SHA, the entire ~4.8GB ci-builder
image pull + sccache compile cycle is skipped. Re-runs on the same
SHA now go straight from submission to training in ~30 seconds
instead of ~3 minutes.
train depends on check-cache + ensure-binary; sources the SHA from
check-cache's output (works whether ensure-binary ran or was
skipped — Argo treats `when:` skip as a satisfied dependency).
Submission script (scripts/argo-alpha-perception.sh) now pre-resolves
commit-sha=HEAD to an actual git SHA via `git rev-parse origin/<branch>`
before submission. This lets the alpine check-cache pod work without
installing git in the container.
Also removed the now-stale `ci-training-h100x2|ci-training-h100-sxm`
case branch from the SM-arch detection — those pools no longer exist
post-pool-cleanup commit a252119fd.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previous cpu request was 8 (limit 16), but L40S-1-48G allocatable cpu
is 7800m (8 vCPU total minus kubelet overhead). Pod couldn't fit on
the very node we wanted it on — autoscaler provisioned the L40S
cleanly but the scheduler then rejected the pod with
"Insufficient cpu" forever.
Fix: requests cpu=6 / mem=16Gi, limits cpu=7 / mem=64Gi. Leaves
~1.8 vCPU and ~27Gi memory headroom for the system daemonsets
(cilium, csi-node, nvidia driver/device-plugin/dcgm-exporter,
gpu-feature-discovery, node-bootstrap, prometheus-node-exporter,
promtail) that the L40S node hosts.
The trainer itself is GPU-bound (kernels do the work), so 6 vCPU is
plenty for the orchestration host process.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reasons:
- h100-sxm: Scaleway account quota is 0/0 for the SXM instance type
(cp_servers_type_H100_SXM_2_80G). The pool was perpetually trying
to maintain size=1 with a creation_error node, which JAMMED the
cluster autoscaler. That blocked L40S scale-up entirely during
today's alpha-perception cluster run.
- h100x2: more expensive than current workloads justify. Every
training path (alpha perception, future PPO) fits on either the
single-GPU H100 or L40S.
Removed:
- Two `scaleway_k8s_pool` resources from infra/modules/kapsule/main.tf
- Six variables (enable + type + max_size for each pool)
- Two outputs (pool_id for each)
- Corresponding inputs in infra/live/production/kapsule/terragrunt.hcl
The live cluster has the SXM pool stuck node manually deleted via
`scw k8s node delete` (this commit-session); the pool resource itself
will be destroyed on next `terragrunt apply`.
Post-cleanup pool inventory:
- platform (DEV1-L × 3)
- ci-training-h100 (H100-1-80G, max 1) <- regular single-GPU
- ci-training-l40s (L40S-1-48G, max 1) <- primary training target
- ci-compile-cpu (POP2-HC, max 4)
- ci-compile-cpu-hm (POP2-HM, max 1)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The actual Kapsule autoscaler_config (infra/modules/kapsule/main.tf
lines 46-52) is scale_down_delay_after_add="10m" +
scale_down_unneeded_time="10m". Combined, a freshly-provisioned node
won't be eligible for scaledown for 10m + another 10m unneeded
before action, so effective grace window is ~20m.
Update the comment in the warmup-gpu template to cite the actual
config rather than the speculative "15m" value. Behavior is
unchanged — the warmup pod still exits immediately and relies on
the grace window to keep the node warm for train.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Removed the 30s sleep. The warmup pod's purpose is to make the L40S
pool autoscaler scale 0 → 1; once the pod lands on the new node
(Scheduled → Running → Succeeded), the node enters Scaleway Kapsule's
scaledown-grace window (~15 min). That single window covers the
entire range of ensure-binary durations (sccache-hit ~10s through
cold ~15 min), so train always lands on a hot node without holding
the warmup pod open.
Trimmed cpu request 100m → 50m and mem 64Mi → 32Mi: the pod runs
~one shell command then exits; tiny resource footprint = faster
scheduling and no kubelet noise.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a parallel warmup-gpu task that runs concurrently with
ensure-binary. The warmup pod is a tiny CPU-only alpine container
scheduled on the gpu-pool's nodeSelector — its presence triggers
cluster-autoscaler scale-up of the L40S pool. After a 30s sleep, the
warmup pod exits; the node enters Scaleway's scaledown grace window
(~10 min), so the train pod lands on a hot node without waiting for
autoscaler provisioning.
No GPU resource request on the warmup pod — that would serialise
warmup and train on the same GPU. nodeSelector + nvidia.com/gpu
toleration are sufficient to force placement on the L40S pool.
Expected savings: ~3-5 min per cold cluster submission. First run
(this session, alpha-perception-4vl7c) showed compile took 113s
(sccache warm) with serial GPU provisioning following; subsequent
submissions should overlap the two stages.
DAG topology:
ensure-binary ──┐
├──> train
warmup-gpu ────┘ (only depends on ensure-binary; warmup is
fire-and-forget infrastructure)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per user direction "no gating, this is the new default": the stacked
Mamba2 -> CfC -> heads design is THE production architecture. There's
no competing-baseline comparison to run. Validation reduces to normal
training metrics (per-horizon val AUC, train loss curve, sanity floor
of >0.5 AUC).
Deletions:
- crates/ml-alpha/src/gate/cfc_vs_mamba2.rs (gate verdict logic)
- crates/ml-alpha/src/gate/mod.rs
- crates/ml-alpha/examples/alpha_gate.rs (gate runner binary)
Renames:
- crates/ml-alpha/src/gate/auc.rs -> crates/ml-alpha/src/eval/auc.rs
- lib.rs: pub mod gate -> pub mod eval (gate implied comparison;
eval doesn't)
Spec amendments:
- Drop the "Gate baseline strategy" amendment (committed earlier
this session)
- Reframe the stacked-architecture amendment as a "decision" not a
"gate"; production path is unambiguous
- Reframe Section 4 "Validation gate: CfC must meet Mamba2" -> just
"Validation: per-horizon val AUC" with the >0.5 sanity floor
Doc cleanups: stale "Mamba2 gate baseline" mentions in build.rs and
pinned_mem.rs replaced with neutral wording. The Argo template
comment about "downstream gate consumption" becomes "for monitoring".
Test status: all 26+ ml-alpha tests pass. AUC tests (6/6) still pass
under the eval:: namespace.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Defines the Mamba2-only baseline for the stacked-vs-baseline gate
verdict as an ablation of the SAME PerceptionTrainer (a --bypass-cfc
flag), not a separate model. Apples-to-apples; same data window,
same hyperparameters, same code path. The only difference is whether
the CfC step is in the loop.
Three ablation options evaluated:
1. --bypass-cfc flag (recommended): Mamba2 -> heads directly
2. --mamba2-state-dim 2 (crippled Mamba2, CfC stays)
3. Frozen CfC initialized to identity (no code branch needed)
Option 1 wins on clarity: it answers "is CfC additive on top of
Mamba2" unambiguously, with the same Mamba2 capacity and same
training regime in both arms.
Concrete next-session work documented (1-2 hours):
- PerceptionTrainerConfig.bypass_cfc: bool + step() branch
- alpha_train --bypass-cfc CLI flag
- alpha-perception-template.yaml workflow parameter + bash branch
- submit both runs, fetch summaries, alpha_gate, commit verdict
gate_verdict logic unchanged — the cfc/mamba2 naming in the report
becomes stacked/bypass at the binding layer; the verdict math is
generic.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Argo WorkflowTemplate at infra/k8s/argo/alpha-perception-template.yaml
runs the stacked Mamba2 -> CfC -> heads PerceptionTrainer on a single
L40S in fr-par-2. Two-stage DAG:
ensure-binary (ci-compile-cpu pool, sccache-backed cargo build of
alpha_train example, SHA-keyed binary cache under
/data/bin/$SHORT_SHA/)
train (ci-training-l40s pool, runs the cached binary against
/data/futures-baseline/mbp10 with predecoded sidecar
cache at /feature-cache/predecoded, writes
alpha_train_summary.json to
/feature-cache/alpha-perception-runs/$SHA/)
Defaults mirror the validated synthetic-overfit smoke config:
epochs=5, seq_len=32, mamba2_state_dim=16, lr_cfc=3e-3,
lr_mamba2=1e-3, n_train_seqs=8000, n_val_seqs=1000, seed=0x4242
Submission script scripts/argo-alpha-perception.sh wraps argo submit
with the standard L40S/H100 cuda-compute-cap mapping. --watch
follows logs.
Workflow nodeSelector pinned to fr-par-2 (consistent with the cluster
topology constraint). ttlStrategy 1h after completion;
activeDeadlineSeconds 4h cap (well above expected ~30-90 min wall).
This is the cluster entrypoint for the stacked perception design.
Once it lands a summary on MinIO, the gate runner (alpha_gate, Task
17) can consume it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The merged Mamba2 -> CfC -> heads design from the 2026-05-16 spec
amendment supersedes the CfC-alone path. Removing the old CfC-only
PerceptionTrainer and renaming the stacked Mamba2CfcTrainer to
PerceptionTrainer (one trainer, clean naming).
Deletions:
- src/trainer/perception.rs (the OLD CfC-only trainer)
- tests/perception_overfit.rs (CfC-only smoke)
- tests/perception_debug_dump.rs (CfC-only trajectory print)
- tests/stacked_overfit.rs (replaced by perception_overfit pointing
at the renamed module)
Renames:
- src/trainer/stacked.rs -> src/trainer/perception.rs
- Mamba2CfcTrainer -> PerceptionTrainer
- Mamba2CfcTrainerConfig -> PerceptionTrainerConfig
- tests/stacked_overfit.rs content -> tests/perception_overfit.rs
CLI rewrite:
examples/alpha_train.rs now drives the stacked PerceptionTrainer.
Per-step inputs are sequences (Vec<Mbp10RawInput>) of length
seq_len; labels come from the LAST position of the window
(per-horizon). Flags: --seq-len, --mamba2-state-dim, --lr-cfc,
--lr-mamba2 (no more --n-hid since hidden_dim is fixed at 128 to
match Mamba2 and CfC by design).
Test status:
- 64 GPU tests pass on local sm_86 (31 lib unit + 33 integration)
- synthetic-overfit (rebranded perception_overfit): 250 steps,
initial=0.5951 -> final=0.1917 (68% drop, well above 40% gate)
- All bit-equiv / finite-diff / invariant tests still PASS
- One ignored test: the gate_artifact integration (waiting for
cluster-trained summary inputs)
The cluster gate (Task 18) now compares stacked-trained AUC vs a
Mamba2-baseline AUC (TBD: stacked vs a simpler "Mamba2 only" config
or an external reference baseline).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Realizes the 2026-05-16 spec amendment merging Mamba2 + CfC into one
stacked architecture (vs the original "compete via gate" framing).
Forward chain:
snap_features × seq_len
-> window pack [1, seq_len, FEATURE_DIM]
-> Mamba2Block.forward_train -> (logit, cache.h_enriched [1, hidden_dim])
-> cfc_step(x=h_enriched, h_old=0) -> h_new
-> heads -> probs [5]
-> BCE(probs, labels)
Backward chain:
BCE -> grad_probs
-> heads_backward -> grad_h_new + grad_W_heads, grad_b_heads
-> cfc_step_backward -> grad_W_in, grad_W_rec, grad_b + grad_x (=grad_h_enriched)
-> Mamba2.backward_from_h_enriched(&cache, &grad_h_enriched_tensor)
-> Mamba2BackwardGrads (full 9-tensor gradient set)
Optimizers (6 total):
- 5 CfC AdamWs (W_in, W_rec, b, heads_w, heads_b) — reused from
PerceptionTrainer's per-param-group pattern
- 1 Mamba2AdamW for all 9 Mamba2 parameter tensors (existing
implementation in mamba2_block.rs)
Synthetic-overfit on constant +1 direction (seq_len=16, state_dim=8,
lr_cfc=3e-3, lr_mamba2=1e-3, 250 steps):
initial_avg=0.5951 → final_avg=0.1917 (68% drop, well past 40% gate).
Monotone descent at all 5 progress checkpoints.
Architectural note (v1): CfC runs with h_old=0 each step (no inter-
step recurrence). With h_old=0, the CfC layer is effectively per-cell
tau-scaled tanh FC. Inter-step CfC state (h_old carrying between
calls) is a v2 extension once the cluster gate validates the v1
foundation.
The cluster gate (Task 18) now has the actual stacked production
trainer to deploy, not a CfC-alone-vs-Mamba2-alone bench.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds grad_x[k] = sum_i d_pre[i] * W_in[i,k] computed by thread 0 of
the cfc_step_backward kernel (after the existing __syncthreads in
the shared-mem sd_pre relay). Required by the stacked Mamba2 -> CfC
design: Mamba2.backward_from_h_enriched needs grad on h_enriched,
which is the CfC's "x" input in the stacked topology.
For the existing CfC-only PerceptionTrainer (x = snap_features, no
upstream learnable layer), grad_x is computed but discarded into a
preallocated buffer.
backward_finite_diff tests still pass (4/4) — the new arg is the
14th positional kernel arg; existing callers updated. perception_
overfit smoke still passes (loss 0.5669 -> 0.0665 in 200 steps).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
GateReport + GateVerdict + load_summary in src/gate/cfc_vs_mamba2.rs.
Verdict criterion per spec Section 4 (with 2026-05-16 stacked
amendment): CfC AUC >= Mamba2 AUC - tolerance at every horizon.
Default tolerance 0.01.
alpha_gate binary takes two AlphaTrainSummary JSON paths (one per
backbone, generated by alpha_train), runs the verdict, emits
phase_a_gate.json with the full report + verdict, exits 0/1.
Tests (5/5 lib unit):
- PASS when CfC >= Mamba2 everywhere
- PASS within tolerance (CfC 0.005 below)
- FAIL at long horizon (h=1000, 6000)
- FAIL at short horizon (h=30)
- delta signs correctly track relative performance
For the cluster gate run (Task 18), the Mamba2 baseline summary is
generated by an existing/separate Mamba2 trainer pass on the same
data window. Apples-to-apples comparison requires identical train +
val seeds and quarters.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLI wraps PerceptionTrainer + MultiHorizonLoader for end-to-end
training. Per-epoch loop:
- train: stream sequences from MultiHorizonLoader, step per
position with reset_hidden_state (K=1 BPTT), accumulate train
loss
- val: separate loader on disjoint seed, accumulate (probs,
labels) per horizon, compute Mann-Whitney U AUC
Emits alpha_train_summary.json with final train loss + per-horizon
val AUC for downstream gate consumption.
Adds PerceptionTrainer::last_probs() — slow-path readback of the
most recent forward's probs. Used by the eval loop to capture
per-position predictions for AUC.
Args: --mbp10-data-dir --predecoded-dir --out --epochs --n-hid
--seq-len --lr --n-train-seqs --n-val-seqs --seed (all with sane
defaults from spec Section 4 Phase A).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Slow-path CPU scalar AUC over (probs, labels) vectors. Mann-Whitney U
formulation with average-rank tie handling. NaN labels filtered
(mirrors generate_labels masking semantics).
Tests (6/6 lib unit):
- perfect separation -> 1.0
- perfect anti-separation -> 0.0
- random uniform pairs -> ~0.5 (within 0.05)
- NaN labels filtered out before computation
- empty class -> 0.5 (defensive fallback)
- all-tied scores -> 0.5 (average-rank correctness)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per user feedback no-phase-naming. Documents the validated-end-to-end
state (loss 0.5669 -> 0.0665 in 200 steps) and the init-sensitivity
caveat for tiny smoke models.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Resolves Task 13 — the synthetic-overfit divergence I thought was a
wiring bug was actually init-sensitivity on the n_hid=32 toy. With
seed=0x4242 + lr=3e-2 + constant +1 direction + 200 steps + reset_
hidden_state per sample, the trainer converges loss 0.5669 -> 0.0665
(88% drop, well under the 60% gate threshold).
The 200-step weight trajectory (debug_long_horizon_weight_trajectory)
shows monotone descent:
step 0: loss=0.6932 hb[0]=0.030 hw[0,0]=-0.124
step 50: loss=0.2332 hb[0]=1.164 hw[0,0]= 0.997
step 100: loss=0.1301 hb[0]=1.599 hw[0,0]= 1.412
step 190: loss=0.0747 hb[0]=1.999 hw[0,0]= 1.777
Heads weights drive monotonically into the correct sigmoid tail.
The chain is sound:
- heads_backward finite-diff at 1% relative
- cfc_step_backward finite-diff at 5% relative
- BCE forward+backward at 5% relative finite-diff
- AdamW invariants (zero-grad + wd, descent on g=theta)
- Graph A capture bit-identical to sequential
- end-to-end overfit on constant +1 = 88% loss drop in 200 steps
Removed the #[ignore] + the speculative "wiring bug" doc comment.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per user feedback "no phase naming, give proper naming to files and
functions": renames src/trainer/phase_a.rs -> perception.rs,
src/data/phase_a_loader.rs -> data/loader.rs, and the corresponding
types (PhaseATrainer -> PerceptionTrainer, PhaseALoader ->
MultiHorizonLoader, PhaseAConfig -> MultiHorizonLoaderConfig,
PhaseASequence -> LabeledSequence). Test files renamed in lock-step.
Adds PerceptionTrainer.step() — full end-to-end forward + heads
backward + cfc_step_backward (K=1 truncated BPTT) + 5 AdamW param
groups (W_in, W_rec, b, heads_w, heads_b). reset_hidden_state()
zeros h_old between independent samples.
KNOWN ISSUE — synthetic-overfit smoke (tests/perception_overfit.rs)
does NOT yet show loss shrinkage on the 200-step budget:
initial_avg=0.6914, final_avg=0.6955 (random-baseline ln(2)=0.693)
The kernels are individually correct (heads_bwd + cfc_bwd finite-diff
at 5% rel, AdamW invariant ‖θ‖ 40->1 in 200 steps). The end-to-end
chain doesn't converge — most likely due to half the CfC cells having
near-1 decay from log-uniform tau init on a zeroed hidden state, so
only the fast cells carry signal. Fix candidates for next session:
- tau init narrower / per-task tuned for the smoke
- longer step budget (1000+) with adjusted LR
- validate end-to-end with explicit print of grad/probs across iters
Task 13 is NOT complete (the gate criterion isn't met). Subsequent
work continues from here.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
multi_horizon_heads_backward: sigmoid + linear chain rule. One block,
HIDDEN_DIM=128 threads. Computes grad_w, grad_b, grad_h_in. No
atomicAdd; per-thread accumulation only.
cfc_step_backward: truncated K=1 BPTT through one CfC time step.
Forward pre/decay/tanh recomputed inside the kernel; emits grad_w_in,
grad_w_rec, grad_b, grad_h_old. tau is held frozen (structural
log-uniform init per Hasani 2022; backprop through tau deferred to
Phase A v2 if the gate needs it). Uses dynamic shared memory for the
d_pre relay between threads (size = 2 * n_hid * 4 bytes).
Tests (4/4 on sm_86) validate via on-GPU finite-difference:
- heads grad_h vs forward(h±eps) → matches at eps=1e-3, rel<=1%
- heads grad_b vs forward(b±eps) → matches at eps=1e-3, rel<=1%
- cfc grad_b vs forward(b±eps) → matches at eps=1e-3, rel<=5%
- cfc grad_h_old vs forward(h_old±eps) → matches at eps=1e-3, rel<=5%
CPU is not the reference (per feedback_no_cpu_test_fallbacks.md). The
kernel is the truth; numerical perturbation validates the analytic
gradient against the kernel's own forward.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reuses ml-features::predecoded::load_or_predecode_mbp10 (no cycle —
ml-features doesn't depend on ml-alpha). Yields seq_len-sized windows
of Mbp10RawInput plus 5-horizon binary labels via
multi_horizon_labels::generate_labels.
Per-snapshot prev_mid / prev_ts_ns / trade_signed_vol come from the
prior snapshot in the source stream (not from the anchor), so the
CfC trunk sees a continuous-time signal across the entire seq.
Labels: NaN at edge positions (no forward window) or tied prices;
BCE kernel masks these (Task 10).
Tests:
- loader_errors_on_missing_root: passes (1/1 inline)
- loader_yields_seq_with_valid_labels: --ignored, runs at gate time
with FOXHUNT_TEST_DATA set
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Captures snap_feature_assemble -> cfc_step -> heads -> projection into
a single replayable graph. Scalars (dt_s, ts_ns, prev_mid, ...) are
frozen at capture time per cudarc 0.19 semantics; the trunk
re-captures when those change. A follow-up task moves scalars into a
device-resident buffer for cross-step replay stability.
Key learning: cudarc's default event-tracking creates cross-stream
dependencies that begin_capture rejects with
CUDA_ERROR_STREAM_CAPTURE_ISOLATION. Pattern (from crates/ml/.../
fused_training.rs): bracket begin/end_capture with
context.disable_event_tracking() / enable_event_tracking(). Mode
remains CU_STREAM_CAPTURE_MODE_RELAXED. Pre-allocate MappedF32Buffer
staging slots as struct fields (host-malloc during/around capture is
also a trigger).
The captured forward writes h_pong directly (no ping-pong swap inside
the captured region — the swap mutates pointer identity which would
invalidate captured kernel args). Heads and projection both read
h_pong.
Tests (3/3 on sm_86):
- graph_a_replay_matches_sequential: captured replay output equals
sequential dispatch on same input at eps<=1e-5 (probs) / 1e-4 (proj)
- graph_a_replay_is_deterministic: 3 consecutive replays produce
bit-identical output
- graph_a_replay_outputs_finite: probs in [0,1], proj finite
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
bce_loss_multi_horizon: fused forward+backward, block tree-reduce (no
atomicAdd), NaN labels masked (drop). Loss = mean over valid; grad =
(p-y)/(p(1-p)) scaled by 1/N_valid.
adamw_step: element-wise AdamW with weight decay; one thread per param.
Tests pass on sm_86:
BCE (4/4): positive+finite loss, near-zero loss when probs match
labels, analytic grad matches GPU-computed finite-difference at
eps=1e-3 / max_relative=5e-2 across 5 perturbation points, NaN
labels mask grad and contribute zero to loss/N_valid.
AdamW (4/4): zero-grad moves param only by weight-decay, positive
grad decreases param, step counter increments, repeated descent
on grad=theta drives ‖θ‖ from 40 to <1 in 200 steps.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CfcTrunk owns weights, ping-pong hidden buffers, and pre-allocated
per-step scratch (snap features, probs, projection output). Modules
and CudaFunction handles cached at new_random so the hot path
avoids reload. forward_snapshot dispatches snap_feature_assemble ->
cfc_step -> heads -> projection sequentially; Graph A capture (Task 11)
will fold these into a single launch.
The Mamba2 prefix (per 2026-05-16 spec amendment) is added in a
follow-up task before Graph A capture.
Tests (5/5 on sm_86):
- probs in [0,1] across all 5 horizons
- hidden state changes after forward
- probs + proj are finite
- layer-norm proj has near-zero mean
- 50-step run leaves hidden finite
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Single-block 8-thread kernel; thread j computes its own 128-dim dot
product, then thread 0 computes block-wide mean/var, then each thread
applies the per-output affine layer-norm. No atomicAdd; reductions are
single-thread (8 elements — negligible cost).
Tests (5/5 on sm_86) assert:
- layer-norm zero-mean output under identity gain
- layer-norm unit-variance output under identity gain
- ln_bias shifts mean uniformly
- ln_gain scales variance (var = gain^2)
- finite output under zero input (variance clamp activates)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per-horizon P(up) at h ∈ {30, 100, 300, 1000, 6000} snapshots forward.
Single-block 5-thread kernel; each thread is its own 128-dim dot
product + sigmoid. No atomicAdd.
Tests (5/5 pass on sm_86) assert invariants only:
- sigmoid output ∈ [0, 1] for all heads
- zero weights + zero bias → 0.5 exactly
- bias = +20 → saturates near 1
- bias = -20 → saturates near 0
- per-head independence (mixed-bias configuration)
Addendum updated to explicitly state no-CPU-mirror discipline per
feedback_no_cpu_test_fallbacks.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Hasani 2022 closed-form CfC recurrence; one thread per hidden unit, no
atomicAdd. Tests assert algebraic invariants (dt=0 -> identity, zero
weights -> h_old * decay, large tau -> h_old preserved, output bound).
Also removes src/cfc/oracle.rs and replaces snap_feature bit-equiv
test with property assertions per feedback_no_cpu_test_fallbacks.md.
CPU mirrors are bug-locks; validation is now via known synthetic
inputs + analytical relations on the GPU output.
12 tests pass on local sm_86 (7 snap_feature invariants + 5 cfc_step
invariants).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per-snapshot 32-dim feature vector (mid log-return, spread, depth, OFI,
trade-flow, dt). Single-block single-thread kernel; uploads via
MappedF32Buffer DtoD into CudaSlice per the addendum Pattern 3.
Bit-equiv tested CPU vs GPU at eps<=1e-5 over (synthetic input,
reserved-slots-are-zero, zero-prev-mid edge case).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mid-execution architecture revision: Mamba2 stays as a sequence
encoder; CfC becomes the layer on top (replacing the Phase 1d.3 MLP
stacker). Gate becomes 'stacked AUC >= Mamba2-only stacker AUC at
every horizon' — proves the CfC layer is additive, rather than CfC
alone beating Mamba2 alone.
Plan 1 kernels (cfc_step, heads, projection, BCE, AdamW, Graph A)
are unchanged. Only CfcTrunk's forward path gains a Mamba2 prefix
that consumes the snapshot stream and emits a 128-dim h_mamba which
CfC reads. The Mamba2 kernel (mamba2_alpha_kernel.cubin) is already
in the build.
Option B (parallel + fused dual-stream) is documented as the Plan 2
fallback if the stacked gate fails.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Typed ABIs (#[repr(C)] SnapshotPayload, FillPayload) backed by
pinned_mem::MappedF32Buffer. write_volatile is the only hot-path
CPU->GPU pathway; GPU reads via device_ptr() with zero HtoD.
Tests pass on local sm_86 (3/3): snapshot round-trip, fill round-trip,
pre-allocation invariant (device pointer stable across writes).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Holds perception outputs (slots 0..13) on-device; slow-path write/snapshot
go through MappedF32Buffer DtoD per the htod/htoh discipline. Slot
semantics documented in design spec Section 7.
Also deletes examples/alpha_mamba_baseline.rs which Task 1 left orphaned
(used the deleted eval + training modules). Task 17 will rebuild the
Mamba2 baseline trainer path inside gate/cfc_vs_mamba2.rs against the
new Phase A loader.
Tests pass on local sm_86 (3/3): round-trip one slot, 32-slot capacity,
multi-slot independent writes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Cargo.toml: drops gbdt; adds memmap2 + approx; keeps ml-core only
(cannot depend on ml: would cycle since ml depends on ml-alpha for
the Mamba2 gate baseline).
build.rs: compiles 7 cubins (mamba2_alpha + 6 new placeholders)
with -O3 --use_fast_math --ftz --fmad. Skips kernels whose source
isn't present yet so partial check-ins work. Every env::var paired
with rerun-if-env-changed per the canonical build pearl.
src/pinned_mem.rs: local copy of MappedF32Buffer (mirrors
ml::cuda_pipeline::mapped_pinned::MappedF32Buffer). Drives the only
permitted CPU<->GPU path per feedback_no_htod_htoh_only_mapped_pinned.
Eventually the move-to-ml-core refactor will deduplicate; out of
scope for the Phase A branch.
Addendum: updates the import path to ml_alpha::pinned_mem.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Plan 1 was written against an older cudarc device-centric API. cudarc 0.19
moved alloc/launch ownership to the stream (partly for CUDA Graph capture
hygiene). This addendum pins:
- MlDevice -> CudaContext -> CudaStream construction
- Cubin load + module + function caching
- MappedF32Buffer staging -> DtoD-async -> CudaSlice (canonical CPU->GPU)
- Slow-path readback via DtoD into a staging MappedF32Buffer
- launch_builder(&func).arg(...).launch(cfg) idiom
- IsvBus and MappedPinnedSnapshotSlot/FillSlot using the real
MappedF32Buffer API (host_slice_mut, read_all, dev_ptr field)
- CUDA Graph A capture via stream.begin_capture / end_capture / instantiate
Plan 1 kernel .cu source, CPU oracles, finite-diff thresholds, smoke
criteria, and gate logic are unchanged. Only Rust binding code uses
these patterns.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Removes mlp/training/eval/backtest/metrics_detail/calibration and the
old example trainers. Preserves multi_horizon_labels, purged_split,
fxcache_reader (Phase A data path), mamba2_block (gate reference).
Subsequent commits populate cfc/heads/isv/pinned/trainer/data/gate.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two complementary additions to validate the minute-horizon alpha
hypothesis at IBKR-realistic costs:
1. `alpha_baseline --decision-stride N`: emits a new action every N
steps; between decisions force action=0 (wait) so an open position
is held rather than re-decided per bar. Cuts per-bar trade counts
~stride× and removes the coin-flip overtrading. Local 2Q sweep
showed stride=200 + scaled training (8K episodes × 25 envs × H=1200)
flipped Sharpe at ¼-tick from -4.29 (per-bar, 3-fold mean) to +1.78,
with std collapsing from ±8.8 to ±1.15. Break-even cost moved from
<¼-tick to ~1-tick — for the first time positive at IBKR-realistic
passive-execution frictions.
2. `alpha_train_stacker --max-rows N`: optional cap on bars consumed
from the fxcache. Used during local 2Q smoke (--max-rows 4M against
the 17.8M-row 9Q fxcache) to fit Mamba2 training on a 4 GB consumer
GPU; on the cluster (--no-cap) it sees all 9Q.
3. New Argo workflow `alpha-cv`: standalone template that compiles
alpha_train_stacker + alpha_baseline + alpha_fill_coeffs.json,
trains the stacker on the 9Q fxcache, then runs 9 sequential
walk-forward folds of alpha_baseline on disjoint 1.9M-bar windows
(one per quarter). Launcher script `scripts/argo-alpha-cv.sh`
mirrors argo-train.sh conventions.
The local 2Q test that motivated this commit is summarised inline in
the alpha-cv template comments; the verdict was "framing was the bug —
once decision cadence matches the multi-minute alpha horizon, the
strategy is positive at IBKR commission".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Scaleway BSSD PVCs (cargo-target-*, sccache-*, feature-cache-pvc,
bin-cache, training-data-pvc, test-data-pvc, platform service PVCs) are
single-zone-locked at create time. The cluster's node pools in main.tf
set `region` but no `zone`, so each autoscale event picks a zone
arbitrarily. When the autoscaler spins up a node in a zone that doesn't
match an existing PVC, scheduling fails with
"1 node(s) didn't match PersistentVolume's node affinity"
until the autoscaler eventually retries in the right zone. We hit this
on the HM pool creation (fixed manually with zone=fr-par-2) and again
on this commit's ensure-binary autoscale.
Track 1 (this commit): add `topology.kubernetes.io/zone: fr-par-2` to
every k8s.scaleway.com/pool-name nodeSelector entry across the 11 argo
workflow templates (34 entries total). The Kubernetes scheduler AND
cluster-autoscaler both honor topology keys when deciding placement /
provisioning — so future autoscale events will only spin up fr-par-2
nodes, and PVC binding is guaranteed.
Track 2 (future, structural): add `zone = "${var.region}-2"` to each
scaleway_k8s_pool in infra/modules/kapsule/main.tf so the pools never
provision in any other zone. Requires terraform-state cleanup (the
GitLab http backend currently has no states; the HM pool was created
out-of-band) and drain/recreate of any existing mixed-zone nodes —
deferred.
Verification: pool=N / paired=N coverage report shows 1:1 pool-name
to topology.kubernetes.io/zone entries in every file. kubectl apply -f
of all 11 templates returned "configured" for each.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
After the cluster's silent-failure incident (75 min between 'alpha
pipeline inputs' and exit, no intervening log line), instrument the
parallel path so any future hang or panic localises to the specific
chunk that died. Each chunk emits start + done lines with chunk_idx,
emit range, warmup_start, row count, and elapsed seconds; the wrapper
also logs dispatch (chunk count + threads + chunk_size) and overall
completion.
Local 1Q smoke (16-thread box):
- dispatching 16 chunks (chunk_size=35485, warmup_bars=2000)
- chunk 0 (cold-start, no warmup shortcut): 19.2s
- chunks 1-15: 13.9-16.5s
- all-chunks-complete log fires before fxcache write
Cost: ~17 info lines per precompute run — negligible. Worth the
observability when the pipeline takes minutes and any future kill
needs root-cause.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Cluster's 9-quarter precompute_features hung silently after the
"alpha pipeline inputs" log line and was killed at ~75 minutes — local
1Q profiling showed the alpha pipeline is strictly single-threaded
(extract_alpha_features is one for-loop over n_output bars with no
rayon usage), so 9Q would have needed ~72 min on one CPU even with no
external interference. That's a kill window large enough to be brittle
under any transient kubelet/argo signal.
Fix: split the emit range across `rayon::current_num_threads()` chunks.
Each chunk gets fresh aggregator state and pre-rolls 2000 leading bars
without emission so Hawkes excitation / Bouchaud EMA / frac-diff FIR /
LOB PCA covariance / microprice EMA / spread_decomp running stats are
saturated before the first emitted row. Trade-feed semantics preserved
via `trades.partition_point` seeding per chunk — each trade still
visits exactly one aggregator chain.
Local 1Q ES benchmark (9-core box):
- before: 2:19 wall, 101% CPU (1 core)
- after: 0:27 wall, 915% CPU (9 cores)
- 5.1× speedup; same row count + Alpha dim 134 + 468MB output
Extrapolated 9Q on cluster's 32-vCPU HM pool: ~4-5 min for the alpha
portion (vs ~72 min before). Well under any plausible kill window.
Numerical caveat: not bit-identical to a fully-sequential run for
chunks k > 0. Aggregators initialise at default state instead of
carrying real history across the chunk seam; the 2000-bar warmup
refills Hawkes's 500-event history twice over and saturates the
longer-memory EMAs, so post-warmup drift is bounded by floating-point
ε. The two existing in-crate unit tests (`test_extract_alpha_features_*`)
still pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Flamegraph of precompute_features on 1Q ES showed 62% of CPU time in
zstd decompression, 6% in DBN FSM parsing, and only 2% in the actual
feature math — single-threaded zstd was the bottleneck, not compute.
Two fixes:
1. Per-quarter parallelism on the volume-bar trades loop (was sequential
`for file in &trade_files`); brings it in line with the OFI path that
already used par_iter.
2. Predecoded sidecar cache in `crates/ml-features/src/predecoded.rs`:
first call to a `.dbn.zst` writes a bincode'd Vec<Mbp10Snapshot> or
Vec<DbnTrade> under `<output_dir>/predecoded/`. Subsequent calls
deserialize the sidecar and skip zstd entirely. An mtime+size header
self-invalidates the sidecar when the source changes — no manual
flush needed when a quarter is re-downloaded.
Local 1Q ES results:
- cold (writes sidecar): 40.7s (was 39.3s; +1.4s for write)
- warm (HIT): 4.7s (8.7× faster)
- zstd in flat perf: 62% → 0% of CPU samples
- sidecar disk per Q: ~150MB
The sidecar layer also auto-dedupes within a single run: the OFI section
re-loads trades, but the second call hits the sidecar that the
volume-bar section wrote moments earlier.
CLI: `--rebuild-predecoded` purges sidecars for cold-path testing or
after a wire-format change to Mbp10Snapshot / DbnTrade. Sidecars also
self-invalidate on format-version mismatch so old caches are skipped
silently rather than mis-deserializing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two 9-quarter precompute_features runs OOM-killed on the existing
ci-compile-cpu pool (POP2-HC-32C-64G, 56Gi cgroup limit):
- train-wq8b8: exit 137 ~12s after "OFI computed" at ~57Gi
- train-2l6p4: exit 137 ~12s after "OFI computed" at ~52Gi peak
(despite the into_iter + drop(feature_vectors) +
normalize_in_place refactors landed in a27cb40a9 + 623ebfcf7,
which trimmed ~12GB of avoidable retention)
The remaining ~52-60GB peak is the irreducible working set at the
post-OFI / pre-alpha-pipeline step:
- 9-quarter MBP-10 snapshots (~10GB)
- 199M front-month trades as Mbp10Trade (~13GB)
- 17.8M bars × features + targets + OFI buffers (~14GB)
- alpha_snapshots (move-handoff from all_snapshots, ~10GB)
- feature/target Vecs (~7GB) + Rust allocator overhead
Adds a dedicated high-memory pool sized for this rare path:
- New scaleway_k8s_pool.ci_compile_cpu_hm (POP2-HM-32C-256G,
32 vCPU + 256GB RAM) with size=0 + min_size=0 autoscaling. Costs
zero when idle; autoscaler provisions one node when a pod targets
`nodeSelector: ci-compile-cpu-hm`.
- New variables: enable_ci_compile_cpu_hm_pool,
ci_compile_cpu_hm_type (default POP2-HM-32C-256G),
ci_compile_cpu_hm_max_size (default 1).
- terragrunt.hcl: enable the pool, max_size=1.
- train-template.yaml: ensure-fxcache nodeSelector pinned to
ci-compile-cpu-hm; memory limit raised 56Gi → 200Gi. The
ci-compile-cpu pool stays as the standard CI compile target for
ensure-binary + every other CPU-heavy task.
Apply with:
cd infra/live/production/kapsule
terragrunt apply -target=module.kapsule.scaleway_k8s_pool.ci_compile_cpu_hm
kubectl apply -n foxhunt -f ../../../../infra/k8s/argo/train-template.yaml
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previous workflow train-4qwtc hit memory-pressure thrash (~56Gi
cgroup.current sitting at the 56Gi pod limit, kernel reclaim
hammering page cache) right after OFI completed on the 9-quarter
17.8M-bar dataset. Two refactors reduce peak by ~12GB:
(1) walk_forward.rs: new `normalize_batch_in_place(&mut features)`
that rewrites the slice in place. The previous `normalize_batch`
`.collect()`s a new Vec — at this dataset size that's a
transient ~6GB peak while both pre- and post-normalised arrays
are alive.
(2) precompute_features.rs:
- call `normalize_batch_in_place` instead of the rebinding form.
- explicit `drop(feature_vectors)` after copying the slice into
`features` — `feature_vectors` would otherwise stay alive
until end-of-main shadowing the ~6GB allocation through
every downstream step.
Combined with the prior `t.into_iter()` refactor (a27cb40a9), the
peak transient drops from ~56GB to ~44GB — well under the 56Gi pod
limit on the existing ci-compile-cpu pool (POP2-HC-32C-64G).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Cleanup of compiler warnings flagged by both local cargo check and the
cluster ensure-binary log. Per `feedback_no_hiding`, every site is
either deleted or wired up — no #[allow] suppressions.
Lib (5 sites):
- gpu_backtest_evaluator.rs:34 — drop unused DevicePtrMut.
- gpu_dqn_trainer.rs:49 — drop unused DevicePtrMut (8 device_ptr_mut
calls don't need the trait import in current cudarc). Line 19852:
drop unnecessary parens around `b * sh2`.
- training_loop.rs:20 — drop unused DevicePtrMut; unbrace single-
symbol use at 5766.
- state_reset_registry.rs:4 — delete the 10-symbol use-block of slot
constants. Names appear in description strings (documentation only),
symbols are never referenced.
Examples (3 sites):
- alpha_dqn_h600_smoke.rs:181, 186 — drop COL_RAW_CLOSE, FEAT_DIM,
FillCoeffs, FillModel imports.
- alpha_baseline.rs:79 — delete unused MappedI32::read. Batched path
uses read_all for N-element action readback; the single-element
method was leftover from the pre-batched legacy path.
Lib + examples now have zero removable warnings. The remaining
unsafe_block lints (each cudarc kernel launch needs unsafe) are
structural and not actionable under the project's -W unsafe-code
policy.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The 9-quarter precompute_features run OOM-killed at ~56Gi on the
ci-compile-cpu pool (POP2-HC-32C-64G) on 2026-05-16. Root cause:
lines 672-684's `t.iter().map(...).collect()` borrows the source
DbnTrade Vec while building the Mbp10Trade Vec — both alive
simultaneously, transient peak ~25GB just from this transformation
for the 199M-trade dataset.
`.into_iter()` consumes the source element-by-element so the
allocation drops as the destination grows, capping peak at the
larger of the two Vecs (~15GB) rather than their sum.
Should let the 9-quarter precompute fit comfortably on the existing
64GB ci-compile-cpu pool without provisioning a high-memory node.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
