The real gateway foxhunt-gw (42653780) was an unmanaged orphan: kapsule state never had a gateway,
public-gateway state referenced a deleted gateway (7a7ffaa5) -> BOTH modules' terragrunt apply would
have created a duplicate gateway (risking cluster egress). Fix:
- kapsule module: removed the legacy gateway/dhcp/gateway_network resources (pre-refactor duplicate,
never in kapsule state; cluster only needs the private_network which stays). kapsule plan now clean.
- public-gateway module (canonical, IPAM mode): state rm the stale 7a7ffaa5 refs + imported the real
gateway/gateway_network/IP (42653780 / 661c8543 / dc419a10); set bastion_enabled=true to match the
live gw. public-gateway plan now 'No changes'. Live gateway untouched throughout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause: global.appConfig.terraformState was missing entirely from the GitLab Helm values, so the
chart rendered terraform_state.enabled=false + object_store disabled -> every /terraform/state API call
403'd ('Object Storage is not enabled for Terraform::StateUploader'), blocking terragrunt. State files
were intact in MinIO (foxhunt-gitlab-artifacts) the whole time. Added terraformState{enabled,bucket,
connection} mirroring artifacts/lfs/packages; helm upgrade -> revision 25 -> TF-state API now 200.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of the stuck cockpit deploy: ci-compile-cpu (32C/64G POP2-HC) hit a fr-par-2 zone stock-out
('Resource POP2-HC-32C-64G is out of stock') -> autoscaler ScaleUpTimedOut -> kaniko build Pending forever.
The big node was sized for the now-dormant Rust cargo compile + precompute_features; the Python fxhnt
cockpit build only needs ~2-6 vCPU. Downsized to POP2-4C-16G (in stock in fr-par-2, far cheaper, scale-to-zero).
Apply blocked in-session: cluster GitLab tokens lack 'api' scope for the TF state backend (403).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The fxhnt-cockpit deploy step applies manifests containing ServiceAccounts (dashboard/dagster), a PVC
(forward track), and a CronJob — none of which the ci-deploy Role permitted, so kubectl apply failed
partway. Also added 'watch' on deployments for 'kubectl rollout status'. Surfaced once the cockpit deploy
finally ran (it previously died at the MinIO log sink first). Rule: this Role must gain a grant for every
new resource KIND a deployed manifest introduces.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the web-dashboard upstream. Proxies peer-to-peer over the tailnet (the kernel-mode proxy can't
reach platform-pool ClusterIPs due to the 100.64/15 pod-CIDR ↔ Tailscale CGNAT overlap, but reaches tailnet
nodes fine). Live CM already patched; this keeps the repo in sync.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Switch from weekly to daily (weekday) evaluation for faster risk response — the leverage (vol-target
+ drawdown CB + corr de-risk) re-evaluates each day so a vol spike de-levers within a day vs a week.
Hysteresis (3% NLV) still gates actual trades, so turnover stays low (in-band days = no-ops). No
return impact; tighter risk only. Weekends excluded (US market closed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After full validation (in-cluster connect via 4004, paper account DU9600528, gates pass, clean logs
post NaN-fix), arm the CronJob to place paper orders. Triple-guarded against real money: paper
account + Gateway paper-mode + bot refuses non-DU without MULTISTRAT_ALLOW_LIVE_CONFIRMED.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pod-to-pod on 4002 reaches IB Gateway with the pod IP as source -> not in the trusted-IP allowlist
(default 127.0.0.1) -> TCP connects but API handshake times out. The gnzsnz image's socat on 4004
relays via localhost so IB Gateway trusts it. (Local port-forward worked on 4002 because it enters
the pod netns = localhost.) Egress netpol now allows 4002+4004; IB_PORT=4004.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Weekly in-cluster CronJob running multistrat_bot.py on IBKR paper via the existing ib-gateway
(connects to ib-gateway:4002 directly, no port-forward). Lean: code via ConfigMap (no image build),
python:3.12-slim + runtime pip. Includes PVC (state/HWM persistence across runs), egress
NetworkPolicy (ib-gateway:4002 + 443 for pip/Yahoo + DNS), and ib-gateway ingress patched to allow
the rebalancer. Schedule Mon 14:35 UTC (~1h after US open). MULTISTRAT_EXECUTE=false by default
(cluster dry-run); flip to true after validating logs. Drawdown CB + leverage cap + idempotency +
paper-guard enforced by the bot.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror of the FOXHUNT_USE_MULTI_HEAD_POLICY pattern (c5036af03).
Adds --band-enabled flag to argo-alpha-rl.sh and band-enabled
workflow parameter wired into the container env block.
Required for Phase 4-C cluster verification of the full Tier 3
no-transaction-band stack (commits e41a73208 → 65c328d3f).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds use-multi-head-policy workflow parameter (default "0") wired
into the container env block as FOXHUNT_USE_MULTI_HEAD_POLICY.
Adds --use-multi-head-policy flag to argo-alpha-rl.sh that passes
through to the workflow.
Enables Phase 2A-E cluster verification with the multi-head policy
mixture (commits 0b3e40150 → c8c81ab7e). Without this flag, the
cluster runs the legacy single-head path identical to pre-Phase-2A
behavior.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sets CUDA_COMPUTE_CAP dynamically from the GPU on the node. Clears
both ml-alpha and ml-core build caches. Captures build.rs stderr in
pod logs for debugging.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Remove: ReplayBuffer, Transition, NStepEntry, push_to_replay,
sample_and_gather, n_step_buffer, replay.rs, --diag-every flag.
PER call sites stubbed with todo!() — replaced by GPU PER in next commits.
Also fixes pre-commit hook to allow todo!() macro (per CLAUDE.md:
"todo!() macro is OK for runtime stubs") while still blocking
TODO/FIXME comment markers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- --diag-every N (default 100): skip device readback on non-diag steps.
Eliminates ~10 stream.sync per step → ~2× throughput at large batch.
- Fix PnL bug: was accumulating raw_rewards (shaped, all batches) instead
of rewards/scale on done steps only.
- Wire diag-every=100 in Argo template (diag JSONL every 100 steps).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
n-backtests default in Argo template: 16 → 128.
PER capacity: max(configured, 4 × b_size) so replay buffer is always
large enough for useful prioritized sampling at any batch size.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Prepared for next run after the γ=0.99 1M run completes:
- γ floor 0.99 → 0.995: horizon 100 → 200 steps (50 seconds).
Real ES directional moves (2-5 points) happen at this scale.
- PER 16384 → 32768: 2048 unique steps of replay depth. Supports
the 200-step γ horizon with margin.
- Min-hold 50 → 100 steps (25 seconds): commit to the full wave.
Short-hold penalty threshold matches.
Safe to push because raw-reward re-normalization eliminates scale
drift in the deeper buffer, and the ±2% scale clamp keeps targets
stable across the 2048-step replay window.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Stored raw_reward alongside scaled reward in Transition. At sample
time, consumer re-applies current reward_scale to raw_reward instead
of using the stale-scale stored reward. This eliminates off-policy
scale drift that caused q_pi_agree to collapse from 0.125 to 1e-22
over 40k steps with PER=32768.
PER capacity set to 16384 (1024 unique steps at b=16) — balances
replay depth against h_t representation staleness.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces the 4-step DAG (check-cache → compile on CPU → warmup GPU →
train on GPU) with a single pod on the L40S that does everything:
git fetch → incremental cargo build (~3s warm) → train.
Eliminates: separate compile node, node autoscale wait, binary
transfer, fxcache step. The ci-builder image has CUDA 13.0 devel +
Rust — compilation and training use the same CUDA libs.
The cargo-target-cuda PVC provides incremental build cache across
runs. LD_LIBRARY_PATH strips the stubs dir so real CUDA libs load.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The MBP-10 predecoded sidecars live on feature-cache-pvc at
/feature-cache/predecoded/, not on training-data-pvc. Points
--predecoded-dir to the correct PVC mount.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The ml-alpha branch uses alpha_rl_train (from ml-alpha crate) with
different CLI args than train_baseline_rl (from ml crate). Adds
model=alpha-rl case that:
- Builds ml-alpha --example alpha_rl_train
- Runs with --mbp10-data-dir, --predecoded-dir, --n-steps 50000,
--n-backtests 16 (matching the local smoke config)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
28 parallel alpha-feature chunks × ~3GB each = 84GB peak at merge.
Capping at 8 threads keeps peak at ~24GB (well within 96Gi limit)
while still saturating I/O. This is a one-time cost — after the
cache is populated on the PVC, subsequent runs skip entirely.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
With cargo-target-cuda PVC persisting across pods, CARGO_INCREMENTAL=1
gives true incremental builds: only recompile changed crates. A
single-crate ml-alpha change rebuilds in ~45s vs 9min full workspace.
sccache removed — it conflicts with incremental compilation and is
redundant when the target dir persists. The binary-by-SHA cache on
the training-data PVC (already implemented) provides the "skip
compile entirely on re-run" fast path.
Expected workflow time reduction:
Before: compile 9m + fxcache 18m + train ~30m = ~57m
After: compile <1m (incremental) + fxcache <1m (PVC cache) + train ~30m = ~32m
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The imbalance bar cache now writes to <mbp10_dir>/.cache/ on the PVC.
The ensure-fxcache step needs write access to persist the cache
across runs (was readOnly: true, causing silent cache-write failures).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two fixes for alpha-rl-9k9x6 (commit 3737feb66) findings:
(1) rl_pi_action_kernel.cu — per-action probability floor P_MIN=0.02
with renorm after softmax. Structural guarantee H(π_sample)
bounded below regardless of logits. 9k9x6 evidence:
- π collapsed to action 1 (100% sample rate) by step 2500
- action_entropy EMA → 0.001 by step 10000 (uniform=2.197)
- entropy_coef controller pegged at COEF_MAX without effect
because PPO update couldn't escape δ-function fixed point
- 0 trades closed in 50000 steps; position monotonically
accumulated to -45871 lots
With P_MIN=0.02, worst-case H(π_sample) ≈ 0.77 nats — well
above the observed collapse and well below uniform.
Per `pearl_blend_formulas_must_have_permanent_floor`.
(2) argo template + script — N_BACKTESTS default 1 → 16. The CLI
default lift in commit 3737feb66 was overridden by both the
argo script and the wftmpl `value: "1"`. 9k9x6 actually ran
b_size=1 despite being framed as the b_size=16 test.
Per `pearl_b_size_1_signal_starvation_blocks_q_learning`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Concurrent submissions at the same SHA (one workflow per fold_idx
for the walk-forward G8 gate) would overwrite each other's
eval_summary.json. Adds a /foldN suffix to the output path so the
aggregator can collect 3+ distinct eval_summary.json files from
/feature-cache/alpha-rl-runs/<sha>/fold0,fold1,fold2/.
Single-fold smokes (n_folds=1) still write to /<sha>/fold0/
directly — backwards-compatible for the prior smoke pattern, just
one level deeper than before.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the minimum-viable implementation of the R9 multi-fold G8 gate
per `pearl_single_window_oos_is_not_oos` ("a single window is NOT
out-of-sample"). The trainer can now:
1. Slice the MBP-10 file list into K equal-sized blocks
(`--n-folds K --fold-idx k`).
2. Train on blocks [0..=k] (passed to MultiHorizonLoader).
3. Run a separate eval phase of `--n-eval-steps` on block [k+1]
using a second loader instance.
4. Drain LobSim trade records gated by a pre-eval head checkpoint
so train-phase trades don't contaminate the eval summary.
5. Compute profit_factor + sharpe + drawdown via existing
`ml_backtesting::artifacts::compute_summary`.
6. Write `eval_summary.json` alongside `alpha_rl_train_summary.json`.
## Manual fan-out (this MVP)
The dispatcher (`scripts/argo-alpha-rl.sh`) gains three new flags
that thread through the Argo template into the CLI: `--fold-idx`,
`--n-folds`, `--n-eval-steps`. To run a 3-fold G8:
./scripts/argo-alpha-rl.sh --n-folds 3 --fold-idx 0 --n-eval-steps 200
./scripts/argo-alpha-rl.sh --n-folds 3 --fold-idx 1 --n-eval-steps 200
(With n_folds=3 the valid fold indices are 0 and 1 — the third block
is the eval window for fold 1. n_folds=K accepts fold_idx ∈ [0, K-2].)
Each submission produces one `eval_summary.json` at the resolved
output dir; the per-fold profit_factor is the value to aggregate.
Manual aggregation for now — automated DAG matrix fan-out + an
in-cluster aggregator pod is a follow-up commit. The aggregator
will mean ± SD the per-fold PFs and gate on `PF > 1.0`.
## What's NOT pure eval
The eval loop calls `step_with_lobsim` (same as train) — Adam steps,
PER updates, controller adaptations all still fire during eval. At
b_size=1 the per-step learning effect is small relative to the
train-phase-accumulated policy, so the eval PF approximates the
OOS performance of the train-end policy. A clean pure-eval mode
(forward + LobSim step only, no backward/Adam/PER) is a follow-up
architectural change; documented inline at the eval phase block.
## Default behaviour unchanged
`--n-folds=1` (default) skips the eval split entirely and uses all
files for training — identical to the prior single-window smoke.
The R9 prior smokes ran in this mode. Default `--fold-idx=0` and
`--n-eval-steps=0` keep prior smoke runs binary-compatible.
## Template + dispatcher changes
* `alpha-rl-template.yaml`: adds 3 new workflow parameters
(`fold-idx`, `n-folds`, `n-eval-steps`) and threads them into
the train container's `alpha_rl_train` invocation.
* `argo-alpha-rl.sh`: adds matching CLI flags with explicit
documentation of the multi-fold dispatching pattern.
## Verified gates
Local sm_86 build + dispatcher syntax clean. Tests unchanged
(the walk-forward path is exercised by cluster smokes, not unit
tests — the loader-slicing logic is straightforward index math).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Closes the rebuild plan's R8 scope: production runner shape for the
integrated RL trainer. Three artifacts wired end-to-end:
1. `crates/ml-alpha/examples/alpha_rl_train.rs` — clap CLI driving
`IntegratedTrainer::step_with_lobsim` against MBP-10 windows
loaded via `MultiHorizonLoader::next_sequence_pair` (R2) for
true `(s_t, s_{t+1})` adjacency. Per `feedback_mbp10_mandatory`,
`--mbp10-data-dir` is required — no synthetic-data fallback in
the production path.
2. `infra/k8s/argo/alpha-rl-template.yaml` — WorkflowTemplate
mirroring alpha-perception's DAG (check-cache → ensure-binary →
train; warmup-gpu parallel). Binary cache slot is
`/data/bin/<sha>/alpha_rl_train` (distinct from `alpha_train`
so the two binaries coexist at the same SHA).
3. `scripts/argo-alpha-rl.sh` — dispatcher with three rebuild-plan
guards baked in.
## Dispatcher guards (per the rebuild plan's feedback list)
`feedback_default_to_l40s_pool` (2026-05-09): default `--gpu-pool`
is `ci-training-l40s` (sm_89). H100 (sm_90) is opt-in for production
scale-up only. Cubins must match the device, so the dispatcher
derives `cuda-compute-cap` from the pool name and threads it into
the workflow params.
`feedback_argo_template_must_apply` (2026-05-21 canonical incident):
`argo submit --from=wftmpl/<name>` reads the cluster CRD, NOT the
on-disk YAML; unknown `-p` parameters silently no-op without a prior
`kubectl apply`. Dispatcher applies the local template BEFORE every
submission (overrideable via `--skip-template-apply` for the rare
case where you've already applied manually).
`feedback_push_before_deploy` (2026-05-20 canonical incident): the
in-cluster `ensure-binary` pod fetches source from `origin/<branch>`,
NOT the local working tree. Submitting before `git push` deploys the
last-pushed SHA, which can lag local diff by N commits. Dispatcher
verifies `git rev-parse HEAD == git rev-parse origin/<branch>` and
hard-errors with the explicit push command otherwise. Bypass via
`--skip-push-check` (only when intentionally deploying a previously-
pushed SHA via `--sha`).
## CLI: gate G8 (NaN abort)
Per `feedback_stop_on_anomaly` + `feedback_kill_runs_on_anomaly_quickly`,
the CLI checks every per-head loss (l_bce / l_q / l_pi / l_v / l_aux
/ l_total) for finiteness after each `step_with_lobsim` call.
Non-finite at any step → write summary with `nan_abort_step` set →
`process::exit(2)`. R9's cluster smoke tail-watcher kills the
workflow on the non-zero exit code, satisfying gate G8 from the
rebuild plan.
## CLI: knobs that ARE on the CLI
Structural / boundary parameters only (per
`pearl_controller_anchors_isv_driven`: every adaptive knob lives in
ISV, not CLI flags):
* `--mbp10-data-dir / --predecoded-dir / --out` — I/O paths.
* `--n-steps` — wall-budget control (1000 R9 smoke / 50k+ prod).
* `--seq-len / --n-backtests / --per-capacity` — structural
sizing. seq_len threads into the loader's multi-resolution
`1:<seq_len>` config; n_backtests into both LobSimCuda and
PerceptionTrainerConfig.n_batch.
* `--seed` — reproducibility per
`pearl_scoped_init_seed_for_reproducibility` (forks deterministic
sub-seeds for dqn / ppo / per).
* `--instrument-mode` — MBP-10 filter (all / front-month / id=N).
* `--gpu-idx` — CUDA device selection.
What's NOT on the CLI: γ / τ / ε / entropy_coef / per_α / reward_scale
/ per-head LRs — all live in ISV[400..417] and are driven by R5's
controllers from EMA-tracked diagnostics. Per the rebuild plan
A1: "every adaptive bound is signal-driven, not tuned."
## Cluster smoke entry point
```bash
# R9 validation smoke (after pre-cluster local CUDA tests green).
./scripts/argo-alpha-rl.sh --n-steps 1000 --instrument-mode front-month
# Production scale-up (gated by R9's multi-fold pass).
./scripts/argo-alpha-rl.sh --n-steps 50000 --n-backtests 32 \
--per-capacity 100000 # GPU sum-tree R-future when capacity > 4096
```
## What's NOT in this commit
The R9 cluster smoke run itself is out of band — this commit ships
the entry points. R9 will execute the pre-cluster validation
checklist + first 1000-step smoke + multi-fold walk-forward G8 gate
per the rebuild plan §"Cluster smoke discipline".
The summary JSON's schema is intentionally narrow (final-step losses
+ replay len + completion state + NaN abort marker). R-future may
add per-epoch breakdowns + per-ISV-slot snapshots once the cluster
smoke tells us which diagnostics are actually load-bearing for kill
decisions.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The Phase 1 multi-resolution layout (10 raw + 10 agg@30 + 12 agg@100)
regressed ALL horizons in alpha-perception-htpp6 (2026-05-22):
- auc_h100: 0.681 -> 0.512 (-0.169)
- auc_h300: 0.617 -> 0.506 (-0.111)
- auc_h1000: 0.576 -> 0.526 (-0.050)
Hypothesis falsified. Root cause: Mamba2+CfC SSM encoder already used
all 32 raw ticks effectively via state-recurrence; replacing 22 raw
ticks with arithmetic-mean aggregates destroyed within-window
microstructure variance (the actual h100 signal) AND broke temporal
continuity that the recurrence relies on. Δt Fourier encoder
couldn't compensate.
Architectural pearl: SSM/RNN/CfC + multi-resolution input is
incompatible without separate-encoder-per-scale or explicit scale
tokens. Transformer-style positional encoding tolerates scale-mixing;
recurrent state updates assume consecutive positions.
Reverts default to '1:32'. Adds explicit single_scale_32() constructor
for callers (harness, tests). Keeps default_three_scale() in code with
deprecation note for future sub-variant experiments. Production
defaults across alpha_train CLI, Argo template, dispatcher script,
ml-backtesting harness now match the proven baseline.
Replaces Option<u32> instrument_id_filter with InstrumentFilter enum {All,
Id(u32), FrontMonth}. FrontMonth runs a two-pass detect over the DBN
stream: pass 1 counts instrument_ids and collects SymbolMapping records,
picks the dominant id, validates it resolves to an ES contract via regex
ES[FGHJKMNQUVXZ]\d{1,2}; pass 2 streams the filtered records.
Motivated by alpha-perception-k54wd: a single-id filter on parent-symbol
ES.FUT data caught Q1 2024 (kept=73M) but kept=0 for Q2-Q9 because ES
front-month rolls quarterly (ESH4 -> ESM4 -> ESU4 -> ESZ4 ...). FrontMonth
self-tunes across the rolls without needing a per-file id table.
Sidecar keys distinguish modes: mbp10 / mbp10_instr<id> / mbp10_front_month.
CLI flag renamed --instrument-id -> --instrument-mode {all,id=N,front-month}
with matching parameter rename in argo-alpha-perception.sh + template.
Adds new workflow parameter instrument-id (default empty string = no filter).
Script forwards via --instrument-id CLI flag to alpha_train when non-empty.
EXTRA_FLAGS template logic appends --instrument-id only when set.
kubectl apply must run first (per feedback_argo_template_must_apply) so the
cluster CRD reflects the new parameter before argo submit.
Mirrors the alpha_train CLI flag (--kernel-step-trace <path>) into
fxt-backtest. The PerceptionTrainerConfig already accepts the path;
this commit just exposes the CLI surface and propagates through:
fxt-backtest run --kernel-step-trace <path>
-> BacktestHarnessConfig.kernel_step_trace_path
-> PerceptionTrainerConfig.kernel_step_trace_path
Sweep YAML gains a `kernel_step_trace: Option<PathBuf>` base field.
Argo lob-backtest-sweep-template gains a `kernel-step-trace` workflow
parameter (default disabled; when set, --features kernel-step-trace
is passed to cargo build AND --kernel-step-trace to fxt-backtest).
Gated behind the `kernel-step-trace` Cargo feature (matching alpha_train).
When disabled (default), zero overhead.
Enables per-step JSONL diagnostic emission from inference kernels --
needed for Smoke 2 deep-dive after sampling histograms (in parallel
investigation) localize the per-horizon dynamics bottleneck.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per spec §2.5 and feedback_no_partial_refactor. Removes:
- LoaderMode enum (single-variant after Sequential removal → deleted entirely)
- next_sequence_sequential + sequential_file_idx + sequential_anchor_idx + last_call_was_file_boundary + is_file_boundary
- last_seen_file_boundary, train_graph_boundary_state fields
- notify_file_boundary method + boundary-state graph recapture logic
- need_attn_pool_bootstrap match — always true now (random mode always bootstraps)
- --loader-mode CLI flag + notify_file_boundary() call
- loader-mode Argo template param
Additional consumers migrated atomically (beyond the 4 files listed in
the plan): ml-alpha/tests/perception_overfit.rs,
ml-alpha/tests/multi_horizon_loader.rs, ml-backtesting/src/harness.rs,
ml-backtesting/tests/trainer_parity.rs,
ml-backtesting/tests/ring3_replay.rs — all referenced LoaderMode or the
removed config fields.
Workspace builds clean at this commit (pre-existing cudarc-cupti example
and ml-crate test errors are unrelated to MTER removal — they fail at
HEAD too). New per-horizon CfC arch lands in subsequent tasks of the
same plan.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Loader can now serve sequences in temporal order (Sequential mode);
when active, the trainer skips attn_pool reset at non-file-boundary
sequence boundaries (next commit will use this for stateful CfC + MTER).
Random mode preserved as default; ZERO behavioral change for existing
runs. Phased commit 1/8 of intervention B per
docs/superpowers/specs/2026-05-21-crt-train-intervention-b-multi-timescale-readout.md
The compile-time feature kernel-step-trace gates inclusion of the ring
code. NEW: --kernel-step-trace <PATH> CLI flag on alpha_train gates
RUNTIME activation:
- Path provided -> ring allocated, drain spawns, JSONL records written
to <PATH> (truncated). One record per kernel emission.
- Path omitted -> ring not allocated, drain not spawned, zero overhead
even with the feature compiled in.
JSONL schema: {"step": u32, "kid": u8, "kname": str, "rt": u8,
"rt_name": str, "payload": {field: f32, ...}}. The payload field names
come from the existing per-(kid, rt) decoders in gpu_log.rs (ported to
serde_json::json! in decode_to_json).
Trainer ring fields are now Option<_>; populated only when the runtime
trace path is Some. Tick kernel, step-counter shadow, smoothness
controller pointer-passing, and Drop all become path-conditional.
The tracing::info!-based drain variant is removed (file-writer is
strictly more useful; tests migrated to assert JSONL file contents).
Argo template:
- New parameter kernel-step-trace-enable (build-time feature opt-in)
- New parameter kernel-step-trace-path (runtime CLI value)
- ensure-binary cache-busts when feature toggles
- training step passes --kernel-step-trace flag conditionally
Per feedback_no_feature_flags: compile-time gate retained because the
ring carries real memory cost (~2 MiB pinned) and per-step write overhead;
the specific name kernel-step-trace narrows scope to this mechanism.
Runtime gate is Option<PathBuf>, not a boolean enable_*.
The cargo-target-cuda PVC persists `$BUILD` across runs. cargo build
mutates Cargo.lock, so the next run's `git checkout $SHA` fails with
"Your local changes to the following files would be overwritten by
checkout: Cargo.lock". Same pattern alpha-perception-template uses
(`git checkout --force $SHA; git clean -fd`) handles this cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the operational gap from P6: when a sweep grid YAML carries
sim_variants, argo-lob-sweep.sh now emits ONE run-sweep-batched task
instead of N fan-out run-cell tasks. The new run-sweep template
invokes `fxt-backtest sweep` against the full grid (base64-encoded
inline as Argo parameter to avoid YAML special-char encoding traps).
The binary's P6 sweep() function handles cell × variant fan-out
internally via BatchedSimConfig::from_grid, so one pod processes all
4 windows × 140 variants sequentially. Trade-off: no inter-window
parallelism in this rev (4 quarters sequential in one pod ≈ firm-bound
2h wall per spec §9). 3-pod scale-out is P7 future work — needs the
script to split the YAML into per-window sub-grids and emit one
run-sweep task per sub-grid.
Backward-compat: legacy (no sim_variants) flow unchanged. The dry-run
of sweep_smoke.yaml continues to emit run-cell-* fan-out tasks; the
dry-run of sweep_deployability.yaml emits one run-sweep-batched task.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
