ml-backtesting/build.rs reads FOXHUNT_CUDA_ARCH (defaults sm_86),
ml-alpha/build.rs reads CUDA_COMPUTE_CAP. Template only set the
latter, so lobsim cubins compiled for sm_86 on H100 (sm_90) →
CUDA_ERROR_NO_BINARY_FOR_GPU at runtime.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sets CUDA_COMPUTE_CAP dynamically from the GPU on the node. Clears
both ml-alpha and ml-core build caches. Captures build.rs stderr in
pod logs for debugging.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Remove: ReplayBuffer, Transition, NStepEntry, push_to_replay,
sample_and_gather, n_step_buffer, replay.rs, --diag-every flag.
PER call sites stubbed with todo!() — replaced by GPU PER in next commits.
Also fixes pre-commit hook to allow todo!() macro (per CLAUDE.md:
"todo!() macro is OK for runtime stubs") while still blocking
TODO/FIXME comment markers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- --diag-every N (default 100): skip device readback on non-diag steps.
Eliminates ~10 stream.sync per step → ~2× throughput at large batch.
- Fix PnL bug: was accumulating raw_rewards (shaped, all batches) instead
of rewards/scale on done steps only.
- Wire diag-every=100 in Argo template (diag JSONL every 100 steps).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
n-backtests default in Argo template: 16 → 128.
PER capacity: max(configured, 4 × b_size) so replay buffer is always
large enough for useful prioritized sampling at any batch size.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Prepared for next run after the γ=0.99 1M run completes:
- γ floor 0.99 → 0.995: horizon 100 → 200 steps (50 seconds).
Real ES directional moves (2-5 points) happen at this scale.
- PER 16384 → 32768: 2048 unique steps of replay depth. Supports
the 200-step γ horizon with margin.
- Min-hold 50 → 100 steps (25 seconds): commit to the full wave.
Short-hold penalty threshold matches.
Safe to push because raw-reward re-normalization eliminates scale
drift in the deeper buffer, and the ±2% scale clamp keeps targets
stable across the 2048-step replay window.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Stored raw_reward alongside scaled reward in Transition. At sample
time, consumer re-applies current reward_scale to raw_reward instead
of using the stale-scale stored reward. This eliminates off-policy
scale drift that caused q_pi_agree to collapse from 0.125 to 1e-22
over 40k steps with PER=32768.
PER capacity set to 16384 (1024 unique steps at b=16) — balances
replay depth against h_t representation staleness.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces the 4-step DAG (check-cache → compile on CPU → warmup GPU →
train on GPU) with a single pod on the L40S that does everything:
git fetch → incremental cargo build (~3s warm) → train.
Eliminates: separate compile node, node autoscale wait, binary
transfer, fxcache step. The ci-builder image has CUDA 13.0 devel +
Rust — compilation and training use the same CUDA libs.
The cargo-target-cuda PVC provides incremental build cache across
runs. LD_LIBRARY_PATH strips the stubs dir so real CUDA libs load.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The MBP-10 predecoded sidecars live on feature-cache-pvc at
/feature-cache/predecoded/, not on training-data-pvc. Points
--predecoded-dir to the correct PVC mount.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The ml-alpha branch uses alpha_rl_train (from ml-alpha crate) with
different CLI args than train_baseline_rl (from ml crate). Adds
model=alpha-rl case that:
- Builds ml-alpha --example alpha_rl_train
- Runs with --mbp10-data-dir, --predecoded-dir, --n-steps 50000,
--n-backtests 16 (matching the local smoke config)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
28 parallel alpha-feature chunks × ~3GB each = 84GB peak at merge.
Capping at 8 threads keeps peak at ~24GB (well within 96Gi limit)
while still saturating I/O. This is a one-time cost — after the
cache is populated on the PVC, subsequent runs skip entirely.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
With cargo-target-cuda PVC persisting across pods, CARGO_INCREMENTAL=1
gives true incremental builds: only recompile changed crates. A
single-crate ml-alpha change rebuilds in ~45s vs 9min full workspace.
sccache removed — it conflicts with incremental compilation and is
redundant when the target dir persists. The binary-by-SHA cache on
the training-data PVC (already implemented) provides the "skip
compile entirely on re-run" fast path.
Expected workflow time reduction:
Before: compile 9m + fxcache 18m + train ~30m = ~57m
After: compile <1m (incremental) + fxcache <1m (PVC cache) + train ~30m = ~32m
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The imbalance bar cache now writes to <mbp10_dir>/.cache/ on the PVC.
The ensure-fxcache step needs write access to persist the cache
across runs (was readOnly: true, causing silent cache-write failures).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two fixes for alpha-rl-9k9x6 (commit 3737feb66) findings:
(1) rl_pi_action_kernel.cu — per-action probability floor P_MIN=0.02
with renorm after softmax. Structural guarantee H(π_sample)
bounded below regardless of logits. 9k9x6 evidence:
- π collapsed to action 1 (100% sample rate) by step 2500
- action_entropy EMA → 0.001 by step 10000 (uniform=2.197)
- entropy_coef controller pegged at COEF_MAX without effect
because PPO update couldn't escape δ-function fixed point
- 0 trades closed in 50000 steps; position monotonically
accumulated to -45871 lots
With P_MIN=0.02, worst-case H(π_sample) ≈ 0.77 nats — well
above the observed collapse and well below uniform.
Per `pearl_blend_formulas_must_have_permanent_floor`.
(2) argo template + script — N_BACKTESTS default 1 → 16. The CLI
default lift in commit 3737feb66 was overridden by both the
argo script and the wftmpl `value: "1"`. 9k9x6 actually ran
b_size=1 despite being framed as the b_size=16 test.
Per `pearl_b_size_1_signal_starvation_blocks_q_learning`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Concurrent submissions at the same SHA (one workflow per fold_idx
for the walk-forward G8 gate) would overwrite each other's
eval_summary.json. Adds a /foldN suffix to the output path so the
aggregator can collect 3+ distinct eval_summary.json files from
/feature-cache/alpha-rl-runs/<sha>/fold0,fold1,fold2/.
Single-fold smokes (n_folds=1) still write to /<sha>/fold0/
directly — backwards-compatible for the prior smoke pattern, just
one level deeper than before.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the minimum-viable implementation of the R9 multi-fold G8 gate
per `pearl_single_window_oos_is_not_oos` ("a single window is NOT
out-of-sample"). The trainer can now:
1. Slice the MBP-10 file list into K equal-sized blocks
(`--n-folds K --fold-idx k`).
2. Train on blocks [0..=k] (passed to MultiHorizonLoader).
3. Run a separate eval phase of `--n-eval-steps` on block [k+1]
using a second loader instance.
4. Drain LobSim trade records gated by a pre-eval head checkpoint
so train-phase trades don't contaminate the eval summary.
5. Compute profit_factor + sharpe + drawdown via existing
`ml_backtesting::artifacts::compute_summary`.
6. Write `eval_summary.json` alongside `alpha_rl_train_summary.json`.
## Manual fan-out (this MVP)
The dispatcher (`scripts/argo-alpha-rl.sh`) gains three new flags
that thread through the Argo template into the CLI: `--fold-idx`,
`--n-folds`, `--n-eval-steps`. To run a 3-fold G8:
./scripts/argo-alpha-rl.sh --n-folds 3 --fold-idx 0 --n-eval-steps 200
./scripts/argo-alpha-rl.sh --n-folds 3 --fold-idx 1 --n-eval-steps 200
(With n_folds=3 the valid fold indices are 0 and 1 — the third block
is the eval window for fold 1. n_folds=K accepts fold_idx ∈ [0, K-2].)
Each submission produces one `eval_summary.json` at the resolved
output dir; the per-fold profit_factor is the value to aggregate.
Manual aggregation for now — automated DAG matrix fan-out + an
in-cluster aggregator pod is a follow-up commit. The aggregator
will mean ± SD the per-fold PFs and gate on `PF > 1.0`.
## What's NOT pure eval
The eval loop calls `step_with_lobsim` (same as train) — Adam steps,
PER updates, controller adaptations all still fire during eval. At
b_size=1 the per-step learning effect is small relative to the
train-phase-accumulated policy, so the eval PF approximates the
OOS performance of the train-end policy. A clean pure-eval mode
(forward + LobSim step only, no backward/Adam/PER) is a follow-up
architectural change; documented inline at the eval phase block.
## Default behaviour unchanged
`--n-folds=1` (default) skips the eval split entirely and uses all
files for training — identical to the prior single-window smoke.
The R9 prior smokes ran in this mode. Default `--fold-idx=0` and
`--n-eval-steps=0` keep prior smoke runs binary-compatible.
## Template + dispatcher changes
* `alpha-rl-template.yaml`: adds 3 new workflow parameters
(`fold-idx`, `n-folds`, `n-eval-steps`) and threads them into
the train container's `alpha_rl_train` invocation.
* `argo-alpha-rl.sh`: adds matching CLI flags with explicit
documentation of the multi-fold dispatching pattern.
## Verified gates
Local sm_86 build + dispatcher syntax clean. Tests unchanged
(the walk-forward path is exercised by cluster smokes, not unit
tests — the loader-slicing logic is straightforward index math).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Closes the rebuild plan's R8 scope: production runner shape for the
integrated RL trainer. Three artifacts wired end-to-end:
1. `crates/ml-alpha/examples/alpha_rl_train.rs` — clap CLI driving
`IntegratedTrainer::step_with_lobsim` against MBP-10 windows
loaded via `MultiHorizonLoader::next_sequence_pair` (R2) for
true `(s_t, s_{t+1})` adjacency. Per `feedback_mbp10_mandatory`,
`--mbp10-data-dir` is required — no synthetic-data fallback in
the production path.
2. `infra/k8s/argo/alpha-rl-template.yaml` — WorkflowTemplate
mirroring alpha-perception's DAG (check-cache → ensure-binary →
train; warmup-gpu parallel). Binary cache slot is
`/data/bin/<sha>/alpha_rl_train` (distinct from `alpha_train`
so the two binaries coexist at the same SHA).
3. `scripts/argo-alpha-rl.sh` — dispatcher with three rebuild-plan
guards baked in.
## Dispatcher guards (per the rebuild plan's feedback list)
`feedback_default_to_l40s_pool` (2026-05-09): default `--gpu-pool`
is `ci-training-l40s` (sm_89). H100 (sm_90) is opt-in for production
scale-up only. Cubins must match the device, so the dispatcher
derives `cuda-compute-cap` from the pool name and threads it into
the workflow params.
`feedback_argo_template_must_apply` (2026-05-21 canonical incident):
`argo submit --from=wftmpl/<name>` reads the cluster CRD, NOT the
on-disk YAML; unknown `-p` parameters silently no-op without a prior
`kubectl apply`. Dispatcher applies the local template BEFORE every
submission (overrideable via `--skip-template-apply` for the rare
case where you've already applied manually).
`feedback_push_before_deploy` (2026-05-20 canonical incident): the
in-cluster `ensure-binary` pod fetches source from `origin/<branch>`,
NOT the local working tree. Submitting before `git push` deploys the
last-pushed SHA, which can lag local diff by N commits. Dispatcher
verifies `git rev-parse HEAD == git rev-parse origin/<branch>` and
hard-errors with the explicit push command otherwise. Bypass via
`--skip-push-check` (only when intentionally deploying a previously-
pushed SHA via `--sha`).
## CLI: gate G8 (NaN abort)
Per `feedback_stop_on_anomaly` + `feedback_kill_runs_on_anomaly_quickly`,
the CLI checks every per-head loss (l_bce / l_q / l_pi / l_v / l_aux
/ l_total) for finiteness after each `step_with_lobsim` call.
Non-finite at any step → write summary with `nan_abort_step` set →
`process::exit(2)`. R9's cluster smoke tail-watcher kills the
workflow on the non-zero exit code, satisfying gate G8 from the
rebuild plan.
## CLI: knobs that ARE on the CLI
Structural / boundary parameters only (per
`pearl_controller_anchors_isv_driven`: every adaptive knob lives in
ISV, not CLI flags):
* `--mbp10-data-dir / --predecoded-dir / --out` — I/O paths.
* `--n-steps` — wall-budget control (1000 R9 smoke / 50k+ prod).
* `--seq-len / --n-backtests / --per-capacity` — structural
sizing. seq_len threads into the loader's multi-resolution
`1:<seq_len>` config; n_backtests into both LobSimCuda and
PerceptionTrainerConfig.n_batch.
* `--seed` — reproducibility per
`pearl_scoped_init_seed_for_reproducibility` (forks deterministic
sub-seeds for dqn / ppo / per).
* `--instrument-mode` — MBP-10 filter (all / front-month / id=N).
* `--gpu-idx` — CUDA device selection.
What's NOT on the CLI: γ / τ / ε / entropy_coef / per_α / reward_scale
/ per-head LRs — all live in ISV[400..417] and are driven by R5's
controllers from EMA-tracked diagnostics. Per the rebuild plan
A1: "every adaptive bound is signal-driven, not tuned."
## Cluster smoke entry point
```bash
# R9 validation smoke (after pre-cluster local CUDA tests green).
./scripts/argo-alpha-rl.sh --n-steps 1000 --instrument-mode front-month
# Production scale-up (gated by R9's multi-fold pass).
./scripts/argo-alpha-rl.sh --n-steps 50000 --n-backtests 32 \
--per-capacity 100000 # GPU sum-tree R-future when capacity > 4096
```
## What's NOT in this commit
The R9 cluster smoke run itself is out of band — this commit ships
the entry points. R9 will execute the pre-cluster validation
checklist + first 1000-step smoke + multi-fold walk-forward G8 gate
per the rebuild plan §"Cluster smoke discipline".
The summary JSON's schema is intentionally narrow (final-step losses
+ replay len + completion state + NaN abort marker). R-future may
add per-epoch breakdowns + per-ISV-slot snapshots once the cluster
smoke tells us which diagnostics are actually load-bearing for kill
decisions.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The Phase 1 multi-resolution layout (10 raw + 10 agg@30 + 12 agg@100)
regressed ALL horizons in alpha-perception-htpp6 (2026-05-22):
- auc_h100: 0.681 -> 0.512 (-0.169)
- auc_h300: 0.617 -> 0.506 (-0.111)
- auc_h1000: 0.576 -> 0.526 (-0.050)
Hypothesis falsified. Root cause: Mamba2+CfC SSM encoder already used
all 32 raw ticks effectively via state-recurrence; replacing 22 raw
ticks with arithmetic-mean aggregates destroyed within-window
microstructure variance (the actual h100 signal) AND broke temporal
continuity that the recurrence relies on. Δt Fourier encoder
couldn't compensate.
Architectural pearl: SSM/RNN/CfC + multi-resolution input is
incompatible without separate-encoder-per-scale or explicit scale
tokens. Transformer-style positional encoding tolerates scale-mixing;
recurrent state updates assume consecutive positions.
Reverts default to '1:32'. Adds explicit single_scale_32() constructor
for callers (harness, tests). Keeps default_three_scale() in code with
deprecation note for future sub-variant experiments. Production
defaults across alpha_train CLI, Argo template, dispatcher script,
ml-backtesting harness now match the proven baseline.
Replaces Option<u32> instrument_id_filter with InstrumentFilter enum {All,
Id(u32), FrontMonth}. FrontMonth runs a two-pass detect over the DBN
stream: pass 1 counts instrument_ids and collects SymbolMapping records,
picks the dominant id, validates it resolves to an ES contract via regex
ES[FGHJKMNQUVXZ]\d{1,2}; pass 2 streams the filtered records.
Motivated by alpha-perception-k54wd: a single-id filter on parent-symbol
ES.FUT data caught Q1 2024 (kept=73M) but kept=0 for Q2-Q9 because ES
front-month rolls quarterly (ESH4 -> ESM4 -> ESU4 -> ESZ4 ...). FrontMonth
self-tunes across the rolls without needing a per-file id table.
Sidecar keys distinguish modes: mbp10 / mbp10_instr<id> / mbp10_front_month.
CLI flag renamed --instrument-id -> --instrument-mode {all,id=N,front-month}
with matching parameter rename in argo-alpha-perception.sh + template.
Adds new workflow parameter instrument-id (default empty string = no filter).
Script forwards via --instrument-id CLI flag to alpha_train when non-empty.
EXTRA_FLAGS template logic appends --instrument-id only when set.
kubectl apply must run first (per feedback_argo_template_must_apply) so the
cluster CRD reflects the new parameter before argo submit.
Mirrors the alpha_train CLI flag (--kernel-step-trace <path>) into
fxt-backtest. The PerceptionTrainerConfig already accepts the path;
this commit just exposes the CLI surface and propagates through:
fxt-backtest run --kernel-step-trace <path>
-> BacktestHarnessConfig.kernel_step_trace_path
-> PerceptionTrainerConfig.kernel_step_trace_path
Sweep YAML gains a `kernel_step_trace: Option<PathBuf>` base field.
Argo lob-backtest-sweep-template gains a `kernel-step-trace` workflow
parameter (default disabled; when set, --features kernel-step-trace
is passed to cargo build AND --kernel-step-trace to fxt-backtest).
Gated behind the `kernel-step-trace` Cargo feature (matching alpha_train).
When disabled (default), zero overhead.
Enables per-step JSONL diagnostic emission from inference kernels --
needed for Smoke 2 deep-dive after sampling histograms (in parallel
investigation) localize the per-horizon dynamics bottleneck.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per spec §2.5 and feedback_no_partial_refactor. Removes:
- LoaderMode enum (single-variant after Sequential removal → deleted entirely)
- next_sequence_sequential + sequential_file_idx + sequential_anchor_idx + last_call_was_file_boundary + is_file_boundary
- last_seen_file_boundary, train_graph_boundary_state fields
- notify_file_boundary method + boundary-state graph recapture logic
- need_attn_pool_bootstrap match — always true now (random mode always bootstraps)
- --loader-mode CLI flag + notify_file_boundary() call
- loader-mode Argo template param
Additional consumers migrated atomically (beyond the 4 files listed in
the plan): ml-alpha/tests/perception_overfit.rs,
ml-alpha/tests/multi_horizon_loader.rs, ml-backtesting/src/harness.rs,
ml-backtesting/tests/trainer_parity.rs,
ml-backtesting/tests/ring3_replay.rs — all referenced LoaderMode or the
removed config fields.
Workspace builds clean at this commit (pre-existing cudarc-cupti example
and ml-crate test errors are unrelated to MTER removal — they fail at
HEAD too). New per-horizon CfC arch lands in subsequent tasks of the
same plan.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Loader can now serve sequences in temporal order (Sequential mode);
when active, the trainer skips attn_pool reset at non-file-boundary
sequence boundaries (next commit will use this for stateful CfC + MTER).
Random mode preserved as default; ZERO behavioral change for existing
runs. Phased commit 1/8 of intervention B per
docs/superpowers/specs/2026-05-21-crt-train-intervention-b-multi-timescale-readout.md
The compile-time feature kernel-step-trace gates inclusion of the ring
code. NEW: --kernel-step-trace <PATH> CLI flag on alpha_train gates
RUNTIME activation:
- Path provided -> ring allocated, drain spawns, JSONL records written
to <PATH> (truncated). One record per kernel emission.
- Path omitted -> ring not allocated, drain not spawned, zero overhead
even with the feature compiled in.
JSONL schema: {"step": u32, "kid": u8, "kname": str, "rt": u8,
"rt_name": str, "payload": {field: f32, ...}}. The payload field names
come from the existing per-(kid, rt) decoders in gpu_log.rs (ported to
serde_json::json! in decode_to_json).
Trainer ring fields are now Option<_>; populated only when the runtime
trace path is Some. Tick kernel, step-counter shadow, smoothness
controller pointer-passing, and Drop all become path-conditional.
The tracing::info!-based drain variant is removed (file-writer is
strictly more useful; tests migrated to assert JSONL file contents).
Argo template:
- New parameter kernel-step-trace-enable (build-time feature opt-in)
- New parameter kernel-step-trace-path (runtime CLI value)
- ensure-binary cache-busts when feature toggles
- training step passes --kernel-step-trace flag conditionally
Per feedback_no_feature_flags: compile-time gate retained because the
ring carries real memory cost (~2 MiB pinned) and per-step write overhead;
the specific name kernel-step-trace narrows scope to this mechanism.
Runtime gate is Option<PathBuf>, not a boolean enable_*.
The cargo-target-cuda PVC persists `$BUILD` across runs. cargo build
mutates Cargo.lock, so the next run's `git checkout $SHA` fails with
"Your local changes to the following files would be overwritten by
checkout: Cargo.lock". Same pattern alpha-perception-template uses
(`git checkout --force $SHA; git clean -fd`) handles this cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the operational gap from P6: when a sweep grid YAML carries
sim_variants, argo-lob-sweep.sh now emits ONE run-sweep-batched task
instead of N fan-out run-cell tasks. The new run-sweep template
invokes `fxt-backtest sweep` against the full grid (base64-encoded
inline as Argo parameter to avoid YAML special-char encoding traps).
The binary's P6 sweep() function handles cell × variant fan-out
internally via BatchedSimConfig::from_grid, so one pod processes all
4 windows × 140 variants sequentially. Trade-off: no inter-window
parallelism in this rev (4 quarters sequential in one pod ≈ firm-bound
2h wall per spec §9). 3-pod scale-out is P7 future work — needs the
script to split the YAML into per-window sub-grids and emit one
run-sweep task per sub-grid.
Backward-compat: legacy (no sim_variants) flow unchanged. The dry-run
of sweep_smoke.yaml continues to emit run-cell-* fan-out tasks; the
dry-run of sweep_deployability.yaml emits one run-sweep-batched task.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three follow-ups to the cold-start floor fix:
1. Kernel: MIN_TRADES_FOR_VAR_CAP gate. After the first trade closed,
`isv_kelly_update_on_close` set `realised_return_var = ret²` — a
single-sample variance proxy that systematically collapses
`cap_units = target_vol / sqrt(var × ann_factor)` near zero for
any biased return. cap_lots → 0 → no further trades despite strong
alpha. Gate the variance-derived cap behind `n_trades_seen >= 10`;
below the threshold cap falls back to host-supplied `max_lots`,
same as the pre-first-trade path. Same gate applied to both
`decision_policy_default` and `decision_policy_program`.
Regression: `post_first_loss_state_does_not_lock_out_further_trades`
reproduces the exact pre-fix state from the smoke (n_trades_seen=1,
var=103.6) and asserts the kernel still fires a long with p=0.8.
2. Aggregate: add `nvidia.com/gpu: 1` resource request. Scaleway's
L40S device plugin mounts libcuda.so.1 into the container only on
GPU-requesting pods; the aggregate logic is CPU-only but the
binary's dynamic loader needs the driver libs. Cheapest correct
fix until a separate CPU-only aggregator binary exists.
3. Smoke YAML: `max_events: 0` (exhaust loader). 100k events is
minutes of ES.FUT, far shorter than the h6000 holding horizon the
model was trained on. Full quarter exercises sustained trading +
variance estimate ramp-up.
All three regression tests pass locally on RTX 3050 Ti.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Production smoke completed end-to-end but produced n_trades=0 across 99,969
decisions — `decision_policy_default` and `decision_policy_program` both
applied a sentinel-skip pattern: if `isv_kelly_d` had not been seeded
(pnl_ema_win == 0), each horizon's signed-size stayed zero, AND each
horizon's aggregation weight (= recent_sharpe) also stayed zero. The
cross-horizon w_sum was therefore 0, final_size was 0, every market_target
was noop. State only updates on trade close → no trade ever fires →
infinite cold-start.
Per pearl_blend_formulas_must_have_permanent_floor (`max(real, floor)`,
not blend) and pearl_kelly_cap_signal_driven_floors, replace the sentinel-
skip with a two-layer floor on each kernel:
1. Kelly fraction: `max(kelly_frac_floor, computed_kelly)` — when state
is sentinel, falls back to the floor directly. Cap_lots falls back
to `max_lots` when realised_return_var is sentinel.
2. Aggregation weight: `max(sharpe_weight_floor, recent_sharpe)` — lets
cross-horizon sum produce a non-zero size before recent_sharpe is
populated. Once a horizon shows positive sharpe it dominates.
Plumbed through `step_decision_with_latency` / `step_decision` as two
new f32 args (atomic contract change, every caller migrated). Defaults
0.20 / 0.10 chosen so a strong-conviction signal (sig_mag ≥ 0.5) fires
1 lot at cold-start under max_lots=5 while weaker signals stay flat
(see `default_kelly_frac_floor` comment for the arithmetic). Exposed
as CLI flags + sweep-grid base/cell overrides.
Regression test `decision_floor_coldstart` proves:
- default floors (0.20/0.10) fire a 1-lot buy with p_h=0.8 and zero state
- zero floors reproduce the original noop bug
Also moves `aggregate` step to the GPU pool because fxt-backtest is
dynamically linked against libcuda.so.1 (the ci-compile-cpu hosts
don't expose CUDA driver libs).
Verified locally on RTX 3050 Ti — workspace cargo check passes, both
regression tests pass, trunk save/load roundtrip still passes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Template was missing feature-cache PVC mount on run-cell; trained
checkpoints + predecoded data live at /feature-cache/* on the
feature-cache-pvc claim. Mount it the same way alpha-perception does.
- sweep_smoke.yaml referenced /data/* (no such mount) and a 40-char
SHA path for the checkpoint dir. alpha-perception-runs subdirs are
9-char short SHAs; data path is /mnt/training-data/* (matches mount).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Working alpha-perception-template references
`foxhunt/foxhunt-training-runtime:latest`. Our lob-backtest-sweep
template stripped the `foxhunt-` prefix and got 404 from the registry
on run-cell + aggregate steps.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CARGO_TARGET_DIR=/cargo-target redirects all build output away from
the in-tree `<crate>/target/` path, so `cp $BUILD/target/release/...`
fails after a successful compile. Mirror alpha-perception-template's
correct pattern: `cp $CARGO_TARGET_DIR/release/...`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pods labeled `component: backtest` only match the `argo-base-egress`
NetworkPolicy (DNS, 443, MinIO, Mattermost). The ensure-binary step
needs SSH-to-gitlab-shell on port 2222, which is gated by
`argo-train-workflow` (selects `component=train`).
Relabel to `train` so the same NetworkPolicy that lets alpha-perception
clone the repo also covers lob-backtest-sweep. Egress shape (compile,
GPU run, PVC writes) is identical to a training run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The template wrote ~/.ssh/config but didn't chmod it. OpenSSH refused
the file with 'Bad owner or permissions on /root/.ssh/config' and the
git clone failed with 'exit status 128' before fxt-backtest could
compile.
Same one-line fix that alpha-perception-template.yaml has had since
forever. Verified: lob-backtest-sweep-<new> ensure-binary now passes
the SSH check and starts cargo build.
Matches the alpha-perception-template (which is the known-good
template in this namespace). Without the fix, ensure-binary +
run-cell + aggregate pods all stayed Init for ~20m with
'MountVolume.SetUp failed for volume "git-ssh-key" : secret
"git-ssh-key" not found' before the workflow timed out.
The actual secret in the foxhunt namespace is named
'argo-git-ssh-key' (see kubectl get secrets -n foxhunt). Same
template misconfig as the training-data PVC name.
The template referenced PVC 'training-data' but the actual claim in
the foxhunt namespace is 'training-data-pvc' (matches the convention
used by other workflows like alpha-perception-template). Without this
fix, ensure-binary + run-cell + aggregate pods all stayed Pending
forever with 'persistentvolumeclaim "training-data" not found',
blocking every smoke / threshold-tuning / deployability sweep.
Verified: smoke sweep lob-backtest-sweep-jp48v scheduled ensure-binary
onto the ci-compile-cpu pool (autoscaler triggered scale-up 0->1
within seconds of resubmission).
Cluster fan-out for the `fxt-backtest sweep` single-machine path.
Reads the same grid YAML format as the binary; runs each cell on a
dedicated GPU pod in parallel; aggregates at the end on a CPU pod.
infra/k8s/argo/lob-backtest-sweep-template.yaml:
WorkflowTemplate `lob-backtest-sweep` with three job templates:
ensure-binary — cache-or-compile fxt-backtest by short-SHA into
/mnt/training-data/bin/<sha>/. Mirrors the
train-multi-seed-template.yaml ensure-binary
shape but for a single binary.
run-cell — single GPU pod (ci-training-l40s default per
feedback_default_to_l40s_pool.md). Receives
cell-name + every Run arg via inputs.parameters.
Writes artifacts to <sweep-root>/<sweep-tag>/<cell>/.
aggregate — CPU pod runs `fxt-backtest aggregate <sweep-dir>`
producing aggregate.parquet + pareto_frontier.json
at the sweep root.
DAG marker `# __SWEEP_CELLS__` replaced at submission time with N
WorkflowTask stanzas (one per cell), and the aggregate's
`dependencies: [ensure-binary]` is rewritten to include every
run-cell-* dep — so aggregate waits for ALL cells.
scripts/argo-lob-sweep.sh:
Companion submission script following the argo-train.sh pattern.
Parses the grid YAML via python3 + PyYAML (no `yq` dependency —
yq isn't used elsewhere in foxhunt scripts; python3+PyYAML is
universal in our CI images). Emits per-cell WorkflowTask stanzas
+ aggregate dependency list, awks them into the template, then
`kubectl apply` + `argo submit`. Supports --dry-run for offline
rendering and --watch for live log following.
Defaults match the spec / pearl set:
- sm_89 / ci-training-l40s default (override via --gpu-pool
ci-training-h100 for sm_90 + 80 GB)
- data root /mnt/training-data/futures-baseline/ES.FUT
- sweep results under /mnt/training-data/sweeps/lob-backtest/<tag>/
- sweep-tag defaults to <basename of grid>-<short-sha>
Verified locally:
- bash -n syntax-check passes
- --help renders
- --dry-run against the existing
config/ml/sweep_decision_stride_example.yaml renders a valid
workflow with 4 cells + correct aggregate dependency list
Live submission is operational work that needs cluster access to
verify; the rendered YAML follows the same conventions as the
existing argo-train.sh workflows that ship in this repo.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Decision-stride S lets a length-K sequence span ((K-1)*S + 1) raw
snapshots instead of K consecutive ones — expands the effective
time-window covered by each sequence at the same K-positions compute
cost. With K=64 and S=4, the window covers 256 ticks (~5s on ES MBP-10
at 20ms-tick) instead of 64 ticks (~1.3s).
Loader (crates/ml-alpha/src/data/loader.rs):
- `MultiHorizonLoaderConfig.decision_stride: usize` (default 1, must
pre-existing call sites add the new field).
- `next_sequence` reads snapshot at `anchor + k * stride`; labels at the
same indices (labels stay in absolute-snapshot horizons regardless of
stride, e.g. h=6000 always means "predict 6000 raw snapshots forward").
- `prev` snapshot for microstructure features (prev_mid, prev_ts_ns)
now points to the prior K-position (`anchor + (k-1)*stride`), NOT the
consecutive-snapshot prior, so `Δt = ts_ns - prev_ts_ns` carries the
actual elapsed time between K-positions (consumed by Mamba2's dt_s and
the planned Phase 2C TGN Fourier features).
- New `#[ignore]` real-data test: `loader_stride_4_yields_correct_spacing`
asserts Δt monotonicity at stride=4.
Mamba2 dt_s (crates/ml-alpha/src/trainer/perception.rs):
- `PerceptionTrainerConfig.decision_stride: usize` plumbs the stride
through. dispatch_train_step + evaluate_batched now use
`dt_s = decision_stride as f32` so Mamba2's selective scan
`exp(-dt * sigmoid(a))` reflects the real elapsed time. With stride=1
the behaviour is identical to before.
CLI (crates/ml-alpha/examples/alpha_train.rs):
- `--decision-stride <S>` flag (default 1) wired into both train and val
loaders + PerceptionTrainerConfig.
Argo workflow:
- `decision-stride` parameter on the template (default "1") +
`--decision-stride` script flag + propagation into the train pod's
alpha_train invocation.
Synthetic smoke (tests/perception_overfit.rs):
- `stacked_trainer_loss_shrinks_with_stride_4` proves the trainer-level
dt_s=4.0 keeps the Mamba2+LN+CfC+GRN chain numerically stable.
Converges 0.32 → 0.0000 (matches stride=1 smoke trajectory — dt_s
scaling didn't break the SSM dynamics).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
mhzs7 reported val mean_auc=0.726 on 3a196382f — but the trainer
constructed both train and val MultiHorizonLoader with the SAME
`mbp10_root: cli.mbp10_data_dir`. The two loaders only differed by
seed. So val sequences were held-out-by-anchor from the same files
train sampled from; not temporally OOS. Per
`pearl_single_window_oos_is_not_oos.md` a single-window result that
doesn't enforce time-ordered separation can collapse across true
walk-forward folds.
Refactor: drop `mbp10_root` from `MultiHorizonLoaderConfig` (which
forced caller to share the dir between train and val). New API takes
an explicit `files: Vec<PathBuf>` — the loader preserves the order
given and does no internal shuffle, so callers control temporal
ordering. Added `discover_mbp10_files_sorted(root)` helper that
enumerates a dir and sorts by filename (chronological under the
`ES.FUT_<YEAR>-Q<n>.dbn.zst` convention).
alpha_train.rs splits the discovered files by 3 new CLI flags:
--cv-fold <k> (default 0)
--cv-n-folds <N> (default 1 — single fold)
--cv-train-window <W> (default 0 — auto)
Single-fold default (cv_n_folds=1): train on all files except the
last, val on the last file. This replaces the old "same files for
both" bug; even runs that don't think about CV now get a temporal
split by default.
Sliding-window CV (cv_n_folds > 1): fold k trains on files
[k..k+W] and validates on file [k+W]. With 9 quarterly files
(2024-Q1..2026-Q1) and `--cv-n-folds 3`, the natural layout is:
fold 0: train 2024-Q1..2024-Q4 (W=4) → val 2025-Q1
fold 1: train 2024-Q2..2025-Q1 → val 2025-Q2
fold 2: train 2024-Q3..2025-Q2 → val 2025-Q3
blind holdout: 2025-Q4, 2026-Q1
Threaded the flags through scripts/argo-alpha-perception.sh and
infra/k8s/argo/alpha-perception-template.yaml so each fold submits
as an independent workflow.
Updated tests/multi_horizon_loader.rs to the new API:
loader_yields_seq_with_valid_labels — exercises discover + load.
loader_errors_on_empty_files — replaces missing-root test.
discover_errors_on_missing_root — pinpoints the discover step.
Honors:
- feedback_no_partial_refactor.md — every consumer migrated atomically.
- feedback_no_legacy_aliases.md — no `mbp10_root` shim left behind.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
cnjfl evidence: val_loss and mean_auc disagree.
val_loss best at e3 (0.5592)
mean_auc best at e4 (0.7670 — new h300 + h6000 peaks)
For downstream trading, ranking quality (AUC) matters more than
probability calibration (BCE loss). New default is mean_auc-based
early stopping, but val_loss/none remain selectable.
AUC is noisier than loss epoch-to-epoch (1-2pt bounces are common
even when long-horizon AUCs are still drifting up under
auto-horizon-weights), so patience defaults bump from 3 → 5.
CLI: --early-stop-metric {val_loss|mean_auc|none} default mean_auc
--early-stop-patience N default 5
Argo template parameters added with matching defaults.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
