SP-chain training has been standardising on L40S since 2026-05-09, but
every invocation required an explicit `--gpu-pool ci-training-l40s`
override. The 2026-05-04 train-mnpf7 incident (sm_90 cubins deployed
to an L40S device, then resubmitted with the explicit override) was
the last incident in a long line of "forgot the pool flag" friction.
`feedback_default_to_l40s_pool.md` codified the user preference; this
commit lands the default in the actual invocation paths.
Changes:
- infra/k8s/argo/train-template.yaml: gpu-pool default H100 → L40S
- infra/k8s/argo/train-multi-seed-template.yaml: same + cuda-compute
-cap default 90 → 89
- scripts/argo-train.sh: docstring / --help / compute-cap fallback
case all flip to L40S as the bare default; H100 becomes opt-in via
`--gpu-pool ci-training-h100` for 80 GB / sm_90 workloads
- scripts/argo-test.sh: --help text aligned
Other architectural defaults (data-source=mbp10 per
feedback_mbp10_mandatory; imbalance-bar-threshold=20.0 per the 2026-
05-10 OOM-prevention fix) are already correct in the template.
Verified via `argo-train.sh dqn --branch sp20-aux-h-fixed --sha HEAD
--baseline --dry-run` — rendered workflow shows cuda-compute-cap=89,
no explicit gpu-pool override (template default L40S in effect).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
At threshold=0.5 the sampler produced 209M bars from 209M trade ticks (1:1
ratio, essentially per-tick) on workflow f5wnd today. Feature extraction hit
54Gi/56Gi memory limit — near OOM. The default was wrong: with EWMA bypassed
(per 1aaf94306), threshold=0.5 contracts of imbalance is below the natural
ES.FUT trade size, so every trade fires a bar.
threshold=20 produces ~5-6M bars matching the volume-bar density baseline
(5.74M bars at 100 contracts). Math: 209M / 5.74M = 36×, so threshold needs
0.5 × 36 ≈ 18 → round to 20.
Three files updated atomically per feedback_no_partial_refactor:
- config/training/dqn-production.toml: 2.5 → 20.0 (wgdc8 left it at 2.5)
- infra/k8s/argo/train-template.yaml: workflow default 0.5 → 20.0
- infra/k8s/argo/train-multi-seed-template.yaml: workflow default 0.5 → 20.0
The 1:1 bar:tick observation alongside the price-continuity proof (22 jumps
out of 209M = ~1e-7 — front-month filter works) confirms the front-month
fix on this branch (6c1ab8850 + 3b5f17913) is correct. Threshold was the
remaining mistuning.
Cache-key implication: changing the threshold invalidates all existing
fxcache files (per the cache-key architectural fix). Next dispatch on this
branch will regenerate fxcache from scratch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Two architectural cleanups, both surfaced by the wgdc8 experiment
### Part 1: volume_bar_size in cache key
Mirrors the imbalance_bar_threshold/ewma_alpha fix from `f7718b376`. The
volume bar size constant (100 contracts/bar) was previously hardcoded and
not in the fxcache key. Tuning it would have hit the same fossilization
bug as imbalance_bar_threshold did pre-fix.
Changes:
- `Hyperparams.volume_bar_size: u64` field added (default 100, matches
`DEFAULT_VOLUME_BAR_SIZE` for backwards compat).
- TrainingProfile loader reads `volume_bar_size` TOML key.
- `calculate_dbn_cache_key_full` signature 7 → 8 args. Hashed via
`to_le_bytes()`. Test `test_cache_key_includes_volume_bar_size`
added; passes alongside the 5 existing tests.
- 4 callers updated atomically (per `feedback_no_partial_refactor`):
`discover_and_load`, `data_loading.rs:146`, `train_baseline_rl.rs:599`,
`precompute_features.rs:259,720`.
- `data_loading.rs:279` now passes `self.hyperparams.volume_bar_size`
to `build_volume_bars` instead of the hardcoded `DEFAULT_VOLUME_BAR_SIZE`.
- New `--volume-bar-size` CLI arg on both binaries (default 100).
- New Argo workflow params `volume-bar-size` (default "100") and
`data-source` (default "mbp10") on both `train-template.yaml` and
`train-multi-seed-template.yaml`. Threaded into precompute + trainer
invocations.
- `scripts/argo-train.sh` exposes `--volume-bar-size <n>` and
`--data-source <s>` for ad-hoc overrides.
### Part 2: OFI front-month filter (latent bug fix)
`crates/ml/examples/precompute_features.rs:539-557` (the OFI/VPIN/Kyle's
Lambda computation branch when MBP-10 + trades data is available) was
loading trades unfiltered for per-bar microstructure feature computation.
The volume bar formation path filters front-month per-file (line 354), but
the OFI path did not.
Effect pre-fix: during contract rollover windows (e.g., ESZ24 → ESH25),
OFI per-bar microstructure features included trades from BOTH contracts
simultaneously, distorting VPIN, Kyle's Lambda, and trade imbalance
signals. Severity in production: small (front-month dominates ES.FUT
volume by 10-100×) but real and present in every prior MBP-10+trades
production run.
Fix: mirror the per-file `filter_front_month` call from the volume bar
path. Volume bar formation and OFI computation now both see the same
in-month trade tape. Added log line shows raw vs filtered count per file
for transparency.
## Why bundled
Both fixes touch trade-data plumbing in `precompute_features.rs` and the
fxcache key contract. Per `feedback_no_partial_refactor`, related
architectural cleanups land atomically. Both surfaced from the same
wgdc8 audit; bundling avoids two cache-key-invalidating commits in
sequence (each would force full fxcache regen).
## Compatibility
- `volume_bar_size` defaults to 100 → existing wgdc7-equivalent runs
reproduce, but with a *new* fxcache key (the f7718b376-era cache file
is unreachable; harmless, can GC manually).
- OFI fix is strictly more correct; no opt-out needed. Existing models
trained on contaminated OFI features may show slight feature
distribution drift on first cache regen — expected, not a regression.
- `data_source = "ohlcv"` Argo param now possible; routes precompute
through volume bar branch directly. wgdc8 experiment uses this to test
bar resolution sensitivity at volume_bar_size=500 (5× DEFAULT).
Tests: 6/6 feature_cache tests pass. Workspace + examples compile clean.
Audit-doc: `docs/dqn-wire-up-audit.md` updated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## The bug (audit 2026-05-09)
`crates/ml/src/feature_cache.rs:calculate_dbn_cache_key_full` hashed only
`(symbol, data_source, dbn_filenames+sizes)` — NOT `imbalance_bar_threshold`
or `imbalance_bar_ewma_alpha`. Combined with `precompute_features.rs:346`
unconditionally calling `build_volume_bars` regardless of `data_source`,
this meant:
1. 14 audited production runs (Apr 11–May 8) all collided on the same
fxcache key (`a3f933aa...` / `c07c960a...`) regardless of TOML
`imbalance_bar_threshold` value
2. The imbalance-bar code path was reachable only via fxcache MISS, which
never happens in production because `ensure-fxcache` always populates
first
3. Every "tuning" of `imbalance_bar_threshold` across 16+ SP runs was a
silent no-op — the system was actually running volume bars at
DEFAULT_VOLUME_BAR_SIZE (100 contracts/bar)
## The fix (this commit)
**Part A — cache key includes bar formation params:**
- `calculate_dbn_cache_key_full` signature: 5 args → 7 args. Two new f64
params hashed via `to_le_bytes()`.
- 4 callers updated atomically (per `feedback_no_partial_refactor`).
- 2 new unit tests (`test_cache_key_includes_bar_threshold`,
`test_cache_key_includes_bar_alpha`) pin the contract.
**Part B — precompute_features actually USES data_source:**
- New CLI args `--imbalance-bar-threshold` (default 0.5) and
`--imbalance-bar-ewma-alpha` (default 0.1) on both train_baseline_rl
and precompute_features.
- `precompute_features.rs:346` now branches: when
`data_source == "mbp10"` AND `mbp10_data_dir.is_some()`, calls
`mbp10_to_imbalance_bars` instead of `build_volume_bars`.
**Argo plumbing:**
- `train-template.yaml` + `train-multi-seed-template.yaml`: new workflow
parameters threaded into BOTH precompute and trainer invocations so
both compute the same fxcache key.
- `scripts/argo-train.sh`: new CLI flags for ad-hoc overrides.
- ensure-fxcache regen path: removed `rm -f /feature-cache/*.fxcache`
(with bar-params now in key, parallel experiments coexist).
## Effects going forward
- Tuning `imbalance_bar_threshold` actually changes bar density
- Configuring `data_source = "mbp10"` actually produces imbalance bars
- Multiple parallel experiments at different thresholds coexist on PVC
- `dqn-production.toml: imbalance_bar_threshold = 0.5` no longer ignored
Default values match prior production behavior → existing wgdc7-equivalent
runs reproduce, just with a *new* fxcache key (the old volume-bar cache
file is still on disk but won't be hit; harmless, can GC manually).
Audit-doc: `docs/dqn-wire-up-audit.md` updated with full context.
Tests: 5/5 feature_cache tests pass, full workspace + examples compile clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The smoke template already compiled `train_baseline_rl`. Now also build
`evaluate_baseline` + `precompute_features` and copy + strip all 3 to
`/data/bin/$SHORT_SHA/` after PASS — exactly the layout that
train-multi-seed-template.yaml's `ensure-binary` cache check
(lines 235-246) looks up by SHA. A smoke-then-train sequence at the same
SHA now hits the cache and skips the ~6-min compile, exiting in the
~1-min pod-startup baseline.
- Drop readOnly:true on training-data PVC mount (cargo writes binaries
into /data/bin/$SHORT_SHA; read-side fxcache + market data unchanged)
- Build all 3 example binaries in a single cargo invocation (sccache-warm)
- Idempotent SHA-keyed dest dir; PASS-only so broken bins never cache
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
compile-and-deploy netpol allows port 2222 (gitlab-shell) and 8181
(webservice) but NOT port 5000 (gitlab-registry). The deps-cache build
needs port 5000 to push the resulting image, so kaniko was failing
with: dial tcp 10.32.4.209:5000: i/o timeout.
ci-pipeline netpol (used by build-ci-image-template, which also pushes
to gitlab-registry via kaniko) does allow port 5000 to app: registry.
Reuse that label.
Setting `app.kubernetes.io/component: deps-cache-build` left the pod
without any matching NetworkPolicy — the cluster's default-deny then
blocked egress to gitlab-shell:2222 and the git clone init container
timed out (verified in refresh-deps-cache-bgttw run).
Switch to `compile-and-deploy` so the pod picks up
argo-compile-and-deploy-workflow netpol's existing allow-list:
gitlab-shell:2222, gitlab-registry, gitlab-webservice, MinIO, etc.
Same egress targets the deps-cache build needs anyway.
The cluster's CronWorkflow CRD (argoproj.io/v1alpha1, current Argo
release) requires `spec.schedules` as an array. Older versions accepted
`spec.schedule` (singular string) as well; the newer CRD rejects it
with: `spec.schedules: Required value`.
Surfaced when applying refresh-deps-cache-template.yaml — the
WorkflowTemplate part applied fine, the CronWorkflow part failed.
The biggest unexplored win against CI cold-start: ship a Docker image
that carries a fully-built /cargo-target-prebuilt/ snapshot of the
workspace's third-party dependency rlibs, and rsync it into the
cargo-target-cpu PVC as the first step of compile-services.
Cold-start compile path before this commit (fresh node, fresh PVC):
1. cargo resolves Cargo.lock ~10s
2. cargo refreshes registry index ~30s
3. cargo compiles ~1500 dep crates ~3-5 min
4. cargo compiles workspace members ~1-2 min
Total cold compile: ~5-8 min
After this commit:
1. seed-deps-cache initContainer pull image ~30-60s (in-cluster registry)
2. rsync /cargo-target-prebuilt/ into PVC ~30-60s (~6-8 GB local)
3. cargo delta-compiles workspace members ~1-2 min
Total cold compile: ~2-4 min
Net saving: 2-4 min per cold-cache compile pod.
Steady-state (warm PVC, stamp file present): the seed initContainer
exits in ~50ms, no rsync.
Files:
- infra/docker/Dockerfile.ci-deps-cache:
Multi-stage. Stage 1 atop ci-builder-cpu, runs
`cargo build --locked --release --workspace || true` (|| true so
transient errors don't kill the cache build), then snapshots
target/release/{deps,.fingerprint,build} into /cargo-target-prebuilt.
Stage 2 thin Ubuntu + rsync, COPY's the snapshot. Default CMD just
prints the contents — actual entrypoint set by initContainer spec.
Used the "build everything" approach over cargo-chef-style stubbing
because it has zero special-case logic for build.rs / examples /
weird workspace shapes; image is ~1-2 GB bigger as a result, but
pulls fast from in-cluster registry.
- infra/k8s/argo/refresh-deps-cache-template.yaml:
WorkflowTemplate that drives a Kaniko build of the Dockerfile and
pushes to gitlab-registry/.../ci-builder-cpu-with-deps:nightly.
Includes a CronWorkflow that runs nightly at 03:00 UTC, suspended
by default (enable manually after first successful run validation).
Mirrors the existing build-ci-image-template.yaml pattern.
- infra/k8s/argo/compile-and-deploy-template.yaml:
Adds a `seed-deps-cache` initContainer to compile-services. Uses
rsync --ignore-existing (don't clobber newer artifacts the PVC may
already have from a prior build) and a stamp file
/cargo-target/cpu_deps_v${DEPS_VERSION}.stamp to short-circuit
re-seeding on warm pods. Tolerates a missing
/cargo-target-prebuilt dir (image not yet published) — emits a
WARN and continues without the seed.
- infra/k8s/argo/kustomization.yaml:
Registers the new template so kustomize/argo deploys it.
- infra/k8s/argo/README-deps-cache.md:
How to refresh manually, enable the cron, bump DEPS_VERSION,
debug a slow seed, and remove this when no longer needed.
Validation:
- python3 yaml.safe_load_all parses both compile-and-deploy and
refresh-deps-cache templates (1 + 2 docs respectively)
- cargo check --workspace --release --locked passes locally
- Dockerfile syntax visually inspected; the multi-stage COPY pattern
matches existing infra/docker/Dockerfile.* conventions
- Argo template structure mirrors build-ci-image-template.yaml which
is known-working in this cluster
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
wild (https://github.com/davidlattimore/wild) is a Rust-native linker,
typically 10-30% faster than mold on large release-LTO links. We have
~5 service binaries each doing release-LTO link, so the saving is
meaningful: ~30-90 sec wall-time on a fresh compile.
Approach (CI-only swap, zero local-dev impact):
1. Dockerfile.ci-builder-cpu installs wild 0.8.0 alongside mold (both
linkers present; revert path is trivial).
2. .cargo/config.toml keeps `-fuse-ld=mold` as the file default so
`cargo build` works locally without requiring wild on PATH.
3. compile-and-deploy-template.yaml's compile-services script does an
in-place sed substitution `mold -> wild` immediately before
`cargo build`, gated on `command -v wild` so a missing binary
silently falls back to mold instead of failing the build.
Why sed-in-script over RUSTFLAGS env var: RUSTFLAGS env var REPLACES
the entire target.<triple>.rustflags array (per cargo docs precedence:
env > target > build, mutually exclusive — they do NOT merge), which
would silently drop our existing -Wl,-z,relro/--as-needed/target-cpu
flags. Sed swap edits one token while preserving everything else.
Validation:
- python3 yaml.safe_load_all parses the template
- cargo check --workspace --release --locked succeeds locally (still
using mold per the unchanged config.toml default)
- Dockerfile syntax visually verified; wild tarball URL confirmed live
via curl https://api.github.com/repos/davidlattimore/wild/releases/latest
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rclone (~50 MB installed) is not used by any Argo step that runs in
ci-builder-cpu. The only Argo step that does use rclone
(build-web-dashboard) runs in node:22-alpine and apk-installs rclone
itself. Removing here saves an unused ~50 MB layer in every
ci-builder-cpu pull on every CI compile pod cold-start.
Tools deliberately KEPT despite the original task suggestion to remove
them:
- terragrunt + tofu — used by terragrunt-apply step in
ci-pipeline-template.yaml (runs in ci-builder-cpu)
- kubectl — used by apply-argo-templates step in
ci-pipeline-template.yaml (runs in ci-builder-cpu)
Removing those would break those pipeline steps.
Expected impact: ~50 MB image size reduction, ~3-7 sec cold-pull
savings per compile pod. Smaller than the original ~250 MB estimate
because terragrunt/tofu/kubectl had to stay.
Validation: visually inspected Dockerfile syntax (no docker daemon
available locally). Grepped infra/k8s/argo/*.yaml for rclone usage —
only build-web-dashboard (node:22-alpine) references it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two small but compounding fixes for the compile-services step:
1. CARGO_BUILD_JOBS=30 matches the cgroup limits.cpu="30" set on the
pod. Without it, cargo asks num_cpus::get_physical() (the host's
whole CPU count, e.g. 64 on a Scaleway PRO2-XL) and over-subscribes
vs the cgroup throttle. Expected: ~5-10% better CPU saturation
under load, fewer context-switch stalls.
2. cargo build --locked skips Cargo.lock resolver work and fails fast
if the lock has drifted (catches accidental Cargo.toml edits that
weren't re-resolved). Expected: ~5-15 sec saved per invocation,
loud error instead of silent re-resolve.
Validation:
- python3 yaml.safe_load_all parses the template cleanly
- cargo check --workspace --release --locked succeeds in foxhunt-brush-test
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mount the existing sccache-cpu PVC and set RUSTC_WRAPPER=sccache for
the regular service-build pipeline. Previously only train-template and
train-multi-seed-template used sccache; the production CI build had
zero rustc-level caching, so every CI run rebuilt every dep from
scratch.
Also set CARGO_INCREMENTAL=0 to override the workspace-wide
[build] incremental = true in .cargo/config.toml. sccache documented
limitation: cannot cache incremental rustc output. Local dev keeps
incremental via config.toml unchanged.
Print sccache --show-stats at end of build for visibility into cache
hit rates.
Mirror of train-template.yaml's sccache pattern; uses the sccache-cpu
PVC (20Gi, scw-bssd-retain) which was already provisioned but unmounted.
Expected impact: rustc cache hit rate jumps from 0% → 50-90% on
subsequent CI runs.
The L40S smoke template uses CARGO_TARGET_DIR=/cargo-target on a
persistent PVC, so every smoke probe builds incrementally on top of
artefacts left by prior probes. File deletions (e.g., regime_conditional.rs
in ff00af68a) can leave dangling rmeta/object references that perturb
downstream codegen between bisect runs — making the bisect result
contingent on which order probes were submitted in, not on the source.
Adds an optional `clean-cache` parameter (default `false`) which runs
`cargo clean -p ml -p ml-dqn --release` before the compile step. Other
crate artefacts (ml-core, ml-supervised, etc.) stay cached so the wipe
is bounded — fresh ml/ml-dqn compile in ~3-5 min vs ~30+ min full clean.
Use case: re-running a52d99613 + ff00af68a with `--clean-cache` to
verify the bisect under controlled build conditions. If the
broken/clean status flips with clean cache, the regression is
build-state-dependent rather than source-line; if it reproduces, the
source regression is real and bisect is definitive.
scripts/argo-smoke.sh exposes `--clean-cache` flag passing through
to the workflow parameter.
Adds a no-wrapper L40S smoke runner alongside the nsys/sanitizer
templates. Same compile + checkout + data-mount layout, no profiler
binary, no sanitizer instrumentation, no MinIO artefact upload — just
a plain test execution against the full training-data PVC.
Use case: validate a smoke test passes on the real 27-month dataset
when local data is too short (laptop only has 1 quarter of MBP-10/
trades, fxcache truncates below the 10-month minimum).
- infra/k8s/argo/smoke-test-template.yaml: WorkflowTemplate
smoke-test, entrypoint smoke-run, default test
multi_fold_convergence::test_multi_fold_convergence.
- infra/k8s/argo/kustomization.yaml: register the new template.
- infra/k8s/argo/argo-workflow-netpol.yaml: extend the
sanitizer/nsys NetworkPolicy podSelector to include smoke-test
(identical egress requirements: git fetch, ci-builder pull,
training-data PVC).
- scripts/argo-smoke.sh: thin wrapper mirroring argo-nsys.sh /
argo-sanitizer.sh (--multi-fold default, --test, --ref, --watch).
Verified: kustomize dry-run clean, kubectl apply -k creates the
template + reconfigures the netpol live in foxhunt namespace.
The original target 'foxhunt-training-artifacts' (cloned from
train-multi-seed-template) does not exist on the cluster MinIO.
Available production buckets: foxhunt-models, foxhunt-training-data,
foxhunt-training-results, foxhunt-binaries, foxhunt-backups,
foxhunt-gitlab-{artifacts,packages,registry}.
foxhunt-training-results is the right home for profiling artefacts
(model evaluation outputs already land there). Upload path is now
foxhunt-training-results/profiles/smoke/<short-sha>/profile-<pod>.nsys-rep.
Note: train-multi-seed-template has the same bucket-name typo —
production --profile uploads have been silently failing. Out of scope
for this commit, separate fix needed there.
NVIDIA's GPU metrics counters require elevated container perms
(/proc/sys/kernel/perf_event_paranoid + nvidia.com/gpu performance
counter device unlock). Containers in our Kapsule pods don't have
those — nsys exits exit-1 immediately with NVGPUCTRPERM.
Default to CUDA/NVTX/osrt traces only — kernel timeline + per-kernel
duration + call-graph context, which is what the bottleneck hunts
actually need. Counters can be re-enabled via --extra-args when
running on a relaxed node.
The test-data-pvc only had 3-4 months of MBP-10 (Q1 2024 only), causing
walk-forward fold generation to fail with 'Need at least 10 months of data'.
Production training uses training-data-pvc mounted at /data with subdirs
futures-baseline / futures-baseline-mbp10 / futures-baseline-trades —
that's the full 27-month dataset.
Switching all three env vars (FOXHUNT_TEST_DATA, FOXHUNT_MBP10_DATA,
FOXHUNT_TRADES_DATA) and the volume/mount to point at the production
data PVC. multi_fold_convergence's 6/2/2 walk-forward (10 months min)
now has ample data.
`grep -q` exits early on first match, closing stdin to upstream
`$TEST_BIN --list`. With pipefail enabled, that SIGPIPE on the upstream
process bubbles up as exit 141 and kills the workflow before the
sanitizer/nsys can launch.
Capture --list output once into a variable, then grep the captured
string (which doesn't pipe to a process). Same pattern for the
already-broken-but-defensive head -30 in the error branch.
Per feedback_mbp10_mandatory.md, MBP-10 + trades are mandatory inputs.
multi_fold_convergence now reads FOXHUNT_MBP10_DATA / FOXHUNT_TRADES_DATA
env vars, forwards them to train_baseline_rl as --mbp10-data-dir /
--trades-data-dir, and hard-fails if either path is missing.
Argo sanitizer-test + nsys-test templates set the env vars to PVC paths
/data/test-data/{mbp10,trades} and FOXHUNT_TEST_DATA points at
/data/test-data/ohlcv (PVC layout has lost+found/trades/mbp10/ohlcv as
top-level subdirs; the symbol ES.FUT lives under ohlcv/).
Without this, the smoke would silently fall back to tick-rule proxy
classification and validate a degraded model variant (no real OFI),
defeating the purpose of L40S validation. Audit doc updated per
Invariant 7.
`set -o pipefail` + `head -1` closing stdin produces SIGPIPE (exit 141)
from upstream `ls`/`grep`. The subshell + || true contains the failure
to the inner pipeline so the script's pipefail doesn't propagate it.
Sanitizer fired exit 141 immediately after the cargo build; nsys
reproduced the same exit on its parallel run, confirming the diagnosis.
The workflow sets CARGO_TARGET_DIR=/cargo-target so cargo writes binaries
to /cargo-target/release/deps/ml-*, not target/release/deps/ml-*. Both
templates were globbing the wrong path and failed with exit 3 (now
caught by the explicit error message).
Compile finished cleanly in 4m14s (sccache cache hit on first run after
re-checkout); test binary lookup then failed and the script exited at
the 'could not locate compiled test binary' check.
Both new workflow components were missing egress permissions and the
default-deny in the foxhunt namespace blocked their git fetch SSH to
gitlab-shell.foxhunt.svc.cluster.local:2222 — caused exit code 128
"Could not read from remote repository" before any GPU work started.
New `argo-sanitizer-nsys-workflow` policy mirrors the ci-pipeline
egress (DNS, K8s API, gitlab SSH on port 2222, MinIO 9000 for log
archive + nsys-rep upload, gitlab registry 5000 for ci-builder image
pull, webservice 8181 for registry token auth, external 80/443 for
mc binary download from dl.min.io).
train-template.yaml had a stray `value: "5"` line directly after
`value: "1.0"` under `- name: spread-ticks`. Stricter YAML parsers
(Go go-yaml used by kustomize) reject duplicate mapping keys and bail
out of the entire kustomize bundle with line-52 errors that mislead
diagnosis to unrelated files.
Permissive parsers (kubectl apply -f's path) silently take the last
value, which made the bug invisible until `kubectl apply -k` was
attempted.
The orphan was a leftover from an earlier edit (likely a max-folds=5
or similar parameter that was renamed but not fully removed).
The unquoted test name string contains '::' (Rust path separator). Some
strict YAML implementations (Go go-yaml when used by certain kustomize
paths) interpret it ambiguously. Quoting makes it always a string scalar.
Templates were applied directly via 'kubectl apply -f' which uses a
permissive parser; the unquoted form worked. Quoting makes them
also kustomize-compatible.
Two new one-shot WorkflowTemplates for validating smoke tests under
GPU-instrumentation tools that don't fit in the laptop's 4 GB VRAM:
- `sanitizer-test`: wraps cargo test smoke under `compute-sanitizer
--tool=memcheck|racecheck|synccheck|initcheck` with --target-processes all
so multi_fold_convergence's spawned `train_baseline_rl` subprocess is also
instrumented. Pre-builds train_baseline_rl example to avoid sanitizer
instrumenting cargo/rustc on the inner spawn. Triages internal-vs-real
errors and exits non-zero only on real bugs.
- `nsys-test`: wraps cargo test smoke under `nsys profile`. Captures CUDA +
NVTX + osrt traces with GPU metrics (ga10x set). Uploads .nsys-rep to
MinIO at foxhunt-training-artifacts/profiles/smoke/<short-sha>/, mirroring
the existing Plan 5 Task 3 production-training profile pattern in
train-multi-seed-template.
Wrapper scripts:
- scripts/argo-sanitizer.sh — `argo submit` wrapper, supports --multi-fold
shortcut for fold-boundary code paths (IQN sync, aux Adam reset,
iqn_readiness reset, MSE clamp) that single-fold tests cannot exercise.
- scripts/argo-nsys.sh — same shape as argo-sanitizer.sh, default test is
performance::test_real_data_single_epoch for broad coverage.
Why L40S: laptop RTX 3050 Ti's 4 GB cannot fit compute-sanitizer's
instrumentation metadata (~2-3x app VRAM) — sanitizer falls back to "didn't
track the launch" with 60k+ internal-allocation errors. L40S 48 GB has
ample headroom for both memcheck and nsys overhead.
Both templates compile cargo test --release --lib --no-run plus cargo build
--release --example train_baseline_rl on the cargo-target PVC. Compile time
dominated by sccache hit rate (production training image: 100% C/C++ cache,
~75% Rust cache after warmup).
Templates registered in kustomization.yaml — apply with `kubectl apply -k
infra/k8s/argo` before first submit.
The first L40S deploy attempt (workflow `train-multi-seed-z2llf`, terminated)
failed at startup with `error: unexpected argument '--fold' found` on every
job: `train_baseline_rl` is a multi-fold walk-forward executor that accepts
`--max-folds K`, NOT `--fold N`. The original P5T1 harness assumed the
opposite and fanned out N seeds × K folds = N*K jobs, each invoking the
binary with `--seed N --fold K`.
User chose Path B: pivot to one job per seed (each runs all K folds via the
existing `--max-folds` mechanism). Per-job runtime is K× longer, but fanout
drops from N*K=30 → N=5 (matches L40S pool capacity better) and the binary
contract becomes the one the binary actually has.
4 surface changes:
1. crates/ml/examples/train_baseline_rl.rs — add `--seed N` CLI arg
(default 42 — historic implicit value). Sets `FOXHUNT_SEED` env var at
startup BEFORE any CUDA module spins up. Logs the seed value at the
training start banner.
2. crates/ml/src/cuda_pipeline/mod.rs — add `global_seed()` (reads
`FOXHUNT_SEED`, default 42) + `mix_seed(base)` (SplitMix64 avalanche
so adjacent global seeds produce uncorrelated module seeds). Six call
sites updated to mix the global seed into their previously-hardcoded
constants:
- trainer/action.rs: GpuActionSelector seed (0xDEAD_BEEF_CAFE) + the
epsilon-greedy fallback StdRng (0xAC7_DEF0).
- cuda_pipeline/gpu_iqn_head.rs: IQN Xavier-init RNG (0x1CA_1234).
- cuda_pipeline/gpu_iql_trainer.rs: V(s) Xavier-init RNG (0x1C1_9ABC).
- cuda_pipeline/gpu_her.rs: random-donor RNG (0x4E4_5678).
- cuda_pipeline/gpu_ppo_collector.rs: rng_seeds Vec for PPO
experience-collector init + reset (0xAA0_5EED).
- trainer/training_loop.rs: per-epoch regime_dropout_seed.
3. infra/k8s/argo/train-multi-seed-template.yaml — drop `fold` parameter
from `train-single` template; binary invoked as `--seed "$SEED"
--max-folds {{workflow.parameters.folds}}` so the walk-forward sweep
happens inside the single training process. Drop `FOLD` env var. Update
the nsys-rep upload filename to drop the fold suffix. Update banners /
doc comments to reflect "one-job-per-seed" semantics.
4. scripts/argo-train.sh — matrix generator drops the inner fold loop.
Each emitted task carries only `seed=${s}` and depends on the same
ensure-fxcache + gpu-warmup. The dry-run synthetic marker switches from
`seed=${s} fold=${f}` to `seed=${s} max_folds=${FOLDS}` so test harnesses
count the new shape correctly.
5. scripts/tests/test_multi_seed_harness.sh — assertions updated:
- `--multi-seed 3 --folds 2` produces 3 tasks (was 6).
- Rendered binary command must include `--max-folds
{{workflow.parameters.folds}}` placeholder.
- Rendered template must declare `folds` workflow parameter (so
`argo submit -p folds=K` overrides the default).
- Rendered binary command must NOT contain any per-fold flag — this
catches the failure mode that broke the first L40S deploy.
- Backward-compat: `--multi-seed 1 --folds 1` preserves the existing
single-template path (no DAG matrix tasks emitted).
6. docs/dqn-wire-up-audit.md — adds 1 Wired row documenting the pivot,
the new `--seed`/`mix_seed` plumbing, all 6 RNG call sites, and the
end-to-end seed-variation verification result.
Validation:
cargo check --workspace clean at 11 warnings (workspace baseline preserved).
cargo build --release --example train_baseline_rl succeeds; --help shows
the new --seed flag with documented default 42.
Seed-variation end-to-end test on RTX 3050 Ti (1 fold × 2 epochs each):
--seed 42 → F0 best Sharpe = -9.7831, best_val_metric = 1.957244,
epoch-2 train Sharpe = -16.12, val_Sharpe = +1.11.
--seed 999 → F0 best Sharpe = +92.9341, best_val_metric = 2.161012,
epoch-2 train Sharpe = +92.93, val_Sharpe = -0.25.
Different best Sharpe / best_val_metric / epoch-2 train + val Sharpe
across seeds proves the seed actually propagates through the RNG init
paths and is not just accepted-and-ignored. The seed=42 numbers match
the prompt's "deterministic baseline" expectation (F0 = -9.7831 was
bit-identical pre-pivot because no global-seed plumbing existed).
./scripts/argo-train.sh dqn --multi-seed 5 --folds 6 --dry-run produces
exactly 5 WorkflowTask markers (train-s0..train-s4), each with
`--max-folds {{workflow.parameters.folds}}` in the binary invocation.
All 3 harness tests PASS:
- test_multi_seed_harness.sh: 5 PASS lines, exit 0.
- test_nsys_harness.sh: 4 PASS lines + ALL PASS, exit 0.
- test_tier_checks.sh: PASS overall (good-fixture passes, bad-fixture
surfaces expected check rejections), exit 0.
Backward compat: existing single-job `argo-train.sh` callers (no
`--multi-seed`, no `--folds`) route to the original `train-template.yaml`
unchanged. `--seed 42` is a no-op offset for the SplitMix64 mix at the call
sites — the trajectory shifts only when the user passes `--seed` explicitly,
matching the prompt's "default 42 (historic implicit value)" requirement.
L40S pool: argo-train.sh defaults `--gpu-pool ci-training-h100`; user passes
`--gpu-pool ci-training-l40s` at deploy time. No script default change
(per constraint 5).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- argo-train.sh: --profile flag forces multi-seed render path so the
nsys wrapper + foxhunt-training-artifacts upload step are visible in
--dry-run YAML without cluster contact (test surface).
- train-multi-seed-template.yaml: new `profile` parameter (default
"false") gates the per-(seed, fold) `nsys profile
--capture-range=cudaProfilerApi` wrapper and the `mc cp` upload to
foxhunt-training-artifacts/profiles/<sha>/. mc binary fetched
on-demand (ci-builder image lacks it). MinIO creds optional —
upload warn-skips if absent.
- Dockerfile.foxhunt-training-runtime: install nsight-systems-cli
unpinned (pinning the stale 2024.4.1.61-1 from earlier plans
breaks builds when apt index advances).
- minio.yaml: add foxhunt-training-artifacts bucket to minio-init.
- compare-nsys-profiles.py: V0 regression detector — compares
cuda_gpu_kern_sum total_ns / epoch_count between two profiles;
exits 1 on >20% slowdown. NVTX per-epoch ranges deferred to T5.
- tests/test_nsys_harness.sh: dry-run grep test — verifies both
required strings appear when --profile is set, and that the
default (no --profile) path keeps profile=false in the rendered
template.
- dqn-wire-up-audit.md: Plan 5 Task 3 row added documenting the
harness + the baseline-capture deferral to T5.
Backward compat: test_multi_seed_harness.sh from P5T1 still PASS.
Adds the orchestration surface for Plan 5 Task 1 — Multi-Seed × Multi-Fold
Validation Harness:
- scripts/argo-train.sh: new --multi-seed N, --folds K, --tag T, --dry-run
flags. Default --multi-seed 1 --folds 1 routes to the existing
train-template.yaml (backward compat — existing callers unchanged). When
N>1 or K>1 the script renders train-multi-seed-template.yaml with an
inline-generated (seed, fold) matrix and either prints the YAML
(--dry-run) or applies + submits it.
- infra/k8s/argo/train-multi-seed-template.yaml: new WorkflowTemplate with
entrypoint multi-seed-matrix → ensure-binary, gpu-warmup, ensure-fxcache,
then N*K parallel train-single instances. Each train-single receives
seed/fold via inputs.parameters and forwards them to the training binary
via --seed/--fold CLI args + SEED/FOLD env vars. The dag.tasks placeholder
`# __MATRIX_TASKS__` is substituted programmatically by argo-train.sh
(awk) — no hand-written 30-task matrix.
- scripts/tests/test_multi_seed_harness.sh: dry-run regression test.
Asserts --multi-seed 3 --folds 2 emits 6 WorkflowTask markers AND
--multi-seed 1 --folds 1 emits zero (single-template path preserved).
Validation:
- argo lint --offline passes on both the source template and the rendered
3x2 / 5x6 outputs.
- test_multi_seed_harness.sh passes locally.
- Single-job dry-run still produces the unchanged train-template YAML.
Note: plan Step 0.1 pre-plan check expects ISV_TOTAL_DIM=72 and seven
ATTN_*_FOCUS_EMA_INDEX slots — both stale (Plan 4 landed
ISV_TOTAL_DIM=117 and the VSN_MAG_EMA / VSN_DIR_EMA / MAMBA2_RETENTION_EMA
slots instead). Plan 4 validation doc never landed (T8 deferred → Plan 5
T5). T1 is pure infrastructure that builds the harness consumed by T5,
so the stale pre-plan expectations do not block this commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The project's Cargo.toml and .cargo/config.toml both set
`incremental = true`, which makes rustc emit per-query save-analysis
artifacts that sccache does not cache. Result: even though
RUSTC_WRAPPER=sccache was set, the rustc calls wrote incremental
state that subsequent builds saw as stale and recompiled anyway.
Setting CARGO_INCREMENTAL=0 in the ensure-binary env forces rustc
to emit pure object output that sccache hashes uniformly on
(source + args). Same SHA → full cache hit; small source change →
only dirty crates recompile.
Does not change the service compile-and-deploy-template path,
which intentionally uses cargo-incremental against a persistent
/cargo-target-cpu PVC for repeated rebuilds of a narrow set of
service binaries.
Applies from the next workflow submission onward (train-5wb4n
already compiled under the old config and is running).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Every new commit triggered a full workspace recompile (~6min for a 2-file
change) in ensure-binary. The binary cache at /data/bin/$SHA is SHA-keyed
so it misses on every new commit. The cargo incremental cache at
/cargo-target/target is file-mtime keyed, which git checkout invalidates
whenever it retouches files.
sccache (already present in ci-builder image, verified via `sccache --version`
in Dockerfile) is content-hash keyed — identical source → identical hit
regardless of mtime. Sits in front of rustc via RUSTC_WRAPPER. Cache dir
on the same cargo-target-cuda PVC (SCCACHE_DIR=/cargo-target/sccache) so:
- Combines with cargo's target/ cache on one volume, no separate PVC
- Disk-local hits, no network round-trip per rustc invocation
- Survives across pods and commits — the git-checkout mtime flip that
breaks cargo's fingerprints doesn't affect content-hash caching
40G cache size cap — the cargo-target-cuda PVC is large enough that
sccache won't evict useful entries under normal use.
Expected behaviour: first build after this commit is still a full rebuild
(sccache cache empty). Subsequent builds for the same CUDA_COMPUTE_CAP
should hit compiled objects at near-zero cost for crates whose source
hasn't changed.
Applied to cluster via `kubectl apply -n foxhunt -f` — WorkflowTemplate
is live before the next ./scripts/argo-train.sh invocation.
Closes task #34.
Two bugs caught by the L40S smoke (train-qhgj6) that couldn't surface on
local RTX-3050 single-fold runs:
1. PER dtoh inside CUDA Graph capture (Fold 1 crash)
Failure: CUDA_ERROR_STREAM_CAPTURE_INVALIDATED at per_prefix_scan on
Fold 1 re-capture. Chain: fused_training parent graph captures →
memcpy_dtoh + cuStreamSynchronize in gpu_replay_buffer::update_priorities_gpu
(health<0.8 diversity path) poisons the stream → subsequent per_sample
kernel on the same stream sees an invalidated capture context.
The prior comment claimed "runs once per epoch, DtoH cost acceptable"
— wrong, it runs every priority update when health<0.8 (common during
Fold handoff when health_cache is re-seeded low). Any dtoh inside
capture invalidates regardless of latency.
Proper fix (no shortcut):
* New kernel actions_sum_scale_reduce_u32 — single-block deterministic
tree reduction over sample_actions (u32) → writes (sum*1000)/n as i32
to a device-accessible slot. No atomics (consistent with the 1/N
determinism policy from commit c82386500).
* mean_action_scaled storage is pinned + device-mapped (cuMemAllocHost
+ cuMemHostGetDevicePointer — same pattern as rng_step_dev_ptr and
size_dev_ptr elsewhere in the file). Zero-copy between host and
device, graph-safe, no explicit free needed (process-exit cleanup,
matches existing pattern).
* pow_alpha_diverse_f32 now takes const int* mean_action_scaled_ptr
and does a plain global load — NOT __ldg. The read-only cache used
by __ldg is not guaranteed coherent with device-mapped host memory;
multi-trial smoke regression caught it (median q_gap collapsed
from 2.0 → 0.15 with __ldg, recovered to 2.8 with plain load).
Verified: multi-trial smoke 5/5 pass, median_q_gap=2.80 (beats 2.00
baseline), Best Sharpe peaks 19-30 per trial. No stream capture
invalidation.
2. evaluate step CLI drift in Argo template
evaluate_baseline's Args struct uses --models-dir and --output (single
file path). Template was passing --checkpoint-dir and --output-dir,
causing clap to reject the invocation. Fixed argument names + added
mkdir for the eval subdir + updated the comment to pin the source of
truth for future drift catches.
Both fixes are graph-capture-clean and match the "wire properly or delete"
discipline. No masking, no feature flags, no dead params.
Persist the current dev-mode shutdown state in YAML so `kustomize build
| kubectl apply` won't quietly restart paused services.
Scope: foxhunt application services only. Core infra (GitLab, Minio,
Argo, Postgres, Redis) unchanged.
Paused:
- api, web-dashboard (dashboard stack, not used for CLI dev)
- trading-service, trading-agent-service, broker-gateway, ib-gateway
(live trading — not running in dev)
- backtesting-service (Argo workflows run standalone without it)
- data-acquisition-service (training uses cached .fxcache, not live feed)
- ml-training-service (training runs via scripts/argo-train.sh)
Each manifest has a restore-command comment documenting the rollback:
`kubectl -n foxhunt scale deployment <name> --replicas=1`.
Per user directive refined from previous commit: pushes to main should
trigger docker image rebuilds (when infra/docker/ changed), nothing else.
All other workflows run via scripts/manual invocation.
Sensor: restored body.ref filter to refs/heads/main (was set to a
never-match sentinel in the previous blanket-disable commit). Comment
updated to document the per-task policy.
DAG: set `when: "false"` on four non-docker tasks:
- build-web-dashboard (was: needs-dashboard)
- test-gate (was: needs-code — the CPU-heavy clippy+test gate)
- apply-argo-templates (was: needs-argo-templates)
- terragrunt-apply (was: needs-infra)
Kept enabled (still gated on detect-changes.docker-images == true):
- rebuild-ci-builder
- rebuild-ci-builder-cpu
- rebuild-runtime
- rebuild-training-runtime
gpu-test was already `when: "false"`; compile-and-deploy lives in a
separate template (manual webhook only, auto-compile-config=false).
Deploy: `kubectl apply -f infra/k8s/argo/events/ci-pipeline-sensor.yaml
infra/k8s/argo/ci-pipeline-template.yaml` (or re-apply via the existing
deploy path if one covers argo/).
Per user directive: pushes to main must not trigger any Argo compute
(not even the CPU-heavy test-gate). Changed the sensor's body.ref
filter value from "refs/heads/main" to a never-matching sentinel.
The eventsource still receives GitLab push webhooks; the sensor
evaluates them and rejects every one.
To re-enable: restore the value to "refs/heads/main" and
`kubectl apply -f infra/k8s/argo/events/ci-pipeline-sensor.yaml`.
Manual trigger paths (unchanged):
- ci-pipeline: argo submit --from=wftmpl/ci-pipeline -p commit-sha=<SHA>
- compile-and-deploy: curl -X POST http://workflow-trigger-eventsource-svc.foxhunt:12001/compile-deploy \
-H 'Content-Type: application/json' -d '{"commit_sha": "<SHA>"}'
Source: docs.nvidia.com/nsight-systems/InstallationGuide
Repo: developer.download.nvidia.com/devtools/repos/ubuntu2404/amd64/
Key: 7fa2af80.pub from CUDA repo
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Output: /data/nsys/ on training-data PVC (survives pod GC)
--show-output=true prints summary to pod logs for mid-run visibility
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Enables nsys profiling for CUDA graph node-level traces on H100.
Use --sanitizer nsys in argo-train.sh to activate.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use --sanitizer nsys to wrap training binary with nsys profile.
Captures CUDA graph node-level traces + GPU metrics.
Output: /workspace/output/nsys_profile.nsys-rep
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pod logs are now archived to MinIO before pod GC. Historical logs
accessible via `argo logs <workflow>` after pod completion.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CLI arg was removed in cost-driven hold timing but Argo template
still passed it, causing exit code 2 on H100 deploy.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
