The smoke template already compiled `train_baseline_rl`. Now also build
`evaluate_baseline` + `precompute_features` and copy + strip all 3 to
`/data/bin/$SHORT_SHA/` after PASS — exactly the layout that
train-multi-seed-template.yaml's `ensure-binary` cache check
(lines 235-246) looks up by SHA. A smoke-then-train sequence at the same
SHA now hits the cache and skips the ~6-min compile, exiting in the
~1-min pod-startup baseline.
- Drop readOnly:true on training-data PVC mount (cargo writes binaries
into /data/bin/$SHORT_SHA; read-side fxcache + market data unchanged)
- Build all 3 example binaries in a single cargo invocation (sccache-warm)
- Idempotent SHA-keyed dest dir; PASS-only so broken bins never cache
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
compile-and-deploy netpol allows port 2222 (gitlab-shell) and 8181
(webservice) but NOT port 5000 (gitlab-registry). The deps-cache build
needs port 5000 to push the resulting image, so kaniko was failing
with: dial tcp 10.32.4.209:5000: i/o timeout.
ci-pipeline netpol (used by build-ci-image-template, which also pushes
to gitlab-registry via kaniko) does allow port 5000 to app: registry.
Reuse that label.
Setting `app.kubernetes.io/component: deps-cache-build` left the pod
without any matching NetworkPolicy — the cluster's default-deny then
blocked egress to gitlab-shell:2222 and the git clone init container
timed out (verified in refresh-deps-cache-bgttw run).
Switch to `compile-and-deploy` so the pod picks up
argo-compile-and-deploy-workflow netpol's existing allow-list:
gitlab-shell:2222, gitlab-registry, gitlab-webservice, MinIO, etc.
Same egress targets the deps-cache build needs anyway.
The cluster's CronWorkflow CRD (argoproj.io/v1alpha1, current Argo
release) requires `spec.schedules` as an array. Older versions accepted
`spec.schedule` (singular string) as well; the newer CRD rejects it
with: `spec.schedules: Required value`.
Surfaced when applying refresh-deps-cache-template.yaml — the
WorkflowTemplate part applied fine, the CronWorkflow part failed.
The biggest unexplored win against CI cold-start: ship a Docker image
that carries a fully-built /cargo-target-prebuilt/ snapshot of the
workspace's third-party dependency rlibs, and rsync it into the
cargo-target-cpu PVC as the first step of compile-services.
Cold-start compile path before this commit (fresh node, fresh PVC):
1. cargo resolves Cargo.lock ~10s
2. cargo refreshes registry index ~30s
3. cargo compiles ~1500 dep crates ~3-5 min
4. cargo compiles workspace members ~1-2 min
Total cold compile: ~5-8 min
After this commit:
1. seed-deps-cache initContainer pull image ~30-60s (in-cluster registry)
2. rsync /cargo-target-prebuilt/ into PVC ~30-60s (~6-8 GB local)
3. cargo delta-compiles workspace members ~1-2 min
Total cold compile: ~2-4 min
Net saving: 2-4 min per cold-cache compile pod.
Steady-state (warm PVC, stamp file present): the seed initContainer
exits in ~50ms, no rsync.
Files:
- infra/docker/Dockerfile.ci-deps-cache:
Multi-stage. Stage 1 atop ci-builder-cpu, runs
`cargo build --locked --release --workspace || true` (|| true so
transient errors don't kill the cache build), then snapshots
target/release/{deps,.fingerprint,build} into /cargo-target-prebuilt.
Stage 2 thin Ubuntu + rsync, COPY's the snapshot. Default CMD just
prints the contents — actual entrypoint set by initContainer spec.
Used the "build everything" approach over cargo-chef-style stubbing
because it has zero special-case logic for build.rs / examples /
weird workspace shapes; image is ~1-2 GB bigger as a result, but
pulls fast from in-cluster registry.
- infra/k8s/argo/refresh-deps-cache-template.yaml:
WorkflowTemplate that drives a Kaniko build of the Dockerfile and
pushes to gitlab-registry/.../ci-builder-cpu-with-deps:nightly.
Includes a CronWorkflow that runs nightly at 03:00 UTC, suspended
by default (enable manually after first successful run validation).
Mirrors the existing build-ci-image-template.yaml pattern.
- infra/k8s/argo/compile-and-deploy-template.yaml:
Adds a `seed-deps-cache` initContainer to compile-services. Uses
rsync --ignore-existing (don't clobber newer artifacts the PVC may
already have from a prior build) and a stamp file
/cargo-target/cpu_deps_v${DEPS_VERSION}.stamp to short-circuit
re-seeding on warm pods. Tolerates a missing
/cargo-target-prebuilt dir (image not yet published) — emits a
WARN and continues without the seed.
- infra/k8s/argo/kustomization.yaml:
Registers the new template so kustomize/argo deploys it.
- infra/k8s/argo/README-deps-cache.md:
How to refresh manually, enable the cron, bump DEPS_VERSION,
debug a slow seed, and remove this when no longer needed.
Validation:
- python3 yaml.safe_load_all parses both compile-and-deploy and
refresh-deps-cache templates (1 + 2 docs respectively)
- cargo check --workspace --release --locked passes locally
- Dockerfile syntax visually inspected; the multi-stage COPY pattern
matches existing infra/docker/Dockerfile.* conventions
- Argo template structure mirrors build-ci-image-template.yaml which
is known-working in this cluster
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
wild (https://github.com/davidlattimore/wild) is a Rust-native linker,
typically 10-30% faster than mold on large release-LTO links. We have
~5 service binaries each doing release-LTO link, so the saving is
meaningful: ~30-90 sec wall-time on a fresh compile.
Approach (CI-only swap, zero local-dev impact):
1. Dockerfile.ci-builder-cpu installs wild 0.8.0 alongside mold (both
linkers present; revert path is trivial).
2. .cargo/config.toml keeps `-fuse-ld=mold` as the file default so
`cargo build` works locally without requiring wild on PATH.
3. compile-and-deploy-template.yaml's compile-services script does an
in-place sed substitution `mold -> wild` immediately before
`cargo build`, gated on `command -v wild` so a missing binary
silently falls back to mold instead of failing the build.
Why sed-in-script over RUSTFLAGS env var: RUSTFLAGS env var REPLACES
the entire target.<triple>.rustflags array (per cargo docs precedence:
env > target > build, mutually exclusive — they do NOT merge), which
would silently drop our existing -Wl,-z,relro/--as-needed/target-cpu
flags. Sed swap edits one token while preserving everything else.
Validation:
- python3 yaml.safe_load_all parses the template
- cargo check --workspace --release --locked succeeds locally (still
using mold per the unchanged config.toml default)
- Dockerfile syntax visually verified; wild tarball URL confirmed live
via curl https://api.github.com/repos/davidlattimore/wild/releases/latest
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rclone (~50 MB installed) is not used by any Argo step that runs in
ci-builder-cpu. The only Argo step that does use rclone
(build-web-dashboard) runs in node:22-alpine and apk-installs rclone
itself. Removing here saves an unused ~50 MB layer in every
ci-builder-cpu pull on every CI compile pod cold-start.
Tools deliberately KEPT despite the original task suggestion to remove
them:
- terragrunt + tofu — used by terragrunt-apply step in
ci-pipeline-template.yaml (runs in ci-builder-cpu)
- kubectl — used by apply-argo-templates step in
ci-pipeline-template.yaml (runs in ci-builder-cpu)
Removing those would break those pipeline steps.
Expected impact: ~50 MB image size reduction, ~3-7 sec cold-pull
savings per compile pod. Smaller than the original ~250 MB estimate
because terragrunt/tofu/kubectl had to stay.
Validation: visually inspected Dockerfile syntax (no docker daemon
available locally). Grepped infra/k8s/argo/*.yaml for rclone usage —
only build-web-dashboard (node:22-alpine) references it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two small but compounding fixes for the compile-services step:
1. CARGO_BUILD_JOBS=30 matches the cgroup limits.cpu="30" set on the
pod. Without it, cargo asks num_cpus::get_physical() (the host's
whole CPU count, e.g. 64 on a Scaleway PRO2-XL) and over-subscribes
vs the cgroup throttle. Expected: ~5-10% better CPU saturation
under load, fewer context-switch stalls.
2. cargo build --locked skips Cargo.lock resolver work and fails fast
if the lock has drifted (catches accidental Cargo.toml edits that
weren't re-resolved). Expected: ~5-15 sec saved per invocation,
loud error instead of silent re-resolve.
Validation:
- python3 yaml.safe_load_all parses the template cleanly
- cargo check --workspace --release --locked succeeds in foxhunt-brush-test
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mount the existing sccache-cpu PVC and set RUSTC_WRAPPER=sccache for
the regular service-build pipeline. Previously only train-template and
train-multi-seed-template used sccache; the production CI build had
zero rustc-level caching, so every CI run rebuilt every dep from
scratch.
Also set CARGO_INCREMENTAL=0 to override the workspace-wide
[build] incremental = true in .cargo/config.toml. sccache documented
limitation: cannot cache incremental rustc output. Local dev keeps
incremental via config.toml unchanged.
Print sccache --show-stats at end of build for visibility into cache
hit rates.
Mirror of train-template.yaml's sccache pattern; uses the sccache-cpu
PVC (20Gi, scw-bssd-retain) which was already provisioned but unmounted.
Expected impact: rustc cache hit rate jumps from 0% → 50-90% on
subsequent CI runs.
The L40S smoke template uses CARGO_TARGET_DIR=/cargo-target on a
persistent PVC, so every smoke probe builds incrementally on top of
artefacts left by prior probes. File deletions (e.g., regime_conditional.rs
in ff00af68a) can leave dangling rmeta/object references that perturb
downstream codegen between bisect runs — making the bisect result
contingent on which order probes were submitted in, not on the source.
Adds an optional `clean-cache` parameter (default `false`) which runs
`cargo clean -p ml -p ml-dqn --release` before the compile step. Other
crate artefacts (ml-core, ml-supervised, etc.) stay cached so the wipe
is bounded — fresh ml/ml-dqn compile in ~3-5 min vs ~30+ min full clean.
Use case: re-running a52d99613 + ff00af68a with `--clean-cache` to
verify the bisect under controlled build conditions. If the
broken/clean status flips with clean cache, the regression is
build-state-dependent rather than source-line; if it reproduces, the
source regression is real and bisect is definitive.
scripts/argo-smoke.sh exposes `--clean-cache` flag passing through
to the workflow parameter.
Adds a no-wrapper L40S smoke runner alongside the nsys/sanitizer
templates. Same compile + checkout + data-mount layout, no profiler
binary, no sanitizer instrumentation, no MinIO artefact upload — just
a plain test execution against the full training-data PVC.
Use case: validate a smoke test passes on the real 27-month dataset
when local data is too short (laptop only has 1 quarter of MBP-10/
trades, fxcache truncates below the 10-month minimum).
- infra/k8s/argo/smoke-test-template.yaml: WorkflowTemplate
smoke-test, entrypoint smoke-run, default test
multi_fold_convergence::test_multi_fold_convergence.
- infra/k8s/argo/kustomization.yaml: register the new template.
- infra/k8s/argo/argo-workflow-netpol.yaml: extend the
sanitizer/nsys NetworkPolicy podSelector to include smoke-test
(identical egress requirements: git fetch, ci-builder pull,
training-data PVC).
- scripts/argo-smoke.sh: thin wrapper mirroring argo-nsys.sh /
argo-sanitizer.sh (--multi-fold default, --test, --ref, --watch).
Verified: kustomize dry-run clean, kubectl apply -k creates the
template + reconfigures the netpol live in foxhunt namespace.
The original target 'foxhunt-training-artifacts' (cloned from
train-multi-seed-template) does not exist on the cluster MinIO.
Available production buckets: foxhunt-models, foxhunt-training-data,
foxhunt-training-results, foxhunt-binaries, foxhunt-backups,
foxhunt-gitlab-{artifacts,packages,registry}.
foxhunt-training-results is the right home for profiling artefacts
(model evaluation outputs already land there). Upload path is now
foxhunt-training-results/profiles/smoke/<short-sha>/profile-<pod>.nsys-rep.
Note: train-multi-seed-template has the same bucket-name typo —
production --profile uploads have been silently failing. Out of scope
for this commit, separate fix needed there.
NVIDIA's GPU metrics counters require elevated container perms
(/proc/sys/kernel/perf_event_paranoid + nvidia.com/gpu performance
counter device unlock). Containers in our Kapsule pods don't have
those — nsys exits exit-1 immediately with NVGPUCTRPERM.
Default to CUDA/NVTX/osrt traces only — kernel timeline + per-kernel
duration + call-graph context, which is what the bottleneck hunts
actually need. Counters can be re-enabled via --extra-args when
running on a relaxed node.
The test-data-pvc only had 3-4 months of MBP-10 (Q1 2024 only), causing
walk-forward fold generation to fail with 'Need at least 10 months of data'.
Production training uses training-data-pvc mounted at /data with subdirs
futures-baseline / futures-baseline-mbp10 / futures-baseline-trades —
that's the full 27-month dataset.
Switching all three env vars (FOXHUNT_TEST_DATA, FOXHUNT_MBP10_DATA,
FOXHUNT_TRADES_DATA) and the volume/mount to point at the production
data PVC. multi_fold_convergence's 6/2/2 walk-forward (10 months min)
now has ample data.
`grep -q` exits early on first match, closing stdin to upstream
`$TEST_BIN --list`. With pipefail enabled, that SIGPIPE on the upstream
process bubbles up as exit 141 and kills the workflow before the
sanitizer/nsys can launch.
Capture --list output once into a variable, then grep the captured
string (which doesn't pipe to a process). Same pattern for the
already-broken-but-defensive head -30 in the error branch.
Per feedback_mbp10_mandatory.md, MBP-10 + trades are mandatory inputs.
multi_fold_convergence now reads FOXHUNT_MBP10_DATA / FOXHUNT_TRADES_DATA
env vars, forwards them to train_baseline_rl as --mbp10-data-dir /
--trades-data-dir, and hard-fails if either path is missing.
Argo sanitizer-test + nsys-test templates set the env vars to PVC paths
/data/test-data/{mbp10,trades} and FOXHUNT_TEST_DATA points at
/data/test-data/ohlcv (PVC layout has lost+found/trades/mbp10/ohlcv as
top-level subdirs; the symbol ES.FUT lives under ohlcv/).
Without this, the smoke would silently fall back to tick-rule proxy
classification and validate a degraded model variant (no real OFI),
defeating the purpose of L40S validation. Audit doc updated per
Invariant 7.
`set -o pipefail` + `head -1` closing stdin produces SIGPIPE (exit 141)
from upstream `ls`/`grep`. The subshell + || true contains the failure
to the inner pipeline so the script's pipefail doesn't propagate it.
Sanitizer fired exit 141 immediately after the cargo build; nsys
reproduced the same exit on its parallel run, confirming the diagnosis.
The workflow sets CARGO_TARGET_DIR=/cargo-target so cargo writes binaries
to /cargo-target/release/deps/ml-*, not target/release/deps/ml-*. Both
templates were globbing the wrong path and failed with exit 3 (now
caught by the explicit error message).
Compile finished cleanly in 4m14s (sccache cache hit on first run after
re-checkout); test binary lookup then failed and the script exited at
the 'could not locate compiled test binary' check.
Both new workflow components were missing egress permissions and the
default-deny in the foxhunt namespace blocked their git fetch SSH to
gitlab-shell.foxhunt.svc.cluster.local:2222 — caused exit code 128
"Could not read from remote repository" before any GPU work started.
New `argo-sanitizer-nsys-workflow` policy mirrors the ci-pipeline
egress (DNS, K8s API, gitlab SSH on port 2222, MinIO 9000 for log
archive + nsys-rep upload, gitlab registry 5000 for ci-builder image
pull, webservice 8181 for registry token auth, external 80/443 for
mc binary download from dl.min.io).
train-template.yaml had a stray `value: "5"` line directly after
`value: "1.0"` under `- name: spread-ticks`. Stricter YAML parsers
(Go go-yaml used by kustomize) reject duplicate mapping keys and bail
out of the entire kustomize bundle with line-52 errors that mislead
diagnosis to unrelated files.
Permissive parsers (kubectl apply -f's path) silently take the last
value, which made the bug invisible until `kubectl apply -k` was
attempted.
The orphan was a leftover from an earlier edit (likely a max-folds=5
or similar parameter that was renamed but not fully removed).
The unquoted test name string contains '::' (Rust path separator). Some
strict YAML implementations (Go go-yaml when used by certain kustomize
paths) interpret it ambiguously. Quoting makes it always a string scalar.
Templates were applied directly via 'kubectl apply -f' which uses a
permissive parser; the unquoted form worked. Quoting makes them
also kustomize-compatible.
Two new one-shot WorkflowTemplates for validating smoke tests under
GPU-instrumentation tools that don't fit in the laptop's 4 GB VRAM:
- `sanitizer-test`: wraps cargo test smoke under `compute-sanitizer
--tool=memcheck|racecheck|synccheck|initcheck` with --target-processes all
so multi_fold_convergence's spawned `train_baseline_rl` subprocess is also
instrumented. Pre-builds train_baseline_rl example to avoid sanitizer
instrumenting cargo/rustc on the inner spawn. Triages internal-vs-real
errors and exits non-zero only on real bugs.
- `nsys-test`: wraps cargo test smoke under `nsys profile`. Captures CUDA +
NVTX + osrt traces with GPU metrics (ga10x set). Uploads .nsys-rep to
MinIO at foxhunt-training-artifacts/profiles/smoke/<short-sha>/, mirroring
the existing Plan 5 Task 3 production-training profile pattern in
train-multi-seed-template.
Wrapper scripts:
- scripts/argo-sanitizer.sh — `argo submit` wrapper, supports --multi-fold
shortcut for fold-boundary code paths (IQN sync, aux Adam reset,
iqn_readiness reset, MSE clamp) that single-fold tests cannot exercise.
- scripts/argo-nsys.sh — same shape as argo-sanitizer.sh, default test is
performance::test_real_data_single_epoch for broad coverage.
Why L40S: laptop RTX 3050 Ti's 4 GB cannot fit compute-sanitizer's
instrumentation metadata (~2-3x app VRAM) — sanitizer falls back to "didn't
track the launch" with 60k+ internal-allocation errors. L40S 48 GB has
ample headroom for both memcheck and nsys overhead.
Both templates compile cargo test --release --lib --no-run plus cargo build
--release --example train_baseline_rl on the cargo-target PVC. Compile time
dominated by sccache hit rate (production training image: 100% C/C++ cache,
~75% Rust cache after warmup).
Templates registered in kustomization.yaml — apply with `kubectl apply -k
infra/k8s/argo` before first submit.
The first L40S deploy attempt (workflow `train-multi-seed-z2llf`, terminated)
failed at startup with `error: unexpected argument '--fold' found` on every
job: `train_baseline_rl` is a multi-fold walk-forward executor that accepts
`--max-folds K`, NOT `--fold N`. The original P5T1 harness assumed the
opposite and fanned out N seeds × K folds = N*K jobs, each invoking the
binary with `--seed N --fold K`.
User chose Path B: pivot to one job per seed (each runs all K folds via the
existing `--max-folds` mechanism). Per-job runtime is K× longer, but fanout
drops from N*K=30 → N=5 (matches L40S pool capacity better) and the binary
contract becomes the one the binary actually has.
4 surface changes:
1. crates/ml/examples/train_baseline_rl.rs — add `--seed N` CLI arg
(default 42 — historic implicit value). Sets `FOXHUNT_SEED` env var at
startup BEFORE any CUDA module spins up. Logs the seed value at the
training start banner.
2. crates/ml/src/cuda_pipeline/mod.rs — add `global_seed()` (reads
`FOXHUNT_SEED`, default 42) + `mix_seed(base)` (SplitMix64 avalanche
so adjacent global seeds produce uncorrelated module seeds). Six call
sites updated to mix the global seed into their previously-hardcoded
constants:
- trainer/action.rs: GpuActionSelector seed (0xDEAD_BEEF_CAFE) + the
epsilon-greedy fallback StdRng (0xAC7_DEF0).
- cuda_pipeline/gpu_iqn_head.rs: IQN Xavier-init RNG (0x1CA_1234).
- cuda_pipeline/gpu_iql_trainer.rs: V(s) Xavier-init RNG (0x1C1_9ABC).
- cuda_pipeline/gpu_her.rs: random-donor RNG (0x4E4_5678).
- cuda_pipeline/gpu_ppo_collector.rs: rng_seeds Vec for PPO
experience-collector init + reset (0xAA0_5EED).
- trainer/training_loop.rs: per-epoch regime_dropout_seed.
3. infra/k8s/argo/train-multi-seed-template.yaml — drop `fold` parameter
from `train-single` template; binary invoked as `--seed "$SEED"
--max-folds {{workflow.parameters.folds}}` so the walk-forward sweep
happens inside the single training process. Drop `FOLD` env var. Update
the nsys-rep upload filename to drop the fold suffix. Update banners /
doc comments to reflect "one-job-per-seed" semantics.
4. scripts/argo-train.sh — matrix generator drops the inner fold loop.
Each emitted task carries only `seed=${s}` and depends on the same
ensure-fxcache + gpu-warmup. The dry-run synthetic marker switches from
`seed=${s} fold=${f}` to `seed=${s} max_folds=${FOLDS}` so test harnesses
count the new shape correctly.
5. scripts/tests/test_multi_seed_harness.sh — assertions updated:
- `--multi-seed 3 --folds 2` produces 3 tasks (was 6).
- Rendered binary command must include `--max-folds
{{workflow.parameters.folds}}` placeholder.
- Rendered template must declare `folds` workflow parameter (so
`argo submit -p folds=K` overrides the default).
- Rendered binary command must NOT contain any per-fold flag — this
catches the failure mode that broke the first L40S deploy.
- Backward-compat: `--multi-seed 1 --folds 1` preserves the existing
single-template path (no DAG matrix tasks emitted).
6. docs/dqn-wire-up-audit.md — adds 1 Wired row documenting the pivot,
the new `--seed`/`mix_seed` plumbing, all 6 RNG call sites, and the
end-to-end seed-variation verification result.
Validation:
cargo check --workspace clean at 11 warnings (workspace baseline preserved).
cargo build --release --example train_baseline_rl succeeds; --help shows
the new --seed flag with documented default 42.
Seed-variation end-to-end test on RTX 3050 Ti (1 fold × 2 epochs each):
--seed 42 → F0 best Sharpe = -9.7831, best_val_metric = 1.957244,
epoch-2 train Sharpe = -16.12, val_Sharpe = +1.11.
--seed 999 → F0 best Sharpe = +92.9341, best_val_metric = 2.161012,
epoch-2 train Sharpe = +92.93, val_Sharpe = -0.25.
Different best Sharpe / best_val_metric / epoch-2 train + val Sharpe
across seeds proves the seed actually propagates through the RNG init
paths and is not just accepted-and-ignored. The seed=42 numbers match
the prompt's "deterministic baseline" expectation (F0 = -9.7831 was
bit-identical pre-pivot because no global-seed plumbing existed).
./scripts/argo-train.sh dqn --multi-seed 5 --folds 6 --dry-run produces
exactly 5 WorkflowTask markers (train-s0..train-s4), each with
`--max-folds {{workflow.parameters.folds}}` in the binary invocation.
All 3 harness tests PASS:
- test_multi_seed_harness.sh: 5 PASS lines, exit 0.
- test_nsys_harness.sh: 4 PASS lines + ALL PASS, exit 0.
- test_tier_checks.sh: PASS overall (good-fixture passes, bad-fixture
surfaces expected check rejections), exit 0.
Backward compat: existing single-job `argo-train.sh` callers (no
`--multi-seed`, no `--folds`) route to the original `train-template.yaml`
unchanged. `--seed 42` is a no-op offset for the SplitMix64 mix at the call
sites — the trajectory shifts only when the user passes `--seed` explicitly,
matching the prompt's "default 42 (historic implicit value)" requirement.
L40S pool: argo-train.sh defaults `--gpu-pool ci-training-h100`; user passes
`--gpu-pool ci-training-l40s` at deploy time. No script default change
(per constraint 5).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- argo-train.sh: --profile flag forces multi-seed render path so the
nsys wrapper + foxhunt-training-artifacts upload step are visible in
--dry-run YAML without cluster contact (test surface).
- train-multi-seed-template.yaml: new `profile` parameter (default
"false") gates the per-(seed, fold) `nsys profile
--capture-range=cudaProfilerApi` wrapper and the `mc cp` upload to
foxhunt-training-artifacts/profiles/<sha>/. mc binary fetched
on-demand (ci-builder image lacks it). MinIO creds optional —
upload warn-skips if absent.
- Dockerfile.foxhunt-training-runtime: install nsight-systems-cli
unpinned (pinning the stale 2024.4.1.61-1 from earlier plans
breaks builds when apt index advances).
- minio.yaml: add foxhunt-training-artifacts bucket to minio-init.
- compare-nsys-profiles.py: V0 regression detector — compares
cuda_gpu_kern_sum total_ns / epoch_count between two profiles;
exits 1 on >20% slowdown. NVTX per-epoch ranges deferred to T5.
- tests/test_nsys_harness.sh: dry-run grep test — verifies both
required strings appear when --profile is set, and that the
default (no --profile) path keeps profile=false in the rendered
template.
- dqn-wire-up-audit.md: Plan 5 Task 3 row added documenting the
harness + the baseline-capture deferral to T5.
Backward compat: test_multi_seed_harness.sh from P5T1 still PASS.
Adds the orchestration surface for Plan 5 Task 1 — Multi-Seed × Multi-Fold
Validation Harness:
- scripts/argo-train.sh: new --multi-seed N, --folds K, --tag T, --dry-run
flags. Default --multi-seed 1 --folds 1 routes to the existing
train-template.yaml (backward compat — existing callers unchanged). When
N>1 or K>1 the script renders train-multi-seed-template.yaml with an
inline-generated (seed, fold) matrix and either prints the YAML
(--dry-run) or applies + submits it.
- infra/k8s/argo/train-multi-seed-template.yaml: new WorkflowTemplate with
entrypoint multi-seed-matrix → ensure-binary, gpu-warmup, ensure-fxcache,
then N*K parallel train-single instances. Each train-single receives
seed/fold via inputs.parameters and forwards them to the training binary
via --seed/--fold CLI args + SEED/FOLD env vars. The dag.tasks placeholder
`# __MATRIX_TASKS__` is substituted programmatically by argo-train.sh
(awk) — no hand-written 30-task matrix.
- scripts/tests/test_multi_seed_harness.sh: dry-run regression test.
Asserts --multi-seed 3 --folds 2 emits 6 WorkflowTask markers AND
--multi-seed 1 --folds 1 emits zero (single-template path preserved).
Validation:
- argo lint --offline passes on both the source template and the rendered
3x2 / 5x6 outputs.
- test_multi_seed_harness.sh passes locally.
- Single-job dry-run still produces the unchanged train-template YAML.
Note: plan Step 0.1 pre-plan check expects ISV_TOTAL_DIM=72 and seven
ATTN_*_FOCUS_EMA_INDEX slots — both stale (Plan 4 landed
ISV_TOTAL_DIM=117 and the VSN_MAG_EMA / VSN_DIR_EMA / MAMBA2_RETENTION_EMA
slots instead). Plan 4 validation doc never landed (T8 deferred → Plan 5
T5). T1 is pure infrastructure that builds the harness consumed by T5,
so the stale pre-plan expectations do not block this commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The project's Cargo.toml and .cargo/config.toml both set
`incremental = true`, which makes rustc emit per-query save-analysis
artifacts that sccache does not cache. Result: even though
RUSTC_WRAPPER=sccache was set, the rustc calls wrote incremental
state that subsequent builds saw as stale and recompiled anyway.
Setting CARGO_INCREMENTAL=0 in the ensure-binary env forces rustc
to emit pure object output that sccache hashes uniformly on
(source + args). Same SHA → full cache hit; small source change →
only dirty crates recompile.
Does not change the service compile-and-deploy-template path,
which intentionally uses cargo-incremental against a persistent
/cargo-target-cpu PVC for repeated rebuilds of a narrow set of
service binaries.
Applies from the next workflow submission onward (train-5wb4n
already compiled under the old config and is running).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Every new commit triggered a full workspace recompile (~6min for a 2-file
change) in ensure-binary. The binary cache at /data/bin/$SHA is SHA-keyed
so it misses on every new commit. The cargo incremental cache at
/cargo-target/target is file-mtime keyed, which git checkout invalidates
whenever it retouches files.
sccache (already present in ci-builder image, verified via `sccache --version`
in Dockerfile) is content-hash keyed — identical source → identical hit
regardless of mtime. Sits in front of rustc via RUSTC_WRAPPER. Cache dir
on the same cargo-target-cuda PVC (SCCACHE_DIR=/cargo-target/sccache) so:
- Combines with cargo's target/ cache on one volume, no separate PVC
- Disk-local hits, no network round-trip per rustc invocation
- Survives across pods and commits — the git-checkout mtime flip that
breaks cargo's fingerprints doesn't affect content-hash caching
40G cache size cap — the cargo-target-cuda PVC is large enough that
sccache won't evict useful entries under normal use.
Expected behaviour: first build after this commit is still a full rebuild
(sccache cache empty). Subsequent builds for the same CUDA_COMPUTE_CAP
should hit compiled objects at near-zero cost for crates whose source
hasn't changed.
Applied to cluster via `kubectl apply -n foxhunt -f` — WorkflowTemplate
is live before the next ./scripts/argo-train.sh invocation.
Closes task #34.
Two bugs caught by the L40S smoke (train-qhgj6) that couldn't surface on
local RTX-3050 single-fold runs:
1. PER dtoh inside CUDA Graph capture (Fold 1 crash)
Failure: CUDA_ERROR_STREAM_CAPTURE_INVALIDATED at per_prefix_scan on
Fold 1 re-capture. Chain: fused_training parent graph captures →
memcpy_dtoh + cuStreamSynchronize in gpu_replay_buffer::update_priorities_gpu
(health<0.8 diversity path) poisons the stream → subsequent per_sample
kernel on the same stream sees an invalidated capture context.
The prior comment claimed "runs once per epoch, DtoH cost acceptable"
— wrong, it runs every priority update when health<0.8 (common during
Fold handoff when health_cache is re-seeded low). Any dtoh inside
capture invalidates regardless of latency.
Proper fix (no shortcut):
* New kernel actions_sum_scale_reduce_u32 — single-block deterministic
tree reduction over sample_actions (u32) → writes (sum*1000)/n as i32
to a device-accessible slot. No atomics (consistent with the 1/N
determinism policy from commit c82386500).
* mean_action_scaled storage is pinned + device-mapped (cuMemAllocHost
+ cuMemHostGetDevicePointer — same pattern as rng_step_dev_ptr and
size_dev_ptr elsewhere in the file). Zero-copy between host and
device, graph-safe, no explicit free needed (process-exit cleanup,
matches existing pattern).
* pow_alpha_diverse_f32 now takes const int* mean_action_scaled_ptr
and does a plain global load — NOT __ldg. The read-only cache used
by __ldg is not guaranteed coherent with device-mapped host memory;
multi-trial smoke regression caught it (median q_gap collapsed
from 2.0 → 0.15 with __ldg, recovered to 2.8 with plain load).
Verified: multi-trial smoke 5/5 pass, median_q_gap=2.80 (beats 2.00
baseline), Best Sharpe peaks 19-30 per trial. No stream capture
invalidation.
2. evaluate step CLI drift in Argo template
evaluate_baseline's Args struct uses --models-dir and --output (single
file path). Template was passing --checkpoint-dir and --output-dir,
causing clap to reject the invocation. Fixed argument names + added
mkdir for the eval subdir + updated the comment to pin the source of
truth for future drift catches.
Both fixes are graph-capture-clean and match the "wire properly or delete"
discipline. No masking, no feature flags, no dead params.
Persist the current dev-mode shutdown state in YAML so `kustomize build
| kubectl apply` won't quietly restart paused services.
Scope: foxhunt application services only. Core infra (GitLab, Minio,
Argo, Postgres, Redis) unchanged.
Paused:
- api, web-dashboard (dashboard stack, not used for CLI dev)
- trading-service, trading-agent-service, broker-gateway, ib-gateway
(live trading — not running in dev)
- backtesting-service (Argo workflows run standalone without it)
- data-acquisition-service (training uses cached .fxcache, not live feed)
- ml-training-service (training runs via scripts/argo-train.sh)
Each manifest has a restore-command comment documenting the rollback:
`kubectl -n foxhunt scale deployment <name> --replicas=1`.
Per user directive refined from previous commit: pushes to main should
trigger docker image rebuilds (when infra/docker/ changed), nothing else.
All other workflows run via scripts/manual invocation.
Sensor: restored body.ref filter to refs/heads/main (was set to a
never-match sentinel in the previous blanket-disable commit). Comment
updated to document the per-task policy.
DAG: set `when: "false"` on four non-docker tasks:
- build-web-dashboard (was: needs-dashboard)
- test-gate (was: needs-code — the CPU-heavy clippy+test gate)
- apply-argo-templates (was: needs-argo-templates)
- terragrunt-apply (was: needs-infra)
Kept enabled (still gated on detect-changes.docker-images == true):
- rebuild-ci-builder
- rebuild-ci-builder-cpu
- rebuild-runtime
- rebuild-training-runtime
gpu-test was already `when: "false"`; compile-and-deploy lives in a
separate template (manual webhook only, auto-compile-config=false).
Deploy: `kubectl apply -f infra/k8s/argo/events/ci-pipeline-sensor.yaml
infra/k8s/argo/ci-pipeline-template.yaml` (or re-apply via the existing
deploy path if one covers argo/).
Per user directive: pushes to main must not trigger any Argo compute
(not even the CPU-heavy test-gate). Changed the sensor's body.ref
filter value from "refs/heads/main" to a never-matching sentinel.
The eventsource still receives GitLab push webhooks; the sensor
evaluates them and rejects every one.
To re-enable: restore the value to "refs/heads/main" and
`kubectl apply -f infra/k8s/argo/events/ci-pipeline-sensor.yaml`.
Manual trigger paths (unchanged):
- ci-pipeline: argo submit --from=wftmpl/ci-pipeline -p commit-sha=<SHA>
- compile-and-deploy: curl -X POST http://workflow-trigger-eventsource-svc.foxhunt:12001/compile-deploy \
-H 'Content-Type: application/json' -d '{"commit_sha": "<SHA>"}'
Source: docs.nvidia.com/nsight-systems/InstallationGuide
Repo: developer.download.nvidia.com/devtools/repos/ubuntu2404/amd64/
Key: 7fa2af80.pub from CUDA repo
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Output: /data/nsys/ on training-data PVC (survives pod GC)
--show-output=true prints summary to pod logs for mid-run visibility
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Enables nsys profiling for CUDA graph node-level traces on H100.
Use --sanitizer nsys in argo-train.sh to activate.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use --sanitizer nsys to wrap training binary with nsys profile.
Captures CUDA graph node-level traces + GPU metrics.
Output: /workspace/output/nsys_profile.nsys-rep
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pod logs are now archived to MinIO before pod GC. Historical logs
accessible via `argo logs <workflow>` after pod completion.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CLI arg was removed in cost-driven hold timing but Argo template
still passed it, causing exit code 2 on H100 deploy.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scaleway Kapsule GPU nodes need NVIDIA driver DKMS compile on cold
start (~15 min). Setting min_size=1 keeps the node warm between runs.
Scale to 0 manually when done training for the day.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
FXCACHE_VERSION in header handles cache invalidation. First attempt
runs precompute — if cache is valid, skips (fast). If version
mismatch, deletes stale cache and regenerates. No more rm on every run.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Version constant in fxcache.rs. validate() rejects stale versions
with clear error message. ensure-fxcache Argo template now deletes
old cache and regenerates unconditionally. No manual PVC cleanup
needed — version mismatch triggers automatic regeneration.
v2: OFI_DIM=20 (20 microstructure features).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
