Mount the existing sccache-cpu PVC and set RUSTC_WRAPPER=sccache for
the regular service-build pipeline. Previously only train-template and
train-multi-seed-template used sccache; the production CI build had
zero rustc-level caching, so every CI run rebuilt every dep from
scratch.
Also set CARGO_INCREMENTAL=0 to override the workspace-wide
[build] incremental = true in .cargo/config.toml. sccache documented
limitation: cannot cache incremental rustc output. Local dev keeps
incremental via config.toml unchanged.
Print sccache --show-stats at end of build for visibility into cache
hit rates.
Mirror of train-template.yaml's sccache pattern; uses the sccache-cpu
PVC (20Gi, scw-bssd-retain) which was already provisioned but unmounted.
Expected impact: rustc cache hit rate jumps from 0% → 50-90% on
subsequent CI runs.
The L40S smoke template uses CARGO_TARGET_DIR=/cargo-target on a
persistent PVC, so every smoke probe builds incrementally on top of
artefacts left by prior probes. File deletions (e.g., regime_conditional.rs
in ff00af68a) can leave dangling rmeta/object references that perturb
downstream codegen between bisect runs — making the bisect result
contingent on which order probes were submitted in, not on the source.
Adds an optional `clean-cache` parameter (default `false`) which runs
`cargo clean -p ml -p ml-dqn --release` before the compile step. Other
crate artefacts (ml-core, ml-supervised, etc.) stay cached so the wipe
is bounded — fresh ml/ml-dqn compile in ~3-5 min vs ~30+ min full clean.
Use case: re-running a52d99613 + ff00af68a with `--clean-cache` to
verify the bisect under controlled build conditions. If the
broken/clean status flips with clean cache, the regression is
build-state-dependent rather than source-line; if it reproduces, the
source regression is real and bisect is definitive.
scripts/argo-smoke.sh exposes `--clean-cache` flag passing through
to the workflow parameter.
Adds a no-wrapper L40S smoke runner alongside the nsys/sanitizer
templates. Same compile + checkout + data-mount layout, no profiler
binary, no sanitizer instrumentation, no MinIO artefact upload — just
a plain test execution against the full training-data PVC.
Use case: validate a smoke test passes on the real 27-month dataset
when local data is too short (laptop only has 1 quarter of MBP-10/
trades, fxcache truncates below the 10-month minimum).
- infra/k8s/argo/smoke-test-template.yaml: WorkflowTemplate
smoke-test, entrypoint smoke-run, default test
multi_fold_convergence::test_multi_fold_convergence.
- infra/k8s/argo/kustomization.yaml: register the new template.
- infra/k8s/argo/argo-workflow-netpol.yaml: extend the
sanitizer/nsys NetworkPolicy podSelector to include smoke-test
(identical egress requirements: git fetch, ci-builder pull,
training-data PVC).
- scripts/argo-smoke.sh: thin wrapper mirroring argo-nsys.sh /
argo-sanitizer.sh (--multi-fold default, --test, --ref, --watch).
Verified: kustomize dry-run clean, kubectl apply -k creates the
template + reconfigures the netpol live in foxhunt namespace.
The original target 'foxhunt-training-artifacts' (cloned from
train-multi-seed-template) does not exist on the cluster MinIO.
Available production buckets: foxhunt-models, foxhunt-training-data,
foxhunt-training-results, foxhunt-binaries, foxhunt-backups,
foxhunt-gitlab-{artifacts,packages,registry}.
foxhunt-training-results is the right home for profiling artefacts
(model evaluation outputs already land there). Upload path is now
foxhunt-training-results/profiles/smoke/<short-sha>/profile-<pod>.nsys-rep.
Note: train-multi-seed-template has the same bucket-name typo —
production --profile uploads have been silently failing. Out of scope
for this commit, separate fix needed there.
NVIDIA's GPU metrics counters require elevated container perms
(/proc/sys/kernel/perf_event_paranoid + nvidia.com/gpu performance
counter device unlock). Containers in our Kapsule pods don't have
those — nsys exits exit-1 immediately with NVGPUCTRPERM.
Default to CUDA/NVTX/osrt traces only — kernel timeline + per-kernel
duration + call-graph context, which is what the bottleneck hunts
actually need. Counters can be re-enabled via --extra-args when
running on a relaxed node.
The test-data-pvc only had 3-4 months of MBP-10 (Q1 2024 only), causing
walk-forward fold generation to fail with 'Need at least 10 months of data'.
Production training uses training-data-pvc mounted at /data with subdirs
futures-baseline / futures-baseline-mbp10 / futures-baseline-trades —
that's the full 27-month dataset.
Switching all three env vars (FOXHUNT_TEST_DATA, FOXHUNT_MBP10_DATA,
FOXHUNT_TRADES_DATA) and the volume/mount to point at the production
data PVC. multi_fold_convergence's 6/2/2 walk-forward (10 months min)
now has ample data.
`grep -q` exits early on first match, closing stdin to upstream
`$TEST_BIN --list`. With pipefail enabled, that SIGPIPE on the upstream
process bubbles up as exit 141 and kills the workflow before the
sanitizer/nsys can launch.
Capture --list output once into a variable, then grep the captured
string (which doesn't pipe to a process). Same pattern for the
already-broken-but-defensive head -30 in the error branch.
Per feedback_mbp10_mandatory.md, MBP-10 + trades are mandatory inputs.
multi_fold_convergence now reads FOXHUNT_MBP10_DATA / FOXHUNT_TRADES_DATA
env vars, forwards them to train_baseline_rl as --mbp10-data-dir /
--trades-data-dir, and hard-fails if either path is missing.
Argo sanitizer-test + nsys-test templates set the env vars to PVC paths
/data/test-data/{mbp10,trades} and FOXHUNT_TEST_DATA points at
/data/test-data/ohlcv (PVC layout has lost+found/trades/mbp10/ohlcv as
top-level subdirs; the symbol ES.FUT lives under ohlcv/).
Without this, the smoke would silently fall back to tick-rule proxy
classification and validate a degraded model variant (no real OFI),
defeating the purpose of L40S validation. Audit doc updated per
Invariant 7.
`set -o pipefail` + `head -1` closing stdin produces SIGPIPE (exit 141)
from upstream `ls`/`grep`. The subshell + || true contains the failure
to the inner pipeline so the script's pipefail doesn't propagate it.
Sanitizer fired exit 141 immediately after the cargo build; nsys
reproduced the same exit on its parallel run, confirming the diagnosis.
The workflow sets CARGO_TARGET_DIR=/cargo-target so cargo writes binaries
to /cargo-target/release/deps/ml-*, not target/release/deps/ml-*. Both
templates were globbing the wrong path and failed with exit 3 (now
caught by the explicit error message).
Compile finished cleanly in 4m14s (sccache cache hit on first run after
re-checkout); test binary lookup then failed and the script exited at
the 'could not locate compiled test binary' check.
Both new workflow components were missing egress permissions and the
default-deny in the foxhunt namespace blocked their git fetch SSH to
gitlab-shell.foxhunt.svc.cluster.local:2222 — caused exit code 128
"Could not read from remote repository" before any GPU work started.
New `argo-sanitizer-nsys-workflow` policy mirrors the ci-pipeline
egress (DNS, K8s API, gitlab SSH on port 2222, MinIO 9000 for log
archive + nsys-rep upload, gitlab registry 5000 for ci-builder image
pull, webservice 8181 for registry token auth, external 80/443 for
mc binary download from dl.min.io).
train-template.yaml had a stray `value: "5"` line directly after
`value: "1.0"` under `- name: spread-ticks`. Stricter YAML parsers
(Go go-yaml used by kustomize) reject duplicate mapping keys and bail
out of the entire kustomize bundle with line-52 errors that mislead
diagnosis to unrelated files.
Permissive parsers (kubectl apply -f's path) silently take the last
value, which made the bug invisible until `kubectl apply -k` was
attempted.
The orphan was a leftover from an earlier edit (likely a max-folds=5
or similar parameter that was renamed but not fully removed).
The unquoted test name string contains '::' (Rust path separator). Some
strict YAML implementations (Go go-yaml when used by certain kustomize
paths) interpret it ambiguously. Quoting makes it always a string scalar.
Templates were applied directly via 'kubectl apply -f' which uses a
permissive parser; the unquoted form worked. Quoting makes them
also kustomize-compatible.
Two new one-shot WorkflowTemplates for validating smoke tests under
GPU-instrumentation tools that don't fit in the laptop's 4 GB VRAM:
- `sanitizer-test`: wraps cargo test smoke under `compute-sanitizer
--tool=memcheck|racecheck|synccheck|initcheck` with --target-processes all
so multi_fold_convergence's spawned `train_baseline_rl` subprocess is also
instrumented. Pre-builds train_baseline_rl example to avoid sanitizer
instrumenting cargo/rustc on the inner spawn. Triages internal-vs-real
errors and exits non-zero only on real bugs.
- `nsys-test`: wraps cargo test smoke under `nsys profile`. Captures CUDA +
NVTX + osrt traces with GPU metrics (ga10x set). Uploads .nsys-rep to
MinIO at foxhunt-training-artifacts/profiles/smoke/<short-sha>/, mirroring
the existing Plan 5 Task 3 production-training profile pattern in
train-multi-seed-template.
Wrapper scripts:
- scripts/argo-sanitizer.sh — `argo submit` wrapper, supports --multi-fold
shortcut for fold-boundary code paths (IQN sync, aux Adam reset,
iqn_readiness reset, MSE clamp) that single-fold tests cannot exercise.
- scripts/argo-nsys.sh — same shape as argo-sanitizer.sh, default test is
performance::test_real_data_single_epoch for broad coverage.
Why L40S: laptop RTX 3050 Ti's 4 GB cannot fit compute-sanitizer's
instrumentation metadata (~2-3x app VRAM) — sanitizer falls back to "didn't
track the launch" with 60k+ internal-allocation errors. L40S 48 GB has
ample headroom for both memcheck and nsys overhead.
Both templates compile cargo test --release --lib --no-run plus cargo build
--release --example train_baseline_rl on the cargo-target PVC. Compile time
dominated by sccache hit rate (production training image: 100% C/C++ cache,
~75% Rust cache after warmup).
Templates registered in kustomization.yaml — apply with `kubectl apply -k
infra/k8s/argo` before first submit.
The first L40S deploy attempt (workflow `train-multi-seed-z2llf`, terminated)
failed at startup with `error: unexpected argument '--fold' found` on every
job: `train_baseline_rl` is a multi-fold walk-forward executor that accepts
`--max-folds K`, NOT `--fold N`. The original P5T1 harness assumed the
opposite and fanned out N seeds × K folds = N*K jobs, each invoking the
binary with `--seed N --fold K`.
User chose Path B: pivot to one job per seed (each runs all K folds via the
existing `--max-folds` mechanism). Per-job runtime is K× longer, but fanout
drops from N*K=30 → N=5 (matches L40S pool capacity better) and the binary
contract becomes the one the binary actually has.
4 surface changes:
1. crates/ml/examples/train_baseline_rl.rs — add `--seed N` CLI arg
(default 42 — historic implicit value). Sets `FOXHUNT_SEED` env var at
startup BEFORE any CUDA module spins up. Logs the seed value at the
training start banner.
2. crates/ml/src/cuda_pipeline/mod.rs — add `global_seed()` (reads
`FOXHUNT_SEED`, default 42) + `mix_seed(base)` (SplitMix64 avalanche
so adjacent global seeds produce uncorrelated module seeds). Six call
sites updated to mix the global seed into their previously-hardcoded
constants:
- trainer/action.rs: GpuActionSelector seed (0xDEAD_BEEF_CAFE) + the
epsilon-greedy fallback StdRng (0xAC7_DEF0).
- cuda_pipeline/gpu_iqn_head.rs: IQN Xavier-init RNG (0x1CA_1234).
- cuda_pipeline/gpu_iql_trainer.rs: V(s) Xavier-init RNG (0x1C1_9ABC).
- cuda_pipeline/gpu_her.rs: random-donor RNG (0x4E4_5678).
- cuda_pipeline/gpu_ppo_collector.rs: rng_seeds Vec for PPO
experience-collector init + reset (0xAA0_5EED).
- trainer/training_loop.rs: per-epoch regime_dropout_seed.
3. infra/k8s/argo/train-multi-seed-template.yaml — drop `fold` parameter
from `train-single` template; binary invoked as `--seed "$SEED"
--max-folds {{workflow.parameters.folds}}` so the walk-forward sweep
happens inside the single training process. Drop `FOLD` env var. Update
the nsys-rep upload filename to drop the fold suffix. Update banners /
doc comments to reflect "one-job-per-seed" semantics.
4. scripts/argo-train.sh — matrix generator drops the inner fold loop.
Each emitted task carries only `seed=${s}` and depends on the same
ensure-fxcache + gpu-warmup. The dry-run synthetic marker switches from
`seed=${s} fold=${f}` to `seed=${s} max_folds=${FOLDS}` so test harnesses
count the new shape correctly.
5. scripts/tests/test_multi_seed_harness.sh — assertions updated:
- `--multi-seed 3 --folds 2` produces 3 tasks (was 6).
- Rendered binary command must include `--max-folds
{{workflow.parameters.folds}}` placeholder.
- Rendered template must declare `folds` workflow parameter (so
`argo submit -p folds=K` overrides the default).
- Rendered binary command must NOT contain any per-fold flag — this
catches the failure mode that broke the first L40S deploy.
- Backward-compat: `--multi-seed 1 --folds 1` preserves the existing
single-template path (no DAG matrix tasks emitted).
6. docs/dqn-wire-up-audit.md — adds 1 Wired row documenting the pivot,
the new `--seed`/`mix_seed` plumbing, all 6 RNG call sites, and the
end-to-end seed-variation verification result.
Validation:
cargo check --workspace clean at 11 warnings (workspace baseline preserved).
cargo build --release --example train_baseline_rl succeeds; --help shows
the new --seed flag with documented default 42.
Seed-variation end-to-end test on RTX 3050 Ti (1 fold × 2 epochs each):
--seed 42 → F0 best Sharpe = -9.7831, best_val_metric = 1.957244,
epoch-2 train Sharpe = -16.12, val_Sharpe = +1.11.
--seed 999 → F0 best Sharpe = +92.9341, best_val_metric = 2.161012,
epoch-2 train Sharpe = +92.93, val_Sharpe = -0.25.
Different best Sharpe / best_val_metric / epoch-2 train + val Sharpe
across seeds proves the seed actually propagates through the RNG init
paths and is not just accepted-and-ignored. The seed=42 numbers match
the prompt's "deterministic baseline" expectation (F0 = -9.7831 was
bit-identical pre-pivot because no global-seed plumbing existed).
./scripts/argo-train.sh dqn --multi-seed 5 --folds 6 --dry-run produces
exactly 5 WorkflowTask markers (train-s0..train-s4), each with
`--max-folds {{workflow.parameters.folds}}` in the binary invocation.
All 3 harness tests PASS:
- test_multi_seed_harness.sh: 5 PASS lines, exit 0.
- test_nsys_harness.sh: 4 PASS lines + ALL PASS, exit 0.
- test_tier_checks.sh: PASS overall (good-fixture passes, bad-fixture
surfaces expected check rejections), exit 0.
Backward compat: existing single-job `argo-train.sh` callers (no
`--multi-seed`, no `--folds`) route to the original `train-template.yaml`
unchanged. `--seed 42` is a no-op offset for the SplitMix64 mix at the call
sites — the trajectory shifts only when the user passes `--seed` explicitly,
matching the prompt's "default 42 (historic implicit value)" requirement.
L40S pool: argo-train.sh defaults `--gpu-pool ci-training-h100`; user passes
`--gpu-pool ci-training-l40s` at deploy time. No script default change
(per constraint 5).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- argo-train.sh: --profile flag forces multi-seed render path so the
nsys wrapper + foxhunt-training-artifacts upload step are visible in
--dry-run YAML without cluster contact (test surface).
- train-multi-seed-template.yaml: new `profile` parameter (default
"false") gates the per-(seed, fold) `nsys profile
--capture-range=cudaProfilerApi` wrapper and the `mc cp` upload to
foxhunt-training-artifacts/profiles/<sha>/. mc binary fetched
on-demand (ci-builder image lacks it). MinIO creds optional —
upload warn-skips if absent.
- Dockerfile.foxhunt-training-runtime: install nsight-systems-cli
unpinned (pinning the stale 2024.4.1.61-1 from earlier plans
breaks builds when apt index advances).
- minio.yaml: add foxhunt-training-artifacts bucket to minio-init.
- compare-nsys-profiles.py: V0 regression detector — compares
cuda_gpu_kern_sum total_ns / epoch_count between two profiles;
exits 1 on >20% slowdown. NVTX per-epoch ranges deferred to T5.
- tests/test_nsys_harness.sh: dry-run grep test — verifies both
required strings appear when --profile is set, and that the
default (no --profile) path keeps profile=false in the rendered
template.
- dqn-wire-up-audit.md: Plan 5 Task 3 row added documenting the
harness + the baseline-capture deferral to T5.
Backward compat: test_multi_seed_harness.sh from P5T1 still PASS.
Adds the orchestration surface for Plan 5 Task 1 — Multi-Seed × Multi-Fold
Validation Harness:
- scripts/argo-train.sh: new --multi-seed N, --folds K, --tag T, --dry-run
flags. Default --multi-seed 1 --folds 1 routes to the existing
train-template.yaml (backward compat — existing callers unchanged). When
N>1 or K>1 the script renders train-multi-seed-template.yaml with an
inline-generated (seed, fold) matrix and either prints the YAML
(--dry-run) or applies + submits it.
- infra/k8s/argo/train-multi-seed-template.yaml: new WorkflowTemplate with
entrypoint multi-seed-matrix → ensure-binary, gpu-warmup, ensure-fxcache,
then N*K parallel train-single instances. Each train-single receives
seed/fold via inputs.parameters and forwards them to the training binary
via --seed/--fold CLI args + SEED/FOLD env vars. The dag.tasks placeholder
`# __MATRIX_TASKS__` is substituted programmatically by argo-train.sh
(awk) — no hand-written 30-task matrix.
- scripts/tests/test_multi_seed_harness.sh: dry-run regression test.
Asserts --multi-seed 3 --folds 2 emits 6 WorkflowTask markers AND
--multi-seed 1 --folds 1 emits zero (single-template path preserved).
Validation:
- argo lint --offline passes on both the source template and the rendered
3x2 / 5x6 outputs.
- test_multi_seed_harness.sh passes locally.
- Single-job dry-run still produces the unchanged train-template YAML.
Note: plan Step 0.1 pre-plan check expects ISV_TOTAL_DIM=72 and seven
ATTN_*_FOCUS_EMA_INDEX slots — both stale (Plan 4 landed
ISV_TOTAL_DIM=117 and the VSN_MAG_EMA / VSN_DIR_EMA / MAMBA2_RETENTION_EMA
slots instead). Plan 4 validation doc never landed (T8 deferred → Plan 5
T5). T1 is pure infrastructure that builds the harness consumed by T5,
so the stale pre-plan expectations do not block this commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The project's Cargo.toml and .cargo/config.toml both set
`incremental = true`, which makes rustc emit per-query save-analysis
artifacts that sccache does not cache. Result: even though
RUSTC_WRAPPER=sccache was set, the rustc calls wrote incremental
state that subsequent builds saw as stale and recompiled anyway.
Setting CARGO_INCREMENTAL=0 in the ensure-binary env forces rustc
to emit pure object output that sccache hashes uniformly on
(source + args). Same SHA → full cache hit; small source change →
only dirty crates recompile.
Does not change the service compile-and-deploy-template path,
which intentionally uses cargo-incremental against a persistent
/cargo-target-cpu PVC for repeated rebuilds of a narrow set of
service binaries.
Applies from the next workflow submission onward (train-5wb4n
already compiled under the old config and is running).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Every new commit triggered a full workspace recompile (~6min for a 2-file
change) in ensure-binary. The binary cache at /data/bin/$SHA is SHA-keyed
so it misses on every new commit. The cargo incremental cache at
/cargo-target/target is file-mtime keyed, which git checkout invalidates
whenever it retouches files.
sccache (already present in ci-builder image, verified via `sccache --version`
in Dockerfile) is content-hash keyed — identical source → identical hit
regardless of mtime. Sits in front of rustc via RUSTC_WRAPPER. Cache dir
on the same cargo-target-cuda PVC (SCCACHE_DIR=/cargo-target/sccache) so:
- Combines with cargo's target/ cache on one volume, no separate PVC
- Disk-local hits, no network round-trip per rustc invocation
- Survives across pods and commits — the git-checkout mtime flip that
breaks cargo's fingerprints doesn't affect content-hash caching
40G cache size cap — the cargo-target-cuda PVC is large enough that
sccache won't evict useful entries under normal use.
Expected behaviour: first build after this commit is still a full rebuild
(sccache cache empty). Subsequent builds for the same CUDA_COMPUTE_CAP
should hit compiled objects at near-zero cost for crates whose source
hasn't changed.
Applied to cluster via `kubectl apply -n foxhunt -f` — WorkflowTemplate
is live before the next ./scripts/argo-train.sh invocation.
Closes task #34.
Two bugs caught by the L40S smoke (train-qhgj6) that couldn't surface on
local RTX-3050 single-fold runs:
1. PER dtoh inside CUDA Graph capture (Fold 1 crash)
Failure: CUDA_ERROR_STREAM_CAPTURE_INVALIDATED at per_prefix_scan on
Fold 1 re-capture. Chain: fused_training parent graph captures →
memcpy_dtoh + cuStreamSynchronize in gpu_replay_buffer::update_priorities_gpu
(health<0.8 diversity path) poisons the stream → subsequent per_sample
kernel on the same stream sees an invalidated capture context.
The prior comment claimed "runs once per epoch, DtoH cost acceptable"
— wrong, it runs every priority update when health<0.8 (common during
Fold handoff when health_cache is re-seeded low). Any dtoh inside
capture invalidates regardless of latency.
Proper fix (no shortcut):
* New kernel actions_sum_scale_reduce_u32 — single-block deterministic
tree reduction over sample_actions (u32) → writes (sum*1000)/n as i32
to a device-accessible slot. No atomics (consistent with the 1/N
determinism policy from commit c82386500).
* mean_action_scaled storage is pinned + device-mapped (cuMemAllocHost
+ cuMemHostGetDevicePointer — same pattern as rng_step_dev_ptr and
size_dev_ptr elsewhere in the file). Zero-copy between host and
device, graph-safe, no explicit free needed (process-exit cleanup,
matches existing pattern).
* pow_alpha_diverse_f32 now takes const int* mean_action_scaled_ptr
and does a plain global load — NOT __ldg. The read-only cache used
by __ldg is not guaranteed coherent with device-mapped host memory;
multi-trial smoke regression caught it (median q_gap collapsed
from 2.0 → 0.15 with __ldg, recovered to 2.8 with plain load).
Verified: multi-trial smoke 5/5 pass, median_q_gap=2.80 (beats 2.00
baseline), Best Sharpe peaks 19-30 per trial. No stream capture
invalidation.
2. evaluate step CLI drift in Argo template
evaluate_baseline's Args struct uses --models-dir and --output (single
file path). Template was passing --checkpoint-dir and --output-dir,
causing clap to reject the invocation. Fixed argument names + added
mkdir for the eval subdir + updated the comment to pin the source of
truth for future drift catches.
Both fixes are graph-capture-clean and match the "wire properly or delete"
discipline. No masking, no feature flags, no dead params.
Persist the current dev-mode shutdown state in YAML so `kustomize build
| kubectl apply` won't quietly restart paused services.
Scope: foxhunt application services only. Core infra (GitLab, Minio,
Argo, Postgres, Redis) unchanged.
Paused:
- api, web-dashboard (dashboard stack, not used for CLI dev)
- trading-service, trading-agent-service, broker-gateway, ib-gateway
(live trading — not running in dev)
- backtesting-service (Argo workflows run standalone without it)
- data-acquisition-service (training uses cached .fxcache, not live feed)
- ml-training-service (training runs via scripts/argo-train.sh)
Each manifest has a restore-command comment documenting the rollback:
`kubectl -n foxhunt scale deployment <name> --replicas=1`.
Per user directive refined from previous commit: pushes to main should
trigger docker image rebuilds (when infra/docker/ changed), nothing else.
All other workflows run via scripts/manual invocation.
Sensor: restored body.ref filter to refs/heads/main (was set to a
never-match sentinel in the previous blanket-disable commit). Comment
updated to document the per-task policy.
DAG: set `when: "false"` on four non-docker tasks:
- build-web-dashboard (was: needs-dashboard)
- test-gate (was: needs-code — the CPU-heavy clippy+test gate)
- apply-argo-templates (was: needs-argo-templates)
- terragrunt-apply (was: needs-infra)
Kept enabled (still gated on detect-changes.docker-images == true):
- rebuild-ci-builder
- rebuild-ci-builder-cpu
- rebuild-runtime
- rebuild-training-runtime
gpu-test was already `when: "false"`; compile-and-deploy lives in a
separate template (manual webhook only, auto-compile-config=false).
Deploy: `kubectl apply -f infra/k8s/argo/events/ci-pipeline-sensor.yaml
infra/k8s/argo/ci-pipeline-template.yaml` (or re-apply via the existing
deploy path if one covers argo/).
Per user directive: pushes to main must not trigger any Argo compute
(not even the CPU-heavy test-gate). Changed the sensor's body.ref
filter value from "refs/heads/main" to a never-matching sentinel.
The eventsource still receives GitLab push webhooks; the sensor
evaluates them and rejects every one.
To re-enable: restore the value to "refs/heads/main" and
`kubectl apply -f infra/k8s/argo/events/ci-pipeline-sensor.yaml`.
Manual trigger paths (unchanged):
- ci-pipeline: argo submit --from=wftmpl/ci-pipeline -p commit-sha=<SHA>
- compile-and-deploy: curl -X POST http://workflow-trigger-eventsource-svc.foxhunt:12001/compile-deploy \
-H 'Content-Type: application/json' -d '{"commit_sha": "<SHA>"}'
Source: docs.nvidia.com/nsight-systems/InstallationGuide
Repo: developer.download.nvidia.com/devtools/repos/ubuntu2404/amd64/
Key: 7fa2af80.pub from CUDA repo
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Output: /data/nsys/ on training-data PVC (survives pod GC)
--show-output=true prints summary to pod logs for mid-run visibility
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Enables nsys profiling for CUDA graph node-level traces on H100.
Use --sanitizer nsys in argo-train.sh to activate.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use --sanitizer nsys to wrap training binary with nsys profile.
Captures CUDA graph node-level traces + GPU metrics.
Output: /workspace/output/nsys_profile.nsys-rep
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pod logs are now archived to MinIO before pod GC. Historical logs
accessible via `argo logs <workflow>` after pod completion.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CLI arg was removed in cost-driven hold timing but Argo template
still passed it, causing exit code 2 on H100 deploy.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scaleway Kapsule GPU nodes need NVIDIA driver DKMS compile on cold
start (~15 min). Setting min_size=1 keeps the node warm between runs.
Scale to 0 manually when done training for the day.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
FXCACHE_VERSION in header handles cache invalidation. First attempt
runs precompute — if cache is valid, skips (fast). If version
mismatch, deletes stale cache and regenerates. No more rm on every run.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Version constant in fxcache.rs. validate() rejects stale versions
with clear error message. ensure-fxcache Argo template now deletes
old cache and regenerates unconditionally. No manual PVC cleanup
needed — version mismatch triggers automatic regeneration.
v2: OFI_DIM=20 (20 microstructure features).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
AlgoGetIds picks algorithms by ID order. On H100 (SM90), the first
valid algo uses split-K/stream-K with internal workspace allocation
that silently produces zero output when replayed via CUDA Graph.
The heuristic selects graph-safe algorithms by design.
Also removes FOXHUNT_NO_GRAPH temp env var and raw DtoH diagnostic probe.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Temporarily sets FOXHUNT_NO_GRAPH=1 in train-best pods to run
forward+backward+Adam ungraphed. If grad_buf is non-zero ungraphed,
the root cause is CUDA Graph capture of cuBLASLt on H100 SM90.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause of DNS deadlock (2026-03-18, 2026-04-12): node-bootstrap
DaemonSet overwrote /etc/resolv.conf with ONLY nameserver 10.32.0.10
(cluster DNS). When nodes rebuild, CoreDNS can't pull its image because
containerd resolves via 10.32.0.10 which requires CoreDNS to be running.
Fix: resolv.conf now has DUAL nameservers:
nameserver 10.32.0.10 # primary: cluster DNS
nameserver 169.254.169.254 # fallback: Scaleway metadata DNS
Containerd tries kube-dns first (fast, resolves .svc.cluster.local),
falls back to VPC resolver when kube-dns is unreachable. Breaks the
chicken-and-egg permanently.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: weight tensors packed sequentially in the flat params buffer
had non-aligned start offsets when preceding tensors had odd element counts
(e.g. bias of 51 atoms = 204 bytes, 204 % 16 = 12). cublasLtMatmul with
CUBLAS_COMPUTE_32F_FAST_TF32 requires 16-byte aligned buffer pointers.
Fix: pad each tensor to 4-element boundary (16 bytes) in both
f32_weight_ptrs_from_base (pointer computation) and compute_total_params
(buffer allocation). Added align4() and padded_byte_offset() helpers,
fixed shrink_perturb skip range and bottleneck gradient offset.
Switched compute type: CUBLAS_COMPUTE_32F → CUBLAS_COMPUTE_32F_FAST_TF32
(forward + backward). Explicit TF32 tensor core path, required by cuBLAS
13.0 on H100 SM90.
Deleted dead bf16_weight_ptrs function.
19/19 smoke tests pass on RTX 3050.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Argo requires task output parameters to flow through DAG arguments →
template inputs, not direct {{tasks.X.outputs}} in container args.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
