Every new commit triggered a full workspace recompile (~6min for a 2-file
change) in ensure-binary. The binary cache at /data/bin/$SHA is SHA-keyed
so it misses on every new commit. The cargo incremental cache at
/cargo-target/target is file-mtime keyed, which git checkout invalidates
whenever it retouches files.
sccache (already present in ci-builder image, verified via `sccache --version`
in Dockerfile) is content-hash keyed — identical source → identical hit
regardless of mtime. Sits in front of rustc via RUSTC_WRAPPER. Cache dir
on the same cargo-target-cuda PVC (SCCACHE_DIR=/cargo-target/sccache) so:
- Combines with cargo's target/ cache on one volume, no separate PVC
- Disk-local hits, no network round-trip per rustc invocation
- Survives across pods and commits — the git-checkout mtime flip that
breaks cargo's fingerprints doesn't affect content-hash caching
40G cache size cap — the cargo-target-cuda PVC is large enough that
sccache won't evict useful entries under normal use.
Expected behaviour: first build after this commit is still a full rebuild
(sccache cache empty). Subsequent builds for the same CUDA_COMPUTE_CAP
should hit compiled objects at near-zero cost for crates whose source
hasn't changed.
Applied to cluster via `kubectl apply -n foxhunt -f` — WorkflowTemplate
is live before the next ./scripts/argo-train.sh invocation.
Closes task #34.
Two bugs caught by the L40S smoke (train-qhgj6) that couldn't surface on
local RTX-3050 single-fold runs:
1. PER dtoh inside CUDA Graph capture (Fold 1 crash)
Failure: CUDA_ERROR_STREAM_CAPTURE_INVALIDATED at per_prefix_scan on
Fold 1 re-capture. Chain: fused_training parent graph captures →
memcpy_dtoh + cuStreamSynchronize in gpu_replay_buffer::update_priorities_gpu
(health<0.8 diversity path) poisons the stream → subsequent per_sample
kernel on the same stream sees an invalidated capture context.
The prior comment claimed "runs once per epoch, DtoH cost acceptable"
— wrong, it runs every priority update when health<0.8 (common during
Fold handoff when health_cache is re-seeded low). Any dtoh inside
capture invalidates regardless of latency.
Proper fix (no shortcut):
* New kernel actions_sum_scale_reduce_u32 — single-block deterministic
tree reduction over sample_actions (u32) → writes (sum*1000)/n as i32
to a device-accessible slot. No atomics (consistent with the 1/N
determinism policy from commit c82386500).
* mean_action_scaled storage is pinned + device-mapped (cuMemAllocHost
+ cuMemHostGetDevicePointer — same pattern as rng_step_dev_ptr and
size_dev_ptr elsewhere in the file). Zero-copy between host and
device, graph-safe, no explicit free needed (process-exit cleanup,
matches existing pattern).
* pow_alpha_diverse_f32 now takes const int* mean_action_scaled_ptr
and does a plain global load — NOT __ldg. The read-only cache used
by __ldg is not guaranteed coherent with device-mapped host memory;
multi-trial smoke regression caught it (median q_gap collapsed
from 2.0 → 0.15 with __ldg, recovered to 2.8 with plain load).
Verified: multi-trial smoke 5/5 pass, median_q_gap=2.80 (beats 2.00
baseline), Best Sharpe peaks 19-30 per trial. No stream capture
invalidation.
2. evaluate step CLI drift in Argo template
evaluate_baseline's Args struct uses --models-dir and --output (single
file path). Template was passing --checkpoint-dir and --output-dir,
causing clap to reject the invocation. Fixed argument names + added
mkdir for the eval subdir + updated the comment to pin the source of
truth for future drift catches.
Both fixes are graph-capture-clean and match the "wire properly or delete"
discipline. No masking, no feature flags, no dead params.
Persist the current dev-mode shutdown state in YAML so `kustomize build
| kubectl apply` won't quietly restart paused services.
Scope: foxhunt application services only. Core infra (GitLab, Minio,
Argo, Postgres, Redis) unchanged.
Paused:
- api, web-dashboard (dashboard stack, not used for CLI dev)
- trading-service, trading-agent-service, broker-gateway, ib-gateway
(live trading — not running in dev)
- backtesting-service (Argo workflows run standalone without it)
- data-acquisition-service (training uses cached .fxcache, not live feed)
- ml-training-service (training runs via scripts/argo-train.sh)
Each manifest has a restore-command comment documenting the rollback:
`kubectl -n foxhunt scale deployment <name> --replicas=1`.
Per user directive refined from previous commit: pushes to main should
trigger docker image rebuilds (when infra/docker/ changed), nothing else.
All other workflows run via scripts/manual invocation.
Sensor: restored body.ref filter to refs/heads/main (was set to a
never-match sentinel in the previous blanket-disable commit). Comment
updated to document the per-task policy.
DAG: set `when: "false"` on four non-docker tasks:
- build-web-dashboard (was: needs-dashboard)
- test-gate (was: needs-code — the CPU-heavy clippy+test gate)
- apply-argo-templates (was: needs-argo-templates)
- terragrunt-apply (was: needs-infra)
Kept enabled (still gated on detect-changes.docker-images == true):
- rebuild-ci-builder
- rebuild-ci-builder-cpu
- rebuild-runtime
- rebuild-training-runtime
gpu-test was already `when: "false"`; compile-and-deploy lives in a
separate template (manual webhook only, auto-compile-config=false).
Deploy: `kubectl apply -f infra/k8s/argo/events/ci-pipeline-sensor.yaml
infra/k8s/argo/ci-pipeline-template.yaml` (or re-apply via the existing
deploy path if one covers argo/).
Per user directive: pushes to main must not trigger any Argo compute
(not even the CPU-heavy test-gate). Changed the sensor's body.ref
filter value from "refs/heads/main" to a never-matching sentinel.
The eventsource still receives GitLab push webhooks; the sensor
evaluates them and rejects every one.
To re-enable: restore the value to "refs/heads/main" and
`kubectl apply -f infra/k8s/argo/events/ci-pipeline-sensor.yaml`.
Manual trigger paths (unchanged):
- ci-pipeline: argo submit --from=wftmpl/ci-pipeline -p commit-sha=<SHA>
- compile-and-deploy: curl -X POST http://workflow-trigger-eventsource-svc.foxhunt:12001/compile-deploy \
-H 'Content-Type: application/json' -d '{"commit_sha": "<SHA>"}'
Source: docs.nvidia.com/nsight-systems/InstallationGuide
Repo: developer.download.nvidia.com/devtools/repos/ubuntu2404/amd64/
Key: 7fa2af80.pub from CUDA repo
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Output: /data/nsys/ on training-data PVC (survives pod GC)
--show-output=true prints summary to pod logs for mid-run visibility
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Enables nsys profiling for CUDA graph node-level traces on H100.
Use --sanitizer nsys in argo-train.sh to activate.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use --sanitizer nsys to wrap training binary with nsys profile.
Captures CUDA graph node-level traces + GPU metrics.
Output: /workspace/output/nsys_profile.nsys-rep
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pod logs are now archived to MinIO before pod GC. Historical logs
accessible via `argo logs <workflow>` after pod completion.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CLI arg was removed in cost-driven hold timing but Argo template
still passed it, causing exit code 2 on H100 deploy.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scaleway Kapsule GPU nodes need NVIDIA driver DKMS compile on cold
start (~15 min). Setting min_size=1 keeps the node warm between runs.
Scale to 0 manually when done training for the day.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
FXCACHE_VERSION in header handles cache invalidation. First attempt
runs precompute — if cache is valid, skips (fast). If version
mismatch, deletes stale cache and regenerates. No more rm on every run.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Version constant in fxcache.rs. validate() rejects stale versions
with clear error message. ensure-fxcache Argo template now deletes
old cache and regenerates unconditionally. No manual PVC cleanup
needed — version mismatch triggers automatic regeneration.
v2: OFI_DIM=20 (20 microstructure features).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
AlgoGetIds picks algorithms by ID order. On H100 (SM90), the first
valid algo uses split-K/stream-K with internal workspace allocation
that silently produces zero output when replayed via CUDA Graph.
The heuristic selects graph-safe algorithms by design.
Also removes FOXHUNT_NO_GRAPH temp env var and raw DtoH diagnostic probe.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Temporarily sets FOXHUNT_NO_GRAPH=1 in train-best pods to run
forward+backward+Adam ungraphed. If grad_buf is non-zero ungraphed,
the root cause is CUDA Graph capture of cuBLASLt on H100 SM90.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause of DNS deadlock (2026-03-18, 2026-04-12): node-bootstrap
DaemonSet overwrote /etc/resolv.conf with ONLY nameserver 10.32.0.10
(cluster DNS). When nodes rebuild, CoreDNS can't pull its image because
containerd resolves via 10.32.0.10 which requires CoreDNS to be running.
Fix: resolv.conf now has DUAL nameservers:
nameserver 10.32.0.10 # primary: cluster DNS
nameserver 169.254.169.254 # fallback: Scaleway metadata DNS
Containerd tries kube-dns first (fast, resolves .svc.cluster.local),
falls back to VPC resolver when kube-dns is unreachable. Breaks the
chicken-and-egg permanently.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: weight tensors packed sequentially in the flat params buffer
had non-aligned start offsets when preceding tensors had odd element counts
(e.g. bias of 51 atoms = 204 bytes, 204 % 16 = 12). cublasLtMatmul with
CUBLAS_COMPUTE_32F_FAST_TF32 requires 16-byte aligned buffer pointers.
Fix: pad each tensor to 4-element boundary (16 bytes) in both
f32_weight_ptrs_from_base (pointer computation) and compute_total_params
(buffer allocation). Added align4() and padded_byte_offset() helpers,
fixed shrink_perturb skip range and bottleneck gradient offset.
Switched compute type: CUBLAS_COMPUTE_32F → CUBLAS_COMPUTE_32F_FAST_TF32
(forward + backward). Explicit TF32 tensor core path, required by cuBLAS
13.0 on H100 SM90.
Deleted dead bf16_weight_ptrs function.
19/19 smoke tests pass on RTX 3050.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Argo requires task output parameters to flow through DAG arguments →
template inputs, not direct {{tasks.X.outputs}} in container args.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- iqn_lambda restored to 0.25 (was 0.0 for NaN isolation)
- STEP_DIAG per-step logging removed (was temporary)
- FOXHUNT_GRAD_DIAG env var removed from Argo template
IQN NaN root cause fixed in 8cbabcef5 (f32-as-bf16 type mismatch).
Ready for H100 baseline deployment.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Temporary diagnostic: logs per-stage gradient norms (pre-clip, post-clip,
post-IQN, final) to identify where H100 gradient collapse occurs.
Remove after root cause is found.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Per-mount readOnly: true is still set on hyperopt/train/evaluate steps.
Only the precompute step gets write access to generate fxcache.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PVC is ReadWriteOnce — can't mount on compile node (different pool).
Precompute step now:
- runs on GPU node (same as training, shares PVC mount)
- depends on fetch-binary (gets binary from GitLab to /workspace/)
- copies binary to /data/bin/ on PVC for future use
- skips if cache already exists (data unchanged)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- precompute_features early-exits if {cache_key}.fxcache already exists
(cache key = SHA256 of filenames + sizes, only changes when data changes)
- compile-and-train copies binaries to PVC /data/bin/ after compile
- precompute step added to compile-and-train DAG (runs after compile,
before training, parallel with GPU warmup)
- precompute template reads binary from PVC (no GitLab download)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Was relying on stale binary at /data/bin/precompute_features on PVC.
Now downloads latest from GitLab Package Registry (uploaded by
compile-and-train). Falls back to PVC binary if download fails.
Also added precompute_features to compile-and-train examples list.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
