Output: /data/nsys/ on training-data PVC (survives pod GC)
--show-output=true prints summary to pod logs for mid-run visibility
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use --sanitizer nsys to wrap training binary with nsys profile.
Captures CUDA graph node-level traces + GPU metrics.
Output: /workspace/output/nsys_profile.nsys-rep
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pod logs are now archived to MinIO before pod GC. Historical logs
accessible via `argo logs <workflow>` after pod completion.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CLI arg was removed in cost-driven hold timing but Argo template
still passed it, causing exit code 2 on H100 deploy.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
FXCACHE_VERSION in header handles cache invalidation. First attempt
runs precompute — if cache is valid, skips (fast). If version
mismatch, deletes stale cache and regenerates. No more rm on every run.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Version constant in fxcache.rs. validate() rejects stale versions
with clear error message. ensure-fxcache Argo template now deletes
old cache and regenerates unconditionally. No manual PVC cleanup
needed — version mismatch triggers automatic regeneration.
v2: OFI_DIM=20 (20 microstructure features).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
AlgoGetIds picks algorithms by ID order. On H100 (SM90), the first
valid algo uses split-K/stream-K with internal workspace allocation
that silently produces zero output when replayed via CUDA Graph.
The heuristic selects graph-safe algorithms by design.
Also removes FOXHUNT_NO_GRAPH temp env var and raw DtoH diagnostic probe.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Temporarily sets FOXHUNT_NO_GRAPH=1 in train-best pods to run
forward+backward+Adam ungraphed. If grad_buf is non-zero ungraphed,
the root cause is CUDA Graph capture of cuBLASLt on H100 SM90.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause of DNS deadlock (2026-03-18, 2026-04-12): node-bootstrap
DaemonSet overwrote /etc/resolv.conf with ONLY nameserver 10.32.0.10
(cluster DNS). When nodes rebuild, CoreDNS can't pull its image because
containerd resolves via 10.32.0.10 which requires CoreDNS to be running.
Fix: resolv.conf now has DUAL nameservers:
nameserver 10.32.0.10 # primary: cluster DNS
nameserver 169.254.169.254 # fallback: Scaleway metadata DNS
Containerd tries kube-dns first (fast, resolves .svc.cluster.local),
falls back to VPC resolver when kube-dns is unreachable. Breaks the
chicken-and-egg permanently.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Argo requires task output parameters to flow through DAG arguments →
template inputs, not direct {{tasks.X.outputs}} in container args.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- iqn_lambda restored to 0.25 (was 0.0 for NaN isolation)
- STEP_DIAG per-step logging removed (was temporary)
- FOXHUNT_GRAD_DIAG env var removed from Argo template
IQN NaN root cause fixed in 8cbabcef5 (f32-as-bf16 type mismatch).
Ready for H100 baseline deployment.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Temporary diagnostic: logs per-stage gradient norms (pre-clip, post-clip,
post-IQN, final) to identify where H100 gradient collapse occurs.
Remove after root cause is found.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Per-mount readOnly: true is still set on hyperopt/train/evaluate steps.
Only the precompute step gets write access to generate fxcache.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PVC is ReadWriteOnce — can't mount on compile node (different pool).
Precompute step now:
- runs on GPU node (same as training, shares PVC mount)
- depends on fetch-binary (gets binary from GitLab to /workspace/)
- copies binary to /data/bin/ on PVC for future use
- skips if cache already exists (data unchanged)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- precompute_features early-exits if {cache_key}.fxcache already exists
(cache key = SHA256 of filenames + sizes, only changes when data changes)
- compile-and-train copies binaries to PVC /data/bin/ after compile
- precompute step added to compile-and-train DAG (runs after compile,
before training, parallel with GPU warmup)
- precompute template reads binary from PVC (no GitLab download)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Was relying on stale binary at /data/bin/precompute_features on PVC.
Now downloads latest from GitLab Package Registry (uploaded by
compile-and-train). Falls back to PVC binary if download fails.
Also added precompute_features to compile-and-train examples list.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
cuda-nvcc-12-4 + cuda-cudart-dev-12-4 (~400MB) added to the CPU builder.
Reverts compile-and-deploy back to ci-builder-cpu (was briefly switched
to full ci-builder which is 6GB+ and unnecessary).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
cudarc build.rs requires nvcc to precompile CUDA kernels to .cubin files.
The ci-builder-cpu image doesn't have the CUDA toolkit installed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previously appended /${SYMBOL} to data-dir AND passed --symbol, causing
double nesting. Now data-dir stays as-is and --symbol handles filtering
via collect_dbn_files_filtered.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rust tracing with JSON subscriber in a container without TTY uses
fully-buffered stdout — logs only flush on buffer full or process exit.
This made H100 training appear stuck for 28+ minutes with no output.
stdbuf -oL forces line buffering so epoch logs appear immediately.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Epoch duration self-balances: bigger GPU → bigger auto-scaled batch →
fewer steps per epoch. The manual cap created 7 different values
(0, 8, 64, 100, 200, 300, 2000) across configs/tests/examples, making
behavior inconsistent between environments.
Removed from: DQNHyperparameters, training profiles (smoketest,
localdev, production), CLI args, Argo templates, hyperopt adapter,
all test overrides, supervised example.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
precompute passes data-dir/SYMBOL to match how train_baseline_rl
calls load_all_bars(data_dir, symbol). compile-and-train gets
FOXHUNT_FEATURE_CACHE_DIR env var and --epochs fix.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reuses compile-training, gpu-warmup, fetch-binary templates from
compile-and-train. Own train step runs train_baseline_rl directly
with --epochs, FOXHUNT_FEATURE_CACHE_DIR, and fxcache validation.
No hyperopt — pure baseline with default params.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- DAG: compile (ci-compile-cpu) + gpu-warmup → train (H100)
- Compiles from source targeting compute cap 90
- fxcache REQUIRED — fails if no .fxcache found
- Error chains printed with {:#} for full cause visibility
- Uses cargo-target-pvc for compile cache + sccache
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
