- DQN/PPO trainers now record epoch, loss, val_loss, batch/s every epoch
- CI training jobs get Prometheus scrape annotations via runner overrides
- Allow Prometheus (foxhunt namespace) to scrape foxhunt-ci pods
- Skip no-model metrics in monitoring service session grouping
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pool mapping:
- ci-compile-cpu (POP2-32C-128G): Rust compile, web dashboard, manifest
- ci-compile (L4): CUDA compile with stubs only
- ci-training (L40S): ALL training (RL + supervised)
Main runner default → ci-compile-cpu
RL runner default → ci-training
.train-rl-base → explicit ci-training node selector
Replaced all stale ci-rl references with correct pool names.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove runtime_class_name="nvidia" from runner configs (applied to ALL
pods, broke compile on CPU-only ci-compile-cpu nodes). Instead use
runtime_class_name_overwrite_allowed + KUBERNETES_RUNTIME_CLASS_NAME
per-job on GPU templates (.train-rl-base, .train-validate-base).
Also routes build-web-dashboard and write-manifest to ci-compile-cpu.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch all MinIO communication from HTTP to HTTPS across the entire stack:
- MinIO deployment: mount TLS secret, tcpSocket probes, HTTPS init-buckets
- 11 service YAMLs: HTTPS rclone endpoint + CA cert volume mount
- Training job template + train.sh: HTTPS for fetch-binaries and uploader
- CI pipeline (.gitlab-ci.yml): all 7 rclone exports use HTTPS + CA cert
- GitLab runner values: minio-ca-cert ConfigMap volume for CI pods
- Rust: CA cert loading via MINIO_CA_CERT_PATH in training_uploader,
storage backend, data_acquisition uploader, and k8s_dispatcher
- Cert generation script (infra/scripts/generate-minio-tls.sh)
Fixes training uploader S3 upload failures caused by with_allow_http(false)
connecting to an HTTP endpoint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RL models (DQN/PPO) only need basic CUDA runtime (~2GB), while supervised
models need cuDNN (~4GB). Splitting saves ~2GB pull time per RL job.
Rename the L4 GPU pool from ci-compile to ci-rl to reflect its actual use.
Add DaemonSet image pre-puller to cache training images on GPU nodes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Set runtime_class_name=nvidia at runner level (all GPU pools have GPUs)
- Fix KUBERNETES_NODE_SELECTOR format: "key=value" not just value
- Replace wrong *_overwrite_allowed (regex) with correct
*_overwrite_max_allowed (max value) for CPU/memory overrides
- Remove per-job KUBERNETES_RUNTIME_CLASS_NAME (runner-level handles it)
- Simplify GPU detection in test/compile (GPU always present now)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
compile-services OOMKilled at 8Gi — ml crate with CUDA needs ~20Gi.
CI variable overrides (KUBERNETES_MEMORY_LIMIT) are silently dropped
by runner config merger for non-auto-generated fields. Increase
defaults to 3500m/12Gi request, 7800m/28Gi limit to fit L4 nodes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Setting runtime_class_name = "" causes Kubernetes to reject pod creation
with "resource name may not be empty". Omitting it entirely means no
runtimeClassName on pod spec by default; GPU jobs override via
KUBERNETES_RUNTIME_CLASS_NAME CI variable.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
L4 ci-compile node was failing with 'no runtime for nvidia' because
fresh nodes take time to install nvidia drivers. Kaniko builds don't
need GPU at all. Now only Rust compile/test and training jobs request
nvidia RuntimeClass via KUBERNETES_RUNTIME_CLASS_NAME CI variable.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- compile-services now depends on check (not test), running in parallel
with test stage — saves ~10 min from critical path
- Runner concurrent bumped 4→10 so all 9 Kaniko builds run simultaneously
- Runner default resources lowered (500m/1Gi) for lightweight Kaniko jobs;
Rust compile/test/training override via KUBERNETES_*_REQUEST CI vars
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All K8s nodeSelectors, CI config, runner config, monitoring docs, and
smoke tests now reference the new pool names provisioned by terragrunt.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
H100 is sm_90, not compatible with existing sm_89 sccache.
Revert runner node selector and resource limits to ci-build (L4).
Training validation stage (manual DQN/TFT smoke tests) stays.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Runner node selector: ci-build (L4) → gpu-training (H100) for faster
compilation (2x+ vCPUs) and native GPU training support
- Resource limits bumped: 20 vCPU / 128Gi (was 10 vCPU / 38Gi)
- New 'train' stage with manual DQN and TFT validation jobs
(50-step smoke tests to verify CUDA + model correctness)
- Requires helm upgrade of gitlab-runner with new values
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CUDA cannot handle linear layers with zero input dimensions.
When num_static_features=0 or num_known_features=0, the
VariableSelectionNetwork constructor creates linear(0, 0, ...)
which triggers CUDA_ERROR_INVALID_VALUE on GPU.
- Make static/future VSN and GRN encoder fields Option<T>
- Skip layer creation when feature count is 0
- Forward pass skips absent feature paths gracefully
- Trainable adapter creates empty placeholder tensors
- Add cilium CNI toleration to training job template and runner
All 103 TFT tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Runner tags were only set during initial Helm registration but lost
on re-registration. Adding tag_list to the TOML config template
ensures tags survive runner pod restarts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Create dedicated ci-build node pool (L4-1-24G, €0.75/hr) replacing
H100 (€3.50+/hr) for CI builds. H100 reserved for training only.
- Add ci-build pool to Terraform (L4-1-24G, scale-to-zero, fr-par-2)
- Update GitLab Runner to target ci-build pool with L4-sized limits
- Change CUDA_COMPUTE_CAP from 90 (H100) to 89 (L4) in CI
- Dockerfile.training keeps CUDA_COMPUTE_CAP=90 for H100 training
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create 20Gi sccache-pvc (scw-bssd) for fast local cache hits
- Mount at /mnt/sccache in CI build pods via runner helm values
- Use SCCACHE_DIR instead of SCCACHE_BUCKET for check/test jobs
- Kaniko builds keep S3 sccache (can't mount PVCs in Kaniko)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Redis image WORKDIR is /data and entrypoint runs chown recursively.
PVC mounted at /data/training was read-only, causing chown to fail
with exit code 1, crashing the Redis sidecar.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Mount training-data-pvc at /data/training in build pods via runner config
- Update real_data_loader to search per-symbol subdirectories (Databento layout)
- Support .dbn.zst (zstd-compressed) files alongside raw .dbn
- Add FOXHUNT_DATA_DIR env var override for CI PVC path
- Extract decode_ohlcv_bars helper (generic over reader type)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace pod_spec strategic merge patch (silently not applied) with
runtime_class_name="nvidia". The nvidia RuntimeClass uses the
nvidia-container-runtime which injects GPU drivers, nvidia-smi, and
/dev/nvidia* devices into all containers automatically.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Strategic merge patches are applied to the Pod object, not PodSpec.
The patch needs {"spec":{"containers":[...]}} not {"containers":[...]}.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- poll_timeout 180s→600s: H100 provisioning takes ~3-5min from scale-to-zero
- build-ci-builder: only auto-run on push with Dockerfile changes (API
pipelines evaluate changes=true, causing unnecessary kaniko builds)
- check: add needs:[] to decouple from prepare stage
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
bindgen_cuda calls nvidia-smi to detect compute capability, which fails
without GPU device access. Setting CUDA_COMPUTE_CAP=90 bypasses this for
compilation. Pod spec GPU request ensures NVIDIA device plugin injects
/dev/nvidia* for test-time CUDA execution.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove orphaned `ci` pool (GP1-XS, 4 vCPU wasted — nothing scheduled to it).
Remove `ci-build` pool (GP1-M) — builds now run on gpu-training (H100-1-80G:
24 vCPU, 240GB, real CUDA). This eliminates the need for CUDA stubs,
separate test-gpu jobs, and ml crate exclusions.
Pool layout after:
always-on DEV1-M (core services, always on)
gitlab GP1-XS (GitLab CE + runner manager, always on)
gpu-training H100-1-80G (CI builds + ML training, scale-to-zero)
gpu-inference L4-1-24G (trading inference, scale-to-zero)
Build pod limits bumped to 16 vCPU / 64GB (from 6/12GB) to use H100 capacity.
Runner now has `gpu` tag — all tests including ml crate run in single job.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add .gitignore entries for .dbn, .safetensors, .onnx and other large binaries
- Point GitLab Runner at internal cluster URL (not external Tailscale)
- Disable nginx proxy_buffering for git clone/push through Tailscale proxy
- Increase socat SSH proxy buffer sizes to 1MB
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Keep existing git.fxhnt.ai hostname for GitLab CE (repoint DNS
to new Tailscale IP after deploy). Add Terraform DNS module to
manage the A record via Terragrunt instead of manual scw CLI.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
