The GPU runners used runtime_class_name=nvidia for CUDA access but
didn't request nvidia.com/gpu as a K8s resource, allowing multiple
training pods to share a GPU without K8s awareness. Add pod_spec
patches to request GPU resources properly so K8s enforces mutual
exclusion between GitLab CI and Argo training jobs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Clear separation: l40s tag → L40S pool, h100 tag → H100 pool.
Updated runner-rl-values.yaml and CI base templates to match.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- DQN/PPO trainers now record epoch, loss, val_loss, batch/s every epoch
- CI training jobs get Prometheus scrape annotations via runner overrides
- Allow Prometheus (foxhunt namespace) to scrape foxhunt-ci pods
- Skip no-model metrics in monitoring service session grouping
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Runner #2 already has tags [kapsule, gpu] but CI jobs still used
kapsule-rl. Also removes stale minio-ca-cert volume from runner values.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RL runner only handles GPU training jobs, so global nvidia runtime
is safe and required (KUBERNETES_RUNTIME_CLASS_NAME override wasn't
being applied). Main runner keeps overwrite-allowed pattern.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pool mapping:
- ci-compile-cpu (POP2-32C-128G): Rust compile, web dashboard, manifest
- ci-compile (L4): CUDA compile with stubs only
- ci-training (L40S): ALL training (RL + supervised)
Main runner default → ci-compile-cpu
RL runner default → ci-training
.train-rl-base → explicit ci-training node selector
Replaced all stale ci-rl references with correct pool names.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tempo was OOMKilled every ~30 min at 512Mi, causing web-gateway OTLP
export errors. Increased to 1Gi.
RL runner had cpu_request_overwrite_max_allowed=4000m but hyperopt RL
jobs need 6000m for parallel PSO trials. Increased to 8000m.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove runtime_class_name="nvidia" from runner configs (applied to ALL
pods, broke compile on CPU-only ci-compile-cpu nodes). Instead use
runtime_class_name_overwrite_allowed + KUBERNETES_RUNTIME_CLASS_NAME
per-job on GPU templates (.train-rl-base, .train-validate-base).
Also routes build-web-dashboard and write-manifest to ci-compile-cpu.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch all MinIO communication from HTTP to HTTPS across the entire stack:
- MinIO deployment: mount TLS secret, tcpSocket probes, HTTPS init-buckets
- 11 service YAMLs: HTTPS rclone endpoint + CA cert volume mount
- Training job template + train.sh: HTTPS for fetch-binaries and uploader
- CI pipeline (.gitlab-ci.yml): all 7 rclone exports use HTTPS + CA cert
- GitLab runner values: minio-ca-cert ConfigMap volume for CI pods
- Rust: CA cert loading via MINIO_CA_CERT_PATH in training_uploader,
storage backend, data_acquisition uploader, and k8s_dispatcher
- Cert generation script (infra/scripts/generate-minio-tls.sh)
Fixes training uploader S3 upload failures caused by with_allow_http(false)
connecting to an HTTP endpoint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RL models (DQN/PPO) only need basic CUDA runtime (~2GB), while supervised
models need cuDNN (~4GB). Splitting saves ~2GB pull time per RL job.
Rename the L4 GPU pool from ci-compile to ci-rl to reflect its actual use.
Add DaemonSet image pre-puller to cache training images on GPU nodes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RL env simulation is single-threaded (~1 core actual). Lowering from
7000m to 2000m request allows DQN + PPO hyperopt to run simultaneously
on one L4 node.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix ServiceAccount: use serviceAccount.name (not deprecated serviceAccountName)
- concurrent=1: one RL job gets full L4 node (7000m/7800m CPU, 16Gi/40Gi RAM)
- Faster iteration: ~2.3x more CPU per job vs splitting across 2 concurrent
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
