Switch all MinIO communication from HTTP to HTTPS across the entire stack:
- MinIO deployment: mount TLS secret, tcpSocket probes, HTTPS init-buckets
- 11 service YAMLs: HTTPS rclone endpoint + CA cert volume mount
- Training job template + train.sh: HTTPS for fetch-binaries and uploader
- CI pipeline (.gitlab-ci.yml): all 7 rclone exports use HTTPS + CA cert
- GitLab runner values: minio-ca-cert ConfigMap volume for CI pods
- Rust: CA cert loading via MINIO_CA_CERT_PATH in training_uploader,
storage backend, data_acquisition uploader, and k8s_dispatcher
- Cert generation script (infra/scripts/generate-minio-tls.sh)
Fixes training uploader S3 upload failures caused by with_allow_http(false)
connecting to an HTTP endpoint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move all compiled binary uploads/downloads from public Scaleway S3 to
MinIO running inside Kapsule. Keeps proprietary binaries and trained
models off the public internet.
- CI compile/upload jobs: Scaleway → MinIO endpoint + credentials
- train.sh: all 3 rclone blocks → RCLONE_CONFIG_MINIO_*
- Deploy stage: apply MinIO manifests + wait for readiness before rollout
- Add infra/k8s/minio/ (Deployment, Service, Secret, PVC, init-buckets Job)
- Update s3-credentials.secret.example → minio-credentials format
IaC Terragrunt jobs and SCW registry pulls intentionally unchanged.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All K8s nodeSelectors, CI config, runner config, monitoring docs, and
smoke tests now reference the new pool names provisioned by terragrunt.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Mass cleanup of crates/ml/:
- Delete 121 dead/broken/superseded example files (-44,098 lines)
- Delete 6 QAT test files (-4,201 lines) — QAT is disabled at runtime
(trainer.rs falls back to FP32 with warning)
- Delete 2 stale markdown files in examples/
- Delete orphaned src/bin/train_tft.rs (unimplemented stub)
- Clean up Cargo.toml: remove stale [[example]] entries, add missing ones
- Fix stale binary references in log_size_test.rs
- Add infra consistency test (35 checks across Dockerfile/train.sh/Cargo.toml)
Remaining examples (7): train_baseline_rl, train_baseline_supervised,
evaluate_baseline, hyperopt_baseline_rl, hyperopt_baseline_supervised,
download_baseline, cuda_test
All 2390 lib tests pass. All examples compile. 35/35 infra checks pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove orphaned `ci` pool (GP1-XS, 4 vCPU wasted — nothing scheduled to it).
Remove `ci-build` pool (GP1-M) — builds now run on gpu-training (H100-1-80G:
24 vCPU, 240GB, real CUDA). This eliminates the need for CUDA stubs,
separate test-gpu jobs, and ml crate exclusions.
Pool layout after:
always-on DEV1-M (core services, always on)
gitlab GP1-XS (GitLab CE + runner manager, always on)
gpu-training H100-1-80G (CI builds + ML training, scale-to-zero)
gpu-inference L4-1-24G (trading inference, scale-to-zero)
Build pod limits bumped to 16 vCPU / 64GB (from 6/12GB) to use H100 capacity.
Runner now has `gpu` tag — all tests including ml crate run in single job.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix container registry region nl-ams → fr-par across all CI jobs
- Add sccache build-args to training image build (no-op fallback for local)
- Add rclone to training Docker runtime for S3 output sync
- Update train.sh: S3 sync on Job completion via rclone env-var config,
--run-id tracking, evaluate preset for walk-forward evaluation
- Add s3-credentials Secret template (.example, apply via kubectl)
- Add design doc and implementation plan
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove deleted foxhunt-deploy COPY from all Dockerfiles, update sccache
default region to fr-par, add build-and-push.sh for building all service
images and pushing to rg.fr-par.scw.cloud/foxhunt/.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- K8s API ACL restricted to Tailscale CGNAT (100.64.0.0/10) + admin IPs
- CI and GPU pools: min_size=0 with lifecycle ignore_changes on size
so autoscaler manages actual node count without Terraform drift
- Add harden.sh script for post-provisioning security lockdown
(ACL setup, exposed service check, bucket ACL verify, registry visibility)
- Update smoke-test.sh default region to fr-par
Security audit results:
- All services ClusterIP only (no LoadBalancer/NodePort)
- No Ingress resources
- Container registry: private
- S3 buckets: HTTP 403 on anonymous access
- Gitea: all ports closed on public IP (Tailscale-only via security group)
- Kapsule nodes: no open ports on external IPs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- train.sh: submit GPU training K8s Jobs with presets
(quick-test, single-model, full-ensemble, hyperopt)
- smoke-test.sh: end-to-end verification of cluster,
networking, databases, services, and node pools
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
