Commit Graph

6 Commits

Author SHA1 Message Date
jgrusewski
10d81b0703 infra(kapsule): consolidate CI builds onto H100 gpu-training pool
Remove orphaned `ci` pool (GP1-XS, 4 vCPU wasted — nothing scheduled to it).
Remove `ci-build` pool (GP1-M) — builds now run on gpu-training (H100-1-80G:
24 vCPU, 240GB, real CUDA). This eliminates the need for CUDA stubs,
separate test-gpu jobs, and ml crate exclusions.

Pool layout after:
  always-on    DEV1-M        (core services, always on)
  gitlab       GP1-XS        (GitLab CE + runner manager, always on)
  gpu-training H100-1-80G    (CI builds + ML training, scale-to-zero)
  gpu-inference L4-1-24G     (trading inference, scale-to-zero)

Build pod limits bumped to 16 vCPU / 64GB (from 6/12GB) to use H100 capacity.
Runner now has `gpu` tag — all tests including ml crate run in single job.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 17:20:26 +01:00
jgrusewski
c8f2dacc5f feat(infra): H100 smoketest pipeline — S3 output sync, sccache, registry fix
- Fix container registry region nl-ams → fr-par across all CI jobs
- Add sccache build-args to training image build (no-op fallback for local)
- Add rclone to training Docker runtime for S3 output sync
- Update train.sh: S3 sync on Job completion via rclone env-var config,
  --run-id tracking, evaluate preset for walk-forward evaluation
- Add s3-credentials Secret template (.example, apply via kubectl)
- Add design doc and implementation plan

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 21:31:22 +01:00
jgrusewski
5fb84a02f0 feat: training pipeline for all 10 ML ensemble models
- Add standalone training binaries: TGGN, KAN, xLSTM, Diffusion (DBN data)
- Update Dockerfile.training: 6 → 16 binaries (all 10 models + hyperopt + baseline)
- Expand train.sh: 4 → 10 models, fix registry URL and GPU pool nodeSelector
- Add GPU overlay manifests for trading-service and ml-training-service
- Create training data PVC and upload pod manifests
- Expand web-gateway model validation: 4 → 10 types (training + tune routes)
- Extend dashboard: 10 model cards grouped by category (RL/Temporal/Graph/Generative)
- Add training image build job to Gitea CI workflow
- Update GPU taint controller to exclude inference pool from tainting
- Fix job-template nodeSelector: gpu → gpu-training

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 20:47:50 +01:00
jgrusewski
f6fb67a58f infra: add build-and-push script and fix Dockerfiles for Scaleway registry
Remove deleted foxhunt-deploy COPY from all Dockerfiles, update sccache
default region to fr-par, add build-and-push.sh for building all service
images and pushing to rg.fr-par.scw.cloud/foxhunt/.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 16:29:43 +01:00
jgrusewski
1441f5fbbb infra: security hardening and autoscale-to-zero for CI/GPU pools
- K8s API ACL restricted to Tailscale CGNAT (100.64.0.0/10) + admin IPs
- CI and GPU pools: min_size=0 with lifecycle ignore_changes on size
  so autoscaler manages actual node count without Terraform drift
- Add harden.sh script for post-provisioning security lockdown
  (ACL setup, exposed service check, bucket ACL verify, registry visibility)
- Update smoke-test.sh default region to fr-par

Security audit results:
- All services ClusterIP only (no LoadBalancer/NodePort)
- No Ingress resources
- Container registry: private
- S3 buckets: HTTP 403 on anonymous access
- Gitea: all ports closed on public IP (Tailscale-only via security group)
- Kapsule nodes: no open ports on external IPs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 15:46:24 +01:00
jgrusewski
4f85d0d475 infra: add training orchestrator and smoke test scripts
- train.sh: submit GPU training K8s Jobs with presets
  (quick-test, single-model, full-ensemble, hyperopt)
- smoke-test.sh: end-to-end verification of cluster,
  networking, databases, services, and node pools

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 14:27:50 +01:00