Commit Graph

360 Commits

Author SHA1 Message Date
jgrusewski
4fb398a531 infra(k8s): switch image refs from Scaleway registry to GitLab registry
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:25:05 +01:00
jgrusewski
678f526225 infra(gitlab): Runner Helm values with K8s executor for ci-build pool
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:23:16 +01:00
jgrusewski
24c1b94fb0 infra(gitlab): Helm values for GitLab CE deployment
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:23:05 +01:00
jgrusewski
1045b190dc infra(gitlab): postgres init job for GitLab database
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:22:54 +01:00
jgrusewski
c8f2dacc5f feat(infra): H100 smoketest pipeline — S3 output sync, sccache, registry fix
- Fix container registry region nl-ams → fr-par across all CI jobs
- Add sccache build-args to training image build (no-op fallback for local)
- Add rclone to training Docker runtime for S3 output sync
- Update train.sh: S3 sync on Job completion via rclone env-var config,
  --run-id tracking, evaluate preset for walk-forward evaluation
- Add s3-credentials Secret template (.example, apply via kubectl)
- Add design doc and implementation plan

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 21:31:22 +01:00
jgrusewski
5fb84a02f0 feat: training pipeline for all 10 ML ensemble models
- Add standalone training binaries: TGGN, KAN, xLSTM, Diffusion (DBN data)
- Update Dockerfile.training: 6 → 16 binaries (all 10 models + hyperopt + baseline)
- Expand train.sh: 4 → 10 models, fix registry URL and GPU pool nodeSelector
- Add GPU overlay manifests for trading-service and ml-training-service
- Create training data PVC and upload pod manifests
- Expand web-gateway model validation: 4 → 10 types (training + tune routes)
- Extend dashboard: 10 model cards grouped by category (RL/Temporal/Graph/Generative)
- Add training image build job to Gitea CI workflow
- Update GPU taint controller to exclude inference pool from tainting
- Fix job-template nodeSelector: gpu → gpu-training

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 20:47:50 +01:00
jgrusewski
5380bd9186 infra: split GPU pool into training (H100) and inference (L4), add auto-taint
- Split single H100 GPU pool into two purpose-specific pools:
  - gpu-training: H100-1-80G (€2.73/hr) for 10-model ensemble training
  - gpu-inference: L4-1-24G (€0.75/hr) for cost-effective trading inference
- Add GPU taint controller DaemonSet that auto-taints new GPU nodes
  with nvidia.com/gpu=true:NoSchedule to prevent non-GPU workloads
- Fix service log directory permissions with emptyDir volumes
  (observability init fails creating /app/logs as non-root user)
- Increase postgres max_connections from 25 to 100
  (7 services each requesting connection pools exhausted the limit)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 17:39:56 +01:00
jgrusewski
3db5c950c7 infra: deploy all services to Kapsule cluster
- Fix DB pool connect/acquire timeouts (50ms→5s) for cluster networking
  (pool-level timeouts, not query timeouts — HFT query timeout stays at 800μs)
- Fix secret key references (DATABASE_PASSWORD→db-password) in all manifests
- Fix api-gateway port (50050→50051) to match actual gRPC listen port
- Fix web-gateway health probe path (/api/health→/health)
- Fix S3 endpoint region (nl-ams→fr-par) in ml-training-service
- Add TLS cert volume mount for ml-training-service
- Add BENZINGA_API_KEY placeholder for backtesting-service startup
- Remove always-on nodeSelector from services (let autoscaler handle)
- Add serve subcommand to ml-training-service container

All 10 pods (3 databases + 7 services) now 1/1 Running.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 17:18:47 +01:00
jgrusewski
f434309f59 infra: provision Kapsule cluster in fr-par with GPU support
Move all infrastructure from nl-ams to fr-par-2 for H100 GPU availability.
Kapsule cluster provisioned with 3 node pools (always-on DEV1-M, CI GP1-XS,
GPU H100-1-80G), databases deployed (Postgres/TimescaleDB, Redis, QuestDB),
and Tailscale subnet router configured.

Key changes:
- Region: nl-ams -> fr-par-2 (GPU types only available in Paris)
- K8s version: 1.30 -> 1.34 with auto-upgrade
- Block storage: migrate b_ssd -> SBS (scaleway_block_volume)
- Kapsule: add VPC private network, delete_additional_resources
- GPU pool: conditional via enable_gpu_pool variable
- K8s manifests: fix registry URLs, secret key names, PG subPath
- Add terragrunt cache/lockfile to .gitignore

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 15:35:40 +01:00
jgrusewski
2a3c107b31 infra: add K8s manifests for full paper trading stack
- Tailscale subnet router (zero public access)
- PostgreSQL, Redis, QuestDB (PVCs on always-on node)
- 7 service deployments (trading, api-gw, broker-gw, ml, backtest, agent, web-gw)
- GPU training Job template + idle reaper CronJob
- Block Storage PV/PVC for shared training data

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 14:27:32 +01:00