Commit Graph

12 Commits

Author SHA1 Message Date
jgrusewski
99f54e3f94 fix(infra): resolve postgres disk-full cascade, add SPOF mitigations
Postgres PVC hit 100% → crash-looped → GitLab webservice couldn't
verify SSH keys → all git operations failed. Root cause: 10Gi PVC
outgrown by GitLab database.

Fixes applied:
- Expand postgres PVC 10Gi → 20Gi
- Expand minio PVC 100Gi → 150Gi (was 81.8% full)
- Expand prometheus PVC 2Gi → 10Gi, retentionSize 1500MB → 8GB
- Scale gitlab-webservice to 2 replicas for HA
- Add PVC autoscaler CronJob (every 15min, auto-expand at 85%)
- Add daily postgres backup CronJob (pg_dump → MinIO, 7d + 4w retention)
- Add PrometheusRule storage alerts (75% warning, 90% critical)
- Add network policies for maintenance job egress

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 18:55:25 +01:00
jgrusewski
7382ffd1e2 feat(infra): QuestDB metrics sink + monitoring network policies
Add QuestDB ILP sink for training metrics, update Prometheus scrape
configs, and fix network policies for monitoring stack connectivity.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 02:42:53 +01:00
jgrusewski
68b6aa8313 feat(infra): migrate container registry from SCW to internal GitLab
Full migration off Scaleway Container Registry to internal GitLab
registry backed by MinIO S3. All 4 images (ci-builder, ci-builder-cpu,
foxhunt-runtime, foxhunt-training-runtime) rebuilt in internal registry.

Registry & images:
- All image refs → gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/
- imagePullSecrets: scw-registry → gitlab-registry
- Kaniko build template: two-step DAG (git-clone → kaniko-build) with shared PVC
- Kaniko layer cache enabled at root/foxhunt/cache
- AWS_ACCESS_KEY_ID: $SCW_ACCESS_KEY → $MINIO_ACCESS_KEY in .gitlab-ci.yml

Network policies:
- ci-pipeline: add HTTP/80, registry/5000, webservice/8181 egress rules

DNS & Tailscale proxy cleanup:
- Remove ci, prometheus, monitor DNS records (no longer exposed)
- Rename s3 → minio DNS record
- Remove Argo UI, Prometheus, monitor nginx server blocks
- Remove argo-htpasswd volume mount
- Tailscale proxy nodeSelector: infra → platform

Terraform cleanup:
- Delete infra/modules/registry/ (SCW CR namespace)
- Delete infra/modules/object-storage/ (SCW S3 buckets)
- Delete infra/modules/secrets/ (SCW secrets)
- Delete corresponding live configs
- TF state backend: S3 → GitLab HTTP

Argo workflows:
- Add events/ (GitLab push eventsource + ci-pipeline sensor)
- ci-pipeline + training templates: SCW → internal registry
- Delete obsolete compile-training-template.yaml

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 13:52:24 +01:00
jgrusewski
b3cc52c13a infra: migrate QuestDB to platform pool (fresh start)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 10:22:57 +01:00
jgrusewski
5084dcd3b2 infra: migrate Postgres to platform pool with PVC migration
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 10:21:20 +01:00
jgrusewski
70672f9878 infra: move Redis to platform pool
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 10:07:58 +01:00
jgrusewski
e25bc8d835 fix(k8s): fix NetworkPolicy label selectors and scope default-deny
Caught during pre-apply review:
- Scope default-deny/allow-dns/allow-monitoring to pods with
  app.kubernetes.io/part-of=foxhunt (was podSelector:{} which would
  block gitlab, grafana, loki sharing the namespace)
- Fix infrastructure.yaml: use app.kubernetes.io/name instead of
  bare app label for postgres/redis/minio/questdb podSelectors
- Add app.kubernetes.io/part-of=foxhunt to all pod templates so
  default-deny and infrastructure ingress rules match correctly

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 23:51:54 +01:00
jgrusewski
2a6ec55755 infra: split foxhunt-secrets into per-concern secrets
db-credentials, jwt-secret, redis-credentials, s3-credentials.
Each service only references the secrets it needs.
Values are REPLACE_IN_DEPLOY markers — real values
injected by scripts/deploy-secrets.sh.

Also updated references in postgres.yaml, migrate.yaml,
and postgres-init.yaml to use db-credentials.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 23:41:13 +01:00
jgrusewski
1a41f9475a refactor(infra): update all manifests for pool rename (always-on→services, ci-build→ci-compile/ci-training)
All K8s nodeSelectors, CI config, runner config, monitoring docs, and
smoke tests now reference the new pool names provisioned by terragrunt.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 10:33:34 +01:00
jgrusewski
5380bd9186 infra: split GPU pool into training (H100) and inference (L4), add auto-taint
- Split single H100 GPU pool into two purpose-specific pools:
  - gpu-training: H100-1-80G (€2.73/hr) for 10-model ensemble training
  - gpu-inference: L4-1-24G (€0.75/hr) for cost-effective trading inference
- Add GPU taint controller DaemonSet that auto-taints new GPU nodes
  with nvidia.com/gpu=true:NoSchedule to prevent non-GPU workloads
- Fix service log directory permissions with emptyDir volumes
  (observability init fails creating /app/logs as non-root user)
- Increase postgres max_connections from 25 to 100
  (7 services each requesting connection pools exhausted the limit)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 17:39:56 +01:00
jgrusewski
f434309f59 infra: provision Kapsule cluster in fr-par with GPU support
Move all infrastructure from nl-ams to fr-par-2 for H100 GPU availability.
Kapsule cluster provisioned with 3 node pools (always-on DEV1-M, CI GP1-XS,
GPU H100-1-80G), databases deployed (Postgres/TimescaleDB, Redis, QuestDB),
and Tailscale subnet router configured.

Key changes:
- Region: nl-ams -> fr-par-2 (GPU types only available in Paris)
- K8s version: 1.30 -> 1.34 with auto-upgrade
- Block storage: migrate b_ssd -> SBS (scaleway_block_volume)
- Kapsule: add VPC private network, delete_additional_resources
- GPU pool: conditional via enable_gpu_pool variable
- K8s manifests: fix registry URLs, secret key names, PG subPath
- Add terragrunt cache/lockfile to .gitignore

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 15:35:40 +01:00
jgrusewski
2a3c107b31 infra: add K8s manifests for full paper trading stack
- Tailscale subnet router (zero public access)
- PostgreSQL, Redis, QuestDB (PVCs on always-on node)
- 7 service deployments (trading, api-gw, broker-gw, ml, backtest, agent, web-gw)
- GPU training Job template + idle reaper CronJob
- Block Storage PV/PVC for shared training data

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 14:27:32 +01:00