Commit Graph

19 Commits

Author SHA1 Message Date
jgrusewski
b31329931f fix(infra): remove MinIO TLS, fix sccache 0% cache hits, update pool selectors
- Remove all HTTPS/TLS from MinIO (plain HTTP for internal cluster traffic)
- Fix sccache 0% cache hit rate (rustls rejected self-signed MinIO cert)
- Remove hardcoded URLs from k8s_dispatcher.rs (S3_ENDPOINT, TRAINING_RUNTIME_IMAGE,
  CALLBACK_ENDPOINT now required env vars)
- Update GitLab registry S3 credentials to HTTP endpoint
- Fix PVC manifest (20Gi → 100Gi to match cluster)
- Fix nodeSelector: infra/foxhunt → platform (match actual node pool)
- Fix rclone trailing backslash causing chmod to be parsed as rclone args
- Remove minio-ca-cert ConfigMap references from all manifests
- Update trading-service GPU overlay to l40s pool

20 files changed, -118 lines net

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 23:53:05 +01:00
jgrusewski
5215366b22 fix(infra): use proxy_pass for API gateway to support gRPC-web
grpc_pass only handles native gRPC (HTTP/2 frames). The web dashboard
uses gRPC-web via Connect protocol (HTTP/1.1 POST requests). Switch
api.fxhnt.ai nginx block to proxy_pass so tonic_web::GrpcWebLayer on
the API service can handle the protocol conversion.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 14:39:20 +01:00
jgrusewski
889ab4588b feat(infra): add web-dashboard k8s deployment on platform pool
- nginx pod serves Vite-built static assets from MinIO S3
- initContainer fetches dist/ from s3://foxhunt-binaries/web-dashboard/
- SPA routing via try_files, aggressive caching for hashed assets
- dashboard.fxhnt.ai proxied to web-dashboard:80 via tailscale nginx
- CI pipeline: build-web-dashboard task (npm ci + build + rclone upload)
- detect-changes: needs-dashboard output for web-dashboard/ paths

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 13:56:59 +01:00
jgrusewski
dc66780465 fix(infra): route dashboard.fxhnt.ai to web-gateway instead of Grafana
Split the combined grafana+dashboard nginx block so dashboard.fxhnt.ai
proxies to web-gateway:3000 (the actual trading dashboard) instead of
Grafana.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 13:53:34 +01:00
jgrusewski
68b6aa8313 feat(infra): migrate container registry from SCW to internal GitLab
Full migration off Scaleway Container Registry to internal GitLab
registry backed by MinIO S3. All 4 images (ci-builder, ci-builder-cpu,
foxhunt-runtime, foxhunt-training-runtime) rebuilt in internal registry.

Registry & images:
- All image refs → gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/
- imagePullSecrets: scw-registry → gitlab-registry
- Kaniko build template: two-step DAG (git-clone → kaniko-build) with shared PVC
- Kaniko layer cache enabled at root/foxhunt/cache
- AWS_ACCESS_KEY_ID: $SCW_ACCESS_KEY → $MINIO_ACCESS_KEY in .gitlab-ci.yml

Network policies:
- ci-pipeline: add HTTP/80, registry/5000, webservice/8181 egress rules

DNS & Tailscale proxy cleanup:
- Remove ci, prometheus, monitor DNS records (no longer exposed)
- Rename s3 → minio DNS record
- Remove Argo UI, Prometheus, monitor nginx server blocks
- Remove argo-htpasswd volume mount
- Tailscale proxy nodeSelector: infra → platform

Terraform cleanup:
- Delete infra/modules/registry/ (SCW CR namespace)
- Delete infra/modules/object-storage/ (SCW S3 buckets)
- Delete infra/modules/secrets/ (SCW secrets)
- Delete corresponding live configs
- TF state backend: S3 → GitLab HTTP

Argo workflows:
- Add events/ (GitLab push eventsource + ci-pipeline sensor)
- ci-pipeline + training templates: SCW → internal registry
- Delete obsolete compile-training-template.yaml

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 13:52:24 +01:00
jgrusewski
57e22c01a8 refactor: update K8s, CI, Docker, Prometheus, scripts, and FXT CLI for api rename
- K8s: rename api-gateway → api manifests, delete web-gateway, update network policies
- CI: rename compile/deploy jobs, delete web-gateway jobs
- Docker: rename service in compose files
- Prometheus: update scrape targets and alert rules
- Scripts: update binary references in build/test/cert scripts
- FXT CLI: rename api_gateway_url → api_url (with serde alias for compat)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 23:46:36 +01:00
jgrusewski
70f5f4ff52 feat(argo): add basic auth login for Argo UI
Add nginx basic auth (htpasswd) to ci.fxhnt.ai proxy block and
switch Argo to server auth mode. Login: jgrusewski / Welcome01.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 22:58:40 +01:00
jgrusewski
370752f253 feat(infra): expose Argo Workflows UI at ci.fxhnt.ai
Rename DNS record from argo→ci and update nginx proxy server_name
to match. Applied via terragrunt and kubectl.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 22:58:40 +01:00
jgrusewski
a2468846e1 feat(argo): add Argo Workflows infrastructure for training orchestration
- Helm values (controller + server on platform node, MinIO artifact repo)
- WorkflowTemplate: parameterized 5-step DAG (fetch→hyperopt→train→eval→upload)
- Nginx proxy for argo.fxhnt.ai → Argo Server :2746
- DNS A record for argo.fxhnt.ai
- MinIO bucket foxhunt-training-results for Argo artifacts
- Kustomization for kubectl apply -k

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 22:58:40 +01:00
jgrusewski
cc209aec7e feat(infra): replace PVC binary distribution with MinIO S3 fetch
Eliminate foxhunt-binaries and training-binaries PVCs — services and
training jobs now fetch binaries from MinIO via rclone initContainers.
This removes L40S GPU autoscale-for-PVC-writes, removes RWO node
affinity constraints, and requires zero image rebuilds.

Changes:
- .gitlab-ci.yml: replace ~115 lines of binary-writer pod logic with
  aws s3 cp to MinIO (~25 lines)
- 8 service YAMLs + 2 GPU overlays: add rclone initContainer + emptyDir
- Training job template: MinIO rclone fetch replaces PVC copy
- Delete foxhunt-binaries-pvc.yaml and training-binaries-pvc.yaml
- Add s3.fxhnt.ai DNS record (Terraform) and nginx proxy block
- Replace pod-writer-deploy skill with MinIO-based deploy skill

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 16:46:57 +01:00
jgrusewski
66d6d0f2a5 chore: delete monitoring_service — absorbed into API Gateway
Remove from workspace members, delete K8s deployment, remove nginx routing.
Training metrics and system health now served directly by API Gateway.

Changes:
- Cargo.toml: remove services/monitoring_service from workspace members
- services/monitoring_service/: deleted entirely (6 files)
- infra/k8s/services/monitoring-service.yaml: deleted
- infra/k8s/network-policies/monitoring-service.yaml: deleted
- infra/k8s/network-policies/api-gateway.yaml: remove egress rule to monitoring-service
- infra/k8s/network-policies/web-gateway.yaml: remove egress rule to monitoring-service
- infra/k8s/services/api-gateway.yaml: remove stale MONITORING_SERVICE_URL env var
- infra/k8s/gitlab/tailscale-proxy.yaml: redirect monitor.fxhnt.ai to api-gateway:50051
- .gitlab-ci.yml: remove monitoring_service from compile, copy, and deploy loops
- services/trading_service/src/services/monitoring.rs: update stale comments to
  reference api_gateway (not monitoring_service) as training metrics host

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 21:26:00 +01:00
jgrusewski
6b67f6193a infra: expose monitoring service via monitor.fxhnt.ai Tailscale proxy
Add DNS record, nginx gRPC proxy block, network policy for Tailscale
ingress, and auto-detect fxhnt.ai in fxt monitor URL derivation.
Show GPU telemetry even when no training sessions are active.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 22:56:40 +01:00
jgrusewski
fea630e9ff infra: move tailscale proxies to platform pool
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 10:08:20 +01:00
jgrusewski
48316d6874 feat(infra): add api.fxhnt.ai DNS + gRPC reverse proxy for API Gateway
Add nginx server block for api.fxhnt.ai with grpc_pass to the
api-gateway ClusterIP service. Add Scaleway DNS A record via Terraform
module. Fix fxt gRPC client to enable TLS when connecting to https://
endpoints (tonic requires explicit ClientTlsConfig).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 22:41:55 +01:00
jgrusewski
3bdefe86d5 feat(monitoring): production dashboards, remove Cockpit, add Kapsule logs
- Remove Cockpit remote_write from GitLab Prometheus (all metrics local)
- Uninstall k8s-monitoring Helm (Alloy no longer pushes to Cockpit)
- Deploy standalone node-exporter DaemonSet + kube-state-metrics
- Add 10 extra Prometheus scrape configs: node-exporter, kube-state-metrics,
  kubelet, cadvisor, DCGM GPU, foxhunt services, Loki, Tempo, Promtail,
  prometheus.io annotated pods
- Provision 9 Grafana dashboards via sidecar (ConfigMap-based):
  overview, services, GPU/training, infrastructure, logs, CI/CD pipelines,
  cluster overview, GPU overview, GitLab services
- Expand Promtail to ingest kube-system + GitLab runner + node-level logs
- Add dashboard.fxhnt.ai DNS record + nginx routing to Grafana
- Update CI deploy job to apply new monitoring manifests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 22:20:32 +01:00
Administrator
fa0e2a3809 infra: HTTPS wildcard cert, subdomain routing, CI/CD pipeline, Grafana
- Wildcard LE cert for *.fxhnt.ai
- Subdomain routing: git/grafana/prometheus.fxhnt.ai
- CI/CD: Kaniko image builds, sccache, deploy stage
- Grafana with Prometheus datasource
- GitLab node upgraded to GP1-XS (16GB)
2026-02-25 01:18:21 +00:00
Administrator
8f98230dc3 fix: .gitignore large files, runner internal URL, proxy buffering
- Add .gitignore entries for .dbn, .safetensors, .onnx and other large binaries
- Point GitLab Runner at internal cluster URL (not external Tailscale)
- Disable nginx proxy_buffering for git clone/push through Tailscale proxy
- Increase socat SSH proxy buffer sizes to 1MB

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 00:09:23 +00:00
jgrusewski
9c37d0edbb infra(gitlab): enable Prometheus, SSH proxy, explicit always-on type
- Enable Prometheus server (GitLab chart bundled) on gitlab node pool
- Add socat SSH proxy sidecar (port 2222 → gitlab-shell) to Tailscale proxy
- Remove nginx stream module (not available in alpine) in favor of socat
- Set unlimited nginx client_max_body_size for large git pushes
- Add workhorse extraArgs for API limits
- Explicit always_on_type = DEV1-M in kapsule terragrunt

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 00:29:13 +01:00
jgrusewski
26e0caf67c fix(gitlab): Helm values schema, Tailscale proxy, DNS record
- Fix certmanager: use installCertmanager top-level key (not certmanager.install)
- Fix redis: global.redis.auth.enabled (not password.enabled)
- Remove grafana (not in GitLab chart), remove top-level minio (use global.minio.enabled)
- Add toolbox backup objectStorage config (required when minio disabled)
- Set HPA minReplicas/maxReplicas=1 for single-node DEV1-L
- Add Tailscale proxy (nginx + tailscale sidecar) for private access
- DNS: git.fxhnt.ai → 100.90.76.85 (foxhunt-gitlab Tailscale IP)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 00:06:10 +01:00