gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,782 Commits 13 Branches 0 Tags
ae704a7b7346da17039e4aeef8265f65c6df40ee
Commit Graph

4 Commits

Author SHA1 Message Date
jgrusewski
b31329931f fix(infra): remove MinIO TLS, fix sccache 0% cache hits, update pool selectors 
- Remove all HTTPS/TLS from MinIO (plain HTTP for internal cluster traffic)
- Fix sccache 0% cache hit rate (rustls rejected self-signed MinIO cert)
- Remove hardcoded URLs from k8s_dispatcher.rs (S3_ENDPOINT, TRAINING_RUNTIME_IMAGE,
  CALLBACK_ENDPOINT now required env vars)
- Update GitLab registry S3 credentials to HTTP endpoint
- Fix PVC manifest (20Gi → 100Gi to match cluster)
- Fix nodeSelector: infra/foxhunt → platform (match actual node pool)
- Fix rclone trailing backslash causing chmod to be parsed as rclone args
- Remove minio-ca-cert ConfigMap references from all manifests
- Update trading-service GPU overlay to l40s pool

20 files changed, -118 lines net

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-06 23:53:05 +01:00
jgrusewski
9dc4d2585f fix(infra): rewrite dashboards with verified metrics, fix KSM + internal DNS 
Rewrite all 5 Grafana dashboards using only confirmed-existing Prometheus
metric names (1023 metrics inventoried). Dashboards now use hardcoded
datasource UIDs instead of unresolvable ${DS_*} template variables.

- Cockpit: 38 panels (service health, gRPC, Argo CI/CD, trading, metrics)
- Training: 33 panels (epochs, loss, Sharpe, Q-values, GPU, hyperopt)
- Trading: 5 rows (overview, latency, gRPC pipeline, data acq, backtesting)
- Infrastructure: 25 panels (cluster, nodes, workloads, storage, GPU, Prometheus)
- Observability: 22 panels (API gateway auth/security/routing, Loki, Tempo)

Fix kube-state-metrics: add part-of label for netpol, fix nodeSelector
(infra→platform), increase memory limit (128→256Mi for OOM), add K8s API
egress rules covering both 10.32.0.0/16 and 172.16.0.0/16 CIDR ranges.

Add training-pods pod-based scrape job for ephemeral Argo workflow pods
exposing metrics on port 9094.

Migrate all image references from localhost:30500 to internal DNS
(gitlab-registry.foxhunt.svc.cluster.local:5000) across 14 YAML files.

Fix import.sh ConfigMap name (grafana-dashboards-infra → infrastructure).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-05 18:55:06 +01:00
jgrusewski
6d731b860d fix(infra): use NodePort localhost:30500 for all container image refs 
containerd on Kapsule nodes uses host DNS which can't resolve
.svc.cluster.local names. Switch all image references from
gitlab-registry.foxhunt.svc.cluster.local:5000 to localhost:30500
(NodePort on the GitLab registry). Also make training runtime image
configurable via TRAINING_RUNTIME_IMAGE env var.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-05 14:31:54 +01:00
jgrusewski
889ab4588b feat(infra): add web-dashboard k8s deployment on platform pool 
- nginx pod serves Vite-built static assets from MinIO S3
- initContainer fetches dist/ from s3://foxhunt-binaries/web-dashboard/
- SPA routing via try_files, aggressive caching for hashed assets
- dashboard.fxhnt.ai proxied to web-dashboard:80 via tailscale nginx
- CI pipeline: build-web-dashboard task (npm ci + build + rclone upload)
- detect-changes: needs-dashboard output for web-dashboard/ paths

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-05 13:56:59 +01:00