gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,782 Commits 13 Branches 0 Tags
ae704a7b7346da17039e4aeef8265f65c6df40ee
Commit Graph

8 Commits

Author SHA1 Message Date
jgrusewski
e4870b17b9 fix: tune log levels across workspace — demote noisy warn to debug/trace 
Reduce log noise for non-critical operational paths: connection retries,
expected fallbacks, graceful degradation, and optional feature absence.
Keeps warn/error for genuine failures requiring attention.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-14 11:35:15 +01:00
jgrusewski
c384ab4b43 fix(infra): replace gitlab.com/prometheus annotations with prometheus.io 
Prometheus was configured to scrape pods via prometheus.io/scrape
annotations, but all services and training jobs used gitlab.com/
prefix from legacy GitLab-managed Prometheus — causing Prometheus
to never discover any foxhunt pods. This resulted in stale/missing
metrics on the Grafana training dashboard.

12 files updated across services/, gpu-overlays/, training/, and
monitoring/ (node-exporter, dcgm-exporter cleanup).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-08 18:39:24 +01:00
jgrusewski
7a9683d5fb feat(infra): GitLab releases with CalVer auto-versioning 
Replace MinIO binary distribution with GitLab Generic Package Registry.
Every code push to main auto-creates a CalVer tag (vYYYY.MM.N),
compiles, uploads binaries to GitLab packages, creates a Release
with auto-generated notes, and deploys via deployment patching.

- New CI templates: create-tag, upload-release
- Modified: compile-services/training upload to GitLab packages
- Modified: deploy-services patches FOXHUNT_RELEASE on deployments
- All 7 service initContainers fetch from GitLab (curl, deploy token)
- Training job-template binary fetch from GitLab (data stays MinIO)
- MinIO retains: sccache, training data, model checkpoints

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-07 20:38:41 +01:00
jgrusewski
b31329931f fix(infra): remove MinIO TLS, fix sccache 0% cache hits, update pool selectors 
- Remove all HTTPS/TLS from MinIO (plain HTTP for internal cluster traffic)
- Fix sccache 0% cache hit rate (rustls rejected self-signed MinIO cert)
- Remove hardcoded URLs from k8s_dispatcher.rs (S3_ENDPOINT, TRAINING_RUNTIME_IMAGE,
  CALLBACK_ENDPOINT now required env vars)
- Update GitLab registry S3 credentials to HTTP endpoint
- Fix PVC manifest (20Gi → 100Gi to match cluster)
- Fix nodeSelector: infra/foxhunt → platform (match actual node pool)
- Fix rclone trailing backslash causing chmod to be parsed as rclone args
- Remove minio-ca-cert ConfigMap references from all manifests
- Update trading-service GPU overlay to l40s pool

20 files changed, -118 lines net

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-06 23:53:05 +01:00
jgrusewski
9dc4d2585f fix(infra): rewrite dashboards with verified metrics, fix KSM + internal DNS 
Rewrite all 5 Grafana dashboards using only confirmed-existing Prometheus
metric names (1023 metrics inventoried). Dashboards now use hardcoded
datasource UIDs instead of unresolvable ${DS_*} template variables.

- Cockpit: 38 panels (service health, gRPC, Argo CI/CD, trading, metrics)
- Training: 33 panels (epochs, loss, Sharpe, Q-values, GPU, hyperopt)
- Trading: 5 rows (overview, latency, gRPC pipeline, data acq, backtesting)
- Infrastructure: 25 panels (cluster, nodes, workloads, storage, GPU, Prometheus)
- Observability: 22 panels (API gateway auth/security/routing, Loki, Tempo)

Fix kube-state-metrics: add part-of label for netpol, fix nodeSelector
(infra→platform), increase memory limit (128→256Mi for OOM), add K8s API
egress rules covering both 10.32.0.0/16 and 172.16.0.0/16 CIDR ranges.

Add training-pods pod-based scrape job for ephemeral Argo workflow pods
exposing metrics on port 9094.

Migrate all image references from localhost:30500 to internal DNS
(gitlab-registry.foxhunt.svc.cluster.local:5000) across 14 YAML files.

Fix import.sh ConfigMap name (grafana-dashboards-infra → infrastructure).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-05 18:55:06 +01:00
jgrusewski
6d731b860d fix(infra): use NodePort localhost:30500 for all container image refs 
containerd on Kapsule nodes uses host DNS which can't resolve
.svc.cluster.local names. Switch all image references from
gitlab-registry.foxhunt.svc.cluster.local:5000 to localhost:30500
(NodePort on the GitLab registry). Also make training runtime image
configurable via TRAINING_RUNTIME_IMAGE env var.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-05 14:31:54 +01:00
jgrusewski
68b6aa8313 feat(infra): migrate container registry from SCW to internal GitLab 
Full migration off Scaleway Container Registry to internal GitLab
registry backed by MinIO S3. All 4 images (ci-builder, ci-builder-cpu,
foxhunt-runtime, foxhunt-training-runtime) rebuilt in internal registry.

Registry & images:
- All image refs → gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/
- imagePullSecrets: scw-registry → gitlab-registry
- Kaniko build template: two-step DAG (git-clone → kaniko-build) with shared PVC
- Kaniko layer cache enabled at root/foxhunt/cache
- AWS_ACCESS_KEY_ID: $SCW_ACCESS_KEY → $MINIO_ACCESS_KEY in .gitlab-ci.yml

Network policies:
- ci-pipeline: add HTTP/80, registry/5000, webservice/8181 egress rules

DNS & Tailscale proxy cleanup:
- Remove ci, prometheus, monitor DNS records (no longer exposed)
- Rename s3 → minio DNS record
- Remove Argo UI, Prometheus, monitor nginx server blocks
- Remove argo-htpasswd volume mount
- Tailscale proxy nodeSelector: infra → platform

Terraform cleanup:
- Delete infra/modules/registry/ (SCW CR namespace)
- Delete infra/modules/object-storage/ (SCW S3 buckets)
- Delete infra/modules/secrets/ (SCW secrets)
- Delete corresponding live configs
- TF state backend: S3 → GitLab HTTP

Argo workflows:
- Add events/ (GitLab push eventsource + ci-pipeline sensor)
- ci-pipeline + training templates: SCW → internal registry
- Delete obsolete compile-training-template.yaml

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-05 13:52:24 +01:00
jgrusewski
57e22c01a8 refactor: update K8s, CI, Docker, Prometheus, scripts, and FXT CLI for api rename 
- K8s: rename api-gateway → api manifests, delete web-gateway, update network policies
- CI: rename compile/deploy jobs, delete web-gateway jobs
- Docker: rename service in compose files
- Prometheus: update scrape targets and alert rules
- Scripts: update binary references in build/test/cert scripts
- FXT CLI: rename api_gateway_url → api_url (with serde alias for compat)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-04 23:46:36 +01:00