gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,782 Commits 13 Branches 0 Tags
ae704a7b7346da17039e4aeef8265f65c6df40ee
Commit Graph

13 Commits

Author SHA1 Message Date
jgrusewski
aabc1d95e5 fix(infra): pin workflow-trigger eventsource to platform pool 
The eventsource had no nodeSelector and landed on ci-compile-cpu
(POP2-HC-32C-64G, €0.56/hr), blocking autoscaler from scaling it
to 0. Now pinned to platform pool — ci-compile-cpu will autoscale
down, saving ~€410/mo.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-14 09:26:31 +01:00
jgrusewski
ddf1082cb5 Revert "fix(ci): retain completed pods 10 min for log access (CPU only)" 
This reverts commit 8e6effb7f6.
 2026-03-14 01:10:34 +01:00
jgrusewski
8e6effb7f6 fix(ci): retain completed pods 10 min for log access (CPU only) 
Add deleteDelayDuration: 600s to podGC on CPU workflow templates and
sensors (ci-pipeline, compile-deploy, build-image). GPU training
workflows keep immediate cleanup to avoid wasting expensive GPU time.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-14 01:10:06 +01:00
jgrusewski
7844523591 feat(ci): add auto-compile ConfigMap toggle (disabled by default) 
Add auto-compile-config ConfigMap with enabled=false. Document the
manual compile-deploy webhook trigger and toggle pattern in the
ci-pipeline sensor. Auto-compile can be enabled per-push by adding
a second trigger to the sensor or via manual webhook.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-12 09:17:52 +01:00
jgrusewski
65f29a8167 fix(infra): update sensor nodeSelectors from DEV1-L to platform pool 
POP2-32C-128G is decommissioned. Sensor pods (ci-pipeline-trigger,
workflow-trigger) now target the platform node pool.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-12 09:07:59 +01:00
jgrusewski
45addfd35d feat(infra): auto-rebuild Docker images in CI pipelines, add build-image webhook 
- Wire build-image endpoint into EventSource + Sensor for manual triggers
- Add rebuild-* DAG tasks to compile-and-deploy and compile-and-train
  templates — Kaniko layer cache makes cached builds ~15-30s
- Fix foxhunt-runtime Dockerfile: rename ubuntu user instead of groupadd
  (GID 1000 already exists in ubuntu:24.04)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-12 08:35:55 +01:00
jgrusewski
6ba52425ea feat(infra): Argo workflow templates, drop cuDNN, GPU hotpath fixes 
- Add compile-and-deploy, train-dqn/ppo/supervised WorkflowTemplates
- Add Argo Events (EventSource, Sensor, Service) for webhook triggers
- Add NetworkPolicy for compile-and-deploy pods (MinIO/DNS/API egress)
- Add convenience scripts: argo-compile-deploy.sh, argo-train.sh
- Drop cuDNN feature flags from all 9 ML crates (zero conv ops in codebase)
- Switch training runtime base to nvidia/cuda:12.9.1-runtime (saves ~800MB)
- Delete unused selective_scan.cu (16KB, zero Rust callers)
- Fix GPU hotpath violations in ml-core (NVTX, gradient utils, capabilities)
- Fix clippy warnings in ml-dqn (VarMap backticks, const fn)
- Add DQN GPU smoketest, backtest evaluator signal adapter fixes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-12 01:44:03 +01:00
jgrusewski
84366d8dd8 fix(infra): pin CI sensor pod to DEV1-L platform nodes 
The Argo Events sensor had no nodeSelector and randomly landed on the
H100 GPU node, preventing autoscale-down. Pin it to DEV1-L to avoid
wasting expensive GPU node hours on a lightweight event listener.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-06 08:45:59 +01:00
jgrusewski
4e71cbf549 fix(ci): use runtime image for deploy step, add podGC to sensor 
- Deploy step: use foxhunt-runtime image (cached on platform node) +
  curl kubectl, replacing bitnami/kubectl:1.31 which doesn't exist.
- Sensor: add podGC:OnPodCompletion and ttlStrategy to workflow spec
  (not inherited from WorkflowTemplate via workflowTemplateRef).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-05 22:46:11 +01:00
jgrusewski
68b6aa8313 feat(infra): migrate container registry from SCW to internal GitLab 
Full migration off Scaleway Container Registry to internal GitLab
registry backed by MinIO S3. All 4 images (ci-builder, ci-builder-cpu,
foxhunt-runtime, foxhunt-training-runtime) rebuilt in internal registry.

Registry & images:
- All image refs → gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/
- imagePullSecrets: scw-registry → gitlab-registry
- Kaniko build template: two-step DAG (git-clone → kaniko-build) with shared PVC
- Kaniko layer cache enabled at root/foxhunt/cache
- AWS_ACCESS_KEY_ID: $SCW_ACCESS_KEY → $MINIO_ACCESS_KEY in .gitlab-ci.yml

Network policies:
- ci-pipeline: add HTTP/80, registry/5000, webservice/8181 egress rules

DNS & Tailscale proxy cleanup:
- Remove ci, prometheus, monitor DNS records (no longer exposed)
- Rename s3 → minio DNS record
- Remove Argo UI, Prometheus, monitor nginx server blocks
- Remove argo-htpasswd volume mount
- Tailscale proxy nodeSelector: infra → platform

Terraform cleanup:
- Delete infra/modules/registry/ (SCW CR namespace)
- Delete infra/modules/object-storage/ (SCW S3 buckets)
- Delete infra/modules/secrets/ (SCW secrets)
- Delete corresponding live configs
- TF state backend: S3 → GitLab HTTP

Argo workflows:
- Add events/ (GitLab push eventsource + ci-pipeline sensor)
- ci-pipeline + training templates: SCW → internal registry
- Delete obsolete compile-training-template.yaml

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-05 13:52:24 +01:00
jgrusewski
8b6588936d fix(fxt,api): wire real gRPC auth login and add api.access permission 
- Replace simulated login in fxt CLI with real AuthServiceClient gRPC call
- Add auth proto to fxt build.rs compile list and lib.rs proto module
- Add api.access permission to JWT claims issued by AuthGrpcService
- Remove orphaned ci-sensor.yaml (replaced by cluster-applied version)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-05 01:42:10 +01:00
jgrusewski
63a454d800 fix(ci): add RBAC for Argo sensor workflow submission 
argo-workflow SA needs workflows/create and workflowtemplates/get
permissions to submit CI pipelines from sensor triggers.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-05 00:54:51 +01:00
jgrusewski
44445aeab4 fix(ci): fix compile-service package mapping and add sensor config 
- Remove unnecessary case map — service names already match Cargo package names
- Add sensor pod serviceAccountName: argo-workflow (RBAC fix)
- Add sensor config to events/ for version tracking

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-05 00:52:03 +01:00