gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,012 Commits 13 Branches 0 Tags
befa7e2f55c55ca81f3f65d5a5259bb3a594e8dd
Commit Graph

9 Commits

Author SHA1 Message Date
jgrusewski
5dbe529ee6 fix(ci): rebuild CI builder images with mold linker 
Both Dockerfiles already install mold v2.35.1 but the registry images
are stale builds without it. This change triggers rebuild-ci-builder
and rebuild-ci-builder-cpu pipeline steps via detect-changes.

.cargo/config.toml uses -fuse-ld=mold — without mold in the image,
linking falls back to the system default (slower).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-07 00:20:09 +01:00
jgrusewski
68b6aa8313 feat(infra): migrate container registry from SCW to internal GitLab 
Full migration off Scaleway Container Registry to internal GitLab
registry backed by MinIO S3. All 4 images (ci-builder, ci-builder-cpu,
foxhunt-runtime, foxhunt-training-runtime) rebuilt in internal registry.

Registry & images:
- All image refs → gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/
- imagePullSecrets: scw-registry → gitlab-registry
- Kaniko build template: two-step DAG (git-clone → kaniko-build) with shared PVC
- Kaniko layer cache enabled at root/foxhunt/cache
- AWS_ACCESS_KEY_ID: $SCW_ACCESS_KEY → $MINIO_ACCESS_KEY in .gitlab-ci.yml

Network policies:
- ci-pipeline: add HTTP/80, registry/5000, webservice/8181 egress rules

DNS & Tailscale proxy cleanup:
- Remove ci, prometheus, monitor DNS records (no longer exposed)
- Rename s3 → minio DNS record
- Remove Argo UI, Prometheus, monitor nginx server blocks
- Remove argo-htpasswd volume mount
- Tailscale proxy nodeSelector: infra → platform

Terraform cleanup:
- Delete infra/modules/registry/ (SCW CR namespace)
- Delete infra/modules/object-storage/ (SCW S3 buckets)
- Delete infra/modules/secrets/ (SCW secrets)
- Delete corresponding live configs
- TF state backend: S3 → GitLab HTTP

Argo workflows:
- Add events/ (GitLab push eventsource + ci-pipeline sensor)
- ci-pipeline + training templates: SCW → internal registry
- Delete obsolete compile-training-template.yaml

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-05 13:52:24 +01:00
jgrusewski
cf88f73379 infra: add rclone to CI builder images for S3 binary uploads 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-28 17:57:08 +01:00
jgrusewski
f13b0df6e8 feat(infra): upgrade CI to L40S + mold linker for faster builds 
- Switch linker from lld to mold (~2-5x faster linking for large binaries)
  - Install mold 2.35.1 in CI builder Dockerfile
  - Update .cargo/config.toml: -fuse-ld=mold
- Upgrade CI build pool: L4-1-24G → L40S-1-48G (~2x training throughput)
  - Increase max_size from 1 to 2 (allows concurrent jobs, fixes scheduling deadlocks)
  - Update runner resource limits for L40S node (24 vCPU, 96GB)
- Update runner-values.yaml comments and .gitlab-ci.yml header

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-27 00:05:23 +01:00
jgrusewski
20a771578f fix(ci): skip GPU-dependent tests on CPU CI nodes, keep CUDA stub for loading 
The backtesting crate links libcuda.so.1 at runtime (via candle CUDA).
GPU-less CI nodes need the stub library so binaries can load, but
model_loader tests that actually use CUDA must be skipped. These will
run on gpu-training nodes separately.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-25 12:59:29 +01:00
jgrusewski
463291115f fix(ci): add libcuda.so.1 stub for test stage on GPU-less nodes 
The backtesting crate dynamically links to libcuda.so.1 via candle.
NVIDIA CUDA dev images include stubs at /usr/local/cuda/lib64/stubs/
but only as libcuda.so (not .so.1). Create the symlink both in the
Dockerfile (for future builds) and inline in the test script (for
the current image).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-25 12:26:04 +01:00
Administrator
bdcd016f33 ci: install protoc 28.3 from GitHub (Ubuntu protoc too old for proto3 optional) 2026-02-25 09:37:16 +00:00
Administrator
39cfb037ea ci: add libfontconfig1-dev to CI builder (plotters dependency) 2026-02-25 09:24:59 +00:00
Administrator
8a73f47624 ci: pre-baked CI builder image, pipeline prepare stage 
- Add Dockerfile.ci-builder (CUDA 12.4 + Rust 1.89 + protoc + sccache + git + lld)
- Add prepare stage to build CI builder image via Kaniko (auto on Dockerfile change, manual otherwise)
- Replace before_script apt-get installs with pre-baked image in .rust-base
- Add git to Dockerfile.service for candle git dependency
- Add sccache stats output to check stage
 2026-02-25 08:22:30 +00:00