Commit Graph

11 Commits

Author SHA1 Message Date
jgrusewski
c6247ea9f8 fix(ci): update Dockerfiles for restructured repo + migrate to SCW registry
- Update COPY paths in all 3 Dockerfiles for crates/bin/services/testing layout
- Migrate service image builds from internal GitLab registry to SCW CR
- Update Kaniko auth to use SCW credentials (nologin + SCW_SECRET_KEY)
- Remove --insecure-registry flags (SCW CR is HTTPS)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 21:09:52 +01:00
jgrusewski
20a771578f fix(ci): skip GPU-dependent tests on CPU CI nodes, keep CUDA stub for loading
The backtesting crate links libcuda.so.1 at runtime (via candle CUDA).
GPU-less CI nodes need the stub library so binaries can load, but
model_loader tests that actually use CUDA must be skipped. These will
run on gpu-training nodes separately.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 12:59:29 +01:00
jgrusewski
463291115f fix(ci): add libcuda.so.1 stub for test stage on GPU-less nodes
The backtesting crate dynamically links to libcuda.so.1 via candle.
NVIDIA CUDA dev images include stubs at /usr/local/cuda/lib64/stubs/
but only as libcuda.so (not .so.1). Create the symlink both in the
Dockerfile (for future builds) and inline in the test script (for
the current image).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 12:26:04 +01:00
Administrator
bdcd016f33 ci: install protoc 28.3 from GitHub (Ubuntu protoc too old for proto3 optional) 2026-02-25 09:37:16 +00:00
Administrator
39cfb037ea ci: add libfontconfig1-dev to CI builder (plotters dependency) 2026-02-25 09:24:59 +00:00
Administrator
8a73f47624 ci: pre-baked CI builder image, pipeline prepare stage
- Add Dockerfile.ci-builder (CUDA 12.4 + Rust 1.89 + protoc + sccache + git + lld)
- Add prepare stage to build CI builder image via Kaniko (auto on Dockerfile change, manual otherwise)
- Replace before_script apt-get installs with pre-baked image in .rust-base
- Add git to Dockerfile.service for candle git dependency
- Add sccache stats output to check stage
2026-02-25 08:22:30 +00:00
jgrusewski
c8f2dacc5f feat(infra): H100 smoketest pipeline — S3 output sync, sccache, registry fix
- Fix container registry region nl-ams → fr-par across all CI jobs
- Add sccache build-args to training image build (no-op fallback for local)
- Add rclone to training Docker runtime for S3 output sync
- Update train.sh: S3 sync on Job completion via rclone env-var config,
  --run-id tracking, evaluate preset for walk-forward evaluation
- Add s3-credentials Secret template (.example, apply via kubectl)
- Add design doc and implementation plan

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 21:31:22 +01:00
jgrusewski
5fb84a02f0 feat: training pipeline for all 10 ML ensemble models
- Add standalone training binaries: TGGN, KAN, xLSTM, Diffusion (DBN data)
- Update Dockerfile.training: 6 → 16 binaries (all 10 models + hyperopt + baseline)
- Expand train.sh: 4 → 10 models, fix registry URL and GPU pool nodeSelector
- Add GPU overlay manifests for trading-service and ml-training-service
- Create training data PVC and upload pod manifests
- Expand web-gateway model validation: 4 → 10 types (training + tune routes)
- Extend dashboard: 10 model cards grouped by category (RL/Temporal/Graph/Generative)
- Add training image build job to Gitea CI workflow
- Update GPU taint controller to exclude inference pool from tainting
- Fix job-template nodeSelector: gpu → gpu-training

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 20:47:50 +01:00
jgrusewski
3db5c950c7 infra: deploy all services to Kapsule cluster
- Fix DB pool connect/acquire timeouts (50ms→5s) for cluster networking
  (pool-level timeouts, not query timeouts — HFT query timeout stays at 800μs)
- Fix secret key references (DATABASE_PASSWORD→db-password) in all manifests
- Fix api-gateway port (50050→50051) to match actual gRPC listen port
- Fix web-gateway health probe path (/api/health→/health)
- Fix S3 endpoint region (nl-ams→fr-par) in ml-training-service
- Add TLS cert volume mount for ml-training-service
- Add BENZINGA_API_KEY placeholder for backtesting-service startup
- Remove always-on nodeSelector from services (let autoscaler handle)
- Add serve subcommand to ml-training-service container

All 10 pods (3 databases + 7 services) now 1/1 Running.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 17:18:47 +01:00
jgrusewski
f6fb67a58f infra: add build-and-push script and fix Dockerfiles for Scaleway registry
Remove deleted foxhunt-deploy COPY from all Dockerfiles, update sccache
default region to fr-par, add build-and-push.sh for building all service
images and pushing to rg.fr-par.scw.cloud/foxhunt/.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 16:29:43 +01:00
jgrusewski
56c33cc1c1 infra: add Dockerfiles for services, web-gateway, and GPU training
- Dockerfile.service: unified multi-stage with sccache S3 support
- Dockerfile.web-gateway: Node dashboard + Rust gateway combined
- Dockerfile.training: CUDA 12.4 with H100 target (compute cap 9.0)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 14:26:54 +01:00