gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,654 Commits 13 Branches 0 Tags
7cbcfce781392f1149ccb90bf0c2ef21f82c19df
Commit Graph

7 Commits

Author SHA1 Message Date
jgrusewski
74a572a5c4 feat(infra): enable MinIO TLS with self-signed CA for in-cluster HTTPS 
Switch all MinIO communication from HTTP to HTTPS across the entire stack:
- MinIO deployment: mount TLS secret, tcpSocket probes, HTTPS init-buckets
- 11 service YAMLs: HTTPS rclone endpoint + CA cert volume mount
- Training job template + train.sh: HTTPS for fetch-binaries and uploader
- CI pipeline (.gitlab-ci.yml): all 7 rclone exports use HTTPS + CA cert
- GitLab runner values: minio-ca-cert ConfigMap volume for CI pods
- Rust: CA cert loading via MINIO_CA_CERT_PATH in training_uploader,
  storage backend, data_acquisition uploader, and k8s_dispatcher
- Cert generation script (infra/scripts/generate-minio-tls.sh)

Fixes training uploader S3 upload failures caused by with_allow_http(false)
connecting to an HTTP endpoint.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-01 11:56:06 +01:00
jgrusewski
995b46a680 feat(infra): migrate binary storage from Scaleway S3 to in-cluster MinIO 
Move all compiled binary uploads/downloads from public Scaleway S3 to
MinIO running inside Kapsule. Keeps proprietary binaries and trained
models off the public internet.

- CI compile/upload jobs: Scaleway → MinIO endpoint + credentials
- train.sh: all 3 rclone blocks → RCLONE_CONFIG_MINIO_*
- Deploy stage: apply MinIO manifests + wait for readiness before rollout
- Add infra/k8s/minio/ (Deployment, Service, Secret, PVC, init-buckets Job)
- Update s3-credentials.secret.example → minio-credentials format

IaC Terragrunt jobs and SCW registry pulls intentionally unchanged.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-01 01:19:39 +01:00
jgrusewski
cd8a45d8a8 fix(gpu): wire PPO GPU experience collection, fix debug log levels, increase hyperopt deadline 
- Wire GPU experience collection into PPO hyperopt adapter with CPU fallback
  (ensure_gpu_data, gpu_collect_trajectories, batch-to-trajectory converter)
- Downgrade 6 ERROR-level debug logs to debug!() in DQN hyperopt adapter
- Increase hyperopt activeDeadlineSeconds from 1h to 6h (job-template + train.sh)
- Mark 3 data-dependent real_data_loader tests as #[ignore]
- Fix 4 clippy map_or → is_none_or warnings in trading_service

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-28 19:58:16 +01:00
jgrusewski
022036cb96 refactor(ml): consolidate 21 training binaries into 2 unified baselines 
Replace 20 per-model training examples with:
- train_baseline_rl: DQN + PPO (renamed from train_baseline)
- train_baseline_supervised: TFT, Mamba2, Liquid, TGGN, TLOB, KAN, xLSTM, Diffusion
  via model factory + UnifiedTrainable generic training loop

Update Dockerfile.training (16→7 binaries), train.sh MODEL_BINARY map,
and job-template.yaml default. -12,759 lines.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-25 23:16:25 +01:00
jgrusewski
c8f2dacc5f feat(infra): H100 smoketest pipeline — S3 output sync, sccache, registry fix 
- Fix container registry region nl-ams → fr-par across all CI jobs
- Add sccache build-args to training image build (no-op fallback for local)
- Add rclone to training Docker runtime for S3 output sync
- Update train.sh: S3 sync on Job completion via rclone env-var config,
  --run-id tracking, evaluate preset for walk-forward evaluation
- Add s3-credentials Secret template (.example, apply via kubectl)
- Add design doc and implementation plan

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-24 21:31:22 +01:00
jgrusewski
5fb84a02f0 feat: training pipeline for all 10 ML ensemble models 
- Add standalone training binaries: TGGN, KAN, xLSTM, Diffusion (DBN data)
- Update Dockerfile.training: 6 → 16 binaries (all 10 models + hyperopt + baseline)
- Expand train.sh: 4 → 10 models, fix registry URL and GPU pool nodeSelector
- Add GPU overlay manifests for trading-service and ml-training-service
- Create training data PVC and upload pod manifests
- Expand web-gateway model validation: 4 → 10 types (training + tune routes)
- Extend dashboard: 10 model cards grouped by category (RL/Temporal/Graph/Generative)
- Add training image build job to Gitea CI workflow
- Update GPU taint controller to exclude inference pool from tainting
- Fix job-template nodeSelector: gpu → gpu-training

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-24 20:47:50 +01:00
jgrusewski
4f85d0d475 infra: add training orchestrator and smoke test scripts 
- train.sh: submit GPU training K8s Jobs with presets
  (quick-test, single-model, full-ensemble, hyperopt)
- smoke-test.sh: end-to-end verification of cluster,
  networking, databases, services, and node pools

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-24 14:27:50 +01:00