Commit Graph

5 Commits

Author SHA1 Message Date
jgrusewski
ddf1082cb5 Revert "fix(ci): retain completed pods 10 min for log access (CPU only)"
This reverts commit 8e6effb7f6.
2026-03-14 01:10:34 +01:00
jgrusewski
8e6effb7f6 fix(ci): retain completed pods 10 min for log access (CPU only)
Add deleteDelayDuration: 600s to podGC on CPU workflow templates and
sensors (ci-pipeline, compile-deploy, build-image). GPU training
workflows keep immediate cleanup to avoid wasting expensive GPU time.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 01:10:06 +01:00
jgrusewski
65f29a8167 fix(infra): update sensor nodeSelectors from DEV1-L to platform pool
POP2-32C-128G is decommissioned. Sensor pods (ci-pipeline-trigger,
workflow-trigger) now target the platform node pool.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-12 09:07:59 +01:00
jgrusewski
45addfd35d feat(infra): auto-rebuild Docker images in CI pipelines, add build-image webhook
- Wire build-image endpoint into EventSource + Sensor for manual triggers
- Add rebuild-* DAG tasks to compile-and-deploy and compile-and-train
  templates — Kaniko layer cache makes cached builds ~15-30s
- Fix foxhunt-runtime Dockerfile: rename ubuntu user instead of groupadd
  (GID 1000 already exists in ubuntu:24.04)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-12 08:35:55 +01:00
jgrusewski
6ba52425ea feat(infra): Argo workflow templates, drop cuDNN, GPU hotpath fixes
- Add compile-and-deploy, train-dqn/ppo/supervised WorkflowTemplates
- Add Argo Events (EventSource, Sensor, Service) for webhook triggers
- Add NetworkPolicy for compile-and-deploy pods (MinIO/DNS/API egress)
- Add convenience scripts: argo-compile-deploy.sh, argo-train.sh
- Drop cuDNN feature flags from all 9 ML crates (zero conv ops in codebase)
- Switch training runtime base to nvidia/cuda:12.9.1-runtime (saves ~800MB)
- Delete unused selective_scan.cu (16KB, zero Rust callers)
- Fix GPU hotpath violations in ml-core (NVTX, gradient utils, capabilities)
- Fix clippy warnings in ml-dqn (VarMap backticks, const fn)
- Add DQN GPU smoketest, backtest evaluator signal adapter fixes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-12 01:44:03 +01:00