feat(ci): route all training to L40S pool (unify GPU resources)

All training jobs (RL + supervised) now use ci-training pool (L40S-1-48G)
instead of splitting RL→L4, supervised→L40S. With parallel hyperopt,
RL benefits from the L40S's more powerful GPU and same 8 vCPU.

Simplifies infrastructure: can potentially decommission L4 pool.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-02-27 22:31:33 +01:00
parent 539f6c2727
commit b160afe038

View File

@@ -424,14 +424,12 @@ build-training:
--destination "${REGISTRY}/training:latest"
# --------------------------------------------------------------------------
# Stage 3c: Training validation — quick smoke test on GPU
# Two base templates: RL (L4, CPU-heavy) and Supervised (L40S, GPU-heavy)
# Stage 3c: Training — all models run on L40S (48GB VRAM, 8 vCPU)
# --------------------------------------------------------------------------
# --------------------------------------------------------------------------
# RL training base: DQN + PPO → ci-compile pool (L4)
# RL models are CPU-bound (environment simulation), use <1GB VRAM.
# No KUBERNETES_NODE_SELECTOR_pool override → runner default pool=ci-compile (L4-1-24G)
# RL training base: DQN + PPO → ci-training pool (L40S)
# RL hyperopt runs parallel PSO trials — CPU+GPU bound.
# --------------------------------------------------------------------------
.train-rl-base:
stage: train
@@ -439,15 +437,15 @@ build-training:
tags:
- kapsule
- gpu
# Runner default node_selector: pool=ci-compile (L4-1-24G)
# training-data-pvc + sccache-pvc mounted via runner config
variables:
# nvidia runtime provided by runner default (runtime_class_name = "nvidia")
# RL hyperopt is CPU-bound (parallel PSO trials). Give full L4 (8 vCPU).
# Route to L40S pool — all training on one GPU type for simplicity
KUBERNETES_NODE_SELECTOR_pool: "pool=ci-training"
# RL hyperopt is CPU-bound (parallel PSO trials). Use full node (8 vCPU).
KUBERNETES_CPU_REQUEST: "6000m"
KUBERNETES_CPU_LIMIT: "7800m"
KUBERNETES_MEMORY_REQUEST: "8Gi"
KUBERNETES_MEMORY_LIMIT: "20Gi"
KUBERNETES_MEMORY_LIMIT: "40Gi"
rules:
- if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
when: manual
@@ -463,7 +461,7 @@ build-training:
# --------------------------------------------------------------------------
# Supervised training base: TFT, Mamba2, TGGN, TLOB, Liquid, KAN, xLSTM, Diffusion → L40S
# Supervised models are GPU-bound — low CPU, high VRAM.
# Supervised models are GPU-bound — low CPU, high VRAM. Same pool as RL.
# --------------------------------------------------------------------------
.train-validate-base:
stage: train