feat(ci): route all training to L40S pool (unify GPU resources)
All training jobs (RL + supervised) now use ci-training pool (L40S-1-48G) instead of splitting RL→L4, supervised→L40S. With parallel hyperopt, RL benefits from the L40S's more powerful GPU and same 8 vCPU. Simplifies infrastructure: can potentially decommission L4 pool. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -424,14 +424,12 @@ build-training:
|
||||
--destination "${REGISTRY}/training:latest"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Stage 3c: Training validation — quick smoke test on GPU
|
||||
# Two base templates: RL (L4, CPU-heavy) and Supervised (L40S, GPU-heavy)
|
||||
# Stage 3c: Training — all models run on L40S (48GB VRAM, 8 vCPU)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# RL training base: DQN + PPO → ci-compile pool (L4)
|
||||
# RL models are CPU-bound (environment simulation), use <1GB VRAM.
|
||||
# No KUBERNETES_NODE_SELECTOR_pool override → runner default pool=ci-compile (L4-1-24G)
|
||||
# RL training base: DQN + PPO → ci-training pool (L40S)
|
||||
# RL hyperopt runs parallel PSO trials — CPU+GPU bound.
|
||||
# --------------------------------------------------------------------------
|
||||
.train-rl-base:
|
||||
stage: train
|
||||
@@ -439,15 +437,15 @@ build-training:
|
||||
tags:
|
||||
- kapsule
|
||||
- gpu
|
||||
# Runner default node_selector: pool=ci-compile (L4-1-24G)
|
||||
# training-data-pvc + sccache-pvc mounted via runner config
|
||||
variables:
|
||||
# nvidia runtime provided by runner default (runtime_class_name = "nvidia")
|
||||
# RL hyperopt is CPU-bound (parallel PSO trials). Give full L4 (8 vCPU).
|
||||
# Route to L40S pool — all training on one GPU type for simplicity
|
||||
KUBERNETES_NODE_SELECTOR_pool: "pool=ci-training"
|
||||
# RL hyperopt is CPU-bound (parallel PSO trials). Use full node (8 vCPU).
|
||||
KUBERNETES_CPU_REQUEST: "6000m"
|
||||
KUBERNETES_CPU_LIMIT: "7800m"
|
||||
KUBERNETES_MEMORY_REQUEST: "8Gi"
|
||||
KUBERNETES_MEMORY_LIMIT: "20Gi"
|
||||
KUBERNETES_MEMORY_LIMIT: "40Gi"
|
||||
rules:
|
||||
- if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
|
||||
when: manual
|
||||
@@ -463,7 +461,7 @@ build-training:
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Supervised training base: TFT, Mamba2, TGGN, TLOB, Liquid, KAN, xLSTM, Diffusion → L40S
|
||||
# Supervised models are GPU-bound — low CPU, high VRAM.
|
||||
# Supervised models are GPU-bound — low CPU, high VRAM. Same pool as RL.
|
||||
# --------------------------------------------------------------------------
|
||||
.train-validate-base:
|
||||
stage: train
|
||||
|
||||
Reference in New Issue
Block a user