Files

jgrusewski 2286b8976a docs: GPU resource optimization design — RL→L4, supervised→L40S

Route DQN/PPO to cheaper L4 (CPU-bound), supervised models to L40S
(GPU-bound). Separate CI templates with tuned resource requests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-02-27 16:15:07 +01:00

3.0 KiB

Raw Blame History

GPU Resource Optimization Design

Goal: Route RL training jobs (DQN, PPO) to cheaper L4 GPUs and supervised jobs to L40S, maximizing throughput and minimizing cost.

Architecture: Two-tier GPU routing with separate CI templates per model family, tuned CPU/memory requests per workload profile.

Tech Stack: GitLab CI, Kubernetes node selectors, Scaleway Kapsule node pools

Current State

L40S node (ci-training pool): Running one job at a time, GPU at 34% utilization, 617MiB/46GB VRAM
L4 node (ci-compile pool): Idle after compile, scaled to zero
DQN/PPO: CPU-bound (environment simulation), use <1GB VRAM, small checkpoints (~233KB)
Supervised models (TFT, Mamba2, TGGN, TLOB, Liquid, KAN, xLSTM, Diffusion): GPU-bound (attention, SSM, denoising), need larger VRAM

Design

Routing Rules

Model Family	GPU Pool	Node	Why
DQN, PPO (+ hyperopt)	ci-compile	L4-1-24G	CPU-bound, <1GB VRAM, L4 has idle CPU after compile
TFT, Mamba2, TGGN, TLOB, Liquid, KAN, xLSTM, Diffusion (+ hyperopt)	ci-training	L40S-1-48G	GPU-bound, benefit from 48GB VRAM and FP8

Resource Requests

RL jobs (L4):

CPU: 3000m request / 3800m limit (high — RL is CPU-bound from env simulation)
Memory: 8Gi request / 16Gi limit
Concurrency: 2 jobs max on L4 (7800m total CPU)

Supervised jobs (L40S):

CPU: 1000m request / 2000m limit (low — GPU-bound)
Memory: 16Gi request / 40Gi limit
Concurrency: 3-4 jobs via CUDA MPS / time-slicing (46GB VRAM shared)

CI Template Changes

New .train-rl-base template (no KUBERNETES_NODE_SELECTOR_POOL override → uses runner default ci-compile):

.train-rl-base:
  stage: train
  image: ${REGISTRY}/training:${CI_COMMIT_SHA}
  tags: [kapsule, gpu]
  variables:
    KUBERNETES_CPU_REQUEST: "3000m"
    KUBERNETES_CPU_LIMIT: "3800m"
    KUBERNETES_MEMORY_REQUEST: "8Gi"
    KUBERNETES_MEMORY_LIMIT: "16Gi"
  # No KUBERNETES_NODE_SELECTOR_POOL → defaults to ci-compile (L4)

Updated .train-validate-base (supervised, explicit L40S):

.train-validate-base:
  stage: train
  image: ${REGISTRY}/training:${CI_COMMIT_SHA}
  tags: [kapsule, gpu]
  variables:
    KUBERNETES_NODE_SELECTOR_POOL: "k8s.scaleway.com/pool-name=ci-training"
    KUBERNETES_CPU_REQUEST: "1000m"
    KUBERNETES_CPU_LIMIT: "2000m"
    KUBERNETES_MEMORY_REQUEST: "16Gi"
    KUBERNETES_MEMORY_LIMIT: "40Gi"

Job Routing

Jobs extending .train-rl-base: train-validate-dqn, train-validate-ppo, hyperopt-dqn, hyperopt-ppo Jobs extending .train-validate-base: all 8 supervised train + 8 supervised hyperopt jobs

Hyperopt Epoch Reduction

Reduce default hyperopt epochs from 15 → 8 for RL models (faster iteration, DQN currently ~1100s/epoch × 15 = 4.5h per trial).

Cost Impact

L4 is ~60% cheaper than L40S per hour
RL jobs move to L4 → immediate cost reduction
Supervised jobs can run concurrently on L40S → higher throughput
Both pools autoscale 0→1 → no idle cost when unused

3.0 KiB Raw Blame History Unescape Escape