2286b8976a
Route DQN/PPO to cheaper L4 (CPU-bound), supervised models to L40S (GPU-bound). Separate CI templates with tuned resource requests. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
84 lines
3.0 KiB
Markdown
84 lines
3.0 KiB
Markdown
# GPU Resource Optimization Design
|
||
|
||
**Goal:** Route RL training jobs (DQN, PPO) to cheaper L4 GPUs and supervised jobs to L40S, maximizing throughput and minimizing cost.
|
||
|
||
**Architecture:** Two-tier GPU routing with separate CI templates per model family, tuned CPU/memory requests per workload profile.
|
||
|
||
**Tech Stack:** GitLab CI, Kubernetes node selectors, Scaleway Kapsule node pools
|
||
|
||
---
|
||
|
||
## Current State
|
||
|
||
- **L40S node** (ci-training pool): Running one job at a time, GPU at 34% utilization, 617MiB/46GB VRAM
|
||
- **L4 node** (ci-compile pool): Idle after compile, scaled to zero
|
||
- **DQN/PPO**: CPU-bound (environment simulation), use <1GB VRAM, small checkpoints (~233KB)
|
||
- **Supervised models** (TFT, Mamba2, TGGN, TLOB, Liquid, KAN, xLSTM, Diffusion): GPU-bound (attention, SSM, denoising), need larger VRAM
|
||
|
||
## Design
|
||
|
||
### Routing Rules
|
||
|
||
| Model Family | GPU Pool | Node | Why |
|
||
|---|---|---|---|
|
||
| DQN, PPO (+ hyperopt) | ci-compile | L4-1-24G | CPU-bound, <1GB VRAM, L4 has idle CPU after compile |
|
||
| TFT, Mamba2, TGGN, TLOB, Liquid, KAN, xLSTM, Diffusion (+ hyperopt) | ci-training | L40S-1-48G | GPU-bound, benefit from 48GB VRAM and FP8 |
|
||
|
||
### Resource Requests
|
||
|
||
**RL jobs (L4):**
|
||
- CPU: 3000m request / 3800m limit (high — RL is CPU-bound from env simulation)
|
||
- Memory: 8Gi request / 16Gi limit
|
||
- Concurrency: 2 jobs max on L4 (7800m total CPU)
|
||
|
||
**Supervised jobs (L40S):**
|
||
- CPU: 1000m request / 2000m limit (low — GPU-bound)
|
||
- Memory: 16Gi request / 40Gi limit
|
||
- Concurrency: 3-4 jobs via CUDA MPS / time-slicing (46GB VRAM shared)
|
||
|
||
### CI Template Changes
|
||
|
||
**New `.train-rl-base` template** (no `KUBERNETES_NODE_SELECTOR_POOL` override → uses runner default `ci-compile`):
|
||
```yaml
|
||
.train-rl-base:
|
||
stage: train
|
||
image: ${REGISTRY}/training:${CI_COMMIT_SHA}
|
||
tags: [kapsule, gpu]
|
||
variables:
|
||
KUBERNETES_CPU_REQUEST: "3000m"
|
||
KUBERNETES_CPU_LIMIT: "3800m"
|
||
KUBERNETES_MEMORY_REQUEST: "8Gi"
|
||
KUBERNETES_MEMORY_LIMIT: "16Gi"
|
||
# No KUBERNETES_NODE_SELECTOR_POOL → defaults to ci-compile (L4)
|
||
```
|
||
|
||
**Updated `.train-validate-base`** (supervised, explicit L40S):
|
||
```yaml
|
||
.train-validate-base:
|
||
stage: train
|
||
image: ${REGISTRY}/training:${CI_COMMIT_SHA}
|
||
tags: [kapsule, gpu]
|
||
variables:
|
||
KUBERNETES_NODE_SELECTOR_POOL: "k8s.scaleway.com/pool-name=ci-training"
|
||
KUBERNETES_CPU_REQUEST: "1000m"
|
||
KUBERNETES_CPU_LIMIT: "2000m"
|
||
KUBERNETES_MEMORY_REQUEST: "16Gi"
|
||
KUBERNETES_MEMORY_LIMIT: "40Gi"
|
||
```
|
||
|
||
### Job Routing
|
||
|
||
Jobs extending `.train-rl-base`: `train-validate-dqn`, `train-validate-ppo`, `hyperopt-dqn`, `hyperopt-ppo`
|
||
Jobs extending `.train-validate-base`: all 8 supervised train + 8 supervised hyperopt jobs
|
||
|
||
### Hyperopt Epoch Reduction
|
||
|
||
Reduce default hyperopt epochs from 15 → 8 for RL models (faster iteration, DQN currently ~1100s/epoch × 15 = 4.5h per trial).
|
||
|
||
### Cost Impact
|
||
|
||
- L4 is ~60% cheaper than L40S per hour
|
||
- RL jobs move to L4 → immediate cost reduction
|
||
- Supervised jobs can run concurrently on L40S → higher throughput
|
||
- Both pools autoscale 0→1 → no idle cost when unused
|