# GPU Resource Optimization Design

**Goal:** Route RL training jobs (DQN, PPO) to cheaper L4 GPUs and supervised jobs to L40S, maximizing throughput and minimizing cost.

**Architecture:** Two-tier GPU routing with separate CI templates per model family, tuned CPU/memory requests per workload profile.

**Tech Stack:** GitLab CI, Kubernetes node selectors, Scaleway Kapsule node pools

---

## Current State

- **L40S node** (ci-training pool): Running one job at a time, GPU at 34% utilization, 617MiB/46GB VRAM
- **L4 node** (ci-compile pool): Idle after compile, scaled to zero
- **DQN/PPO**: CPU-bound (environment simulation), use <1GB VRAM, small checkpoints (~233KB)
- **Supervised models** (TFT, Mamba2, TGGN, TLOB, Liquid, KAN, xLSTM, Diffusion): GPU-bound (attention, SSM, denoising), need larger VRAM

## Design

### Routing Rules

| Model Family | GPU Pool | Node | Why |
|---|---|---|---|
| DQN, PPO (+ hyperopt) | ci-compile | L4-1-24G | CPU-bound, <1GB VRAM, L4 has idle CPU after compile |
| TFT, Mamba2, TGGN, TLOB, Liquid, KAN, xLSTM, Diffusion (+ hyperopt) | ci-training | L40S-1-48G | GPU-bound, benefit from 48GB VRAM and FP8 |

### Resource Requests

**RL jobs (L4):**
- CPU: 3000m request / 3800m limit (high — RL is CPU-bound from env simulation)
- Memory: 8Gi request / 16Gi limit
- Concurrency: 2 jobs max on L4 (7800m total CPU)

**Supervised jobs (L40S):**
- CPU: 1000m request / 2000m limit (low — GPU-bound)
- Memory: 16Gi request / 40Gi limit
- Concurrency: 3-4 jobs via CUDA MPS / time-slicing (46GB VRAM shared)

### CI Template Changes

**New `.train-rl-base` template** (no `KUBERNETES_NODE_SELECTOR_POOL` override → uses runner default `ci-compile`):
```yaml
.train-rl-base:
  stage: train
  image: ${REGISTRY}/training:${CI_COMMIT_SHA}
  tags: [kapsule, gpu]
  variables:
    KUBERNETES_CPU_REQUEST: "3000m"
    KUBERNETES_CPU_LIMIT: "3800m"
    KUBERNETES_MEMORY_REQUEST: "8Gi"
    KUBERNETES_MEMORY_LIMIT: "16Gi"
  # No KUBERNETES_NODE_SELECTOR_POOL → defaults to ci-compile (L4)
```

**Updated `.train-validate-base`** (supervised, explicit L40S):
```yaml
.train-validate-base:
  stage: train
  image: ${REGISTRY}/training:${CI_COMMIT_SHA}
  tags: [kapsule, gpu]
  variables:
    KUBERNETES_NODE_SELECTOR_POOL: "k8s.scaleway.com/pool-name=ci-training"
    KUBERNETES_CPU_REQUEST: "1000m"
    KUBERNETES_CPU_LIMIT: "2000m"
    KUBERNETES_MEMORY_REQUEST: "16Gi"
    KUBERNETES_MEMORY_LIMIT: "40Gi"
```

### Job Routing

Jobs extending `.train-rl-base`: `train-validate-dqn`, `train-validate-ppo`, `hyperopt-dqn`, `hyperopt-ppo`
Jobs extending `.train-validate-base`: all 8 supervised train + 8 supervised hyperopt jobs

### Hyperopt Epoch Reduction

Reduce default hyperopt epochs from 15 → 8 for RL models (faster iteration, DQN currently ~1100s/epoch × 15 = 4.5h per trial).

### Cost Impact

- L4 is ~60% cheaper than L40S per hour
- RL jobs move to L4 → immediate cost reduction
- Supervised jobs can run concurrently on L40S → higher throughput
- Both pools autoscale 0→1 → no idle cost when unused