From 2286b8976a6fde3d7cebf18b5057b9d3601fa4f4 Mon Sep 17 00:00:00 2001
From: jgrusewski <jgrusewski@localhost>
Date: Fri, 27 Feb 2026 16:15:07 +0100
Subject: [PATCH] =?UTF-8?q?docs:=20GPU=20resource=20optimization=20design?=
 =?UTF-8?q?=20=E2=80=94=20RL=E2=86=92L4,=20supervised=E2=86=92L40S?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Route DQN/PPO to cheaper L4 (CPU-bound), supervised models to L40S
(GPU-bound). Separate CI templates with tuned resource requests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---
 ...-02-27-gpu-resource-optimization-design.md | 83 +++++++++++++++++++
 1 file changed, 83 insertions(+)
 create mode 100644 docs/plans/2026-02-27-gpu-resource-optimization-design.md

diff --git a/docs/plans/2026-02-27-gpu-resource-optimization-design.md b/docs/plans/2026-02-27-gpu-resource-optimization-design.md
new file mode 100644
index 000000000..3edbd3af4
--- /dev/null
+++ b/docs/plans/2026-02-27-gpu-resource-optimization-design.md
@@ -0,0 +1,83 @@
+# GPU Resource Optimization Design
+
+**Goal:** Route RL training jobs (DQN, PPO) to cheaper L4 GPUs and supervised jobs to L40S, maximizing throughput and minimizing cost.
+
+**Architecture:** Two-tier GPU routing with separate CI templates per model family, tuned CPU/memory requests per workload profile.
+
+**Tech Stack:** GitLab CI, Kubernetes node selectors, Scaleway Kapsule node pools
+
+---
+
+## Current State
+
+- **L40S node** (ci-training pool): Running one job at a time, GPU at 34% utilization, 617MiB/46GB VRAM
+- **L4 node** (ci-compile pool): Idle after compile, scaled to zero
+- **DQN/PPO**: CPU-bound (environment simulation), use <1GB VRAM, small checkpoints (~233KB)
+- **Supervised models** (TFT, Mamba2, TGGN, TLOB, Liquid, KAN, xLSTM, Diffusion): GPU-bound (attention, SSM, denoising), need larger VRAM
+
+## Design
+
+### Routing Rules
+
+| Model Family | GPU Pool | Node | Why |
+|---|---|---|---|
+| DQN, PPO (+ hyperopt) | ci-compile | L4-1-24G | CPU-bound, <1GB VRAM, L4 has idle CPU after compile |
+| TFT, Mamba2, TGGN, TLOB, Liquid, KAN, xLSTM, Diffusion (+ hyperopt) | ci-training | L40S-1-48G | GPU-bound, benefit from 48GB VRAM and FP8 |
+
+### Resource Requests
+
+**RL jobs (L4):**
+- CPU: 3000m request / 3800m limit (high — RL is CPU-bound from env simulation)
+- Memory: 8Gi request / 16Gi limit
+- Concurrency: 2 jobs max on L4 (7800m total CPU)
+
+**Supervised jobs (L40S):**
+- CPU: 1000m request / 2000m limit (low — GPU-bound)
+- Memory: 16Gi request / 40Gi limit
+- Concurrency: 3-4 jobs via CUDA MPS / time-slicing (46GB VRAM shared)
+
+### CI Template Changes
+
+**New `.train-rl-base` template** (no `KUBERNETES_NODE_SELECTOR_POOL` override → uses runner default `ci-compile`):
+```yaml
+.train-rl-base:
+  stage: train
+  image: ${REGISTRY}/training:${CI_COMMIT_SHA}
+  tags: [kapsule, gpu]
+  variables:
+    KUBERNETES_CPU_REQUEST: "3000m"
+    KUBERNETES_CPU_LIMIT: "3800m"
+    KUBERNETES_MEMORY_REQUEST: "8Gi"
+    KUBERNETES_MEMORY_LIMIT: "16Gi"
+  # No KUBERNETES_NODE_SELECTOR_POOL → defaults to ci-compile (L4)
+```
+
+**Updated `.train-validate-base`** (supervised, explicit L40S):
+```yaml
+.train-validate-base:
+  stage: train
+  image: ${REGISTRY}/training:${CI_COMMIT_SHA}
+  tags: [kapsule, gpu]
+  variables:
+    KUBERNETES_NODE_SELECTOR_POOL: "k8s.scaleway.com/pool-name=ci-training"
+    KUBERNETES_CPU_REQUEST: "1000m"
+    KUBERNETES_CPU_LIMIT: "2000m"
+    KUBERNETES_MEMORY_REQUEST: "16Gi"
+    KUBERNETES_MEMORY_LIMIT: "40Gi"
+```
+
+### Job Routing
+
+Jobs extending `.train-rl-base`: `train-validate-dqn`, `train-validate-ppo`, `hyperopt-dqn`, `hyperopt-ppo`
+Jobs extending `.train-validate-base`: all 8 supervised train + 8 supervised hyperopt jobs
+
+### Hyperopt Epoch Reduction
+
+Reduce default hyperopt epochs from 15 → 8 for RL models (faster iteration, DQN currently ~1100s/epoch × 15 = 4.5h per trial).
+
+### Cost Impact
+
+- L4 is ~60% cheaper than L40S per hour
+- RL jobs move to L4 → immediate cost reduction
+- Supervised jobs can run concurrently on L40S → higher throughput
+- Both pools autoscale 0→1 → no idle cost when unused