foxhunt/docs/plans/2026-02-27-gpu-resource-optimization-implementation.md

# GPU Resource Optimization Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Route RL training (DQN, PPO) to cheaper L4 GPUs and supervised models to L40S, with tuned CPU/memory per workload.

**Architecture:** Add a `.train-rl-base` CI template (no node selector override → runner default ci-compile/L4). Update `.train-validate-base` CPU down for GPU-bound supervised models. Switch 3 RL jobs to new template. Reduce RL hyperopt epochs 15→8.

**Tech Stack:** GitLab CI YAML, Kubernetes node selectors via runner config overwrite variables

---

### Task 1: Add `.train-rl-base` CI template

**Files:**
- Modify: `.gitlab-ci.yml:413` (insert new template BEFORE existing `.train-validate-base`)

**Step 1: Insert `.train-rl-base` template above `.train-validate-base`**

Find the comment block starting at line 410 and insert the new template before it. The new template goes right before the `# Runs the training binary...` comment block:

```yaml
# --------------------------------------------------------------------------
# RL training base: DQN + PPO → ci-compile pool (L4)
# RL models are CPU-bound (environment simulation), use <1GB VRAM.
# No KUBERNETES_NODE_SELECTOR_POOL → runner default = ci-compile (L4-1-24G)
# --------------------------------------------------------------------------
.train-rl-base:
  stage: train
  image: ${REGISTRY}/training:${CI_COMMIT_SHA}
  tags:
    - kapsule
    - gpu
  variables:
    # RL is CPU-bound — give it most of the L4's 7800m allocatable CPU.
    # 3000m request allows 2 concurrent RL jobs on one L4 node.
    KUBERNETES_CPU_REQUEST: "3000m"
    KUBERNETES_CPU_LIMIT: "3800m"
    KUBERNETES_MEMORY_REQUEST: "8Gi"
    KUBERNETES_MEMORY_LIMIT: "16Gi"
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      when: manual
      allow_failure: true
    - if: $CI_PIPELINE_SOURCE == "schedule" && $TRAIN_VALIDATE == "true"
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"
      when: manual
      allow_failure: true
  before_script:
    - export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')
    - nvidia-smi
    - mkdir -p ${CI_PROJECT_DIR}/output

```

Insert this BEFORE the existing line 410 (`# Runs the training binary for a few steps...`).

**Step 2: Update `.train-validate-base` comment and CPU**

In the existing `.train-validate-base` (now shifted down), update:
- Comment: change to clarify it's for supervised models on L40S
- CPU request: `"2000m"` → `"1000m"`
- CPU limit: `"3500m"` → `"2000m"`

The updated `.train-validate-base` should read:

```yaml
# --------------------------------------------------------------------------
# Supervised training base: TFT, Mamba2, TGGN, TLOB, Liquid, KAN, xLSTM, Diffusion → L40S
# Supervised models are GPU-bound — low CPU, high VRAM. Routes to ci-training (L40S-1-48G).
# --------------------------------------------------------------------------
.train-validate-base:
  stage: train
  image: ${REGISTRY}/training:${CI_COMMIT_SHA}
  tags:
    - kapsule
    - gpu
  variables:
    # Route to L40S pool (48GB VRAM for large supervised models)
    KUBERNETES_NODE_SELECTOR_POOL: "k8s.scaleway.com/pool-name=ci-training"
    # Supervised is GPU-bound — minimal CPU. Low request allows 3-4 concurrent jobs.
    KUBERNETES_CPU_REQUEST: "1000m"
    KUBERNETES_CPU_LIMIT: "2000m"
    KUBERNETES_MEMORY_REQUEST: "16Gi"
    KUBERNETES_MEMORY_LIMIT: "40Gi"
```

**Step 3: Commit**

```bash
git add .gitlab-ci.yml
git commit -m "feat(ci): add .train-rl-base template for L4 routing, tune supervised CPU"
```

---

### Task 2: Switch RL jobs to `.train-rl-base`

**Files:**
- Modify: `.gitlab-ci.yml` — 3 jobs: `train-validate-rl`, `hyperopt-ppo`, `hyperopt-dqn`

**Step 1: Change `train-validate-rl` extends**

Find `train-validate-rl:` and change:
```yaml
# BEFORE:
train-validate-rl:
  extends: .train-validate-base

# AFTER:
train-validate-rl:
  extends: .train-rl-base
```

**Step 2: Change `hyperopt-ppo` extends**

Find `hyperopt-ppo:` and change:
```yaml
# BEFORE:
hyperopt-ppo:
  extends: .train-validate-base

# AFTER:
hyperopt-ppo:
  extends: .train-rl-base
```

**Step 3: Change `hyperopt-dqn` extends**

Find `hyperopt-dqn:` and change:
```yaml
# BEFORE:
hyperopt-dqn:
  extends: .train-validate-base

# AFTER:
hyperopt-dqn:
  extends: .train-rl-base
```

**Step 4: Commit**

```bash
git add .gitlab-ci.yml
git commit -m "feat(ci): route RL jobs (DQN, PPO) to L4 via .train-rl-base"
```

---

### Task 3: Reduce RL hyperopt epochs 15 → 8

**Files:**
- Modify: `.gitlab-ci.yml` — `hyperopt-ppo` and `hyperopt-dqn` scripts

**Step 1: Change `hyperopt-ppo` epochs**

In the `hyperopt-ppo` script section, find `--epochs 15` and change to `--epochs 8`.

**Step 2: Change `hyperopt-dqn` epochs**

In the `hyperopt-dqn` script section, find `--epochs 15` and change to `--epochs 8`.

**Step 3: Commit**

```bash
git add .gitlab-ci.yml
git commit -m "feat(ci): reduce RL hyperopt epochs 15→8 for faster iteration"
```

---

### Task 4: Validate pipeline and push

**Step 1: Lint the YAML**

```bash
python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" && echo "YAML valid"
```

Expected: `YAML valid`

**Step 2: Verify job routing by checking extends**

```bash
grep -E '(train-validate-rl|hyperopt-ppo|hyperopt-dqn):' -A1 .gitlab-ci.yml | grep extends
```

Expected: All 3 show `extends: .train-rl-base`

```bash
grep -E '(train-validate-tft|train-validate-mamba2|hyperopt-tft|hyperopt-mamba2):' -A1 .gitlab-ci.yml | grep extends
```

Expected: All show `extends: .train-validate-base`

**Step 3: Verify epochs**

```bash
grep -B5 'epochs 8' .gitlab-ci.yml | grep -E '(hyperopt-ppo|hyperopt-dqn)'
```

Expected: Both RL hyperopt jobs show epochs 8.

```bash
grep -B5 'epochs 15' .gitlab-ci.yml | head -20
```

Expected: Only supervised hyperopt jobs (tft, mamba2, liquid, tggn, tlob, kan, xlstm, diffusion) still have `--epochs 15`.

**Step 4: Push and verify**

```bash
git push origin main
```

Trigger a pipeline via Rails console and verify:
- RL jobs (train-validate-rl, hyperopt-ppo, hyperopt-dqn) schedule on ci-compile (L4) node
- Supervised jobs schedule on ci-training (L40S) node
- RL hyperopt shows `--epochs 8` in log output