gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(infra): increase Tempo memory to 1Gi, RL runner CPU limits to 8000m

Browse Source 
Tempo was OOMKilled every ~30 min at 512Mi, causing web-gateway OTLP
export errors. Increased to 1Gi.

RL runner had cpu_request_overwrite_max_allowed=4000m but hyperopt RL
jobs need 6000m for parallel PSO trials. Increased to 8000m.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-03-01 13:04:26 +01:00
parent cfdd878fff
commit a36fce0b87
2 changed files with 7 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
infra/k8s/gitlab/runner-rl-values.yaml
View File

@@ -34,10 +34,11 @@ runners:
image = "rust:1.89-slim"
privileged = false
node_selector_overwrite_allowed = ".*"
cpu_request_overwrite_max_allowed = "4000m"
cpu_limit_overwrite_max_allowed = "4000m"
memory_request_overwrite_max_allowed = "16Gi"
memory_limit_overwrite_max_allowed = "16Gi"
# Hyperopt RL jobs need 6000m CPU + 20Gi mem (PSO parallel trials on L4 8 vCPU)
cpu_request_overwrite_max_allowed = "8000m"
cpu_limit_overwrite_max_allowed = "8000m"
memory_request_overwrite_max_allowed = "24Gi"
memory_limit_overwrite_max_allowed = "24Gi"
poll_timeout = 600
# Allow CI jobs to override runtimeClassName via KUBERNETES_RUNTIME_CLASS_NAME
runtime_class_name_overwrite_allowed = ".*"

4
infra/k8s/monitoring/tempo.yaml
View File

@@ -77,10 +77,10 @@ spec:
resources:
requests:
cpu: 100m
memory: 128Mi
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
memory: 1Gi
readinessProbe:
httpGet:
path: /ready