From 16021a0d4f654d42362d47bd3e1bc1db148e4ac7 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Mon, 25 May 2026 00:04:28 +0200 Subject: [PATCH] fix(argo): cap RAYON_NUM_THREADS=8 in fxcache to prevent OOM MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 28 parallel alpha-feature chunks × ~3GB each = 84GB peak at merge. Capping at 8 threads keeps peak at ~24GB (well within 96Gi limit) while still saturating I/O. This is a one-time cost — after the cache is populated on the PVC, subsequent runs skip entirely. Co-Authored-By: Claude Opus 4.7 --- infra/k8s/argo/train-multi-seed-template.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/infra/k8s/argo/train-multi-seed-template.yaml b/infra/k8s/argo/train-multi-seed-template.yaml index 983907bb4..0642fb753 100644 --- a/infra/k8s/argo/train-multi-seed-template.yaml +++ b/infra/k8s/argo/train-multi-seed-template.yaml @@ -365,6 +365,7 @@ spec: if [ ! -x "$BINARY" ]; then echo "ERROR: precompute_features not found at $BINARY"; exit 1 fi + export RAYON_NUM_THREADS=8 if $BINARY \ --data-dir /data/futures-baseline \ --mbp10-data-dir /data/futures-baseline-mbp10 \