Two 9-quarter precompute_features runs OOM-killed on the existing
ci-compile-cpu pool (POP2-HC-32C-64G, 56Gi cgroup limit):
- train-wq8b8: exit 137 ~12s after "OFI computed" at ~57Gi
- train-2l6p4: exit 137 ~12s after "OFI computed" at ~52Gi peak
(despite the into_iter + drop(feature_vectors) +
normalize_in_place refactors landed in a27cb40a9 + 623ebfcf7,
which trimmed ~12GB of avoidable retention)
The remaining ~52-60GB peak is the irreducible working set at the
post-OFI / pre-alpha-pipeline step:
- 9-quarter MBP-10 snapshots (~10GB)
- 199M front-month trades as Mbp10Trade (~13GB)
- 17.8M bars × features + targets + OFI buffers (~14GB)
- alpha_snapshots (move-handoff from all_snapshots, ~10GB)
- feature/target Vecs (~7GB) + Rust allocator overhead
Adds a dedicated high-memory pool sized for this rare path:
- New scaleway_k8s_pool.ci_compile_cpu_hm (POP2-HM-32C-256G,
32 vCPU + 256GB RAM) with size=0 + min_size=0 autoscaling. Costs
zero when idle; autoscaler provisions one node when a pod targets
`nodeSelector: ci-compile-cpu-hm`.
- New variables: enable_ci_compile_cpu_hm_pool,
ci_compile_cpu_hm_type (default POP2-HM-32C-256G),
ci_compile_cpu_hm_max_size (default 1).
- terragrunt.hcl: enable the pool, max_size=1.
- train-template.yaml: ensure-fxcache nodeSelector pinned to
ci-compile-cpu-hm; memory limit raised 56Gi → 200Gi. The
ci-compile-cpu pool stays as the standard CI compile target for
ensure-binary + every other CPU-heavy task.
Apply with:
cd infra/live/production/kapsule
terragrunt apply -target=module.kapsule.scaleway_k8s_pool.ci_compile_cpu_hm
kubectl apply -n foxhunt -f ../../../../infra/k8s/argo/train-template.yaml
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
86de265fc8