ci: increase compile-training pod timeout to 20 min

When all 8 per-service compile jobs fire simultaneously (e.g. on
Cargo.toml change), the single POP2-32C-128G node is fully loaded.
Training compile pod can't schedule until service jobs finish.
KUBERNETES_POLL_TIMEOUT=1200 gives it time to wait.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-03-04 22:47:29 +01:00
parent 9743545794
commit d537ee87e5

View File

@@ -599,6 +599,9 @@ compile-training:
# CUDA stubs from ci-builder image — compiles CUDA kernels without GPU hardware.
# L4 GPU = compute capability 8.9 → candle-kernels generates sm_89 PTX
CUDA_COMPUTE_CAP: "89"
# When all 8 service jobs fire (Cargo.toml change), the node is fully loaded.
# Increase pod scheduling timeout so training waits for service jobs to finish.
KUBERNETES_POLL_TIMEOUT: "1200"
rules:
- if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
changes: