diag: enable FOXHUNT_NO_GRAPH in Argo train template — isolate H100 graph issue

Temporarily sets FOXHUNT_NO_GRAPH=1 in train-best pods to run
forward+backward+Adam ungraphed. If grad_buf is non-zero ungraphed,
the root cause is CUDA Graph capture of cuBLASLt on H100 SM90.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-04-13 23:44:10 +02:00
parent 407b3b04ff
commit d6ada5266b

View File

@@ -400,6 +400,8 @@ spec:
value: ":4096:8"
- name: FOXHUNT_FEATURE_CACHE_DIR
value: /feature-cache
- name: FOXHUNT_NO_GRAPH
value: "1"
resources:
requests:
nvidia.com/gpu: "1"
@@ -484,6 +486,8 @@ spec:
value: ":4096:8"
- name: FOXHUNT_FEATURE_CACHE_DIR
value: /feature-cache
- name: FOXHUNT_NO_GRAPH
value: "1"
resources:
requests:
nvidia.com/gpu: "1"