diag: enable FOXHUNT_NO_GRAPH in Argo train template — isolate H100 graph issue
Temporarily sets FOXHUNT_NO_GRAPH=1 in train-best pods to run forward+backward+Adam ungraphed. If grad_buf is non-zero ungraphed, the root cause is CUDA Graph capture of cuBLASLt on H100 SM90. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -400,6 +400,8 @@ spec:
|
||||
value: ":4096:8"
|
||||
- name: FOXHUNT_FEATURE_CACHE_DIR
|
||||
value: /feature-cache
|
||||
- name: FOXHUNT_NO_GRAPH
|
||||
value: "1"
|
||||
resources:
|
||||
requests:
|
||||
nvidia.com/gpu: "1"
|
||||
@@ -484,6 +486,8 @@ spec:
|
||||
value: ":4096:8"
|
||||
- name: FOXHUNT_FEATURE_CACHE_DIR
|
||||
value: /feature-cache
|
||||
- name: FOXHUNT_NO_GRAPH
|
||||
value: "1"
|
||||
resources:
|
||||
requests:
|
||||
nvidia.com/gpu: "1"
|
||||
|
||||
Reference in New Issue
Block a user