fix: force line-buffered stdout in Argo training pod

Rust tracing with JSON subscriber in a container without TTY uses
fully-buffered stdout — logs only flush on buffer full or process exit.
This made H100 training appear stuck for 28+ minutes with no output.
stdbuf -oL forces line buffering so epoch logs appear immediately.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-04-02 16:03:30 +02:00
parent 3d53735363
commit 34df9adc8f

View File

@@ -497,7 +497,7 @@ spec:
echo "=== Training best: $MODEL ({{workflow.parameters.train-epochs}} epochs) ==="
${BINARY} \
stdbuf -oL ${BINARY} \
--model "$MODEL" \
--symbol {{workflow.parameters.symbol}} \
--tx-cost-bps {{workflow.parameters.tx-cost-bps}} \