fix: force line-buffered stdout in Argo training pod
Rust tracing with JSON subscriber in a container without TTY uses fully-buffered stdout — logs only flush on buffer full or process exit. This made H100 training appear stuck for 28+ minutes with no output. stdbuf -oL forces line buffering so epoch logs appear immediately. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -497,7 +497,7 @@ spec:
|
||||
|
||||
echo "=== Training best: $MODEL ({{workflow.parameters.train-epochs}} epochs) ==="
|
||||
|
||||
${BINARY} \
|
||||
stdbuf -oL ${BINARY} \
|
||||
--model "$MODEL" \
|
||||
--symbol {{workflow.parameters.symbol}} \
|
||||
--tx-cost-bps {{workflow.parameters.tx-cost-bps}} \
|
||||
|
||||
Reference in New Issue
Block a user