diag(sp15-wave5): stderr checkpoints in run_full_step step-0 ungraphed path

Browse Source

L40S smoke train-jfbzr (commit 23e9a1f78) segfaults at exit 139 in the
first run_full_step invocation, after "GPU training guard initialized
(epoch loop)" and before any HEALTH_DIAG output. Static analysis
debugger couldn't reproduce locally (test_data/futures-baseline/ lacks
.fxcache).

This commit adds eprintln! checkpoints at each phase boundary in the
ungraphed step-0 path of FusedTrainer::run_full_step (PER sample,
PopArt, counters, spectral norm, TLOB forward, forward_main, DDQN,
aux_ops, post_aux, NaN checks).

stderr is line-flushed (vs stdout buffered through tracing JSON
formatter), so the last printed checkpoint identifies the SEGV site.
Each fires once per fold's first step (graph capture absorbs subsequent
steps), so log overhead is minimal.

Diagnostic-only commit — no behavior change. Will be reverted after the
next L40S smoke localizes the SEGV and the root-cause fix lands.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

...

This commit is contained in:

jgrusewski

2026-05-07 12:21:12 +02:00

parent 23e9a1f78c

commit e9d9afbd61

2 changed files with 19 additions and 0 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

2

docs/dqn-wire-up-audit.md

File diff suppressed because one or more lines are too long