obs(ml-alpha): per-N-step train_loss log line for liveness + intra-epoch monitor

Adds a single tracing::info!("step") emitted every 500 training steps:

  if epoch_train_steps % 500 == 0 {
      tracing::info!(epoch, step = epoch_train_steps, loss = loss, "step");
  }

Consumed by /tmp/alpha_monitor.py (v2: per-epoch trajectories + ISV +
liveness) to render intra-epoch loss trajectory and detect stalls
faster than the once-per-epoch granularity allowed.

At 8000 steps/epoch and ~36s/epoch on L40S → ~16 step lines per epoch
per fold, well below the cluster log volume budget.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-18 22:33:44 +02:00
parent 008f65d894
commit 004b662c80

View File

@@ -455,6 +455,19 @@ fn main() -> Result<()> {
epoch_train_steps += 1;
train_loss_running += loss;
train_steps += 1;
// Periodic per-step log (every 500 steps) for liveness +
// intra-epoch trajectory monitoring. Cheap (~1 line/min on
// L40S at 8000 steps/epoch / 36s = 220 steps/s ⇒ ~16 lines
// per 500-step interval per epoch). Consumed by
// /tmp/alpha_monitor.py.
if epoch_train_steps % 500 == 0 {
tracing::info!(
epoch,
step = epoch_train_steps,
loss = loss,
"step"
);
}
snap_batch.clear();
label_batch.clear();
}