obs(ml-alpha): per-N-step train_loss log line for liveness + intra-epoch monitor
Adds a single tracing::info!("step") emitted every 500 training steps:
if epoch_train_steps % 500 == 0 {
tracing::info!(epoch, step = epoch_train_steps, loss = loss, "step");
}
Consumed by /tmp/alpha_monitor.py (v2: per-epoch trajectories + ISV +
liveness) to render intra-epoch loss trajectory and detect stalls
faster than the once-per-epoch granularity allowed.
At 8000 steps/epoch and ~36s/epoch on L40S → ~16 step lines per epoch
per fold, well below the cluster log volume budget.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -455,6 +455,19 @@ fn main() -> Result<()> {
|
||||
epoch_train_steps += 1;
|
||||
train_loss_running += loss;
|
||||
train_steps += 1;
|
||||
// Periodic per-step log (every 500 steps) for liveness +
|
||||
// intra-epoch trajectory monitoring. Cheap (~1 line/min on
|
||||
// L40S at 8000 steps/epoch / 36s = 220 steps/s ⇒ ~16 lines
|
||||
// per 500-step interval per epoch). Consumed by
|
||||
// /tmp/alpha_monitor.py.
|
||||
if epoch_train_steps % 500 == 0 {
|
||||
tracing::info!(
|
||||
epoch,
|
||||
step = epoch_train_steps,
|
||||
loss = loss,
|
||||
"step"
|
||||
);
|
||||
}
|
||||
snap_batch.clear();
|
||||
label_batch.clear();
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user