jgrusewski 47c8b783c4 plan5(task1B): metric aggregation across multi-seed runs ...

Companion to Plan 5 Task 1A (multi-seed Argo DAG). Adds the post-run
aggregation pipeline:

- scripts/gather-multi-seed-metrics.sh: thin wrapper that pulls Argo logs
  for every workflow tagged foxhunt-tag=<tag>, concatenates them into
  /tmp/all-logs-<tag>.txt, then dispatches to the Python aggregator.

- scripts/aggregate-multi-seed-metrics.py: stdlib-only HEALTH_DIAG parser.
  Recognises both bare and JSON-envelope-wrapped HEALTH_DIAG[<epoch>]
  lines, parses every <block>[<key=val> ...] segment, and emits per-
  (epoch, metric_name) mean / std / median / min / max across streams
  (where each stream is one (seed, fold) training run, attributed by
  pod-name prefix when present, else by epoch-rewind detection — naive
  epoch=0 trigger over-segments the multi-line per-epoch HEALTH_DIAG
  output, fixed by requiring epoch < last_epoch to start a new stream).
  Output JSON schema matches the plan example (top-level: tag,
  multi_seed, folds, warmup_end_epoch, streams_seen, aggregates;
  per-entry: epoch, mean, std, median, min, max, n_samples).

- scripts/aggregate-norm-stats.py: stdlib-only merger for
  norm_stats_foldN_seedM.json files. Per-fold output collapses N seeds
  into mean/median/std/std_dispersion arrays consumed by the
  `evaluate` step's inference normaliser.

- scripts/requirements.txt: declares numpy/scipy/matplotlib for downstream
  Plan 5 T4-T5 tier-validation/plotting scripts. The aggregators
  themselves are stdlib-only (statistics module).

Smoke-validated on /tmp/p4t6-cleanroom-smoke.log: detects 3 streams
(matching the 3 fold runs in that log), 90 unique metrics, n_samples=3
per (epoch, metric), schema matches plan.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-26 10:58:42 +02:00

494 B

Raw Blame History

View Raw