47c8b783c4
Companion to Plan 5 Task 1A (multi-seed Argo DAG). Adds the post-run aggregation pipeline: - scripts/gather-multi-seed-metrics.sh: thin wrapper that pulls Argo logs for every workflow tagged foxhunt-tag=<tag>, concatenates them into /tmp/all-logs-<tag>.txt, then dispatches to the Python aggregator. - scripts/aggregate-multi-seed-metrics.py: stdlib-only HEALTH_DIAG parser. Recognises both bare and JSON-envelope-wrapped HEALTH_DIAG[<epoch>] lines, parses every <block>[<key=val> ...] segment, and emits per- (epoch, metric_name) mean / std / median / min / max across streams (where each stream is one (seed, fold) training run, attributed by pod-name prefix when present, else by epoch-rewind detection — naive epoch=0 trigger over-segments the multi-line per-epoch HEALTH_DIAG output, fixed by requiring epoch < last_epoch to start a new stream). Output JSON schema matches the plan example (top-level: tag, multi_seed, folds, warmup_end_epoch, streams_seen, aggregates; per-entry: epoch, mean, std, median, min, max, n_samples). - scripts/aggregate-norm-stats.py: stdlib-only merger for norm_stats_foldN_seedM.json files. Per-fold output collapses N seeds into mean/median/std/std_dispersion arrays consumed by the `evaluate` step's inference normaliser. - scripts/requirements.txt: declares numpy/scipy/matplotlib for downstream Plan 5 T4-T5 tier-validation/plotting scripts. The aggregators themselves are stdlib-only (statistics module). Smoke-validated on /tmp/p4t6-cleanroom-smoke.log: detects 3 streams (matching the 3 fold runs in that log), 90 unique metrics, n_samples=3 per (epoch, metric), schema matches plan. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
14 lines
494 B
Plaintext
14 lines
494 B
Plaintext
# Python dependencies for scripts/ utilities.
|
||
#
|
||
# Used by:
|
||
# - aggregate-multi-seed-metrics.py (Plan 5 Task 1B.2)
|
||
# - aggregate-norm-stats.py (Plan 5 Task 1B.3)
|
||
#
|
||
# numpy / scipy: not currently imported by the aggregators (they use
|
||
# stdlib `statistics`), but reserved for downstream metric-band plotting
|
||
# and tier-validation scripts (Plan 5 Tasks 4–5) which use scipy.stats
|
||
# for confidence intervals and matplotlib for visualisations.
|
||
numpy>=1.26
|
||
scipy>=1.11
|
||
matplotlib>=3.8
|