47c8b783c4
Companion to Plan 5 Task 1A (multi-seed Argo DAG). Adds the post-run aggregation pipeline: - scripts/gather-multi-seed-metrics.sh: thin wrapper that pulls Argo logs for every workflow tagged foxhunt-tag=<tag>, concatenates them into /tmp/all-logs-<tag>.txt, then dispatches to the Python aggregator. - scripts/aggregate-multi-seed-metrics.py: stdlib-only HEALTH_DIAG parser. Recognises both bare and JSON-envelope-wrapped HEALTH_DIAG[<epoch>] lines, parses every <block>[<key=val> ...] segment, and emits per- (epoch, metric_name) mean / std / median / min / max across streams (where each stream is one (seed, fold) training run, attributed by pod-name prefix when present, else by epoch-rewind detection — naive epoch=0 trigger over-segments the multi-line per-epoch HEALTH_DIAG output, fixed by requiring epoch < last_epoch to start a new stream). Output JSON schema matches the plan example (top-level: tag, multi_seed, folds, warmup_end_epoch, streams_seen, aggregates; per-entry: epoch, mean, std, median, min, max, n_samples). - scripts/aggregate-norm-stats.py: stdlib-only merger for norm_stats_foldN_seedM.json files. Per-fold output collapses N seeds into mean/median/std/std_dispersion arrays consumed by the `evaluate` step's inference normaliser. - scripts/requirements.txt: declares numpy/scipy/matplotlib for downstream Plan 5 T4-T5 tier-validation/plotting scripts. The aggregators themselves are stdlib-only (statistics module). Smoke-validated on /tmp/p4t6-cleanroom-smoke.log: detects 3 streams (matching the 3 fold runs in that log), 90 unique metrics, n_samples=3 per (epoch, metric), schema matches plan. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
68 lines
2.2 KiB
Bash
Executable File
68 lines
2.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Gather per-seed HEALTH_DIAG metrics from a multi-seed Argo run and pipe
|
|
# them through aggregate-multi-seed-metrics.py for cross-seed statistics.
|
|
#
|
|
# Plan 5 Task 1B.1.
|
|
#
|
|
# Usage:
|
|
# ./scripts/gather-multi-seed-metrics.sh <tag> [output.json]
|
|
#
|
|
# Where <tag> matches the foxhunt-tag label set via `argo-train.sh --tag`.
|
|
# The script:
|
|
# 1. Pulls all logs for workflows labelled foxhunt-tag=<tag> into a
|
|
# flat text file (/tmp/all-logs-<tag>.txt).
|
|
# 2. Invokes aggregate-multi-seed-metrics.py to produce a single JSON
|
|
# report with per-(epoch, metric) mean/std/median/min/max across
|
|
# seeds.
|
|
#
|
|
# Defaults: --multi-seed and --folds are read from the MULTI_SEED / FOLDS
|
|
# environment vars (default 5 / 6). Override via env or by editing here.
|
|
set -euo pipefail
|
|
|
|
TAG="${1:?Usage: gather-multi-seed-metrics.sh <tag> [output.json]}"
|
|
OUT="${2:-/tmp/multi-seed-${TAG}.json}"
|
|
LOG_FILE="/tmp/all-logs-${TAG}.txt"
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
|
|
|
if ! command -v argo >/dev/null 2>&1; then
|
|
echo "ERROR: argo CLI not found in PATH" >&2
|
|
exit 1
|
|
fi
|
|
|
|
if ! command -v python3 >/dev/null 2>&1; then
|
|
echo "ERROR: python3 not found in PATH" >&2
|
|
exit 1
|
|
fi
|
|
|
|
echo "=== Pulling logs for workflows tagged foxhunt-tag=${TAG} ==="
|
|
# `argo logs @latest` is a single-workflow tail; for multi-workflow
|
|
# selection we list workflows by label, then concatenate their logs.
|
|
WORKFLOWS=$(argo list -n foxhunt --selector "foxhunt-tag=${TAG}" -o name 2>/dev/null || true)
|
|
if [[ -z "${WORKFLOWS}" ]]; then
|
|
echo "ERROR: no workflows found with foxhunt-tag=${TAG}" >&2
|
|
exit 1
|
|
fi
|
|
|
|
: > "${LOG_FILE}"
|
|
for wf in ${WORKFLOWS}; do
|
|
echo " -> $wf"
|
|
argo logs -n foxhunt --no-color "$wf" >> "${LOG_FILE}" 2>&1 || {
|
|
echo " WARN: failed to fetch logs for $wf" >&2
|
|
}
|
|
done
|
|
|
|
LOG_LINES=$(wc -l < "${LOG_FILE}")
|
|
echo "Collected ${LOG_LINES} log lines → ${LOG_FILE}"
|
|
|
|
echo "=== Aggregating per-(epoch, metric) statistics ==="
|
|
python3 "${REPO_ROOT}/scripts/aggregate-multi-seed-metrics.py" \
|
|
--input "${LOG_FILE}" \
|
|
--multi-seed "${MULTI_SEED:-5}" \
|
|
--folds "${FOLDS:-6}" \
|
|
--tag "${TAG}" \
|
|
--output "${OUT}"
|
|
|
|
echo "Aggregated → ${OUT}"
|