diff --git a/docs/superpowers/specs/2026-05-18-auc-regret-controller-design.md b/docs/superpowers/specs/2026-05-18-auc-regret-controller-design.md index 54406fa0c..d034c155f 100644 --- a/docs/superpowers/specs/2026-05-18-auc-regret-controller-design.md +++ b/docs/superpowers/specs/2026-05-18-auc-regret-controller-design.md @@ -1,11 +1,37 @@ # AUC-Regret Controller Design Spec -**Status:** DESIGN — written during 3-fold validation of `004b662c8` (Option 2 input-feature fix); awaiting user review before plan handoff -**Date:** 2026-05-18 -**Branch:** `ml-alpha-phase-a` (next commit) +**Status:** ⛔ SUPERSEDED — empirically falsified by the diagnostic this spec proposed. See §S below. +**Date:** 2026-05-18 (drafted), 2026-05-18 (superseded same day) +**Branch:** `ml-alpha-phase-a` **Owner:** ml-alpha team -## 0. Context +## S. Superseded — empirical falsification (2026-05-18) + +The spec's core motivation was that `λ` saturating at the BCE-z-score ceiling **does not predict AUC gain**, citing a 50% hit-rate observed on Option-2 fold 0 (gm67g). That was a 4-saturated-event sample on a config (Option 2 / commit `004b662c8`) that subsequently proved to be a *regression* vs the actual production baseline (ISV-σ at `410ab6b0e`). + +When the same diagnostic is run on the **real ISV-σ baseline** (rxm5t/r57lx/x24d6 — 3 folds, 215 total horizon-epoch observations, logs retrieved from `argo-logs` bucket on MinIO): + +| | n | AUC up next epoch | Hit rate | Median ΔAUC | +|---|---|---|---|---| +| λ saturated (≥1.99) | 13 | 8 / 5 down | **61.5%** | **+0.00245** | +| λ not saturated | 202 | 96 / 106 down | 47.5% | −0.00120 | + +**+14 pp positive saturation→AUC-up signal.** Median ΔAUC sign is correct (positive on the boosted horizon, negative when no boost). The BCE-z-score controller is doing what it's supposed to do — boosting horizons whose AUC then improves. + +The diagnostic n=13 saturated events is small; a binomial z-test against the null hypothesis (p=0.484) gives p≈0.34 — not statistically significant at p<0.05. But the conservative reading is clear: **no evidence the controller is misaligned with AUC**. The premise of this spec ("controller is fighting AUC") is unsupported by data. + +**Decision**: do not implement. The BCE-z-score controller is retained at production. The production HEAD (`e004d6c21`) keeps the existing ISV-σ + adaptive Z_SCALE design. + +This spec is kept in tree as a historical record of an architectural hypothesis that was tested and falsified. Future controller work (if pursued) should: +1. Re-verify the diagnostic on the latest production config before redesigning. +2. Require statistically significant evidence (n ≥ 50 saturated events, p < 0.05) before committing implementation cycles. +3. Consider the AUC-regret framework only if BCE-z-score saturation→AUC correlation drops below 50%. + +The methodology of the diagnostic itself is valuable and worth preserving — see `/tmp/isv_sigma_logs/` and the saturation-hit-rate Python snippet in the conversation transcript for the analytical pattern. + +--- + +## 0. Context (original draft preserved below as record) The per-horizon λ controller in `horizon_lambda.cu` scales each horizon's trunk-grad contribution in `heads_grn_bwd` by `λ[h] ∈ [1.0, 2.0]`. Per `pearl_adam_normalizes_loss_weights`, scaling head-grad would be Adam-cancelled; scaling **trunk-grad works** because trunk parameters are shared across horizons — boosting one horizon's contribution shifts the trunk's effective update direction in that horizon's favor while Adam normalizes the aggregate.