gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

spec(ml-alpha): mark AUC-regret controller SUPERSEDED — empirically falsified

Browse Source 
The motivating "50% saturation hit-rate" observation came from gm67g
fold 0 (Option-2 config, commit 004b662c8 — itself a -0.003 mean_auc
regression vs ISV-σ at 410ab6b0e). Re-running the same diagnostic on
the actual production baseline (ISV-σ 3 folds: rxm5t/r57lx/x24d6,
logs retrieved from MinIO argo-logs bucket) on 215 horizon-epoch
observations:

  saturated λ→AUC up:     8/13 = 61.5%   median Δauc = +0.0025
  non-saturated→AUC up:  96/202 = 47.5%   median Δauc = -0.0012
  difference:            +14pp in favor of the controller working

The BCE-z-score controller is empirically correlated with the
optimization target. No evidence it's misaligned with AUC. Spec
premise falsified.

Spec retained in tree as historical record. The diagnostic
methodology itself (saturation→Δauc analysis on archived MinIO logs)
is the durable artifact and is documented in the supersede block.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-18 23:24:01 +02:00
parent e004d6c217
commit 6b7920474d
1 changed files with 30 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
docs/superpowers/specs/2026-05-18-auc-regret-controller-design.md
View File

@@ -1,11 +1,37 @@
# AUC-Regret Controller Design Spec
**Status:** DESIGN — written during 3-fold validation of `004b662c8` (Option 2 input-feature fix); awaiting user review before plan handoff
**Date:** 2026-05-18
**Branch:** `ml-alpha-phase-a` (next commit)
**Status:** ⛔ SUPERSEDED — empirically falsified by the diagnostic this spec proposed. See §S below.
**Date:** 2026-05-18 (drafted), 2026-05-18 (superseded same day)
**Branch:** `ml-alpha-phase-a`
**Owner:** ml-alpha team
## 0. Context
## S. Superseded — empirical falsification (2026-05-18)
The spec's core motivation was that `λ` saturating at the BCE-z-score ceiling **does not predict AUC gain**, citing a 50% hit-rate observed on Option-2 fold 0 (gm67g). That was a 4-saturated-event sample on a config (Option 2 / commit `004b662c8`) that subsequently proved to be a *regression* vs the actual production baseline (ISV-σ at `410ab6b0e`).
When the same diagnostic is run on the **real ISV-σ baseline** (rxm5t/r57lx/x24d6 — 3 folds, 215 total horizon-epoch observations, logs retrieved from `argo-logs` bucket on MinIO):
| | n | AUC up next epoch | Hit rate | Median ΔAUC |
|---|---|---|---|---|
| λ saturated (≥1.99) | 13 | 8 / 5 down | **61.5%** | **+0.00245** |
| λ not saturated | 202 | 96 / 106 down | 47.5% | −0.00120 |
**+14 pp positive saturation→AUC-up signal.** Median ΔAUC sign is correct (positive on the boosted horizon, negative when no boost). The BCE-z-score controller is doing what it's supposed to do — boosting horizons whose AUC then improves.
The diagnostic n=13 saturated events is small; a binomial z-test against the null hypothesis (p=0.484) gives p≈0.34 — not statistically significant at p<0.05. But the conservative reading is clear: **no evidence the controller is misaligned with AUC**. The premise of this spec ("controller is fighting AUC") is unsupported by data.
**Decision**: do not implement. The BCE-z-score controller is retained at production. The production HEAD (`e004d6c21`) keeps the existing ISV-σ + adaptive Z_SCALE design.
This spec is kept in tree as a historical record of an architectural hypothesis that was tested and falsified. Future controller work (if pursued) should:
1. Re-verify the diagnostic on the latest production config before redesigning.
2. Require statistically significant evidence (n ≥ 50 saturated events, p < 0.05) before committing implementation cycles.
3. Consider the AUC-regret framework only if BCE-z-score saturation→AUC correlation drops below 50%.
The methodology of the diagnostic itself is valuable and worth preserving — see `/tmp/isv_sigma_logs/` and the saturation-hit-rate Python snippet in the conversation transcript for the analytical pattern.
---
## 0. Context (original draft preserved below as record)
The per-horizon λ controller in `horizon_lambda.cu` scales each horizon's trunk-grad contribution in `heads_grn_bwd` by `λ[h] ∈ [1.0, 2.0]`. Per `pearl_adam_normalizes_loss_weights`, scaling head-grad would be Adam-cancelled; scaling **trunk-grad works** because trunk parameters are shared across horizons — boosting one horizon's contribution shifts the trunk's effective update direction in that horizon's favor while Adam normalizes the aggregate.