From 1fc100ae74722cf424563c79db334e08bbdf283b Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Mon, 18 May 2026 23:06:29 +0200 Subject: [PATCH] =?UTF-8?q?spec(ml-alpha):=20AUC-regret=20controller=20?= =?UTF-8?q?=E2=80=94=20replace=20BCE-z-score=20signal=20with=20per-horizon?= =?UTF-8?q?-regret?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The current λ controller boosts horizons by BCE-EMA z-score, which conflates three distinct causes of high BCE — only one of which (under-trained horizon) benefits from boosting. The other two (intrinsically harder horizon, calibration drift) are unaffected by gradient-magnitude lifts. Empirical motivation (gm67g fold-0, this branch's 3-fold run): when λ_h6000 saturated at 2.0, h6000's next-epoch AUC went UP 2/4 times and DOWN 2/4 times. 50% hit rate ⇒ the BCE saturation signal is misaligned with the optimization objective. This spec replaces BCE-z-score with AUC-regret: best_auc[h] = running max of per-horizon validation AUC regret[h] = max(0, best_auc[h] - current_auc[h]) regret_max_ema = EMA of max_h regret[h] λ[h] = clamp(1.0, 2.0, 1 + regret[h] / regret_max_ema) Properties: - Aligned with the objective (AUC, not BCE) - Naturally bounded (regret ∈ [0, 1]) - "At personal best" → λ=1.0 (no wasted boost) - Auto-saturation by design (max-regret horizon → ceiling) - Cold-start clean (e0: best=current, regret=0, uniform λ) - Zero hardcoded magic beyond bootstrap epsilons Implementation surface ~250 LOC: - Split horizon_lambda kernel: horizon_loss_ema (per-step) + horizon_lambda (per-epoch, AUC-regret math) - Trainer state: drop z_max_ema, add best_auc + regret_max_ema - Per-epoch entry point: trainer.update_lambda_from_auc() - Extend isv snapshot log line with best_auc_h* + regret_h* Test plan: - Local 9/9 perception_overfit - New synthetic-AUC unit test (controller correctness) - Cluster 5-epoch smoke + 3-fold A/B vs Option-2 baseline (004b662c8) - Success: mean_auc lifts ≥ +0.005 AND median sat→next-AUC delta positive Awaiting user review before plan handoff. Co-Authored-By: Claude Opus 4.7 --- ...2026-05-18-auc-regret-controller-design.md | 293 ++++++++++++++++++ 1 file changed, 293 insertions(+) create mode 100644 docs/superpowers/specs/2026-05-18-auc-regret-controller-design.md diff --git a/docs/superpowers/specs/2026-05-18-auc-regret-controller-design.md b/docs/superpowers/specs/2026-05-18-auc-regret-controller-design.md new file mode 100644 index 000000000..54406fa0c --- /dev/null +++ b/docs/superpowers/specs/2026-05-18-auc-regret-controller-design.md @@ -0,0 +1,293 @@ +# AUC-Regret Controller Design Spec + +**Status:** DESIGN — written during 3-fold validation of `004b662c8` (Option 2 input-feature fix); awaiting user review before plan handoff +**Date:** 2026-05-18 +**Branch:** `ml-alpha-phase-a` (next commit) +**Owner:** ml-alpha team + +## 0. Context + +The per-horizon λ controller in `horizon_lambda.cu` scales each horizon's trunk-grad contribution in `heads_grn_bwd` by `λ[h] ∈ [1.0, 2.0]`. Per `pearl_adam_normalizes_loss_weights`, scaling head-grad would be Adam-cancelled; scaling **trunk-grad works** because trunk parameters are shared across horizons — boosting one horizon's contribution shifts the trunk's effective update direction in that horizon's favor while Adam normalizes the aggregate. + +The current controller computes `λ[h]` from a **z-score of `loss_ema[h]`** (per-horizon EMA of unweighted BCE): + +``` +z[h] = (loss_ema[h] - mean) / std +Z_SCALE_ISV = 1.0 / max(z_max_ema, 0.1) +λ[h] = clamp(1.0, 2.0, 1.0 + Z_SCALE_ISV * z[h]) +``` + +**Empirical problem (gm67g fold-0, 20 epochs):** when `λ_h6000` saturated at 2.0 (4 of 20 epochs), h6000 AUC went UP on the next epoch in 2/4 cases and DOWN in 2/4. Saturation does not predict AUC gain — the BCE signal is misaligned with the optimization objective. + +**Root cause:** BCE measures distance from labels, not predictability. A horizon can have high BCE for three reasons, only one of which benefits from boosting: + +| Cause | Boost helps? | +|---|---| +| Genuinely under-trained — encoder hasn't found the signal yet | **Yes** | +| Intrinsically harder horizon (e.g., h6000 has more inherent price-walk uncertainty) | **No** — BCE floor is structural | +| Calibration drift (probs miscalibrated but ranking fine — `pearl_auc_accuracy_gap_means_miscalibration`) | **No** — AUC unchanged by ranking-preserving recalibration | + +For our 5 horizons {30, 100, 300, 1000, 6000}, cause #2 dominates h6000. The current controller almost always identifies h6000 as the "worst" horizon and boosts it — but the BCE floor is structural, so the boost spends encoder capacity on a horizon whose loss can't get materially lower. + +**This spec replaces BCE-z-score with AUC-regret as the controller signal.** The new controller boosts horizons that have fallen below their *personal-best* AUC — i.e., horizons that demonstrably *were* learnable and now aren't. + +Out of scope (separate work): +- Changing the consumption point (still `heads_grn_bwd` per-horizon trunk-grad scaling). +- Changing the λ envelope `[1.0, 2.0]`. +- Adding a meta-learner over λ. Per `pearl_controller_anchors_isv_driven`, the controller stays signal-driven. +- Trainable controllers. The whole point is principled signal-driven adaptation. + +## 1. Goal + +Per-epoch validation AUC drives λ. A horizon that's at its personal-best AUC contributes uniform grad (λ=1). A horizon that's regressed from its personal-best AUC contributes boosted grad (λ up to 2). The historical-max regret across horizons maps exactly to the ceiling (Z_SCALE_ISV self-calibrating). + +**Falsifiable success criterion:** at this commit, on the same `seed=16962` 3-fold CV at 30 epochs as the Option-2 baseline, **mean_auc lifts by ≥ +0.005** (one full noise σ from prior runs) AND **the median saturation→next-epoch-AUC delta is positive** (i.e., when the controller boosts, the boosted horizon actually improves more often than not). + +## 2. Math + +### State (carried across training steps) + +| Variable | Shape | Description | +|---|---|---| +| `best_auc[h]` | [5] | Running max of per-horizon validation AUC since training start | +| `regret_max_ema` | [1] | EMA of `max_h regret[h]` across epochs (the adaptive scale anchor) | + +Both initialised to 0 (sentinel — first-observation bootstrap per `pearl_first_observation_bootstrap`). + +### Per-epoch update (after validation completes) + +Input: `current_auc[5]` from the validation pass (already computed for the `validation` log line). + +```text +# 1. Update best_auc (running max — strict, no decay). +for h in 0..5: + best_auc[h] = max(best_auc[h], current_auc[h]) + +# 2. Compute regret. ≥ 0 by construction. +regret[h] = max(0.0, best_auc[h] - current_auc[h]) + +# 3. Update regret_max_ema (first-observation bootstrap, fixed α=0.1). +r_max = max_h regret[h] +if regret_max_ema <= 0: + regret_max_ema = r_max # sentinel bootstrap +else: + regret_max_ema = (1-α) * regret_max_ema + α * r_max + +# 4. Adaptive scale — by design, observed-max-regret maps to ceiling. +Z_SCALE_ISV = (LAMBDA_CEILING - LAMBDA_FLOOR) / max(regret_max_ema, REGRET_FLOOR) + = 1.0 / max(regret_max_ema, 0.001) + +# 5. Per-horizon λ. +for h in 0..5: + raw = 1.0 + Z_SCALE_ISV * regret[h] + λ[h] = clamp(LAMBDA_FLOOR=1.0, LAMBDA_CEILING=2.0, raw) +``` + +### Properties + +- **At personal best**: `regret[h]=0` → `λ[h]=1.0`. Uniform encoder grad. +- **Max regret**: `regret[h]=regret_max_ema` → `λ[h]=2.0`. Full boost. +- **No regret anywhere**: `r_max=0` → `regret_max_ema` floor at `0.001` keeps `Z_SCALE_ISV` finite. All `regret[h]=0` → all `λ[h]=1.0`. Controller dormant. +- **Cold start (e0)**: `best_auc` is 0 → after first validation, `best_auc[h] = current_auc[h]` for all h → `regret[h]=0` → uniform λ. Controller engages only at e1+. +- **Persistent regression**: if a horizon stays below its best for many epochs, regret stays high, λ stays boosted. The controller doesn't time out. +- **No hardcoded magic numbers** beyond bootstrap epsilons (`REGRET_FLOOR=0.001`, EMA `α=0.1` matching the existing loss-EMA cadence) and the clamp envelope (kept at `[1.0, 2.0]` for direct comparability with the current controller). + +## 3. Removed: BCE-z-score signal + +The existing kernel computes `loss_ema[h]` from `loss_per_horizon` emitted by the BCE kernel, z-scores across horizons, EMA-tracks `z_max`, and produces λ. + +**This pipeline is fully replaced.** Specifically: + +- `loss_ema[5]` state → **REMOVED** from the controller. (Still emitted by the BCE kernel for telemetry, but no longer consumed.) +- `z_max_ema[1]` state → **REMOVED**. +- `log_sigma_h[5]` output → **KEPT as-is**. The closed-form Kendall σ from loss_ema is preserved for telemetry (BCE kernel ignores it per the σ-sidecar commit `17eb82511`, so it's a pure observability output). σ still uses `loss_ema` internally; that loss_ema computation moves to a dedicated `horizon_loss_ema` kernel. + +After this commit: +- `horizon_loss_ema.cu` (new, tiny): maintains `loss_ema[5]` and `log_sigma_h[5]`. Called from the trainer at the same point in the step as today. +- `horizon_lambda.cu` (rewritten): consumes `current_auc[5]` + maintains `best_auc[5]` + `regret_max_ema[1]`, emits `lambda[5]`. Called from the trainer **once per epoch** (after validation), not per step. + +The per-step call frequency change is significant. See §5 for trainer wiring. + +## 4. Inputs / outputs + +### `horizon_loss_ema` kernel (new, split off from current) + +```cuda +extern "C" __global__ void horizon_loss_ema( + const float* __restrict__ loss_per_horizon, // [5] from BCE kernel + float* __restrict__ loss_ema, // [5] state read+write + float* __restrict__ log_sigma_h // [5] output (sidecar) +); +``` + +Single thread, single block. Updates `loss_ema[h]` with sentinel bootstrap + fixed α=0.1; computes `log_sigma_h[h] = max(log(0.5), 0.5 * log(loss_ema[h] + ε))` (preserves the existing σ-sidecar behavior). Called per-step from the trainer hot loop (same place `horizon_ema_and_lambda` is called today). + +### `horizon_lambda` kernel (rewritten) + +```cuda +extern "C" __global__ void horizon_lambda( + const float* __restrict__ current_auc, // [5] from this epoch's validation + float* __restrict__ best_auc, // [5] state read+write + float* __restrict__ regret_max_ema, // [1] state read+write + float* __restrict__ lambda // [5] output +); +``` + +Single thread, single block. Implements the math in §2. Called **once per epoch** from `alpha_train.rs` after validation completes. + +The trainer passes `current_auc` from the existing per-horizon AUC computation (the same values that populate the `auc_h30..auc_h6000` keys on the `validation` log line and `best_val_auc` in the summary). No new computation needed. + +## 5. Trainer wiring (`crates/ml-alpha/src/trainer/perception.rs` + `alpha_train.rs`) + +### New `PerceptionTrainer` state + +```rust +// AUC-regret controller state. +best_auc_d: CudaSlice, // [5], zero-init (sentinel) +regret_max_ema_d: CudaSlice, // [1], zero-init (sentinel) +horizon_lambda_fn: CudaFunction, // existing handle, same name +// New: +horizon_loss_ema_fn: CudaFunction, // split off from horizon_lambda +``` + +### Removed state + +```rust +z_max_ema_d: CudaSlice, // [1] — DROPPED (was used by BCE z-score controller) +``` + +`loss_ema_d` and `log_sigma_h_d` remain (both still used by the new `horizon_loss_ema` kernel + BCE kernel's σ-sidecar signature stability). + +### Per-step trainer call (in K-loop body) + +```rust +// Was: launch horizon_ema_and_lambda(loss_per_h, loss_ema, z_max_ema, lambda, log_sigma_h) +// Now: launch horizon_loss_ema(loss_per_h, loss_ema, log_sigma_h) +self.stream.launch_builder(&self.horizon_loss_ema_fn) + .arg(&self.loss_per_horizon_d) + .arg(&mut self.loss_ema_d) + .arg(&mut self.log_sigma_h_d) + .launch(cfg_single)?; +``` + +λ does NOT update per-step anymore. It uses the value computed at the last epoch boundary. + +### Per-epoch trainer call (new entry point, called from `alpha_train.rs`) + +```rust +impl PerceptionTrainer { + pub fn update_lambda_from_auc(&mut self, current_auc: &[f32; N_HORIZONS]) -> Result<()> { + // Upload current_auc to device, launch horizon_lambda kernel, + // λ buffer is now populated for the next training epoch's K-loop. + } +} +``` + +In `alpha_train.rs` after computing `per_horizon_auc` for the validation pass: + +```rust +trainer.update_lambda_from_auc(&per_horizon_auc) + .context("update lambda from validation AUC")?; +``` + +### Cold-start handling + +At e0, before validation runs, `best_auc_d = [0; 5]` and `regret_max_ema_d = [0]`. The trainer hot loop's first epoch reads `lambda` which is also zero-initialised → all-zero λ → kernel sentinel logic (`l <= 0 ? 1.0 : l`) in `grn_bwd` already converts to uniform λ=1.0 per `s_lambda` cooperative-load logic. **No special-case code needed in the trainer**; the existing sentinel handles e0 transparently. + +After validation at end of e0, `update_lambda_from_auc` runs with `current_auc[h] = first-observation`. Since `best_auc[h] = 0` initially, `best_auc[h] := max(0, current_auc[h]) = current_auc[h]` → regret=0 everywhere → λ=1.0. Same uniform behavior as the sentinel, just emitted from the kernel. + +E1 onwards, regret can become non-zero whenever a horizon dips below its observed maximum. + +## 6. Telemetry / observability + +The existing `isv snapshot` log line at validation end currently logs `ema_h*` (loss_ema) and `lam_h*` (the λ values). **Add `best_auc_h*` and `regret_h*` to that log line** so the per-epoch trajectory in `/tmp/alpha_status.py wf ` shows the full controller state. + +```rust +tracing::info!( + epoch, + // existing: + ema_h30 = ema[0], /* ... */ ema_h6000 = ema[4], + lam_h30 = lam[0], /* ... */ lam_h6000 = lam[4], + // new (AUC-regret state): + best_auc_h30 = best_auc[0], /* ... */ best_auc_h6000 = best_auc[4], + regret_h30 = regret[0], /* ... */ regret_h6000 = regret[4], + "isv snapshot" +); +``` + +The monitor's `parse_train_log` already handles unknown extra fields gracefully (regex only captures explicit captures); add a parse pass for these new fields. + +## 7. Failure modes & mitigations + +1. **All horizons at personal best forever** — `regret=0` everywhere, λ=1.0 uniform forever. The controller is dormant. *This is the desired behavior* — if the model never regresses on any horizon, the controller has nothing useful to do. The encoder learns uniformly. + +2. **Best-AUC drift on noisy validation** — if a validation seed quirk produces an outlier-high AUC on some horizon at e3, `best_auc[h]` pins to that outlier, and every subsequent epoch shows positive regret. Boost stays on indefinitely. + + **Mitigation**: validation uses a deterministic seed (`val_loader` seed = `train_seed + 0xC0FFEE`); same validation data every epoch. So outliers are not from sampling noise — they're real model behavior on that fixed eval set. If the model genuinely degraded, persistent regret is the correct signal. + + Open question: should `best_auc` decay with time? Possible failure: model finds a high-AUC local optimum at e3 then diverges. With strict-max best_auc, the controller will keep trying to drag it back; with EMA-smoothed best_auc, it would let go. **Spec choice: strict max (no decay)**. Rationale: the strict signal is more interpretable; if the model genuinely diverges, early-stop fires (patience=5 on mean_auc) and we end the run anyway. + +3. **Saturation rate too high / too low** — if `regret_max_ema` is too small (most regrets near 0), `Z_SCALE_ISV` is large and even tiny regrets saturate λ. The `REGRET_FLOOR=0.001` clamps this; effectively the smallest regret that produces λ=2.0 saturation is ≥ 0.001 in absolute AUC. That's ~0.1 percentage points — reasonable. If you think it's too sensitive, raise the floor. + +4. **K-loop reads λ before per-epoch update** — the K-loop reads λ values from `self.lambda_d`. After epoch N's validation, we update λ via `update_lambda_from_auc`. The next training epoch's K-loop reads the new λ. Race-free because validation happens between training epochs on the same stream. + +5. **Cuda-graph capture compatibility** — the K-loop runs inside a captured CUDA graph. The graph reads `lambda_d` device pointer; we update the *contents* of that buffer between captures (per epoch), not the pointer. Compatible with graph capture (data update is allowed; pointer change isn't). Same model the current controller uses. + +## 8. Test plan + +### Local (sm_86, RTX 3050) + +1. **Build + unit test**: `cargo test -p ml-alpha --release --test perception_overfit` — all 9 tests pass. The overfit smoke validates the K-loop end-to-end including the new controller wiring. + +2. **Synthetic controller test** (new): construct synthetic `current_auc` sequences and assert the controller produces expected λ values. + - All-zero current_auc → all λ = 1.0 (cold start). + - Monotonically-improving AUC → all λ = 1.0 (no regret). + - One horizon regresses by 0.05 from best → that horizon's λ ≈ 2.0 (saturates at observed-max-regret), others ≈ 1.0. + - All horizons regress equally → all λ ≈ 2.0 (uniform max). + - File: `crates/ml-alpha/tests/horizon_lambda_auc_regret.rs`. + +### Cluster + +3. **5-epoch single-fold smoke** at the new commit. Check the `isv snapshot` log lines show non-zero regret values from e1 onwards, and `best_auc_h*` is monotonically non-decreasing. Confirm λ is correlated with regret (high regret → high λ). + +4. **3-fold validation** at same `seed=16962`, 30 epochs. Compare to Option-2 baseline (`004b662c8`): + - Primary success criterion: `mean_auc` lifts by ≥ +0.005 averaged over 3 folds. + - Secondary: median saturation→next-epoch-AUC delta is positive. + - If primary fails but secondary positive: the controller is doing the right thing per-horizon but doesn't move the mean — possible the per-horizon AUC distribution is fine and the gain is in stability (std reduction). Report and discuss. + +## 9. Implementation surface (file-level summary) + +| File | Change | +|---|---| +| `crates/ml-alpha/cuda/horizon_loss_ema.cu` | NEW — small kernel (loss_ema EMA + closed-form log σ) | +| `crates/ml-alpha/cuda/horizon_lambda.cu` | REWRITE — AUC-regret math (signature change: now consumes `current_auc[5]`) | +| `crates/ml-alpha/build.rs` | Add `horizon_loss_ema` to kernel list | +| `crates/ml-alpha/src/trainer/perception.rs` | Replace `z_max_ema_d` with `best_auc_d` + `regret_max_ema_d`. Replace per-step `horizon_ema_and_lambda` launch with per-step `horizon_loss_ema`. Add per-epoch `update_lambda_from_auc(per_horizon_auc)` method. | +| `crates/ml-alpha/examples/alpha_train.rs` | Call `trainer.update_lambda_from_auc(&per_horizon_auc)` after the existing `validation` log line. Extend the `isv snapshot` log line with `best_auc_h*` and `regret_h*` fields. | +| `crates/ml-alpha/tests/horizon_lambda_auc_regret.rs` | NEW — synthetic controller correctness tests | +| `/tmp/alpha_monitor.py` | Extend `parse_train_log` to capture `best_auc_h*` and `regret_h*` from the `isv snapshot` line; expose in the wf-detail status view | + +Estimate: ~250 LOC across kernels + Rust, plus ~50 LOC of new tests. + +## 10. Open questions for review + +1. **Best-AUC decay vs strict-max?** Spec choice: strict max. Alternative: EMA-smoothed best (e.g., α=0.05) lets the controller "forget" outliers. *Recommendation*: strict-max for the first round; revisit if outlier-pinning becomes a problem. + +2. **Per-epoch update vs per-K-step?** Spec choice: per-epoch (the cheapest, most-aligned cadence). Alternative: per-step λ update using a running estimate of validation AUC (e.g., online AUC from the training mini-batch's logits). *Recommendation*: per-epoch is sufficient — λ is consumed 64 times per training step in the K-loop, so even per-epoch λ gets ~512K applications per fold. + +3. **Should the existing σ sidecar move out of horizon_loss_ema too?** Currently σ is wired into `loss_ema` computation. The σ kernel produces `log_sigma_h` from `loss_ema` in closed form (the σ-sidecar from `17eb82511`). Cleanest: bundle σ into `horizon_loss_ema` (one fused kernel). Alternative: split σ into its own kernel for clarity. *Recommendation*: bundle for now (one fewer kernel launch per step); split if it becomes a clarity issue. + +4. **`REGRET_FLOOR` value?** Spec choice: `0.001` (0.1pp AUC). Sensitivity: smaller floor → controller saturates on smaller regressions (more reactive); larger floor → only big regressions trigger boost. *Recommendation*: 0.001 is the right order of magnitude; tunable but no need to make ISV-driven for the first round. + +5. **Should we also expose `is_personal_best[5]` as a per-epoch boolean log field?** Useful for monitoring "which horizons just hit a new high?" Tiny addition. *Recommendation*: yes, costs 5 booleans / epoch. + +## 11. References + +- `pearl_controller_anchors_isv_driven` — every controller anchor/target/cap is signal-driven, never hardcoded. +- `pearl_adam_normalizes_loss_weights` — Adam cancels uniform per-loss weight lifts; effective lever is trunk-grad scaling, not loss aggregate. +- `pearl_first_observation_bootstrap` — sentinel = 0, first observation replaces (not blends). +- `pearl_audit_unboundedness_for_implicit_asymmetry` — asymmetric clamp (boost-only) preserved. +- `pearl_zscore_normalization_for_magnitude_asymmetric_signals` — historical motivation for the prior z-score design; explicitly REPLACED here because the signal (BCE) was wrong, not the math. +- `pearl_input_feature_redundancy_loadbearing` (2026-05-18) — the Option-2 input-feature fix this builds on top of. + +The current controller's saturation pattern (gm67g fold-0 ISV trajectory in `/tmp/alpha_monitor/workflows/alpha-perception-gm67g.json`) is the empirical motivation for this redesign.