gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

spec(ml-alpha): AUC-regret controller — replace BCE-z-score signal with per-horizon-regret

Browse Source 
The current λ controller boosts horizons by BCE-EMA z-score, which conflates
three distinct causes of high BCE — only one of which (under-trained
horizon) benefits from boosting. The other two (intrinsically harder
horizon, calibration drift) are unaffected by gradient-magnitude lifts.

Empirical motivation (gm67g fold-0, this branch's 3-fold run):
when λ_h6000 saturated at 2.0, h6000's next-epoch AUC went UP 2/4
times and DOWN 2/4 times. 50% hit rate ⇒ the BCE saturation signal
is misaligned with the optimization objective.

This spec replaces BCE-z-score with AUC-regret:

  best_auc[h]     = running max of per-horizon validation AUC
  regret[h]       = max(0, best_auc[h] - current_auc[h])
  regret_max_ema  = EMA of max_h regret[h]
  λ[h]            = clamp(1.0, 2.0, 1 + regret[h] / regret_max_ema)

Properties:
  - Aligned with the objective (AUC, not BCE)
  - Naturally bounded (regret ∈ [0, 1])
  - "At personal best" → λ=1.0 (no wasted boost)
  - Auto-saturation by design (max-regret horizon → ceiling)
  - Cold-start clean (e0: best=current, regret=0, uniform λ)
  - Zero hardcoded magic beyond bootstrap epsilons

Implementation surface ~250 LOC:
  - Split horizon_lambda kernel: horizon_loss_ema (per-step) +
    horizon_lambda (per-epoch, AUC-regret math)
  - Trainer state: drop z_max_ema, add best_auc + regret_max_ema
  - Per-epoch entry point: trainer.update_lambda_from_auc()
  - Extend isv snapshot log line with best_auc_h* + regret_h*

Test plan:
  - Local 9/9 perception_overfit
  - New synthetic-AUC unit test (controller correctness)
  - Cluster 5-epoch smoke + 3-fold A/B vs Option-2 baseline (004b662c8)
  - Success: mean_auc lifts ≥ +0.005 AND median sat→next-AUC delta positive

Awaiting user review before plan handoff.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-18 23:06:29 +02:00
parent 004b662c80
commit 1fc100ae74
1 changed files with 293 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

293
docs/superpowers/specs/2026-05-18-auc-regret-controller-design.md Normal file
View File

@@ -0,0 +1,293 @@
# AUC-Regret Controller Design Spec
**Status:** DESIGN — written during 3-fold validation of `004b662c8` (Option 2 input-feature fix); awaiting user review before plan handoff
**Date:** 2026-05-18
**Branch:** `ml-alpha-phase-a` (next commit)
**Owner:** ml-alpha team
## 0. Context
The per-horizon λ controller in `horizon_lambda.cu` scales each horizon's trunk-grad contribution in `heads_grn_bwd` by `λ[h] ∈ [1.0, 2.0]`. Per `pearl_adam_normalizes_loss_weights`, scaling head-grad would be Adam-cancelled; scaling **trunk-grad works** because trunk parameters are shared across horizons — boosting one horizon's contribution shifts the trunk's effective update direction in that horizon's favor while Adam normalizes the aggregate.
The current controller computes `λ[h]` from a **z-score of `loss_ema[h]`** (per-horizon EMA of unweighted BCE):
```
z[h] = (loss_ema[h] - mean) / std
Z_SCALE_ISV = 1.0 / max(z_max_ema, 0.1)
λ[h] = clamp(1.0, 2.0, 1.0 + Z_SCALE_ISV * z[h])
```
**Empirical problem (gm67g fold-0, 20 epochs):** when `λ_h6000` saturated at 2.0 (4 of 20 epochs), h6000 AUC went UP on the next epoch in 2/4 cases and DOWN in 2/4. Saturation does not predict AUC gain — the BCE signal is misaligned with the optimization objective.
**Root cause:** BCE measures distance from labels, not predictability. A horizon can have high BCE for three reasons, only one of which benefits from boosting:
| Cause | Boost helps? |
|---|---|
| Genuinely under-trained — encoder hasn't found the signal yet | **Yes** |
| Intrinsically harder horizon (e.g., h6000 has more inherent price-walk uncertainty) | **No** — BCE floor is structural |
| Calibration drift (probs miscalibrated but ranking fine — `pearl_auc_accuracy_gap_means_miscalibration`) | **No** — AUC unchanged by ranking-preserving recalibration |
For our 5 horizons {30, 100, 300, 1000, 6000}, cause #2 dominates h6000. The current controller almost always identifies h6000 as the "worst" horizon and boosts it — but the BCE floor is structural, so the boost spends encoder capacity on a horizon whose loss can't get materially lower.
**This spec replaces BCE-z-score with AUC-regret as the controller signal.** The new controller boosts horizons that have fallen below their *personal-best* AUC — i.e., horizons that demonstrably *were* learnable and now aren't.
Out of scope (separate work):
- Changing the consumption point (still `heads_grn_bwd` per-horizon trunk-grad scaling).
- Changing the λ envelope `[1.0, 2.0]`.
- Adding a meta-learner over λ. Per `pearl_controller_anchors_isv_driven`, the controller stays signal-driven.
- Trainable controllers. The whole point is principled signal-driven adaptation.
## 1. Goal
Per-epoch validation AUC drives λ. A horizon that's at its personal-best AUC contributes uniform grad (λ=1). A horizon that's regressed from its personal-best AUC contributes boosted grad (λ up to 2). The historical-max regret across horizons maps exactly to the ceiling (Z_SCALE_ISV self-calibrating).
**Falsifiable success criterion:** at this commit, on the same `seed=16962` 3-fold CV at 30 epochs as the Option-2 baseline, **mean_auc lifts by ≥ +0.005** (one full noise σ from prior runs) AND **the median saturation→next-epoch-AUC delta is positive** (i.e., when the controller boosts, the boosted horizon actually improves more often than not).
## 2. Math
### State (carried across training steps)
| Variable | Shape | Description |
|---|---|---|
| `best_auc[h]` | [5] | Running max of per-horizon validation AUC since training start |
| `regret_max_ema` | [1] | EMA of `max_h regret[h]` across epochs (the adaptive scale anchor) |
Both initialised to 0 (sentinel — first-observation bootstrap per `pearl_first_observation_bootstrap`).
### Per-epoch update (after validation completes)
Input: `current_auc[5]` from the validation pass (already computed for the `validation` log line).
```text
# 1. Update best_auc (running max — strict, no decay).
for h in 0..5:
best_auc[h] = max(best_auc[h], current_auc[h])
# 2. Compute regret. ≥ 0 by construction.
regret[h] = max(0.0, best_auc[h] - current_auc[h])
# 3. Update regret_max_ema (first-observation bootstrap, fixed α=0.1).
r_max = max_h regret[h]
if regret_max_ema <= 0:
regret_max_ema = r_max # sentinel bootstrap
else:
regret_max_ema = (1-α) * regret_max_ema + α * r_max
# 4. Adaptive scale — by design, observed-max-regret maps to ceiling.
Z_SCALE_ISV = (LAMBDA_CEILING - LAMBDA_FLOOR) / max(regret_max_ema, REGRET_FLOOR)
= 1.0 / max(regret_max_ema, 0.001)
# 5. Per-horizon λ.
for h in 0..5:
raw = 1.0 + Z_SCALE_ISV * regret[h]
λ[h] = clamp(LAMBDA_FLOOR=1.0, LAMBDA_CEILING=2.0, raw)
```
### Properties
- **At personal best**: `regret[h]=0` → `λ[h]=1.0`. Uniform encoder grad.
- **Max regret**: `regret[h]=regret_max_ema` → `λ[h]=2.0`. Full boost.
- **No regret anywhere**: `r_max=0` → `regret_max_ema` floor at `0.001` keeps `Z_SCALE_ISV` finite. All `regret[h]=0` → all `λ[h]=1.0`. Controller dormant.
- **Cold start (e0)**: `best_auc` is 0 → after first validation, `best_auc[h] = current_auc[h]` for all h → `regret[h]=0` → uniform λ. Controller engages only at e1+.
- **Persistent regression**: if a horizon stays below its best for many epochs, regret stays high, λ stays boosted. The controller doesn't time out.
- **No hardcoded magic numbers** beyond bootstrap epsilons (`REGRET_FLOOR=0.001`, EMA `α=0.1` matching the existing loss-EMA cadence) and the clamp envelope (kept at `[1.0, 2.0]` for direct comparability with the current controller).
## 3. Removed: BCE-z-score signal
The existing kernel computes `loss_ema[h]` from `loss_per_horizon` emitted by the BCE kernel, z-scores across horizons, EMA-tracks `z_max`, and produces λ.
**This pipeline is fully replaced.** Specifically:
- `loss_ema[5]` state → **REMOVED** from the controller. (Still emitted by the BCE kernel for telemetry, but no longer consumed.)
- `z_max_ema[1]` state → **REMOVED**.
- `log_sigma_h[5]` output → **KEPT as-is**. The closed-form Kendall σ from loss_ema is preserved for telemetry (BCE kernel ignores it per the σ-sidecar commit `17eb82511`, so it's a pure observability output). σ still uses `loss_ema` internally; that loss_ema computation moves to a dedicated `horizon_loss_ema` kernel.
After this commit:
- `horizon_loss_ema.cu` (new, tiny): maintains `loss_ema[5]` and `log_sigma_h[5]`. Called from the trainer at the same point in the step as today.
- `horizon_lambda.cu` (rewritten): consumes `current_auc[5]` + maintains `best_auc[5]` + `regret_max_ema[1]`, emits `lambda[5]`. Called from the trainer **once per epoch** (after validation), not per step.
The per-step call frequency change is significant. See §5 for trainer wiring.
## 4. Inputs / outputs
### `horizon_loss_ema` kernel (new, split off from current)
```cuda
extern "C" __global__ void horizon_loss_ema(
const float* __restrict__ loss_per_horizon, // [5] from BCE kernel
float* __restrict__ loss_ema, // [5] state read+write
float* __restrict__ log_sigma_h // [5] output (sidecar)
);
```
Single thread, single block. Updates `loss_ema[h]` with sentinel bootstrap + fixed α=0.1; computes `log_sigma_h[h] = max(log(0.5), 0.5 * log(loss_ema[h] + ε))` (preserves the existing σ-sidecar behavior). Called per-step from the trainer hot loop (same place `horizon_ema_and_lambda` is called today).
### `horizon_lambda` kernel (rewritten)
```cuda
extern "C" __global__ void horizon_lambda(
const float* __restrict__ current_auc, // [5] from this epoch's validation
float* __restrict__ best_auc, // [5] state read+write
float* __restrict__ regret_max_ema, // [1] state read+write
float* __restrict__ lambda // [5] output
);
```
Single thread, single block. Implements the math in §2. Called **once per epoch** from `alpha_train.rs` after validation completes.
The trainer passes `current_auc` from the existing per-horizon AUC computation (the same values that populate the `auc_h30..auc_h6000` keys on the `validation` log line and `best_val_auc` in the summary). No new computation needed.
## 5. Trainer wiring (`crates/ml-alpha/src/trainer/perception.rs` + `alpha_train.rs`)
### New `PerceptionTrainer` state
```rust
// AUC-regret controller state.
best_auc_d: CudaSlice<f32>, // [5], zero-init (sentinel)
regret_max_ema_d: CudaSlice<f32>, // [1], zero-init (sentinel)
horizon_lambda_fn: CudaFunction, // existing handle, same name
// New:
horizon_loss_ema_fn: CudaFunction, // split off from horizon_lambda
```
### Removed state
```rust
z_max_ema_d: CudaSlice<f32>, // [1] — DROPPED (was used by BCE z-score controller)
```
`loss_ema_d` and `log_sigma_h_d` remain (both still used by the new `horizon_loss_ema` kernel + BCE kernel's σ-sidecar signature stability).
### Per-step trainer call (in K-loop body)
```rust
// Was: launch horizon_ema_and_lambda(loss_per_h, loss_ema, z_max_ema, lambda, log_sigma_h)
// Now: launch horizon_loss_ema(loss_per_h, loss_ema, log_sigma_h)
self.stream.launch_builder(&self.horizon_loss_ema_fn)
.arg(&self.loss_per_horizon_d)
.arg(&mut self.loss_ema_d)
.arg(&mut self.log_sigma_h_d)
.launch(cfg_single)?;
```
λ does NOT update per-step anymore. It uses the value computed at the last epoch boundary.
### Per-epoch trainer call (new entry point, called from `alpha_train.rs`)
```rust
impl PerceptionTrainer {
pub fn update_lambda_from_auc(&mut self, current_auc: &[f32; N_HORIZONS]) -> Result<()> {
// Upload current_auc to device, launch horizon_lambda kernel,
// λ buffer is now populated for the next training epoch's K-loop.
}
}
```
In `alpha_train.rs` after computing `per_horizon_auc` for the validation pass:
```rust
trainer.update_lambda_from_auc(&per_horizon_auc)
.context("update lambda from validation AUC")?;
```
### Cold-start handling
At e0, before validation runs, `best_auc_d = [0; 5]` and `regret_max_ema_d = [0]`. The trainer hot loop's first epoch reads `lambda` which is also zero-initialised → all-zero λ → kernel sentinel logic (`l <= 0 ? 1.0 : l`) in `grn_bwd` already converts to uniform λ=1.0 per `s_lambda` cooperative-load logic. **No special-case code needed in the trainer**; the existing sentinel handles e0 transparently.
After validation at end of e0, `update_lambda_from_auc` runs with `current_auc[h] = first-observation`. Since `best_auc[h] = 0` initially, `best_auc[h] := max(0, current_auc[h]) = current_auc[h]` → regret=0 everywhere → λ=1.0. Same uniform behavior as the sentinel, just emitted from the kernel.
E1 onwards, regret can become non-zero whenever a horizon dips below its observed maximum.
## 6. Telemetry / observability
The existing `isv snapshot` log line at validation end currently logs `ema_h*` (loss_ema) and `lam_h*` (the λ values). **Add `best_auc_h*` and `regret_h*` to that log line** so the per-epoch trajectory in `/tmp/alpha_status.py wf <name>` shows the full controller state.
```rust
tracing::info!(
epoch,
// existing:
ema_h30 = ema[0], /* ... */ ema_h6000 = ema[4],
lam_h30 = lam[0], /* ... */ lam_h6000 = lam[4],
// new (AUC-regret state):
best_auc_h30 = best_auc[0], /* ... */ best_auc_h6000 = best_auc[4],
regret_h30 = regret[0], /* ... */ regret_h6000 = regret[4],
"isv snapshot"
);
```
The monitor's `parse_train_log` already handles unknown extra fields gracefully (regex only captures explicit captures); add a parse pass for these new fields.
## 7. Failure modes & mitigations
1. **All horizons at personal best forever** — `regret=0` everywhere, λ=1.0 uniform forever. The controller is dormant. *This is the desired behavior* — if the model never regresses on any horizon, the controller has nothing useful to do. The encoder learns uniformly.
2. **Best-AUC drift on noisy validation** — if a validation seed quirk produces an outlier-high AUC on some horizon at e3, `best_auc[h]` pins to that outlier, and every subsequent epoch shows positive regret. Boost stays on indefinitely.
**Mitigation**: validation uses a deterministic seed (`val_loader` seed = `train_seed + 0xC0FFEE`); same validation data every epoch. So outliers are not from sampling noise — they're real model behavior on that fixed eval set. If the model genuinely degraded, persistent regret is the correct signal.
Open question: should `best_auc` decay with time? Possible failure: model finds a high-AUC local optimum at e3 then diverges. With strict-max best_auc, the controller will keep trying to drag it back; with EMA-smoothed best_auc, it would let go. **Spec choice: strict max (no decay)**. Rationale: the strict signal is more interpretable; if the model genuinely diverges, early-stop fires (patience=5 on mean_auc) and we end the run anyway.
3. **Saturation rate too high / too low** — if `regret_max_ema` is too small (most regrets near 0), `Z_SCALE_ISV` is large and even tiny regrets saturate λ. The `REGRET_FLOOR=0.001` clamps this; effectively the smallest regret that produces λ=2.0 saturation is ≥ 0.001 in absolute AUC. That's ~0.1 percentage points — reasonable. If you think it's too sensitive, raise the floor.
4. **K-loop reads λ before per-epoch update** — the K-loop reads λ values from `self.lambda_d`. After epoch N's validation, we update λ via `update_lambda_from_auc`. The next training epoch's K-loop reads the new λ. Race-free because validation happens between training epochs on the same stream.
5. **Cuda-graph capture compatibility** — the K-loop runs inside a captured CUDA graph. The graph reads `lambda_d` device pointer; we update the *contents* of that buffer between captures (per epoch), not the pointer. Compatible with graph capture (data update is allowed; pointer change isn't). Same model the current controller uses.
## 8. Test plan
### Local (sm_86, RTX 3050)
1. **Build + unit test**: `cargo test -p ml-alpha --release --test perception_overfit` — all 9 tests pass. The overfit smoke validates the K-loop end-to-end including the new controller wiring.
2. **Synthetic controller test** (new): construct synthetic `current_auc` sequences and assert the controller produces expected λ values.
- All-zero current_auc → all λ = 1.0 (cold start).
- Monotonically-improving AUC → all λ = 1.0 (no regret).
- One horizon regresses by 0.05 from best → that horizon's λ ≈ 2.0 (saturates at observed-max-regret), others ≈ 1.0.
- All horizons regress equally → all λ ≈ 2.0 (uniform max).
- File: `crates/ml-alpha/tests/horizon_lambda_auc_regret.rs`.
### Cluster
3. **5-epoch single-fold smoke** at the new commit. Check the `isv snapshot` log lines show non-zero regret values from e1 onwards, and `best_auc_h*` is monotonically non-decreasing. Confirm λ is correlated with regret (high regret → high λ).
4. **3-fold validation** at same `seed=16962`, 30 epochs. Compare to Option-2 baseline (`004b662c8`):
- Primary success criterion: `mean_auc` lifts by ≥ +0.005 averaged over 3 folds.
- Secondary: median saturation→next-epoch-AUC delta is positive.
- If primary fails but secondary positive: the controller is doing the right thing per-horizon but doesn't move the mean — possible the per-horizon AUC distribution is fine and the gain is in stability (std reduction). Report and discuss.
## 9. Implementation surface (file-level summary)
| File | Change |
|---|---|
| `crates/ml-alpha/cuda/horizon_loss_ema.cu` | NEW — small kernel (loss_ema EMA + closed-form log σ) |
| `crates/ml-alpha/cuda/horizon_lambda.cu` | REWRITE — AUC-regret math (signature change: now consumes `current_auc[5]`) |
| `crates/ml-alpha/build.rs` | Add `horizon_loss_ema` to kernel list |
| `crates/ml-alpha/src/trainer/perception.rs` | Replace `z_max_ema_d` with `best_auc_d` + `regret_max_ema_d`. Replace per-step `horizon_ema_and_lambda` launch with per-step `horizon_loss_ema`. Add per-epoch `update_lambda_from_auc(per_horizon_auc)` method. |
| `crates/ml-alpha/examples/alpha_train.rs` | Call `trainer.update_lambda_from_auc(&per_horizon_auc)` after the existing `validation` log line. Extend the `isv snapshot` log line with `best_auc_h*` and `regret_h*` fields. |
| `crates/ml-alpha/tests/horizon_lambda_auc_regret.rs` | NEW — synthetic controller correctness tests |
| `/tmp/alpha_monitor.py` | Extend `parse_train_log` to capture `best_auc_h*` and `regret_h*` from the `isv snapshot` line; expose in the wf-detail status view |
Estimate: ~250 LOC across kernels + Rust, plus ~50 LOC of new tests.
## 10. Open questions for review
1. **Best-AUC decay vs strict-max?** Spec choice: strict max. Alternative: EMA-smoothed best (e.g., α=0.05) lets the controller "forget" outliers. *Recommendation*: strict-max for the first round; revisit if outlier-pinning becomes a problem.
2. **Per-epoch update vs per-K-step?** Spec choice: per-epoch (the cheapest, most-aligned cadence). Alternative: per-step λ update using a running estimate of validation AUC (e.g., online AUC from the training mini-batch's logits). *Recommendation*: per-epoch is sufficient — λ is consumed 64 times per training step in the K-loop, so even per-epoch λ gets ~512K applications per fold.
3. **Should the existing σ sidecar move out of horizon_loss_ema too?** Currently σ is wired into `loss_ema` computation. The σ kernel produces `log_sigma_h` from `loss_ema` in closed form (the σ-sidecar from `17eb82511`). Cleanest: bundle σ into `horizon_loss_ema` (one fused kernel). Alternative: split σ into its own kernel for clarity. *Recommendation*: bundle for now (one fewer kernel launch per step); split if it becomes a clarity issue.
4. **`REGRET_FLOOR` value?** Spec choice: `0.001` (0.1pp AUC). Sensitivity: smaller floor → controller saturates on smaller regressions (more reactive); larger floor → only big regressions trigger boost. *Recommendation*: 0.001 is the right order of magnitude; tunable but no need to make ISV-driven for the first round.
5. **Should we also expose `is_personal_best[5]` as a per-epoch boolean log field?** Useful for monitoring "which horizons just hit a new high?" Tiny addition. *Recommendation*: yes, costs 5 booleans / epoch.
## 11. References
- `pearl_controller_anchors_isv_driven` — every controller anchor/target/cap is signal-driven, never hardcoded.
- `pearl_adam_normalizes_loss_weights` — Adam cancels uniform per-loss weight lifts; effective lever is trunk-grad scaling, not loss aggregate.
- `pearl_first_observation_bootstrap` — sentinel = 0, first observation replaces (not blends).
- `pearl_audit_unboundedness_for_implicit_asymmetry` — asymmetric clamp (boost-only) preserved.
- `pearl_zscore_normalization_for_magnitude_asymmetric_signals` — historical motivation for the prior z-score design; explicitly REPLACED here because the signal (BCE) was wrong, not the math.
- `pearl_input_feature_redundancy_loadbearing` (2026-05-18) — the Option-2 input-feature fix this builds on top of.
The current controller's saturation pattern (gm67g fold-0 ISV trajectory in `/tmp/alpha_monitor/workflows/alpha-perception-gm67g.json`) is the empirical motivation for this redesign.