fix(rl): B-6 — ISV-driven adaptive asymmetric Wiener-α (Bayesian shrinkage)
B-5 (asymmetric α with static α_slow=0.001) revealed the static parameter problem: provably bounds cascades (avg_win peak $2k vs B-4's $40k) BUT over-conservative in train (dckcc step 800: avg_l > avg_w → Kelly says don't trade → model can't discover edges; wr_ema crashed to 0.145). The fundamental tension: static α_slow can't satisfy BOTH - Train convergence: asymmetry must FADE so model learns from real data - Boundary safety: asymmetry must ENGAGE at every fold to prevent cascade B-6 RESOLVES this via Bayesian shrinkage: trust(n) = min(1, cum_dones / n_full_threshold) [Phase 1] stability = exp(-CV × cv_gain) [Phase 2] trust_eff = trust(n) × stability α_slow_eff = α_slow_min + (α_fast − α_slow_min) × trust_eff Phase 1 (data-quantity): trust grows with cum_dones; reset_session_state zeroes cum_dones → asymmetry RESUMES at every boundary. Math: at n=0 α_slow_eff = α_slow_min = 0.001 (full skepticism). At n=n_full = 30k trades: α_slow_eff = α_fast = 0.05 (full standard Wiener). Phase 2 (data-quality): Welford CV of reward magnitude gates trust. Stable signal (CV→0): stability=1, trust opens normally. Volatile signal (CV high): stability→0, asymmetry persists. cv_gain=0 disables Phase 2. Per-EMA asymmetry direction encodes Kelly safety semantics: avg_win: slow-up (skeptical of wins), fast-down avg_loss: fast-up (admit losses), slow-down (slow forget) wr_ema: slow-up (skeptical of high WR), fast-down ISV slots (all signal-derived from cum_dones + Welford): 721 RL_EMA_ALPHA_SLOW_MIN_INDEX = 0.001 722 RL_EMA_TRUST_FULL_THRESHOLD_INDEX = 30000 723 RL_EMA_CV_GAIN_INDEX = 1.0 Threshold calibration (n_full=30k): - Train: ~1000 cluster steps for trust to fully open → asymmetry active during cold-start (first 30 steps, cascade prevention) then fades. - Eval: 30 dones/step × 500 eval steps = 15k dones → trust climbs to 0.5 by eval end → partial protection throughout eval. Composes: - B-3 Kelly fractional-trust (Kelly SIZING gated by cum_dones) - B-6 EMA asymmetric-α (Kelly INPUTS biased conservative by cum_dones) Both fade as data accumulates; both reset at boundary. Spec: docs/superpowers/specs/2026-06-01-ema-asymmetric-trust-with-cv-gain.md Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -0,0 +1,259 @@
|
||||
# Asymmetric Wiener-α EMA with Trust Schedule and Welford-CV Gain (B-6)
|
||||
|
||||
**Date:** 2026-06-01
|
||||
**Status:** Draft (post B-5 cluster diagnosis)
|
||||
**Parent docs:**
|
||||
- `2026-05-31-pos-max-ema-cold-start-redesign.md` (B-2)
|
||||
- `2026-05-31-eval-boundary-addendum-reward-scale-and-train-spike.md` (A3)
|
||||
**Pearls referenced:**
|
||||
- `pearl_controller_anchors_isv_driven` — every controller anchor ISV-driven
|
||||
- `feedback_adaptive_not_tuned` — signal-driven, not tuned constants
|
||||
- `feedback_isv_for_adaptive_bounds` — bounds in ISV
|
||||
- `pearl_first_observation_bootstrap` — the underlying anti-pattern
|
||||
- `pearl_adaptive_carryover_discipline` — predictive vs normalization EMAs
|
||||
- `pearl_multiplicative_controllers_need_bounded_step_and_noise_floor`
|
||||
|
||||
---
|
||||
|
||||
## 1. Motivation
|
||||
|
||||
The B-4 multiplicative/additive caps failed math gap (1.5^N unbounded; additive cap never engaged). B-5 introduced **asymmetric Wiener-α** with α_slow=0.001 in safety direction → provably biased estimator → Kelly under-sized by construction.
|
||||
|
||||
Cluster diagnosis (alpha-rl-dckcc step 800, B-5):
|
||||
- avg_w max peak: **$2,037** (vs B-4's $40,911 — **20× reduction** ✓)
|
||||
- wr_ema max: **0.50** (vs B-4's 0.88 — never exceeded ✓)
|
||||
- BUT wr_ema crashed to 0.145 by step 800, avg_l > avg_w → Kelly says don't trade
|
||||
- Model trains on near-zero positions → can't discover edges
|
||||
|
||||
**The fundamental tension:** static α_slow can't satisfy both:
|
||||
- Train: asymmetry must FADE so model learns from real magnitudes
|
||||
- Eval boundary: asymmetry must ENGAGE so no cascade
|
||||
|
||||
## 2. Goals
|
||||
|
||||
### G1 — ISV-driven adaptive α_slow
|
||||
Replace static α_slow=0.001 with signal-derived value that adapts to data confidence. Per `pearl_controller_anchors_isv_driven`, every controller anchor must be ISV-driven (signal-derived, not just stored).
|
||||
|
||||
### G2 — Boundary safety via resettable counter
|
||||
Asymmetry must resume at every eval boundary. Use `RL_CUMULATIVE_DONES_INDEX` (slot 660) which is reset by `reset_session_state` — same counter as B-3 Kelly trust schedule. Single source of truth for "data confidence".
|
||||
|
||||
### G3 — Train convergence
|
||||
After sufficient data accumulation, asymmetry fades to standard Wiener-α=0.05. Model learns from real signal magnitudes; b̂ and wr̂ approach true values.
|
||||
|
||||
### G4 — Volatility-aware skepticism
|
||||
Beyond cumulative count, the EFFECTIVE confidence depends on signal stability. A volatile signal (high CV) demands more skepticism even at high trade counts. Phase 2 adds Welford-CV gain modulating the trust schedule.
|
||||
|
||||
### G5 — Non-goals
|
||||
- Don't change asymmetry DIRECTION (slow-up for wins/wr, fast-up for losses) — that encodes the loss-aversion safety semantic
|
||||
- Don't replace Wiener-α with Kalman/median estimators — out of scope, bigger refactor
|
||||
- Don't add additional EMAs beyond avg_win, avg_loss, wr_ema in this phase
|
||||
|
||||
## 3. Design
|
||||
|
||||
### 3.1 Phase 1 — Trust schedule on α_slow (data-confidence)
|
||||
|
||||
```
|
||||
trust(n) = min(1, n_cumulative_dones / n_full_threshold)
|
||||
α_slow_eff = α_slow_min + (α_fast − α_slow_min) × trust(n)
|
||||
```
|
||||
|
||||
For each EMA, asymmetric direction blends α_fast and α_slow_eff:
|
||||
```c
|
||||
// avg_win: slow-up, fast-down (skeptical of wins)
|
||||
alpha = (step_avg > prev) ? α_slow_eff : α_fast;
|
||||
|
||||
// avg_loss: fast-up, slow-down (admit losses, slow forget)
|
||||
alpha = (step_avg > prev) ? α_fast : α_slow_eff;
|
||||
|
||||
// wr_ema: slow-up, fast-down (skeptical of high WR)
|
||||
alpha = (step_wr > prev) ? α_slow_eff : α_fast;
|
||||
```
|
||||
|
||||
#### Math properties
|
||||
|
||||
**Cold-start** (n=0): α_slow_eff = α_slow_min = 0.001 → full asymmetry → b̂ underestimated → conservative Kelly.
|
||||
|
||||
**Mid-training** (n=n_full/2): α_slow_eff = (α_slow_min + α_fast)/2 = 0.0255 → moderate asymmetry → b̂ slightly biased low.
|
||||
|
||||
**Post-warmup** (n ≥ n_full): α_slow_eff = α_fast = 0.05 → standard Wiener → unbiased.
|
||||
|
||||
**Eval boundary** (reset_session_state zeros cum_dones): α_slow_eff = α_slow_min → asymmetry RESUMES → conservative re-engagement.
|
||||
|
||||
#### Threshold calibration
|
||||
|
||||
`n_full_threshold = 30,000` derived from:
|
||||
|
||||
1. **Hoeffding for proportions**: `n = 1.844 / ε²` for confidence 95%. For wr precision ε=0.05: n=738. For ε=0.01 (very tight): n=18,440.
|
||||
|
||||
2. **For heavy-tailed magnitudes**: variance bounds don't apply. Need larger n for reliable mean estimation. Conservative choice: 30k.
|
||||
|
||||
3. **Eval phase coverage**: at b=1024 with ~30 dones/step, eval phase (500 steps) generates ~15k dones → trust climbs to 0.5 by eval end. Asymmetry remains partially active throughout eval. Good safety property.
|
||||
|
||||
4. **Train phase**: ~1000 training steps for trust to fully open (~30k dones / 30 dones/step). Sufficient for cascade prevention during cold-start (first 30 steps where peak occurs).
|
||||
|
||||
### 3.2 Phase 2 — Welford-CV gain on trust (volatility-aware)
|
||||
|
||||
The Phase 1 trust schedule treats all data as equally trustworthy after n_full. But noisy signals demand more skepticism. Phase 2 modulates trust by the signal's Coefficient of Variation:
|
||||
|
||||
```
|
||||
CV(signal) = σ_welford / μ_welford (from existing Welford triplet)
|
||||
stability = exp(-CV × γ_cv) // bounded [0, 1]
|
||||
trust_eff = trust(n) × stability
|
||||
α_slow_eff = α_slow_min + (α_fast − α_slow_min) × trust_eff
|
||||
```
|
||||
|
||||
Where:
|
||||
- `γ_cv` (ISV slot 723, default 1.0): sensitivity to CV. Higher = more skeptical of volatile signals.
|
||||
- CV=0 (perfectly stable): stability=1, trust_eff = trust(n)
|
||||
- CV=1 (σ=μ, classical "noisy"): stability=exp(-1)=0.37, trust gated to 37% of n-based value
|
||||
- CV=2 (σ=2μ, highly volatile): stability=0.14, heavily skeptical
|
||||
|
||||
#### Math properties (Phase 2)
|
||||
|
||||
**Stable regime** (low CV): trust opens normally with data accumulation.
|
||||
|
||||
**Volatile regime** (high CV): trust capped low even with abundant data → α_slow_eff stays near α_slow_min → continued conservatism in volatile markets. Matches surfer-trading philosophy: be skeptical when the wave isn't clean.
|
||||
|
||||
**Adaptive across regimes**: as market changes volatility, asymmetry naturally tightens/loosens. No manual retuning needed.
|
||||
|
||||
### 3.3 Welford signals available
|
||||
|
||||
The signal_variance_update kernel maintains Welford triplets for:
|
||||
- `RL_REWARD_MAGNITUDE_EMA_INDEX` (slot 614) — used by reward_scale controller
|
||||
- `RL_PPO_RATIO_VAR_M2_INDEX` (slot ~422)
|
||||
- `RL_ADV_VAR_M2_INDEX`, `RL_ENTROPY_OBS_VAR_M2_INDEX`, etc.
|
||||
|
||||
For our purposes we need Welford on:
|
||||
- `RL_AVG_WIN_USD_EMA_INDEX` (slot 678) — per-step avg_win signal
|
||||
- `RL_AVG_LOSS_USD_EMA_INDEX` (slot 679) — per-step avg_loss signal
|
||||
- `RL_WIN_RATE_EMA_INDEX` (slot 677) — per-step wr signal
|
||||
|
||||
Two options:
|
||||
- **A**: Extend signal_variance_update to also track avg_win/loss/wr triplets (new Welford slots)
|
||||
- **B**: Approximate CV using existing reward_magnitude Welford (slot 614) — same underlying volatility
|
||||
|
||||
Option B is simpler and avoids slot proliferation. The reward magnitude EMA captures overall signal volatility, which is correlated with the per-EMA volatility we'd otherwise track separately.
|
||||
|
||||
**Decision**: Phase 2 uses Option B (existing Welford). If empirically inadequate, Phase 2.5 adds per-EMA Welford.
|
||||
|
||||
## 4. ISV slot allocation
|
||||
|
||||
| Slot | Name | Default | Description |
|
||||
|---|---|---:|---|
|
||||
| 721 | `RL_EMA_ALPHA_SLOW_MIN_INDEX` | 0.001 | Minimum α for asymmetric admission (= full skepticism) |
|
||||
| 722 | `RL_EMA_TRUST_FULL_THRESHOLD_INDEX` | 30,000 | Cumulative dones for trust=1.0 (asymmetry fades) |
|
||||
| 723 | `RL_EMA_CV_GAIN_INDEX` | 1.0 | Sensitivity to signal CV for Phase 2 gating. Set to 0.0 to disable Phase 2 (pure Phase 1) |
|
||||
|
||||
All ISV-CONFIGURABLE constants. The α_slow_eff itself is signal-DRIVEN (computed at runtime from cum_dones + Welford).
|
||||
|
||||
## 5. Implementation
|
||||
|
||||
### 5.1 Kernel updates
|
||||
|
||||
**`rl_avg_win_loss_ema_update.cu`** + **`rl_win_rate_ema_update.cu`**:
|
||||
```c
|
||||
const float a_slow_min = isv[RL_EMA_ALPHA_SLOW_MIN_INDEX];
|
||||
const float a_fast = EMA_ALPHA_FAST; // 0.05
|
||||
const float n_trades = isv[RL_CUMULATIVE_DONES_INDEX];
|
||||
const float n_full = isv[RL_EMA_TRUST_FULL_THRESHOLD_INDEX];
|
||||
const float cv_gain = isv[RL_EMA_CV_GAIN_INDEX];
|
||||
|
||||
// Phase 1: trust schedule
|
||||
float trust = (n_full > 0.0f) ? fminf(1.0f, n_trades / n_full) : 1.0f;
|
||||
|
||||
// Phase 2: CV gain (only if cv_gain > 0)
|
||||
if (cv_gain > 0.0f) {
|
||||
const float wf_count = isv[RL_REWARD_MAG_VAR_COUNT_INDEX];
|
||||
if (wf_count > 1.0f) {
|
||||
const float wf_m2 = isv[RL_REWARD_MAG_VAR_M2_INDEX];
|
||||
const float wf_mean = isv[RL_REWARD_MAG_VAR_MEAN_INDEX];
|
||||
const float wf_var = wf_m2 / (wf_count - 1.0f);
|
||||
const float cv = (wf_mean > 1e-6f) ? sqrtf(wf_var) / wf_mean : 0.0f;
|
||||
const float stability = expf(-cv * cv_gain);
|
||||
trust *= stability;
|
||||
}
|
||||
}
|
||||
|
||||
const float a_slow_eff = a_slow_min + (a_fast - a_slow_min) * trust;
|
||||
|
||||
// Asymmetric direction (per-EMA semantics)
|
||||
// avg_win: slow-up, fast-down
|
||||
// avg_loss: fast-up, slow-down (mirror)
|
||||
// wr_ema: slow-up, fast-down
|
||||
const float alpha = (X > prev) ? a_slow_eff : a_fast; // avg_win, wr_ema
|
||||
// OR for avg_loss:
|
||||
const float alpha = (X > prev) ? a_fast : a_slow_eff;
|
||||
```
|
||||
|
||||
### 5.2 Bootstrap entries (`with_controllers_bootstrapped`)
|
||||
|
||||
```rust
|
||||
(RL_EMA_ALPHA_SLOW_MIN_INDEX, 0.001_f32), // was just _SLOW_, renamed
|
||||
(RL_EMA_TRUST_FULL_THRESHOLD_INDEX, 30000.0_f32),
|
||||
(RL_EMA_CV_GAIN_INDEX, 1.0_f32), // Phase 2 enabled by default
|
||||
```
|
||||
|
||||
### 5.3 Diag emission
|
||||
|
||||
Add to `build_diag_value`:
|
||||
```rust
|
||||
"ema_alpha_slow_min": isv[RL_EMA_ALPHA_SLOW_MIN_INDEX],
|
||||
"ema_trust_full_threshold": isv[RL_EMA_TRUST_FULL_THRESHOLD_INDEX],
|
||||
"ema_cv_gain": isv[RL_EMA_CV_GAIN_INDEX],
|
||||
```
|
||||
|
||||
### 5.4 Test schema update
|
||||
|
||||
`tests/eval_diag_emission.rs`: EXPECTED_LEAVES 649 → 651 (+2 new leaves; one slot renamed).
|
||||
|
||||
## 6. Validation gates
|
||||
|
||||
### V1 — Math invariants
|
||||
- At step 1 (cold-start, cum_dones=0): observed alpha = α_slow_min when admission direction is safety, α_fast otherwise.
|
||||
- At step 1000 (cum_dones ≫ n_full): observed alpha → α_fast in both directions.
|
||||
- At eval[1] (post-reset_session_state): observed alpha = α_slow_min again.
|
||||
|
||||
### V2 — Cascade bounds
|
||||
- avg_win peak ≤ $5k (vs B-4 $40k, B-5 $2k baseline)
|
||||
- wr_ema max ≤ 0.50 in first 100 steps; can rise above as trust ramps in train
|
||||
- avg_loss > avg_win NOT permanent (should normalize as data accumulates)
|
||||
|
||||
### V3 — Train convergence
|
||||
- By step 1000+: wr_ema ≈ true wr (0.30-0.40 range typical for this codebase)
|
||||
- avg_win > avg_loss for profitable runs (model finds edges)
|
||||
- Kelly active sizing (not constantly at f_floor)
|
||||
|
||||
### V4 — Cluster eval pnl
|
||||
- vs B-3 (x56wn): -$118M
|
||||
- vs B-4 (zh96b, killed): partial data showed similar peaks
|
||||
- vs B-5 (dckcc): pending (likely terrible due to over-conservatism)
|
||||
- Target: ≤ -$30M (10× improvement)
|
||||
|
||||
## 7. Risks
|
||||
|
||||
- **Risk α (low)**: n_full_threshold = 30,000 too high → asymmetry persists into train, hampers learning. Mitigation: ISV-tunable, can lower to 5,000 if needed.
|
||||
- **Risk β (medium)**: Welford CV from reward_magnitude (Option B) inadequate proxy for per-EMA volatility. Mitigation: Phase 2.5 adds per-EMA Welford if validation fails.
|
||||
- **Risk γ (low)**: CV gain γ_cv=1.0 too aggressive → trust permanently low in volatile markets. Mitigation: tunable, can disable Phase 2 by setting γ_cv=0.
|
||||
- **Risk δ (low)**: Resetting cum_dones at boundary disrupts both Kelly trust AND EMA asymmetry → double safety effect at boundary might over-suppress. Mitigation: empirical — eval result reveals if too conservative; can split into separate counters if needed.
|
||||
|
||||
## 8. Generalization
|
||||
|
||||
The pattern is:
|
||||
|
||||
> **Every adaptive EMA whose value gates downstream behavior should use asymmetric Wiener-α with the SAFETY direction admitting at a rate that depends on (a) cumulative data confidence and (b) signal volatility. The asymmetry direction encodes the controller's risk preference; the magnitude adapts to data.**
|
||||
|
||||
This is the same statistical principle as **Bayesian shrinkage**: estimates are pulled toward a conservative prior in the absence of strong evidence, and toward empirical values as evidence accumulates. Welford CV provides the noise floor that gates the shrinkage rate.
|
||||
|
||||
Future controllers using this template:
|
||||
- popart envelope (slot 714) — currently uses fast-up slow-decay; could be ISV-driven too
|
||||
- inventory variance EMA — currently has dead-signal hold; could integrate this template
|
||||
- regime_observer EMAs — currently varied semantics; unify under this framework
|
||||
|
||||
## 9. Done means
|
||||
|
||||
- 3 ISV slots allocated (or 2 if reusing 721)
|
||||
- Both EMA kernels updated with trust-gated asymmetric α
|
||||
- Diag emission for the new state
|
||||
- Local smoke shows cold-start asymmetry → mid-train fading → eval boundary re-engagement
|
||||
- Cluster validation V1-V4 pass
|
||||
- Pearl update: refine `pearl_first_observation_bootstrap` to include trust-gated asymmetric α as the canonical fix pattern
|
||||
Reference in New Issue
Block a user