From d49fbbe4e2d2ac7ac7f3bdacc22e4404c6b40176 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Mon, 4 May 2026 00:08:39 +0200 Subject: [PATCH] spec(sp11): reframe curiosity as feature + add permanent floor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit User correction: curiosity is the *fix* for the ep1-peak overfitting pathology, not a hazard to defend against. Reframed §7 from "Risks" to "Design notes" — curiosity bound is a signal-relative scale, not a defensive cap. Fix the contradiction this exposed in the formula: previous `curiosity_pressure = stagnant_or_worse * curiosity_bound` went to zero when improving, which would cancel the always-on exploration the §7 narrative now relies on. Replace with permanent-floor pattern per pearl_blend_formulas_must_have_permanent_floor: curiosity_floor = 0.2 * curiosity_bound (CURIOSITY_PERMANENT_FRACTION) curiosity_dynamic = stagnant_or_worse * curiosity_bound curiosity_pressure = max(curiosity_dynamic, curiosity_floor) Now curiosity is always ≥ 20% of bound (anti-overfitting baseline) and rises toward the bound when stagnant (stagnation breaker). Updated unit-test guidance to assert pressure > 0 even at z=+10. CURIOSITY_PERMANENT_FRACTION=0.2 added to Invariant-1 fraction list. Saboteur-rising-with-improvement reframed as adversarial-load feature rather than over-stress risk. --- ...-04-sp11-reward-as-controlled-subsystem.md | 59 +++++++++++++------ 1 file changed, 41 insertions(+), 18 deletions(-) diff --git a/docs/superpowers/specs/2026-05-04-sp11-reward-as-controlled-subsystem.md b/docs/superpowers/specs/2026-05-04-sp11-reward-as-controlled-subsystem.md index 48b2fcef2..e17868d71 100644 --- a/docs/superpowers/specs/2026-05-04-sp11-reward-as-controlled-subsystem.md +++ b/docs/superpowers/specs/2026-05-04-sp11-reward-as-controlled-subsystem.md @@ -136,8 +136,14 @@ for (int c = 0; c < 6; c++) { float curiosity_bound = isv[PNL_REWARD_MAGNITUDE_EMA] * 0.3f; // 0.3 is Invariant-1 fraction scratch[scratch_curiosity_bound] = curiosity_bound; -// Curiosity pressure: scales [0, curiosity_bound] by (1 - improving) -float curiosity_pressure = stagnant_or_worse * curiosity_bound; +// Curiosity pressure: ALWAYS-ON exploration signal scaled by stagnation. +// When improving: floor preserves baseline exploration (anti-overfitting). +// When stagnating: rises toward curiosity_bound to break out of fixed point. +// Permanent floor pattern per pearl_blend_formulas_must_have_permanent_floor — +// max() form, not blend, so curiosity never zeros even when policy is winning. +float curiosity_floor = CURIOSITY_PERMANENT_FRACTION * curiosity_bound; // 0.2 = Invariant-1 fraction +float curiosity_dynamic = stagnant_or_worse * curiosity_bound; +float curiosity_pressure = fmaxf(curiosity_dynamic, curiosity_floor); scratch[scratch_curiosity_pressure] = curiosity_pressure; // Saboteur intensity: continuous mapping of z → [SABOTEUR_MIN, SABOTEUR_MAX] @@ -157,11 +163,13 @@ float adaptive_floor = fmaxf(WEIGHT_HARD_FLOOR, 0.5f * min_grad_ratio); // 0.5 scratch[scratch_weight_floor] = adaptive_floor; ``` -The constants `0.3f` (curiosity fraction), `0.5f` (floor fraction), and -`0.1f` (engagement floor) are Invariant-1 fractions: they bound rates, not -regimes, and remain valid across all regimes. The thresholds themselves -(what counts as "improving", "stagnant", "regressing") are entirely -encoded in the Z-score and sigmoid — no hardcoded threshold survives. +The constants `0.3f` (curiosity fraction of PnL), `0.5f` (floor fraction +of min grad ratio), `0.2f` (CURIOSITY_PERMANENT_FRACTION — exploration +floor as fraction of curiosity_bound), and `0.1f` (engagement floor) are +Invariant-1 fractions: they bound rates, not regimes, and remain valid +across all regimes. The thresholds themselves (what counts as "improving", +"stagnant", "regressing") are entirely encoded in the Z-score and sigmoid +— no hardcoded threshold survives. ### 3.5 Consumer migrations @@ -234,7 +242,7 @@ Plus modifications to existing kernels: ## 6. Tests -- Unit test: synthetic z=0 → curiosity at midpoint; z>0 → curiosity drops; z<0 → curiosity max +- Unit test: synthetic z=0 → curiosity at midpoint; z<0 → curiosity rises toward curiosity_bound; z>0 → curiosity drops to floor (0.2 × curiosity_bound), NOT zero — assert `curiosity_pressure > 0` even at z=+10 - Unit test: weight floor ≥ WEIGHT_HARD_FLOOR always - Unit test: saboteur_intensity ∈ [SABOTEUR_MIN, SABOTEUR_MAX] - Contract test: 20 new slots have FoldReset entries + dispatch arms @@ -242,19 +250,34 @@ Plus modifications to existing kernels: - L40S smoke: 5-epoch run shows weights drifting from initial sentinels, curiosity oscillating with stagnation signal, saboteur_intensity reactive -## 7. Risks and open questions +## 7. Design notes and open questions -- **Risk 1**: Curiosity bonus could destabilize a working policy. Mitigated - by `CURIOSITY_BOUND_INDEX` = 0.3 × PnL EMA — curiosity is always at most - 30% of PnL signal magnitude. Permanent floor pattern means it never - zeros out either, so we always have some exploration. -- **Risk 2**: Per-component weight reweighting could starve a component. +- **Curiosity is a feature, not a hazard**: The current pathology is that + the model converges to a fixed point at ep1 and then declines. Curiosity- + driven exploration is the corrective mechanism — we *want* the model to + keep exploring before settling. The `CURIOSITY_BOUND_INDEX = 0.3 × PnL + EMA` is not a defensive cap against curiosity; it's a signal-relative + scale that keeps the bonus magnitude commensurate with the PnL signal so + it doesn't drown out the primary objective. The permanent floor pattern + (per `pearl_blend_formulas_must_have_permanent_floor`) ensures curiosity + never fully zeros out — exploration pressure is maintained even after + convergence, which is exactly what addresses the ep1-peak overfitting. + In other words: more curiosity when stagnant (stagnation breaker), some + curiosity always (overfitting prevention). +- **Component starvation guard**: Per-component weight reweighting could in + principle starve a contributor. Mitigated by adaptive floor + `≥ 0.5 × min_grad_ratio` ensuring even the least-contributing component + keeps a non-trivial budget. Mitigated by adaptive floor `≥ 0.5 × min_grad_ratio` ensuring even the least-contributing component keeps a non-trivial budget. -- **Risk 3**: Saboteur intensity rising on improvement could over-stress the - policy. Mitigated by [0.5, 2.0] hard bound and engagement-rate self- - correction (per `pearl_engagement_rate_self_correction`). -- **Risk 4**: First-fold cold-start: with only 1-2 val emits available, the +- **Saboteur intensity adapts both directions**: When val sharpe is + improving, saboteur intensity rises to keep the policy under realistic + adversarial load (you don't want a model that only works when execution + is perfect). When stagnating, intensity drops so the policy gets a + cleaner gradient. The [0.5, 2.0] hard bound and engagement-rate self- + correction (per `pearl_engagement_rate_self_correction`) keep this + bounded. +- **First-fold cold-start**: With only 1-2 val emits available, the Z-score is noisy. Mitigated by Pearl A sentinel-bootstrap (first observation replaces sentinel directly) and Pearl D Wiener-α (smooth as data accumulates).