spec(sp11): reframe curiosity as feature + add permanent floor
User correction: curiosity is the *fix* for the ep1-peak overfitting pathology, not a hazard to defend against. Reframed §7 from "Risks" to "Design notes" — curiosity bound is a signal-relative scale, not a defensive cap. Fix the contradiction this exposed in the formula: previous `curiosity_pressure = stagnant_or_worse * curiosity_bound` went to zero when improving, which would cancel the always-on exploration the §7 narrative now relies on. Replace with permanent-floor pattern per pearl_blend_formulas_must_have_permanent_floor: curiosity_floor = 0.2 * curiosity_bound (CURIOSITY_PERMANENT_FRACTION) curiosity_dynamic = stagnant_or_worse * curiosity_bound curiosity_pressure = max(curiosity_dynamic, curiosity_floor) Now curiosity is always ≥ 20% of bound (anti-overfitting baseline) and rises toward the bound when stagnant (stagnation breaker). Updated unit-test guidance to assert pressure > 0 even at z=+10. CURIOSITY_PERMANENT_FRACTION=0.2 added to Invariant-1 fraction list. Saboteur-rising-with-improvement reframed as adversarial-load feature rather than over-stress risk.
This commit is contained in:
@@ -136,8 +136,14 @@ for (int c = 0; c < 6; c++) {
|
||||
float curiosity_bound = isv[PNL_REWARD_MAGNITUDE_EMA] * 0.3f; // 0.3 is Invariant-1 fraction
|
||||
scratch[scratch_curiosity_bound] = curiosity_bound;
|
||||
|
||||
// Curiosity pressure: scales [0, curiosity_bound] by (1 - improving)
|
||||
float curiosity_pressure = stagnant_or_worse * curiosity_bound;
|
||||
// Curiosity pressure: ALWAYS-ON exploration signal scaled by stagnation.
|
||||
// When improving: floor preserves baseline exploration (anti-overfitting).
|
||||
// When stagnating: rises toward curiosity_bound to break out of fixed point.
|
||||
// Permanent floor pattern per pearl_blend_formulas_must_have_permanent_floor —
|
||||
// max() form, not blend, so curiosity never zeros even when policy is winning.
|
||||
float curiosity_floor = CURIOSITY_PERMANENT_FRACTION * curiosity_bound; // 0.2 = Invariant-1 fraction
|
||||
float curiosity_dynamic = stagnant_or_worse * curiosity_bound;
|
||||
float curiosity_pressure = fmaxf(curiosity_dynamic, curiosity_floor);
|
||||
scratch[scratch_curiosity_pressure] = curiosity_pressure;
|
||||
|
||||
// Saboteur intensity: continuous mapping of z → [SABOTEUR_MIN, SABOTEUR_MAX]
|
||||
@@ -157,11 +163,13 @@ float adaptive_floor = fmaxf(WEIGHT_HARD_FLOOR, 0.5f * min_grad_ratio); // 0.5
|
||||
scratch[scratch_weight_floor] = adaptive_floor;
|
||||
```
|
||||
|
||||
The constants `0.3f` (curiosity fraction), `0.5f` (floor fraction), and
|
||||
`0.1f` (engagement floor) are Invariant-1 fractions: they bound rates, not
|
||||
regimes, and remain valid across all regimes. The thresholds themselves
|
||||
(what counts as "improving", "stagnant", "regressing") are entirely
|
||||
encoded in the Z-score and sigmoid — no hardcoded threshold survives.
|
||||
The constants `0.3f` (curiosity fraction of PnL), `0.5f` (floor fraction
|
||||
of min grad ratio), `0.2f` (CURIOSITY_PERMANENT_FRACTION — exploration
|
||||
floor as fraction of curiosity_bound), and `0.1f` (engagement floor) are
|
||||
Invariant-1 fractions: they bound rates, not regimes, and remain valid
|
||||
across all regimes. The thresholds themselves (what counts as "improving",
|
||||
"stagnant", "regressing") are entirely encoded in the Z-score and sigmoid
|
||||
— no hardcoded threshold survives.
|
||||
|
||||
### 3.5 Consumer migrations
|
||||
|
||||
@@ -234,7 +242,7 @@ Plus modifications to existing kernels:
|
||||
|
||||
## 6. Tests
|
||||
|
||||
- Unit test: synthetic z=0 → curiosity at midpoint; z>0 → curiosity drops; z<0 → curiosity max
|
||||
- Unit test: synthetic z=0 → curiosity at midpoint; z<0 → curiosity rises toward curiosity_bound; z>0 → curiosity drops to floor (0.2 × curiosity_bound), NOT zero — assert `curiosity_pressure > 0` even at z=+10
|
||||
- Unit test: weight floor ≥ WEIGHT_HARD_FLOOR always
|
||||
- Unit test: saboteur_intensity ∈ [SABOTEUR_MIN, SABOTEUR_MAX]
|
||||
- Contract test: 20 new slots have FoldReset entries + dispatch arms
|
||||
@@ -242,19 +250,34 @@ Plus modifications to existing kernels:
|
||||
- L40S smoke: 5-epoch run shows weights drifting from initial sentinels,
|
||||
curiosity oscillating with stagnation signal, saboteur_intensity reactive
|
||||
|
||||
## 7. Risks and open questions
|
||||
## 7. Design notes and open questions
|
||||
|
||||
- **Risk 1**: Curiosity bonus could destabilize a working policy. Mitigated
|
||||
by `CURIOSITY_BOUND_INDEX` = 0.3 × PnL EMA — curiosity is always at most
|
||||
30% of PnL signal magnitude. Permanent floor pattern means it never
|
||||
zeros out either, so we always have some exploration.
|
||||
- **Risk 2**: Per-component weight reweighting could starve a component.
|
||||
- **Curiosity is a feature, not a hazard**: The current pathology is that
|
||||
the model converges to a fixed point at ep1 and then declines. Curiosity-
|
||||
driven exploration is the corrective mechanism — we *want* the model to
|
||||
keep exploring before settling. The `CURIOSITY_BOUND_INDEX = 0.3 × PnL
|
||||
EMA` is not a defensive cap against curiosity; it's a signal-relative
|
||||
scale that keeps the bonus magnitude commensurate with the PnL signal so
|
||||
it doesn't drown out the primary objective. The permanent floor pattern
|
||||
(per `pearl_blend_formulas_must_have_permanent_floor`) ensures curiosity
|
||||
never fully zeros out — exploration pressure is maintained even after
|
||||
convergence, which is exactly what addresses the ep1-peak overfitting.
|
||||
In other words: more curiosity when stagnant (stagnation breaker), some
|
||||
curiosity always (overfitting prevention).
|
||||
- **Component starvation guard**: Per-component weight reweighting could in
|
||||
principle starve a contributor. Mitigated by adaptive floor
|
||||
`≥ 0.5 × min_grad_ratio` ensuring even the least-contributing component
|
||||
keeps a non-trivial budget.
|
||||
Mitigated by adaptive floor `≥ 0.5 × min_grad_ratio` ensuring even the
|
||||
least-contributing component keeps a non-trivial budget.
|
||||
- **Risk 3**: Saboteur intensity rising on improvement could over-stress the
|
||||
policy. Mitigated by [0.5, 2.0] hard bound and engagement-rate self-
|
||||
correction (per `pearl_engagement_rate_self_correction`).
|
||||
- **Risk 4**: First-fold cold-start: with only 1-2 val emits available, the
|
||||
- **Saboteur intensity adapts both directions**: When val sharpe is
|
||||
improving, saboteur intensity rises to keep the policy under realistic
|
||||
adversarial load (you don't want a model that only works when execution
|
||||
is perfect). When stagnating, intensity drops so the policy gets a
|
||||
cleaner gradient. The [0.5, 2.0] hard bound and engagement-rate self-
|
||||
correction (per `pearl_engagement_rate_self_correction`) keep this
|
||||
bounded.
|
||||
- **First-fold cold-start**: With only 1-2 val emits available, the
|
||||
Z-score is noisy. Mitigated by Pearl A sentinel-bootstrap (first
|
||||
observation replaces sentinel directly) and Pearl D Wiener-α (smooth as
|
||||
data accumulates).
|
||||
|
||||
Reference in New Issue
Block a user