Folds two new producer signals into SP4 scope, eliminating the earlier
"carved out" exception:
WEIGHT DECAY (per param-group, 7 new ISV slots):
λ = |w·g| / max(||w||², ε)
Derived from equilibrium analysis of d/dt(||w||²) = 2(w·g) - 2λ||w||².
The equilibrium gradient-projection-onto-weight-direction divided by
weight norm. Same theoretical-derivation category as Adam β values.
EMA half-life α=0.005 (~140 steps). Bootstrap 1.0.
L1 LAMBDA (trunk only, 1 new ISV slot — NOVEL PEARL):
λ = (mean(|g|) / mean(|w|)) × D
where D = (log K - H_observed) / log K is gradient-direction entropy deficit
and H_observed = -Σ p[i]·log p[i], p[i] = ||g[:,i]|| / Σ ||g[:,j]||
L1 regularization-strength derives from gradient-direction entropy
deficit across input features. When gradient is uniform across features
(D≈0): network hasn't differentiated, λ=0 (no pruning). When gradient
concentrates on few features (D≈1): network has identified what matters,
λ ramps up to prune the rest. Self-curriculum — L1 strength tracks the
emergence of feature differentiation.
This extends pearl_adaptive_moe_lambda (regularization strength = EMA-
tracked deficit of regularized quantity) to feature-redundancy domain.
Pearl-name candidate (post-validation): pearl_signal_driven_regularisation_strength.
Bootstrap λ=0 means cold-start = no L1 pruning; ramp-up only after
gradient differentiates. Worst-case behavior is "L1 disabled" — graceful.
Total ISV slot count: 28 → 36. Total producer kernels: 28 → 36.
Effort estimate: 3000-4000 → 3500-4500 LOC, 1-1.5 → 1.5-2 weeks.
Acknowledged limitations updated: removed item #8 (carve-out) since
no carve-outs remain. Added items for L1 pearl novelty (untested) and
weight decay equilibrium-formula non-stationarity. Both have graceful
worst-case behavior and explicit validation criteria (#10 and #11) to
detect anomalies.
SP4 now closes 100% of the magnitude/regularization surface — no
hardcoded scalars remain in the entire chain. AdamW config fields for
weight_decay and l1_lambda removed from HyperParams to prevent
accidental hardcoding regression.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
21b6058e07