mjzfk diag (commit 53aeef099) showed PPO clip ε pegged at MAX=0.5
for 100% of the 50k-step run, despite kl_pi_ema median = 5.3e-9 and
max = 3.4e-4 (well below KL_TARGET=0.01). The widened clip band is
why ratio_clamp settled at (1+0.5)*10=15 instead of 12, and why π
took no meaningful updates — KL was essentially zero meaning the
policy wasn't moving.
## Root cause
The controller's adaptation was `ratio = KL_TARGET / max(kl_ema, 1e-6)`
— a multiplicative formula with no per-step bound. With kl_ema=1e-11
the kernel sees:
ratio = 0.01 / max(1e-11, 1e-6) = 0.01 / 1e-6 = 10000
target = eps_prev * 10000 = clamped to EPS_MAX
First-observation replace-directly then locks ε at MAX immediately,
and the Wiener α-floor=0.4 blend keeps it there forever.
The cold-start `if (kl_ema == 0.0f) return;` gate only caught EXACT
zero — first observable but tiny KL (typical: cold-start LR not yet
producing measurable policy drift) blows past the gate and saturates
the multiplier.
Same dangerous pattern existed in `rl_target_tau_controller` (uses
`q_div / DIV_TARGET` ratio with no per-step bound) — hadn't bitten
because q_divergence_norm naturally lives in the [0.01, 0.1] range,
but a quiet initialisation could hit it the same way.
## Fix: Schulman-style bounded adaptive KL
Both controllers now use a discrete-step adjustment:
* input > target × TOLERANCE (1.5) → ratio = ADJUST_RATE (1.5)
* input < target / TOLERANCE → ratio = 1/ADJUST_RATE
* in-band → ratio = 1.0 (hold)
Per-step adjustment is bounded at 1.5× (50% expansion / 33%
shrinkage), so no single observation can swing the output across the
[MIN, MAX] range regardless of how outlier-tiny or outlier-huge it
is. After several consecutive out-of-band observations the output
drifts smoothly toward MIN/MAX, but the response is dampened.
## Noise-floor gate
In addition to the bounded step, both controllers now hold their
output when the input EMA is below a noise floor:
KL_NOISE_FLOOR = KL_TARGET × 0.01 = 1e-4 (ppo_clip)
DIV_NOISE_FLOOR = DIV_TARGET × 0.01 = 1e-4 (target_tau)
Two orders of magnitude below the design target = "policy isn't
actually updating" / "Q hasn't started learning" / numerical noise.
Reacting to this signal can only mis-tune the controller — we'd
rather hold a sane default than chase noise.
The TARGET-derived floors (rather than absolute constants) mean
adjusting KL_TARGET / DIV_TARGET shifts the floors proportionally —
consistent with the existing pattern.
## ISV discipline
Per `feedback_isv_for_adaptive_bounds`: KL_TARGET and DIV_TARGET
themselves are PPO/DQN design constants (like REWARD_CLAMP_WIN =
1.0 in apply_reward_scale, or LR_BOOTSTRAP = 1e-3 in
rl_lr_controller). The NOISE_FLOOR derives from them, and the
TOLERANCE / ADJUST_RATE constants are structural Schulman-recipe
parameters that don't adapt at runtime. Existing diag exposes both
controllers' outputs (isv_out.ppo_clip_eps, isv_out.target_tau) and
inputs (isv_ema_in.kl_pi, isv_ema_in.q_divergence) so the
controller behaviour is fully observable from the JSONL — no new
slots needed.
## Verified gates (local sm_86)
G1 isv_bootstrap ✅
G3 controllers ✅ (controllers still move outputs when fed real EMAs)
G4 target_update ✅
integrated_smoke ✅
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
6a58ac9465