R9 cluster smoke (alpha-rl-fv7xz step 7) caught a real production
bug the local sm_86 smoke missed: with b_size=1 and 1000 steps of
real ES MBP-10 data, the 4 controllers I'd previously declared
"intentionally hardcoded with documented rationale" (τ, ε, n_roll,
scale) ALL exhibited cold-start migration to clamp bounds before any
real signal arrived.
The smoking-gun trace (per-step diag JSONL):
step 0: scale=400 ε=0.32 n_roll=1638 τ=0.0034 dones=0
step 1: scale=640 ε=0.39 n_roll=1311 τ=0.0024 dones=0
step 6: scale=972 ε=0.49 n_roll=429 τ=0.0011 dones=0
step 7: scale=583 ε=0.50 n_roll=360 τ=0.0011 DONES=1 rew_sum=485,385.84
The real realized PnL on step 7's closed trade was $832. The
reward_scale controller had migrated from bootstrap 1.0 → 972
during steps 1-6 (no signal, ratio degenerated to 1/EPS_PNL=1000
→ clamped to MAX → Wiener α=0.4 pulled prev toward MAX every step).
At step 7's first closed trade, scale=583 multiplied the real
$832 PnL into a 485,256 reward, fed straight into Q/V backward.
l_v spiked to 15.9 from that single step before the controller
recovered.
## Initial diagnosis: wrong
Two commits ago I added a pearl
(pearl_hardcoded_bootstrap_target_collision) claiming
multiplicative-target controllers were SAFE from the bootstrap-
target-coincidence anti-pattern because their bootstrap IS the
initial prev. That was wrong. They have a DIFFERENT failure mode
that's just as bad: at sentinel input the ratio degenerates (to 0
or ∞), the target slams to a clamp, and the Wiener α-floor drags
prev toward the clamp every step until signal arrives.
## Fix: cold-start gate (4 kernels, ~2 lines each)
```cuda
const float input_ema = isv[input_slot];
if (input_ema == 0.0f) return; // hold bootstrap; adapt only on real signal
```
Applied to:
* rl_target_tau_controller (multiplicative, q_div input)
* rl_ppo_clip_controller (multiplicative, kl_pi input)
* rl_rollout_steps_controller (multiplicative, adv_var_ratio input)
* rl_reward_scale_controller (reciprocal, mean_abs_pnl input)
The bootstrap value stays canonical (PPO ε=0.2, target-net τ=0.005,
PPO rollout=2048, reward scale=1.0 raw passthrough) until the first
non-zero EMA observation. Only then does the per-step Wiener blend
begin moving prev toward the formula's target. This matches
`pearl_first_observation_bootstrap`'s "sentinel = 0 means no data —
adapt against signal, not noise" mandate, just applied at the
controller layer rather than the EMA producer layer.
## Why local sm_86 smoke missed this
The R9 G3 local test seeded every EMA input with a non-zero value
BEFORE firing the controllers. That's a real-signal scenario by
construction — the cold-start gate is a no-op there. The test still
proves "controllers respond to real signal" but cannot detect
"controllers misbehave at sentinel input" because it never feeds
sentinel input.
Adding a `g3b_controllers_hold_bootstrap_at_sentinel_input` test
would be sensible for follow-up. For now the cluster smoke is the
canonical witness — re-run will confirm scale/ε/n_roll/τ all hold
at bootstrap until the first closed trade.
## Pearl updated
The existing `pearl_hardcoded_bootstrap_target_collision.md` is
amended to document BOTH failure modes (additive vs multiplicative/
reciprocal) and BOTH fixes (derive-from-input for additive, cold-
start gate for multiplicative). The canonical incident
(alpha-rl-fv7xz step 7) is captured with the actual scale=583 ×
$832 = 485k trace.
## Verified gates (post-fix, local sm_86)
G1 isv_bootstrap ✅ unchanged (bootstrap values intact)
G3 controllers_emit ✅ all 7 still move when fed real EMAs
(test pre-seeds non-zero inputs)
G4 target_soft_update ✅ unchanged
G6 r7d_per_wiring ✅ unchanged
R3 ema/advantage (3 tests) ✅ unchanged
R4 action kernels (3 tests) ✅ unchanged
end integrated_trainer_smoke ✅ all 5 head losses finite, unchanged
## Next: re-submit cluster smoke
The fix is local. Push + ./scripts/argo-alpha-rl.sh --n-steps 1000
will re-validate on real ES MBP-10 with the cold-start gates active.
Expected diag at step 7: scale=1.0 (bootstrap, unchanged) for the
first trade close — no 485k reward spike. The diff between this and
the prior smoke is the load-bearing signal that the fix worked.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
069f31286c