Two coordinated fixes for the alpha-rl-frt7s findings:
## Issue 1: n_rollout_steps controller was write-only
ISV consumer audit confirmed: 7 of 8 RL controllers had a non-
controller consumer in the per-step path; n_rollout_steps had ZERO.
The controller adapted its output between 256-8192 but nothing read
it. Bit-identical losses between cvf86 and frt7s confirmed: even
fixing the target (0.1 → 5.0) and putting the controller into
healthy HOLD/SHRINK/WIDEN distribution had zero behavioral impact
because no downstream code gated on the emitted value.
### Fix: wire as DQN-replay + PPO+V K-loop multiplier
step_with_lobsim now wraps (sample_and_gather + step_synthetic +
PER priority update) in a K-loop where:
K = clamp(isv[RL_N_ROLLOUT_STEPS_INDEX] / 1024, 1, 8)
Mapping:
* isv[404] = 256 (MIN) → K = 1 (current behavior)
* isv[404] = 2048 (BOOTSTRAP) → K = 2
* isv[404] = 8192 (MAX) → K = 8
Each iteration re-samples PER (different transitions per Adam step)
and runs full Q + π + V forward + backward + Adam. Adapts the
training:env ratio so noisy-advantages regimes get more gradient
samples per env step without slowing env stepping. Directly
addresses the b_size=1 gradient starvation that left l_q stuck at
2.82 in frt7s.
Semantic fit: n_rollout_steps's design intent ("noisy advantages →
need more samples per update") now drives "more training updates
per env step" — equivalent semantics, fits the b_size=1
architecture without requiring a PPO rollout buffer refactor.
`last_k_updates` field tracks the per-step K value for diag.
## Issue 2: LR plateau-decay Q-lock
frt7s deep dive showed:
* Q best=2.3230 locked at step ~783 from a brief downward
excursion during early-training noise
* loss_ema range across 50k steps: [2.323, 3.113]; mean 2.819,
std 0.104
* ZERO steps had loss_ema < best in entire run (let alone <
best × 0.99 = 2.30 threshold)
* 7 LR halvings drove all heads to LR_MIN = 1e-5 by step 7783
* At 1e-5, Q's per-step Adam update is too small to escape;
l_q stayed at ~2.82 for 42k more steps
The plateau-decay is CORRECTLY identifying "model has stopped
improving" — the fix isn't to make plateau detection less
sensitive (loosening threshold to 0.95/0.90 still finds zero
improvements). The fix is to raise the floor LR so the model
has enough learning rate to escape the noise-locked best.
### Fix: LR_MIN 1e-5 → 1e-4 + WARMUP_STEPS 500 → 2000
* LR_MIN raised 10× — even at the plateau-decay floor the model
gets meaningful gradient. Still 10× below LR_BOOTSTRAP=1e-3
so the controller has full dynamic range.
* WARMUP_STEPS raised 4× — gives loss_ema 2000 observations
(≈145 EMA half-lives at α=0.05) to settle BEFORE best is
locked. Prevents the "lucky early excursion locks unreachable
bar" failure mode.
## Diag bake-in
JSONL gains `k_updates` field (per-step K value from the n_rollout
loop) so post-hoc analysis can correlate the K-multiplier with
loss trajectories.
## Verified gates (local sm_86)
G1 isv_bootstrap ✅
G3 controllers ✅
G4 target_update ✅
integrated_smoke ✅
## Quality-first scope decision
User requested "quality over speed". Considered alternatives:
* Building a proper PPO rollout buffer (Issue 1) — significant
refactor, ~1-2 days. K-loop interpretation chosen instead
because it (a) matches the controller's design intent, (b)
requires no buffer/gradient-accumulation infrastructure, (c)
directly addresses Q learning starvation by giving more
gradient samples per env step.
* Encoder LR decoupling (Issue 2) — encoder receives gradient
from all head backward kernels with their own LRs; treating
the encoder separately would require restructuring all
backward kernels. LR_MIN raise + WARMUP extension gives the
same benefit at the head level without that scope.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1d8ef94848