8df1c7eea2f4e280aeb89c2b341d9e8135e46465
3 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
jgrusewski
|
083a88f7c3 |
feat(rl): adaptive controller floors — 12 controllers, all signal-driven
Replaces hardcoded thresholds AND clamp bounds across 12 RL controllers
with observed-signal-driven ISV-slot bounds. Eliminates the architectural
failure mode that surfaced in walk-forward fold 0 (alpha-rl-m9cx5) and
fold 1 (alpha-rl-jgdh6): under Phase 4.5 advantage normalization, eight
controllers (PPO clip, target_tau, rollout_steps, entropy_coef, per_α,
gamma, reward_scale, q_distill_lambda) saturated at extrema within ~50
steps and stayed pinned for the rest of training — same pattern across
two different data slices, confirming the bug is structural rather than
data-size-dependent.
Mechanism: each controller's noise floor was hardcoded as a small
fraction of its target (`_NOISE_FLOOR_FRAC = 0.01f`), calibrated against
a pre-Phase-4.5 signal regime. Phase 4.5 normalization reduces operating
KL by ~50× — observed signal stays below the 1%-of-target floor, the
Schulman widen path fires continuously (asymmetric in the wrong
direction), and ε hits MAX 0.50 within ~50 training steps. ksll2's full-
data run (n_folds=1) happened to escape via a single above-band KL
observation that triggered tighten; both walk-forward folds (3 and 6
files) did not.
Per `feedback_adaptive_not_tuned`, `feedback_isv_for_adaptive_bounds`,
`pearl_controller_anchors_isv_driven`: every threshold now derives from
observed signal statistics (Welford online variance) rather than
constants calibrated against a prior signal regime. User explicitly
expanded scope mid-implementation: "if all clamps ISV bound should be
added to this spec and tasks" — applying the principle consistently
means hardcoded MIN/MAX clamp bounds count too, not just the saturating
noise floors. 12 controllers, single atomic commit per
`feedback_no_partial_refactor`.
Spec: docs/superpowers/specs/2026-05-30-adaptive-controller-floor-design.md
Plan: docs/superpowers/plans/2026-05-30-adaptive-controller-floor-plan.md
# New kernel
- rl_signal_variance_update.cu (80 LOC) — Welford online variance.
Single-thread single-block. Sentinel-zero skip per pearl. Per-controller
Welford triple (count, mean, M²) drives every adaptive noise floor.
# Per-controller refactors (12 .cu files)
- ppo_clip + target_tau + rollout_steps + entropy_coef + per_α —
adaptive noise floor = max(target × 0.5, sqrt(observed_var) × 2);
asymmetric Schulman (tighten on single observation, widen requires 3
consecutive below-band).
- gamma (Special G) — hardcoded GAMMA_MIN = 0.995 → adaptive via
Welford MEAN of trade duration. Per spec Q6 resolution: the Welford
mean's natural N-smoothing lag breaks the gamma↔trade_duration
feedback loop without explicit step-period gating. 100-observation
warmup falls back to EMA before Welford has enough samples.
- reward_scale (Special R) — asymmetric DECREASE rate cap (5% per
step) on both bootstrap-replace and Wiener-blend paths; bootstrap-
fraction floor (10% of bootstrap) until 100 trades close.
- q_distill_lambda (Special Q) — hardcoded MIN_LAMBDA = 0.05 → adaptive
via Welford on q_distill_kl_ema: max(0.001, std × 0.05).
- v_blend_alpha (Phase 4.4) — 5 hardcoded constants → 5 ISV slots
(DEAD_SIGNAL_FLOOR, TARGET_TRACK_RATIO, SCHULMAN_STEP, EMA_ALPHA,
BOOTSTRAP_ALPHA) + adaptive dead-signal floor from V_scalar magnitude
variance.
- ppo_ratio_clamp — adaptive MIN/MAX from observed log-ratio variance.
Architectural 2.0 absolute floor preserved (don't degenerate to
vanilla policy gradient).
- reward_clamp — V_BOUND_FLOOR, V_BOUND_EWMA_ALPHA, MIN_WIN, MIN_RATIO,
MAX_RATIO, MIN/MAX_MARGIN, MARGIN_TOLERANCE/ADJUST_RATE,
CLIP_RATE_EMA_ALPHA → 10 ISV slots.
- gate_threshold — hardcoded alpha = 0.01 → ISV slot 638.
- Clamp-bound expansion: EPS_MIN/MAX, TAU_MIN, ROLLOUT_MIN/MAX,
COEF_MIN/MAX, PER_ALPHA_MIN/MAX, GAMMA_MAX, MAX_LAMBDA, KL_TOLERANCE,
LAMBDA_RAMP_RATE, LAMBDA_DECAY_RATE → all ISV.
- WIENER_ALPHA_FLOOR shared across 9 controllers → single ISV slot 659.
# Trainer integration (integrated.rs)
- rl_signal_variance_update kernel loaded + helper method
`launch_rl_signal_variance_update`.
- Per-step Welford launches for 9 controller inputs, placed between
EMA producers and rl_fused_controllers in the per-step pipeline.
- Input-slot lookup array for rl_fused_controllers updated: rollout_steps
now consumes RL_ADV_VAR_PRE_NORM_INDEX (emitted by
rl_advantage_normalize before in-place normalize) instead of the
post-norm advantage_var_ratio (definitionally ~0 under Phase 4.5).
- ~30 new ISV bootstrap entries in with_controllers_bootstrapped.
# ISV slot allocation (isv_slots.rs)
- 72 new slots, RL_SLOTS_END 588 → 660. +288 bytes mapped-pinned.
- 9 Welford variance triples + 5 asymmetric Schulman counters +
3 new input signals (var_pre_norm, gamma_min_adaptive, reward_mag) +
v_blend (5 + 3 Welford) + ppo_ratio_clamp (1 + 3 Welford) +
reward_clamp (5) + gate_threshold (1) + q_distill (1) + 20 clamp bounds +
shared wiener floor.
# Bootstrap-clamp consistency fix (caught by Task 16 testing)
The original draft bootstrapped RL_EPS_BOOTSTRAP at 0.01 (= KL target),
but EPS_MIN was 0.05 — bootstrap value below clamp range. First post-
bootstrap step always snapped ε to MIN regardless of signal direction
("snap to MIN" behavior the unit tests surfaced). Fixed by bootstrapping
ε at MIN (0.05) so asymmetric Schulman operates from a valid state.
# Validation
- cargo build --release: clean
- cargo build --tests: clean
- 3 GPU Welford kernel tests (G1 constant→0var, G2 sequence→known var,
G3 sentinel skip): all pass
- 5 GPU adaptive-floor invariant tests (G3 PPO clip holds at bootstrap
when signal below floor, G4 tightens on single above-band, G5 widens
only after 3 consecutive below-band, G6 post-warmup uses Welford mean,
G6 pre-warmup uses EMA): all pass
- integrated_trainer_smoke (full GPU pipeline): passes
- 1k local smoke at b=128: 1000/1000 steps, completed_clean, no NaN,
controllers genuinely adapting (γ 0.90→0.974, per_α 0.40→0.52,
q_distill_λ 0.05→0.21, ε held at 0.05 = MIN per Phase 4.5 small-KL
regime — was previously stuck at MAX 0.50 throughout fold 0/1)
- compute-sanitizer memcheck b=128 5 steps: 0 errors
Cluster validation (G7-G9) submitted as follow-up runs.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
96abc364b5 |
fix(rl): asymmetric Q→π distillation λ controller — ramp fast, decay slow
Root cause of q_pi_agree collapse: the symmetric Schulman controller decayed λ from 0.5 to 0.001 in 34 steps (÷1.2/step), killing Q→π coupling. Once decoupled, Q and π learned independent policies, making q_pi_agree drop to 1e-22. Three fixes: 1. MIN_LAMBDA 0.001 → 0.05: Q pull never drops below 5% strength 2. Asymmetric rates: ramp 1.2×/step (4 steps to double), decay 0.998×/step (347 steps to halve). Coupling establishes fast and persists for thousands of steps. 3. Dead zone tolerance 1.5× → 3.0×: natural KL fluctuation stays in-band instead of triggering constant ramp/decay cycles. Seed λ raised to 0.1 so distillation is active from step 0. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jgrusewski
|
e87e0b0774 |
feat(rl): adaptive λ_distill controller + reward_scale MIN ISV
Two architectural fixes from rljzl in-flight analysis (ultrathink
deep dive on actions a7/a8 + per-action calibration):
(1) λ_distill: static → controller-driven via Schulman bounded step
wwcsz showed Q→π KL EMA dropped 2.10 → 0.30 with λ=0.01, then
rljzl bumped to 0.05. Static λ is design intuition; KL is the
natural feedback signal:
if KL > target × 1.5 → λ *= 1.2 (Q not landing, pull harder)
if KL < target / 1.5 → λ /= 1.2 (Q absorbed, relax)
Bounds [MIN=0.001, MAX=1.0]. Target KL seeded 0.1 (slot 491).
New kernel `rl_q_distill_lambda_controller.cu`. Runs after the
distill kernel writes KL_EMA each step.
(2) REWARD_SCALE_MIN: hardcoded 1e-3 → ISV-driven 1e-4
wwcsz audit (mean_abs_pnl_ema mean=920, max=49437, p99=high):
the controller wanted scale ≈ 3.5e-4 when EMA spiked to 2871
but pegged at 1e-3, letting scaled rewards exceed unit support
and wasting C51 atom resolution on outliers. ISV slot 492
permits runtime re-tuning; default 1e-4 admits one more order
of magnitude before pegging. Per user-stated "floors and clamp
bounds" exemption — ISV-resident for tunability, not because
required.
Diag exposes q_distill_kl_target + reward_scale_min so the new
adaptation chains are observable.
Investigation (ultrathink): actions 7/8 (TrailTighten/TrailLoosen)
have ZERO consumers across the codebase. Spec'd as "ISV mutation"
in actions_to_market_targets.cu header but no slot, no mutation
kernel, no stop-check kernel. ~10% of wwcsz policy mass goes to
dead no-ops. Documented in
`pearl_dead_trail_stop_actions_a7_a8.md` — implementation
deferred to its own SP (per-batch trail_distance buffer +
mutation kernel + stop-check integration with LobSim apply_fill_to_pos
per `pearl-stop-checks-run-at-deadline-cadence`). N_ACTIONS=9
preserved; alternative refactor to 7 actions captured as
"Path B" in the pearl.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|