fix(rl): surfer-scaffold v5.2 — drop wr_excess clamp, full scaffold when novice
alpha-rl-mjsmr (4a4953b01 step 99-249, 2026-06-02): v5.1 with wr_ema=0.248
(below break_even 0.30) produced scaffold_weight = 0.500, not the expected
1.0. Math: `max(0, wr_ema - break_even) = 0` → sigmoid(0) = 0.5 → competence
= 0.5 → w_competence = 0.5. So a novice agent below break-even got
HALF-scaffold instead of full scaffold (entropy dropped to 1.27 at step 249
because of the early directional signal, then ramped back up to 1.96 once
w_decay took over — unstable).
Fix: drop the `max(0, ...)` clamp. Use signed `wr_distance = wr_ema -
break_even` so:
wr=0.20 → sigmoid(30·-0.10) ≈ 0.05 → w_competence ≈ 0.95 (full scaffold ✓)
wr=0.30 → sigmoid(0) = 0.50 → w_competence = 0.50 (half at BE)
wr=0.40 → sigmoid(30·+0.10) ≈ 0.95 → w_competence ≈ 0.05 (pure pnl ✓)
This is the proper symmetric fade around break_even; the asymmetry was a
v5 original-design error. Combined with v5.1 `w_decay = max(0, 2·frac-1)`,
the scaffold should now:
- hold near 1.0 while wr < break_even (novice)
- smoothly fade through 0.5 as wr crosses break_even
- drop toward 0 as wr settles above break_even AND PH decay subsides
Local invariant test:
reward_alignment_surfer_scaffold_invariants OK: 251 rows validated
train[0] bootstrap = 0.966 (unchanged)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -19,19 +19,26 @@
|
||||
// RL_SURFER_SCAFFOLD_WEIGHT_INDEX (753) — w ∈ [0,1] consumed by
|
||||
// rl_fused_reward_pipeline.cu Phase 5
|
||||
//
|
||||
// Math:
|
||||
// competence = confidence(n_trades) × sigmoid(k_sharp × max(0, wr_ema − break_even))
|
||||
// Math (v5.2):
|
||||
// competence = confidence(n_trades) × sigmoid(k_sharp × (wr_ema − break_even))
|
||||
// w_competence = 1 − competence
|
||||
// w_decay = max(0, 2 × frac_alerted − 1) // FIX v5.1: only fires above 50%
|
||||
// w_decay = max(0, 2 × frac_alerted − 1) // v5.1: fires above 50% baseline
|
||||
// w = clamp(max(w_competence, w_decay), 0, 1)
|
||||
//
|
||||
// At training start (n_trades=0, wr_ema=0): competence ≈ 0, w_competence ≈ 1 → w=1 (full scaffold)
|
||||
// After warmup and wr_ema crossing break-even: competence → 1, w → 0 (pure pnl)
|
||||
// If edge-decay PH alerts on >50% of batches: w_decay re-engages PROPORTIONAL to
|
||||
// excess above 50% baseline (alpha-rl-zf6s5 step 5719 verdict 2026-06-02 showed
|
||||
// train-time PH inherently alerts on 75-95% of batches, so the v5 original
|
||||
// `min(1, 2×frac)` pinned w=1 forever even when wr crossed break-even; the new
|
||||
// `max(0, 2×frac − 1)` only fires above 50% and reaches 1.0 only at 100% alert).
|
||||
// v5.2 fix (alpha-rl-mjsmr step 99-249 verdict 2026-06-02): dropped the
|
||||
// `max(0, …)` clamp on `wr_excess`. With the clamp, wr<break_even mapped to
|
||||
// sigmoid(0)=0.5 (half-scaffold at novice agent — wrong). Now wr=0.20 maps
|
||||
// to sigmoid(−3)≈0.05 ⇒ competence≈0.05 ⇒ w_competence≈0.95 (full scaffold,
|
||||
// correct novice-bias). Symmetric around break_even — competence saturates
|
||||
// fast both directions for k_sharp=30.
|
||||
//
|
||||
// Behavioral table:
|
||||
// wr_ema wr-bk sigmoid competence (n_trades≫warmup) w_competence
|
||||
// 0.20 -0.10 0.047 0.05 0.95 (full scaffold)
|
||||
// 0.27 -0.03 0.29 0.29 0.71
|
||||
// 0.30 0.00 0.50 0.50 0.50 (half at break-even)
|
||||
// 0.33 +0.03 0.71 0.71 0.29
|
||||
// 0.40 +0.10 0.953 0.95 0.05 (pure pnl)
|
||||
//
|
||||
// Per `feedback_no_atomicadd`: single-thread launch (1×1×1), no atomic ops.
|
||||
// Per `feedback_cpu_is_read_only`: pure device-side.
|
||||
@@ -71,11 +78,12 @@ extern "C" __global__ void rl_surfer_scaffold_controller(float* isv) {
|
||||
const float warmup_scale = fmaxf(warmup_n * 0.3f, 1.0f); // floor to avoid div-by-zero
|
||||
const float confidence = sigmoidf((n_trades - warmup_n) / warmup_scale);
|
||||
|
||||
// Win-rate excess above break-even, gated nonneg.
|
||||
const float wr_excess = fmaxf(0.0f, wr_ema - break_even);
|
||||
// Win-rate distance from break-even (signed — v5.2 drops the max(0,...) clamp
|
||||
// so wr<break_even properly maps to low competence ⇒ full scaffold).
|
||||
const float wr_distance = wr_ema - break_even;
|
||||
|
||||
// Competence: agent must have BOTH trade-count confidence AND wr above break-even.
|
||||
const float competence = confidence * sigmoidf(k_sharp * wr_excess);
|
||||
const float competence = confidence * sigmoidf(k_sharp * wr_distance);
|
||||
|
||||
// Inverse: scaffold weight from competence.
|
||||
const float w_competence = 1.0f - competence;
|
||||
|
||||
Reference in New Issue
Block a user