fix(rl): R9 local-smoke prep — two test-fixture bugs
Two `pearl_tests_must_prove_not_lock_observations` violations
surfaced during the R9 pre-cluster validation sweep on the dev RTX
3050 Ti (sm_86). Neither was a bug in the trainer or kernels — both
were test fixtures that asserted observed-value coincidences rather
than invariants. Per the canonical pearl, observed-value tests
become bug-locks (the SP16 T3 sp16_phase3_alpha_low_in_steady_state
incident was an assertion `α<0.40` matching the bug itself).
## g3_per_step_controllers_move_isv_outputs_when_fed_real_emas
The fixture fed `RL_TD_KURTOSIS_EMA = 10.0` to the rl_per_alpha
controller, expecting ISV[405] to move off its bootstrap 0.6 after
the Wiener blend. But the kernel's target formula at td_kurtosis=10
maps to **exactly** the bootstrap:
target = 0.4 + 0.2 × (10 − 3) / 7 = 0.6
The Wiener blend `(1−α)·prev + α·target` then produces 0.6 from any
α, so the controller can't move off bootstrap. The assertion was
asserting a coincidence — fixed by picking `td_kurtosis = 20.0`
which lands at `target = 0.886`, distinct from the 0.6 bootstrap.
With the fix all 7 controllers move (γ→0.9, τ→0.023, ε→0.14,
coef→0.0154, n_roll→2867, per_α→0.714, scale→0.608).
The kernel itself is correct — the test was wrong.
## integrated_trainer_step_with_lobsim_runs_without_panic
Asserted `λ_sum ≈ 1.0` for the loss-balance λs. But
`LossLambdas::default()` returns each λ=1.0 (sum = 5.0) with the
`/5.0` divide applied at the trainer's loss-combine site so each
head's contribution is `lambda/5.0`. The "sum=1" assertion was
based on a normalization that the trainer never used. Loosened to
the actual invariant we care about ("every head has a finite
positive λ so the encoder receives real-valued gradient") which
survives any future controller-driven λ re-weighting.
## R9 local-smoke results (all gates green on sm_86)
```
G1 isv_bootstrap ✅ γ=0.99 τ=0.005 ε=0.2
coef=0.01 n_roll=2048
per_α=0.6 scale=1.0
R3 r3_ema_advantage (3 tests) ✅ bootstrap + per-step EMA +
advantage/return formula
R4 r4_action_kernels (3 tests) ✅ Thompson + argmax + log_pi
G3 controllers_emit ✅ all 7 ISV outputs moved
G4 target_soft_update ✅ Polyak τ=0.005 applied
G6 r7d_per_wiring ✅ buffer 0→5→8 + sample size
end integrated_trainer_smoke ✅ all 5 head losses finite
```
Confirms R7c-data + R7d run end-to-end on real CUDA. Next R9 step
(cluster smoke via scripts/argo-alpha-rl.sh + multi-fold G8) requires
git push + cluster credits — paused per the chosen R9 path "stop
before cluster submission."
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -105,16 +105,26 @@ fn integrated_trainer_step_with_lobsim_runs_without_panic() {
|
||||
assert!(stats.l_v.is_finite(), "l_v is not finite: {}", stats.l_v);
|
||||
assert!(stats.l_total.is_finite(), "l_total is not finite: {}", stats.l_total);
|
||||
|
||||
// λs sum to ~1.0 (loss-balance default before any controller fires).
|
||||
let lambda_sum = stats.lambdas.bce
|
||||
+ stats.lambdas.q
|
||||
+ stats.lambdas.pi
|
||||
+ stats.lambdas.v
|
||||
+ stats.lambdas.aux;
|
||||
assert!(
|
||||
(lambda_sum - 1.0).abs() < 1e-4,
|
||||
"loss-balance λs should sum to 1.0; got {lambda_sum}"
|
||||
);
|
||||
// Each λ is finite + positive. The per-head divide-by-5 happens
|
||||
// at the trainer's loss-combine site (see `loss_balance.rs::Default`
|
||||
// doc), so each λ defaults to 1.0 (sum 5.0) on first observation.
|
||||
// Asserting the SUM equals a specific value locks the test against
|
||||
// any future controller-driven re-weighting (canonical
|
||||
// `pearl_tests_must_prove_not_lock_observations`); the invariant
|
||||
// we actually care about is "every head has a finite positive
|
||||
// weight so the encoder gets a real-valued gradient signal."
|
||||
for (name, lam) in [
|
||||
("bce", stats.lambdas.bce),
|
||||
("q", stats.lambdas.q),
|
||||
("pi", stats.lambdas.pi),
|
||||
("v", stats.lambdas.v),
|
||||
("aux", stats.lambdas.aux),
|
||||
] {
|
||||
assert!(
|
||||
lam.is_finite() && lam > 0.0,
|
||||
"lambda[{name}] must be finite + positive; got {lam}"
|
||||
);
|
||||
}
|
||||
|
||||
eprintln!(
|
||||
"R6 OK — step_with_lobsim end-to-end: l_bce={} l_q={} l_pi={} l_v={} l_aux={} l_total={}",
|
||||
|
||||
@@ -118,7 +118,13 @@ fn g3_per_step_controllers_move_isv_outputs_when_fed_real_emas() {
|
||||
(RL_KL_PI_EMA_INDEX, 0.1), // → rl_ppo_clip
|
||||
(RL_ENTROPY_OBSERVED_EMA_INDEX, 0.5), // → rl_entropy_coef
|
||||
(RL_ADVANTAGE_VAR_RATIO_EMA_INDEX, 5.0), // → rl_rollout_steps
|
||||
(RL_TD_KURTOSIS_EMA_INDEX, 10.0), // → rl_per_alpha
|
||||
// td_kurtosis = 20.0 chosen so the per_α controller's target
|
||||
// (0.4 + 0.2 × (20 − 3) / 7 = 0.886) is distinct from the
|
||||
// bootstrap 0.6 — a fixture value of 10.0 mapped exactly to
|
||||
// target = 0.6, making the Wiener blend produce 0.6 regardless
|
||||
// of α and rendering the "moved off bootstrap" assertion
|
||||
// unfalsifiable (canonical `pearl_tests_must_prove_not_lock_observations`).
|
||||
(RL_TD_KURTOSIS_EMA_INDEX, 20.0), // → rl_per_alpha
|
||||
(RL_MEAN_ABS_PNL_EMA_INDEX, 50.0), // → rl_reward_scale
|
||||
];
|
||||
for (slot, obs_val) in inputs {
|
||||
|
||||
Reference in New Issue
Block a user