gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(ml-alpha): Phase 2A-D B1 — gate LR multiplier (ISV slot 790)

Browse Source 
Addresses the gate-not-learning failure mode observed at 3-seed
local mid-smoke verification of HEAD 5f10bcde3: gate_entropy_mean
stayed at MAX (Δ -0.0003 over 2000 steps) and max(gate_argmax_mass)
stuck at 1.0. Root cause: at uniform gate (g_k ≈ 1/K), the chain
grad_pi_probs → grad_gate_probs has small magnitude — gate Adam
moves W_gate too slowly to escape the uniform basin.

## Mechanism

* New ISV slot 790 RL_POLICY_GATE_LR_MULTIPLIER_INDEX, bootstrap
  5.0 inside the flag-gated block (slots 761-764 +790).
  RL_SLOTS_END = 791.
* Per-step LR-set site (only inside if self.use_multi_head_policy):
    lr_gate = lr_pi * isv[790]
    self.mhp_w_gate_adam.lr = lr_gate
    self.mhp_b_gate_adam.lr = lr_gate
* Head Adams (mhp_w_heads_adam, mhp_b_heads_adam) keep raw lr_pi.

## Verification (single-seed seed=42, flag-on, b=128, 2000 steps)

* gate_entropy_mean trajectory (5× multiplier vs baseline 5f10bcde3):
    step 0     1.0986 → 1.0986
    step 100   1.0986 → 1.0968 (Δ -0.0018)
    step 500   1.0985 → 1.0968
    step 1000  1.0985 → 1.0944
    step 1500  1.0985 → 1.0822
    step 1999  1.0983 → 1.0829 (Δ -0.0157)
  → 52× more gate gradient flow at 5× LR multiplier.
* gate_probs_mean differentiated: [0.333, 0.333, 0.333] →
  [0.252, 0.382, 0.366]. Head 0 (Short-bias, init = +0.5 on
  ShortLarge) deselected by 0.081 — semantically sensible because
  test data is 2024-Q1 ES futures (bullish/range-bound regime).
* max(gate_argmax_mass) = 1.0 still (head 1 wins 100% of batches
  by margin 0.382 vs 0.366) — soft routing emerging in gate_probs
  but argmax has not yet crossed over.

## Status

5.0 multiplier is "GATE_MOVING (partial pass)" per the dispatch's
threshold (entropy ≤ 1.05 not crossed). Slot is ISV-tunable so
escalation to 10×-20× is a single-line bootstrap change. The 52×
local improvement projected to cluster scale (b=1024 × 20k steps,
~80× more gradient signal) suggests the gate should specialize
clearly at cluster — but a stronger local multiplier first
verifies the dose-response curve hasn't saturated at 5×.

## Other verification gates

* 13/13 invariants PASS (unchanged).
* FOXHUNT_USE_MULTI_HEAD_POLICY=0 ./scripts/determinism-check.sh
  --quick: exit 0. isv_state checksum byte-identical to pre-B1
  (slot 790 zero-bootstrap contributes 0² to checksum).
* FOXHUNT_USE_MULTI_HEAD_POLICY=1 determinism-check.sh: exit 0.
* Pre-commit: 0 atomicAdd, 0 raw memcpy_htod/dtoh, 0 TODO.

## Linked
* Phase 2A-A: 0b3e40150 (foundation, inert)
* Phase 2A-B: e22da61cf (backward + aux KL prior)
* Phase 2A-C: 3e36f4a0e (trainer integration behind flag)
* Phase 2A-C+: 5f10bcde3 (device-aggregated gate diag)
* Phase 2A-D verdict: 3-seed pnl t=+0.223 (NOISE), G4 FAIL
  (argmax_mass=1.0) → motivated B1.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-06-03 13:49:20 +02:00
parent 5f10bcde3a
commit 6f3639bfbf
2 changed files with 38 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
crates/ml-alpha/src/rl/isv_slots.rs
View File

@@ -1886,6 +1886,24 @@ pub const RL_POLICY_GATE_ENTROPY_MEAN_INDEX: usize = 781;
/// single action.
pub const RL_POLICY_PER_HEAD_ENTROPY_MEAN_BASE_INDEX: usize = 782;
/// Multiplier applied to the gate Adam learning rates (W_gate, b_gate) on
/// top of the shared `RL_LR_PI_INDEX` (slot 411) value. Bootstrap 5.0 per
/// Phase 2A-D fix B1 (2026-06-03): 3-seed mid-smoke at HEAD 5f10bcde3
/// (flag-on) showed gate_entropy_mean pinned at MAX log(K) ≈ 1.0983 for
/// 2000 steps with argmax_mass = 1.000 — the gate gradient
/// `grad_gate_probs[k] = Σ_a pi_probs_k[k,a]·grad_pi_probs[a]` is
/// structurally weak near uniform g_k ≈ 1/K because all heads contribute
/// similarly, so W_gate moves too slowly to escape the uniform basin.
/// Scaling only the gate Adams (NOT the head Adams) accelerates routing
/// learning without destabilising head training. Range [0.5, 50.0] for
/// the ISV controller — outside that we expect either degenerate noise
/// (too high) or no effect (too low). Slot defaults to 0.0 sentinel when
/// the multi-head flag is off; the bootstrap write at slot 790 happens
/// inside the same `if self.use_multi_head_policy { ... }` block as
/// slots 761-764, so flag-off bit-equality is preserved (0² contributes
/// 0 to the `isv_state` checksum).
pub const RL_POLICY_GATE_LR_MULTIPLIER_INDEX: usize = 790;
/// Last RL-allocated slot index (exclusive).
/// Pre-risk-stack: 662. Post-Fix-B: 685. Post-v9 (warmup boundary): 696.
/// Post-regime-observer (F1.1): 716. Post-warmed-flag addendum: 717.
@@ -1902,4 +1920,5 @@ pub const RL_POLICY_PER_HEAD_ENTROPY_MEAN_BASE_INDEX: usize = 782;
/// Post-rollout-buffer+GAE Phase 1B-A (3 PPO control slots): 760.
/// Post-multi-head-policy Phase 2A-A (4 policy-gating slots): 764.
/// Post-multi-head-policy Phase 2A-D (25 aggregate diag slots): 789.
pub const RL_SLOTS_END: usize = 790;
/// Post-Phase 2A-D fix B1 (gate LR multiplier slot 790): 790.
pub const RL_SLOTS_END: usize = 791;

21
crates/ml-alpha/src/trainer/integrated.rs
View File

@@ -4167,11 +4167,16 @@ impl IntegratedTrainer {
// * head_entropy_floor = log(N_ACTIONS)·0.4 ≈ 0.959
// * aux_prior_beta = 0.05 (5% of policy-loss magnitude)
if self.use_multi_head_policy {
let mhp_isv_constants: [(usize, f32); 4] = [
// Phase 2A-D fix B1 (2026-06-03): slot 790 gate-LR multiplier
// bootstrap 5.0 — addresses gate-not-learning observed in
// 3-seed mid-smoke at HEAD 5f10bcde3 (gate_entropy stayed
// pinned at MAX, argmax_mass = 1.0). See slot doc-comment.
let mhp_isv_constants: [(usize, f32); 5] = [
(crate::rl::isv_slots::RL_POLICY_NUM_HEADS_INDEX, 3.0_f32),
(crate::rl::isv_slots::RL_POLICY_GATING_ENTROPY_FLOOR_INDEX, 0.549_f32),
(crate::rl::isv_slots::RL_POLICY_HEAD_ENTROPY_FLOOR_INDEX, 0.959_f32),
(crate::rl::isv_slots::RL_POLICY_AUX_PRIOR_BETA_INDEX, 0.05_f32),
(crate::rl::isv_slots::RL_POLICY_GATE_LR_MULTIPLIER_INDEX, 5.0_f32),
];
for (slot, value) in mhp_isv_constants.iter() {
let slot_i32 = *slot as i32;
@@ -5522,11 +5527,21 @@ impl IntegratedTrainer {
// Phase 2A-C: mirror lr_pi onto the multi-head Adam state when
// the gate is on. ISV[RL_LR_PI_INDEX] remains the single source
// of truth — no separate slot for multi-head LR.
//
// Phase 2A-D fix B1 (2026-06-03): gate Adams use lr_pi × ISV[790]
// (RL_POLICY_GATE_LR_MULTIPLIER_INDEX, bootstrap 5.0) so the
// routing gradient — structurally weak near uniform g_k ≈ 1/K —
// can escape the uniform basin without amplifying head training.
// Heads keep raw lr_pi.
if self.use_multi_head_policy {
let gate_lr_mult = self.read_isv_host(
crate::rl::isv_slots::RL_POLICY_GATE_LR_MULTIPLIER_INDEX,
);
let lr_gate = lr_pi * gate_lr_mult;
if let Some(a) = self.mhp_w_heads_adam.as_mut() { a.lr = lr_pi; }
if let Some(a) = self.mhp_b_heads_adam.as_mut() { a.lr = lr_pi; }
if let Some(a) = self.mhp_w_gate_adam.as_mut() { a.lr = lr_pi; }
if let Some(a) = self.mhp_b_gate_adam.as_mut() { a.lr = lr_pi; }
if let Some(a) = self.mhp_w_gate_adam.as_mut() { a.lr = lr_gate; }
if let Some(a) = self.mhp_b_gate_adam.as_mut() { a.lr = lr_gate; }
}
self.value_w_adam.lr = lr_v;
self.value_b_adam.lr = lr_v;