feat(sp13): P0b — aux_w deficit+stagnation controller + Hold cost lift
P0a smoke (train-67gqb on f934ea171) returned PARTIAL: aux_dir_acc climbed
0.149 → 0.483 in 10 epochs (signal exists), but val_win_rate stuck at 0.4638
and observed_hold_rate climbed to 0.479 despite controller saturating at
2.4× base. Two findings drive P0b:
(1) aux_w=0.05 (SP11-era clamp) starves the aux head of gradient. Replace
inverted formula at training_loop.rs SP11 site with deficit+stagnation:
deficit = max(0, target - short_ema)
improve = max(0, short_ema - long_ema)
stag = (deficit > 0.005) ? clamp(1 - improve/deficit, 0, 1) : 0
aux_w = base × (1 + 5 × deficit) × (1 - 0.7 × stag),
clamped [0.3×base, 3.0×base]
base 0.05 → 0.5 (10× lift). Stagnation decay prevents permanent
destabilisation in data-limited case. Formula extracted as host helper
`compute_aux_w_p0b(target, short, long)` for unit-test coverage.
(2) HOLD_COST_BASE=0.001 was too weak — max cost 0.005/bar × 30-bar hold =
0.15 cumulative vs ±5-10 reward range = <3% of magnitude. Lift to 0.005
(max 0.025/bar × 30 bars = 0.75 cumulative ≈ 10-15% of capped reward).
Genuinely deters lazy Hold without crippling MFT use. Constructor
static-init unchanged: it still writes the (now-lifted) HOLD_COST_BASE
constant to slot 380 at fold boundary.
Per pearl_event_driven_reward_density_alignment tension already addressed
in P0a spec; the lift doesn't change the architecture, just the calibration.
Per feedback_isv_for_adaptive_bounds: base/gain/decay/floor/ceil are
numerical anchors; target/short/long EMAs read from ISV.
Tests: 3 new unit tests for controller formula (aux_w_at_target_returns_base,
aux_w_stagnation_decays_to_floor, aux_w_improving_amplifies_above_base).
14 SP13 GPU oracle tests + 14 SP12 reward-math tests still green; no kernel
changes.
Audit-doc updated with P0b section per Invariant 7.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -5801,3 +5801,63 @@ Per-bar Hold cost is per-bar reward shaping, which `pearl_event_driven_reward_de
|
||||
### Files (atomic P0a commit)
|
||||
|
||||
15 files / 2316 lines added: 5 new kernels + 9 modifications + 1 mod.rs wire-up. Per `feedback_no_partial_refactor`: all consumers of the new ISV slots wired in this single commit.
|
||||
|
||||
---
|
||||
|
||||
## SP13 P0b — aux_w deficit+stagnation controller + Hold cost lift (2026-05-04)
|
||||
|
||||
**Spec**: `docs/superpowers/specs/2026-05-04-sp13-redefine-success-for-predictive-skill.md` v3 §"Change 5"
|
||||
**Plan**: `docs/superpowers/plans/2026-05-04-sp13-redefine-success-for-predictive-skill.md` v3 §"Phase 0b"
|
||||
**Trigger**: P0a smoke `train-67gqb` on `f934ea171` returned PARTIAL — `aux_dir_acc` climbed 0.149 → 0.483 over 10 epochs (signal exists), but `val_win_rate` stuck at 0.4638 (Q-head not consuming aux signal) and `observed_hold_rate` climbed to 0.479 (Hold cost too weak; controller saturated at 2.4× base, model rationally absorbed it).
|
||||
|
||||
### Decision A — corrected aux_w controller (replaces SP11 inverted formula)
|
||||
|
||||
`training_loop.rs` SP11-era site (post Plan 4 Task 6 Commit B):
|
||||
```rust
|
||||
// REPLACED:
|
||||
let aux_w = (0.1 × learning_health × (1 - tanh(0.1 × sharpe))).clamp(0.05, 0.3);
|
||||
```
|
||||
The SP11 formula LOWERED aux_w when sharpe was high — exactly wrong for the failure mode where high sharpe + low WR signals the structural-extraction trap (aux head needs MORE gradient when policy is over-fitting noise structure, not less).
|
||||
|
||||
Replaced with deficit+stagnation controller per spec §"Change 5":
|
||||
```rust
|
||||
deficit = max(0, target_dir_acc - short_ema)
|
||||
improvement = max(0, short_ema - long_ema)
|
||||
stagnation = (deficit > 0.005) ? clamp(1 - improvement/deficit, 0, 1) : 0
|
||||
aux_w_raw = AUX_W_BASE × (1 + AUX_W_DEFICIT_GAIN × deficit) × (1 - AUX_W_STAGNATION_DECAY × stagnation)
|
||||
aux_w = clamp(aux_w_raw, [AUX_W_BASE × 0.3, AUX_W_BASE × 3.0])
|
||||
```
|
||||
Reads `target_dir_acc` from ISV[372], `short_ema` from ISV[373], `long_ema` from ISV[374] (all P0a slots). `AUX_W_BASE = 0.5` is a 10× lift from the SP11 0.05 floor — the lift is the fix, not a confounder, because the SP11 formula was structurally wrong (hard floor was acting as the dominant suppression mechanism). Stagnation decay prevents permanent destabilisation in the data-limited case ("we tried, it didn't help, stop destabilising Q").
|
||||
|
||||
Formula extracted as a top-level `pub(crate) fn compute_aux_w_p0b(target, short, long) -> f32` helper (`training_loop.rs`) for unit-test coverage. The callsite reads ISV slots and invokes the helper; setter `fused.set_aux_weight()` is unchanged (idempotent, no graph re-capture invalidation).
|
||||
|
||||
### Decision B — `HOLD_COST_BASE` lift 0.001 → 0.005
|
||||
|
||||
The P0a Hold-pricing controller wired the per-bar Hold cost in `experience_kernels.cu` reading from ISV[380]. The constant lift is a single-line change in `sp13_isv_slots.rs` — the controller scales `1 + 5 × max(0, observed - target)` clamped to `[0.5×, 5.0×]`, so with `BASE = 0.001` the maximum cost is 0.005/bar × 30-bar hold = 0.15 cumulative vs ±5-10 reward range = <3% of magnitude. Model rationally absorbed the cost. Lift to `BASE = 0.005`: max 0.025/bar × 30 bars = 0.75 cumulative ≈ 10–15% of capped reward, genuinely deters lazy Hold without crippling MFT use. Constructor static-init unchanged: it still writes the (now-lifted) `HOLD_COST_BASE` constant to slot 380 at fold boundary; controller now scales the lifted base.
|
||||
|
||||
### Constants added (`sp13_isv_slots.rs`)
|
||||
|
||||
5 new aux_w controller anchors per `feedback_isv_for_adaptive_bounds.md` (numerical anchors only; bounds derived from ISV signals):
|
||||
|
||||
| Constant | Value | Role |
|
||||
|---|---|---|
|
||||
| `AUX_W_BASE` | 0.5 | aux_w at target (steady-state). 10× SP11 floor. |
|
||||
| `AUX_W_DEFICIT_GAIN` | 5.0 | `aux_w = base × (1 + 5 × deficit)` — deficit-driven amplification |
|
||||
| `AUX_W_STAGNATION_DECAY` | 0.7 | `(1 - 0.7 × stagnation)` — pulls back when not improving |
|
||||
| `AUX_W_HARD_FLOOR_RATIO` | 0.3 | Floor = base × 0.3 = 0.15 — prevents complete suppression |
|
||||
| `AUX_W_HARD_CEIL_RATIO` | 3.0 | Ceil = base × 3.0 = 1.5 — prevents Q destabilisation |
|
||||
|
||||
`HOLD_COST_BASE` lifted in-place; no slot index changes; constructor write site unchanged.
|
||||
|
||||
### Test coverage
|
||||
|
||||
3 new host-side unit tests in `crates/ml/src/trainers/dqn/trainer/tests.rs::aux_w_controller_tests`:
|
||||
- `aux_w_at_target_returns_base` — short = target → aux_w = AUX_W_BASE
|
||||
- `aux_w_stagnation_decays_to_floor` — short stuck below target with no improvement → aux_w drops ≥3× vs improving regime; respects floor
|
||||
- `aux_w_improving_amplifies_above_base` — short pulled ahead of long → aux_w = 0.625 (above base, deficit-amplified)
|
||||
|
||||
14 SP13 GPU oracle tests + 14 SP12 reward-math tests still green; no kernel changes (P0b is a constants + host-side controller change only).
|
||||
|
||||
### Files (atomic P0b commit)
|
||||
|
||||
3 files: `sp13_isv_slots.rs` (constants), `training_loop.rs` (controller + helper extraction), `tests.rs` (3 unit tests). No new kernels, no new ISV slots, no fingerprint change.
|
||||
|
||||
Reference in New Issue
Block a user