feat(sp13): P0b — aux_w deficit+stagnation controller + Hold cost lift

P0a smoke (train-67gqb on f934ea171) returned PARTIAL: aux_dir_acc climbed
0.149 → 0.483 in 10 epochs (signal exists), but val_win_rate stuck at 0.4638
and observed_hold_rate climbed to 0.479 despite controller saturating at
2.4× base. Two findings drive P0b:

(1) aux_w=0.05 (SP11-era clamp) starves the aux head of gradient. Replace
    inverted formula at training_loop.rs SP11 site with deficit+stagnation:
        deficit  = max(0, target - short_ema)
        improve  = max(0, short_ema - long_ema)
        stag     = (deficit > 0.005) ? clamp(1 - improve/deficit, 0, 1) : 0
        aux_w    = base × (1 + 5 × deficit) × (1 - 0.7 × stag),
                   clamped [0.3×base, 3.0×base]
    base 0.05 → 0.5 (10× lift). Stagnation decay prevents permanent
    destabilisation in data-limited case. Formula extracted as host helper
    `compute_aux_w_p0b(target, short, long)` for unit-test coverage.

(2) HOLD_COST_BASE=0.001 was too weak — max cost 0.005/bar × 30-bar hold =
    0.15 cumulative vs ±5-10 reward range = <3% of magnitude. Lift to 0.005
    (max 0.025/bar × 30 bars = 0.75 cumulative ≈ 10-15% of capped reward).
    Genuinely deters lazy Hold without crippling MFT use. Constructor
    static-init unchanged: it still writes the (now-lifted) HOLD_COST_BASE
    constant to slot 380 at fold boundary.

Per pearl_event_driven_reward_density_alignment tension already addressed
in P0a spec; the lift doesn't change the architecture, just the calibration.
Per feedback_isv_for_adaptive_bounds: base/gain/decay/floor/ceil are
numerical anchors; target/short/long EMAs read from ISV.

Tests: 3 new unit tests for controller formula (aux_w_at_target_returns_base,
aux_w_stagnation_decays_to_floor, aux_w_improving_amplifies_above_base).
14 SP13 GPU oracle tests + 14 SP12 reward-math tests still green; no kernel
changes.

Audit-doc updated with P0b section per Invariant 7.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-05 08:10:35 +02:00
parent f934ea1719
commit bdc5cb8bb2
4 changed files with 251 additions and 19 deletions

View File

@@ -60,12 +60,37 @@ pub const SKILL_BONUS_CAP_RATIO_DEFAULT: f32 = 0.3;
/// (~10 ticks of price for ES.FUT at ~0.25 tick value) but cumulative across
/// long Hold runs becomes meaningful relative to the SP12 capped trade reward
/// (±5..10).
pub const HOLD_COST_BASE: f32 = 0.001;
///
/// P0b lift (2026-05-04): `HOLD_COST_BASE` raised 0.001 → 0.005 after P0a smoke
/// (`train-67gqb` on `f934ea171`) showed `observed_hold_rate` climbing to 0.479
/// despite the controller saturating at 2.4× base — model rationally absorbed
/// the cost because it was <3% of reward magnitude (0.001 × 5 ceiling × 30-bar
/// hold = 0.15 cumulative vs ±5..10 reward range). Lifted base × 5 ceiling now
/// yields 0.025/bar × 30 bars = 0.75 cumulative ≈ 1015% of capped reward —
/// genuinely deters lazy Hold without crippling MFT use. Per-bar slot wiring
/// unchanged: constructor static-init still writes `HOLD_COST_BASE` to slot 380
/// at fold boundary; controller now scales the (lifted) base.
pub const HOLD_COST_BASE: f32 = 0.005;
pub const HOLD_RATE_TARGET_DEFAULT: f32 = 0.20;
pub const HOLD_COST_CONTROLLER_GAIN: f32 = 5.0;
pub const HOLD_COST_FLOOR_RATIO: f32 = 0.5; // never below 0.5 × base
pub const HOLD_COST_CEIL_RATIO: f32 = 5.0; // never above 5.0 × base
/// SP13 P0b: aux_w deficit + stagnation controller anchors. Replaces the
/// SP11-era inverted formula `aux_w = 0.1 × health × (1 - sharpe).clamp(0.05, 0.3)`
/// at `training_loop.rs` (which LOWERED aux when sharpe was high — exactly
/// wrong for the failure mode where high sharpe + low WR is the structural-
/// extraction trap). The new controller drives aux_w from the deficit
/// `target_dir_acc - short_ema` and decays back to floor when the short EMA
/// fails to pull ahead of the long EMA (data-limited stagnation). Per
/// `feedback_isv_for_adaptive_bounds.md`: base/gain/decay/floor/ceil are
/// numerical anchors; target/short/long EMAs read from ISV. Spec §"Change 5".
pub const AUX_W_BASE: f32 = 0.5; // 10× lift from SP11 0.05 floor
pub const AUX_W_DEFICIT_GAIN: f32 = 5.0; // aux_w = base × (1 + 5 × deficit)
pub const AUX_W_STAGNATION_DECAY: f32 = 0.7; // (1 - 0.7 × stagnation)
pub const AUX_W_HARD_FLOOR_RATIO: f32 = 0.3; // never below 0.3 × base
pub const AUX_W_HARD_CEIL_RATIO: f32 = 3.0; // never above 3.0 × base
/// Sentinel for both dir-acc EMAs at fold boundary: 0.5 is the random-guessing
/// baseline for binary directional accuracy. First observation replaces this
/// directly per `pearl_first_observation_bootstrap` so the EMA tracks the new

View File

@@ -617,3 +617,97 @@ fn test_14d_roundtrip_preserves_intensities() {
"ensemble_intensity must survive roundtrip"
);
}
/// SP13 P0b — aux_w deficit + stagnation controller unit tests.
///
/// The kernel-tested machinery (apply_pearls EMAs, dir_acc reduction) lives in
/// `sp13_phase0_oracle_tests.rs`. These tests cover the host-side controller
/// formula `compute_aux_w_p0b(target, short, long)` extracted in P0b for
/// testability, exercising the two boundary regimes the spec calls out:
/// 1. at-target: deficit=0 → aux_w collapses to base
/// 2. fully-stagnant: short below target with no improvement → decays to floor
mod aux_w_controller_tests {
use crate::cuda_pipeline::sp13_isv_slots::{
AUX_W_BASE, AUX_W_HARD_FLOOR_RATIO,
};
use super::super::training_loop::compute_aux_w_p0b;
/// At-target steady state: short = target, no deficit, no stagnation
/// activation → aux_w = base × 1 × 1 = base.
#[test]
fn aux_w_at_target_returns_base() {
let target = 0.55_f32;
let short = 0.55_f32;
let long = 0.50_f32;
let aux_w = compute_aux_w_p0b(target, short, long);
assert!(
(aux_w - AUX_W_BASE).abs() < 1e-6,
"expected base={}, got {}",
AUX_W_BASE, aux_w,
);
}
/// Stagnation regime: short stuck below target with zero improvement
/// (short == long) → stagnation = 1.0 → (1 - 0.7 × 1) = 0.3 decay factor
/// drives aux_w_raw close to floor without crippling Q-head training.
/// With deficit=0.05: aux_w_raw = 0.5 × 1.25 × 0.3 = 0.1875. The hard floor
/// 0.15 = base × 0.3 prevents further suppression even at extreme stagnation
/// (the 0.3 decay factor cannot push below floor unless deficit→0, in which
/// case the deficit-gain term collapses anyway). The contract: under
/// stagnation, aux_w must be much closer to floor than the matching
/// improving regime would give.
#[test]
fn aux_w_stagnation_decays_to_floor() {
let target = 0.55_f32;
let short = 0.50_f32;
let long = 0.50_f32; // no improvement → stagnation = 1
let aux_w_stagnant = compute_aux_w_p0b(target, short, long);
let floor = AUX_W_BASE * AUX_W_HARD_FLOOR_RATIO;
// Sanity: never below the hard floor.
assert!(
aux_w_stagnant >= floor - 1e-6,
"must respect floor={}; got {}",
floor, aux_w_stagnant,
);
// Same deficit but with full improvement (long < short) → stagnation = 0.
let long_improving = 0.45_f32;
let aux_w_improving = compute_aux_w_p0b(target, short, long_improving);
// Stagnation must drag aux_w toward floor: ≥3× drop versus improving.
assert!(
aux_w_improving / aux_w_stagnant >= 3.0,
"stagnation factor 0.3 must decay aux_w by ≥3× (improving={}, stagnant={})",
aux_w_improving, aux_w_stagnant,
);
// Concrete value check: 0.5 × 1.25 × 0.3 = 0.1875.
let expected = 0.1875_f32;
assert!(
(aux_w_stagnant - expected).abs() < 1e-5,
"stagnant value should be {}, got {}",
expected, aux_w_stagnant,
);
}
/// Improving below target: short pulled ahead of long, no stagnation, but
/// deficit-gain still amplifies. With target=0.55, short=0.50, long=0.45:
/// deficit=0.05, improvement=0.05, stagnation=0.0
/// aux_w_raw = 0.5 × (1 + 5 × 0.05) × 1.0 = 0.5 × 1.25 = 0.625
/// Within ceiling 1.5, so result = 0.625.
#[test]
fn aux_w_improving_amplifies_above_base() {
let target = 0.55_f32;
let short = 0.50_f32;
let long = 0.45_f32;
let aux_w = compute_aux_w_p0b(target, short, long);
let expected = 0.625_f32;
assert!(
(aux_w - expected).abs() < 1e-5,
"improving regime should amplify to {}, got {}",
expected, aux_w,
);
assert!(aux_w > AUX_W_BASE, "must rise above base when improving with deficit");
}
}

View File

@@ -93,6 +93,47 @@ fn shannon_entropy_normalized(dist: &[f32]) -> f32 {
h / (dist.len() as f32).ln()
}
/// SP13 P0b aux_w deficit + stagnation controller (extracted for testability).
///
/// Replaces the SP11-era inverted formula
/// `aux_w = (0.1 × learning_health × (1 - tanh(0.1 × sharpe))).clamp(0.05, 0.3)`
/// which lowered aux_w when sharpe was high (exact wrong direction for the
/// failure mode where high sharpe + low WR signals structural-extraction trap).
///
/// Mechanism per spec §"Change 5":
/// ```text
/// deficit = max(0, target - short)
/// improvement = max(0, short - long)
/// stagnation = (deficit > 0.005) ? clamp(1 - improvement/deficit, 0, 1) : 0
/// aux_w_raw = AUX_W_BASE × (1 + 5 × deficit) × (1 - 0.7 × stagnation)
/// aux_w = clamp(aux_w_raw, [0.3 × base, 3.0 × base])
/// ```
///
/// The 0.005 deficit-floor on the stagnation gate avoids divide-by-near-zero
/// when at/above target — at that point we are satisfied, not stagnant, so
/// stagnation is forced to 0 and the (1 + 5 × deficit) term is also ≈ 1, so
/// `aux_w` collapses cleanly to the base.
pub(crate) fn compute_aux_w_p0b(target: f32, short: f32, long: f32) -> f32 {
use crate::cuda_pipeline::sp13_isv_slots::{
AUX_W_BASE, AUX_W_DEFICIT_GAIN, AUX_W_STAGNATION_DECAY,
AUX_W_HARD_FLOOR_RATIO, AUX_W_HARD_CEIL_RATIO,
};
let deficit = (target - short).max(0.0);
let improvement = (short - long).max(0.0);
let stagnation = if deficit > 0.005 {
(1.0 - improvement / deficit).clamp(0.0, 1.0)
} else {
0.0
};
let aux_w_raw = AUX_W_BASE
* (1.0 + AUX_W_DEFICIT_GAIN * deficit)
* (1.0 - AUX_W_STAGNATION_DECAY * stagnation);
aux_w_raw.clamp(
AUX_W_BASE * AUX_W_HARD_FLOOR_RATIO,
AUX_W_BASE * AUX_W_HARD_CEIL_RATIO,
)
}
/// Metrics returned from `process_epoch_boundary` for downstream logging.
pub(crate) struct EpochBoundaryMetrics {
pub avg_loss: f32,
@@ -4163,25 +4204,37 @@ impl DQNTrainer {
}
}
// Plan 4 Task 6 Commit B: refresh the aux-loss weight before the
// next training-step graph capture picks it up.
//
// aux_weight = clamp(0.1 × ISV[LEARNING_HEALTH] × (1 - tanh(SHARPE_SCALE × sharpe_ema)),
// 0.05, 0.3)
//
// SHARPE_SCALE = 0.1 — matches Plan 3's controller convention for
// sharpe-driven α adaptation. Numerical-stability bounds [0.05, 0.3]
// per Invariant 1 carve-out. Coefficient `0.1` is the standard
// aux-loss base weight per spec §4.E.6 (only tuned multiplicand).
// The setter is idempotent and does NOT invalidate the captured
// graph — the new value takes effect on the next graph re-capture
// (fold boundary, lr change, etc.).
// SP13 P0b (2026-05-04): aux_w deficit + stagnation controller.
// Replaces SP11-era inverted formula
// aux_w = (0.1 × learning_health × (1 - tanh(0.1 × sharpe_ema))).clamp(0.05, 0.3)
// which LOWERED aux when sharpe was high — exactly wrong for the
// failure mode (high sharpe + low WR is the structural-extraction
// trap; aux head needs MORE gradient when policy is over-fitting
// a noise structure, not less). New mechanism per spec §"Change 5":
// deficit = max(0, target_dir_acc - short_ema)
// improvement = max(0, short_ema - long_ema)
// stagnation = (deficit > 0.005) ? clamp(1 - improvement/deficit, 0, 1) : 0
// aux_w_raw = base × (1 + 5 × deficit) × (1 - 0.7 × stagnation)
// aux_w = clamp(aux_w_raw, [0.3 × base, 3.0 × base])
// Stagnation decay backs aux_w to floor when the short EMA fails
// to pull ahead of the long EMA — "we tried, it didn't help, stop
// destabilising Q." Hard floor 0.3× base prevents permanent
// suppression (the SP11 inversion bug). The setter is idempotent
// and does NOT invalidate the captured graph — new value takes
// effect on the next re-capture (fold boundary, lr change, etc.).
if let Some(ref mut fused) = self.fused_ctx {
use crate::cuda_pipeline::gpu_dqn_trainer::LEARNING_HEALTH_INDEX;
let learning_health = fused.read_isv_signal_at(LEARNING_HEALTH_INDEX);
const SHARPE_SCALE: f32 = 0.1;
let sharpe_tanh = (SHARPE_SCALE * self.training_sharpe_ema).tanh();
let aux_w = (0.1_f32 * learning_health * (1.0 - sharpe_tanh)).clamp(0.05, 0.3);
use crate::cuda_pipeline::sp13_isv_slots::{
TARGET_DIR_ACC_INDEX, AUX_DIR_ACC_SHORT_EMA_INDEX,
AUX_DIR_ACC_LONG_EMA_INDEX,
};
let target = fused.read_isv_signal_at(TARGET_DIR_ACC_INDEX);
let short_ema = fused.read_isv_signal_at(AUX_DIR_ACC_SHORT_EMA_INDEX);
let long_ema = fused.read_isv_signal_at(AUX_DIR_ACC_LONG_EMA_INDEX);
let aux_w = compute_aux_w_p0b(target, short_ema, long_ema);
tracing::debug!(
"aux_w controller: target={:.3} short={:.3} long={:.3} aux_w={:.3}",
target, short_ema, long_ema, aux_w,
);
fused.set_aux_weight(aux_w);
}

View File

@@ -5801,3 +5801,63 @@ Per-bar Hold cost is per-bar reward shaping, which `pearl_event_driven_reward_de
### Files (atomic P0a commit)
15 files / 2316 lines added: 5 new kernels + 9 modifications + 1 mod.rs wire-up. Per `feedback_no_partial_refactor`: all consumers of the new ISV slots wired in this single commit.
---
## SP13 P0b — aux_w deficit+stagnation controller + Hold cost lift (2026-05-04)
**Spec**: `docs/superpowers/specs/2026-05-04-sp13-redefine-success-for-predictive-skill.md` v3 §"Change 5"
**Plan**: `docs/superpowers/plans/2026-05-04-sp13-redefine-success-for-predictive-skill.md` v3 §"Phase 0b"
**Trigger**: P0a smoke `train-67gqb` on `f934ea171` returned PARTIAL — `aux_dir_acc` climbed 0.149 → 0.483 over 10 epochs (signal exists), but `val_win_rate` stuck at 0.4638 (Q-head not consuming aux signal) and `observed_hold_rate` climbed to 0.479 (Hold cost too weak; controller saturated at 2.4× base, model rationally absorbed it).
### Decision A — corrected aux_w controller (replaces SP11 inverted formula)
`training_loop.rs` SP11-era site (post Plan 4 Task 6 Commit B):
```rust
// REPLACED:
let aux_w = (0.1 × learning_health × (1 - tanh(0.1 × sharpe))).clamp(0.05, 0.3);
```
The SP11 formula LOWERED aux_w when sharpe was high — exactly wrong for the failure mode where high sharpe + low WR signals the structural-extraction trap (aux head needs MORE gradient when policy is over-fitting noise structure, not less).
Replaced with deficit+stagnation controller per spec §"Change 5":
```rust
deficit = max(0, target_dir_acc - short_ema)
improvement = max(0, short_ema - long_ema)
stagnation = (deficit > 0.005) ? clamp(1 - improvement/deficit, 0, 1) : 0
aux_w_raw = AUX_W_BASE × (1 + AUX_W_DEFICIT_GAIN × deficit) × (1 - AUX_W_STAGNATION_DECAY × stagnation)
aux_w = clamp(aux_w_raw, [AUX_W_BASE × 0.3, AUX_W_BASE × 3.0])
```
Reads `target_dir_acc` from ISV[372], `short_ema` from ISV[373], `long_ema` from ISV[374] (all P0a slots). `AUX_W_BASE = 0.5` is a 10× lift from the SP11 0.05 floor — the lift is the fix, not a confounder, because the SP11 formula was structurally wrong (hard floor was acting as the dominant suppression mechanism). Stagnation decay prevents permanent destabilisation in the data-limited case ("we tried, it didn't help, stop destabilising Q").
Formula extracted as a top-level `pub(crate) fn compute_aux_w_p0b(target, short, long) -> f32` helper (`training_loop.rs`) for unit-test coverage. The callsite reads ISV slots and invokes the helper; setter `fused.set_aux_weight()` is unchanged (idempotent, no graph re-capture invalidation).
### Decision B — `HOLD_COST_BASE` lift 0.001 → 0.005
The P0a Hold-pricing controller wired the per-bar Hold cost in `experience_kernels.cu` reading from ISV[380]. The constant lift is a single-line change in `sp13_isv_slots.rs` — the controller scales `1 + 5 × max(0, observed - target)` clamped to `[0.5×, 5.0×]`, so with `BASE = 0.001` the maximum cost is 0.005/bar × 30-bar hold = 0.15 cumulative vs ±5-10 reward range = <3% of magnitude. Model rationally absorbed the cost. Lift to `BASE = 0.005`: max 0.025/bar × 30 bars = 0.75 cumulative ≈ 1015% of capped reward, genuinely deters lazy Hold without crippling MFT use. Constructor static-init unchanged: it still writes the (now-lifted) `HOLD_COST_BASE` constant to slot 380 at fold boundary; controller now scales the lifted base.
### Constants added (`sp13_isv_slots.rs`)
5 new aux_w controller anchors per `feedback_isv_for_adaptive_bounds.md` (numerical anchors only; bounds derived from ISV signals):
| Constant | Value | Role |
|---|---|---|
| `AUX_W_BASE` | 0.5 | aux_w at target (steady-state). 10× SP11 floor. |
| `AUX_W_DEFICIT_GAIN` | 5.0 | `aux_w = base × (1 + 5 × deficit)` — deficit-driven amplification |
| `AUX_W_STAGNATION_DECAY` | 0.7 | `(1 - 0.7 × stagnation)` — pulls back when not improving |
| `AUX_W_HARD_FLOOR_RATIO` | 0.3 | Floor = base × 0.3 = 0.15 — prevents complete suppression |
| `AUX_W_HARD_CEIL_RATIO` | 3.0 | Ceil = base × 3.0 = 1.5 — prevents Q destabilisation |
`HOLD_COST_BASE` lifted in-place; no slot index changes; constructor write site unchanged.
### Test coverage
3 new host-side unit tests in `crates/ml/src/trainers/dqn/trainer/tests.rs::aux_w_controller_tests`:
- `aux_w_at_target_returns_base` — short = target → aux_w = AUX_W_BASE
- `aux_w_stagnation_decays_to_floor` — short stuck below target with no improvement → aux_w drops ≥3× vs improving regime; respects floor
- `aux_w_improving_amplifies_above_base` — short pulled ahead of long → aux_w = 0.625 (above base, deficit-amplified)
14 SP13 GPU oracle tests + 14 SP12 reward-math tests still green; no kernel changes (P0b is a constants + host-side controller change only).
### Files (atomic P0b commit)
3 files: `sp13_isv_slots.rs` (constants), `training_loop.rs` (controller + helper extraction), `tests.rs` (3 unit tests). No new kernels, no new ISV slots, no fingerprint change.