diff --git a/crates/ml/src/trainers/dqn/state_reset_registry.rs b/crates/ml/src/trainers/dqn/state_reset_registry.rs index 768848470..c3f0021d3 100644 --- a/crates/ml/src/trainers/dqn/state_reset_registry.rs +++ b/crates/ml/src/trainers/dqn/state_reset_registry.rs @@ -1176,7 +1176,7 @@ impl StateResetRegistry { RegistryEntry { name: "sp16_phase2_hold_cost_scale_adaptive", category: ResetCategory::FoldReset, - description: "ISV[HOLD_COST_SCALE_INDEX=461] — SP16 Phase 2 (2026-05-08) adaptive Hold cost scale (multiplicative scalar applied to the static per-bar Hold cost ISV[HOLD_COST_INDEX=380] at the 3 consumer sites in experience_kernels.cu — segment_complete branch line ~3089, per-bar positioned-Hold branch line ~3553, per-bar flat-Hold branch line ~3617). Producer kernel `hold_cost_scale_update_kernel` reads ISV[HOLD_RATE_OBSERVED_EMA_INDEX=382] and ISV[HOLD_RATE_TARGET_INDEX=381] (same input chain as SP16-P1 MIN_HOLD_TEMPERATURE — same per-epoch cadence, same Pearl-A bootstrap pattern) and maps the overrun into a [1, 25] multiplier via `target_scale = SCALE_MIN + (SCALE_MAX − SCALE_MIN) × clamp(max(0, observed - target) / max(target, 0.01), 0, 1)`, slow-EMA-blends into slot 461. Per train-multi-seed-pfh9n post-mortem: realised Hold rate climbed 0.25 → 0.52 across training while the cost penalty stayed at ~0.006 (~100× smaller than per-bar reward magnitudes — popart=0.97, cf=0.65). Hold became effectively free, allowing the structural Q-bias toward zero-variance Hold to dominate. The adaptive scale pushes effective cost to ~0.15/bar at 100% overrun (observed=2× target), competitive with per-bar reward magnitudes but well below the SP12 asymmetric cap [-10, +5] so no clamp interaction. FoldReset sentinel SENTINEL_HOLD_COST_SCALE=0.0 — Pearl-A first-observation bootstrap: the consumer's cold-start fallback (scale=1.0 when slot ≤ 0 or out-of-bounds) fires for exactly one launch, then the producer's first valid observation lands a real scale and the consumer reads it thereafter. Cold-start sentinel guard at the producer: when observed_hold_rate is at SENTINEL_HOLD_RATE_OBSERVED=0.0 (matches sp13_hold_rate_observed_ema fold-reset sentinel), the kernel keeps slot 461 unchanged — the consumer falls back to scale=1.0. α=0.05 mid-cadence EMA. Bounds [1, 25] — Category-1 dimensional safety floors per `feedback_isv_for_adaptive_bounds.md`. SCALE_MIN=1: the floor is `no extra penalty` (lower bound enforced by overrun being clamped at 0). SCALE_MAX=25: at this ceiling, a single Hold bar's effective cost ~0.15 — competitive with per-bar reward magnitudes ~0.01-0.1 but well below the SP12 cap, so no clamp interaction. Same launch-ordering constraint as SP16-P1: producer fires per-epoch boundary AFTER the per-step hold_rate_observer chain populates slot 382 (per `pearl_canary_input_freshness_launch_order.md`).", + description: "ISV[HOLD_COST_SCALE_INDEX=461] — SP18 v2: RETIRED — replaced by D-leg structural reward (sentinel 0.0 keeps slot bit-identical for checkpoint compat). The SP16 Phase 2 adaptive Hold cost scale chain is superseded by the SP18 D-leg structural Hold opportunity-cost reward (compute_sp18_hold_opportunity_cost in trade_physics.cuh + ISV[483..493) cap producer chain) per DD7=c. The 3 consumer sites in experience_kernels.cu that previously multiplied isv[ISV_HOLD_COST_IDX=380] by isv[461] are migrated to the structural reward in Phase 2 of SP18 (atomic 3-site migration per `feedback_no_partial_refactor`). Producer kernel `hold_cost_scale_update_kernel` is DELETED in Phase 1 atomic with the cost-application sites per DD7=c. The slot stays ALLOCATED with sentinel 0.0 to preserve checkpoint layout fingerprint compatibility — RESERVED-gap pattern from SP14-C.1 (deleted slots are NOT compacted; the surviving fingerprint just keeps the slot in the bus and the FoldReset machinery rewrites the sentinel each fold so consumers reading via stale code paths fall back to scale=1.0 rather than reading uninitialised garbage). Plan: docs/superpowers/plans/2026-05-08-sp18-reward-shape-hold-attractor.md § DD7 + Pre-Phase Task PP.4.", }, RegistryEntry { name: "sp14_audit_4b_min_hold_temperature_adaptive", @@ -1213,32 +1213,32 @@ impl StateResetRegistry { RegistryEntry { name: "sp16_t3_hcs_target_mean", category: ResetCategory::FoldReset, - description: "ISV[HCS_TARGET_MEAN_INDEX=462] — SP16 T3 Welford running mean of target_scale for the HOLD_COST_SCALE producer kernel. Sentinel 0.0; first valid observation initialises via Welford increment in `hold_cost_scale_update_kernel.cu`. Combined with the cold-start fallback (sample_count<3 → α=1.0), the first 3 epochs of each fold force REPLACE; from epoch 3 onward Wiener-optimal α emerges from Welford state.", + description: "ISV[HCS_TARGET_MEAN_INDEX=462] — SP18 v2: RETIRED — replaced by D-leg structural reward (sentinel 0.0 keeps slot bit-identical for checkpoint compat). The SP16 T3 Welford accumulator for HOLD_COST_SCALE is no longer driven (producer kernel deleted in SP18 Phase 1 per DD7=c). The slot stays ALLOCATED in the layout fingerprint per the SP14-C.1 RESERVED-gap pattern. Original semantics preserved here for archaeology: SP16 T3 Welford running mean of target_scale for the HOLD_COST_SCALE producer kernel; sentinel 0.0 enabled Pearl-A bootstrap with cold-start fallback (sample_count<3 → α=1.0). See SP18 spec § DD7 + plan PP.4.", }, RegistryEntry { name: "sp16_t3_hcs_target_m2", category: ResetCategory::FoldReset, - description: "ISV[HCS_TARGET_M2_INDEX=463] — SP16 T3 Welford ΣΔ² (sample-variance accumulator) for target_scale in the HOLD_COST_SCALE producer kernel. Sentinel 0.0; tracks variance of `target_scale` series across each fold. Used to compute `sample_var = M2 / (n - 1)` once `n ≥ 3`.", + description: "ISV[HCS_TARGET_M2_INDEX=463] — SP18 v2: RETIRED — replaced by D-leg structural reward (sentinel 0.0 keeps slot bit-identical for checkpoint compat). Original semantics: SP16 T3 Welford ΣΔ² (sample-variance accumulator) for target_scale in the HOLD_COST_SCALE producer kernel. See SP18 spec § DD7 + plan PP.4.", }, RegistryEntry { name: "sp16_t3_hcs_diff_mean", category: ResetCategory::FoldReset, - description: "ISV[HCS_DIFF_MEAN_INDEX=464] — SP16 T3 Welford running mean of consecutive `target_scale` one-step diffs in the HOLD_COST_SCALE producer kernel. Sentinel 0.0; first diff observation lands when sample_count reaches 2.", + description: "ISV[HCS_DIFF_MEAN_INDEX=464] — SP18 v2: RETIRED — replaced by D-leg structural reward (sentinel 0.0 keeps slot bit-identical for checkpoint compat). Original semantics: SP16 T3 Welford running mean of consecutive `target_scale` one-step diffs. See SP18 spec § DD7 + plan PP.4.", }, RegistryEntry { name: "sp16_t3_hcs_diff_m2", category: ResetCategory::FoldReset, - description: "ISV[HCS_DIFF_M2_INDEX=465] — SP16 T3 Welford ΣΔ² for the consecutive-diff series in the HOLD_COST_SCALE producer kernel. Sentinel 0.0; provides `diff_var` half of the Wiener formula α = diff_var / (diff_var + sample_var + ε).", + description: "ISV[HCS_DIFF_M2_INDEX=465] — SP18 v2: RETIRED — replaced by D-leg structural reward (sentinel 0.0 keeps slot bit-identical for checkpoint compat). Original semantics: SP16 T3 Welford ΣΔ² for the consecutive-diff series; provided `diff_var` half of the Wiener formula. See SP18 spec § DD7 + plan PP.4.", }, RegistryEntry { name: "sp16_t3_hcs_prev_target", category: ResetCategory::FoldReset, - description: "ISV[HCS_PREV_TARGET_INDEX=466] — SP16 T3 previous-epoch `target_scale` for the HOLD_COST_SCALE producer kernel. Sentinel 0.0; written every launch from the freshly-computed `target_scale` so the next launch's diff calculation has a basis.", + description: "ISV[HCS_PREV_TARGET_INDEX=466] — SP18 v2: RETIRED — replaced by D-leg structural reward (sentinel 0.0 keeps slot bit-identical for checkpoint compat). Original semantics: SP16 T3 previous-epoch `target_scale` for the HOLD_COST_SCALE producer kernel. See SP18 spec § DD7 + plan PP.4.", }, RegistryEntry { name: "sp16_t3_hcs_sample_count", category: ResetCategory::FoldReset, - description: "ISV[HCS_SAMPLE_COUNT_INDEX=467] — SP16 T3 Welford observation counter for the HOLD_COST_SCALE producer kernel. Sentinel 0.0; incremented every launch (excluding the cold-start sentinel-guard early-return). The cold-start fallback (sample_count < WELFORD_MIN_SAMPLE_COUNT=3 → α = 1.0) keeps the producer responsive in epochs 1-2 of each fold while Welford state stabilises.", + description: "ISV[HCS_SAMPLE_COUNT_INDEX=467] — SP18 v2: RETIRED — replaced by D-leg structural reward (sentinel 0.0 keeps slot bit-identical for checkpoint compat). Original semantics: SP16 T3 Welford observation counter for the HOLD_COST_SCALE producer kernel; cold-start fallback (sample_count < WELFORD_MIN_SAMPLE_COUNT=3 → α = 1.0). See SP18 spec § DD7 + plan PP.4.", }, RegistryEntry { name: "sp16_t3_mht_target_mean", @@ -2227,4 +2227,72 @@ mod tests { ); } } + + /// Lock SP18 v2 Pre-Phase Task PP.4: SP16 P2/T3 HOLD_COST_SCALE chain is + /// retired by DD7=c (replaced by D-leg structural Hold opportunity-cost + /// reward). Slots 461 (HOLD_COST_SCALE_INDEX) and [462..468) (HCS_*) + /// remain ALLOCATED in the registry with sentinel 0.0 to preserve + /// checkpoint layout-fingerprint compatibility (SP14-C.1 RESERVED-gap + /// pattern), but their descriptions are updated to the canonical SP18 + /// retirement marker so future readers know the producer kernel + 3 + /// consumer sites in `experience_kernels.cu` are deleted by SP18 Phase 1. + /// MHT_* slots [468..474) are NOT retired — MIN_HOLD_TEMPERATURE remains + /// in the SP16 P1 observation chain per DD7=c "keep the observation + /// chain". SP13 P0a slots [380..383) likewise remain (observation only — + /// the per-step hold_rate_observer chain that feeds MIN_HOLD_TEMPERATURE). + #[test] + fn sp16_t2_t3_slots_marked_retired() { + let r = StateResetRegistry::new(); + let by_name: std::collections::HashMap<&str, &RegistryEntry> = + r.all().iter().map(|e| (e.name, e)).collect(); + + let retired: &[&str] = &[ + "sp16_phase2_hold_cost_scale_adaptive", // slot 461 + "sp16_t3_hcs_target_mean", // slot 462 + "sp16_t3_hcs_target_m2", // slot 463 + "sp16_t3_hcs_diff_mean", // slot 464 + "sp16_t3_hcs_diff_m2", // slot 465 + "sp16_t3_hcs_prev_target", // slot 466 + "sp16_t3_hcs_sample_count", // slot 467 + ]; + + for name in retired { + let entry = by_name.get(name).unwrap_or_else(|| { + panic!("SP18 v2 PP.4: retired SP16 P2/T3 entry '{}' missing from registry", name) + }); + assert!( + entry.description.contains("SP18 v2: RETIRED"), + "SP18 v2 PP.4: entry '{}' description must contain 'SP18 v2: RETIRED' marker; got: {}", + name, entry.description + ); + assert_eq!( + entry.category, + ResetCategory::FoldReset, + "Retired SP16 P2/T3 entry '{}' stays FoldReset (sentinel rewrite preserves \ + bit-identical pre-SP18 cold-start; the slot just isn't driven by a producer \ + any more); got: {:?}", name, entry.category + ); + } + + // Verify the MHT_* slots [468..474) and SP13 P0a slots [380..383) + // remain UN-retired — DD7=c preserves the observation chain. + let preserved: &[&str] = &[ + "sp16_t3_mht_target_mean", // slot 468 + "sp16_t3_mht_sample_count", // slot 473 + // SP13 P0a slots are not in the registry by these names — they + // are part of the per-step hold_rate_observer chain rather than + // FoldReset slots. We just assert MHT_* are not flagged retired. + ]; + for name in preserved { + let entry = by_name.get(name).unwrap_or_else(|| { + panic!("Expected SP16 T3 MHT_* entry '{}' to exist but registry lookup failed", name) + }); + assert!( + !entry.description.contains("SP18 v2: RETIRED"), + "SP16 T3 MHT entry '{}' must NOT carry SP18 retirement marker (DD7=c keeps \ + the MIN_HOLD_TEMPERATURE observation chain); got: {}", + name, entry.description + ); + } + } } diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index b905d8975..3dd0a1f5a 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -2,6 +2,36 @@ **Status:** Populated during Plan 1 Task 6 (A.5 orphan audit). Updated on every commit per Invariant 7. +## 2026-05-08 — SP18 v2 Pre-Phase Task PP.4: SP16 P2/T3 retirement marker + +Pre-Phase scaffolding marking the SP16 P2/T3 HOLD_COST_SCALE chain as RETIRED per DD7=c. The SP18 D-leg structural Hold opportunity-cost reward (compute_sp18_hold_opportunity_cost in trade_physics.cuh + ISV[483..493) cap producer chain) replaces the SP13 P0a + SP16 P2 + SP16 T3 reactive per-bar Hold-cost chain. The retirement is **slot-level only** at this stage — Phase 1 of SP18 deletes the producer kernel + 3 consumer sites + host controller atomically per `feedback_no_partial_refactor`. + +**Slots retired** (description updated; sentinel preserved at 0.0 for checkpoint compat): + +| Slot | Name | Reason | +|---|---|---| +| 461 | sp16_phase2_hold_cost_scale_adaptive | HOLD_COST_SCALE producer kernel deleted in Phase 1 | +| 462 | sp16_t3_hcs_target_mean | Welford accumulator no longer driven | +| 463 | sp16_t3_hcs_target_m2 | Welford accumulator no longer driven | +| 464 | sp16_t3_hcs_diff_mean | Welford accumulator no longer driven | +| 465 | sp16_t3_hcs_diff_m2 | Welford accumulator no longer driven | +| 466 | sp16_t3_hcs_prev_target | Welford accumulator no longer driven | +| 467 | sp16_t3_hcs_sample_count | Welford accumulator no longer driven | + +**Slots preserved** (DD7=c — "keep the observation chain"): + +| Slot | Name | Why preserved | +|---|---|---| +| 380..383 | SP13 P0a HOLD_COST + HOLD_RATE_TARGET + HOLD_RATE_OBSERVED_EMA | Per-step hold_rate_observer kernel still writes; SP18 D-leg HEALTH_DIAG observability reads observed Hold rate | +| 460 | MIN_HOLD_TEMPERATURE_ADAPTIVE | Separate producer; controls min-hold soft penalty (`compute_min_hold_penalty` in trade_physics.cuh), NOT in SP18 deletion scope | +| 468..474 | sp16_t3_mht_* | MHT Welford accumulators for MIN_HOLD_TEMPERATURE — still active | + +**Pattern**: SP14-C.1 RESERVED-gap precedent — retired slots stay in the `layout_fingerprint_seed` and the `state_reset_registry` (sentinel 0.0 rewritten each fold) so checkpoints with the old slot names continue to load. Slot consumers in legacy code paths fall back to scale=1.0 when reading sentinel 0.0 (bit-identical pre-Phase-2 v3-P0a Hold-cost magnitude). The Phase 1 atomic delete removes the **producer kernel and consumer sites**, not the slot allocation. New `sp16_t2_t3_slots_marked_retired` lock test asserts the retirement marker is in place; pairs with the existing `sp18_fold_reset_entries_present` test for full Pre-Phase coverage. + +**Atomicity envelope**: Pre-Phase scaffolding only — descriptions updated, registry entries kept at FoldReset, sentinels unchanged. The producer kernel deletion + 3-site consumer migration is Phase 1 of SP18 (atomic per `feedback_no_partial_refactor`). + +**Plan**: `docs/superpowers/plans/2026-05-08-sp18-reward-shape-hold-attractor.md` § Pre-Phase Task PP.4 + § DD7. + ## 2026-05-08 — SP18 v2 Pre-Phase Task PP.3: 22 fold-reset registry entries + dispatch arms Pre-Phase scaffolding completing the FoldReset wire-up for the 22 SP18 ISV slots from PP.2: