From 662bf7eb361f49f6c6313abf3e6e2540591c911d Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sat, 9 May 2026 01:20:08 +0200 Subject: [PATCH] =?UTF-8?q?plan(sp18=20v2):=20Phase=200=20Task=200.5=20?= =?UTF-8?q?=E2=80=94=20pearl=20candidate=20draft?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drafts `pearl_diagnostic_decomposition_before_reward_intervention` per the plan's Task 0.5 deliverable. Captures the discipline pattern applied in Tasks 0.1–0.2: Before changing reward weights or Bellman target arithmetic to fight a Q-attractor, instrument a per-action decomposition of the existing reward components AND a trajectory observable on the Q-target distribution. The instrumentation is observability-only and lands atomically. One epoch of dispatch surfaces whether the suspected pathology is real BEFORE any production-path code changes. Pearl is DRAFT — pending the Task 0.3 (reviewer L40S 1-epoch dispatch) + Task 0.4 (KILL CRITERION evaluation) outcomes. Promotion to user memory + cross-reference to MEMORY.md is gated on both legs returning PROCEED. Cross-references: pearl_canary_input_freshness_launch_order, pearl_first_observation_bootstrap, pearl_wiener_alpha_floor_for_ nonstationary, feedback_no_partial_refactor, feedback_wire_everything_up. No production code changes — single doc-only commit. Plan: docs/superpowers/plans/2026-05-08-sp18-reward-shape-hold-attractor.md § Phase 0 Task 0.5. Co-Authored-By: Claude Opus 4.7 (1M context) --- ...ecomposition_before_reward_intervention.md | 94 +++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 docs/superpowers/notes/pearl_diagnostic_decomposition_before_reward_intervention.md diff --git a/docs/superpowers/notes/pearl_diagnostic_decomposition_before_reward_intervention.md b/docs/superpowers/notes/pearl_diagnostic_decomposition_before_reward_intervention.md new file mode 100644 index 000000000..0dcde38fe --- /dev/null +++ b/docs/superpowers/notes/pearl_diagnostic_decomposition_before_reward_intervention.md @@ -0,0 +1,94 @@ +# Pearl draft: diagnostic decomposition before reward intervention + +> **Status:** DRAFT — pending Phase 0 Task 0.4 KILL CRITERION reviewer evaluation. If the 1-epoch L40S baseline (Task 0.3) confirms the SP18 D-leg PROCEED criterion (`|micro_hold_ema| < 0.5 × max(|popart_long_ema|, |popart_short_ema|)` AND Hold% rising) AND the B-leg PROCEED criterion (V_SHARE slope > 0.05 over HD0→HD3 AND `v_share[HD3] > 0.7`), this pearl is promoted to user memory. If the run shows ABORT or INCONCLUSIVE, this draft stays in `docs/superpowers/notes/` and the plan is revisited. +> +> **Source:** SP18 v2 Phase 0 (2026-05-08), `docs/superpowers/plans/2026-05-08-sp18-reward-shape-hold-attractor.md`. + +## Pattern + +Before changing reward weights or Bellman target arithmetic to fight a Q-attractor, instrument a **per-action decomposition** of the existing reward components AND a **trajectory observable** on the Q-target distribution. The instrumentation is observability-only — no consumer wired — and lands atomically (kernel + launcher + HEALTH_DIAG emit + GPU oracle test in one commit). One epoch of dispatch surfaces whether the suspected pathology is real BEFORE any production-path code changes. + +## Why + +The SP18 plan amends the SP13 P0a + SP16 P2 + SP16 T3 reactive per-bar Hold-cost chain because vtj9r showed `Hold%` climbing HD2→HD3 +13.4 pts with `v_share` monotone HD0→HD4 (0.33→0.83). Two competing hypotheses for the persistent Q(Hold) attractor: + +1. **D-leg (reward-shape):** the per-bar Hold cost (`-ISV[380] × ISV[461]` capped at `[-0.625, -0.0025]`) is too small relative to Long/Short reward magnitudes; the Q-target distribution for Hold is structurally tighter than Long/Short, so the optimizer converges on Hold even when the directional signal exists. + +2. **B-leg (TD(λ) Q(s') bootstrap bias):** `q_next = rewards` self-bootstrap at `gpu_experience_collector.rs:4143` creates a degenerate target distribution — Q(s,a) regresses toward the **mean reward** rather than the **expected discounted return**, so V_SHARE elevates (V dominates A) and action selection collapses to the lowest-variance action (Hold). + +Both hypotheses predict the same symptom (Q(Hold) attractor) but require different fixes. **Instrumenting BEFORE fixing tells us which is real.** + +The Phase 0 instrumentation: + +- **D-leg observable** — per-action reward decomposition (`mean(r_micro)`, `mean(r_opp_cost)`, `mean(r_popart)`, `mean(|reward|)`, `fire_rate`) per direction-axis bin (Hold/Long/Short/Flat) via a single block-tree-reduce kernel reading the existing `reward_components_per_sample` buffer + `actions_out`. Emits a single HEALTH_DIAG `reward_decomp` line per epoch. Surfaces empirically whether Hold's reward distribution is structurally tighter than Long/Short's. + +- **B-leg observable** — V_SHARE slope across last 4 epochs `(EMA[now] - EMA[now-4]) / 4` per branch (host-side ring buffer reading the SP17 V_SHARE EMA at slots [478..482)) + TD-error magnitude EMA `mean(|td_errors[b]|)` via a single-block kernel reading the existing `td_errors_buf [B]` post-train-step. Emits two HEALTH_DIAG lines per epoch (`v_share_traj`, `td_error_pre`). Surfaces the V_SHARE monotone trend (B-leg synergy hypothesis: bootstrap-bias drives V_SHARE elevation; flattening or reversal post-fix confirms the hypothesis) and establishes the pre-fix `mean(|TD-error|)` baseline for the B-DD9 ratio gate. + +KILL CRITERION (per plan Task 0.4): + +``` +ABORT D-leg if |r_micro_hold_ema| > 2 × |r_micro_long_ema| + AND Hold% trajectory is FALLING. +ABORT B-leg if v_share_slope < 0 + AND v_share[HD0] < 0.5. + +PROCEED on combined if D-leg PROCEED criterion AND B-leg PROCEED criterion BOTH fire. +INCONCLUSIVE if either ABORT/PROCEED is mixed — convene human review. +``` + +The KILL CRITERION is **NOT** a fix gate — it's a sanity check that the diagnostic instrumentation produces interpretable readings before the Phase 9 smoke verdict is trusted. + +## When to apply + +- A persistent Q-attractor symptom has multiple competing structural explanations (reward-shape vs Bellman-target vs network-architecture). +- Each fix is a multi-commit atomic refactor (per `feedback_no_partial_refactor`) that is expensive to roll back. +- The existing per-step buffers already carry the data needed for the decomposition (i.e. you don't need new collectors — just a new producer kernel reading the existing buffers). +- The instrumentation can be reduced to a small atomic commit (kernel + launcher + HEALTH_DIAG emit + GPU oracle test in one commit per task per `feedback_wire_everything_up`). + +## When NOT to apply + +- The decomposition would require new per-step buffers that hot-path code doesn't already produce (then the diagnostic itself is a multi-commit refactor — at that point write the spec for the fix and skip Phase 0). +- The hypothesis already has a closed-form smoke test (e.g. SP14-C aux loss CE plateau at `ln(K)` — no decomposition needed; gradient flow audit suffices). +- Each candidate fix is a one-line constant change (Phase 0 is overkill for a single LR or α tuning). + +## Implementation pattern (canonical SP18 v2) + +``` +Phase 0 commit 1: D-leg decomposition kernel + launcher + HEALTH_DIAG + GPU oracle test. +Phase 0 commit 2: B-leg V_SHARE trend (host ring buffer) + TD-error magnitude EMA kernel + + HEALTH_DIAG lines + GPU oracle tests. +Phase 0 commit 3: this pearl draft. + +Reviewer: push branch, dispatch 1-epoch L40S `--label sp18-phase0-decomp`. +Reviewer: tail HEALTH_DIAG `reward_decomp` + `v_share_traj` + `td_error_pre` lines. +Reviewer: evaluate KILL CRITERION → PROCEED / ABORT / INCONCLUSIVE. +``` + +The Phase 0 cost is **2 small commits + 1 epoch L40S dispatch + 1 reviewer reading-of-HEALTH-DIAG**. The cost saved is the multi-week atomic D-leg refactor (Phase 1–3, 16 tasks across 3 phases) AND the B-leg target-net forward pass refactor (Phase 4–5, 9 tasks) for hypothesis branches that turn out to be wrong. + +## Bit-identical checkpoint compatibility + +Phase 0 producer kernels write to **new ISV slots** (`[493..497)` for the B-leg: `TD_ERROR_MAG_EMA`, `Q_NEXT_TARGET_P99`, `Q_NEXT_MINUS_REWARD_P99`, `V_SHARE_TREND_DIAG`) — never to existing slots. Cold-start sentinels (0.0 for diagnostic slots; 1.0 for the one-shot `POPART_RESET_FLAG`) keep the layout fingerprint compatible with checkpoints from prior superprojects. Observers never modify production-path buffers (e.g. `td_errors_buf` is read-only in the producer; `reward_components_per_sample` is read by the diag kernel BEFORE the existing `reward_component_ema_kernel` zeros it, per `pearl_canary_input_freshness_launch_order`). + +## Cross-references + +- `pearl_canary_input_freshness_launch_order` — ordering guarantee for buffers consumed by multiple producers. +- `pearl_first_observation_bootstrap` — sentinel = 0.0; first observation REPLACES directly. +- `pearl_wiener_alpha_floor_for_nonstationary` — α=0.4 floor for non-stationary control loops; Phase 0 uses fixed α=0.4 for the TD-error magnitude EMA pending the Phase 4 q_next_target Welford machinery. +- `feedback_no_partial_refactor` — atomic kernel + launcher + emit + test per task. +- `feedback_wire_everything_up` — every kernel/buffer/slot wired in same commit; no orphans. + +## Promotion checklist (post-Phase 0 KILL CRITERION) + +If Task 0.3 + 0.4 evaluation confirms PROCEED: + +1. Move this draft to user memory (`/home/jgrusewski/.claude/projects/.../memory/pearl_diagnostic_decomposition_before_reward_intervention.md`). +2. Add the slug to `MEMORY.md` § "Patterns / Pearls (technique discoveries)" → "Controllers and signals" or "Architectural constraints" depending on which KILL CRITERION leg fired. +3. Add empirical anchor to the pearl: vtj9r baseline + post-Phase-0 `reward_decomp` + `v_share_traj` numbers (HD0–HD4 trajectory tables) so future SPs comparing against this pattern have hard data. +4. Update SP18 spec § "Why combined scope" with a backreference to the pearl as the discipline source. + +If Task 0.4 evaluation returns ABORT or INCONCLUSIVE: + +- Leave this draft in `docs/superpowers/notes/`. +- Add a "rejected" subsection summarising why the pattern did NOT generalise on this run. +- Revise the SP18 plan to address the surprise; do not promote.