gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

plan(sp18 v2): Phase 0 Task 0.5 — pearl candidate draft

Browse Source 
Drafts `pearl_diagnostic_decomposition_before_reward_intervention` per
the plan's Task 0.5 deliverable. Captures the discipline pattern
applied in Tasks 0.1–0.2:

  Before changing reward weights or Bellman target arithmetic to fight
  a Q-attractor, instrument a per-action decomposition of the existing
  reward components AND a trajectory observable on the Q-target
  distribution. The instrumentation is observability-only and lands
  atomically. One epoch of dispatch surfaces whether the suspected
  pathology is real BEFORE any production-path code changes.

Pearl is DRAFT — pending the Task 0.3 (reviewer L40S 1-epoch
dispatch) + Task 0.4 (KILL CRITERION evaluation) outcomes. Promotion
to user memory + cross-reference to MEMORY.md is gated on both legs
returning PROCEED.

Cross-references: pearl_canary_input_freshness_launch_order,
pearl_first_observation_bootstrap, pearl_wiener_alpha_floor_for_
nonstationary, feedback_no_partial_refactor, feedback_wire_everything_up.

No production code changes — single doc-only commit.

Plan: docs/superpowers/plans/2026-05-08-sp18-reward-shape-hold-attractor.md
      § Phase 0 Task 0.5.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-09 01:20:08 +02:00
parent 27f8e332da
commit 662bf7eb36
1 changed files with 94 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

94
docs/superpowers/notes/pearl_diagnostic_decomposition_before_reward_intervention.md Normal file
View File

@@ -0,0 +1,94 @@
# Pearl draft: diagnostic decomposition before reward intervention
> **Status:** DRAFT — pending Phase 0 Task 0.4 KILL CRITERION reviewer evaluation. If the 1-epoch L40S baseline (Task 0.3) confirms the SP18 D-leg PROCEED criterion (`|micro_hold_ema| < 0.5 × max(|popart_long_ema|, |popart_short_ema|)` AND Hold% rising) AND the B-leg PROCEED criterion (V_SHARE slope > 0.05 over HD0→HD3 AND `v_share[HD3] > 0.7`), this pearl is promoted to user memory. If the run shows ABORT or INCONCLUSIVE, this draft stays in `docs/superpowers/notes/` and the plan is revisited.
>
> **Source:** SP18 v2 Phase 0 (2026-05-08), `docs/superpowers/plans/2026-05-08-sp18-reward-shape-hold-attractor.md`.
## Pattern
Before changing reward weights or Bellman target arithmetic to fight a Q-attractor, instrument a **per-action decomposition** of the existing reward components AND a **trajectory observable** on the Q-target distribution. The instrumentation is observability-only — no consumer wired — and lands atomically (kernel + launcher + HEALTH_DIAG emit + GPU oracle test in one commit). One epoch of dispatch surfaces whether the suspected pathology is real BEFORE any production-path code changes.
## Why
The SP18 plan amends the SP13 P0a + SP16 P2 + SP16 T3 reactive per-bar Hold-cost chain because vtj9r showed `Hold%` climbing HD2→HD3 +13.4 pts with `v_share` monotone HD0→HD4 (0.33→0.83). Two competing hypotheses for the persistent Q(Hold) attractor:
1. **D-leg (reward-shape):** the per-bar Hold cost (`-ISV[380] × ISV[461]` capped at `[-0.625, -0.0025]`) is too small relative to Long/Short reward magnitudes; the Q-target distribution for Hold is structurally tighter than Long/Short, so the optimizer converges on Hold even when the directional signal exists.
2. **B-leg (TD(λ) Q(s') bootstrap bias):** `q_next = rewards` self-bootstrap at `gpu_experience_collector.rs:4143` creates a degenerate target distribution — Q(s,a) regresses toward the **mean reward** rather than the **expected discounted return**, so V_SHARE elevates (V dominates A) and action selection collapses to the lowest-variance action (Hold).
Both hypotheses predict the same symptom (Q(Hold) attractor) but require different fixes. **Instrumenting BEFORE fixing tells us which is real.**
The Phase 0 instrumentation:
- **D-leg observable** — per-action reward decomposition (`mean(r_micro)`, `mean(r_opp_cost)`, `mean(r_popart)`, `mean(|reward|)`, `fire_rate`) per direction-axis bin (Hold/Long/Short/Flat) via a single block-tree-reduce kernel reading the existing `reward_components_per_sample` buffer + `actions_out`. Emits a single HEALTH_DIAG `reward_decomp` line per epoch. Surfaces empirically whether Hold's reward distribution is structurally tighter than Long/Short's.
- **B-leg observable** — V_SHARE slope across last 4 epochs `(EMA[now] - EMA[now-4]) / 4` per branch (host-side ring buffer reading the SP17 V_SHARE EMA at slots [478..482)) + TD-error magnitude EMA `mean(|td_errors[b]|)` via a single-block kernel reading the existing `td_errors_buf [B]` post-train-step. Emits two HEALTH_DIAG lines per epoch (`v_share_traj`, `td_error_pre`). Surfaces the V_SHARE monotone trend (B-leg synergy hypothesis: bootstrap-bias drives V_SHARE elevation; flattening or reversal post-fix confirms the hypothesis) and establishes the pre-fix `mean(|TD-error|)` baseline for the B-DD9 ratio gate.
KILL CRITERION (per plan Task 0.4):
```
ABORT D-leg if |r_micro_hold_ema| > 2 × |r_micro_long_ema|
AND Hold% trajectory is FALLING.
ABORT B-leg if v_share_slope < 0
AND v_share[HD0] < 0.5.
PROCEED on combined if D-leg PROCEED criterion AND B-leg PROCEED criterion BOTH fire.
INCONCLUSIVE if either ABORT/PROCEED is mixed — convene human review.
```
The KILL CRITERION is **NOT** a fix gate — it's a sanity check that the diagnostic instrumentation produces interpretable readings before the Phase 9 smoke verdict is trusted.
## When to apply
- A persistent Q-attractor symptom has multiple competing structural explanations (reward-shape vs Bellman-target vs network-architecture).
- Each fix is a multi-commit atomic refactor (per `feedback_no_partial_refactor`) that is expensive to roll back.
- The existing per-step buffers already carry the data needed for the decomposition (i.e. you don't need new collectors — just a new producer kernel reading the existing buffers).
- The instrumentation can be reduced to a small atomic commit (kernel + launcher + HEALTH_DIAG emit + GPU oracle test in one commit per task per `feedback_wire_everything_up`).
## When NOT to apply
- The decomposition would require new per-step buffers that hot-path code doesn't already produce (then the diagnostic itself is a multi-commit refactor — at that point write the spec for the fix and skip Phase 0).
- The hypothesis already has a closed-form smoke test (e.g. SP14-C aux loss CE plateau at `ln(K)` — no decomposition needed; gradient flow audit suffices).
- Each candidate fix is a one-line constant change (Phase 0 is overkill for a single LR or α tuning).
## Implementation pattern (canonical SP18 v2)
```
Phase 0 commit 1: D-leg decomposition kernel + launcher + HEALTH_DIAG + GPU oracle test.
Phase 0 commit 2: B-leg V_SHARE trend (host ring buffer) + TD-error magnitude EMA kernel
+ HEALTH_DIAG lines + GPU oracle tests.
Phase 0 commit 3: this pearl draft.
Reviewer: push branch, dispatch 1-epoch L40S `--label sp18-phase0-decomp`.
Reviewer: tail HEALTH_DIAG `reward_decomp` + `v_share_traj` + `td_error_pre` lines.
Reviewer: evaluate KILL CRITERION → PROCEED / ABORT / INCONCLUSIVE.
```
The Phase 0 cost is **2 small commits + 1 epoch L40S dispatch + 1 reviewer reading-of-HEALTH-DIAG**. The cost saved is the multi-week atomic D-leg refactor (Phase 1–3, 16 tasks across 3 phases) AND the B-leg target-net forward pass refactor (Phase 4–5, 9 tasks) for hypothesis branches that turn out to be wrong.
## Bit-identical checkpoint compatibility
Phase 0 producer kernels write to **new ISV slots** (`[493..497)` for the B-leg: `TD_ERROR_MAG_EMA`, `Q_NEXT_TARGET_P99`, `Q_NEXT_MINUS_REWARD_P99`, `V_SHARE_TREND_DIAG`) — never to existing slots. Cold-start sentinels (0.0 for diagnostic slots; 1.0 for the one-shot `POPART_RESET_FLAG`) keep the layout fingerprint compatible with checkpoints from prior superprojects. Observers never modify production-path buffers (e.g. `td_errors_buf` is read-only in the producer; `reward_components_per_sample` is read by the diag kernel BEFORE the existing `reward_component_ema_kernel` zeros it, per `pearl_canary_input_freshness_launch_order`).
## Cross-references
- `pearl_canary_input_freshness_launch_order` — ordering guarantee for buffers consumed by multiple producers.
- `pearl_first_observation_bootstrap` — sentinel = 0.0; first observation REPLACES directly.
- `pearl_wiener_alpha_floor_for_nonstationary` — α=0.4 floor for non-stationary control loops; Phase 0 uses fixed α=0.4 for the TD-error magnitude EMA pending the Phase 4 q_next_target Welford machinery.
- `feedback_no_partial_refactor` — atomic kernel + launcher + emit + test per task.
- `feedback_wire_everything_up` — every kernel/buffer/slot wired in same commit; no orphans.
## Promotion checklist (post-Phase 0 KILL CRITERION)
If Task 0.3 + 0.4 evaluation confirms PROCEED:
1. Move this draft to user memory (`/home/jgrusewski/.claude/projects/.../memory/pearl_diagnostic_decomposition_before_reward_intervention.md`).
2. Add the slug to `MEMORY.md` § "Patterns / Pearls (technique discoveries)" → "Controllers and signals" or "Architectural constraints" depending on which KILL CRITERION leg fired.
3. Add empirical anchor to the pearl: vtj9r baseline + post-Phase-0 `reward_decomp` + `v_share_traj` numbers (HD0–HD4 trajectory tables) so future SPs comparing against this pattern have hard data.
4. Update SP18 spec § "Why combined scope" with a backreference to the pearl as the discipline source.
If Task 0.4 evaluation returns ABORT or INCONCLUSIVE:
- Leave this draft in `docs/superpowers/notes/`.
- Add a "rejected" subsection summarising why the pattern did NOT generalise on this run.
- Revise the SP18 plan to address the surprise; do not promote.