docs(superpowers): edge investigation decision record + F4-gating/reward-alignment specs & plans
Session 2026-06-05: investigation-first falsification gated the F4 oracle build (7-14 days) and walked back through the reward/edge question to a code-verified root cause — the MBP-10 decoder corrupts the inside quote (dbn_parser.rs:903 'for lvl in 1..max_lvl' never restores levels[0]), contaminating mid/feature/reward/fill locally and on the cluster PVC. - notes/2026-06-05-edge-investigation-decision-record.md: full 8-verdict arc (incl. 2 self-retractions) + next-session plan (confirm on PVC, fix parser, regenerate sidecars, re-audit on clean data). - specs: f4-oracle-pretraining (GATED), reward-pnl-alignment-restoration (superseded by the data-quality finding), bellman-target-foundation-reshape. - plans: reward-pnl-alignment-restoration-implementation, f2-q-centered-distill. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -0,0 +1,59 @@
|
||||
# Decision Record — Edge Investigation & MBP-10 Root Cause (2026-06-05)
|
||||
|
||||
**Branch:** `ml-alpha-regime-observer`
|
||||
**Outcome in one line:** Started to build F4 oracle pretraining (7-14 days); instead, investigation-first falsification gated F4, walked back through the reward/edge question, and **found a code-verified root-cause bug — the MBP-10 decoder corrupts the inside quote (`levels[0]`) — that contaminates every mid/feature/reward/fill, locally AND on the cluster PVC.**
|
||||
|
||||
---
|
||||
|
||||
## The arc (each step a falsifiable verdict)
|
||||
|
||||
1. **F4 oracle pretraining — GATED, not built.** Fresh-eyes research (4 parallel omnisearch agents) + a 1-hour measurement showed F4 would warm-start Q onto a reward only **Pearson 0.28** aligned with USD pnl (gate 0.70). Root cause of that: slot-753 (`surfer_scaffold_weight`) 0.0 "pure-pnl" bootstrap is **cosmetic** — the controller (`rl_surfer_scaffold_controller.cu:101`) overwrites it to ~0.97 every step (no force-pin gate). External literature (Cal-QL/WSRL/IDAAC, 2023-2025): value-pretraining is an optimization, not a generalization, lever. → Pearl: `pearl_reward_misalign_blocks_f4_slot753_override`.
|
||||
|
||||
2. **Falsification: value generalizes, agent loses on both folds.** `loss.v`/`loss.q` eval/train ratios 0.91×/0.83× (value does NOT overfit), yet eval pnl deeply negative. → reward *target*, not Q-init, was the suspected lever.
|
||||
|
||||
3. **Phase A force-pin (committed) — works, but reveals more.** Added slot 824 `RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX` + `FOXHUNT_PIN_SURFER_SCAFFOLD` env flag + kernel gate (commit `d8447475c`). RA-G1 ✓ (slot 753 held 0.0), RA-G2 ✓ (determinism). But pinning to pure-pnl **collapsed trading** (entropy 0.015, eval 0 trades) — a b=128 popart-starvation confound, disambiguated by enabling popart-normalize (trading recovered to entropy 1.87, 1,278 eval trades) — **yet still −$3.57M eval.** A *healthy* policy on aligned-intent reward still loses → reward alignment weakened as the lever. → Pearls: `pearl_shaping_is_dual_purpose_align_vs_explore`.
|
||||
|
||||
4. **Data-edge audit (short horizons) — no tradeable edge.** Simple non-RL predictor, train Q1→eval Q2: the only OOS-persistent signal is the bid-ask bounce, untradeable net of spread. → `pearl_data_edge_audit_no_tradeable_oos_edge_local`.
|
||||
|
||||
5. **LobSim cost audit — sim is HONEST.** Book-walk fills cross the spread (enter ask / exit bid); MTM-at-mid telescopes to ~0 per closed trade; commission 0 (minor). Eval loss is genuine. → refuted the "free bounce" hypothesis (addendum to the pearl above).
|
||||
|
||||
6. **Long-horizon edge audit — "edge exists!" (FALSE alarm).** On *mid prices*, a ~6hr mean-reversion looked tradeable (OOS t=5.06). → `pearl_edge_exists_at_multihour_mean_reversion` (now RETRACTED).
|
||||
|
||||
7. **Rule-based baseline with REAL fills — RETRACTS #6.** Enter ask / exit bid: loses in every θ/H cell OOS. The "edge" was **phantom reversion into degraded quotes** (`corr(|z|,spread)=0.91`; 99.8% of signals fire on spread>5pt). **Lesson: never trust mid/abstract-cost backtests without bid/ask fill validation.** → `pearl_local_data_degraded_edge_was_quote_artifact`.
|
||||
|
||||
8. **Degradation source — ROOT CAUSE (code-verified).** The degraded top-of-book is a **decoder bug**, not thin liquidity (no time-clustering; 89% single-snapshot spikes; 6.4% crossed; L1 strictly inside L0 in 100% of wide cases; reconstruction from L1 is clean). → `pearl_mbp10_level0_decode_bug_root_cause`.
|
||||
|
||||
---
|
||||
|
||||
## The root cause (the deliverable)
|
||||
|
||||
`crates/data/src/providers/databento/dbn_parser.rs`:
|
||||
- **L876-883** writes the single MBP-10 *event's* `(price,size)` into `levels[0]` — but an event is a delta to one level at depth `mbp10.depth`, not the inside.
|
||||
- **L902-903** the "Phase E.1 fix" copies the real book `mbp10.levels[1..]` but the loop **starts at 1**, never overwriting the corrupt L0 with the canonical `mbp10.levels[0]`.
|
||||
|
||||
**Blast radius:** the same parser decodes the 148GB cluster PVC. Corrupts mid, microprice, spread, L0-OFI features, the Phase-5 mid-based MTM reward, and the LOB-sim `levels[0]` fill reference — i.e. the agent has been training and trading on phantom inside quotes (~10% wide, 6.4% crossed). Falsifies the old "cluster data is fine" claim.
|
||||
|
||||
---
|
||||
|
||||
## Decision & next action
|
||||
|
||||
**Adopt the MBP-10 L0 decode fix as the next work, ahead of any RL/reward/architecture effort.** Every prior intervention (F4, F2, F5, reward-alignment, 64 commits) operated on a corrupt data substrate; fixing it is the prerequisite for any trustworthy result.
|
||||
|
||||
Plan (next session):
|
||||
1. **Confirm on PVC first** — run the diagnostic dump (crossed %, L1-inside-L0 %, spread p50/p99) on a cluster PVC ES file to verify the corruption is present there too (inferred from shared code, not yet measured).
|
||||
2. **Fix** `dbn_parser.rs`: line 903 `1..max_lvl` → `0..max_lvl`; drop the `update_level(0,...)` at 876-883. Add a unit test asserting no crossed/locked L0 on a known record set. (TDD; ~½ day code.)
|
||||
3. **Regenerate predecoded sidecars** — the cache validates by SOURCE mtime/size (`ml-features/src/predecoded.rs`), so a parser change will NOT auto-invalidate. Delete + regenerate local `test_data` and cluster PVC sidecars (148GB — the heavy step; Argo job).
|
||||
4. **Re-validate data** (spread p50≈1 tick, crossed≈0), then **re-run the edge audit on clean data** (short AND long horizons, with real fills), then re-assess whether RL training has a real substrate.
|
||||
|
||||
---
|
||||
|
||||
## Artifacts & state
|
||||
|
||||
- **Committed (branch `ml-alpha-regime-observer`):** Phase A force-pin (slot 824, `d8447475c`); `scripts/measure_reward_alignment.py` (`e57b07657`).
|
||||
- **Reverted:** the `FOXHUNT_POPART_NORMALIZE` disambiguation scaffold in `integrated.rs` (its finding is in `pearl_shaping_is_dual_purpose_align_vs_explore`). Re-add if testing popart on clean data.
|
||||
- **Specs/plans (uncommitted, this session):** `docs/superpowers/specs/2026-06-05-f4-oracle-pretraining-infrastructure.md` (GATED), `docs/superpowers/specs/2026-06-05-reward-pnl-alignment-restoration.md` (superseded by the data-quality finding), `docs/superpowers/plans/2026-06-05-reward-pnl-alignment-restoration-implementation.md`.
|
||||
- **Memory pearls added:** reward-misalign-blocks-f4 / shaping-dual-purpose / data-edge-audit / edge-exists (retracted) / local-data-degraded / **mbp10-level0-decode-bug-root-cause**.
|
||||
- **Diagnostic data:** `/tmp/diag_q1.csv`, `/tmp/edge*_*.csv`, `/tmp/bt_*.csv`.
|
||||
|
||||
## Methodology note
|
||||
This session is a clean demonstration of `feedback_investigation_first_falsification_methodology`: spec→measure→falsify→STOP-on-surprise, applied through **8 sequential verdicts including 2 self-retractions** (F4 gating; the "edge exists" reversal). It converted a planned 2-week F4 build into the discovery of a ~2-line root-cause bug. Cost: a handful of ~10-min local smokes + parallel research/dump agents. No cluster spend.
|
||||
40
docs/superpowers/notes/2026-06-05-next-session-prompt.md
Normal file
40
docs/superpowers/notes/2026-06-05-next-session-prompt.md
Normal file
@@ -0,0 +1,40 @@
|
||||
# Next session resume prompt
|
||||
|
||||
Copy-paste this into a new Claude Code session to pick up Phase 7c F4 oracle pretrain work:
|
||||
|
||||
---
|
||||
|
||||
Resume foxhunt Phase 7c F4 oracle pretraining work.
|
||||
|
||||
Branch: `ml-alpha-regime-observer` at HEAD `7376b1c67` (6 commits ahead; pushed). F2 (slot 817) + F5 (slot 823) + Option A gate-patches engaged. F2+F5+OptA = SOTA local at eval_pnl=-$415,237 (best-of-session, +$29k vs Phase 5). Cluster sk2z9 bled 3× faster than F2-alone with 6:1 Short:Long directional bias → killed at step 519 → F4 oracle pretrain is the escalation per spec §5.3 outcome 3.
|
||||
|
||||
**Read these memory pearls first** (in this order):
|
||||
1. `project_phase7_state_2026_06_05.md` — full Phase 7 state, branch + slots + verdicts
|
||||
2. `pearl_f2_q_centered_distill_necessary_not_sufficient.md` — why F2 alone failed
|
||||
3. `pearl_f5_state_mask_neutralized_by_downstream_gates.md` — F5 gate-interop pattern
|
||||
4. `pearl_f2_plus_f5_unlock_q_signal.md` — why combined helps; cluster bleed
|
||||
5. `reference_outcome_head_supervised_pattern.md` — `outcome_head.rs` is the F4 Phase 7c.2 precedent
|
||||
6. `feedback_investigation_first_falsification_methodology.md` — methodology that broke 64-commit going-in-circles
|
||||
|
||||
**Read this spec next**:
|
||||
- `docs/superpowers/specs/2026-06-05-f4-oracle-pretraining-infrastructure.md` (733 lines, 6 phases 7c.0-7c.5, ~7-14 days bounded)
|
||||
|
||||
**Next step**: write the F4 implementation plan at `docs/superpowers/plans/2026-06-05-f4-oracle-pretraining-implementation.md` (similar 1500-LOC structure to F2 plan), then dispatch Phase 7c.0 oracle pipeline coder (2-3 days).
|
||||
|
||||
**Top 3 risks per F4 spec §7**:
|
||||
1. Oracle heuristic itself wrong → F4-G1 Pearson < 0.7 → STOP at Phase 7c.0
|
||||
2. Reward function identity drift → MUST use realistic-Rust+CUDA oracle (NOT Python)
|
||||
3. Q drift undoes warm-start → F4-G4/G6 are detection gates
|
||||
|
||||
**Key code references**:
|
||||
- F2 kernel: `crates/ml-alpha/cuda/rl_q_pi_distill_grad.cu` Step 2 hinge (lines ~101-150)
|
||||
- F5 kernel: `crates/ml-alpha/cuda/rl_state_action_mask.cu`
|
||||
- OutcomeHead pattern: `crates/ml-alpha/src/rl/outcome_head.rs`
|
||||
- F2 plan template: `docs/superpowers/plans/2026-06-04-f2-q-centered-distill-implementation.md`
|
||||
|
||||
**Open questions for the plan** (resolve before dispatching coder):
|
||||
- Reward function versioning hash protocol (Phase 5 slots + Phase 7a/7b slots + γ + K → filename embedding)
|
||||
- Held-out subset for F4-G3 (last 10% of fold-1 train OR fold-1 eval discarded)
|
||||
- Encoder LR for pretraining (same as RL fine-tune for MVP simplicity OR separate higher LR)
|
||||
|
||||
Proceed with writing the F4 implementation plan, then begin Phase 7c.0.
|
||||
Reference in New Issue
Block a user