# Decision Record — Edge Investigation & MBP-10 Root Cause (2026-06-05)

**Branch:** `ml-alpha-regime-observer`
**Outcome in one line:** Started to build F4 oracle pretraining (7-14 days); instead, investigation-first falsification gated F4, walked back through the reward/edge question, and **found a code-verified root-cause bug — the MBP-10 decoder corrupts the inside quote (`levels[0]`) — that contaminates every mid/feature/reward/fill, locally AND on the cluster PVC.**

---

## The arc (each step a falsifiable verdict)

1. **F4 oracle pretraining — GATED, not built.** Fresh-eyes research (4 parallel omnisearch agents) + a 1-hour measurement showed F4 would warm-start Q onto a reward only **Pearson 0.28** aligned with USD pnl (gate 0.70). Root cause of that: slot-753 (`surfer_scaffold_weight`) 0.0 "pure-pnl" bootstrap is **cosmetic** — the controller (`rl_surfer_scaffold_controller.cu:101`) overwrites it to ~0.97 every step (no force-pin gate). External literature (Cal-QL/WSRL/IDAAC, 2023-2025): value-pretraining is an optimization, not a generalization, lever. → Pearl: `pearl_reward_misalign_blocks_f4_slot753_override`.

2. **Falsification: value generalizes, agent loses on both folds.** `loss.v`/`loss.q` eval/train ratios 0.91×/0.83× (value does NOT overfit), yet eval pnl deeply negative. → reward *target*, not Q-init, was the suspected lever.

3. **Phase A force-pin (committed) — works, but reveals more.** Added slot 824 `RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX` + `FOXHUNT_PIN_SURFER_SCAFFOLD` env flag + kernel gate (commit `d8447475c`). RA-G1 ✓ (slot 753 held 0.0), RA-G2 ✓ (determinism). But pinning to pure-pnl **collapsed trading** (entropy 0.015, eval 0 trades) — a b=128 popart-starvation confound, disambiguated by enabling popart-normalize (trading recovered to entropy 1.87, 1,278 eval trades) — **yet still −$3.57M eval.** A *healthy* policy on aligned-intent reward still loses → reward alignment weakened as the lever. → Pearls: `pearl_shaping_is_dual_purpose_align_vs_explore`.

4. **Data-edge audit (short horizons) — no tradeable edge.** Simple non-RL predictor, train Q1→eval Q2: the only OOS-persistent signal is the bid-ask bounce, untradeable net of spread. → `pearl_data_edge_audit_no_tradeable_oos_edge_local`.

5. **LobSim cost audit — sim is HONEST.** Book-walk fills cross the spread (enter ask / exit bid); MTM-at-mid telescopes to ~0 per closed trade; commission 0 (minor). Eval loss is genuine. → refuted the "free bounce" hypothesis (addendum to the pearl above).

6. **Long-horizon edge audit — "edge exists!" (FALSE alarm).** On *mid prices*, a ~6hr mean-reversion looked tradeable (OOS t=5.06). → `pearl_edge_exists_at_multihour_mean_reversion` (now RETRACTED).

7. **Rule-based baseline with REAL fills — RETRACTS #6.** Enter ask / exit bid: loses in every θ/H cell OOS. The "edge" was **phantom reversion into degraded quotes** (`corr(|z|,spread)=0.91`; 99.8% of signals fire on spread>5pt). **Lesson: never trust mid/abstract-cost backtests without bid/ask fill validation.** → `pearl_local_data_degraded_edge_was_quote_artifact`.

8. **Degradation source — ROOT CAUSE (code-verified).** The degraded top-of-book is a **decoder bug**, not thin liquidity (no time-clustering; 89% single-snapshot spikes; 6.4% crossed; L1 strictly inside L0 in 100% of wide cases; reconstruction from L1 is clean). → `pearl_mbp10_level0_decode_bug_root_cause`.

---

## The root cause (the deliverable)

`crates/data/src/providers/databento/dbn_parser.rs`:
- **L876-883** writes the single MBP-10 *event's* `(price,size)` into `levels[0]` — but an event is a delta to one level at depth `mbp10.depth`, not the inside.
- **L902-903** the "Phase E.1 fix" copies the real book `mbp10.levels[1..]` but the loop **starts at 1**, never overwriting the corrupt L0 with the canonical `mbp10.levels[0]`.

**Blast radius:** the same parser decodes the 148GB cluster PVC. Corrupts mid, microprice, spread, L0-OFI features, the Phase-5 mid-based MTM reward, and the LOB-sim `levels[0]` fill reference — i.e. the agent has been training and trading on phantom inside quotes (~10% wide, 6.4% crossed). Falsifies the old "cluster data is fine" claim.

---

## Decision & next action

**Adopt the MBP-10 L0 decode fix as the next work, ahead of any RL/reward/architecture effort.** Every prior intervention (F4, F2, F5, reward-alignment, 64 commits) operated on a corrupt data substrate; fixing it is the prerequisite for any trustworthy result.

Plan (next session):
1. **Confirm on PVC first** — run the diagnostic dump (crossed %, L1-inside-L0 %, spread p50/p99) on a cluster PVC ES file to verify the corruption is present there too (inferred from shared code, not yet measured).
2. **Fix** `dbn_parser.rs`: line 903 `1..max_lvl` → `0..max_lvl`; drop the `update_level(0,...)` at 876-883. Add a unit test asserting no crossed/locked L0 on a known record set. (TDD; ~½ day code.)
3. **Regenerate predecoded sidecars** — the cache validates by SOURCE mtime/size (`ml-features/src/predecoded.rs`), so a parser change will NOT auto-invalidate. Delete + regenerate local `test_data` and cluster PVC sidecars (148GB — the heavy step; Argo job).
4. **Re-validate data** (spread p50≈1 tick, crossed≈0), then **re-run the edge audit on clean data** (short AND long horizons, with real fills), then re-assess whether RL training has a real substrate.

---

## Artifacts & state

- **Committed (branch `ml-alpha-regime-observer`):** Phase A force-pin (slot 824, `d8447475c`); `scripts/measure_reward_alignment.py` (`e57b07657`).
- **Reverted:** the `FOXHUNT_POPART_NORMALIZE` disambiguation scaffold in `integrated.rs` (its finding is in `pearl_shaping_is_dual_purpose_align_vs_explore`). Re-add if testing popart on clean data.
- **Specs/plans (uncommitted, this session):** `docs/superpowers/specs/2026-06-05-f4-oracle-pretraining-infrastructure.md` (GATED), `docs/superpowers/specs/2026-06-05-reward-pnl-alignment-restoration.md` (superseded by the data-quality finding), `docs/superpowers/plans/2026-06-05-reward-pnl-alignment-restoration-implementation.md`.
- **Memory pearls added:** reward-misalign-blocks-f4 / shaping-dual-purpose / data-edge-audit / edge-exists (retracted) / local-data-degraded / **mbp10-level0-decode-bug-root-cause**.
- **Diagnostic data:** `/tmp/diag_q1.csv`, `/tmp/edge*_*.csv`, `/tmp/bt_*.csv`.

## Methodology note
This session is a clean demonstration of `feedback_investigation_first_falsification_methodology`: spec→measure→falsify→STOP-on-surprise, applied through **8 sequential verdicts including 2 self-retractions** (F4 gating; the "edge exists" reversal). It converted a planned 2-week F4 build into the discovery of a ~2-line root-cause bug. Cost: a handful of ~10-min local smokes + parallel research/dump agents. No cluster spend.
