docs(superpowers): edge investigation decision record + F4-gating/reward-alignment specs & plans
Session 2026-06-05: investigation-first falsification gated the F4 oracle build (7-14 days) and walked back through the reward/edge question to a code-verified root cause — the MBP-10 decoder corrupts the inside quote (dbn_parser.rs:903 'for lvl in 1..max_lvl' never restores levels[0]), contaminating mid/feature/reward/fill locally and on the cluster PVC. - notes/2026-06-05-edge-investigation-decision-record.md: full 8-verdict arc (incl. 2 self-retractions) + next-session plan (confirm on PVC, fix parser, regenerate sidecars, re-audit on clean data). - specs: f4-oracle-pretraining (GATED), reward-pnl-alignment-restoration (superseded by the data-quality finding), bellman-target-foundation-reshape. - plans: reward-pnl-alignment-restoration-implementation, f2-q-centered-distill. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -0,0 +1,59 @@
|
||||
# Decision Record — Edge Investigation & MBP-10 Root Cause (2026-06-05)
|
||||
|
||||
**Branch:** `ml-alpha-regime-observer`
|
||||
**Outcome in one line:** Started to build F4 oracle pretraining (7-14 days); instead, investigation-first falsification gated F4, walked back through the reward/edge question, and **found a code-verified root-cause bug — the MBP-10 decoder corrupts the inside quote (`levels[0]`) — that contaminates every mid/feature/reward/fill, locally AND on the cluster PVC.**
|
||||
|
||||
---
|
||||
|
||||
## The arc (each step a falsifiable verdict)
|
||||
|
||||
1. **F4 oracle pretraining — GATED, not built.** Fresh-eyes research (4 parallel omnisearch agents) + a 1-hour measurement showed F4 would warm-start Q onto a reward only **Pearson 0.28** aligned with USD pnl (gate 0.70). Root cause of that: slot-753 (`surfer_scaffold_weight`) 0.0 "pure-pnl" bootstrap is **cosmetic** — the controller (`rl_surfer_scaffold_controller.cu:101`) overwrites it to ~0.97 every step (no force-pin gate). External literature (Cal-QL/WSRL/IDAAC, 2023-2025): value-pretraining is an optimization, not a generalization, lever. → Pearl: `pearl_reward_misalign_blocks_f4_slot753_override`.
|
||||
|
||||
2. **Falsification: value generalizes, agent loses on both folds.** `loss.v`/`loss.q` eval/train ratios 0.91×/0.83× (value does NOT overfit), yet eval pnl deeply negative. → reward *target*, not Q-init, was the suspected lever.
|
||||
|
||||
3. **Phase A force-pin (committed) — works, but reveals more.** Added slot 824 `RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX` + `FOXHUNT_PIN_SURFER_SCAFFOLD` env flag + kernel gate (commit `d8447475c`). RA-G1 ✓ (slot 753 held 0.0), RA-G2 ✓ (determinism). But pinning to pure-pnl **collapsed trading** (entropy 0.015, eval 0 trades) — a b=128 popart-starvation confound, disambiguated by enabling popart-normalize (trading recovered to entropy 1.87, 1,278 eval trades) — **yet still −$3.57M eval.** A *healthy* policy on aligned-intent reward still loses → reward alignment weakened as the lever. → Pearls: `pearl_shaping_is_dual_purpose_align_vs_explore`.
|
||||
|
||||
4. **Data-edge audit (short horizons) — no tradeable edge.** Simple non-RL predictor, train Q1→eval Q2: the only OOS-persistent signal is the bid-ask bounce, untradeable net of spread. → `pearl_data_edge_audit_no_tradeable_oos_edge_local`.
|
||||
|
||||
5. **LobSim cost audit — sim is HONEST.** Book-walk fills cross the spread (enter ask / exit bid); MTM-at-mid telescopes to ~0 per closed trade; commission 0 (minor). Eval loss is genuine. → refuted the "free bounce" hypothesis (addendum to the pearl above).
|
||||
|
||||
6. **Long-horizon edge audit — "edge exists!" (FALSE alarm).** On *mid prices*, a ~6hr mean-reversion looked tradeable (OOS t=5.06). → `pearl_edge_exists_at_multihour_mean_reversion` (now RETRACTED).
|
||||
|
||||
7. **Rule-based baseline with REAL fills — RETRACTS #6.** Enter ask / exit bid: loses in every θ/H cell OOS. The "edge" was **phantom reversion into degraded quotes** (`corr(|z|,spread)=0.91`; 99.8% of signals fire on spread>5pt). **Lesson: never trust mid/abstract-cost backtests without bid/ask fill validation.** → `pearl_local_data_degraded_edge_was_quote_artifact`.
|
||||
|
||||
8. **Degradation source — ROOT CAUSE (code-verified).** The degraded top-of-book is a **decoder bug**, not thin liquidity (no time-clustering; 89% single-snapshot spikes; 6.4% crossed; L1 strictly inside L0 in 100% of wide cases; reconstruction from L1 is clean). → `pearl_mbp10_level0_decode_bug_root_cause`.
|
||||
|
||||
---
|
||||
|
||||
## The root cause (the deliverable)
|
||||
|
||||
`crates/data/src/providers/databento/dbn_parser.rs`:
|
||||
- **L876-883** writes the single MBP-10 *event's* `(price,size)` into `levels[0]` — but an event is a delta to one level at depth `mbp10.depth`, not the inside.
|
||||
- **L902-903** the "Phase E.1 fix" copies the real book `mbp10.levels[1..]` but the loop **starts at 1**, never overwriting the corrupt L0 with the canonical `mbp10.levels[0]`.
|
||||
|
||||
**Blast radius:** the same parser decodes the 148GB cluster PVC. Corrupts mid, microprice, spread, L0-OFI features, the Phase-5 mid-based MTM reward, and the LOB-sim `levels[0]` fill reference — i.e. the agent has been training and trading on phantom inside quotes (~10% wide, 6.4% crossed). Falsifies the old "cluster data is fine" claim.
|
||||
|
||||
---
|
||||
|
||||
## Decision & next action
|
||||
|
||||
**Adopt the MBP-10 L0 decode fix as the next work, ahead of any RL/reward/architecture effort.** Every prior intervention (F4, F2, F5, reward-alignment, 64 commits) operated on a corrupt data substrate; fixing it is the prerequisite for any trustworthy result.
|
||||
|
||||
Plan (next session):
|
||||
1. **Confirm on PVC first** — run the diagnostic dump (crossed %, L1-inside-L0 %, spread p50/p99) on a cluster PVC ES file to verify the corruption is present there too (inferred from shared code, not yet measured).
|
||||
2. **Fix** `dbn_parser.rs`: line 903 `1..max_lvl` → `0..max_lvl`; drop the `update_level(0,...)` at 876-883. Add a unit test asserting no crossed/locked L0 on a known record set. (TDD; ~½ day code.)
|
||||
3. **Regenerate predecoded sidecars** — the cache validates by SOURCE mtime/size (`ml-features/src/predecoded.rs`), so a parser change will NOT auto-invalidate. Delete + regenerate local `test_data` and cluster PVC sidecars (148GB — the heavy step; Argo job).
|
||||
4. **Re-validate data** (spread p50≈1 tick, crossed≈0), then **re-run the edge audit on clean data** (short AND long horizons, with real fills), then re-assess whether RL training has a real substrate.
|
||||
|
||||
---
|
||||
|
||||
## Artifacts & state
|
||||
|
||||
- **Committed (branch `ml-alpha-regime-observer`):** Phase A force-pin (slot 824, `d8447475c`); `scripts/measure_reward_alignment.py` (`e57b07657`).
|
||||
- **Reverted:** the `FOXHUNT_POPART_NORMALIZE` disambiguation scaffold in `integrated.rs` (its finding is in `pearl_shaping_is_dual_purpose_align_vs_explore`). Re-add if testing popart on clean data.
|
||||
- **Specs/plans (uncommitted, this session):** `docs/superpowers/specs/2026-06-05-f4-oracle-pretraining-infrastructure.md` (GATED), `docs/superpowers/specs/2026-06-05-reward-pnl-alignment-restoration.md` (superseded by the data-quality finding), `docs/superpowers/plans/2026-06-05-reward-pnl-alignment-restoration-implementation.md`.
|
||||
- **Memory pearls added:** reward-misalign-blocks-f4 / shaping-dual-purpose / data-edge-audit / edge-exists (retracted) / local-data-degraded / **mbp10-level0-decode-bug-root-cause**.
|
||||
- **Diagnostic data:** `/tmp/diag_q1.csv`, `/tmp/edge*_*.csv`, `/tmp/bt_*.csv`.
|
||||
|
||||
## Methodology note
|
||||
This session is a clean demonstration of `feedback_investigation_first_falsification_methodology`: spec→measure→falsify→STOP-on-surprise, applied through **8 sequential verdicts including 2 self-retractions** (F4 gating; the "edge exists" reversal). It converted a planned 2-week F4 build into the discovery of a ~2-line root-cause bug. Cost: a handful of ~10-min local smokes + parallel research/dump agents. No cluster spend.
|
||||
40
docs/superpowers/notes/2026-06-05-next-session-prompt.md
Normal file
40
docs/superpowers/notes/2026-06-05-next-session-prompt.md
Normal file
@@ -0,0 +1,40 @@
|
||||
# Next session resume prompt
|
||||
|
||||
Copy-paste this into a new Claude Code session to pick up Phase 7c F4 oracle pretrain work:
|
||||
|
||||
---
|
||||
|
||||
Resume foxhunt Phase 7c F4 oracle pretraining work.
|
||||
|
||||
Branch: `ml-alpha-regime-observer` at HEAD `7376b1c67` (6 commits ahead; pushed). F2 (slot 817) + F5 (slot 823) + Option A gate-patches engaged. F2+F5+OptA = SOTA local at eval_pnl=-$415,237 (best-of-session, +$29k vs Phase 5). Cluster sk2z9 bled 3× faster than F2-alone with 6:1 Short:Long directional bias → killed at step 519 → F4 oracle pretrain is the escalation per spec §5.3 outcome 3.
|
||||
|
||||
**Read these memory pearls first** (in this order):
|
||||
1. `project_phase7_state_2026_06_05.md` — full Phase 7 state, branch + slots + verdicts
|
||||
2. `pearl_f2_q_centered_distill_necessary_not_sufficient.md` — why F2 alone failed
|
||||
3. `pearl_f5_state_mask_neutralized_by_downstream_gates.md` — F5 gate-interop pattern
|
||||
4. `pearl_f2_plus_f5_unlock_q_signal.md` — why combined helps; cluster bleed
|
||||
5. `reference_outcome_head_supervised_pattern.md` — `outcome_head.rs` is the F4 Phase 7c.2 precedent
|
||||
6. `feedback_investigation_first_falsification_methodology.md` — methodology that broke 64-commit going-in-circles
|
||||
|
||||
**Read this spec next**:
|
||||
- `docs/superpowers/specs/2026-06-05-f4-oracle-pretraining-infrastructure.md` (733 lines, 6 phases 7c.0-7c.5, ~7-14 days bounded)
|
||||
|
||||
**Next step**: write the F4 implementation plan at `docs/superpowers/plans/2026-06-05-f4-oracle-pretraining-implementation.md` (similar 1500-LOC structure to F2 plan), then dispatch Phase 7c.0 oracle pipeline coder (2-3 days).
|
||||
|
||||
**Top 3 risks per F4 spec §7**:
|
||||
1. Oracle heuristic itself wrong → F4-G1 Pearson < 0.7 → STOP at Phase 7c.0
|
||||
2. Reward function identity drift → MUST use realistic-Rust+CUDA oracle (NOT Python)
|
||||
3. Q drift undoes warm-start → F4-G4/G6 are detection gates
|
||||
|
||||
**Key code references**:
|
||||
- F2 kernel: `crates/ml-alpha/cuda/rl_q_pi_distill_grad.cu` Step 2 hinge (lines ~101-150)
|
||||
- F5 kernel: `crates/ml-alpha/cuda/rl_state_action_mask.cu`
|
||||
- OutcomeHead pattern: `crates/ml-alpha/src/rl/outcome_head.rs`
|
||||
- F2 plan template: `docs/superpowers/plans/2026-06-04-f2-q-centered-distill-implementation.md`
|
||||
|
||||
**Open questions for the plan** (resolve before dispatching coder):
|
||||
- Reward function versioning hash protocol (Phase 5 slots + Phase 7a/7b slots + γ + K → filename embedding)
|
||||
- Held-out subset for F4-G3 (last 10% of fold-1 train OR fold-1 eval discarded)
|
||||
- Encoder LR for pretraining (same as RL fine-tune for MVP simplicity OR separate higher LR)
|
||||
|
||||
Proceed with writing the F4 implementation plan, then begin Phase 7c.0.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,465 @@
|
||||
# Reward–PnL Alignment Restoration — Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Restore reward↔pnl alignment by making the surfer-scaffold weight (slot 753) actually controllable at runtime, prove the credit-assignment co-fix is viable with a cheap diagnostic gate (RA-G4), and only then complete the GAE multi-epoch update — displacing F4 oracle pretraining as the next intervention.
|
||||
|
||||
**Architecture:** Two independent defects. **D1 (alignment):** `rl_surfer_scaffold_controller.cu:101` overwrites slot 753 every step, making its 0.0 "pure-pnl" bootstrap cosmetic — fixed by a force-pin gate. **D2 (credit assignment):** aligned reward is sparse and the advantage kernel is done-gated 1-step TD — fixed (conditionally) by completing the already-scaffolded GAE multi-epoch PPO update. A cheap diagnostic (RA-G4) gates the expensive D2 build.
|
||||
|
||||
**Tech Stack:** Rust 1.85 (ml-alpha crate), CUDA 12.4 (pre-compiled cubins via build.rs, no nvrtc), ISV bus (device-resident f32 control vector), Tier 1.5 local smoke (RTX 3050, b=128), Argo cluster (L40S, b=1024).
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-06-05-reward-pnl-alignment-restoration.md`
|
||||
|
||||
---
|
||||
|
||||
## Phase plan & gates (read first)
|
||||
|
||||
| Phase | What | Gates | STOP-if |
|
||||
|---|---|---|---|
|
||||
| **A** | Force-pin slot 753 (D1) + re-measure harness | RA-G1 (weight==0 when pinned), RA-G2 (determinism), RA-G3 (Pearson≥0.70) | RA-G1/G2 fail |
|
||||
| **B** | GAE-advantage↔pnl viability diagnostic | **RA-G4 (advantage↔pnl Pearson≥0.70)** | RA-G4<0.70 → STOP, do NOT build Phase C |
|
||||
| **C** | Complete 1B-C multi-epoch PPO over rollout buffer (D2) — CONDITIONAL | RA-G5 (no eval regression), RA-G2 | RA-G5 regresses |
|
||||
| **D** | Cluster verify | RA-G6 (cluster eval > sk2z9) | branch per spec §5 |
|
||||
|
||||
**Phase A and B are fully specified below. Phase C and D are structured outlines** — Phase C's exact code depends on the RA-G4 verdict and a rollout-scaffold audit (its first task), so per `feedback_investigation_first_falsification_methodology` it is NOT pre-coded speculatively. Expand Phase C into full TDD tasks only after RA-G4 passes.
|
||||
|
||||
---
|
||||
|
||||
## File structure
|
||||
|
||||
| File | Change | Responsibility |
|
||||
|---|---|---|
|
||||
| `crates/ml-alpha/src/rl/isv_slots.rs` | Add slot 824, bump `RL_SLOTS_END`→825 | Slot constant + invariant test |
|
||||
| `crates/ml-alpha/cuda/rl_surfer_scaffold_controller.cu` | Add `#define` + force-pin early-return | The D1 gate (device-side) |
|
||||
| `crates/ml-alpha/src/trainer/integrated.rs` | Env-gated bootstrap entry + diag field | Wire pin flag; observability |
|
||||
| `crates/ml-alpha/src/trainer/eval_diag_emission.rs` | Bump `EXPECTED_LEAVES` +1 | Keep leaf-count gate honest |
|
||||
| `scripts/measure_reward_alignment.py` | New | Reusable Pearson(reward,pnl) gate harness |
|
||||
| `crates/ml-alpha/src/trainer/rollout_collection.rs` | (Phase B) per-close pnl capture + advantage diag | RA-G4 plumbing |
|
||||
|
||||
---
|
||||
|
||||
## Phase A — Force-pin slot 753 (D1)
|
||||
|
||||
### Task A1: Add ISV slot 824 + bump RL_SLOTS_END
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml-alpha/src/rl/isv_slots.rs:2346` (after slot 823) and `:2376` (`RL_SLOTS_END`)
|
||||
- Test: `crates/ml-alpha/src/rl/isv_slots.rs` (inline `#[cfg(test)]`)
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Add to the `#[cfg(test)] mod tests` block in `isv_slots.rs` (create the block at end of file if none exists):
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn force_pin_slot_allocated_below_end() {
|
||||
// Phase A (2026-06-05) — reward-pnl alignment force-pin.
|
||||
assert_eq!(RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX, 824);
|
||||
assert_eq!(RL_SLOTS_END, 825);
|
||||
assert!(RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX < RL_SLOTS_END);
|
||||
// Must not collide with the prior last slot (F5 mask = 823).
|
||||
assert_ne!(RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX, RL_F5_STATE_MASK_ENABLED_INDEX);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo test -p ml-alpha --lib force_pin_slot_allocated_below_end`
|
||||
Expected: FAIL — `RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX` not found (does not compile).
|
||||
|
||||
- [ ] **Step 3: Add the slot constant and bump the end**
|
||||
|
||||
Insert after line 2346 (`pub const RL_F5_STATE_MASK_ENABLED_INDEX: usize = 823;`):
|
||||
|
||||
```rust
|
||||
/// Phase A reward-pnl alignment (2026-06-05) — force-pin gate for the
|
||||
/// surfer-scaffold weight controller. When `> 0.5`,
|
||||
/// `rl_surfer_scaffold_controller.cu` returns WITHOUT writing slot 753,
|
||||
/// so slot 753 holds its bootstrap value (pure-pnl = 0.0) for the whole
|
||||
/// run instead of being overwritten to w≈0.97 every step.
|
||||
///
|
||||
/// Root cause it fixes: the slot-753 0.0 bootstrap was cosmetic — the
|
||||
/// controller's unconditional write re-enabled all four Phase-5
|
||||
/// non-potential shaping terms, giving Pearson(reward,pnl)=0.28 (gate
|
||||
/// 0.70). See `pearl_reward_misalign_blocks_f4_slot753_override`.
|
||||
///
|
||||
/// Bootstrap 0.0 (OFF — controller writes as before). Set to 1.0 via the
|
||||
/// `FOXHUNT_PIN_SURFER_SCAFFOLD=1` env flag to pin pure-pnl. Clamp
|
||||
/// `{0.0, 1.0}` (binary).
|
||||
pub const RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX: usize = 824;
|
||||
```
|
||||
|
||||
Then update the `RL_SLOTS_END` doc-chain (line ~2375) and value (line 2376):
|
||||
|
||||
```rust
|
||||
/// Post-Phase 7b F5 (state-conditional action mask gate, slot 823): 823.
|
||||
/// Post-Phase A reward-pnl alignment (surfer-scaffold force-pin, slot 824): 824.
|
||||
pub const RL_SLOTS_END: usize = 825;
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo test -p ml-alpha --lib force_pin_slot_allocated_below_end`
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/ml-alpha/src/rl/isv_slots.rs
|
||||
git commit -m "feat(ml-alpha): Phase A — RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX slot 824"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task A2: Add the force-pin gate to the controller kernel
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml-alpha/cuda/rl_surfer_scaffold_controller.cu` (add `#define` near line 57; add early-return near line 65)
|
||||
|
||||
> CUDA kernels have no unit-test harness in this repo (verified via smoke + determinism, Tasks A6/A7). This task is a build-and-inspect step; behavioral verification is RA-G1 in Task A6.
|
||||
|
||||
- [ ] **Step 1: Add the slot `#define`**
|
||||
|
||||
After line 57 (`#define RL_SURFER_WARMUP_TRADES_INDEX 756`), add:
|
||||
|
||||
```cuda
|
||||
#define RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX 824
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add the early-return gate**
|
||||
|
||||
In `rl_surfer_scaffold_controller`, immediately after the thread guard at line 65 (`if (threadIdx.x != 0 || blockIdx.x != 0) return;`), insert:
|
||||
|
||||
```cuda
|
||||
// Phase A force-pin (2026-06-05): when set, leave slot 753 at its
|
||||
// bootstrap value (pure-pnl) instead of overwriting it. Fixes the
|
||||
// cosmetic-bootstrap bug (pearl_reward_misalign_blocks_f4_slot753_override).
|
||||
if (isv[RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX] > 0.5f) {
|
||||
return; // do NOT write isv[RL_SURFER_SCAFFOLD_WEIGHT_INDEX]
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Build to verify the cubin compiles**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo build -p ml-alpha --profile=dev-release 2>&1 | tail -5`
|
||||
Expected: build succeeds (build.rs recompiles `rl_surfer_scaffold_controller.cubin` via nvcc; no errors).
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/ml-alpha/cuda/rl_surfer_scaffold_controller.cu
|
||||
git commit -m "feat(ml-alpha): Phase A — force-pin gate in surfer-scaffold controller"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task A3: Env-gated bootstrap entry
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml-alpha/src/trainer/integrated.rs:4416` (bootstrap table, after the F5 entry)
|
||||
|
||||
- [ ] **Step 1: Read the env flag and add the bootstrap entry**
|
||||
|
||||
Immediately before the closing `];` of the `isv_constants` array at line 4417, after the F5 entry (line 4416), insert:
|
||||
|
||||
```rust
|
||||
// Phase A reward-pnl alignment (2026-06-05) — surfer-scaffold
|
||||
// force-pin. Env-gated per the FOXHUNT_USE_ROLLOUT precedent
|
||||
// (commit 779c03b9d): unset/0 → 0.0 (controller writes slot
|
||||
// 753 as before, behaviour unchanged); =1 → 1.0 (controller
|
||||
// skips its write, slot 753 holds its 0.0 pure-pnl bootstrap).
|
||||
(crate::rl::isv_slots::RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX, {
|
||||
let pin: bool = std::env::var("FOXHUNT_PIN_SURFER_SCAFFOLD")
|
||||
.ok()
|
||||
.and_then(|v| v.parse::<u8>().ok())
|
||||
.map(|n| n > 0)
|
||||
.unwrap_or(false);
|
||||
if pin { 1.0_f32 } else { 0.0_f32 }
|
||||
}),
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Build to verify it compiles**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo build -p ml-alpha --profile=dev-release 2>&1 | tail -5`
|
||||
Expected: build succeeds.
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/ml-alpha/src/trainer/integrated.rs
|
||||
git commit -m "feat(ml-alpha): Phase A — FOXHUNT_PIN_SURFER_SCAFFOLD env flag → slot 824 bootstrap"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task A4: Diag observability field + leaf-count bump
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml-alpha/src/trainer/integrated.rs:12576` (after `surfer_scaffold_weight`)
|
||||
- Modify: `crates/ml-alpha/src/trainer/eval_diag_emission.rs` (`EXPECTED_LEAVES`)
|
||||
|
||||
- [ ] **Step 1: Add the diag field**
|
||||
|
||||
Immediately after the `surfer_scaffold_weight` entry (line 12575-12576), insert:
|
||||
|
||||
```rust
|
||||
"surfer_scaffold_force_pin":
|
||||
isv[crate::rl::isv_slots::RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX],
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Find the current EXPECTED_LEAVES value**
|
||||
|
||||
Run: `grep -n "EXPECTED_LEAVES" crates/ml-alpha/src/trainer/eval_diag_emission.rs`
|
||||
Expected: one `const EXPECTED_LEAVES: usize = N;` line. Note `N`.
|
||||
|
||||
- [ ] **Step 3: Bump EXPECTED_LEAVES by 1**
|
||||
|
||||
Edit that constant from `N` to `N + 1` (one new leaf added under `rewards`). Update any adjacent comment that records the count.
|
||||
|
||||
- [ ] **Step 4: Verify the leaf-count assertion passes**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo test -p ml-alpha --lib eval_diag 2>&1 | tail -20`
|
||||
Expected: the diag leaf-count test passes (if it fails with `actual N+1 expected N`, the bump in Step 3 was missed).
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/ml-alpha/src/trainer/integrated.rs crates/ml-alpha/src/trainer/eval_diag_emission.rs
|
||||
git commit -m "feat(ml-alpha): Phase A — surfer_scaffold_force_pin diag field (+1 leaf)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task A5: Reusable alignment-measurement harness
|
||||
|
||||
**Files:**
|
||||
- Create: `scripts/measure_reward_alignment.py`
|
||||
|
||||
- [ ] **Step 1: Write the harness**
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
"""Measure reward<->pnl alignment from a diag.jsonl.
|
||||
|
||||
RA-G3 gate: Pearson(rewards.sum, trading.realized_pnl_usd_delta) >= 0.70.
|
||||
Usage: python3 scripts/measure_reward_alignment.py <run_dir_or_diag.jsonl> [--json]
|
||||
Exit 0 if Pearson >= 0.70, 1 if below, 2 on bad input.
|
||||
"""
|
||||
import json, os, sys
|
||||
|
||||
def diag_path(arg):
|
||||
if os.path.isdir(arg):
|
||||
return os.path.join(arg, "diag.jsonl")
|
||||
return arg
|
||||
|
||||
def pearson(x, y):
|
||||
n = len(x)
|
||||
if n < 3:
|
||||
return None
|
||||
mx, my = sum(x)/n, sum(y)/n
|
||||
sxy = sum((a-mx)*(b-my) for a, b in zip(x, y))
|
||||
sxx = sum((a-mx)**2 for a in x)
|
||||
syy = sum((b-my)**2 for b in y)
|
||||
if sxx == 0 or syy == 0:
|
||||
return None
|
||||
return sxy/(sxx*syy)**0.5
|
||||
|
||||
def main():
|
||||
args = [a for a in sys.argv[1:] if not a.startswith("--")]
|
||||
want_json = "--json" in sys.argv
|
||||
if not args:
|
||||
print("usage: measure_reward_alignment.py <run_dir_or_diag.jsonl>", file=sys.stderr)
|
||||
return 2
|
||||
path = diag_path(args[0])
|
||||
if not os.path.exists(path):
|
||||
print("no diag at %s" % path, file=sys.stderr)
|
||||
return 2
|
||||
rew, pnl = [], []
|
||||
for line in open(path):
|
||||
try:
|
||||
d = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
r = d.get("rewards", {}).get("sum")
|
||||
p = d.get("trading", {}).get("realized_pnl_usd_delta")
|
||||
if r is None or p is None:
|
||||
continue
|
||||
rew.append(float(r)); pnl.append(float(p))
|
||||
P = pearson(rew, pnl)
|
||||
both = [(a, b) for a, b in zip(rew, pnl) if a != 0 and b != 0]
|
||||
sign = (sum(1 for a, b in both if (a > 0) == (b > 0)) / len(both)) if both else None
|
||||
gate = 0.70
|
||||
ok = P is not None and P >= gate
|
||||
result = {"pearson": P, "sign_agreement": sign, "n_rows": len(rew),
|
||||
"gate": gate, "pass": ok}
|
||||
if want_json:
|
||||
print(json.dumps(result))
|
||||
else:
|
||||
print("rows=%d Pearson=%s sign_agree=%s gate>=%.2f -> %s" % (
|
||||
len(rew),
|
||||
"n/a" if P is None else "%.4f" % P,
|
||||
"n/a" if sign is None else "%.3f" % sign,
|
||||
gate, "PASS" if ok else "FAIL"))
|
||||
return 0 if ok else 1
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Smoke the harness on the existing baseline diag (expect FAIL = 0.28)**
|
||||
|
||||
Run: `python3 scripts/measure_reward_alignment.py /tmp/foxhunt-mid-smoke --`
|
||||
Expected: `rows=2000 Pearson=0.2775 sign_agree=0.655 gate>=0.70 -> FAIL` (exit 1). Confirms the harness reproduces the baseline measurement.
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add scripts/measure_reward_alignment.py
|
||||
git commit -m "feat(scripts): reward-pnl alignment measurement harness (RA-G3 gate)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task A6: Tier 1.5 smoke with pin engaged → RA-G1 + RA-G3
|
||||
|
||||
**Files:** none (behavioral verification)
|
||||
|
||||
- [ ] **Step 1: Run the pinned mid-smoke**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
FOXHUNT_PIN_SURFER_SCAFFOLD=1 ./scripts/local-mid-smoke.sh --seed 42 --out /tmp/foxhunt-pin-smoke
|
||||
```
|
||||
Expected: completes ~10 min, writes `/tmp/foxhunt-pin-smoke/diag.jsonl` + `eval_summary.json`, no NaN.
|
||||
|
||||
- [ ] **Step 2: RA-G1 — verify slot 753 held at 0 the whole run**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
python3 -c "import json; vs=[json.loads(l)['rewards']['surfer_scaffold_weight'] for l in open('/tmp/foxhunt-pin-smoke/diag.jsonl')]; print('min=%.4f max=%.4f pin_field=%.1f'%(min(vs),max(vs), json.loads(open('/tmp/foxhunt-pin-smoke/diag.jsonl').readline())['rewards']['surfer_scaffold_force_pin']))"
|
||||
```
|
||||
Expected: `min=0.0000 max=0.0000 pin_field=1.0`. **RA-G1 PASS** iff weight is 0.0 across all rows and the pin field reads 1.0.
|
||||
**STOP-if-fail:** weight ≠ 0 means the kernel gate (Task A2) is not wired to the controller write — re-verify the early-return placement is BEFORE the `isv[753]=w` line.
|
||||
|
||||
- [ ] **Step 3: RA-G3 — measure restored alignment**
|
||||
|
||||
Run: `python3 scripts/measure_reward_alignment.py /tmp/foxhunt-pin-smoke`
|
||||
Expected: `-> PASS` with Pearson ≥ 0.70 (pure-pnl reward ≈ pnl-delta by construction; this confirms the shaping was the misalignment source).
|
||||
**Note (spec §7 risk 1):** RA-G3 is near-tautological — it confirms the pin removed the shaping, NOT that the agent trades better. The real viability test is RA-G4 (Phase B). If RA-G3 is still < 0.70 after pinning, the residual is magnitude clamp/popart (`pearl_popart_blind_to_session_signals`) → raise a clamp sub-spec before proceeding.
|
||||
|
||||
---
|
||||
|
||||
### Task A7: Determinism → RA-G2
|
||||
|
||||
**Files:** none
|
||||
|
||||
- [ ] **Step 1: Run determinism check with pin engaged**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
FOXHUNT_PIN_SURFER_SCAFFOLD=1 ./scripts/determinism-check.sh --quick
|
||||
```
|
||||
Expected: exit 0 — same-seed runs bit-equal (`diag.jsonl` byte-equal modulo `elapsed_s`, `eval_summary.json` byte-equal).
|
||||
**STOP-if-fail:** the pin/diag introduced nondeterminism — bisect via `--debug-dump-rl`. (Adding a constant slot + early-return should be deterministic; an ISV-length-dependent checksum divergence is the most likely culprit — investigate the ISV checksum path.)
|
||||
|
||||
> Note: this gate is same-seed bit-equality, NOT bit-equality-vs-HEAD. The new slot 824 + diag leaf intentionally change the schema vs HEAD; that is expected and not a regression.
|
||||
|
||||
---
|
||||
|
||||
### Task A8: Phase A wrap-up commit + verdict record
|
||||
|
||||
- [ ] **Step 1: Record the Phase A verdict**
|
||||
|
||||
Append a one-line result to the spec's decision log (RA-G1/G2/G3 pass/fail + measured Pearson). If all pass, Phase A is green and Phase B is unlocked.
|
||||
|
||||
```bash
|
||||
git add docs/superpowers/specs/2026-06-05-reward-pnl-alignment-restoration.md
|
||||
git commit -m "docs: Phase A verdict — RA-G1/G2/G3 results"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase B — Credit-assignment viability diagnostic (RA-G4)
|
||||
|
||||
> **Goal:** measure `Pearson(GAE_advantage_at_open, realized_trade_pnl)` cheaply (no training rearchitecture) to decide whether the expensive Phase C build is justified. The rollout buffer (`rollout_buffer.rs`) already computes `advantages_d`/`returns_d` via `compute_gae`; collection (`rollout_collection.rs`) fills it as a side-effect under `FOXHUNT_USE_ROLLOUT=1`. What it does NOT currently store is per-step realized USD pnl — Task B1 audits that gap.
|
||||
|
||||
### Task B1: Audit the rollout-collection → pnl mapping (investigation, no code)
|
||||
|
||||
**Deliverable:** a written finding (append to the spec) answering:
|
||||
- [ ] Where does `rollout_collection.rs` capture per-step data, and is `trading.realized_pnl_usd_delta` (or the device buffer behind it) reachable at the same `(b, t)` granularity as `advantages_d`?
|
||||
- [ ] At which `t` does a trade "close" (done flag) and where is its realized pnl available on-device?
|
||||
- [ ] Can the open-step `advantages_d[b, t_open]` be paired with the realized pnl of the position that opened at `t_open`? (trade-tracking through the buffer)
|
||||
|
||||
**STOP-if:** the pnl signal is not reachable at buffer granularity without a >200 LOC collection refactor → fall back to the **coarser RA-G4 proxy** in Task B2-alt (per-step diag correlation) and note the precision loss.
|
||||
|
||||
### Task B2: Emit GAE-advantage + realized-pnl pairs to diag
|
||||
|
||||
Based on B1, add a deterministic diag emission (per `pearl_fleet_fraction_not_aggregate` — emit fleet fractions, not just aggregates) of:
|
||||
- mean/median `advantages_d` at close (done) steps,
|
||||
- co-located `realized_pnl_usd_delta`,
|
||||
- per-batch fraction with advantage-sign == pnl-sign.
|
||||
|
||||
Mirror the existing per-batch reduction pattern (the F2 diag counters, `RL_F2_DIAG_*`). Wire under `FOXHUNT_USE_ROLLOUT=1` only (bit-equal when off).
|
||||
|
||||
### Task B2-alt (fallback): coarse per-step proxy
|
||||
|
||||
If B1 STOPs: compute `Pearson(returns_d-mean-at-close, realized_pnl_usd_delta)` host-side from diag rows (cross-granularity, weaker), and label the RA-G4 result "coarse proxy — precision-limited."
|
||||
|
||||
### Task B3: Run smoke and evaluate RA-G4
|
||||
|
||||
- [ ] Run: `FOXHUNT_PIN_SURFER_SCAFFOLD=1 FOXHUNT_USE_ROLLOUT=1 ./scripts/local-mid-smoke.sh --seed 42 --out /tmp/foxhunt-pin-gae-smoke`
|
||||
- [ ] Compute `Pearson(GAE_advantage_at_open, realized_trade_pnl)` from the emitted diag.
|
||||
- [ ] **RA-G4 gate: ≥ 0.70.**
|
||||
|
||||
**STOP-if RA-G4 < 0.70:** Per spec §5.4, do NOT build Phase C and do NOT revisit F4. A failure means multi-step credit assignment on aligned reward still does not recover pnl-aligned advantages → the fault is deeper (reward shape itself or genuine regime shift). Escalate to a reward-shape / regime investigation. Record the verdict in the spec decision log.
|
||||
|
||||
### Task B4: Determinism + commit
|
||||
|
||||
- [ ] `FOXHUNT_PIN_SURFER_SCAFFOLD=1 FOXHUNT_USE_ROLLOUT=1 ./scripts/determinism-check.sh --quick` → exit 0 (RA-G2).
|
||||
- [ ] Commit the diagnostic + spec verdict.
|
||||
|
||||
---
|
||||
|
||||
## Phase C — Complete 1B-C multi-epoch PPO over rollout buffer (D2) — CONDITIONAL on RA-G4
|
||||
|
||||
> **Do NOT expand into TDD tasks until RA-G4 passes.** Phase C is the substantive (~3-6 day) build that replaces the per-step done-gated TD update with a multi-epoch PPO update over the GAE buffer. Its exact code depends on the rollout-scaffold audit (Task C1) and must honour `pearl_foxhunt_trainer_is_genuinely_single_step` (the per-step training split is the load-bearing change).
|
||||
|
||||
Structured task outline:
|
||||
- **C1 (audit):** map the current per-step PPO update path in `integrated.rs` (the `ss_pi_grad_logits_d` distill write site ~5607 + Adam steps) and the `rollout_collection.rs` collection loop. Identify exactly where per-step training is replaced by buffer-replay multi-epoch update. Produce the Phase C TDD task list from this audit.
|
||||
- **C2:** implement the multi-epoch minibatch loop over `RolloutBuffer` (re-evaluate heads on stored `h_t_d`, PPO ratio from `log_pi_old_d`, advantages from `advantages_d`), gated behind `FOXHUNT_USE_ROLLOUT=1`.
|
||||
- **C3:** determinism (RA-G2) + bit-equality-when-off.
|
||||
- **C4:** mid-smoke with pin + rollout → **RA-G5 (eval pnl ≥ −$415,237 F2+F5 SOTA, not worse)**. STOP-if-regresses (recall Phase 1A regressed here).
|
||||
|
||||
**Gates:** RA-G5, RA-G2. **STOP-if:** RA-G5 regresses → diagnose with fleet-fraction diag before cluster.
|
||||
|
||||
---
|
||||
|
||||
## Phase D — Cluster verify (RA-G6) — CONDITIONAL on Phase C
|
||||
|
||||
Structured checklist:
|
||||
- **D1:** confirm local HEAD == origin (push), apply Argo template (`feedback_argo_template_must_apply`).
|
||||
- **D2:** submit Tier 2 — `./scripts/argo-alpha-rl.sh --n-steps 20000 --n-eval-steps 5000 --n-folds 3 --fold-idx 1 --n-backtests 1024 --band-enabled 1` with `FOXHUNT_PIN_SURFER_SCAFFOLD=1 FOXHUNT_USE_ROLLOUT=1` threaded into the workflow env (verify the Argo template forwards these env vars; add them if not).
|
||||
- **D3:** wait ~70 min; analyze `eval_summary.json`.
|
||||
- **Gate RA-G6:** `total_pnl_usd` > F2+F5 sk2z9 cluster baseline.
|
||||
- **Branch (spec §5):** RA-G6 pass → reward alignment was the lever; commit. RA-G6 fail with RA-G4 pass → encoder/generalization branch legitimately isolated (NOT another reward reshape; consider IDAAC decoupling / overfit-rejection ensemble, or — only now — a properly-done warm-start).
|
||||
|
||||
---
|
||||
|
||||
## Self-review
|
||||
|
||||
**Spec coverage:**
|
||||
- D1 force-pin → Tasks A1-A3 ✓ | RA-G1 → A6 ✓ | RA-G2 → A7, B4 ✓ | RA-G3 → A5, A6 ✓
|
||||
- §3.4 re-measure harness → A5 ✓ | §3.2 RA-G4 diagnostic → Phase B ✓ | RA-G4 → B3 ✓
|
||||
- §3.3 1B-C (D2) → Phase C ✓ | RA-G5 → C4 ✓ | RA-G6 → D3 ✓
|
||||
- §7 risk 1 (RA-G3 tautology) called out in A6 Step 3 ✓ | §7 risk 3 (magnitude clamp) called out in A6 Step 3 ✓ | §7 risk 4 (1B-C cost) → Phase C kept conditional + audit-first ✓
|
||||
- Phase 1A landmine (§1.3) → RA-G4 gates the D2 spend; RA-G5 watches for the regression ✓
|
||||
|
||||
**Placeholder scan:** Phase A and B-task-B1/B3/B4 contain complete code/commands. Phase B Tasks B2/B2-alt and all of Phase C/D are *intentionally* structured outlines (not placeholders) because their exact code is gated behind RA-G4 and a scaffold audit — pre-coding them would violate the investigation-first methodology and likely be wrong. Each carries its gate and STOP rule.
|
||||
|
||||
**Type/name consistency:** `RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX` (824), `FOXHUNT_PIN_SURFER_SCAFFOLD` env flag, `surfer_scaffold_force_pin` diag key — used consistently across A1/A2/A3/A4/A6. `RL_SLOTS_END=825` consistent A1. Harness path `scripts/measure_reward_alignment.py` consistent A5/A6.
|
||||
|
||||
---
|
||||
|
||||
## Execution note
|
||||
|
||||
Phase A is the fully-coded, low-risk unit (force-pin + harness + 3 gates, ~0.5-1 day) and is independently valuable: even if Phase B STOPs, Phase A leaves slot 753 actually controllable and a reusable alignment gate in CI. Recommend executing Phase A end-to-end, reviewing RA-G1/G2/G3, then deciding Phase B with the RA-G4 verdict in hand.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,733 @@
|
||||
# F4 — Oracle Pretraining Infrastructure Spec
|
||||
|
||||
> **For agentic workers:** This spec defines the INFRASTRUCTURE for F4 oracle pretraining; it does NOT itself implement F4. Implementation follows in a separate plan after this spec is approved.
|
||||
|
||||
**Status:** Draft 1 (2026-06-05)
|
||||
**Predecessor:** `docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md` (F4 §3.4)
|
||||
**Phase 7a/7b empirical context:** F2 + F5 + OptA committed (HEAD `7376b1c67` on `ml-alpha-regime-observer`); cluster bleed −$6.4M @ step 519 → projected −$247M @ step 20k (sk2z9). F2 alone hinge_zero_rate saturated to 1.0 (mq2pc descendant); F5 alone surfaced 6:1 Short:Long bias; F2+F5 produced trade rate but 3× faster bleed than F2-alone.
|
||||
|
||||
**Branch target:** TBD (off post-Phase-7b commit `7376b1c67` once this spec is approved).
|
||||
|
||||
**Estimated wall-clock to evaluable F4-MVP cluster verdict:** ~7-10 working days (see §6). Honest range: 7 days if every phase passes first attempt; 14+ days if Phase 7c.0 oracle ground-truth fails or Phase 7c.5 cluster regresses.
|
||||
|
||||
**Scope discipline:** This is an INFRASTRUCTURE spec. It defines what must be BUILT to make F4 evaluable. It does NOT specify F4 hyperparameters, prescribe LOC counts at the kernel level, or replace the implementation plan that must follow. F4-Full (multi-epoch, full-action distribution, encoder-freeze variants) is explicitly OUT OF SCOPE; only F4-MVP is in scope.
|
||||
|
||||
**Linked specs (load-bearing context):**
|
||||
|
||||
- `docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md` — F4 §3.4 (predecessor), F2 §3.2 (Phase 7a, committed), F5 §3.5 (Phase 7b, committed)
|
||||
- `docs/superpowers/specs/2026-06-01-reward-policy-alignment-investigation.md` — Pearson-misalignment foundation, established the reward function shape that F4 oracle must match
|
||||
- `docs/superpowers/specs/2026-06-02-fast-dev-cycle.md` — Tier 1 / 1.5 / 2 funnel that all phase gates use
|
||||
- `docs/superpowers/specs/2026-06-02-determinism-foundation.md` — bit-equality guarantee F4 must preserve
|
||||
|
||||
**Linked pearls (load-bearing):**
|
||||
|
||||
- `pearl_foxhunt_pi_trained_by_q_distillation_not_ppo` — π's only gradient source is Q-distill; F4's pretrained Q is what π distills from
|
||||
- `pearl_foxhunt_trainer_is_genuinely_single_step` — `[B]`-only buffer shape; F4-MVP intentionally preserves this constraint
|
||||
- `pearl_advantage_kernel_is_done_gated_td_not_gae` — V regression operates at T_eff=1; F4 oracle returns are γ-discounted not bootstrap-1-step
|
||||
- `pearl_diag_pnl_fields_are_shaped_reward_not_usd` — load-bearing reward-function-identity caveat; F4 oracle MUST consume the same shaped reward function the trainer consumes (NOT raw USD pnl)
|
||||
- `pearl_phase5_term_4_is_almost_potential` — Ng-Harada-Russell verdict on Phase 5 shaping terms; F4 oracle inherits the same non-potential terms (entry cost, MTM, short-hold penalty)
|
||||
- `pearl_determinism_achieved` — same-seed bit-equality, F4 pretrain phase must preserve
|
||||
- `pearl_fleet_fraction_not_aggregate` — at b=1024+ aggregate diag hides per-batch reality; F4 oracle quality gates must report fleet fractions
|
||||
- `pearl_reward_signal_anti_aligned_with_pnl` — the load-bearing motivation for any oracle-style supervised warm-start
|
||||
- `pearl_surfer_baseline_was_train_only_never_eval` — F4 gates must measure eval pnl, not train; train→eval transfer is itself a gate
|
||||
|
||||
---
|
||||
|
||||
## §1. Why F4, why now
|
||||
|
||||
### §1.1 The Phase 7 empirical trajectory
|
||||
|
||||
Three cluster runs after Phase 5 substrate, each with a distinct mechanism activation profile:
|
||||
|
||||
| Cluster | Phase | What was enabled | Mechanism observed | Eval pnl (cluster) |
|
||||
|---|---|---|---|---|
|
||||
| h6btt | Phase 5 single-head | (baseline; pre-Phase-7) | Hold-collapse to 98%; no opens | bleed ≈ $4.18/batch/step |
|
||||
| mq2pc | Phase 5 + multi-head | (baseline) | TrailLoosen-collapse to 90%; oscillating attractors | bleed ≈ $19/batch/step |
|
||||
| sk2z9 | Phase 7a F2 + Phase 7b F5 + OptA Hold-suppression | F2 hinged distill target + F5 flat-state mask + Gate-aware Hold suppression | F2 hinge active (hinge_zero_rate 1.0→0.91 across training; advantage_max=1.86); F5 forced opens (6:1 Short:Long); trade rate 7× higher than mq2pc | bleed −$6.4M @ step 519, linear projection −$247M @ step 20k (3× faster than F2-alone) |
|
||||
|
||||
### §1.2 What this proves about Phase 7's foundation
|
||||
|
||||
F2 + F5 mechanism gates BOTH pass (verifiable from cluster diag: hinge_zero_rate dropped, advantage_max > 0, flat-state mask correctness 100%, action distribution from flat ∈ {Long, Short}). The Phase 7 spec §5.3 attribution table maps this row exactly:
|
||||
|
||||
```
|
||||
Phase 7b: F2 mech PASS | F2-G5 surfer (n/a, F5 active) | F5 mech PASS | F5-G8 FAIL
|
||||
→ "F5 has no marginal benefit OR introduces new pathology; revert F5, commit F2-alone,
|
||||
escalate residual to encoder / alignment work."
|
||||
```
|
||||
|
||||
But F2-alone also failed (sk2z9 sibling run with F5=0 showed hinge_zero_rate 1.0 saturation — Q never discovers Open advantage from cold start; random exploration cannot produce enough above-baseline samples to gradient-train the Q-head out of the inaction attractor). The empirical reading is:
|
||||
|
||||
- **F5 alone (forced opens)** loses money because random forced opens cannot beat commission drag in the no-edge regime fraction (predicted by Phase 7 spec §3.5 expected failure mode #1).
|
||||
- **F2 alone (gradient redirection)** zeroes everything because cold-start Q has no above-baseline samples; hinge_zero_rate saturates at 1.0 (predicted by Phase 7 spec §3.2 expected failure mode #1 "mass-zero distill target").
|
||||
- **F2 + F5 (gradient + forced data)** activates F2's hinge but feeds it the noise distribution of random forced opens; F2 learns "no Open action has positive advantage in expectation" because the data shows it, faster than the encoder can extract real edge signal.
|
||||
|
||||
Per spec §5.3 attribution outcome 3 / §5.4 escalation path:
|
||||
|
||||
> "If F2 + F5 both pass mechanism gates but eval pnl stays at baseline OR worsens: the inaction attractor was a *symptom* of a deeper problem. Neither is solvable by another Bellman-target reshape… F4 (oracle pretraining) becomes the right next dispatch if the encoder hypothesis dominates."
|
||||
|
||||
The encoder hypothesis is now favored: the agent's Q-head cannot bootstrap discriminating-action-quality estimates from cold-start random-policy-rollout-collected data because the signal-to-noise ratio is too low. The cluster data shows the agent CAN find some directional signal (F5-only directional bias 6:1 is non-random; F2's advantage_max=1.86 is non-zero) — but cannot find the *Q-quality* of specific actions in specific states fast enough to break the inaction attractor.
|
||||
|
||||
### §1.3 The disciplined response per `feedback_going_in_circles_pattern`
|
||||
|
||||
The previous spec's §5.3 attribution row identifies F4 as the next dispatch when F2+F5 mechanism gates pass and outcome fails. That row also requires that no further Bellman-target reshape (F1, F3, more F5 variants) be attempted. F4 attacks a structurally different axis — not the gradient, not the choice-set, but the *initial conditions* of Q. F4 either fixes the problem (cluster eval improves) or refutes the "encoder cannot bootstrap" hypothesis (cluster eval still fails) — and that refutation would itself be progress, since it would isolate the problem to genuine reward-pnl misalignment or train→eval distribution shift.
|
||||
|
||||
F4 is therefore the textbook next step under the spec's own escalation logic. This spec defines the infrastructure to make F4 evaluable.
|
||||
|
||||
---
|
||||
|
||||
## §2. F4 mechanism
|
||||
|
||||
### §2.1 Restated definition
|
||||
|
||||
Per the predecessor spec §3.4:
|
||||
|
||||
> "For each historical state `s_t` in the training data, compute an oracle Q-value `Q_oracle(s_t, a)` using perfect-foresight knowledge of future outcomes. Supervised-train Q to match `Q_oracle` for one or more epochs, THEN RL fine-tune. The agent inherits a Q function that *already knows* Hold is not always optimal because the supervised loss reflects realised returns from each action with perfect foresight."
|
||||
|
||||
In formal terms:
|
||||
|
||||
```
|
||||
Q_oracle(s_t, a) = E_{trajectory | a_t = a, π* thereafter} [ Σ_{k=0..∞} γ^k · r_shaped(s_{t+k}, a_{t+k}) ]
|
||||
```
|
||||
|
||||
where:
|
||||
- `r_shaped` is the trainer's reward function (Phase 5 MTM + Phase 5 trail-max ceiling + base pnl + Phase 7a hinge if applicable). Per `pearl_diag_pnl_fields_are_shaped_reward_not_usd`, this is NOT raw USD pnl.
|
||||
- `π*` is the per-state-after-t optimal policy. "Optimal" is defined by the same `r_shaped` function (the F4-MVP simplification is recursive — oracle's notion of optimal is bootstrapped from greedy-on-oracle-Q one step lookahead; see §2.3 below).
|
||||
- `γ = 0.975` (foxhunt's current C51 / IQN discount factor).
|
||||
|
||||
### §2.2 Two scope tiers
|
||||
|
||||
#### F4-MVP (this spec's scope)
|
||||
|
||||
- Oracle Q for the **argmax action only**. Specifically: for each state `s_t`, compute one `(s_t, a_argmax, Q_argmax)` triple where `a_argmax` is the highest-realized-return action under a chosen oracle policy.
|
||||
- Supervised MSE regression for **ONE epoch over the training fold** BEFORE the RL loop begins.
|
||||
- Q head + encoder both training during the supervised phase (no freeze).
|
||||
- Single fold (fold 1) only.
|
||||
- Pretrain writes to the SAME Q head weights that the Bellman target writes to during RL fine-tune.
|
||||
|
||||
#### F4-Full (out of scope, eventual)
|
||||
|
||||
- Oracle Q for all 11 actions per state.
|
||||
- Multi-epoch supervised regression (curriculum, annealing).
|
||||
- Per-fold oracle generation across walk-forward folds.
|
||||
- Encoder-only pretrain phase, Q-head-only pretrain phase, joint variants.
|
||||
- Distillation regularization to prevent catastrophic Q drift during RL fine-tune.
|
||||
- Online oracle update during RL (Dyna-style).
|
||||
|
||||
### §2.3 Mathematical relationship to the existing Bellman target
|
||||
|
||||
The trainer's existing Bellman update is:
|
||||
|
||||
```
|
||||
Q_target(s, a) = r(s, a) + γ · max_{a'} Q_target_net(s', a')
|
||||
L_q(s, a) = (Q_θ(s, a) − Q_target(s, a))²
|
||||
```
|
||||
|
||||
F4-MVP inserts a supervised-regression phase BEFORE the RL phase:
|
||||
|
||||
```
|
||||
Phase F4-pretrain (epochs=1):
|
||||
for each (s_t, a_argmax, Q_argmax) in oracle dataset:
|
||||
Q_θ(s_t, a_argmax) ← trained via MSE against Q_argmax
|
||||
# All other actions Q_θ(s_t, a≠argmax) receive no supervised loss (F4-MVP).
|
||||
# Encoder gradient flows through Q head as in standard supervised training.
|
||||
|
||||
Phase RL fine-tune (existing trainer):
|
||||
Q_θ is now initialized to "the oracle-argmax-q-shape Q"
|
||||
Standard Bellman fine-tune proceeds; Q_θ learns the off-argmax actions via the Bellman target;
|
||||
Q_θ for argmax actions starts near-correct rather than random.
|
||||
```
|
||||
|
||||
The relationship to the inaction-bias problem (Phase 7 spec §2.1 equation c):
|
||||
|
||||
```
|
||||
Pre-F4: Q_init(s, a) ≈ random → Q_θ(s, no-op) > Q_θ(s, open) by entry_cost in expectation
|
||||
→ F2 hinge zeroes Open mass → no learning signal → collapse
|
||||
|
||||
Post-F4-MVP: Q_init(s, a_argmax) ≈ Q_oracle(s, a_argmax)
|
||||
→ for edge-bearing states where a_argmax = Open, Q_θ(s, Open) > Q_θ(s, Hold) at init
|
||||
→ F2 hinge passes for Open at edge-bearing states → distill mass on Open → learning proceeds
|
||||
→ for no-edge states where a_argmax = Hold, Q_θ(s, Open) < Q_θ(s, Hold) at init
|
||||
→ F2 hinge zeroes Open mass → distill mass on Hold → surfer-style patience preserved
|
||||
```
|
||||
|
||||
F4-MVP + F2 + (optionally) F5 is the conjecture this spec puts under test. Empirical falsification of that conjecture is encoded in the F4-G7/G8/G9 cluster outcome gates in §5.
|
||||
|
||||
### §2.4 The "optimally thereafter" simplification (load-bearing)
|
||||
|
||||
A truly optimal oracle would require knowing `Q*_oracle` for every state — which is the very thing F4 is trying to bootstrap. F4-MVP uses a *greedy-1-step-lookahead-on-realized-return* approximation:
|
||||
|
||||
```
|
||||
For each (s_t):
|
||||
For each action a ∈ A:
|
||||
# Simulate the action through the trainer's LobSim for K steps (γ-discounted).
|
||||
# K is chosen large enough that γ^K is negligible (e.g., γ=0.975 → K=400 → γ^K ≈ 5e-5).
|
||||
# The first action is `a`; subsequent actions follow a heuristic policy:
|
||||
# - if in-position-and-pnl > 2·entry_cost: close
|
||||
# - if in-position-and-trail-breached: close
|
||||
# - else: Hold
|
||||
# This is the "frictionless trail-stop heuristic policy" π_oracle.
|
||||
rollout_K = simulate_lobsim(s_t, a, π_oracle, K_steps=400)
|
||||
Q_oracle(s_t, a) = Σ_k γ^k · r_shaped(rollout_K[k])
|
||||
a_argmax = argmax_a Q_oracle(s_t, a)
|
||||
```
|
||||
|
||||
This is a *heuristic* oracle (not a fixed-point oracle). The justification is that under any reasonable trading regime:
|
||||
|
||||
- The heuristic captures the dominant pnl driver (taking the trade and exiting on edge depletion).
|
||||
- Heuristic-optimal Q values bound the random-policy Q values from above by a large margin.
|
||||
- F4-MVP's job is to give Q a "good direction to start," not to compute the unreachable theoretical optimum. The RL fine-tune phase refines around the bootstrapped baseline.
|
||||
|
||||
Two-step honest framing for the implementation plan:
|
||||
|
||||
1. The oracle is NOT optimal in the game-theoretic sense; it is a *strong heuristic baseline* under perfect foresight.
|
||||
2. F4-G1 (oracle-ground-truth gate, §5) tests whether the heuristic correlates with realized pnl on held-out states. If F4-G1 fails, the heuristic itself is broken (a tighter heuristic policy is required) — this is BEFORE we go anywhere near the trainer.
|
||||
|
||||
### §2.5 Citation chain
|
||||
|
||||
Oracle pretraining for offline-to-online RL is a well-established direction:
|
||||
|
||||
- Levine et al. 2020 ("Offline RL: Tutorial, Review, and Perspectives," arXiv:2005.01643) — survey establishing the framework.
|
||||
- Kumar et al. 2020 ("Conservative Q-Learning," arXiv:2006.04779, NeurIPS) — CQL pessimism for offline-to-online warm-start.
|
||||
- Nair et al. 2020 ("AWAC: Accelerating Online RL with Offline Datasets," arXiv:2006.09359) — advantage-weighted regression bridging.
|
||||
- Kostrikov et al. 2021 ("Offline RL with Implicit Q-Learning," arXiv:2110.06169) — expectile regression sidestepping explicit pessimism.
|
||||
|
||||
F4-MVP is a simplified Nair-2020 (AWAC-style) warm-start: supervised regression on perfect-foresight return labels, then RL fine-tune. The trade-offs vs CQL pessimism / IQL expectile are out of scope for F4-MVP; revisit in F4-Full if MVP shows benefit.
|
||||
|
||||
For the heuristic oracle specifically: closest precedent is Schaul et al. 2015 ("Prioritized Experience Replay," arXiv:1511.05952) which uses TD-error-rank as a heuristic priority; F4-MVP uses heuristic-realized-return as a label rather than as a sampling weight. The mechanism difference is the level of supervision (PER reweights existing TD targets; F4 supplies new MSE targets).
|
||||
|
||||
---
|
||||
|
||||
## §3. What infrastructure is needed
|
||||
|
||||
This is the load-bearing section. F4-MVP requires building four pieces of infrastructure foxhunt does not currently have.
|
||||
|
||||
### §3.1 Oracle computation pipeline
|
||||
|
||||
**Purpose:** Given an MBP-10 trading session trace, produce a per-state oracle Q-value `Q_oracle(s_t, a_argmax)` and identify `a_argmax`.
|
||||
|
||||
**Where it runs.** Two design options:
|
||||
|
||||
| Option | Runs in | Pros | Cons |
|
||||
|---|---|---|---|
|
||||
| §3.1.A — Python preprocess (offline) | Python script over MBP-10 fixtures; outputs binary | Fast iteration on oracle policy logic; no CUDA-kernel changes; reuses `crates/ml-alpha/cuda/...` reward kernel definitions via JSON-mirrored math | Risk of math drift between Python and CUDA reward kernel; requires regeneration on every reward-shape change |
|
||||
| §3.1.B — Rust + CUDA on-the-fly | A new binary `bin/foxhunt-oracle-gen` invoking the same `rl_fused_reward_pipeline.cu` and LobSim CUDA kernels the trainer uses | No math-drift risk (uses exact trainer reward); reusable for ablation studies | Higher infra cost; longer iteration cycle; CUDA dev environment required |
|
||||
|
||||
**Recommendation:** §3.1.B (Rust + CUDA on-the-fly). The math-drift risk is load-bearing: per `pearl_diag_pnl_fields_are_shaped_reward_not_usd`, the diag's `_usd` fields were misleading because the reward function was shaped differently than supposed. F4 oracle must be COMPUTED with the trainer's reward function, not a Python re-derivation, or F4 will fail F4-G1 ground-truth gate due to reward-function-identity bugs that are silent.
|
||||
|
||||
**Cost analysis (back-of-envelope, must be verified in Phase 7c.0).**
|
||||
|
||||
- 24 months × ES × MBP-10 events ≈ ~10⁸ states (per `pearl_mbp10_data_is_structurally_broken`, cluster PVC has 148GB coherent 24-month ES MBP-10; foxhunt's effective training-event throughput is ~10⁷-10⁸ per fold)
|
||||
- F4-MVP needs only `(s_t, a_argmax, Q_argmax)` per state. Per state: ~16 bytes (state-id i64 + a_argmax i32 + Q_argmax f32). Per fold: ~1.6GB at 10⁸ states.
|
||||
- However: producing `a_argmax` requires evaluating all 11 actions per state (per §2.4 algorithm), each requiring K=400-step LobSim rollout. At GPU-batched throughput ~10⁵ states/sec (extrapolated from existing LobSim benchmarks), full-fold oracle generation = 10⁸ / 10⁵ = ~10³ seconds = ~17 minutes per fold (highly speculative; Phase 7c.0 must measure).
|
||||
- Per-action rollout is embarrassingly parallel across states and across actions. GPU implementation should batch them efficiently.
|
||||
|
||||
**Storage volume.** F4-MVP per-fold ≈ 1.6GB. Acceptable on local 4GB RTX 3050 (oracle generation is one-shot, not held in GPU memory during the trainer step). PVC storage on cluster is ample (Scaleway Kapsule has terabytes available).
|
||||
|
||||
**Frictionless vs realistic LobSim.** Two sub-options:
|
||||
|
||||
| Sub-option | LobSim accounting | Pros | Cons |
|
||||
|---|---|---|---|
|
||||
| §3.1.B.1 — Frictionless oracle | Slippage = 0; fills at mid; commission = 0 | Simpler; faster compute; tighter upper bound on achievable pnl | Oracle Q values overestimate real-world achievable; risk of Q being miscalibrated when RL fine-tune introduces friction |
|
||||
| §3.1.B.2 — Realistic oracle | Use the SAME LobSim friction model the trainer uses (slippage, commission, MAS) | Math-identity with trainer; F4-pretrained Q is correctly calibrated | Slower compute; oracle complexity bound by LobSim complexity |
|
||||
|
||||
**Recommendation:** §3.1.B.2 (realistic oracle). Same reasoning as §3.1.B vs §3.1.A: math-identity with trainer is load-bearing. Better to spend the compute than to debug calibration mismatches later.
|
||||
|
||||
**Action-state coupling (load-bearing).** Several of foxhunt's 11 actions depend on `pos_state`, `unit_state`, and `trail_distance`:
|
||||
|
||||
- `ACTION_HOLD` is universally available but its effect depends on whether the trail is breached.
|
||||
- `ACTION_TRAIL_LOOSEN` is a no-op at trail-max cap.
|
||||
- `ACTION_PYRAMID_LONG/SHORT` is unavailable from flat (per F5 mask), and effect depends on `unit_state` (how many units are already open).
|
||||
- `ACTION_FLAT_FROM_LONG/SHORT` is a no-op when flat.
|
||||
- `ACTION_HALF_FLAT_LONG/SHORT` reduces position size by 50%.
|
||||
|
||||
The oracle simulator must track `pos_state` and `unit_state` per (s_t, a) branch through the K-step rollout. This means each (state, action) branch is a separate stateful simulation. Per-state cost is therefore O(K × |A|) LobSim steps = 400 × 11 = ~4400 simulator transitions per state. Over 10⁸ states → 4.4 × 10¹¹ simulator transitions. This is large; Phase 7c.0 cost estimate may need to recalibrate by:
|
||||
|
||||
- Pruning: pre-mask actions that are no-ops per state (saves ~50% of work for most flat states).
|
||||
- Stride sampling: rather than oracle-label every state, label every Nth state (N=10-100) and let RL handle the rest. F4-MVP must specify N.
|
||||
|
||||
**Time horizon truncation.** The oracle integral is over an infinite-horizon discounted return. Truncate at K such that `γ^K · max_possible_reward < ε`. For γ=0.975, K=400 → γ^K ≈ 5e-5 — adequate if max reward magnitude is bounded (it is, via PopArt and trail-max). Use K=400 unless Phase 7c.0 finds the truncation introduces meaningful bias.
|
||||
|
||||
**Per-action MTM accounting.** The oracle MUST consume the same reward function as the trainer:
|
||||
|
||||
```
|
||||
r_shaped(s, a, s_next) =
|
||||
r_base(s, a, s_next) # base pnl-delta or popart-whitened
|
||||
+ RL_MTM_REWARD_WEIGHT_INDEX · MTM_delta(s, s_next) # Phase 5 MTM
|
||||
+ Phase_5_short_hold_penalty(s, a, s_next) # Phase 5 term 1 (non-potential)
|
||||
+ Phase_5_long_ride_bonus(s, a, s_next) # Phase 5 term 3 (non-potential)
|
||||
+ Phase_5_hold_bonus(s, a, s_next) # Phase 5 term 4 (gate-zeroed, almost-potential)
|
||||
+ Phase_5_entry_cost # Phase 5 term 2 (non-potential)
|
||||
+ Phase_7a_F2_hinge_signal # if F2 enabled — but F2 is the consumer not a producer
|
||||
```
|
||||
|
||||
Per `pearl_phase5_term_4_is_almost_potential`, three of Phase 5's four terms are explicitly non-potential. F4 oracle MUST include them (otherwise Q is calibrated for a different reward function than the trainer uses → F4-G3 will fail). The oracle reward function is the SAME function the trainer reads from `rl_fused_reward_pipeline.cu`.
|
||||
|
||||
**Versioning.** When the reward function changes (e.g., Phase 5 MTM weight tuned, new shaping term added), the oracle must be regenerated. Implementation plan must:
|
||||
|
||||
- Compute a hash of the reward-function configuration (Phase 5 weights + Phase 7a F2 mode + γ + K).
|
||||
- Embed the hash in the oracle file name.
|
||||
- Trainer reads the hash and refuses to load oracle if hash mismatches current ISV state.
|
||||
|
||||
### §3.2 Storage format and access path
|
||||
|
||||
**Format options.**
|
||||
|
||||
| Option | Format | Pros | Cons |
|
||||
|---|---|---|---|
|
||||
| §3.2.A — Predecoded binary (foxhunt convention) | `.predecoded.bin` with same MMap pattern as MBP-10 trace | Reuses existing infra (`crates/ml-alpha/src/data/loader.rs:85-98`); zero-copy I/O | Custom serialization needed |
|
||||
| §3.2.B — Parquet | Columnar Arrow-compatible | Industry standard; introspectable | Adds Arrow dep to ml-alpha; per-row decode cost |
|
||||
| §3.2.C — TFRecord / Numpy | Cross-tool compatibility | Easy to inspect from Python | Foreign to foxhunt's Rust toolchain |
|
||||
|
||||
**Recommendation:** §3.2.A (predecoded binary). Reuses foxhunt's existing MMap convention. Naming pattern:
|
||||
|
||||
```
|
||||
<predecoded_dir>/ES.FUT_<quarter>.dbn.zst.oracle_q_argmax_v<reward_hash>.predecoded.bin
|
||||
```
|
||||
|
||||
Where the file body is a sequence of `(state_global_id: u64, a_argmax: u8, q_argmax: f32)` records. Each record is 13 bytes; aligned to 16 for fast vectorized load.
|
||||
|
||||
The trainer's existing data loader (`crates/ml-alpha/src/data/loader.rs`) already MMaps MBP-10 traces — extending it to MMap a parallel oracle file is straightforward.
|
||||
|
||||
**Indexing.** The trainer references states by `(file_idx, byte_offset)` already (per existing MBP-10 loader). The oracle file uses the same indexing: each state in the MBP-10 trace has a corresponding entry in the oracle file at the same record offset. State-by-state alignment is therefore zero-cost (no hash table, no lookup, just record-index arithmetic).
|
||||
|
||||
**Cluster delivery.** PVC-mounted, same as MBP-10 training data:
|
||||
|
||||
```
|
||||
/mnt/training-data/futures-baseline/ES.FUT/ES.FUT_<quarter>.dbn.zst.oracle_q_argmax_v<hash>.predecoded.bin
|
||||
```
|
||||
|
||||
The Argo template `argo-alpha-rl.sh` would need to mount the new file (no template change required since PVC already exposes the directory).
|
||||
|
||||
**Versioning protocol.**
|
||||
|
||||
1. Reward-function hash computed at compile time from `crates/ml-alpha/src/rl/isv_slots.rs` Phase 5 + Phase 7a bootstrap values + γ.
|
||||
2. Oracle generation binary embeds the hash in the filename.
|
||||
3. Trainer at load time reads the hash and: (a) if it matches → load; (b) if it mismatches → log a warning AND refuse to load (force regeneration).
|
||||
4. CI / cluster automation regenerates oracle when the hash changes (Argo workflow step).
|
||||
|
||||
### §3.3 Supervised pretraining phase
|
||||
|
||||
**Where in the trainer it runs.** Before the existing `step_with_lobsim_gpu` loop's first call:
|
||||
|
||||
```rust
|
||||
pub fn step_with_lobsim_gpu(...) -> Result<...> {
|
||||
if self.f4_pretrain_enabled() && !self.f4_pretrain_done {
|
||||
self.pretrain_from_oracle()?; // ← NEW
|
||||
self.f4_pretrain_done = true;
|
||||
}
|
||||
// Existing logic from here.
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
**New training loop body (F4-MVP).**
|
||||
|
||||
```rust
|
||||
fn pretrain_from_oracle(&mut self) -> Result<()> {
|
||||
let n_batches = oracle_dataset.len() / BATCH_SIZE; // BATCH_SIZE = 128 local, 1024 cluster
|
||||
for batch_idx in 0..n_batches {
|
||||
// 1. Sample BATCH_SIZE (state_id, a_argmax, q_argmax) records.
|
||||
let (state_batch, a_argmax_batch, q_argmax_batch) = oracle_dataset.sample(BATCH_SIZE, prng_seed);
|
||||
|
||||
// 2. Run encoder forward on state_batch (existing forward kernel chain).
|
||||
self.encoder_forward(&state_batch)?;
|
||||
|
||||
// 3. Q head forward to produce Q_θ(s, a_argmax) — read only at a_argmax positions.
|
||||
self.q_head_forward(&state_batch)?;
|
||||
let q_pred = self.gather_q_at_action(&a_argmax_batch); // [B] — Q_θ(s, a_argmax)
|
||||
|
||||
// 4. MSE loss: (q_pred - q_argmax_batch).pow(2).mean()
|
||||
let loss = (q_pred - q_argmax_batch).pow(2).mean();
|
||||
|
||||
// 5. Backward + Adam through Q head AND encoder.
|
||||
loss.backward();
|
||||
self.optimizer.step();
|
||||
|
||||
// 6. Diag emit: pretrain step + MSE + max |Δ|
|
||||
self.emit_pretrain_diag(batch_idx, loss);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
**Encoder freeze (F4-MVP decision).** Both Q head AND encoder train during pretraining (encoder unfrozen). Rationale: foxhunt's encoder (mamba2-based) shares weight with the policy head; freezing the encoder would also freeze the policy's perceptual capacity. F4-Full may revisit encoder-only or Q-only freeze variants once F4-MVP results are in.
|
||||
|
||||
**Number of pretrain epochs.** ONE epoch. Adaptive stopping (e.g., until MSE plateaus) is F4-Full scope.
|
||||
|
||||
**Gradient flow specifics (load-bearing).**
|
||||
|
||||
- The MSE loss writes to Q head weights (the same weights the Bellman target writes to during RL fine-tune). NO separate "pretrain head" — F4-MVP intentionally uses the production Q head.
|
||||
- The encoder gradient flows from the Q head's L2 loss back through the trunk. This means encoder weights at the start of RL fine-tune are a *function of the oracle pretrain*. If the oracle is biased (e.g., heuristic policy is bad), the encoder is biased in that direction.
|
||||
- Gradient clipping: use the existing trainer's per-batch gradient norm clip (slot TBD; verify in Phase 7c.0).
|
||||
- Learning rate: use the existing Q-head Adam learning rate. NO new LR controller for pretrain (F4-MVP simplification). If MSE diverges, this is a STOP signal per F4-G2.
|
||||
|
||||
**New ISV slots.**
|
||||
|
||||
| Slot ID | Name | Bootstrap | Effect when bootstrap |
|
||||
|---|---|---|---|
|
||||
| 824 (next free) | `RL_F4_PRETRAIN_MODE_INDEX` | 0.0 (OFF) | 0=skip, 1=MVP one-epoch, 2=full multi-epoch (reserved) |
|
||||
| 825 | `RL_F4_PRETRAIN_BATCH_COUNT_INDEX` | 0.0 (read-only counter) | Filled by trainer with `n_batches` at start of pretrain; diag exposes for verification |
|
||||
| 826 | `RL_F4_PRETRAIN_FINAL_MSE_INDEX` | 0.0 (read-only) | Filled by trainer at end of pretrain; F4-G2 reads this |
|
||||
| 827 | `RL_F4_PRETRAIN_DONE_INDEX` | 0.0 | Set to 1.0 once pretrain completes; trainer reads to gate "skip if already done" |
|
||||
|
||||
Slot ID ranges are illustrative; Phase 7c.0 must verify current `RL_SLOTS_END` and allocate from there.
|
||||
|
||||
**Determinism.** The pretrain phase must preserve `pearl_determinism_achieved`. Oracle reads use the established seeded-PRNG pattern (per the IQN race fix + OutcomeHead seed fix in commit `c75c45a93`). The pretrain epoch's sample order is deterministic for a given seed.
|
||||
|
||||
### §3.4 Trainer integration points
|
||||
|
||||
Per `pearl_foxhunt_trainer_is_genuinely_single_step`, the trainer is `[B]`-only. F4-MVP intentionally preserves this:
|
||||
|
||||
- Pretrain samples are `[B]`-shaped (one batch = B independent states, one action each).
|
||||
- No rollout buffer needed — the supervised loss is computed over individual transitions.
|
||||
- The pretrain loop is a separate code path in `integrated.rs`; it does not interfere with the trainer's main step body.
|
||||
|
||||
**New struct fields on `IntegratedTrainer`.**
|
||||
|
||||
```rust
|
||||
pub struct IntegratedTrainer {
|
||||
// ... existing fields ...
|
||||
|
||||
/// F4 oracle storage handle (memory-mapped predecoded binary).
|
||||
pub oracle_storage: Option<crate::data::oracle_loader::OracleStorage>,
|
||||
|
||||
/// F4 pretrain step counter (for diag emit).
|
||||
pub f4_pretrain_step: u64,
|
||||
|
||||
/// F4 pretrain done flag (idempotent skip if already done).
|
||||
pub f4_pretrain_done: bool,
|
||||
}
|
||||
```
|
||||
|
||||
**New code paths.**
|
||||
|
||||
1. `OracleStorage::load_from_predecoded(path) -> Self` — MMap and validate hash.
|
||||
2. `IntegratedTrainer::pretrain_from_oracle(&mut self) -> Result<()>` — the supervised loop body.
|
||||
3. `IntegratedTrainer::oracle_batch_sample(&mut self, batch_size, prng_seed) -> (StateIds, ActionIds, QValues)` — deterministic batch sampler.
|
||||
4. `IntegratedTrainer::gather_q_at_action(actions: &[u8]) -> CudaSlice<f32>` — utility to gather Q_θ values at specific action indices (similar to the existing `gather` pattern in `rl_q_pi_distill_grad.cu`).
|
||||
|
||||
**Determinism wiring.**
|
||||
|
||||
- The oracle sample PRNG seeds from the existing trainer seed.
|
||||
- The kernel reads from `oracle_q_d` (a new CudaSlice<f32>) and `oracle_a_argmax_d` (a new CudaSlice<u8>) are race-free (single thread per batch reads its row).
|
||||
- The MSE gradient kernel uses the existing Adam optimizer step.
|
||||
|
||||
**Code structure.** A new module `crates/ml-alpha/src/data/oracle_loader.rs` for the predecoded-binary MMap; a new module `crates/ml-alpha/src/trainer/f4_pretrain.rs` for the pretrain loop body. Both files are new but the surface area is small (~200-400 LOC each, mostly trivial).
|
||||
|
||||
---
|
||||
|
||||
## §4. F4-MVP scope boundary
|
||||
|
||||
The discipline boundary of F4-MVP is the single most important section of this spec for preventing scope creep into F4-Full.
|
||||
|
||||
### §4.1 What F4-MVP IS
|
||||
|
||||
- ONE supervised pretrain epoch (no curriculum, no annealing, no early-stopping on MSE).
|
||||
- argmax-only oracle: one `(s_t, a_argmax, Q_argmax)` triple per state.
|
||||
- Q-head + encoder both training during pretrain (no freeze).
|
||||
- Single fold (fold 1) only.
|
||||
- Pretrain writes to SAME Q head weights that Bellman target writes to during RL fine-tune.
|
||||
- NO separate "pretrain head."
|
||||
- NO distillation regularization to prevent post-pretrain Q drift.
|
||||
- NO online oracle update during RL.
|
||||
|
||||
### §4.2 What F4-MVP is NOT
|
||||
|
||||
| Out-of-scope feature | Why excluded from MVP | When to revisit |
|
||||
|---|---|---|
|
||||
| Multi-epoch supervised | F4-Full; MVP tests whether one-epoch warm-start moves the needle | F4-MVP cluster passes F4-G4; user wants to explore further pretrain |
|
||||
| Full-distribution oracle (all 11 actions) | F4-Full; 11× compute cost; MVP-only-argmax is the cheapest meaningful test | F4-MVP fails F4-G3 because non-argmax actions need supervision |
|
||||
| Encoder freeze | F4-Full; risks overfitting trade-off | F4-MVP cluster shows encoder catastrophic-shift post-pretrain |
|
||||
| Per-fold oracle generation (walk-forward) | F4-Full; MVP tests fold-1 only | F4-MVP fold-1 passes and we need to verify the win generalizes across folds |
|
||||
| Action-curriculum (oracle gradually reveals actions) | F4-Full | F4-MVP fails F4-G6 because Q drifts too far from oracle during RL |
|
||||
| Reward-shape-curriculum (oracle gradually reveals shaping) | F4-Full | Phase 5 shaping terms cause oracle to over-recommend wait |
|
||||
| Adversarial / smarter-than-heuristic oracle | F4-Full | F4-G1 ground-truth gate suggests heuristic ceiling is too low |
|
||||
| Behavioral cloning from human demonstrations | Out of scope entirely | No human demonstrations exist in foxhunt |
|
||||
| Imitation learning from a learned-policy expert | Out of scope entirely | A learned expert policy must itself be trained — recursive |
|
||||
| Online oracle update (Dyna-style) | F4-Full; major arch change | F4-MVP cluster passes and we want to tighten oracle as policy improves |
|
||||
| Hindsight relabeling (HER) | Out of scope; orthogonal | Separate spec if pursued |
|
||||
| F4 + F5 combined (Phase 7c.6 if needed) | Sequenced after F4-MVP-only result | F4-MVP fails F4-G7 outcome and Phase 7c.6 is invoked per §6 |
|
||||
|
||||
---
|
||||
|
||||
## §5. Empirical falsification gates
|
||||
|
||||
Three categories: **Pretrain-phase gates** (verify oracle pipeline is correct), **RL fine-tune phase gates** (verify F4 changes downstream training), **Cluster outcome gates** (verify F4 actually helps).
|
||||
|
||||
### §5.1 Pretrain-phase gates — verify oracle pipeline correctness
|
||||
|
||||
Run during Phase 7c.0 / Phase 7c.1 / Phase 7c.2 / Phase 7c.3 of the implementation plan.
|
||||
|
||||
| Gate ID | Metric | Pass condition | When measured | What it falsifies |
|
||||
|---|---|---|---|---|
|
||||
| F4-G1 (oracle ground-truth) | Pearson correlation between `Q_oracle(s_t, a_argmax)` and the actually-realized discounted return when an agent forced to take `a_argmax` is run forward K=400 steps on held-out data | ≥ 0.7 | Phase 7c.0 (offline) | If fails: oracle policy heuristic is wrong (re-design heuristic or escalate to F4-Full smarter oracle) |
|
||||
| F4-G2 (pretrain loss decreases) | Final MSE after 1 epoch / initial MSE at epoch start | ≤ 1e-2 (i.e., MSE drops ≥ 100×) | Phase 7c.2 (local Tier 1.5) | If fails: supervised loss path is broken (kernel bug, wrong gradient flow, oracle hash mismatch) |
|
||||
| F4-G3 (Q-head outputs match oracle) | On held-out state subset: `argmax_a Q_θ(s_t, a) == a_argmax` agreement rate | ≥ 70% | Phase 7c.3 (local Tier 1.5) | If fails: encoder/Q-head capacity insufficient OR oracle is degenerate (most a_argmax are Hold) |
|
||||
|
||||
**Honest caveat.** F4-G1's 0.7 Pearson threshold is calibrated against `pearl_reward_signal_anti_aligned_with_pnl` baseline (0.11-0.45 random-policy). 0.7 is a *substantial* improvement target; if F4-G1 produces 0.5-0.7, this may still be worthwhile but is a STOP-and-evaluate signal. If F4-G1 produces < 0.5, the oracle heuristic is broken and Phase 7c.1+ MUST NOT proceed (the pipeline would be wired to wrong labels).
|
||||
|
||||
### §5.2 RL fine-tune phase gates — verify F4 changes downstream training
|
||||
|
||||
Run during Phase 7c.4 of the implementation plan, with F4 pretrain ON and standard RL fine-tune proceeding.
|
||||
|
||||
| Gate ID | Metric | Pass condition | When measured | What it falsifies |
|
||||
|---|---|---|---|---|
|
||||
| F4-G4 (RL improves vs cold-start) | Steps until `l_q` reaches the cold-start cluster `l_q` at step N | ≤ 0.3 × N | Phase 7c.4 (local Tier 1.5) | If fails: pretrained Q drifts away from oracle too fast during RL (need distillation regularization, F4-Full) |
|
||||
| F4-G5 (early policy quality) | Action entropy at step 100 of RL | ≥ 1.5 nats | Phase 7c.4 | If fails: pretrained policy collapsed during pretrain (encoder overfit or oracle too argmax-degenerate) |
|
||||
| F4-G6 (no surfer-attractor migration) | F2's `hinge_zero_rate` (existing diag from F2 commit) at step 1000 | ≤ 0.5 | Phase 7c.4 | If fails: F4's pretrained Q does not prevent F2's hinge from saturating (Q still has Hold > Open everywhere) |
|
||||
|
||||
### §5.3 Cluster outcome gates — verify F4 actually helps
|
||||
|
||||
Run during Phase 7c.5 of the implementation plan, at cluster Tier 2 (b=1024, 20k train + 5k eval, fold 1, ~70 min wallclock).
|
||||
|
||||
| Gate ID | Metric | Pass condition | When measured | What it falsifies |
|
||||
|---|---|---|---|---|
|
||||
| F4-G7 (cluster eval pnl) | `eval_summary.total_pnl_usd` | ≥ F2+F5 (sk2z9 baseline) cluster eval pnl + $10M / session | Phase 7c.5 | If fails: F4-MVP does not solve the eval-fail problem; consider F4-Full or escalate to model-architecture |
|
||||
| F4-G8 (no degenerate attractor) | Max single-action share at eval steps 5k and 15k | < 80% | Phase 7c.5 | If fails: F4 has migrated to a different degenerate (e.g., always-Open-Long); diagnose before committing |
|
||||
| F4-G9 (train→eval transfer) | `eval_pnl / train_pnl` | ≥ 0.7 | Phase 7c.5 | If fails: F4 does not solve the train→eval generalization gap; problem is upstream of Q-init (encoder generalization, regime shift) |
|
||||
|
||||
**F4-G9 is the most consequential gate.** Per `pearl_surfer_baseline_was_train_only_never_eval`, the train→eval gap has been the long-standing failure mode (eval ≈ 5% of train at b=128 currently). F4 explicitly aims to address this by giving Q a starting point that already encodes "what works on held-out data." If F4 still produces ~5% eval-to-train ratio, F4-MVP has not solved the problem and we have refuted the encoder-Q-bootstrap hypothesis. F4-G9 failure is a *progress signal* — it isolates the next investigation direction.
|
||||
|
||||
### §5.4 Stop-on-fail rules
|
||||
|
||||
- **F4-G1 fails (oracle ground-truth Pearson < 0.7):** STOP at Phase 7c.0; redesign oracle heuristic or escalate to F4-Full smarter oracle. DO NOT proceed to Phase 7c.1+.
|
||||
- **F4-G2 fails (MSE does not decrease):** STOP at Phase 7c.2; the supervised loss path is broken (kernel bug, hash mismatch, or gradient-flow bug). Debug before continuing.
|
||||
- **F4-G3 fails (Q-head does not match oracle on held-out):** STOP at Phase 7c.3; either encoder capacity insufficient, or oracle is degenerate (overwhelmingly Hold). Either escalate to F4-Full (full-distribution oracle) or accept and proceed to Phase 7c.4 with degraded expectation.
|
||||
- **F4-G4, F4-G5, or F4-G6 fail at smoke:** Do not cluster. Diagnose with `pearl_fleet_fraction_not_aggregate`-discipline diag; consider F4-Full distillation regularization.
|
||||
- **F4-G7 fails at cluster but G8 + G9 are healthy:** F4-MVP insufficient; consider Phase 7c.6 (F4 + F5) or escalate to F4-Full.
|
||||
- **F4-G9 fails at cluster regardless of G7:** F4-MVP has refuted the encoder-Q-bootstrap hypothesis. STOP; escalate to model-architecture work (e.g., different encoder, attention modifications) or train→eval regime-shift work.
|
||||
|
||||
### §5.5 The "going in circles" guard
|
||||
|
||||
Per `feedback_going_in_circles_pattern` and `feedback_investigation_first_falsification_methodology`: F4 is the third independent intervention axis tested in Phase 7 (gradient axis = F2; choice-set axis = F5; initial-conditions axis = F4). If F4-MVP fails on F4-G7 AND F4-G9, the predetermined next action is NOT another Bellman-target reshape (per Phase 7 spec §5.4 → all the easy variants are now tested). The next direction is upstream: encoder, reward function, or train/eval regime structure. This guard prevents the empirical loop from spawning a Phase 8 / 9 / 10 series of more-Bellman-reshapes.
|
||||
|
||||
---
|
||||
|
||||
## §6. Implementation phases
|
||||
|
||||
Each phase has a predetermined gate and a STOP-if-fail rule. Phases run sequentially.
|
||||
|
||||
### Phase 7c.0 — Oracle pipeline (build + verify) — ~2-3 days
|
||||
|
||||
Build the offline oracle generator. Verify ground-truth correlation.
|
||||
|
||||
- Build `bin/foxhunt-oracle-gen` (a new binary).
|
||||
- Implement realistic LobSim rollout for each (s_t, a) over K=400 steps.
|
||||
- Implement heuristic-policy continuation per §2.4.
|
||||
- Output `(state_global_id, a_argmax, q_argmax)` triples in predecoded.bin format.
|
||||
- Test against hand-computed ground truth on 10 sample states (verify by inspection).
|
||||
- Generate oracle for fold-1 training data (one quarter of ES MBP-10).
|
||||
- Compute F4-G1: Pearson correlation between oracle `Q_argmax` and forced-rollout realized return on held-out fold-1-eval states.
|
||||
|
||||
**Gates:** F4-G1 only.
|
||||
|
||||
**STOP if:** F4-G1 < 0.7. Redesign heuristic or accept F4-Full as required prerequisite.
|
||||
|
||||
### Phase 7c.1 — Storage + access path (build + verify) — ~1-2 days
|
||||
|
||||
Wire oracle reads into the trainer.
|
||||
|
||||
- Add `crates/ml-alpha/src/data/oracle_loader.rs` (MMap predecoded.bin).
|
||||
- Add hash validation + version check.
|
||||
- Add `OracleStorage::sample(batch_size, prng_seed)` with bit-deterministic behavior.
|
||||
- Write a Rust unit test that two same-seed sample calls produce identical (state_id, a, q) batches.
|
||||
- Add minimal Tier 1 correctness test (load + sample + validate hash).
|
||||
|
||||
**Gates:** Determinism unit test only (no behavioral gate yet).
|
||||
|
||||
**STOP if:** Determinism test fails. Per `pearl_determinism_achieved`, this MUST hold.
|
||||
|
||||
### Phase 7c.2 — Pretraining MSE loss (build + verify) — ~2-3 days
|
||||
|
||||
Add `pretrain_from_oracle()` method.
|
||||
|
||||
- Add new ISV slots 824-827 per §3.3.
|
||||
- Implement `IntegratedTrainer::pretrain_from_oracle()` body (MSE loss, backprop, Adam step).
|
||||
- Default `RL_F4_PRETRAIN_MODE_INDEX = 0.0` (OFF) — pretrain skipped by default.
|
||||
- Add diag emit for `pretrain_mse_initial`, `pretrain_mse_final`, `pretrain_steps_completed`.
|
||||
- Run local Tier 1.5 smoke with `RL_F4_PRETRAIN_MODE_INDEX = 1.0` and verify MSE decreases (F4-G2).
|
||||
|
||||
**Gates:** F4-G2 (MSE decreases ≥ 100×).
|
||||
|
||||
**STOP if:** F4-G2 fails. Debug gradient flow / hash mismatch.
|
||||
|
||||
### Phase 7c.3 — Verify pretrained Q matches oracle — ~1 day
|
||||
|
||||
Confirm F4-G3 on held-out subset.
|
||||
|
||||
- Add a Rust integration test that runs the full pretrain on local fold-1 train fixtures, then evaluates `argmax_a Q_θ` vs `a_argmax` on held-out states.
|
||||
- Emit per-state agreement rate.
|
||||
- Report aggregate F4-G3 metric.
|
||||
|
||||
**Gates:** F4-G3 (≥ 70% argmax agreement).
|
||||
|
||||
**STOP if:** F4-G3 < 50%. Likely either degenerate oracle (mostly Hold) or insufficient encoder capacity; consider F4-Full full-distribution oracle before proceeding.
|
||||
|
||||
### Phase 7c.4 — RL fine-tune from pretrained Q (smoke + verify) — ~1 day
|
||||
|
||||
Run Tier 1.5 smoke with full F4 pipeline.
|
||||
|
||||
- Configure: F2 ON, F5 OFF (clean F4 isolation), F4 pretrain ON, Band ON (for surfer principle preservation).
|
||||
- Run mid-smoke (~10 min wallclock).
|
||||
- Verify F4-G4, F4-G5, F4-G6 against post-pretrain diag fields.
|
||||
- Run `determinism-check.sh --quick` to verify pretrain phase preserves bit-equality.
|
||||
|
||||
**Gates:** F4-G4, F4-G5, F4-G6, determinism preserved.
|
||||
|
||||
**STOP if:** Any RL fine-tune gate fails. Diagnose before cluster.
|
||||
|
||||
### Phase 7c.5 — Cluster verify — ~80 min wallclock + analysis time
|
||||
|
||||
Submit Argo run.
|
||||
|
||||
- Configure: single-head, Band ON, F2 ON, F5 OFF, F4 ON.
|
||||
- Submit cluster Tier 2 (b=1024, 20k train + 5k eval, fold 1).
|
||||
- Wait ~70 min for completion.
|
||||
- Analyze `eval_summary.json` for F4-G7, F4-G8, F4-G9.
|
||||
|
||||
**Gates:** F4-G7 (eval pnl improvement), F4-G8 (no new attractor), F4-G9 (train→eval transfer).
|
||||
|
||||
**STOP if:** F4-G7 fails AND F4-G9 fails → escalate per §5.4. F4-G7 alone fails AND F4-G9 passes → consider Phase 7c.6 (F4 + F5 combined).
|
||||
|
||||
### Phase 7c.6 (CONDITIONAL) — Re-engage F5 with F4 pretrain — ~80 min cluster
|
||||
|
||||
ONLY if Phase 7c.5 shows F4 alone insufficient on F4-G7 BUT F4-G9 healthy (train→eval transfer improved).
|
||||
|
||||
- Configure: single-head, Band ON, F2 ON, F5 ON, F4 ON.
|
||||
- Submit cluster Tier 2.
|
||||
- Compare F4 + F5 vs F4-alone vs sk2z9 (F2 + F5 + OptA without F4).
|
||||
|
||||
**Gates:** F4-G7 vs F4-alone (must show marginal improvement from F5).
|
||||
|
||||
**STOP if:** Phase 7c.6 fails. F4-Full or model-architecture work is the next direction.
|
||||
|
||||
### Total time estimate
|
||||
|
||||
| Phase | Duration | Cumulative |
|
||||
|---|---|---|
|
||||
| 7c.0 | 2-3 days | 2-3 days |
|
||||
| 7c.1 | 1-2 days | 3-5 days |
|
||||
| 7c.2 | 2-3 days | 5-8 days |
|
||||
| 7c.3 | 1 day | 6-9 days |
|
||||
| 7c.4 | 1 day | 7-10 days |
|
||||
| 7c.5 | 1-2 days (incl. analysis) | 8-12 days |
|
||||
| 7c.6 (conditional) | 1-2 days | 9-14 days |
|
||||
|
||||
**Honest range: 7-14 working days for F4-MVP cluster verdict.** Faster if every phase passes first attempt; slower if Phase 7c.0 oracle heuristic needs iteration (likely) or Phase 7c.5 cluster surfaces new pathology (also likely given Phase 7 history).
|
||||
|
||||
---
|
||||
|
||||
## §7. Surprises, risks, and out-of-scope
|
||||
|
||||
### §7.1 Honest risks
|
||||
|
||||
1. **Oracle ground-truth requires non-trivial LobSim simulation.** §2.4's heuristic policy is itself a hypothesis. The Phase 7c.0 ground-truth gate (F4-G1) is non-trivial to pass. If heuristic-realized-return correlates poorly with achievable return, the oracle is broken before we get to the trainer. Mitigation: spend the time on the heuristic; consider hybrid heuristic + RL-trained-on-small-fold for the oracle policy.
|
||||
|
||||
2. **Q drift during RL fine-tune may undo the warm-start.** Without distillation regularization (KL between current Q and pretrained Q, à la AWAC), the RL Bellman target may move Q rapidly away from oracle. F4-MVP intentionally excludes this regularizer for scope reasons, but F4-G4 (RL improvement rate) and F4-G6 (hinge_zero_rate) will detect rapid drift. If drift is severe, F4-Full distillation is required.
|
||||
|
||||
3. **Encoder may overfit during pretraining.** One epoch with no dropout / weight decay could push the encoder to memorize the oracle's training-fold features. Mitigation: F4-G5 (action entropy at step 100 of RL) detects overfit-collapse. If encoder overfit is detected, F4-Full encoder-freeze variant is needed.
|
||||
|
||||
4. **F4 does not solve regime shift.** If the test data has different action-quality structure than the train data, F4 trained on train will give a Q that is correctly bootstrapped for train but wrong for eval. This is a fundamental limitation. F4-G9 (train→eval transfer ratio) measures the impact. If F4-G9 fails, the problem is regime-shift, not Q-init — F4 does not solve it.
|
||||
|
||||
5. **Reward function identity is load-bearing.** Per `pearl_diag_pnl_fields_are_shaped_reward_not_usd`, the trainer's `r_shaped` is non-trivial (Phase 5 MTM + entry cost + short-hold penalty + long-ride bonus). The oracle MUST use the EXACT SAME function. Re-derivation in Python (§3.1.A rejected option) was rejected precisely because of this. If §3.1.B math-identity is not maintained, F4 will appear to fail for the wrong reasons.
|
||||
|
||||
6. **Determinism requirement is non-trivial.** Per `pearl_determinism_achieved`, the bit-equality discipline took ~170 LOC of fixes across multiple subsystems. The pretrain phase introduces new kernel paths (gather Q at action, MSE loss, etc.) — each must be carefully implemented to preserve same-seed bit-equality. F4-MVP's Phase 7c.4 includes a `determinism-check.sh` gate; do not skip.
|
||||
|
||||
7. **Cluster vs local divergence.** Per `pearl_cluster_log_is_ground_truth`, local b=128 PPO diverges even when cluster b=1024 trains stable. F4-G4/G5/G6 are local-Tier-1.5 gates; F4-G7/G8/G9 are cluster gates. They MAY produce different verdicts. The discipline is: do not skip local gates (catch obvious bugs early), but do not over-weight local results (cluster is ground truth).
|
||||
|
||||
### §7.2 Out of scope (explicit)
|
||||
|
||||
- F4-Full (multi-epoch, full-distribution oracle) — separate spec eventually.
|
||||
- Encoder-only pretraining (Q frozen, encoder trained on auxiliary task) — separate spec.
|
||||
- Adversarial / RL-trained oracle (smarter than argmax heuristic) — separate spec.
|
||||
- Online oracle update during RL (Dyna-style) — separate spec, major arch change.
|
||||
- Hindsight relabeling (HER) — orthogonal direction.
|
||||
- F4 + F5 combined (Phase 7c.6 if needed) — conditional within this spec.
|
||||
- Multi-fold walk-forward oracle generation — F4-Full.
|
||||
- Distillation regularization to prevent Q drift — F4-Full.
|
||||
- Confidence-gate position-aware wiring (`pearl_confidence_gate_pos_state_unwired`) — pre-existing, orthogonal.
|
||||
- GAE / rollout buffer — orthogonal; per `pearl_foxhunt_trainer_is_genuinely_single_step` not blocked by F4.
|
||||
|
||||
---
|
||||
|
||||
## §8. Decision-relevant alternatives surveyed
|
||||
|
||||
Briefly note alternatives ruled out at this stage. Full analysis is in the predecessor spec §3.
|
||||
|
||||
- **Architectural changes (e.g., transformer policy, attention modifications).** Out of scope for this dispatch; would require new encoder + retrain everything. Becomes the next direction if F4 fails F4-G9.
|
||||
- **Reward shaping F1 (per-step inaction cost).** Per Phase 7 spec §3.1: non-potential per Ng-Harada-Russell 1999, violates surfer principle, criterion-7 score 2. F4 has criterion-7 score 5 (preserves surfer by encoding patient behavior in oracle Q labels).
|
||||
- **F3 action-space reduction.** Per Phase 7 spec §3.3: criterion-7 score 1 (total surfer violation), removes documented architectural progress, not reversible. Rejected.
|
||||
- **Behavioral cloning from human expert.** No human expert demonstrations exist for foxhunt; out of scope.
|
||||
- **Imitation learning from a heuristic policy.** A heuristic policy would itself need to be designed; oracle is a stronger signal (perfect-foresight return vs heuristic action sequence).
|
||||
|
||||
**Explicit statement:** F4-MVP is the most disciplined next step because:
|
||||
|
||||
(a) F2+F5 mechanism gates pass but outcome gates fail at cluster (per `sk2z9` empirical data + spec §5.3 attribution outcome 3).
|
||||
|
||||
(b) F4 has the strongest theoretical foundation among unimplemented options (Levine 2020 / Nair 2020 / Kumar 2020 / Kostrikov 2021 — offline RL is a mature direction).
|
||||
|
||||
(c) F4-MVP scope is bounded to ~7-14 days with predetermined gates at each phase.
|
||||
|
||||
(d) The methodology `feedback_investigation_first_falsification_methodology` requires spec → plan → implement (not direct dive). This is the spec.
|
||||
|
||||
(e) F4 attacks a distinct mechanistic axis (initial conditions) from F2 (gradient) and F5 (choice-set). The Phase 7 axis-completion logic is preserved.
|
||||
|
||||
---
|
||||
|
||||
## §9. Open questions for the implementer
|
||||
|
||||
The implementation plan that follows this spec must resolve at least these questions:
|
||||
|
||||
1. **Trade-boundary marking in MBP-10.** Where in MBP-10 data layout do we identify trade boundaries for oracle return computation? The existing predecoded format includes mid-price events — does the oracle use mid-prices or top-of-book bid/ask for fill simulation? Verify in Phase 7c.0.
|
||||
|
||||
2. **Reward function versioning protocol.** Exact hash function for reward-function-state. Include: Phase 5 weight slots (MTM, entry cost, etc.), Phase 7a F2 slots, Phase 7b F5 slots, γ. Spec the hash format in the plan.
|
||||
|
||||
3. **Slippage / market impact accounting.** Should the oracle account for slippage? Phase 7 spec §3.5 LobSim trail-stop kernel includes a slippage model. Use the same? Or assume frictionless? Recommendation: §3.1.B.2 (realistic). Confirm in Phase 7c.0.
|
||||
|
||||
4. **Per-state action-availability tracking.** How is oracle action-availability tracked through the K-step rollout? Specifically, `unit_state` (current pyramid layer count) affects PYRAMID action effects. Spec the per-(s_t, a) state propagation.
|
||||
|
||||
5. **Oracle for train fold only or train+eval folds?** F4-MVP says train only (no oracle for eval). But F4-G3 (held-out match rate) requires SOME eval-state oracle to compute. Resolve: use a strict train-fold sub-split for in-pretrain held-out (e.g., last 10% of fold-1-train), OR generate oracle for fold-1-eval purely for the gate (and discard for training). Recommendation: latter — gate-only oracle on eval is acceptable.
|
||||
|
||||
6. **ISV slot range.** Per current state (`grep RL_SLOTS_END` shows 824), allocate 824-827 (or shift if collisions). Confirm in Phase 7c.2 step 1.
|
||||
|
||||
7. **Encoder LR for pretraining.** Use same LR as RL fine-tune? Or a separate (typically larger) supervised LR? F4-MVP simplification says: same LR. If F4-G2 shows MSE plateau too high, revisit.
|
||||
|
||||
8. **Diag emission discipline.** Per `pearl_fleet_fraction_not_aggregate`, F4 pretrain diag emissions MUST include per-batch fractions (e.g., "fraction of pretrain batches where MSE > threshold") not just aggregate `worst_X`. Specify diag schema in Phase 7c.2.
|
||||
|
||||
9. **Batch size for pretrain.** Use same B as RL (128 local / 1024 cluster)? Or different? F4-MVP simplification: same B. Recommend confirming compute headroom in Phase 7c.0.
|
||||
|
||||
10. **Resumption semantics.** What happens if pretrain is interrupted (kernel crash, OOM)? F4-MVP simplification: ALL or NOTHING — pretrain restarts from scratch on resume. `RL_F4_PRETRAIN_DONE_INDEX` is the idempotent gate. F4-Full would add checkpoint/resume.
|
||||
|
||||
---
|
||||
|
||||
## §10. Document discipline notes
|
||||
|
||||
- **No implementation code in this spec.** Pseudocode in §3.3 is design clarification, not implementation prescription.
|
||||
- **No new files outside this one spec.** The §3.1 / §3.4 references to `bin/foxhunt-oracle-gen` and `crates/ml-alpha/src/data/oracle_loader.rs` are forward references for the implementation plan, not files this spec creates.
|
||||
- **Honest scope.** F4 is multi-week even at MVP scope. The 7-14 day range in §6 is honest about variance. Do not commit to F4-MVP if there is < 2 weeks of focused capacity available; better to defer.
|
||||
- **Theoretical claims cited.** Every claim in §2.5, §7, §8 has a citation to predecessor spec, pearl, or external paper.
|
||||
- **Predetermined falsification at every phase.** §5.4 and §6's "STOP if" rules are not optional; they are the methodology's load-bearing discipline per `feedback_investigation_first_falsification_methodology`.
|
||||
- **No dispatch of further agents from this spec.** The output is the spec file. The implementation plan is a separate dispatch.
|
||||
|
||||
---
|
||||
|
||||
## §11. Decision log entry
|
||||
|
||||
If F4-MVP is approved for implementation, log the decision per `feedback_v7_gem_methodology` §decision-log:
|
||||
|
||||
```
|
||||
Decision: Adopt F4-MVP (oracle pretraining with argmax-only realistic-LobSim heuristic oracle) as Phase 7c, sequenced after Phase 7a (F2 committed) and Phase 7b (F5 committed). F4-MVP is the disciplined response per Phase 7 spec §5.3 attribution outcome 3 + §5.4 escalation path when F2 + F5 mechanism gates pass but cluster outcome regresses.
|
||||
Rejected: F4-Full (multi-epoch, full-distribution, encoder-freeze variants — deferred until F4-MVP cluster verdict).
|
||||
Rationale: F2 + F5 + OptA (sk2z9) demonstrated that gradient-axis and choice-set-axis interventions are mechanically operational but cannot bootstrap Q-quality estimates from cold-start random-policy data. F4 attacks the distinct initial-conditions axis: warm-start Q with perfect-foresight realized-return labels BEFORE RL fine-tune begins. This addresses the "encoder cannot bootstrap" hypothesis directly.
|
||||
|
||||
F4 is theoretically grounded in Levine 2020 (Offline RL Survey), Nair 2020 (AWAC), Kumar 2020 (CQL), Kostrikov 2021 (IQL). F4-MVP is a simplified Nair-2020-style warm-start. F4-MVP scope is bounded by §4 (one epoch, argmax-only oracle, fold-1 only, no distillation regularizer, no encoder freeze).
|
||||
|
||||
F4-MVP is implementable with bounded scope (~7-14 days) across 6 sequential phases (7c.0 — 7c.5) with predetermined gates F4-G1 through F4-G9, three of which (F4-G1, F4-G2, F4-G3) gate progression of the pipeline itself. Phase 7c.6 is conditional re-engagement of F5.
|
||||
|
||||
Date: 2026-06-05
|
||||
Falsification trigger (oracle pipeline): F4-G1 per §5.1
|
||||
Falsification trigger (RL fine-tune): F4-G4 + F4-G5 + F4-G6 per §5.2
|
||||
Falsification trigger (outcome): F4-G7 + F4-G9 per §5.3
|
||||
Going-in-circles trigger: §5.4 F4-G9 failure → STOP, no further Bellman-target reshape
|
||||
Pre-registered failure prediction: §5.4 "F4-G9 failure refutes encoder-Q-bootstrap hypothesis"
|
||||
```
|
||||
@@ -0,0 +1,158 @@
|
||||
# Reward–PnL Alignment Restoration Spec
|
||||
|
||||
> **For agentic workers:** This spec defines WHY and WHAT for restoring reward↔pnl alignment as the next intervention, displacing F4 oracle pretraining. It does NOT itself implement the fix; an implementation plan follows after approval.
|
||||
|
||||
**Status:** Draft 1 (2026-06-05)
|
||||
**Displaces (sequencing):** `docs/superpowers/specs/2026-06-05-f4-oracle-pretraining-infrastructure.md` — F4 is GATED behind this spec per the foundation-reshape spec's own §5.4 fork (alignment-branch must be ruled out before encoder-branch).
|
||||
**Predecessors:** `docs/superpowers/specs/2026-06-01-reward-policy-alignment-investigation.md` (diagnosis), `docs/superpowers/specs/2026-06-02-trainer-rollout-buffer-gae.md` (the rollout/GAE scaffold this spec completes), `docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md` (§5.4 escalation fork).
|
||||
**Branch target:** off `7376b1c67` on `ml-alpha-regime-observer` (or a fresh branch once approved).
|
||||
**Estimated wall-clock:** Phase A+B (force-pin + cheap diagnostic gate) ~1-2 days; Phase C (1B-C credit-assignment completion) ~3-6 days IF Phase B passes. Honest range 1-2 days (if Phase B STOPs) to ~8 days (full build + cluster). Less than F4's 7-14 days, and on the **confirmed** lever.
|
||||
|
||||
**Linked pearls (load-bearing):**
|
||||
- `pearl_reward_misalign_blocks_f4_slot753_override` — the measurement + mechanism this spec acts on
|
||||
- `pearl_reward_signal_anti_aligned_with_pnl` — the original diagnosis (4 ADDENDA)
|
||||
- `pearl_advantage_kernel_is_done_gated_td_not_gae` — why aligned-but-sparse reward starves at T_eff=1
|
||||
- `pearl_foxhunt_trainer_is_genuinely_single_step` — why the credit-assignment fix is real work, not a flag flip
|
||||
- `pearl_phase5_term_4_is_almost_potential` — the non-potential shaping terms slot 753 re-enables
|
||||
- `pearl_diag_pnl_fields_are_shaped_reward_not_usd` — `realized_pnl_usd_delta` is the authoritative USD signal the gates use
|
||||
- `pearl_pure_pnl_mode_starves_b16_controllers` — pure-pnl at b<1024 starves controllers (use b=1024 for reward-distribution changes)
|
||||
- `feedback_investigation_first_falsification_methodology` — the discipline; cheapest-informative-step-first
|
||||
|
||||
---
|
||||
|
||||
## §1. Why this, why now
|
||||
|
||||
### §1.1 Three measured findings (2026-06-05)
|
||||
|
||||
1. **Reward is misaligned, measured.** `Pearson(rewards.sum, trading.realized_pnl_usd_delta) = 0.28` on the mid-smoke diag (`/tmp/foxhunt-mid-smoke/diag.jsonl`, b=128 seed=42, 2000 rows). Gate for trustworthy Q learning is ≥ 0.70. Sign-agreement = 0.655 (improved from the original 0.23-0.32; the dominant failure has shifted from sign-flips to **magnitude decoupling** — among worst-5% pnl steps losing $42k-$155k, reward range was [-1509, +591], i.e. some catastrophic-loss steps received positive reward).
|
||||
|
||||
2. **The "pure-pnl" bootstrap is cosmetic — verified against live code.** `integrated.rs:4293` sets `RL_SURFER_SCAFFOLD_WEIGHT_INDEX` (753) bootstrap = 0.0, but `rl_surfer_scaffold_controller.cu:101` writes `isv[753] = w` **unconditionally** with **no force-pin gate** (grep-confirmed absent). Runtime diag shows `surfer_scaffold_weight` pinned at **0.966 → 0.974** across all 2000 rows. All four Phase-5 non-potential shaping terms (`rl_fused_reward_pipeline.cu:344-373`, each gated `w*…`) are therefore live at ~97% strength regardless of the bootstrap.
|
||||
|
||||
3. **Falsification: the value function is NOT the fault — the reward target is.** value-error-vs-policy-error, eval/train median ratios: `loss.v` = 0.91×, `loss.q` = 0.83×, `loss.pi` = 1.05×. The value function generalizes fine (eval error ≈ train, even lower), the policy agrees with Q (`q_pi_agree_ema=0.95`, `q_value_abs_max=4.92` by end of train) — yet eval pnl = -$4.96M, wr 0.317, PF 0.64. *Low value error + high Q-π agreement + negative pnl on both folds* ⇒ the agent **faithfully optimizes a misaligned objective**. This simultaneously falsifies (a) F4 (Q-init is not the fault; Q learns its target fine) and (b) generalization machinery / IDAAC decoupling (value does not overfit). It confirms the **reward target** is the lever.
|
||||
|
||||
### §1.2 Why not F4
|
||||
|
||||
Per `pearl_reward_misalign_blocks_f4_slot753_override` and four independent omnisearch research agents (2026-06-05): F4 would label its oracle Q against this same `r_shaped` (math-identity is load-bearing in the F4 spec §3.1.B), warm-starting Q onto a 0.28-aligned target. F4-G1 cannot detect this (both sides of its Pearson are in shaped units — tautology). External literature is decisive: value-pretraining is an optimization lever, not a generalization lever (IDAAC, Raileanu & Fergus 2021); one-epoch-MSE-then-finetune triggers the WSRL (ICLR 2025) recalibration collapse; argmax-on-realized-path is look-ahead leakage. **Fix the target before warm-starting onto it.**
|
||||
|
||||
### §1.3 The Phase 1A landmine (why this is not a 10-LOC fix)
|
||||
|
||||
Force-pinning slot 753 to 0 gives reward ≈ pnl-delta → Pearson → ~1.0 **trivially** (the metric becomes near-tautological). But pure-pnl ALONE was already tried (Phase 1A, `pearl_reward_signal_anti_aligned_with_pnl` ADDENDUM 2026-06-02d) and **regressed eval −15.5%**, because aligned reward is *sparse* (pnl realizes at close events only) and the advantage kernel is done-gated 1-step TD (`compute_advantage_return.cu:43`, `advantages[b] = is_done ? (ret-vt) : 0`, T_eff=1) — only ~6.75 non-zero advantages per b=128 step. Aligned-but-sparse reward without multi-step credit assignment starves the gradient. **Alignment and credit-assignment are co-required.** This spec sequences a cheap diagnostic gate (Phase B) to confirm the co-fix will work *before* committing to the credit-assignment build (Phase C).
|
||||
|
||||
---
|
||||
|
||||
## §2. The fault mechanism
|
||||
|
||||
```
|
||||
slot 753 bootstrap = 0.0 (cosmetic)
|
||||
↓ rl_surfer_scaffold_controller.cu:101 writes isv[753] = w UNCONDITIONALLY (no pin gate)
|
||||
↓ w = max(1 − competence, w_decay); cold-start n_trades=0 → competence≈0 → w≈0.97 every step
|
||||
↓ rl_fused_reward_pipeline.cu:344-373: 4 non-potential Phase-5 shaping terms live at 0.97
|
||||
↓ r_shaped only 0.28-Pearson-aligned with realized USD pnl (magnitude decoupled)
|
||||
↓ V regresses r_shaped (loss.v fits fine on train AND eval); Q distills from V/Bellman
|
||||
↓ π distills from Q (q_pi_agree 0.95) → π faithfully optimizes the misaligned objective
|
||||
↓ eval pnl negative DESPITE low value error and high Q-π agreement
|
||||
```
|
||||
|
||||
The two independent defects:
|
||||
- **D1 (alignment):** the controller silently re-enables anti-aligned shaping; the 0.0 bootstrap never takes effect at runtime.
|
||||
- **D2 (credit assignment):** even with aligned (sparse) reward, the done-gated 1-step TD advantage kernel cannot propagate close-event pnl back to the opening decision. The GAE rollout scaffold exists (`rollout_buffer.rs`, `rollout.rs`, `rollout_collection.rs`, GAE kernel, commits `5cd2f8703`+`779c03b9d`) but is **collection-only** (Phase 1B-B): training still runs per-step TD; the multi-epoch PPO update over the buffer (Phase 1B-C) was deferred per STOP-on-unexpected-finding.
|
||||
|
||||
---
|
||||
|
||||
## §3. What to build
|
||||
|
||||
### §3.1 Force-pin for slot 753 (D1 fix) — bounded
|
||||
|
||||
A pin gate so the bootstrap value is honored at runtime. Two design options (resolve in the plan):
|
||||
|
||||
| Option | Mechanism | Pros | Cons |
|
||||
|---|---|---|---|
|
||||
| §3.1.A New ISV pin slot | `RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX` (next free slot); controller reads it and `if (pin > 0.5) return;` before writing `isv[753]` | ISV-native; A/B-toggleable; determinism-preserving | one new slot |
|
||||
| §3.1.B Reuse bootstrap-as-pin | controller skips the write when an existing "design constant locked" flag is set | no new slot | conflates two concerns; less explicit |
|
||||
|
||||
**Recommendation:** §3.1.A. Mirror the F2/F5 slot-allocation pattern (`isv_slots.rs`, bootstrap table `integrated.rs:4382-4417`). Bootstrap pin = 0.0 (OFF → bit-equal to HEAD); set to 1.0 only in the alignment experiment config. The pinned *value* of slot 753 is then whatever its bootstrap says (0.0 = pure-pnl, or a chosen partial-shaping weight for ablation).
|
||||
|
||||
### §3.2 Cheap diagnostic gate (decides whether D2 build is justified) — bounded
|
||||
|
||||
With slot 753 pinned (aligned reward) AND `FOXHUNT_USE_ROLLOUT=1` (GAE computed as side-effect, training still TD), add a diagnostic that measures **`Pearson(GAE_advantage, realized_pnl_at_close)`** in the rollout buffer. This answers: *does multi-step GAE credit assignment on the aligned reward produce advantages that align with realized pnl?* — WITHOUT the expensive 1B-C training rearchitecture. This is the load-bearing investigation-first gate.
|
||||
|
||||
### §3.3 Complete Phase 1B-C (D2 fix) — substantive, CONDITIONAL on §3.2 passing
|
||||
|
||||
Multi-epoch PPO update over the GAE rollout buffer, replacing the per-step done-gated TD update path. Per `rollout_collection.rs:24-37` this is where per-step training is split. Scope this in the plan against the existing scaffold; do NOT rebuild the buffer/GAE/collection (already done in 1B-B).
|
||||
|
||||
### §3.4 Re-measure harness
|
||||
|
||||
A reusable script (`scripts/measure_reward_alignment.py`, NOT in root) that computes `Pearson(rewards.sum, realized_pnl_usd_delta)` and sign-agreement from any diag.jsonl, so every smoke reports the alignment number as a first-class gate.
|
||||
|
||||
---
|
||||
|
||||
## §4. Scope boundary
|
||||
|
||||
### §4.1 In scope (MVP)
|
||||
- Force-pin slot 753 (D1).
|
||||
- Diagnostic: GAE-advantage↔pnl Pearson (the gate).
|
||||
- IF gate passes: complete 1B-C multi-epoch PPO over rollout buffer (D2).
|
||||
- Re-measure harness + falsification gates at smoke and cluster.
|
||||
|
||||
### §4.2 Out of scope
|
||||
- F4 oracle pretraining (gated behind this spec; revisit only if aligned reward + GAE still fails eval).
|
||||
- Magnitude-clamp / popart redesign (the secondary finding; separate spec if Pearson stalls after D1+D2).
|
||||
- New reward shaping terms (this spec REMOVES misaligned shaping, adds none).
|
||||
- Generalization machinery (IDAAC decoupling, ensembles, augmentation) — only if D1+D2 succeed and a *residual* train→eval gap remains.
|
||||
- A reward redesign beyond pinning existing weights (e.g. potential-based reshaping) — F4-Full-style; separate spec.
|
||||
|
||||
---
|
||||
|
||||
## §5. Falsification gates
|
||||
|
||||
| Gate | Metric | Pass | When | Falsifies |
|
||||
|---|---|---|---|---|
|
||||
| RA-G1 (pin works) | runtime `surfer_scaffold_weight` with pin=1.0, bootstrap=0.0 | == 0.0 across all rows | Phase A smoke | If fails: pin gate not wired to the controller write |
|
||||
| RA-G2 (determinism) | `determinism-check.sh --quick` seeds 42/43/44 | exit 0, eval_summary bit-equal | Phase A | If fails: pin/diag introduced nondeterminism |
|
||||
| RA-G3 (alignment restored) | `Pearson(rewards.sum, realized_pnl_usd_delta)` with pin=1.0 bootstrap=0.0 | ≥ 0.70 | Phase A smoke | (near-tautological with pure-pnl; confirms the pin removed the shaping) |
|
||||
| **RA-G4 (credit-assignment viability — the load-bearing gate)** | `Pearson(GAE_advantage, realized_pnl_at_close)` in rollout buffer, aligned reward + USE_ROLLOUT=1 | ≥ 0.70 | Phase B smoke | **If fails: GAE on aligned reward still does not align advantages with pnl → STOP; the fault is deeper than credit assignment (reward shape itself / regime). Do NOT build 1B-C.** |
|
||||
| RA-G5 (no eval regression) | smoke eval pnl with D1+D2 | ≥ F2+F5 SOTA −$415,237 (not worse than current) | Phase C smoke | If fails: D2 build did not deliver; diagnose before cluster (recall Phase 1A regressed here) |
|
||||
| RA-G6 (cluster eval) | `eval_summary.total_pnl_usd` at Tier 2 b=1024 20k+5k fold 1 | > F2+F5 sk2z9 cluster baseline | Phase D cluster | If fails AND RA-G4 passed: aligned reward+GAE insufficient; THEN encoder/generalization or F4 is the next fork |
|
||||
|
||||
**Stop-on-fail discipline:**
|
||||
- RA-G4 is the spec's economic linchpin: it costs ~1 day (no training rearchitecture) and decides whether the ~3-6 day Phase C build is justified. **If RA-G4 < 0.70, STOP — do not build 1B-C.** A failure here means even perfect multi-step credit assignment on the aligned reward does not recover pnl-aligned advantages, which would point at deeper reward-shape or regime-shift causes (and would, notably, also predict F4 failure — so it is decisive either way).
|
||||
- RA-G6 failure with RA-G4 pass cleanly isolates the residual to encoder/generalization or genuine OOS regime shift — the foundation-reshape §5.4 "encoder branch," now legitimately reached.
|
||||
|
||||
---
|
||||
|
||||
## §6. Implementation phases
|
||||
|
||||
### Phase A — Force-pin slot 753 (~0.5-1 day)
|
||||
Add pin slot + controller gate + bootstrap-table entry. Smoke with pin=1.0 bootstrap=0.0. **Gates:** RA-G1, RA-G2, RA-G3. **STOP if** RA-G1/G2 fail.
|
||||
|
||||
### Phase B — Credit-assignment viability diagnostic (~0.5-1 day)
|
||||
Add `Pearson(GAE_advantage, realized_pnl_at_close)` diag to the rollout-buffer path. Smoke with pin=1.0 + `FOXHUNT_USE_ROLLOUT=1` (training unchanged). **Gate:** RA-G4. **STOP if** RA-G4 < 0.70 — escalate to reward-shape/regime investigation, NOT 1B-C, NOT F4.
|
||||
|
||||
### Phase C — Complete 1B-C multi-epoch PPO over rollout buffer (~3-6 days) — CONDITIONAL on RA-G4
|
||||
Replace per-step done-gated TD update with multi-epoch PPO over the GAE buffer. Mid-smoke. **Gates:** RA-G5, RA-G2 (determinism preserved). **STOP if** RA-G5 regresses.
|
||||
|
||||
### Phase D — Cluster verify (~80 min + analysis)
|
||||
Tier 2 b=1024, 20k+5k, fold 1, pin=1.0 + USE_ROLLOUT=1 + 1B-C. **Gate:** RA-G6. Branch per §5.
|
||||
|
||||
---
|
||||
|
||||
## §7. Risks
|
||||
|
||||
1. **RA-G3 is near-tautological** — pinning to pure-pnl makes reward≈pnl by construction. That is why RA-G4 (advantage↔pnl, NOT reward↔pnl) is the real gate. Do not over-read RA-G3.
|
||||
2. **Phase 1A precedent** — pure-pnl regressed eval −15.5%. This spec's whole bet is that the regression was a credit-assignment (D2) artifact, not an alignment (D1) artifact. RA-G4 tests that bet cheaply before the Phase C spend. If RA-G4 fails, the bet is wrong and we stop early — cheaply.
|
||||
3. **Magnitude decoupling may persist after D1** — if removing shaping still leaves Pearson < 0.70 because of clamp/popart magnitude crushing (`pearl_popart_blind_to_session_signals`), a clamp/popart fix becomes the next sub-spec. RA-G3 catches this.
|
||||
4. **1B-C cost is the dominant uncertainty** — `pearl_foxhunt_trainer_is_genuinely_single_step` warns multi-step training is real work. The buffer+GAE+collection exist; the multi-epoch PPO update path does not. Plan must scope 1B-C against the existing scaffold and STOP if it balloons beyond the estimate.
|
||||
5. **b<1024 controller starvation** — per `pearl_pure_pnl_mode_starves_b16_controllers`, validate reward-distribution changes at b=1024; treat local b=128 smoke as directional only (`pearl_cluster_log_is_ground_truth`).
|
||||
|
||||
---
|
||||
|
||||
## §8. Decision log entry
|
||||
|
||||
```
|
||||
Decision: Adopt Reward–PnL Alignment Restoration as the next intervention, GATING F4 oracle pretraining behind it. Sequenced: force-pin slot 753 (D1) → cheap GAE-advantage↔pnl diagnostic (RA-G4) → conditional 1B-C credit-assignment completion (D2) → cluster.
|
||||
Rejected (for now): F4 oracle pretraining — falsified as the lever (value generalizes; Q learns its target fine; the fault is the misaligned reward TARGET, not Q-init). External lit (Cal-QL/WSRL/IDAAC, 2023-2025) concurs value-pretrain is the wrong tool here.
|
||||
Rationale: measured Pearson(reward,pnl)=0.28 (gate 0.70); slot-753 controller override makes the pure-pnl bootstrap cosmetic; value-error-vs-policy-error falsification (0.91×/0.83×/1.05× eval-vs-train) shows value generalizes while the agent loses money on both folds — the objective is misaligned, not the estimator.
|
||||
Date: 2026-06-05
|
||||
Falsification trigger (viability): RA-G4 < 0.70 → STOP before 1B-C build.
|
||||
Going-in-circles guard: this is the alignment branch of foundation-reshape §5.4; if D1+D2 pass RA-G4/G5 but RA-G6 fails at cluster, the encoder/generalization branch is legitimately isolated — NOT another reward reshape.
|
||||
```
|
||||
Reference in New Issue
Block a user