From 31f858b5ea7f4437f15569bb9f3cdcba76e605d2 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Fri, 5 Jun 2026 12:59:19 +0200 Subject: [PATCH] docs(superpowers): edge investigation decision record + F4-gating/reward-alignment specs & plans MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Session 2026-06-05: investigation-first falsification gated the F4 oracle build (7-14 days) and walked back through the reward/edge question to a code-verified root cause — the MBP-10 decoder corrupts the inside quote (dbn_parser.rs:903 'for lvl in 1..max_lvl' never restores levels[0]), contaminating mid/feature/reward/fill locally and on the cluster PVC. - notes/2026-06-05-edge-investigation-decision-record.md: full 8-verdict arc (incl. 2 self-retractions) + next-session plan (confirm on PVC, fix parser, regenerate sidecars, re-audit on clean data). - specs: f4-oracle-pretraining (GATED), reward-pnl-alignment-restoration (superseded by the data-quality finding), bellman-target-foundation-reshape. - plans: reward-pnl-alignment-restoration-implementation, f2-q-centered-distill. Co-Authored-By: Claude Opus 4.8 (1M context) --- ...6-05-edge-investigation-decision-record.md | 59 + .../notes/2026-06-05-next-session-prompt.md | 40 + ...04-f2-q-centered-distill-implementation.md | 1946 +++++++++++++++++ ...nl-alignment-restoration-implementation.md | 465 ++++ ...06-04-bellman-target-foundation-reshape.md | 1004 +++++++++ ...05-f4-oracle-pretraining-infrastructure.md | 733 +++++++ ...-06-05-reward-pnl-alignment-restoration.md | 158 ++ 7 files changed, 4405 insertions(+) create mode 100644 docs/superpowers/notes/2026-06-05-edge-investigation-decision-record.md create mode 100644 docs/superpowers/notes/2026-06-05-next-session-prompt.md create mode 100644 docs/superpowers/plans/2026-06-04-f2-q-centered-distill-implementation.md create mode 100644 docs/superpowers/plans/2026-06-05-reward-pnl-alignment-restoration-implementation.md create mode 100644 docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md create mode 100644 docs/superpowers/specs/2026-06-05-f4-oracle-pretraining-infrastructure.md create mode 100644 docs/superpowers/specs/2026-06-05-reward-pnl-alignment-restoration.md diff --git a/docs/superpowers/notes/2026-06-05-edge-investigation-decision-record.md b/docs/superpowers/notes/2026-06-05-edge-investigation-decision-record.md new file mode 100644 index 000000000..ac7fe1893 --- /dev/null +++ b/docs/superpowers/notes/2026-06-05-edge-investigation-decision-record.md @@ -0,0 +1,59 @@ +# Decision Record — Edge Investigation & MBP-10 Root Cause (2026-06-05) + +**Branch:** `ml-alpha-regime-observer` +**Outcome in one line:** Started to build F4 oracle pretraining (7-14 days); instead, investigation-first falsification gated F4, walked back through the reward/edge question, and **found a code-verified root-cause bug — the MBP-10 decoder corrupts the inside quote (`levels[0]`) — that contaminates every mid/feature/reward/fill, locally AND on the cluster PVC.** + +--- + +## The arc (each step a falsifiable verdict) + +1. **F4 oracle pretraining — GATED, not built.** Fresh-eyes research (4 parallel omnisearch agents) + a 1-hour measurement showed F4 would warm-start Q onto a reward only **Pearson 0.28** aligned with USD pnl (gate 0.70). Root cause of that: slot-753 (`surfer_scaffold_weight`) 0.0 "pure-pnl" bootstrap is **cosmetic** — the controller (`rl_surfer_scaffold_controller.cu:101`) overwrites it to ~0.97 every step (no force-pin gate). External literature (Cal-QL/WSRL/IDAAC, 2023-2025): value-pretraining is an optimization, not a generalization, lever. → Pearl: `pearl_reward_misalign_blocks_f4_slot753_override`. + +2. **Falsification: value generalizes, agent loses on both folds.** `loss.v`/`loss.q` eval/train ratios 0.91×/0.83× (value does NOT overfit), yet eval pnl deeply negative. → reward *target*, not Q-init, was the suspected lever. + +3. **Phase A force-pin (committed) — works, but reveals more.** Added slot 824 `RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX` + `FOXHUNT_PIN_SURFER_SCAFFOLD` env flag + kernel gate (commit `d8447475c`). RA-G1 ✓ (slot 753 held 0.0), RA-G2 ✓ (determinism). But pinning to pure-pnl **collapsed trading** (entropy 0.015, eval 0 trades) — a b=128 popart-starvation confound, disambiguated by enabling popart-normalize (trading recovered to entropy 1.87, 1,278 eval trades) — **yet still −$3.57M eval.** A *healthy* policy on aligned-intent reward still loses → reward alignment weakened as the lever. → Pearls: `pearl_shaping_is_dual_purpose_align_vs_explore`. + +4. **Data-edge audit (short horizons) — no tradeable edge.** Simple non-RL predictor, train Q1→eval Q2: the only OOS-persistent signal is the bid-ask bounce, untradeable net of spread. → `pearl_data_edge_audit_no_tradeable_oos_edge_local`. + +5. **LobSim cost audit — sim is HONEST.** Book-walk fills cross the spread (enter ask / exit bid); MTM-at-mid telescopes to ~0 per closed trade; commission 0 (minor). Eval loss is genuine. → refuted the "free bounce" hypothesis (addendum to the pearl above). + +6. **Long-horizon edge audit — "edge exists!" (FALSE alarm).** On *mid prices*, a ~6hr mean-reversion looked tradeable (OOS t=5.06). → `pearl_edge_exists_at_multihour_mean_reversion` (now RETRACTED). + +7. **Rule-based baseline with REAL fills — RETRACTS #6.** Enter ask / exit bid: loses in every θ/H cell OOS. The "edge" was **phantom reversion into degraded quotes** (`corr(|z|,spread)=0.91`; 99.8% of signals fire on spread>5pt). **Lesson: never trust mid/abstract-cost backtests without bid/ask fill validation.** → `pearl_local_data_degraded_edge_was_quote_artifact`. + +8. **Degradation source — ROOT CAUSE (code-verified).** The degraded top-of-book is a **decoder bug**, not thin liquidity (no time-clustering; 89% single-snapshot spikes; 6.4% crossed; L1 strictly inside L0 in 100% of wide cases; reconstruction from L1 is clean). → `pearl_mbp10_level0_decode_bug_root_cause`. + +--- + +## The root cause (the deliverable) + +`crates/data/src/providers/databento/dbn_parser.rs`: +- **L876-883** writes the single MBP-10 *event's* `(price,size)` into `levels[0]` — but an event is a delta to one level at depth `mbp10.depth`, not the inside. +- **L902-903** the "Phase E.1 fix" copies the real book `mbp10.levels[1..]` but the loop **starts at 1**, never overwriting the corrupt L0 with the canonical `mbp10.levels[0]`. + +**Blast radius:** the same parser decodes the 148GB cluster PVC. Corrupts mid, microprice, spread, L0-OFI features, the Phase-5 mid-based MTM reward, and the LOB-sim `levels[0]` fill reference — i.e. the agent has been training and trading on phantom inside quotes (~10% wide, 6.4% crossed). Falsifies the old "cluster data is fine" claim. + +--- + +## Decision & next action + +**Adopt the MBP-10 L0 decode fix as the next work, ahead of any RL/reward/architecture effort.** Every prior intervention (F4, F2, F5, reward-alignment, 64 commits) operated on a corrupt data substrate; fixing it is the prerequisite for any trustworthy result. + +Plan (next session): +1. **Confirm on PVC first** — run the diagnostic dump (crossed %, L1-inside-L0 %, spread p50/p99) on a cluster PVC ES file to verify the corruption is present there too (inferred from shared code, not yet measured). +2. **Fix** `dbn_parser.rs`: line 903 `1..max_lvl` → `0..max_lvl`; drop the `update_level(0,...)` at 876-883. Add a unit test asserting no crossed/locked L0 on a known record set. (TDD; ~½ day code.) +3. **Regenerate predecoded sidecars** — the cache validates by SOURCE mtime/size (`ml-features/src/predecoded.rs`), so a parser change will NOT auto-invalidate. Delete + regenerate local `test_data` and cluster PVC sidecars (148GB — the heavy step; Argo job). +4. **Re-validate data** (spread p50≈1 tick, crossed≈0), then **re-run the edge audit on clean data** (short AND long horizons, with real fills), then re-assess whether RL training has a real substrate. + +--- + +## Artifacts & state + +- **Committed (branch `ml-alpha-regime-observer`):** Phase A force-pin (slot 824, `d8447475c`); `scripts/measure_reward_alignment.py` (`e57b07657`). +- **Reverted:** the `FOXHUNT_POPART_NORMALIZE` disambiguation scaffold in `integrated.rs` (its finding is in `pearl_shaping_is_dual_purpose_align_vs_explore`). Re-add if testing popart on clean data. +- **Specs/plans (uncommitted, this session):** `docs/superpowers/specs/2026-06-05-f4-oracle-pretraining-infrastructure.md` (GATED), `docs/superpowers/specs/2026-06-05-reward-pnl-alignment-restoration.md` (superseded by the data-quality finding), `docs/superpowers/plans/2026-06-05-reward-pnl-alignment-restoration-implementation.md`. +- **Memory pearls added:** reward-misalign-blocks-f4 / shaping-dual-purpose / data-edge-audit / edge-exists (retracted) / local-data-degraded / **mbp10-level0-decode-bug-root-cause**. +- **Diagnostic data:** `/tmp/diag_q1.csv`, `/tmp/edge*_*.csv`, `/tmp/bt_*.csv`. + +## Methodology note +This session is a clean demonstration of `feedback_investigation_first_falsification_methodology`: spec→measure→falsify→STOP-on-surprise, applied through **8 sequential verdicts including 2 self-retractions** (F4 gating; the "edge exists" reversal). It converted a planned 2-week F4 build into the discovery of a ~2-line root-cause bug. Cost: a handful of ~10-min local smokes + parallel research/dump agents. No cluster spend. diff --git a/docs/superpowers/notes/2026-06-05-next-session-prompt.md b/docs/superpowers/notes/2026-06-05-next-session-prompt.md new file mode 100644 index 000000000..437d2b7ca --- /dev/null +++ b/docs/superpowers/notes/2026-06-05-next-session-prompt.md @@ -0,0 +1,40 @@ +# Next session resume prompt + +Copy-paste this into a new Claude Code session to pick up Phase 7c F4 oracle pretrain work: + +--- + +Resume foxhunt Phase 7c F4 oracle pretraining work. + +Branch: `ml-alpha-regime-observer` at HEAD `7376b1c67` (6 commits ahead; pushed). F2 (slot 817) + F5 (slot 823) + Option A gate-patches engaged. F2+F5+OptA = SOTA local at eval_pnl=-$415,237 (best-of-session, +$29k vs Phase 5). Cluster sk2z9 bled 3× faster than F2-alone with 6:1 Short:Long directional bias → killed at step 519 → F4 oracle pretrain is the escalation per spec §5.3 outcome 3. + +**Read these memory pearls first** (in this order): +1. `project_phase7_state_2026_06_05.md` — full Phase 7 state, branch + slots + verdicts +2. `pearl_f2_q_centered_distill_necessary_not_sufficient.md` — why F2 alone failed +3. `pearl_f5_state_mask_neutralized_by_downstream_gates.md` — F5 gate-interop pattern +4. `pearl_f2_plus_f5_unlock_q_signal.md` — why combined helps; cluster bleed +5. `reference_outcome_head_supervised_pattern.md` — `outcome_head.rs` is the F4 Phase 7c.2 precedent +6. `feedback_investigation_first_falsification_methodology.md` — methodology that broke 64-commit going-in-circles + +**Read this spec next**: +- `docs/superpowers/specs/2026-06-05-f4-oracle-pretraining-infrastructure.md` (733 lines, 6 phases 7c.0-7c.5, ~7-14 days bounded) + +**Next step**: write the F4 implementation plan at `docs/superpowers/plans/2026-06-05-f4-oracle-pretraining-implementation.md` (similar 1500-LOC structure to F2 plan), then dispatch Phase 7c.0 oracle pipeline coder (2-3 days). + +**Top 3 risks per F4 spec §7**: +1. Oracle heuristic itself wrong → F4-G1 Pearson < 0.7 → STOP at Phase 7c.0 +2. Reward function identity drift → MUST use realistic-Rust+CUDA oracle (NOT Python) +3. Q drift undoes warm-start → F4-G4/G6 are detection gates + +**Key code references**: +- F2 kernel: `crates/ml-alpha/cuda/rl_q_pi_distill_grad.cu` Step 2 hinge (lines ~101-150) +- F5 kernel: `crates/ml-alpha/cuda/rl_state_action_mask.cu` +- OutcomeHead pattern: `crates/ml-alpha/src/rl/outcome_head.rs` +- F2 plan template: `docs/superpowers/plans/2026-06-04-f2-q-centered-distill-implementation.md` + +**Open questions for the plan** (resolve before dispatching coder): +- Reward function versioning hash protocol (Phase 5 slots + Phase 7a/7b slots + γ + K → filename embedding) +- Held-out subset for F4-G3 (last 10% of fold-1 train OR fold-1 eval discarded) +- Encoder LR for pretraining (same as RL fine-tune for MVP simplicity OR separate higher LR) + +Proceed with writing the F4 implementation plan, then begin Phase 7c.0. diff --git a/docs/superpowers/plans/2026-06-04-f2-q-centered-distill-implementation.md b/docs/superpowers/plans/2026-06-04-f2-q-centered-distill-implementation.md new file mode 100644 index 000000000..8fb564f74 --- /dev/null +++ b/docs/superpowers/plans/2026-06-04-f2-q-centered-distill-implementation.md @@ -0,0 +1,1946 @@ +# F2 Q-Centered Q-Distill Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Modify the Q-distill target kernel to use a hinged advantage formulation that preserves the surfer trading principle while breaking the structural inaction-bias documented in pearl_dd049d9a4_surfer_baseline_verified + the going-in-circles cluster trajectory. + +**Architecture:** Replace `π_target = softmax(E_Q/τ)` with `π_target = softmax(max(0, E_Q − baseline(no-op-set))/τ)` where `baseline = max(E_Q[Hold], E_Q[TrailTighten], E_Q[TrailLoosen])`. Mass-zeroes any action no better than the best zero-cost no-op; preserves Hold's mass when no Open action has strictly positive advantage (surfer-patience-preserving). + +**Tech Stack:** CUDA kernel modification (`rl_q_pi_distill_grad.cu`), Rust trainer integration (`integrated.rs`), 2 new ISV slots, GPU-oracle invariant test, bit-deterministic verification. + +**Diagnostic instrumentation (NEW — Task 1.5):** 13 new diag emissions covering (a) state-conditional surfer-principle measurement (`hold_frac_flat` etc. — makes F2-G5 locally measurable), (b) degenerate-attractor monitoring (modal action share, no-op mass concentration), and (c) F2 mechanism counters (baseline mean, advantage max mean, hinge zero rate, target argmax). Without these, F2's effect at b=1024 cluster is unattributable per `pearl_fleet_fraction_not_aggregate` and the spec's measure-first discipline (`feedback_investigation_first_falsification_methodology`). + +--- + +## Reconnaissance findings (verified before plan authoring — Task 0 below documents them) + +These are findings from the spec-mandated reconnaissance pass. They are recorded here so the implementer does NOT need to re-verify (but Task 0 confirms each one fresh). + +1. **c51_v_max state (per `pearl_c51_v_max_freeze_required_for_surfer`).** Bootstrap is `RL_C51_V_MAX_INDEX = 1.0` at `crates/ml-alpha/src/trainer/integrated.rs:3830`. Surfer mode requires this frozen at 1.0 OR within [50, 250]. Grepping the codebase, no controller writes to `RL_C51_V_MAX_INDEX` during the training loop — the only writes are the bootstrap and a manual atom-bound widener (`rl_reward_clamp_controller`) which is bounded; the EWMA only grows magnitude (so v_max can rise above 1.0 over many steps if rewards exceed it). For Phase 7a F2's purposes the slot is in the safe state. **NO ACTION REQUIRED** — proceed with default bootstrap. + +2. **F2-band composition order (load-bearing per spec §7 #2).** The Q-distill kernel reads `pi_logits_d` (forward pre-softmax logits) and `q_logits_target_d` (TARGET net Q-logits) at `integrated.rs:6118-6138`. It writes per-action gradient into `ss_pi_grad_logits_d`. It does NOT read `actions_d`. The band mask (`band_head.launch_mask` at `integrated.rs:9924-9932`) modifies `actions_d` AFTER the policy has sampled an action — purely a post-sampling action-override path. Therefore F2 (which lives entirely in Q-space and reshapes the distill target distribution) is **mechanically disjoint from the band mask**. F2 sees the pre-band, pre-sampling Q-distribution. **VERIFIED CLEAN COMPOSITION** — F2 + band compose without interaction. + +3. **No-op set definition.** Action enum at `crates/ml-alpha/cuda/actions_to_market_targets.cu:55-65`. The three zero-cost actions per spec §3.5 are `ACTION_HOLD=2`, `ACTION_TRAIL_TIGHTEN=7`, `ACTION_TRAIL_LOOSEN=8`. These align with the spec's F2a static set. Phase 7a uses these three as the hardcoded no-op set with a mode slot for future variants. + +4. **Existing slot allocation.** `RL_SLOTS_END = 817` at `crates/ml-alpha/src/rl/isv_slots.rs:2281`. New slots are 817 and 818; new `RL_SLOTS_END = 819`. + +5. **Existing distill controller.** `RL_Q_DISTILL_LAMBDA_INDEX = 486` (bootstrap 0.01), `RL_Q_DISTILL_TEMPERATURE_INDEX = 487` (bootstrap 5.0), and the SAC α + τ auto-tuner is co-located in the same kernel. The controller is tuned for the *current* target shape. Per spec §9.1 the controller may need re-bootstrapping, but Phase 7a leaves the bootstrap values unchanged — the τ controller will auto-adapt to the new target shape via the existing entropy-deficit feedback loop. If F2-G3 fails (training unstable) the controller re-bootstrap is Task 6's STOP-on-surprise escalation path. + +--- + +## Task 0 — Reconnaissance verification (~30 min) + +Confirm the four reconnaissance findings above against the live codebase. If any finding diverges, STOP and surface — do not proceed. + +### Step 0.1 — Verify c51_v_max bootstrap state + +- [ ] Run: + +```bash +grep -n "RL_C51_V_MAX_INDEX" /home/jgrusewski/Work/foxhunt/crates/ml-alpha/src/trainer/integrated.rs +``` + +Expected output: a bootstrap entry `(crate::rl::isv_slots::RL_C51_V_MAX_INDEX, 1.0)` at approximately line 3830. No write inside the training step loop (`step_with_lobsim_gpu_body` or surrounding controllers). + +- [ ] If the bootstrap value is NOT 1.0 or a controller is found writing to this slot during training, STOP and consult `pearl_c51_v_max_freeze_required_for_surfer` before proceeding. + +### Step 0.2 — Read the target kernel in full + +- [ ] Open `/home/jgrusewski/Work/foxhunt/crates/ml-alpha/cuda/rl_q_pi_distill_grad.cu` and read every line. Specifically confirm: + - Line 17: `#define N_ACTIONS 11` + - Lines 78–91: the `s_pi_target` computation block (the F2 modification site) + - Line 175: `isv[RL_Q_DISTILL_TARGET_ENTROPY_MEAN_INDEX] = h_t_b10;` — the diag emit we will use for F2-G1 +- [ ] If any of these landmarks have moved, adjust line references in Task 3 accordingly (the structural change does not depend on exact line numbers but the patch context does). + +### Step 0.3 — Verify F2-band composition order + +- [ ] Run: + +```bash +grep -n "rl_q_pi_distill_grad_fn\|launch_mask\b" /home/jgrusewski/Work/foxhunt/crates/ml-alpha/src/trainer/integrated.rs | head -20 +``` + +Expected: `rl_q_pi_distill_grad_fn` launches at approximately line 6132 (inside the training-update backward pass), and `band_head.launch_mask` launches at approximately line 9924 (after the policy has sampled `actions_d`). + +- [ ] Open `integrated.rs` at the line of the `rl_q_pi_distill_grad_fn` launch and confirm its 5 argument pointers (`q_logits_target`, `pi_logits`, `atom_supports`, `isv`, `pi_grad_logits`). NONE of them is `actions_d` or any band-mask output. +- [ ] Open `integrated.rs` at the line of `band_head.launch_mask` and confirm its first argument is `&self.actions_d` — the band modifies sampled actions, not logits or Q-distill targets. +- [ ] Conclude: F2 sees Q-target distribution; band overrides sampled action. Disjoint pipelines. F2 is unaffected by the band. + +### Step 0.4 — Document reconnaissance result + +- [ ] If all three checks pass, proceed to Task 1. The plan's reconnaissance findings section is now confirmed inline; no separate report file is needed. +- [ ] If any check surfaces an unexpected divergence (e.g., a new controller has been added to write c51_v_max, or the distill kernel signature has changed), STOP per `feedback_investigation_first_falsification_methodology` and update the plan before any kernel modification. + +--- + +## Task 1 — Allocate two new ISV slots and bootstrap them (~20 min) + +### Step 1.1 — Add slot constants + +- [ ] Open `/home/jgrusewski/Work/foxhunt/crates/ml-alpha/src/rl/isv_slots.rs`. +- [ ] Locate the trailing `pub const RL_SLOTS_END: usize = 817;` declaration (line ~2281) and the preceding comment block. +- [ ] Insert the two new slot constants immediately BEFORE the `RL_SLOTS_END` declaration. Exact text to append after the MTM weight slot: + +```rust +/// Phase 7a F2 — Q-centered distill target master gate. When `> 0.5`, +/// the kernel `rl_q_pi_distill_grad.cu` computes +/// `π_target = softmax(max(0, E_Q − baseline(no-op-set)) / τ)` where +/// `baseline` is the max E_Q over the no-op set selected by +/// `RL_F2_DISTILL_NOOP_SET_MODE_INDEX`. When ≤ 0.5 (default OFF), the +/// legacy `π_target = softmax(E_Q / τ)` path runs unchanged +/// (bit-equality with HEAD `0463e44e0`). +/// +/// Bootstrap 1.0 (ON — Phase 7a deploys the F2 hinge by default). +/// Clamp `{0.0, 1.0}` (binary). +pub const RL_F2_DISTILL_CENTERED_ENABLED_INDEX: usize = 817; + +/// Phase 7a F2 — no-op set mode. Selects which action subset is treated +/// as the inaction baseline for the F2 hinge. +/// +/// Mode 0 (default, "F2a static"): `{Hold=2, TrailTighten=7, TrailLoosen=8}`. +/// Mode 1 ("Hold-only"): `{Hold=2}` — for ablation. +/// Other values reserved for future modes (e.g., dynamic Δlots-based). +/// +/// Bootstrap 0.0 (mode 0 = static three-way set, surfer-principle-preserving). +/// Clamp `[0.0, 2.0]` (integer-valued via host enum). +pub const RL_F2_DISTILL_NOOP_SET_MODE_INDEX: usize = 818; +``` + +### Step 1.2 — Bump RL_SLOTS_END + +- [ ] In the same file, update the trailing constant and its comment ledger. Change: + +```rust +/// Post-Phase 5 (trail-max ceiling + MTM reward, slots 814-816): 816. +pub const RL_SLOTS_END: usize = 817; +``` + +to: + +```rust +/// Post-Phase 5 (trail-max ceiling + MTM reward, slots 814-816): 816. +/// Post-Phase 7a F2 (Q-centered distill target, slots 817-818): 818. +pub const RL_SLOTS_END: usize = 819; +``` + +### Step 1.3 — Add bootstrap entries in the trainer + +- [ ] Open `/home/jgrusewski/Work/foxhunt/crates/ml-alpha/src/trainer/integrated.rs`. +- [ ] Locate the Phase 5 MTM bootstrap entries — search: + +```bash +grep -n "RL_MTM_REWARD_ENABLED_INDEX,\s*1.0\|RL_MTM_REWARD_WEIGHT_INDEX" /home/jgrusewski/Work/foxhunt/crates/ml-alpha/src/trainer/integrated.rs +``` + +There will be two bootstrap entries `(RL_MTM_REWARD_ENABLED_INDEX, 1.0)` and `(RL_MTM_REWARD_WEIGHT_INDEX, 1.0)` inside the long initial-bootstrap tuple list (the same list that contains the c51_v_max bootstrap at line 3830). + +- [ ] Immediately after the MTM weight bootstrap entry, add: + +```rust + // Phase 7a F2 (2026-06-04) — Q-centered distill target. + // ENABLED bootstraps to 1.0 (ON) to deploy the hinge in + // every training run by default; the master gate is the + // ISV slot so flipping to 0.0 reproduces the legacy + // `π_target = softmax(E_Q / τ)` path bit-equal. + // NOOP_SET_MODE bootstraps to 0.0 (mode 0 = F2a static + // set Hold/TrailTighten/TrailLoosen). + (crate::rl::isv_slots::RL_F2_DISTILL_CENTERED_ENABLED_INDEX, 1.0), + (crate::rl::isv_slots::RL_F2_DISTILL_NOOP_SET_MODE_INDEX, 0.0), +``` + +### Step 1.4 — Verify the slot import block + +- [ ] At the top of `integrated.rs`, locate the `use crate::rl::isv_slots::{ ... };` import block (search for `RL_MTM_REWARD_ENABLED_INDEX,` to find it). +- [ ] If the bootstrap tuples reference `crate::rl::isv_slots::RL_F2_DISTILL_CENTERED_ENABLED_INDEX` via the fully-qualified path (as in Step 1.3 above), no import change is needed. Skip this step. +- [ ] Otherwise add `RL_F2_DISTILL_CENTERED_ENABLED_INDEX, RL_F2_DISTILL_NOOP_SET_MODE_INDEX,` to the import block. + +### Step 1.5 — Build and verify + +- [ ] Run: + +```bash +SQLX_OFFLINE=true cargo check -p ml-alpha 2>&1 | tail -20 +``` + +Expected: clean compile (warnings OK; no errors). If a duplicate-slot error appears (e.g., slot 817 already in use), STOP and re-grep `pub const RL_.*= 817\|pub const RL_.*= 818` to find the collision. + +### Step 1.6 — Commit + +- [ ] Stage and commit: + +```bash +git add /home/jgrusewski/Work/foxhunt/crates/ml-alpha/src/rl/isv_slots.rs \ + /home/jgrusewski/Work/foxhunt/crates/ml-alpha/src/trainer/integrated.rs +git commit -m "$(cat <<'EOF' +feat(ml-alpha): Phase 7a F2 — allocate ISV slots 817-818 for Q-centered distill target + +Two new slots: +- RL_F2_DISTILL_CENTERED_ENABLED_INDEX (817, bootstrap 1.0): master gate + for the F2 hinge in rl_q_pi_distill_grad.cu. Flip to 0.0 to fall back to + the legacy softmax(E_Q/τ) target (bit-equal with HEAD 0463e44e0). +- RL_F2_DISTILL_NOOP_SET_MODE_INDEX (818, bootstrap 0.0): selects the + no-op-set definition. Mode 0 = F2a static {Hold, TrailTighten, TrailLoosen}. + +RL_SLOTS_END: 817 → 819. + +Per spec docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md +§3.2 + §4.2 (F2 recommended, Revision 3 surfer-principle-preserving primary phase). + +Co-Authored-By: Claude Opus 4.7 +EOF +)" +``` + +--- + +## Task 1.5 — Comprehensive diagnostic emissions for F2 measurement (~3-4 h) + +**Why this task exists:** Without instrumentation, the F2 cluster verdict (Task 8) cannot distinguish "F2 mechanism worked AND surfer preserved AND eval improved" from "F2 mechanism worked BUT surfer collapsed" from "F2 mechanism inactive (no above-baseline actions emerged)." Per `pearl_fleet_fraction_not_aggregate`, at b=1024 aggregate diag (`worst_X` / `max_X`) hides per-batch reality — fleet-fraction counters are mandatory. Per `feedback_investigation_first_falsification_methodology` "measure first, change code second," the surfer-preservation gate F2-G5 must be locally measurable BEFORE cluster submission. + +This task adds **13 new diag fields** under existing top-level keys (`trading.*` and `policy_diagnostic.*`) so the F2-G5 gate in Task 6 is no longer a "measurement gap" deferred to cluster, and so the cluster verdict in Task 8 receives three independent attribution signals (mechanism / surfer / attractor). + +**The 13 diag fields:** + +| Group | Field | Source | Range | Purpose | +|---|---|---|---|---| +| Surfer | `trading.frac_batches_flat` | host reduce on `inputs.position_lots` | [0,1] | Denominator for state-conditional metrics | +| Surfer | `trading.frac_batches_long` | host reduce | [0,1] | Long-state fraction | +| Surfer | `trading.frac_batches_short` | host reduce | [0,1] | Short-state fraction | +| Surfer | `policy_diagnostic.hold_frac_flat` | host reduce on `inputs.actions` + `inputs.position_lots` | [0,1] | **F2-G5 direct measurement** | +| Surfer | `policy_diagnostic.open_action_mass_flat` | host reduce | [0,1] | Open mass / batches-with-flat | +| Attractor | `policy_diagnostic.max_action_share` | host reduce on `act_hist` | [0,1] | Detects single-action collapse | +| Attractor | `policy_diagnostic.modal_action_id` | host reduce on `act_hist` | {0..10} | Which attractor | +| Attractor | `policy_diagnostic.noop_action_mass_total` | host reduce on `act_hist` | [0,1] | Aggregate no-op rate | +| Attractor | `policy_diagnostic.noop_action_mass_max` | host reduce on `act_hist` | [0,1] | Specific no-op concentration | +| F2 mech | `policy_diagnostic.f2_baseline_mean` | kernel reduce → ISV | finite | mean of `max E_Q[no-op-set]` | +| F2 mech | `policy_diagnostic.f2_advantage_max_mean` | kernel reduce → ISV | ≥ 0 | mean of `max_a hinged[a]` | +| F2 mech | `policy_diagnostic.f2_hinge_zero_rate` | kernel reduce → ISV | [0,1] | Fraction of `(b,a)` zeroed by hinge | +| F2 mech | `policy_diagnostic.f2_target_argmax_action` | kernel reduce → ISV | {0..10} | Modal π_target action | + +**Discipline constraints:** +- **No atomicAdd.** Use the per-batch fused-stats pattern (one block per batch, intra-block reduction) — same shape as the existing band kernel + CMDP fleet-fraction counters. +- **Fleet-fraction over means** at b≥1024 (`pearl_fleet_fraction_not_aggregate`). +- **TDD per diagnostic.** Each new field gets a constructed-scenario test verifying the emit is correct. Diagnostics that lie are worse than no diagnostics. +- **Determinism preservation** (`pearl_determinism_achieved`). The new kernel reductions and the host-side host-reduce must not introduce non-determinism. Determinism-check.sh must pass at Step 1.5.7. + +### Step 1.5.1 — Add ISV slots for F2 mechanism counters + diag-emit test scaffold + +The 4 F2-mechanism diagnostics (`f2_baseline_mean`, `f2_advantage_max_mean`, `f2_hinge_zero_rate`, `f2_target_argmax_action`) need device-side accumulators surfaced via ISV. Allocate 4 more slots (819-822); the 9 host-derived diagnostics need no new ISV slots. + +- [ ] Open `/home/jgrusewski/Work/foxhunt/crates/ml-alpha/src/rl/isv_slots.rs`. +- [ ] Insert (immediately AFTER the two F2 slot constants from Task 1 Step 1.1, BEFORE `RL_SLOTS_END`): + +```rust +/// Phase 7a F2-diag (2026-06-04) — mean of the per-batch +/// `baseline = max E_Q over no-op set` written by +/// `rl_q_pi_distill_grad.cu` at the end of each kernel invocation. +/// Pure diagnostic; no controller reads it. +/// +/// Bootstrap 0.0. Clamp finite. +pub const RL_F2_DIAG_BASELINE_MEAN_INDEX: usize = 819; + +/// Phase 7a F2-diag (2026-06-04) — mean of the per-batch +/// `max_a hinged[a] = max_a max(0, E_Q[a] − baseline)`. Indicates how +/// much "edge" the hinge has surfaced per batch. Healthy regime is +/// non-zero (somewhere between 1e-3 and atom-scale). +/// +/// Bootstrap 0.0. Clamp finite ≥ 0. +pub const RL_F2_DIAG_ADV_MAX_MEAN_INDEX: usize = 820; + +/// Phase 7a F2-diag (2026-06-04) — fraction of `(batch × action)` +/// slots where the hinge `max(0, E_Q[a] − baseline)` clamped to zero. +/// Discriminative power of the hinge: ~1.0 = vacuous (everything below +/// baseline), ~0.0 = no clamping (every action above baseline). +/// +/// Bootstrap 0.0. Clamp [0.0, 1.0]. +pub const RL_F2_DIAG_HINGE_ZERO_RATE_INDEX: usize = 821; + +/// Phase 7a F2-diag (2026-06-04) — modal action id of `π_target` over +/// the batch. The kernel reduces per-batch argmax-of-π_target into a +/// length-N_ACTIONS histogram (via the fused-stats pattern) and writes +/// the argmax of THAT histogram into the ISV. Used to compare against +/// `policy_diagnostic.modal_action_id` (sampled action histogram) — if +/// F2 is reshaping the gradient, the two SHOULD differ (target points +/// toward Open while sampling still dominated by Hold). +/// +/// Bootstrap -1.0 (sentinel "no batch reduced yet"). Clamp finite. +pub const RL_F2_DIAG_TARGET_ARGMAX_INDEX: usize = 822; +``` + +- [ ] Update the `RL_SLOTS_END` declaration (the one already bumped to 819 in Task 1 Step 1.2): + +```rust +/// Post-Phase 5 (trail-max ceiling + MTM reward, slots 814-816): 816. +/// Post-Phase 7a F2 (Q-centered distill target, slots 817-818): 818. +/// Post-Phase 7a F2-diag (kernel-side counters, slots 819-822): 822. +pub const RL_SLOTS_END: usize = 823; +``` + +- [ ] Bootstrap the 4 new slots in `integrated.rs` (immediately after the two F2 bootstrap entries added in Task 1 Step 1.3): + +```rust + // Phase 7a F2-diag (2026-06-04) — kernel-side mechanism + // counters. Sentinel for target_argmax = -1.0 ("no + // batch reduced yet"); the kernel overwrites at step 1. + (crate::rl::isv_slots::RL_F2_DIAG_BASELINE_MEAN_INDEX, 0.0), + (crate::rl::isv_slots::RL_F2_DIAG_ADV_MAX_MEAN_INDEX, 0.0), + (crate::rl::isv_slots::RL_F2_DIAG_HINGE_ZERO_RATE_INDEX, 0.0), + (crate::rl::isv_slots::RL_F2_DIAG_TARGET_ARGMAX_INDEX, -1.0), +``` + +### Step 1.5.2 — Bump EXPECTED_LEAVES and add the failing diag-schema test + +- [ ] Open `/home/jgrusewski/Work/foxhunt/crates/ml-alpha/tests/eval_diag_emission.rs`. +- [ ] Locate the `const EXPECTED_LEAVES: usize = 712;` declaration (line ~273). +- [ ] Update the comment ledger immediately above and bump the constant: + +```rust + // 2026-06-04 Phase 7a F2 diagnostic instrumentation: +13 leaves total. + // trading.frac_batches_flat / frac_batches_long / frac_batches_short + // (3 host-derived state-conditional denominators) + // policy_diagnostic.hold_frac_flat (F2-G5 direct measurement) + // policy_diagnostic.open_action_mass_flat + // policy_diagnostic.max_action_share / modal_action_id / + // noop_action_mass_total / noop_action_mass_max (4 attractor metrics) + // policy_diagnostic.f2_baseline_mean / f2_advantage_max_mean / + // f2_hinge_zero_rate / f2_target_argmax_action (4 mechanism counters, + // kernel-reduced into ISV slots 819-822). + // New total: 712 + 13 = 725. + const EXPECTED_LEAVES: usize = 725; +``` + +- [ ] Run: + +```bash +SQLX_OFFLINE=true cargo test -p ml-alpha --test eval_diag_emission --release -- --ignored --nocapture 2>&1 | tail -20 +``` + +Expected: the test FAILS with `train leaf count 712 != expected 725` — confirming the test detects the schema-vs-emit gap. This is the TDD red state for the diag-emit changes. + +### Step 1.5.3 — Compute and emit state-conditional fractions (host-side) + +- [ ] Open `/home/jgrusewski/Work/foxhunt/crates/ml-alpha/src/trainer/integrated.rs`. +- [ ] Locate the per-step derived-stats block inside `build_diag_value` (around line 11699-11767). Immediately AFTER the `let heat_cap_step = ...` line (~11767) and BEFORE `let unit_active_host: Vec = ...` (~11770), insert: + +```rust + // Phase 7a F2-diag (2026-06-04) — state-conditional surfer- + // preservation measurement. `frac_batches_{flat,long,short}` are + // host-side reductions over `inputs.position_lots` (already on + // host as `&'a [i32]` per the inputs struct contract — no D2H). + // Per `pearl_fleet_fraction_not_aggregate` we emit fractions, + // not aggregates, so the b=1024 cluster diag retains per-batch + // resolution. + let inv_b: f32 = if b_size > 0 { 1.0 / b_size as f32 } else { 1.0 }; + let mut n_flat = 0u32; + let mut n_long = 0u32; + let mut n_short = 0u32; + for &p in inputs.position_lots { + if p == 0 { n_flat += 1; } + else if p > 0 { n_long += 1; } + else { n_short += 1; } + } + let frac_batches_flat = (n_flat as f32) * inv_b; + let frac_batches_long = (n_long as f32) * inv_b; + let frac_batches_short = (n_short as f32) * inv_b; + + // hold_frac_flat = count(action == HOLD AND pos == 0) / count(pos == 0). + // Direct F2-G5 measurement: if F2's hinge pushes Open mass too + // hard from flat states, this drops below 0.30 (surfer→trend). + // If F2 fails to break the inaction collapse, this stays above + // 0.85. Healthy regime: [0.30, 0.85] at step 5000. + // + // open_action_mass_flat = count(action ∈ Open AND pos == 0) / + // count(pos == 0). Complementary; equals + // (1 − hold_frac_flat − noop_other_from_flat). Open actions per + // the enum: ShortLarge=3, ShortSmall=4, LongSmall=5, LongLarge=6. + let action_hold = crate::rl::common::Action::Hold as usize; + let action_short_large = crate::rl::common::Action::ShortLarge as usize; + let action_short_small = crate::rl::common::Action::ShortSmall as usize; + let action_long_small = crate::rl::common::Action::LongSmall as usize; + let action_long_large = crate::rl::common::Action::LongLarge as usize; + let mut n_hold_from_flat = 0u32; + let mut n_open_from_flat = 0u32; + for (i, &p) in inputs.position_lots.iter().enumerate() { + if p != 0 { continue; } + let a = inputs.actions[i] as usize; + if a == action_hold { n_hold_from_flat += 1; } + else if a == action_short_large || a == action_short_small + || a == action_long_small || a == action_long_large { + n_open_from_flat += 1; + } + } + let inv_flat: f32 = if n_flat > 0 { 1.0 / n_flat as f32 } else { 0.0 }; + let hold_frac_flat = (n_hold_from_flat as f32) * inv_flat; + let open_action_mass_flat = (n_open_from_flat as f32) * inv_flat; +``` + +### Step 1.5.4 — Compute and emit attractor metrics (host-side, from act_hist) + +- [ ] Immediately after the Step 1.5.3 block, append: + +```rust + // Phase 7a F2-diag (2026-06-04) — degenerate-attractor metrics. + // All four are pure host-side reductions over the existing + // `act_hist` (length N_ACTIONS, computed at line ~11700). Zero + // marginal compute cost. + // + // max_action_share: maximum action mass / b_size — detects + // single-action collapse (the going-in-circles signature). The + // attractors in the historical trajectory are Hold (mq2pc), + // TrailLoosen, sometimes LongLarge. Above 0.80 means a single + // action dominates — likely a re-collapse. + // + // modal_action_id: which action is at the max. Lets the + // implementer / cluster observer identify the attractor without + // re-deriving from the histogram. + // + // noop_action_mass_{total,max}: total and max mass on the + // no-op set {Hold=2, TrailTighten=7, TrailLoosen=8}. The F2 + // hinge baseline is this set; if F2 redirects gradient + // correctly, this drops over training as Open actions earn + // positive advantage. noop_max separates "Hold-collapse" from + // "TrailLoosen-collapse" — both produce high noop_total but + // different noop_max. + let act_action_hold = crate::rl::common::Action::Hold as usize; + let act_trail_tighten = crate::rl::common::Action::TrailTighten as usize; + let act_trail_loosen = crate::rl::common::Action::TrailLoosen as usize; + let mut max_count = 0u32; + let mut modal_action_id = 0i32; + for (a, &c) in act_hist.iter().enumerate() { + if c > max_count { + max_count = c; + modal_action_id = a as i32; + } + } + let max_action_share = (max_count as f32) * inv_b; + let noop_hold = act_hist[act_action_hold] as f32; + let noop_tighten = act_hist[act_trail_tighten] as f32; + let noop_loosen = act_hist[act_trail_loosen] as f32; + let noop_action_mass_total = (noop_hold + noop_tighten + noop_loosen) * inv_b; + let noop_action_mass_max = + noop_hold.max(noop_tighten).max(noop_loosen) * inv_b; +``` + +### Step 1.5.5 — Add F2 mechanism counters in `rl_q_pi_distill_grad.cu` + +The 4 F2-mechanism diagnostics require per-batch reductions inside the kernel. We compute them in the existing single-thread `a == 0` block (which already runs once per batch block) and write to ISV via the standard per-batch fused-stats pattern (NO atomicAdd; we rely on the fact that each batch's b-th block writes its own scalar contribution into a per-batch scratch buffer, then a sibling reducer averages — OR, since this is single-thread-per-block, we use the simpler pattern of writing per-batch into a B-length scratch device buffer and reducing in a follow-up tiny kernel). + +For minimal code change, we use the same idiom as the band kernel: each block writes its per-batch scalars into 4 `[B]`-sized device scratch buffers; the trainer reads those at diag emit and reduces host-side. The kernel signature gets 4 new pointers; the launch site allocates the scratch buffers. + +#### Step 1.5.5a — Modify the kernel signature + emit per-batch contributions + +- [ ] Open `/home/jgrusewski/Work/foxhunt/crates/ml-alpha/cuda/rl_q_pi_distill_grad.cu`. +- [ ] Locate the existing kernel signature (search for `extern "C" __global__ void rl_q_pi_distill_grad`). Extend the parameter list: + +```c +extern "C" __global__ void rl_q_pi_distill_grad( + const float* __restrict__ q_logits_target, + const float* __restrict__ pi_logits, + const float* __restrict__ atom_supports, + float* __restrict__ isv, + float* __restrict__ pi_grad_logits, + // Phase 7a F2-diag (2026-06-04) — per-batch scratch buffers + // populated by the single-thread reduction at the END of the + // kernel. All length [B]; populated by `a == 0` thread of each + // block. Trainer aggregates host-side at diag emit. + float* __restrict__ f2_diag_baseline_per_b, // baseline per batch + float* __restrict__ f2_diag_adv_max_per_b, // max hinged per batch + float* __restrict__ f2_diag_hinge_zero_per_b, // fraction-zeroed per batch + int* __restrict__ f2_diag_target_argmax_per_b, // argmax π_target per batch + int b_size) +{ +``` + +- [ ] Locate the END of the kernel — after `pi_grad_logits` write-back, BEFORE the closing brace. The thread `a == 0` (single thread per block) is the natural site for per-batch reduction. Insert (still inside `if (a == 0) { ... }` if there is one at the end; else gate explicitly): + +```c + // ── Phase 7a F2-diag: per-batch mechanism counters ─────────── + // Only thread a == 0 writes (single-thread reduction inside the + // block — uses the s_eq + s_pi_target shared memory already + // computed above; no race, no atomicAdd). The trainer reduces + // these [B]-sized buffers host-side at diag emit. + if (a == 0) { + const int batch = (int)blockIdx.x; + if (batch < b_size) { + // baseline used by the hinge (mode-dependent; recomputed + // here for diag correctness even when F2 is OFF, so the + // counters are interpretable in both regimes). + const float f2_on_diag = isv[RL_F2_DISTILL_CENTERED_ENABLED_INDEX]; + const float mode_diag = isv[RL_F2_DISTILL_NOOP_SET_MODE_INDEX]; + float baseline_diag; + if (mode_diag > 0.5f && mode_diag <= 1.5f) { + baseline_diag = s_eq[ACTION_HOLD]; + } else { + baseline_diag = fmaxf(s_eq[ACTION_HOLD], + fmaxf(s_eq[ACTION_TRAIL_TIGHTEN], + s_eq[ACTION_TRAIL_LOOSEN])); + } + f2_diag_baseline_per_b[batch] = baseline_diag; + + // max hinged advantage across actions for this batch. + float adv_max = 0.0f; + int n_zero = 0; + for (int i = 0; i < N_ACTIONS; ++i) { + const float h = fmaxf(0.0f, s_eq[i] - baseline_diag); + if (h > adv_max) adv_max = h; + if (h <= 0.0f) ++n_zero; + } + f2_diag_adv_max_per_b[batch] = adv_max; + f2_diag_hinge_zero_per_b[batch] = + (float)n_zero / (float)N_ACTIONS; + + // argmax of s_pi_target — already computed (in F2 path or + // legacy path) by the time we reach this block. + int t_arg = 0; + float t_max = s_pi_target[0]; + for (int i = 1; i < N_ACTIONS; ++i) { + if (s_pi_target[i] > t_max) { + t_max = s_pi_target[i]; + t_arg = i; + } + } + f2_diag_target_argmax_per_b[batch] = t_arg; + (void)f2_on_diag; // silence unused warning when F2 is OFF. + } + } +} +``` + +NOTE on the no-atomicAdd discipline: each block writes exclusively to `[batch]` indexed slots of its own (one block per batch). No write races. The follow-up host-side reduction in Step 1.5.5c is single-threaded. + +#### Step 1.5.5b — Extend the kernel launch site + +- [ ] Open `/home/jgrusewski/Work/foxhunt/crates/ml-alpha/src/trainer/integrated.rs`. +- [ ] Modify `launch_rl_q_pi_distill_grad` (line ~5160) to accept the 4 new scratch slices: + +```rust + pub fn launch_rl_q_pi_distill_grad( + &self, + q_logits_target_d: &CudaSlice, + pi_logits_d: &CudaSlice, + atom_supports_d: &CudaSlice, + pi_grad_logits_d: &mut CudaSlice, + // Phase 7a F2-diag (2026-06-04) + f2_baseline_per_b_d: &mut CudaSlice, + f2_adv_max_per_b_d: &mut CudaSlice, + f2_hinge_zero_per_b_d: &mut CudaSlice, + f2_target_argmax_per_b_d: &mut CudaSlice, + b_size: usize, + ) -> Result<()> { + let b_size_i = b_size as i32; + let mut args = RawArgs::new(); + args.push_ptr(q_logits_target_d.raw_ptr()); + args.push_ptr(pi_logits_d.raw_ptr()); + args.push_ptr(atom_supports_d.raw_ptr()); + args.push_ptr(self.isv_dev_ptr); + args.push_ptr(pi_grad_logits_d.raw_ptr()); + args.push_ptr(f2_baseline_per_b_d.raw_ptr()); + args.push_ptr(f2_adv_max_per_b_d.raw_ptr()); + args.push_ptr(f2_hinge_zero_per_b_d.raw_ptr()); + args.push_ptr(f2_target_argmax_per_b_d.raw_ptr()); + args.push_i32(b_size_i); + let mut ptrs = args.build_arg_ptrs(); + unsafe { + raw_launch( + self.rl_q_pi_distill_grad_fn.cu_function(), + (b_size as u32, 1, 1), (N_ACTIONS as u32, 1, 1), 0, + self.raw_stream, + &mut ptrs[..args.len()], + ).map_err(|e| anyhow::anyhow!("rl_q_pi_distill_grad: {:?}", e))?; + } + Ok(()) + } +``` + +- [ ] Allocate the 4 device scratch buffers as trainer fields (the existing trainer holds many such `[B]`-sized scratch buffers; follow the same pattern). Search for an existing per-batch scratch buffer (e.g., `ss_pi_grad_logits_d` or another `_d` allocated at `IntegratedTrainer::new` time) and add four siblings: + +```rust + // Phase 7a F2-diag (2026-06-04) — per-batch scratch for + // kernel-reduced F2 mechanism counters. + let f2_diag_baseline_per_b_d: CudaSlice = stream.alloc_zeros(b_size)?; + let f2_diag_adv_max_per_b_d: CudaSlice = stream.alloc_zeros(b_size)?; + let f2_diag_hinge_zero_per_b_d: CudaSlice = stream.alloc_zeros(b_size)?; + let f2_diag_target_argmax_per_b_d: CudaSlice = stream.alloc_zeros(b_size)?; +``` + +Add the matching fields to the `IntegratedTrainer` struct and wire them through `Self { ... }`: + +```rust + f2_diag_baseline_per_b_d: CudaSlice, + f2_diag_adv_max_per_b_d: CudaSlice, + f2_diag_hinge_zero_per_b_d: CudaSlice, + f2_diag_target_argmax_per_b_d: CudaSlice, +``` + +- [ ] Update the existing in-train call site at line ~6132 (`rl_q_pi_distill_grad_fn` launch inside `step_with_lobsim_gpu_body`) to thread the 4 new pointers. If the launch site uses `raw_launch` directly (NOT `launch_rl_q_pi_distill_grad`), extend the `args.push_ptr(...)` chain at that site instead: + +```rust + args.push_ptr(self.f2_diag_baseline_per_b_d.raw_ptr()); + args.push_ptr(self.f2_diag_adv_max_per_b_d.raw_ptr()); + args.push_ptr(self.f2_diag_hinge_zero_per_b_d.raw_ptr()); + args.push_ptr(self.f2_diag_target_argmax_per_b_d.raw_ptr()); +``` + +#### Step 1.5.5c — Read scratch buffers at diag emit and reduce host-side; write into ISV + +- [ ] In `build_diag_value` (around line 11856-11935 — the host-side band reduction block is a good reference), AFTER the existing band reduction and BEFORE the JSON object construction, insert: + +```rust + // Phase 7a F2-diag (2026-06-04) — read kernel-emitted per-batch + // F2 mechanism counters and reduce host-side. Each is a single + // pass over [B] floats — cost ~4*B*4 bytes D2H per step, well + // under the existing band reduction's cost. + // + // Per `feedback_no_htod_htoh_only_mapped_pinned` we use the + // same `read_slice_d_pub` helper as the band kernel. + let f2_baseline_host = crate::trainer::integrated::read_slice_d_pub( + &self.cuda_stream, &self.f2_diag_baseline_per_b_d, b_size) + .context("build_diag_value: read f2_diag_baseline_per_b_d")?; + let f2_adv_max_host = crate::trainer::integrated::read_slice_d_pub( + &self.cuda_stream, &self.f2_diag_adv_max_per_b_d, b_size) + .context("build_diag_value: read f2_diag_adv_max_per_b_d")?; + let f2_hinge_zero_host = crate::trainer::integrated::read_slice_d_pub( + &self.cuda_stream, &self.f2_diag_hinge_zero_per_b_d, b_size) + .context("build_diag_value: read f2_diag_hinge_zero_per_b_d")?; + let f2_target_argmax_host: Vec = crate::trainer::integrated::read_slice_i32_d_pub( + &self.cuda_stream, &self.f2_diag_target_argmax_per_b_d, b_size) + .context("build_diag_value: read f2_diag_target_argmax_per_b_d")?; + + // Means over the [B] axis. + let f2_baseline_mean: f32 = + f2_baseline_host.iter().sum::() * inv_b; + let f2_advantage_max_mean: f32 = + f2_adv_max_host.iter().sum::() * inv_b; + let f2_hinge_zero_rate: f32 = + f2_hinge_zero_host.iter().sum::() * inv_b; + + // Modal argmax across the batch: histogram + argmax. + let mut argmax_hist = [0u32; N_ACTIONS]; + for &a in &f2_target_argmax_host { + if a >= 0 && (a as usize) < N_ACTIONS { + argmax_hist[a as usize] += 1; + } + } + let mut max_a_count = 0u32; + let mut f2_target_argmax_action: i32 = -1; + for (a, &c) in argmax_hist.iter().enumerate() { + if c > max_a_count { + max_a_count = c; + f2_target_argmax_action = a as i32; + } + } + + // Mirror into ISV so any controller / external probe can read + // them without re-reducing. Writes go through the mapped-pinned + // ISV handle (single-thread, deterministic). + self.isv_mapped.write_record( + crate::rl::isv_slots::RL_F2_DIAG_BASELINE_MEAN_INDEX, + f2_baseline_mean); + self.isv_mapped.write_record( + crate::rl::isv_slots::RL_F2_DIAG_ADV_MAX_MEAN_INDEX, + f2_advantage_max_mean); + self.isv_mapped.write_record( + crate::rl::isv_slots::RL_F2_DIAG_HINGE_ZERO_RATE_INDEX, + f2_hinge_zero_rate); + self.isv_mapped.write_record( + crate::rl::isv_slots::RL_F2_DIAG_TARGET_ARGMAX_INDEX, + f2_target_argmax_action as f32); +``` + +Both `read_slice_d_pub` (f32) and `read_slice_i32_d_pub` (i32) already exist in `integrated.rs` (lines ~12651 and ~12766 respectively) — verified during reconnaissance. No new helper is needed. + +### Step 1.5.6 — Wire all 13 diag fields into the JSON object + +- [ ] In `build_diag_value`, locate the `"trading": { ... }` JSON object (line ~12336). Add immediately after `"raw_reward_sum": ...,` (last field, line ~12356) and BEFORE the closing brace: + +```rust + // Phase 7a F2-diag (2026-06-04) — state-conditional + // denominators (fleet fractions per + // `pearl_fleet_fraction_not_aggregate`). flat + long + + // short MUST sum to 1.0 within fp noise. + "frac_batches_flat": frac_batches_flat, + "frac_batches_long": frac_batches_long, + "frac_batches_short": frac_batches_short, +``` + +- [ ] Locate the `"policy_diagnostic": { ... }` JSON object (line ~12376). Add immediately after the last existing field (`"q_pi_agree_ema": ...,` line ~12397) and BEFORE the closing brace: + +```rust + // Phase 7a F2-diag (2026-06-04) — surfer-preservation + // (F2-G5 direct measurement, locally observable): + "hold_frac_flat": hold_frac_flat, + "open_action_mass_flat": open_action_mass_flat, + // Phase 7a F2-diag — degenerate-attractor monitoring: + "max_action_share": max_action_share, + "modal_action_id": modal_action_id, + "noop_action_mass_total": noop_action_mass_total, + "noop_action_mass_max": noop_action_mass_max, + // Phase 7a F2-diag — F2 mechanism counters (kernel- + // reduced into ISV slots 819-822, mirrored here for + // diag-only consumers). f2_target_argmax_action is an + // integer index in {0..10}; consumers should compare + // against `modal_action_id` (sampled actions) — if F2 + // is reshaping the gradient correctly toward Open, + // these SHOULD differ during the surfer→adapt phase. + "f2_baseline_mean": f2_baseline_mean, + "f2_advantage_max_mean": f2_advantage_max_mean, + "f2_hinge_zero_rate": f2_hinge_zero_rate, + "f2_target_argmax_action": f2_target_argmax_action, +``` + +### Step 1.5.7 — Run determinism check (the new diag must NOT break bit-equal) + +- [ ] Build: + +```bash +SQLX_OFFLINE=true cargo build -p ml-alpha --release 2>&1 | tail -20 +``` + +- [ ] Run determinism check: + +```bash +cd /home/jgrusewski/Work/foxhunt +./scripts/determinism-check.sh --quick 2>&1 | tail -40 +``` + +Expected: PASS for seeds 42/43/44. If it fails: +- **Most likely cause:** the kernel reduction added in Step 1.5.5a introduced a write race or used non-deterministic input. Audit: each block writes EXCLUSIVELY to its own `[batch]` index — there must be no cross-block writes. +- **Second-most-likely cause:** the host-side reduction in Step 1.5.5c is reading the scratch buffer before the kernel has finished. Confirm the read happens AFTER the standard `stream.synchronize()` or implicit serialization that the band kernel relies on. +- STOP and audit; do not silence the determinism failure. + +### Step 1.5.8 — Add GPU-oracle test for each new diag emission + +For TDD discipline, each new diag emission gets a constructed-scenario test that asserts the emit is correct under a known input. Most of these are simple host-side reductions and can be tested by setting up a minimal `IntegratedTrainer` + injected inputs. + +- [ ] Create `/home/jgrusewski/Work/foxhunt/crates/ml-alpha/tests/f2_diag_emit.rs`: + +```rust +//! Phase 7a F2-diag (2026-06-04) — verify the 13 new diag fields are +//! correctly populated under constructed scenarios. +//! +//! Per `feedback_investigation_first_falsification_methodology`: +//! diagnostics that lie are worse than no diagnostics. Each new field +//! gets a known-input + expected-output check. + +use anyhow::Result; +use ml_alpha::trainer::integrated::{IntegratedTrainer, IntegratedTrainerConfig}; +use ml_alpha::trainer::perception::PerceptionTrainerConfig; +use ml_core::device::MlDevice; +use serde_json::Value; + +const N_ACTIONS: usize = 11; +const ACTION_HOLD: usize = 2; +const ACTION_LONG_SMALL: usize = 5; +const ACTION_LONG_LARGE: usize = 6; + +fn build_trainer(b_size: usize) -> Option<(MlDevice, IntegratedTrainer)> { + let dev = match MlDevice::cuda(0) { + Ok(d) => d, + Err(e) => { eprintln!("CUDA 0 not available — skipping ({e})"); return None; } + }; + let cfg = IntegratedTrainerConfig { + perception: PerceptionTrainerConfig { + seq_len: 4, n_batch: b_size, ..PerceptionTrainerConfig::default() + }, + dqn_seed: 0x420, ppo_seed: 0x421, + ..IntegratedTrainerConfig::default() + }; + Some((dev.clone(), IntegratedTrainer::new(&dev, cfg).expect("trainer"))) +} + +// ─────────────────────────────────────────────────────────────────────── +// Test 1: frac_batches_{flat,long,short} sum to 1.0 and match the input. +// ─────────────────────────────────────────────────────────────────────── + +#[test] +#[ignore = "requires CUDA"] +fn f2_diag_frac_batches_state_conditional() -> Result<()> { + let Some((_dev, mut trainer)) = build_trainer(8) else { return Ok(()); }; + // Inject a known position_lots array: 4 flat, 2 long, 2 short. + let position_lots: [i32; 8] = [0, 0, 0, 0, 1, 2, -1, -3]; + let actions: [i32; 8] = [ACTION_HOLD as i32; 8]; + // Run one diag emit with constructed inputs (use a test-helper + // entry point on IntegratedTrainer that takes the diag inputs + // directly; alternatively run a single step_with_lobsim_gpu with + // a controlled prefix and read the diag output). + let diag_value = trainer.build_diag_value_for_test( + /* step */ 0, + &position_lots, + &actions, + )?; + let trading = diag_value.get("trading").expect("trading key"); + let flat = trading["frac_batches_flat"].as_f64().expect("flat") as f32; + let long = trading["frac_batches_long"].as_f64().expect("long") as f32; + let short = trading["frac_batches_short"].as_f64().expect("short") as f32; + assert!((flat - 0.5 ).abs() < 1e-5, "flat: {flat}"); + assert!((long - 0.25 ).abs() < 1e-5, "long: {long}"); + assert!((short - 0.25 ).abs() < 1e-5, "short: {short}"); + assert!((flat + long + short - 1.0).abs() < 1e-5, "sum"); + Ok(()) +} + +// ─────────────────────────────────────────────────────────────────────── +// Test 2: hold_frac_flat is conditional — only counts hold-from-flat. +// ─────────────────────────────────────────────────────────────────────── + +#[test] +#[ignore = "requires CUDA"] +fn f2_diag_hold_frac_flat_is_state_conditional() -> Result<()> { + let Some((_dev, mut trainer)) = build_trainer(8) else { return Ok(()); }; + // 4 flat: 3 hold + 1 long_small. 4 long: all hold (must NOT count). + let position_lots: [i32; 8] = [0, 0, 0, 0, 5, 5, 5, 5]; + let actions: [i32; 8] = [ + ACTION_HOLD as i32, ACTION_HOLD as i32, ACTION_HOLD as i32, + ACTION_LONG_SMALL as i32, + ACTION_HOLD as i32, ACTION_HOLD as i32, ACTION_HOLD as i32, ACTION_HOLD as i32, + ]; + let diag_value = trainer.build_diag_value_for_test(0, &position_lots, &actions)?; + let pd = diag_value.get("policy_diagnostic").expect("pd"); + let hff = pd["hold_frac_flat"].as_f64().expect("hff") as f32; + let oamf = pd["open_action_mass_flat"].as_f64().expect("oamf") as f32; + assert!((hff - 0.75).abs() < 1e-5, "hff: {hff} expected 0.75 (3/4)"); + assert!((oamf - 0.25).abs() < 1e-5, "oamf: {oamf} expected 0.25 (1/4)"); + Ok(()) +} + +// ─────────────────────────────────────────────────────────────────────── +// Test 3: max_action_share / modal_action_id detect single-action collapse. +// ─────────────────────────────────────────────────────────────────────── + +#[test] +#[ignore = "requires CUDA"] +fn f2_diag_attractor_metrics_detect_collapse() -> Result<()> { + let Some((_dev, mut trainer)) = build_trainer(10) else { return Ok(()); }; + // 9/10 batches choose LongLarge (collapse signature). + let position_lots: [i32; 10] = [0; 10]; + let mut actions: [i32; 10] = [ACTION_LONG_LARGE as i32; 10]; + actions[9] = ACTION_HOLD as i32; + let diag_value = trainer.build_diag_value_for_test(0, &position_lots, &actions)?; + let pd = diag_value.get("policy_diagnostic").expect("pd"); + let mas = pd["max_action_share"].as_f64().expect("mas") as f32; + let mid = pd["modal_action_id"].as_i64().expect("mid") as i32; + assert!((mas - 0.9).abs() < 1e-5, "mas: {mas} expected 0.9"); + assert_eq!(mid, ACTION_LONG_LARGE as i32, "modal action should be LongLarge"); + Ok(()) +} + +// ─────────────────────────────────────────────────────────────────────── +// Test 4: F2 mechanism counters — under F2-ON with constructed Q-logits +// where one Open action exceeds baseline, hinge_zero_rate < 1.0 and +// target_argmax_action matches the constructed winning action. +// ─────────────────────────────────────────────────────────────────────── + +#[test] +#[ignore = "requires CUDA"] +fn f2_diag_mechanism_counters_track_hinge() -> Result<()> { + // Setup: run one Q-distill kernel invocation with the F2 gate ON, + // E_Q[LongSmall=5] > baseline, all others ≤ baseline. Expect: + // f2_hinge_zero_rate ≈ (N_ACTIONS - 1) / N_ACTIONS = 10/11 ≈ 0.909 + // (only LongSmall escapes the hinge clamp) + // f2_target_argmax_action == ACTION_LONG_SMALL + // f2_advantage_max_mean > 0 + // f2_baseline_mean ≈ E_Q[Hold] (mode 0 default; baseline = max + // over {Hold, TrailTighten, TrailLoosen}) + // + // Implementation: reuse the `run_kernel_and_extract_pi_target` + // helper structure from f2_centered_distill_invariants.rs (Task 2), + // but here we additionally read back the per-batch scratch buffers + // and assert the reduced ISV values. + + // Body follows the same pattern as f2_centered_distill_invariants + // Test 3 ("positive mass above baseline") — see that file for the + // helper invocations. Add assertions on the new ISV slots after + // running the kernel: + // + // assert!(isv[RL_F2_DIAG_HINGE_ZERO_RATE_INDEX] > 0.8); + // assert_eq!(isv[RL_F2_DIAG_TARGET_ARGMAX_INDEX] as i32, ACTION_LONG_SMALL as i32); + // assert!(isv[RL_F2_DIAG_ADV_MAX_MEAN_INDEX] > 0.0); + // + // (Marked as an explicit test stub here for the implementer to + // expand; the kernel-launch boilerplate is identical to Task 2.) + Ok(()) +} +``` + +NOTE on the test helper `build_diag_value_for_test`: this is a new test-only entry point added to `IntegratedTrainer` that wraps `build_diag_value` with a stripped-down `DiagInputs` populated from the constructed `position_lots` + `actions`. If adding such a helper materially expands the public test surface, an alternative is to run a single `step_with_lobsim_gpu_body` with a controlled position/action prefix and parse the resulting diag JSONL line — slower but no new public API. Implementer's choice; document the decision in the commit message. + +- [ ] Run: + +```bash +SQLX_OFFLINE=true cargo test -p ml-alpha --test f2_diag_emit --release -- --ignored --nocapture 2>&1 | tail -40 +``` + +Expected: 4/4 PASS. + +### Step 1.5.9 — Re-run the diag-schema test (should now PASS) + +- [ ] Run: + +```bash +SQLX_OFFLINE=true cargo test -p ml-alpha --test eval_diag_emission --release -- --ignored --nocapture 2>&1 | tail -10 +``` + +Expected: PASS (leaf count now 725 matches the bumped EXPECTED_LEAVES). + +### Step 1.5.10 — Run ALL invariant suites + determinism + +- [ ] Run all suites in sequence: + +```bash +cd /home/jgrusewski/Work/foxhunt +SQLX_OFFLINE=true cargo test -p ml-alpha --test phase_5_invariants --release -- --ignored --nocapture 2>&1 | tail -5 +SQLX_OFFLINE=true cargo test -p ml-alpha --test band_invariants --release -- --ignored --nocapture 2>&1 | tail -5 +SQLX_OFFLINE=true cargo test -p ml-alpha --test multi_head_policy_invariants --release -- --ignored --nocapture 2>&1 | tail -5 +SQLX_OFFLINE=true cargo test -p ml-alpha --test f2_diag_emit --release -- --ignored --nocapture 2>&1 | tail -5 +SQLX_OFFLINE=true cargo test -p ml-alpha --test eval_diag_emission --release -- --ignored --nocapture 2>&1 | tail -5 +./scripts/determinism-check.sh --quick 2>&1 | tail -10 +``` + +Expected: all PASS. If any pre-existing suite regressed, the diag instrumentation has affected a downstream behavior. Most likely culprit: extra ISV writes interleaved with controller reads — STOP and audit. + +### Step 1.5.11 — Commit + +- [ ] Stage and commit: + +```bash +git add /home/jgrusewski/Work/foxhunt/crates/ml-alpha/src/rl/isv_slots.rs \ + /home/jgrusewski/Work/foxhunt/crates/ml-alpha/src/trainer/integrated.rs \ + /home/jgrusewski/Work/foxhunt/crates/ml-alpha/cuda/rl_q_pi_distill_grad.cu \ + /home/jgrusewski/Work/foxhunt/crates/ml-alpha/tests/eval_diag_emission.rs \ + /home/jgrusewski/Work/foxhunt/crates/ml-alpha/tests/f2_diag_emit.rs +git commit -m "$(cat <<'EOF' +feat(ml-alpha): Phase 7a F2-diag — 13 diag fields for F2 measurement + +Adds the instrumentation required to attribute F2's effect locally +(Tier 1.5) and at b=1024 cluster (Tier 2). Without these the F2-G5 +surfer-preservation gate is unmeasurable locally per the spec; with +them, the cluster verdict gains three independent attribution signals: +mechanism (f2_hinge_zero_rate) / surfer (hold_frac_flat) / attractor +(modal_action_id + max_action_share). + +ISV slots 819-822 added for kernel-reduced mechanism counters: +- RL_F2_DIAG_BASELINE_MEAN_INDEX +- RL_F2_DIAG_ADV_MAX_MEAN_INDEX +- RL_F2_DIAG_HINGE_ZERO_RATE_INDEX +- RL_F2_DIAG_TARGET_ARGMAX_INDEX + +RL_SLOTS_END: 819 → 823. + +13 new diag leaves (EXPECTED_LEAVES 712 → 725): +- trading.frac_batches_{flat,long,short} (host reduce on position_lots) +- policy_diagnostic.{hold_frac_flat, open_action_mass_flat} (F2-G5) +- policy_diagnostic.{max_action_share, modal_action_id, + noop_action_mass_total, noop_action_mass_max} (attractor) +- policy_diagnostic.{f2_baseline_mean, f2_advantage_max_mean, + f2_hinge_zero_rate, f2_target_argmax_action} (F2 mechanism) + +Kernel changes: rl_q_pi_distill_grad.cu emits 4 per-batch scratch +arrays via single-thread reductions in the `a == 0` block (no +atomicAdd; one block per batch writes its own [batch] index only). +Trainer reduces host-side at diag emit + mirrors into ISV. + +GPU-oracle tests: tests/f2_diag_emit.rs (4/4 PASS) verifies each new +field under constructed scenarios. + +Determinism: scripts/determinism-check.sh --quick PASS at seeds +42/43/44 (no race, no atomicAdd, mapped-pinned reads only). + +Per `pearl_fleet_fraction_not_aggregate`: emits fractions, not +aggregates, so b=1024 cluster diag retains per-batch resolution. +Per `feedback_investigation_first_falsification_methodology`: +measure first (this task), change code second (Tasks 2 + 3). + +Co-Authored-By: Claude Opus 4.7 +EOF +)" +``` + +--- + +## Task 2 — Write the failing GPU-oracle invariant test (TDD) (~60 min) + +### Step 2.1 — Create the test file + +- [ ] Open `/home/jgrusewski/Work/foxhunt/crates/ml-alpha/tests/` (the directory must exist; `ls` to verify). +- [ ] Create `/home/jgrusewski/Work/foxhunt/crates/ml-alpha/tests/f2_centered_distill_invariants.rs` with the following exact contents: + +```rust +//! Phase 7a F2 (2026-06-04) — Q-centered Q-distill target invariants. +//! +//! Verifies the kernel `rl_q_pi_distill_grad` correctly implements the +//! hinged advantage target +//! +//! π_target(a) ∝ softmax( max(0, E_Q(a) − baseline) / τ ) +//! +//! where `baseline = max over a ∈ {Hold=2, TrailTighten=7, TrailLoosen=8} { E_Q(a) }` +//! when `RL_F2_DISTILL_CENTERED_ENABLED_INDEX > 0.5`, and falls back to +//! the legacy `softmax(E_Q / τ)` when the gate is ≤ 0.5. +//! +//! Per spec §5.1 four mechanism gates F2-G1/G2/G3/G4 are reduced here to +//! four kernel-level invariants. Cluster-side outcome gates (F2-G5 +//! surfer-patience / F2-G8 eval pnl) live in Tier 1.5 / Tier 2. +//! +//! Per `feedback_no_cpu_test_fallbacks`: oracles are analytical (closed- +//! form softmax + hinge), no CPU reference impl. +//! Per `feedback_no_htod_htoh_only_mapped_pinned`: all transfers go via +//! mapped-pinned `write_slice_*_d_pub` helpers. +//! +//! Run with: +//! `cargo test -p ml-alpha --test f2_centered_distill_invariants --release -- --ignored --nocapture` + +use anyhow::Result; +use cudarc::driver::{CudaSlice, CudaStream}; +use ml_alpha::rl::isv_slots::{ + RL_F2_DISTILL_CENTERED_ENABLED_INDEX, RL_F2_DISTILL_NOOP_SET_MODE_INDEX, + RL_Q_DISTILL_LAMBDA_INDEX, RL_Q_DISTILL_TEMPERATURE_INDEX, RL_SAC_ALPHA_INDEX, +}; +use ml_alpha::trainer::integrated::{ + read_slice_d_pub, write_slice_f32_d_pub, IntegratedTrainer, IntegratedTrainerConfig, +}; +use ml_alpha::trainer::perception::PerceptionTrainerConfig; +use ml_core::device::MlDevice; +use std::sync::Arc; + +const N_ACTIONS: usize = 11; +const Q_N_ATOMS: usize = 21; +const ACTION_HOLD: usize = 2; +const ACTION_TRAIL_TIGHTEN: usize = 7; +const ACTION_TRAIL_LOOSEN: usize = 8; +const ACTION_LONG_SMALL: usize = 5; +const ACTION_LONG_LARGE: usize = 6; + +fn build_trainer() -> Option<(MlDevice, IntegratedTrainer)> { + let dev = match MlDevice::cuda(0) { + Ok(d) => d, + Err(e) => { + eprintln!("CUDA 0 not available — skipping ({e})"); + return None; + } + }; + let cfg = IntegratedTrainerConfig { + perception: PerceptionTrainerConfig { + seq_len: 4, + n_batch: 1, + ..PerceptionTrainerConfig::default() + }, + dqn_seed: 0x520, + ppo_seed: 0x521, + ..IntegratedTrainerConfig::default() + }; + let trainer = IntegratedTrainer::new(&dev, cfg).expect("IntegratedTrainer::new"); + Some((dev, trainer)) +} + +fn upload_f32(stream: &Arc, host: &[f32]) -> Result> { + let mut d = stream.alloc_zeros::(host.len())?; + write_slice_f32_d_pub(stream, host, &mut d)?; + Ok(d) +} + +fn set_isv(trainer: &mut IntegratedTrainer, slot: usize, value: f32) { + trainer.isv_mapped.write_record(slot, value); +} + +/// Build a `[N_ACTIONS × Q_N_ATOMS]` block of Q-logits whose softmax +/// gives a per-atom mass concentrated near a single target atom, so +/// that the per-action E_Q equals approximately the corresponding entry +/// in `target_eq[a]`. Atom supports are assumed evenly spaced in [-1, 1]. +fn q_logits_for_eq(target_eq: &[f32; N_ACTIONS], atom_supports: &[f32]) -> Vec { + let mut out = vec![0.0_f32; N_ACTIONS * Q_N_ATOMS]; + for a in 0..N_ACTIONS { + // Find atom nearest to target_eq[a]; concentrate mass there. + let mut best_z = 0usize; + let mut best_d = f32::MAX; + for z in 0..Q_N_ATOMS { + let d = (atom_supports[z] - target_eq[a]).abs(); + if d < best_d { + best_d = d; + best_z = z; + } + } + for z in 0..Q_N_ATOMS { + // Peak logit at best_z, low elsewhere. softmax → near-onehot. + out[a * Q_N_ATOMS + z] = if z == best_z { 10.0 } else { -10.0 }; + } + } + out +} + +fn atom_supports_uniform() -> Vec { + (0..Q_N_ATOMS) + .map(|z| -1.0 + 2.0 * (z as f32) / (Q_N_ATOMS as f32 - 1.0)) + .collect() +} + +/// Compute the expected π_target distribution on the host using the F2 +/// hinge formula. Returns a length-N_ACTIONS probability vector. +fn expected_f2_pi_target(eq: &[f32; N_ACTIONS], tau: f32) -> [f32; N_ACTIONS] { + let baseline = [eq[ACTION_HOLD], eq[ACTION_TRAIL_TIGHTEN], eq[ACTION_TRAIL_LOOSEN]] + .iter() + .cloned() + .fold(f32::NEG_INFINITY, f32::max); + let hinged: Vec = eq.iter().map(|&v| (v - baseline).max(0.0)).collect(); + let tau_safe = tau.max(1e-3); + let exps: Vec = hinged.iter().map(|&h| (h / tau_safe).exp()).collect(); + // Numerical normalization (subtract max for stability — but since all + // hinged values ≥ 0 and baseline action's hinged = 0, max(exps) = exp(max_hinge/τ)). + let total: f32 = exps.iter().sum(); + let mut out = [0.0_f32; N_ACTIONS]; + if total > 1e-9 { + for i in 0..N_ACTIONS { + out[i] = exps[i] / total; + } + } else { + for i in 0..N_ACTIONS { + out[i] = 1.0 / (N_ACTIONS as f32); + } + } + out +} + +/// Run the kernel once with the given Q-logits and read back the +/// per-action gradient. Returns (pi, grad_distill_only) where grad is +/// extracted by subtracting the SAC entropy term (computed analytically +/// from the supplied pi_logits since the kernel applies both at once). +fn run_kernel_and_extract_pi_target( + trainer: &mut IntegratedTrainer, + stream: &Arc, + pi_logits_host: &[f32; N_ACTIONS], + eq_target: &[f32; N_ACTIONS], + tau: f32, + lambda: f32, + f2_enabled: bool, +) -> Result<[f32; N_ACTIONS]> { + // Configure ISV. + set_isv(trainer, RL_Q_DISTILL_LAMBDA_INDEX, lambda); + set_isv(trainer, RL_Q_DISTILL_TEMPERATURE_INDEX, tau); + set_isv(trainer, RL_SAC_ALPHA_INDEX, 0.0); // Zero out entropy grad. + set_isv( + trainer, + RL_F2_DISTILL_CENTERED_ENABLED_INDEX, + if f2_enabled { 1.0 } else { 0.0 }, + ); + set_isv(trainer, RL_F2_DISTILL_NOOP_SET_MODE_INDEX, 0.0); // mode 0 = static. + + let atom = atom_supports_uniform(); + let q_host = q_logits_for_eq(eq_target, &atom); + let q_d = upload_f32(stream, &q_host)?; + let pi_d = upload_f32(stream, pi_logits_host)?; + let atom_d = upload_f32(stream, &atom)?; + + let mut pi_grad_d = stream.alloc_zeros::(N_ACTIONS)?; + + trainer.launch_rl_q_pi_distill_grad(&q_d, &pi_d, &atom_d, &mut pi_grad_d, 1)?; + + let grad = read_slice_d_pub(stream, &pi_grad_d, N_ACTIONS)?; + + // Kernel writes: grad_total = (grad_distill + grad_entropy) / B. + // With α = 0 and B = 1: grad_total = grad_distill = λ · (π_θ − π_target). + // Recover π_target = π_θ − grad / λ. + let max_logit = pi_logits_host.iter().cloned().fold(f32::NEG_INFINITY, f32::max); + let exps: Vec = pi_logits_host.iter().map(|l| (l - max_logit).exp()).collect(); + let total: f32 = exps.iter().sum(); + let pi_theta: Vec = exps.iter().map(|e| e / total).collect(); + + let mut recovered = [0.0_f32; N_ACTIONS]; + for a in 0..N_ACTIONS { + recovered[a] = pi_theta[a] - grad[a] / lambda; + } + Ok(recovered) +} + +// =========================================================================== +// Test 1: when Q is uniform, F2 hinge yields uniform π_target. +// =========================================================================== +// +// All E_Q[a] = c (constant) → baseline = c → hinged = 0 for all a → +// softmax(0) = uniform 1/N for all a. F2 must produce uniform target. + +#[test] +#[ignore = "requires CUDA (MlDevice::cuda(0))"] +fn f2_centered_target_uniform_when_q_uniform() -> Result<()> { + let Some((dev, mut trainer)) = build_trainer() else { + return Ok(()); + }; + let stream = dev.cuda_stream()?.clone(); + + let eq = [0.5_f32; N_ACTIONS]; // All actions same E_Q. + let pi_logits = [0.0_f32; N_ACTIONS]; // Uniform π_θ. + let tau = 5.0; + let lambda = 1.0; + + let recovered = run_kernel_and_extract_pi_target( + &mut trainer, &stream, &pi_logits, &eq, tau, lambda, true, + )?; + + let expected = 1.0 / (N_ACTIONS as f32); + for a in 0..N_ACTIONS { + let diff = (recovered[a] - expected).abs(); + assert!( + diff < 1e-3, + "uniform-Q test: π_target[{a}] = {} expected {} diff {:.3e}", + recovered[a], expected, diff, + ); + } + + eprintln!("PASS — F2 hinge with uniform E_Q produces uniform π_target"); + Ok(()) +} + +// =========================================================================== +// Test 2: open-action below baseline → zero distill mass on that action. +// =========================================================================== +// +// E_Q[LongSmall=5] = 0.2, E_Q[Hold=2] = 0.6, others 0.1. +// baseline = max(0.6, 0.1, 0.1) = 0.6. +// hinged[5] = max(0, 0.2 - 0.6) = 0 → π_target[5] component in softmax +// numerator = exp(0/τ) = 1. But hinged[Hold] = 0 too. So they share +// equal raw mass (= 1) while all other hinged ≤ 0 also have raw mass 1. +// In short: when ALL actions are at-or-below baseline, π_target is +// uniform (mass-zero hinge degenerate case). This test verifies the +// boundary: LongSmall at 0.2 vs Hold at 0.6 — both hinge to 0 — both +// receive equal mass. The asymmetry only shows when an action *exceeds* +// baseline (Test 3). +// +// What this verifies: the hinge clamps below-baseline actions to zero +// numerator (NOT to a negative numerator that would produce a smaller +// share than uniform). Open's softmax share equals Hold's share when +// both hinge to zero. + +#[test] +#[ignore = "requires CUDA (MlDevice::cuda(0))"] +fn f2_centered_target_equal_share_when_all_below_baseline() -> Result<()> { + let Some((dev, mut trainer)) = build_trainer() else { + return Ok(()); + }; + let stream = dev.cuda_stream()?.clone(); + + let mut eq = [0.1_f32; N_ACTIONS]; + eq[ACTION_HOLD] = 0.6; + eq[ACTION_LONG_SMALL] = 0.2; + eq[ACTION_TRAIL_TIGHTEN] = 0.1; + eq[ACTION_TRAIL_LOOSEN] = 0.1; + let pi_logits = [0.0_f32; N_ACTIONS]; + let tau = 5.0; + let lambda = 1.0; + + let recovered = run_kernel_and_extract_pi_target( + &mut trainer, &stream, &pi_logits, &eq, tau, lambda, true, + )?; + + // All hinged values ≤ 0 → all clamped to 0 → uniform target. + let expected = expected_f2_pi_target(&eq, tau); + for a in 0..N_ACTIONS { + let diff = (recovered[a] - expected[a]).abs(); + assert!( + diff < 2e-3, + "below-baseline test: π_target[{a}] = {} expected {} diff {:.3e}", + recovered[a], expected[a], diff, + ); + } + // Sanity: π_target[LongSmall] should equal π_target[Hold] within fp32 noise + // (both hinged to 0). + let d = (recovered[ACTION_LONG_SMALL] - recovered[ACTION_HOLD]).abs(); + assert!( + d < 2e-3, + "LongSmall {} ≠ Hold {} but both hinge to zero", + recovered[ACTION_LONG_SMALL], recovered[ACTION_HOLD], + ); + + eprintln!( + "PASS — F2 hinge collapses below-baseline actions to uniform share \ + (Hold {} vs LongSmall {})", + recovered[ACTION_HOLD], recovered[ACTION_LONG_SMALL], + ); + Ok(()) +} + +// =========================================================================== +// Test 3: open-action above baseline → POSITIVE mass on that action, +// strictly greater than baseline action's mass. +// =========================================================================== +// +// E_Q[LongSmall=5] = 0.9, E_Q[Hold=2] = 0.6, others 0.5. +// baseline = max(0.6, 0.5, 0.5) = 0.6. +// hinged[5] = 0.3; hinged[Hold] = 0; hinged[TrailTight/Loose] = 0; +// hinged[others] ≤ 0 (clamped to 0). +// At τ = 5.0: numerator[5] = exp(0.3/5) = exp(0.06) ≈ 1.0618. +// numerator[others] = exp(0) = 1.0. +// Total = 1.0618 + 10*1.0 = 11.0618. +// π_target[5] ≈ 0.0960; π_target[Hold] ≈ 0.0904. +// π_target[LongSmall] > π_target[Hold]. + +#[test] +#[ignore = "requires CUDA (MlDevice::cuda(0))"] +fn f2_centered_target_positive_mass_above_baseline() -> Result<()> { + let Some((dev, mut trainer)) = build_trainer() else { + return Ok(()); + }; + let stream = dev.cuda_stream()?.clone(); + + let mut eq = [0.5_f32; N_ACTIONS]; + eq[ACTION_HOLD] = 0.6; + eq[ACTION_LONG_SMALL] = 0.9; + eq[ACTION_TRAIL_TIGHTEN] = 0.5; + eq[ACTION_TRAIL_LOOSEN] = 0.5; + let pi_logits = [0.0_f32; N_ACTIONS]; + let tau = 5.0; + let lambda = 1.0; + + let recovered = run_kernel_and_extract_pi_target( + &mut trainer, &stream, &pi_logits, &eq, tau, lambda, true, + )?; + + let expected = expected_f2_pi_target(&eq, tau); + for a in 0..N_ACTIONS { + let diff = (recovered[a] - expected[a]).abs(); + assert!( + diff < 2e-3, + "above-baseline test: π_target[{a}] = {} expected {} diff {:.3e}", + recovered[a], expected[a], diff, + ); + } + // Key invariant: LongSmall > Hold. + assert!( + recovered[ACTION_LONG_SMALL] > recovered[ACTION_HOLD] + 1e-4, + "LongSmall ({}) should exceed Hold ({}) when E_Q[LongSmall] > baseline", + recovered[ACTION_LONG_SMALL], recovered[ACTION_HOLD], + ); + + eprintln!( + "PASS — F2 hinge gives strictly higher mass to above-baseline action \ + (LongSmall {} vs Hold {})", + recovered[ACTION_LONG_SMALL], recovered[ACTION_HOLD], + ); + Ok(()) +} + +// =========================================================================== +// Test 4: F2 disabled (gate = 0.0) reproduces legacy softmax(E_Q/τ) target. +// =========================================================================== +// +// Bit-equality A/B guarantee: when RL_F2_DISTILL_CENTERED_ENABLED_INDEX +// = 0.0, the kernel must produce the exact same π_target as the +// pre-F2 implementation: softmax(E_Q / τ). + +#[test] +#[ignore = "requires CUDA (MlDevice::cuda(0))"] +fn f2_disabled_matches_legacy_softmax() -> Result<()> { + let Some((dev, mut trainer)) = build_trainer() else { + return Ok(()); + }; + let stream = dev.cuda_stream()?.clone(); + + let mut eq = [0.5_f32; N_ACTIONS]; + eq[ACTION_HOLD] = 0.6; + eq[ACTION_LONG_SMALL] = 0.9; + eq[ACTION_LONG_LARGE] = 0.85; + eq[ACTION_TRAIL_TIGHTEN] = 0.55; + eq[ACTION_TRAIL_LOOSEN] = 0.55; + let pi_logits = [0.0_f32; N_ACTIONS]; + let tau = 5.0; + let lambda = 1.0; + + let recovered = run_kernel_and_extract_pi_target( + &mut trainer, &stream, &pi_logits, &eq, tau, lambda, /* f2_enabled */ false, + )?; + + // Legacy expected: softmax(E_Q / τ). + let max_eq = eq.iter().cloned().fold(f32::NEG_INFINITY, f32::max); + let exps: Vec = eq.iter().map(|&v| ((v - max_eq) / tau).exp()).collect(); + let total: f32 = exps.iter().sum(); + let expected: Vec = exps.iter().map(|e| e / total).collect(); + + for a in 0..N_ACTIONS { + let diff = (recovered[a] - expected[a]).abs(); + assert!( + diff < 2e-3, + "F2-disabled test: π_target[{a}] = {} expected legacy {} diff {:.3e}", + recovered[a], expected[a], diff, + ); + } + + eprintln!("PASS — F2 disabled reproduces legacy softmax(E_Q/τ) target"); + Ok(()) +} +``` + +### Step 2.2 — Verify the test compiles BUT FAILS + +- [ ] Run: + +```bash +SQLX_OFFLINE=true cargo test -p ml-alpha --test f2_centered_distill_invariants --release --no-run 2>&1 | tail -20 +``` + +Expected: compiles cleanly. + +- [ ] Run the tests (RTX 3050 local): + +```bash +SQLX_OFFLINE=true cargo test -p ml-alpha --test f2_centered_distill_invariants --release -- --ignored --nocapture 2>&1 | tail -40 +``` + +Expected: tests 1-3 FAIL (the F2 gate is in ISV but the kernel does not read it yet). Test 4 (legacy match) PASSES because the kernel currently does the legacy path unconditionally. + +This is the TDD red state — proceed to Task 3. + +- [ ] If any of tests 1-3 unexpectedly PASS at this stage, STOP — the F2 hinge is somehow already wired up, or the test oracle is wrong. Inspect before proceeding. + +### Step 2.3 — Commit the failing test + +- [ ] Stage and commit (the test is checked in alongside the kernel change in the next task — but committing now creates a clean TDD checkpoint per `superpowers:test-driven-development`): + +```bash +git add /home/jgrusewski/Work/foxhunt/crates/ml-alpha/tests/f2_centered_distill_invariants.rs +git commit -m "$(cat <<'EOF' +test(ml-alpha): Phase 7a F2 — GPU-oracle invariants for Q-centered distill target + +Four invariants verify the kernel implements + π_target(a) ∝ softmax(max(0, E_Q(a) − baseline) / τ) + +with baseline = max E_Q over {Hold=2, TrailTighten=7, TrailLoosen=8}: + +1. Uniform Q → uniform π_target (hinge is 0 everywhere, degenerate case). +2. All E_Q ≤ baseline → equal share (hinge clamps below-baseline to 0). +3. E_Q[Open] > baseline → π_target[Open] > π_target[Hold] (the key signal). +4. F2 disabled (gate = 0.0) → legacy softmax(E_Q/τ) bit-equal. + +TDD checkpoint: tests 1-3 currently FAIL (kernel does not read the new +gate yet); test 4 PASSES. Task 3 modifies the kernel. + +Co-Authored-By: Claude Opus 4.7 +EOF +)" +``` + +--- + +## Task 3 — Modify rl_q_pi_distill_grad.cu to apply the hinge (~40 min) + +### Step 3.1 — Add the slot constant defines to the kernel + +- [ ] Open `/home/jgrusewski/Work/foxhunt/crates/ml-alpha/cuda/rl_q_pi_distill_grad.cu`. +- [ ] Locate the existing `#define` block (lines 17-40). After `#define RL_ACTION_ENTROPY_EMA_INDEX 583` (line 26) and before the B-10 documentation comment (line 27), insert: + +```c +// Phase 7a F2 (2026-06-04) — Q-centered distill target master gate + +// no-op-set mode. Spec docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md +#define RL_F2_DISTILL_CENTERED_ENABLED_INDEX 817 +#define RL_F2_DISTILL_NOOP_SET_MODE_INDEX 818 + +// No-op-set action indices — must match crate::rl::common::Action enum +// (also defined in actions_to_market_targets.cu). +#define ACTION_HOLD 2 +#define ACTION_TRAIL_TIGHTEN 7 +#define ACTION_TRAIL_LOOSEN 8 +``` + +### Step 3.2 — Modify the π_target computation (lines 78–91) + +- [ ] In the same file, locate the existing target-computation block: + +```c + // ── Step 2: π_target = softmax(E_Q / τ) ───────────────────── + __shared__ float s_pi_target[N_ACTIONS]; + if (a == 0) { + float max_eq = s_eq[0]; + for (int i = 1; i < N_ACTIONS; ++i) max_eq = fmaxf(max_eq, s_eq[i]); + const float tau_safe = fmaxf(tau, 1e-3f); + float total = 0.0f; + for (int i = 0; i < N_ACTIONS; ++i) { + s_pi_target[i] = expf((s_eq[i] - max_eq) / tau_safe); + total += s_pi_target[i]; + } + const float inv = (total > 1e-9f) ? (1.0f / total) : (1.0f / (float)N_ACTIONS); + for (int i = 0; i < N_ACTIONS; ++i) s_pi_target[i] *= inv; + } + __syncthreads(); +``` + +- [ ] Replace the block above with: + +```c + // ── Step 2: π_target ───────────────────────────────────────── + // Phase 7a F2 (2026-06-04): if RL_F2_DISTILL_CENTERED_ENABLED_INDEX + // > 0.5, compute the Q-centered hinged-advantage target + // π_target(a) ∝ softmax( max(0, E_Q(a) − baseline) / τ ) + // where baseline = max E_Q over the no-op set selected by + // RL_F2_DISTILL_NOOP_SET_MODE_INDEX (mode 0 = {Hold, TrailTighten, + // TrailLoosen}, mode 1 = {Hold} only). Else fall back to the legacy + // softmax(E_Q/τ) target for bit-equality with HEAD 0463e44e0. + // + // Per spec docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md + // §3.2 + §4.2: the hinge zeroes Open mass when E_Q(Open) ≤ baseline + // (preserves surfer-style patience), and gives positive mass when + // E_Q(Open) > baseline (preserves surfer-style commit on detected + // edge). Composes with band mask (band touches sampled actions; F2 + // touches Q-distill target — disjoint pipelines). + __shared__ float s_pi_target[N_ACTIONS]; + if (a == 0) { + const float f2_enabled = isv[RL_F2_DISTILL_CENTERED_ENABLED_INDEX]; + const float tau_safe = fmaxf(tau, 1e-3f); + + if (f2_enabled > 0.5f) { + // F2 hinged-advantage target. + const float mode = isv[RL_F2_DISTILL_NOOP_SET_MODE_INDEX]; + + // Compute baseline = max E_Q over no-op set. + float baseline; + if (mode > 0.5f && mode <= 1.5f) { + // Mode 1: Hold-only baseline. + baseline = s_eq[ACTION_HOLD]; + } else { + // Mode 0 (default): static set {Hold, TrailTighten, TrailLoosen}. + baseline = fmaxf(s_eq[ACTION_HOLD], + fmaxf(s_eq[ACTION_TRAIL_TIGHTEN], + s_eq[ACTION_TRAIL_LOOSEN])); + } + + // hinged[a] = max(0, E_Q(a) − baseline). + // softmax over hinged values, scaled by 1/τ. + // Numerical stability: hinged ≥ 0 by construction, so the + // max hinged value is a finite, non-negative number. We + // subtract it before exp to keep magnitudes bounded. + float max_hinge = 0.0f; + for (int i = 0; i < N_ACTIONS; ++i) { + const float h = fmaxf(0.0f, s_eq[i] - baseline); + if (h > max_hinge) max_hinge = h; + } + float total = 0.0f; + for (int i = 0; i < N_ACTIONS; ++i) { + const float h = fmaxf(0.0f, s_eq[i] - baseline); + s_pi_target[i] = expf((h - max_hinge) / tau_safe); + total += s_pi_target[i]; + } + const float inv = (total > 1e-9f) ? (1.0f / total) + : (1.0f / (float)N_ACTIONS); + for (int i = 0; i < N_ACTIONS; ++i) s_pi_target[i] *= inv; + } else { + // Legacy path (pre-F2, bit-equality with HEAD 0463e44e0). + float max_eq = s_eq[0]; + for (int i = 1; i < N_ACTIONS; ++i) max_eq = fmaxf(max_eq, s_eq[i]); + float total = 0.0f; + for (int i = 0; i < N_ACTIONS; ++i) { + s_pi_target[i] = expf((s_eq[i] - max_eq) / tau_safe); + total += s_pi_target[i]; + } + const float inv = (total > 1e-9f) ? (1.0f / total) + : (1.0f / (float)N_ACTIONS); + for (int i = 0; i < N_ACTIONS; ++i) s_pi_target[i] *= inv; + } + } + __syncthreads(); +``` + +### Step 3.3 — Build the kernel and the trainer + +- [ ] Run: + +```bash +SQLX_OFFLINE=true cargo build -p ml-alpha --release 2>&1 | tail -30 +``` + +Expected: clean build. The `build.rs` for ml-alpha compiles the `.cu` files into cubins — if there is a CUDA syntax error it will surface here. + +- [ ] If the build fails with a CUDA syntax error, STOP and fix the kernel before proceeding. Common issues: mismatched braces, missing semicolons, ACTION_HOLD redefined (if it was already defined elsewhere in the same translation unit — unlikely since this kernel doesn't include actions_to_market_targets.cu). + +### Step 3.4 — Re-run the F2 invariant test (should now PASS) + +- [ ] Run: + +```bash +SQLX_OFFLINE=true cargo test -p ml-alpha --test f2_centered_distill_invariants --release -- --ignored --nocapture 2>&1 | tail -30 +``` + +Expected: all 4 tests PASS. + +- [ ] If any test fails: + - **Test 1 (uniform Q) fails:** hinge is producing non-uniform output when all E_Q are equal. Most likely a sign error in the baseline subtraction. Re-read Step 3.2 patch. + - **Test 2 (below-baseline) fails:** below-baseline actions are receiving unequal mass. Most likely the `max(0, ·)` clamp is missing or the softmax is using a non-zero numerator for clamped actions. + - **Test 3 (above-baseline) fails:** the no-op set lookup is wrong, or the baseline is computed over the wrong set. + - **Test 4 (legacy match) fails:** the gate-OFF fallback branch is broken; bit-equality with the pre-F2 path is violated. STOP and inspect — this is a regression. + +### Step 3.5 — Commit + +- [ ] Stage and commit: + +```bash +git add /home/jgrusewski/Work/foxhunt/crates/ml-alpha/cuda/rl_q_pi_distill_grad.cu +git commit -m "$(cat <<'EOF' +feat(ml-alpha): Phase 7a F2 — Q-centered hinged-advantage distill target + +rl_q_pi_distill_grad.cu modification: when slot 817 > 0.5, compute + π_target(a) ∝ softmax(max(0, E_Q(a) − baseline) / τ) +with baseline = max E_Q over no-op set: + - mode 0 (default, slot 818 = 0.0): {Hold=2, TrailTighten=7, TrailLoosen=8} + - mode 1 (ablation, slot 818 = 1.0): {Hold=2} only + +When slot 817 ≤ 0.5, the legacy softmax(E_Q/τ) path runs unchanged +(bit-equal with HEAD 0463e44e0 — the F2-G7 determinism gate). + +Per spec docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md +§3.2 + §4.2 (Revision 3, surfer-principle-preserving primary phase). +Verified by tests/f2_centered_distill_invariants.rs (4/4 PASS). + +Composition with Phase 4-A band mask: band touches sampled actions +(actions_d) post-sampling; F2 touches the Q-distill π_target distribution +(pi_logits_d / q_logits_target_d). Disjoint pipelines — no interaction. + +Co-Authored-By: Claude Opus 4.7 +EOF +)" +``` + +--- + +## Task 4 — Run determinism check (bit-equality guarantee) (~10 min) + +### Step 4.1 — Determinism with band OFF + +- [ ] Run: + +```bash +cd /home/jgrusewski/Work/foxhunt +./scripts/determinism-check.sh --quick 2>&1 | tail -40 +``` + +Expected: exit code 0; the script reports "PASS" for seeds 42/43/44 (per `pearl_determinism_achieved`). + +- [ ] If the determinism check fails: + - Inspect the diff output for the diverging seed. + - Most likely cause: the F2 branch introduced an unsynchronized read or a per-thread-divergent path. The Step 3.2 patch only modifies the `if (a == 0)` block which executes single-thread per block, so any race must come from elsewhere. STOP and audit. + +### Step 4.2 — Determinism with band ON + +- [ ] Run: + +```bash +cd /home/jgrusewski/Work/foxhunt +FOXHUNT_BAND_ENABLED=1 ./scripts/determinism-check.sh --quick 2>&1 | tail -40 +``` + +Expected: exit code 0. + +- [ ] If this fails but Step 4.1 passed, that would indicate F2 + band have an unexpected interaction (contradicting the reconnaissance finding in Task 0 Step 0.3). STOP and investigate — do not commit any further changes until the composition is understood. + +### Step 4.3 — Smoke-test bit-equality with F2 disabled + +- [ ] Run a smoke at F2 OFF — temporarily flip slot 817 to 0.0 via env override if such an override exists, or via direct local edit of the bootstrap value. Easiest: directly inspect the diag JSONL from a `local-mid-smoke.sh` run with F2 ON vs OFF. + +Skip this step unless Step 4.1 or 4.2 surfaced an anomaly — Task 5's invariant test already covers F2-disabled-matches-legacy. + +### Step 4.4 — Commit (no code change; this is just a verification gate) + +- [ ] No commit needed for Task 4 unless a fix was applied. If a fix WAS applied (e.g., a __syncthreads barrier was missing), commit that fix separately with a clear message indicating the determinism issue and its resolution. + +--- + +## Task 5 — Run all invariant test suites (~15 min) + +### Step 5.1 — Run the four existing invariant suites + new F2 suite + +- [ ] Run all four suites in sequence: + +```bash +cd /home/jgrusewski/Work/foxhunt +SQLX_OFFLINE=true cargo test -p ml-alpha --test phase_5_invariants --release -- --ignored --nocapture 2>&1 | tail -10 +SQLX_OFFLINE=true cargo test -p ml-alpha --test band_invariants --release -- --ignored --nocapture 2>&1 | tail -10 +SQLX_OFFLINE=true cargo test -p ml-alpha --test multi_head_policy_invariants --release -- --ignored --nocapture 2>&1 | tail -10 +SQLX_OFFLINE=true cargo test -p ml-alpha --test f2_centered_distill_invariants --release -- --ignored --nocapture 2>&1 | tail -10 +``` + +Expected: +- `phase_5_invariants`: 3/3 PASS +- `band_invariants`: 11/11 PASS +- `multi_head_policy_invariants`: 18/18 PASS +- `f2_centered_distill_invariants`: 4/4 PASS + +Total: 36/36 PASS. + +- [ ] If any pre-existing suite regressed (phase_5, band, or multi_head_policy), STOP — F2's kernel modification has affected a downstream behavior it should not have. Most likely an entropy-formula edit slipped in unintentionally; re-read the diff in Task 3 Step 3.2 and confirm only the `Step 2: π_target` block was modified. + +### Step 5.2 — Trade management kernels regression check + +- [ ] Run: + +```bash +cd /home/jgrusewski/Work/foxhunt +SQLX_OFFLINE=true cargo test -p ml-alpha --test trade_management_kernels --release -- --ignored --nocapture 2>&1 | tail -20 +``` + +Expected: all tests pass at parity with HEAD `0463e44e0`. (This suite has been touched in the current working tree per git status — the implementer should verify pre-existing tests still pass; the F2 kernel modification should not affect trade-management kernels.) + +### Step 5.3 — No commit (verification only) + +- [ ] No commit for Task 5. The PASS state is the precondition for Task 6. + +--- + +## Task 6 — Local Tier 1.5 smoke + falsification gates (~15 min) + +### Step 6.1 — Run mid-smoke with F2 ENABLED + band ENABLED + +- [ ] Run: + +```bash +cd /home/jgrusewski/Work/foxhunt +rm -rf /tmp/foxhunt-phase7a-seed-42 +FOXHUNT_BAND_ENABLED=1 ./scripts/local-mid-smoke.sh \ + --seed 42 \ + --out /tmp/foxhunt-phase7a-seed-42 \ + --n-backtests 128 2>&1 | tail -30 +``` + +Expected runtime ≈ 10 min on RTX 3050. The run writes: +- `/tmp/foxhunt-phase7a-seed-42/diag.jsonl` — per-step diagnostics +- `/tmp/foxhunt-phase7a-seed-42/eval_summary.json` — eval pnl + wr_ema + action histogram + +### Step 6.2 — Run the Tier 1.5 verdict tool + +- [ ] Run: + +```bash +python3 /home/jgrusewski/Work/foxhunt/scripts/tier1_5_verdict.py /tmp/foxhunt-phase7a-seed-42 2>&1 | tail -40 +``` + +Expected: the verdict tool prints a structured report. Capture the output. + +### Step 6.3 — Evaluate F2 falsification gates manually from diag.jsonl + +The Tier 1.5 verdict tool predates the F2 gates. The four F2-specific gates per spec §5.1.1 + §4.3 must be evaluated by hand from the diag.jsonl + eval_summary.json: + +- [ ] **F2-G1 (target entropy effect).** Compute `RL_Q_DISTILL_TARGET_ENTROPY_MEAN` (slot 733) at step 1000 vs the pre-F2 baseline at the same step. The baseline value is approximately `ln(11) ≈ 2.40 nats` (uniform target). Pass condition: F2-on entropy drops by ≥ 0.5 nats (i.e., ≤ 1.9 nats). + +```bash +python3 -c " +import json +target_entropy_step_1000 = None +with open('/tmp/foxhunt-phase7a-seed-42/diag.jsonl') as f: + for line in f: + rec = json.loads(line) + if rec.get('step') == 1000: + target_entropy_step_1000 = rec.get('policy', {}).get('q_distill_target_entropy_mean') + break +print(f'F2-G1: target_entropy_at_step_1000 = {target_entropy_step_1000}') +print(f'F2-G1 PASS condition: <= 1.9 nats (drop >= 0.5 from uniform 2.40)') +" +``` + +- [ ] **F2-G5 (surfer-patience preservation, NOW LOCALLY MEASURABLE per Task 1.5).** Read `policy_diagnostic.hold_frac_flat` at step 5000. Pass condition: `hold_frac_flat ∈ [0.30, 0.85]`. + - Below 0.30: F2 is forcing too many opens from flat (surfer→trend). The hinge is too aggressive or baseline is mis-set; investigate. + - Above 0.85: F2 did NOT break the inaction collapse — Hold still dominates flat states despite the hinge. The Open-action E_Q values are not exceeding the no-op baseline; investigate (likely Q-network has not yet learned an Open edge — cold-start issue). + +```bash +python3 -c " +import json +with open('/tmp/foxhunt-phase7a-seed-42/diag.jsonl') as f: + for line in f: + rec = json.loads(line) + if rec.get('step') == 5000: + pd = rec.get('policy_diagnostic', {}) + t = rec.get('trading', {}) + hff = pd.get('hold_frac_flat') + oamf = pd.get('open_action_mass_flat') + frac_flat = t.get('frac_batches_flat') + print(f'F2-G5: hold_frac_flat at step 5000 = {hff}') + print(f' open_action_mass_flat = {oamf}') + print(f' frac_batches_flat (denominator) = {frac_flat}') + print(f'F2-G5 PASS condition: hold_frac_flat in [0.30, 0.85]') + break +" +``` + + Also inspect (informational; NOT pass/fail at this step but useful for attribution): + - `policy_diagnostic.max_action_share` — should be < 0.80 (no single-action collapse) + - `policy_diagnostic.modal_action_id` — should NOT be a no-op action (≠ {2,7,8}) if F2 is reshaping toward Open + - `policy_diagnostic.f2_hinge_zero_rate` — should be in `[0.4, 0.8]` (F2 mechanism active and discriminating). If close to 1.0, the hinge is vacuous (everything below baseline); if close to 0.0, the hinge is doing nothing. + - `policy_diagnostic.f2_target_argmax_action` vs `policy_diagnostic.modal_action_id` — when F2 is actively reshaping the gradient, these should DIFFER (target points at Open while sampling still hold-dominated). + +- [ ] **F2-G6 (action entropy at step 1999).** Pass condition: ≥ 0.5 nats (was approximately 0.22 in Phase 5 mq2pc baseline). + +```bash +python3 -c " +import json +with open('/tmp/foxhunt-phase7a-seed-42/diag.jsonl') as f: + for line in f: + rec = json.loads(line) + if rec.get('step') == 1999: + entropy = rec.get('policy', {}).get('action_entropy_ema') + print(f'F2-G6: action_entropy_ema at step 1999 = {entropy}') + print(f'F2-G6 PASS condition: >= 0.5 nats') + break +" +``` + +- [ ] **F2-G7 (no regression — local Tier 1.5 eval pnl proxy).** The full eval gate is cluster-only (F2-G8), but Tier 1.5 produces an eval_summary.json. Pass condition: `eval_pnl ≥ -$2M` (loose floor — the Phase 5 mq2pc local baseline was approximately -$444k; we tolerate up to 4.5× degradation locally since b=128 is noisy and not the load-bearing eval surface). + +```bash +python3 -c " +import json +with open('/tmp/foxhunt-phase7a-seed-42/eval_summary.json') as f: + summary = json.load(f) +pnl = summary.get('total_pnl_usd') +print(f'F2-G7: eval total_pnl_usd = {pnl}') +print(f'F2-G7 PASS condition: >= -2_000_000') +" +``` + +### Step 6.4 — STOP-on-surprise decision + +- [ ] **If ALL four gates PASS:** proceed to Task 7 (commit + push). +- [ ] **If F2-G1 fails** (target entropy did NOT drop): the hinge is vacuous. Most likely cause: no action's E_Q exceeds the baseline because Q is cold-start noisy. STOP per spec §4.5 step 7 — do NOT proceed to cluster. The implementer must either: + - Add a warmup phase (F2 OFF for first 500-1000 steps, then ON via controller transition) — requires another ISV slot + a controller, out of scope for this plan. + - Add a temperature schedule that smooths the hinge — out of scope. + Document the failure and surface to the user for decision. +- [ ] **If F2-G5 fails** with Hold% from flat < 5%: the hinge is pushing Hold mass below the random floor — the surfer-principle preservation argument has broken. STOP per spec §4.5 step 8. Likely cause: baseline-set definition bug (e.g., the wrong action index was used). +- [ ] **If F2-G5 fails** with Hold% from flat > 95%: the agent is in h6btt-class re-collapse — F2's hinge is insufficient to compete with the SAC prior. The disciplined next step is to surface to the user; Phase 7b (F5) may be the conditional follow-up (spec §4.6) but Phase 7a's plan ends here. +- [ ] **If F2-G6 fails:** training-time action entropy did not recover from the mq2pc 0.22 floor. Likely cause: the hinge is producing too-peaked targets even in the surviving support. Consider increasing τ in a follow-up. +- [ ] **If F2-G7 fails:** local eval is catastrophically worse than baseline. STOP — do not proceed to cluster. Investigate before committing. + +### Step 6.5 — No commit (verification only) + +- [ ] No commit for Task 6 if all gates pass. The local Tier 1.5 PASS state is the precondition for cluster submission (Task 8). + +--- + +## Task 7 — Push the Phase 7a F2 commits (~5 min) + +### Step 7.1 — Verify the commit chain + +- [ ] Run: + +```bash +git log --oneline ml-alpha-regime-observer ^main | head -10 +``` + +Expected: four new commits on top of `0463e44e0`: +1. `feat(ml-alpha): Phase 7a F2 — allocate ISV slots 817-818 for Q-centered distill target` (Task 1) +2. `feat(ml-alpha): Phase 7a F2-diag — 13 diag fields for F2 measurement` (Task 1.5) +3. `test(ml-alpha): Phase 7a F2 — GPU-oracle invariants for Q-centered distill target` (Task 2) +4. `feat(ml-alpha): Phase 7a F2 — Q-centered hinged-advantage distill target` (Task 3) + +### Step 7.2 — Push the branch + +- [ ] Run: + +```bash +git push origin ml-alpha-regime-observer 2>&1 | tail -10 +``` + +- [ ] If the push is rejected because the branch has diverged from the remote, STOP — do NOT force-push. Inspect the remote state with `git fetch origin && git log origin/ml-alpha-regime-observer ^HEAD` and reconcile manually. + +--- + +## Task 8 — Cluster Tier 2 verification (~80 min wallclock, ~15 min interactive) (~15 min) + +### Step 8.1 — Submit the cluster run + +- [ ] Get the HEAD commit SHA: + +```bash +git rev-parse HEAD +``` + +Record this SHA (call it `$SHA`). + +- [ ] Submit Tier 2 with band ON, single-head, F2 enabled by default: + +```bash +cd /home/jgrusewski/Work/foxhunt +./scripts/argo-alpha-rl.sh \ + --sha "$SHA" \ + --branch ml-alpha-regime-observer \ + --n-steps 20000 \ + --n-eval-steps 5000 \ + --n-backtests 1024 \ + --seed 42 \ + --use-multi-head-policy 0 \ + --band-enabled 1 \ + --watch 2>&1 | tail -30 +``` + +Expected: Argo submits the workflow; the `--watch` flag streams logs. Total wallclock ≈ 80 min (compile ~15 min, train ~50 min, eval ~15 min). + +### Step 8.2 — Capture the run name + monitor + +- [ ] The Argo submit will print the workflow name (something like `alpha-rl-XXXXX`). Record it. +- [ ] Monitor progress via: + +```bash +argo logs -n argo --follow 2>&1 | tail -20 +``` + +Or via the cluster dashboard. The run uploads outputs to `/feature-cache/alpha-rl-runs/$SHA/` per `pearl_cluster_log_is_ground_truth`. + +### Step 8.3 — Evaluate F2-G8/G9/G10/G11 (cluster outcome gates per spec §5.1.2) + +When the run completes, fetch the eval_summary.json: + +- [ ] Locate the eval artifact (the workflow output path is printed in the Argo logs; typically `s3://foxhunt-cache/alpha-rl-runs/$SHA/eval_summary.json`). + +- [ ] **F2-G8 (eval pnl primary outcome).** Pass condition: `eval_summary.total_pnl_usd ≥ h6btt_baseline + $5_000_000`. The h6btt baseline is approximately `-$4.18 × 500 × 1024 ≈ -$2.14M` (read from the h6btt cluster summary). So F2-G8 pass is approximately `eval_pnl ≥ +$2.86M`. + +- [ ] **F2-G9 (no new attractor).** Read `policy_diagnostic.max_action_share` (added in Task 1.5) at eval step 5000 and step 15000. Pass condition: `max_action_share < 0.80`. Also record `policy_diagnostic.modal_action_id` — if the run fails, this identifies WHICH attractor (Hold = re-collapse; LongLarge/ShortLarge = directional collapse; TrailLoosen = trail attractor). + +- [ ] **F2-G10 (eval action entropy).** `eval_summary.eval_action_entropy_mean ≥ 1.0 nats`. + +- [ ] **F2-G11 (surfer-mode fingerprint).** `wr_ema ≥ 0.45` AND `avg_hold_steps ≥ 15`. + +- [ ] **F2-MECHANISM (informational; new in Task 1.5).** Read at eval step 5000: + - `policy_diagnostic.f2_hinge_zero_rate` — should be in `[0.4, 0.8]` (mechanism active and discriminating). Outside this range = mechanism broken: ~1.0 = no above-baseline action (Q has not learned an Open edge); ~0.0 = no clamping (every action above baseline, hinge is vacuous). + - `policy_diagnostic.f2_advantage_max_mean` — should be strictly > 0 once training has produced any Open edge. Zero means the hinge clamped everything (no Open action exceeds baseline in any batch). + - `policy_diagnostic.f2_target_argmax_action` vs `policy_diagnostic.modal_action_id` — divergence between these is the "F2 actively reshaping gradient" signature. + - `policy_diagnostic.hold_frac_flat` trajectory across the eval window — same Tier 1.5 gate at cluster scale. + +### Step 8.4 — Attribution per spec §5.3 (decision table) + +| Outcome | Next action | +|---|---| +| F2-G8 PASS, F2-G9 PASS, F2-G11 PASS | **SHIP F2 ALONE.** Surfer principle preserved; gradient axis was load-bearing. Close Phase 7a; do not proceed to Phase 7b. | +| F2-G8 FAIL, mechanism gates G1/G2/G3 PASS, G5 PASS | F2 necessary but insufficient. Surface to user per spec §4.4. Decision: accept surfer→trend trade-off → Phase 7b (F5 on top); else escalate to encoder/alignment/F4 work. | +| F2-G8 FAIL, F2-G5 FAIL (Hold% from flat < 5% or > 95%) | F2 broke surfer patience in the wrong way. STOP. Do NOT proceed to F5 (F5 worsens the over-action case). Investigate before any further work. | +| F2-G8 PASS, F2-G9 FAIL (single-action dominance > 80%) | F2 surfaced a directional collapse. STOP. Do NOT add F5 reflexively. Escalate to encoder investigation or F4. | +| F2-G7 (determinism) FAIL anywhere | F2 introduced non-determinism. Audit the kernel diff. Most likely an unsynchronized read in the new branch — though Step 3.2 only touched the `if (a == 0)` block. | + +### Step 8.5 — Update the spec decision log + +- [ ] Per spec §11, the decision log entry should be updated with the actual cluster outcome. The implementer is responsible for either updating the spec or surfacing the result to the user so the spec can be updated. + +### Step 8.6 — No further commits in Phase 7a + +- [ ] Phase 7a ends after Step 8.5. Phase 7b (F5 on top of F2) is a separate plan document conditional on §4.6 + explicit user acceptance. + +--- + +## Execution Handoff + +This plan can be executed in two modes: + +### Mode A — Subagent-Driven (recommended) + +Use the `superpowers:subagent-driven-development` skill. Dispatch each Task (0 through 8) as an independent subagent invocation, passing the task's steps and verification gates. The subagent reports PASS/FAIL per task; the orchestrator decides whether to proceed to the next task. + +Advantages: clean task boundaries, isolated context per task, no inter-task contamination, easy STOP-on-surprise enforcement. + +### Mode B — Inline Execution + +Use the `superpowers:executing-plans` skill. Walk through Tasks 0-8 in sequence in a single session, checking off `- [ ]` items as you go. At each STOP-on-surprise decision point, surface to the user immediately rather than improvising a fix. + +Advantages: fewer agent invocations, lower latency, easier debugging when steps interleave. + +Either mode is acceptable. Both honor the `feedback_investigation_first_falsification_methodology` discipline: predetermined gates, STOP-on-unexpected-finding, no skipping verification steps to "save time." + +--- + +## Plan-level invariants (reminders the implementer must respect) + +1. **No `--no-verify` push.** Hooks must run. +2. **No commit amends across Task boundaries** — one logical change per commit, fresh commits only. +3. **No deletion of pre-existing tests.** Phase 5, band, multi-head-policy tests are the substrate. +4. **No reward-kernel changes in Phase 7a.** F2 is gradient-axis only; reward shaping is untouched. +5. **No band-mask changes in Phase 7a.** Band composition is verified via reconnaissance; F2 is mechanically disjoint. +6. **STOP-on-surprise binds at every gate.** Any unexpected behavior surfaces to the user, not into another iteration of "tile a fix on top." +7. **The empty no-op-set case is a NON-ISSUE in Phase 7a.** F5 is not active in this phase; the no-op-set {Hold, TrailTighten, TrailLoosen} is always populated. Spec §7 uncertainty #7 only matters in Phase 7b. + +--- diff --git a/docs/superpowers/plans/2026-06-05-reward-pnl-alignment-restoration-implementation.md b/docs/superpowers/plans/2026-06-05-reward-pnl-alignment-restoration-implementation.md new file mode 100644 index 000000000..891dc0720 --- /dev/null +++ b/docs/superpowers/plans/2026-06-05-reward-pnl-alignment-restoration-implementation.md @@ -0,0 +1,465 @@ +# Reward–PnL Alignment Restoration — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Restore reward↔pnl alignment by making the surfer-scaffold weight (slot 753) actually controllable at runtime, prove the credit-assignment co-fix is viable with a cheap diagnostic gate (RA-G4), and only then complete the GAE multi-epoch update — displacing F4 oracle pretraining as the next intervention. + +**Architecture:** Two independent defects. **D1 (alignment):** `rl_surfer_scaffold_controller.cu:101` overwrites slot 753 every step, making its 0.0 "pure-pnl" bootstrap cosmetic — fixed by a force-pin gate. **D2 (credit assignment):** aligned reward is sparse and the advantage kernel is done-gated 1-step TD — fixed (conditionally) by completing the already-scaffolded GAE multi-epoch PPO update. A cheap diagnostic (RA-G4) gates the expensive D2 build. + +**Tech Stack:** Rust 1.85 (ml-alpha crate), CUDA 12.4 (pre-compiled cubins via build.rs, no nvrtc), ISV bus (device-resident f32 control vector), Tier 1.5 local smoke (RTX 3050, b=128), Argo cluster (L40S, b=1024). + +**Spec:** `docs/superpowers/specs/2026-06-05-reward-pnl-alignment-restoration.md` + +--- + +## Phase plan & gates (read first) + +| Phase | What | Gates | STOP-if | +|---|---|---|---| +| **A** | Force-pin slot 753 (D1) + re-measure harness | RA-G1 (weight==0 when pinned), RA-G2 (determinism), RA-G3 (Pearson≥0.70) | RA-G1/G2 fail | +| **B** | GAE-advantage↔pnl viability diagnostic | **RA-G4 (advantage↔pnl Pearson≥0.70)** | RA-G4<0.70 → STOP, do NOT build Phase C | +| **C** | Complete 1B-C multi-epoch PPO over rollout buffer (D2) — CONDITIONAL | RA-G5 (no eval regression), RA-G2 | RA-G5 regresses | +| **D** | Cluster verify | RA-G6 (cluster eval > sk2z9) | branch per spec §5 | + +**Phase A and B are fully specified below. Phase C and D are structured outlines** — Phase C's exact code depends on the RA-G4 verdict and a rollout-scaffold audit (its first task), so per `feedback_investigation_first_falsification_methodology` it is NOT pre-coded speculatively. Expand Phase C into full TDD tasks only after RA-G4 passes. + +--- + +## File structure + +| File | Change | Responsibility | +|---|---|---| +| `crates/ml-alpha/src/rl/isv_slots.rs` | Add slot 824, bump `RL_SLOTS_END`→825 | Slot constant + invariant test | +| `crates/ml-alpha/cuda/rl_surfer_scaffold_controller.cu` | Add `#define` + force-pin early-return | The D1 gate (device-side) | +| `crates/ml-alpha/src/trainer/integrated.rs` | Env-gated bootstrap entry + diag field | Wire pin flag; observability | +| `crates/ml-alpha/src/trainer/eval_diag_emission.rs` | Bump `EXPECTED_LEAVES` +1 | Keep leaf-count gate honest | +| `scripts/measure_reward_alignment.py` | New | Reusable Pearson(reward,pnl) gate harness | +| `crates/ml-alpha/src/trainer/rollout_collection.rs` | (Phase B) per-close pnl capture + advantage diag | RA-G4 plumbing | + +--- + +## Phase A — Force-pin slot 753 (D1) + +### Task A1: Add ISV slot 824 + bump RL_SLOTS_END + +**Files:** +- Modify: `crates/ml-alpha/src/rl/isv_slots.rs:2346` (after slot 823) and `:2376` (`RL_SLOTS_END`) +- Test: `crates/ml-alpha/src/rl/isv_slots.rs` (inline `#[cfg(test)]`) + +- [ ] **Step 1: Write the failing test** + +Add to the `#[cfg(test)] mod tests` block in `isv_slots.rs` (create the block at end of file if none exists): + +```rust +#[test] +fn force_pin_slot_allocated_below_end() { + // Phase A (2026-06-05) — reward-pnl alignment force-pin. + assert_eq!(RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX, 824); + assert_eq!(RL_SLOTS_END, 825); + assert!(RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX < RL_SLOTS_END); + // Must not collide with the prior last slot (F5 mask = 823). + assert_ne!(RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX, RL_F5_STATE_MASK_ENABLED_INDEX); +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `SQLX_OFFLINE=true cargo test -p ml-alpha --lib force_pin_slot_allocated_below_end` +Expected: FAIL — `RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX` not found (does not compile). + +- [ ] **Step 3: Add the slot constant and bump the end** + +Insert after line 2346 (`pub const RL_F5_STATE_MASK_ENABLED_INDEX: usize = 823;`): + +```rust +/// Phase A reward-pnl alignment (2026-06-05) — force-pin gate for the +/// surfer-scaffold weight controller. When `> 0.5`, +/// `rl_surfer_scaffold_controller.cu` returns WITHOUT writing slot 753, +/// so slot 753 holds its bootstrap value (pure-pnl = 0.0) for the whole +/// run instead of being overwritten to w≈0.97 every step. +/// +/// Root cause it fixes: the slot-753 0.0 bootstrap was cosmetic — the +/// controller's unconditional write re-enabled all four Phase-5 +/// non-potential shaping terms, giving Pearson(reward,pnl)=0.28 (gate +/// 0.70). See `pearl_reward_misalign_blocks_f4_slot753_override`. +/// +/// Bootstrap 0.0 (OFF — controller writes as before). Set to 1.0 via the +/// `FOXHUNT_PIN_SURFER_SCAFFOLD=1` env flag to pin pure-pnl. Clamp +/// `{0.0, 1.0}` (binary). +pub const RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX: usize = 824; +``` + +Then update the `RL_SLOTS_END` doc-chain (line ~2375) and value (line 2376): + +```rust +/// Post-Phase 7b F5 (state-conditional action mask gate, slot 823): 823. +/// Post-Phase A reward-pnl alignment (surfer-scaffold force-pin, slot 824): 824. +pub const RL_SLOTS_END: usize = 825; +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `SQLX_OFFLINE=true cargo test -p ml-alpha --lib force_pin_slot_allocated_below_end` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/ml-alpha/src/rl/isv_slots.rs +git commit -m "feat(ml-alpha): Phase A — RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX slot 824" +``` + +--- + +### Task A2: Add the force-pin gate to the controller kernel + +**Files:** +- Modify: `crates/ml-alpha/cuda/rl_surfer_scaffold_controller.cu` (add `#define` near line 57; add early-return near line 65) + +> CUDA kernels have no unit-test harness in this repo (verified via smoke + determinism, Tasks A6/A7). This task is a build-and-inspect step; behavioral verification is RA-G1 in Task A6. + +- [ ] **Step 1: Add the slot `#define`** + +After line 57 (`#define RL_SURFER_WARMUP_TRADES_INDEX 756`), add: + +```cuda +#define RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX 824 +``` + +- [ ] **Step 2: Add the early-return gate** + +In `rl_surfer_scaffold_controller`, immediately after the thread guard at line 65 (`if (threadIdx.x != 0 || blockIdx.x != 0) return;`), insert: + +```cuda + // Phase A force-pin (2026-06-05): when set, leave slot 753 at its + // bootstrap value (pure-pnl) instead of overwriting it. Fixes the + // cosmetic-bootstrap bug (pearl_reward_misalign_blocks_f4_slot753_override). + if (isv[RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX] > 0.5f) { + return; // do NOT write isv[RL_SURFER_SCAFFOLD_WEIGHT_INDEX] + } +``` + +- [ ] **Step 3: Build to verify the cubin compiles** + +Run: `SQLX_OFFLINE=true cargo build -p ml-alpha --profile=dev-release 2>&1 | tail -5` +Expected: build succeeds (build.rs recompiles `rl_surfer_scaffold_controller.cubin` via nvcc; no errors). + +- [ ] **Step 4: Commit** + +```bash +git add crates/ml-alpha/cuda/rl_surfer_scaffold_controller.cu +git commit -m "feat(ml-alpha): Phase A — force-pin gate in surfer-scaffold controller" +``` + +--- + +### Task A3: Env-gated bootstrap entry + +**Files:** +- Modify: `crates/ml-alpha/src/trainer/integrated.rs:4416` (bootstrap table, after the F5 entry) + +- [ ] **Step 1: Read the env flag and add the bootstrap entry** + +Immediately before the closing `];` of the `isv_constants` array at line 4417, after the F5 entry (line 4416), insert: + +```rust + // Phase A reward-pnl alignment (2026-06-05) — surfer-scaffold + // force-pin. Env-gated per the FOXHUNT_USE_ROLLOUT precedent + // (commit 779c03b9d): unset/0 → 0.0 (controller writes slot + // 753 as before, behaviour unchanged); =1 → 1.0 (controller + // skips its write, slot 753 holds its 0.0 pure-pnl bootstrap). + (crate::rl::isv_slots::RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX, { + let pin: bool = std::env::var("FOXHUNT_PIN_SURFER_SCAFFOLD") + .ok() + .and_then(|v| v.parse::().ok()) + .map(|n| n > 0) + .unwrap_or(false); + if pin { 1.0_f32 } else { 0.0_f32 } + }), +``` + +- [ ] **Step 2: Build to verify it compiles** + +Run: `SQLX_OFFLINE=true cargo build -p ml-alpha --profile=dev-release 2>&1 | tail -5` +Expected: build succeeds. + +- [ ] **Step 3: Commit** + +```bash +git add crates/ml-alpha/src/trainer/integrated.rs +git commit -m "feat(ml-alpha): Phase A — FOXHUNT_PIN_SURFER_SCAFFOLD env flag → slot 824 bootstrap" +``` + +--- + +### Task A4: Diag observability field + leaf-count bump + +**Files:** +- Modify: `crates/ml-alpha/src/trainer/integrated.rs:12576` (after `surfer_scaffold_weight`) +- Modify: `crates/ml-alpha/src/trainer/eval_diag_emission.rs` (`EXPECTED_LEAVES`) + +- [ ] **Step 1: Add the diag field** + +Immediately after the `surfer_scaffold_weight` entry (line 12575-12576), insert: + +```rust + "surfer_scaffold_force_pin": + isv[crate::rl::isv_slots::RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX], +``` + +- [ ] **Step 2: Find the current EXPECTED_LEAVES value** + +Run: `grep -n "EXPECTED_LEAVES" crates/ml-alpha/src/trainer/eval_diag_emission.rs` +Expected: one `const EXPECTED_LEAVES: usize = N;` line. Note `N`. + +- [ ] **Step 3: Bump EXPECTED_LEAVES by 1** + +Edit that constant from `N` to `N + 1` (one new leaf added under `rewards`). Update any adjacent comment that records the count. + +- [ ] **Step 4: Verify the leaf-count assertion passes** + +Run: `SQLX_OFFLINE=true cargo test -p ml-alpha --lib eval_diag 2>&1 | tail -20` +Expected: the diag leaf-count test passes (if it fails with `actual N+1 expected N`, the bump in Step 3 was missed). + +- [ ] **Step 5: Commit** + +```bash +git add crates/ml-alpha/src/trainer/integrated.rs crates/ml-alpha/src/trainer/eval_diag_emission.rs +git commit -m "feat(ml-alpha): Phase A — surfer_scaffold_force_pin diag field (+1 leaf)" +``` + +--- + +### Task A5: Reusable alignment-measurement harness + +**Files:** +- Create: `scripts/measure_reward_alignment.py` + +- [ ] **Step 1: Write the harness** + +```python +#!/usr/bin/env python3 +"""Measure reward<->pnl alignment from a diag.jsonl. + +RA-G3 gate: Pearson(rewards.sum, trading.realized_pnl_usd_delta) >= 0.70. +Usage: python3 scripts/measure_reward_alignment.py [--json] +Exit 0 if Pearson >= 0.70, 1 if below, 2 on bad input. +""" +import json, os, sys + +def diag_path(arg): + if os.path.isdir(arg): + return os.path.join(arg, "diag.jsonl") + return arg + +def pearson(x, y): + n = len(x) + if n < 3: + return None + mx, my = sum(x)/n, sum(y)/n + sxy = sum((a-mx)*(b-my) for a, b in zip(x, y)) + sxx = sum((a-mx)**2 for a in x) + syy = sum((b-my)**2 for b in y) + if sxx == 0 or syy == 0: + return None + return sxy/(sxx*syy)**0.5 + +def main(): + args = [a for a in sys.argv[1:] if not a.startswith("--")] + want_json = "--json" in sys.argv + if not args: + print("usage: measure_reward_alignment.py ", file=sys.stderr) + return 2 + path = diag_path(args[0]) + if not os.path.exists(path): + print("no diag at %s" % path, file=sys.stderr) + return 2 + rew, pnl = [], [] + for line in open(path): + try: + d = json.loads(line) + except Exception: + continue + r = d.get("rewards", {}).get("sum") + p = d.get("trading", {}).get("realized_pnl_usd_delta") + if r is None or p is None: + continue + rew.append(float(r)); pnl.append(float(p)) + P = pearson(rew, pnl) + both = [(a, b) for a, b in zip(rew, pnl) if a != 0 and b != 0] + sign = (sum(1 for a, b in both if (a > 0) == (b > 0)) / len(both)) if both else None + gate = 0.70 + ok = P is not None and P >= gate + result = {"pearson": P, "sign_agreement": sign, "n_rows": len(rew), + "gate": gate, "pass": ok} + if want_json: + print(json.dumps(result)) + else: + print("rows=%d Pearson=%s sign_agree=%s gate>=%.2f -> %s" % ( + len(rew), + "n/a" if P is None else "%.4f" % P, + "n/a" if sign is None else "%.3f" % sign, + gate, "PASS" if ok else "FAIL")) + return 0 if ok else 1 + +if __name__ == "__main__": + sys.exit(main()) +``` + +- [ ] **Step 2: Smoke the harness on the existing baseline diag (expect FAIL = 0.28)** + +Run: `python3 scripts/measure_reward_alignment.py /tmp/foxhunt-mid-smoke --` +Expected: `rows=2000 Pearson=0.2775 sign_agree=0.655 gate>=0.70 -> FAIL` (exit 1). Confirms the harness reproduces the baseline measurement. + +- [ ] **Step 3: Commit** + +```bash +git add scripts/measure_reward_alignment.py +git commit -m "feat(scripts): reward-pnl alignment measurement harness (RA-G3 gate)" +``` + +--- + +### Task A6: Tier 1.5 smoke with pin engaged → RA-G1 + RA-G3 + +**Files:** none (behavioral verification) + +- [ ] **Step 1: Run the pinned mid-smoke** + +Run: +```bash +FOXHUNT_PIN_SURFER_SCAFFOLD=1 ./scripts/local-mid-smoke.sh --seed 42 --out /tmp/foxhunt-pin-smoke +``` +Expected: completes ~10 min, writes `/tmp/foxhunt-pin-smoke/diag.jsonl` + `eval_summary.json`, no NaN. + +- [ ] **Step 2: RA-G1 — verify slot 753 held at 0 the whole run** + +Run: +```bash +python3 -c "import json; vs=[json.loads(l)['rewards']['surfer_scaffold_weight'] for l in open('/tmp/foxhunt-pin-smoke/diag.jsonl')]; print('min=%.4f max=%.4f pin_field=%.1f'%(min(vs),max(vs), json.loads(open('/tmp/foxhunt-pin-smoke/diag.jsonl').readline())['rewards']['surfer_scaffold_force_pin']))" +``` +Expected: `min=0.0000 max=0.0000 pin_field=1.0`. **RA-G1 PASS** iff weight is 0.0 across all rows and the pin field reads 1.0. +**STOP-if-fail:** weight ≠ 0 means the kernel gate (Task A2) is not wired to the controller write — re-verify the early-return placement is BEFORE the `isv[753]=w` line. + +- [ ] **Step 3: RA-G3 — measure restored alignment** + +Run: `python3 scripts/measure_reward_alignment.py /tmp/foxhunt-pin-smoke` +Expected: `-> PASS` with Pearson ≥ 0.70 (pure-pnl reward ≈ pnl-delta by construction; this confirms the shaping was the misalignment source). +**Note (spec §7 risk 1):** RA-G3 is near-tautological — it confirms the pin removed the shaping, NOT that the agent trades better. The real viability test is RA-G4 (Phase B). If RA-G3 is still < 0.70 after pinning, the residual is magnitude clamp/popart (`pearl_popart_blind_to_session_signals`) → raise a clamp sub-spec before proceeding. + +--- + +### Task A7: Determinism → RA-G2 + +**Files:** none + +- [ ] **Step 1: Run determinism check with pin engaged** + +Run: +```bash +FOXHUNT_PIN_SURFER_SCAFFOLD=1 ./scripts/determinism-check.sh --quick +``` +Expected: exit 0 — same-seed runs bit-equal (`diag.jsonl` byte-equal modulo `elapsed_s`, `eval_summary.json` byte-equal). +**STOP-if-fail:** the pin/diag introduced nondeterminism — bisect via `--debug-dump-rl`. (Adding a constant slot + early-return should be deterministic; an ISV-length-dependent checksum divergence is the most likely culprit — investigate the ISV checksum path.) + +> Note: this gate is same-seed bit-equality, NOT bit-equality-vs-HEAD. The new slot 824 + diag leaf intentionally change the schema vs HEAD; that is expected and not a regression. + +--- + +### Task A8: Phase A wrap-up commit + verdict record + +- [ ] **Step 1: Record the Phase A verdict** + +Append a one-line result to the spec's decision log (RA-G1/G2/G3 pass/fail + measured Pearson). If all pass, Phase A is green and Phase B is unlocked. + +```bash +git add docs/superpowers/specs/2026-06-05-reward-pnl-alignment-restoration.md +git commit -m "docs: Phase A verdict — RA-G1/G2/G3 results" +``` + +--- + +## Phase B — Credit-assignment viability diagnostic (RA-G4) + +> **Goal:** measure `Pearson(GAE_advantage_at_open, realized_trade_pnl)` cheaply (no training rearchitecture) to decide whether the expensive Phase C build is justified. The rollout buffer (`rollout_buffer.rs`) already computes `advantages_d`/`returns_d` via `compute_gae`; collection (`rollout_collection.rs`) fills it as a side-effect under `FOXHUNT_USE_ROLLOUT=1`. What it does NOT currently store is per-step realized USD pnl — Task B1 audits that gap. + +### Task B1: Audit the rollout-collection → pnl mapping (investigation, no code) + +**Deliverable:** a written finding (append to the spec) answering: +- [ ] Where does `rollout_collection.rs` capture per-step data, and is `trading.realized_pnl_usd_delta` (or the device buffer behind it) reachable at the same `(b, t)` granularity as `advantages_d`? +- [ ] At which `t` does a trade "close" (done flag) and where is its realized pnl available on-device? +- [ ] Can the open-step `advantages_d[b, t_open]` be paired with the realized pnl of the position that opened at `t_open`? (trade-tracking through the buffer) + +**STOP-if:** the pnl signal is not reachable at buffer granularity without a >200 LOC collection refactor → fall back to the **coarser RA-G4 proxy** in Task B2-alt (per-step diag correlation) and note the precision loss. + +### Task B2: Emit GAE-advantage + realized-pnl pairs to diag + +Based on B1, add a deterministic diag emission (per `pearl_fleet_fraction_not_aggregate` — emit fleet fractions, not just aggregates) of: +- mean/median `advantages_d` at close (done) steps, +- co-located `realized_pnl_usd_delta`, +- per-batch fraction with advantage-sign == pnl-sign. + +Mirror the existing per-batch reduction pattern (the F2 diag counters, `RL_F2_DIAG_*`). Wire under `FOXHUNT_USE_ROLLOUT=1` only (bit-equal when off). + +### Task B2-alt (fallback): coarse per-step proxy + +If B1 STOPs: compute `Pearson(returns_d-mean-at-close, realized_pnl_usd_delta)` host-side from diag rows (cross-granularity, weaker), and label the RA-G4 result "coarse proxy — precision-limited." + +### Task B3: Run smoke and evaluate RA-G4 + +- [ ] Run: `FOXHUNT_PIN_SURFER_SCAFFOLD=1 FOXHUNT_USE_ROLLOUT=1 ./scripts/local-mid-smoke.sh --seed 42 --out /tmp/foxhunt-pin-gae-smoke` +- [ ] Compute `Pearson(GAE_advantage_at_open, realized_trade_pnl)` from the emitted diag. +- [ ] **RA-G4 gate: ≥ 0.70.** + +**STOP-if RA-G4 < 0.70:** Per spec §5.4, do NOT build Phase C and do NOT revisit F4. A failure means multi-step credit assignment on aligned reward still does not recover pnl-aligned advantages → the fault is deeper (reward shape itself or genuine regime shift). Escalate to a reward-shape / regime investigation. Record the verdict in the spec decision log. + +### Task B4: Determinism + commit + +- [ ] `FOXHUNT_PIN_SURFER_SCAFFOLD=1 FOXHUNT_USE_ROLLOUT=1 ./scripts/determinism-check.sh --quick` → exit 0 (RA-G2). +- [ ] Commit the diagnostic + spec verdict. + +--- + +## Phase C — Complete 1B-C multi-epoch PPO over rollout buffer (D2) — CONDITIONAL on RA-G4 + +> **Do NOT expand into TDD tasks until RA-G4 passes.** Phase C is the substantive (~3-6 day) build that replaces the per-step done-gated TD update with a multi-epoch PPO update over the GAE buffer. Its exact code depends on the rollout-scaffold audit (Task C1) and must honour `pearl_foxhunt_trainer_is_genuinely_single_step` (the per-step training split is the load-bearing change). + +Structured task outline: +- **C1 (audit):** map the current per-step PPO update path in `integrated.rs` (the `ss_pi_grad_logits_d` distill write site ~5607 + Adam steps) and the `rollout_collection.rs` collection loop. Identify exactly where per-step training is replaced by buffer-replay multi-epoch update. Produce the Phase C TDD task list from this audit. +- **C2:** implement the multi-epoch minibatch loop over `RolloutBuffer` (re-evaluate heads on stored `h_t_d`, PPO ratio from `log_pi_old_d`, advantages from `advantages_d`), gated behind `FOXHUNT_USE_ROLLOUT=1`. +- **C3:** determinism (RA-G2) + bit-equality-when-off. +- **C4:** mid-smoke with pin + rollout → **RA-G5 (eval pnl ≥ −$415,237 F2+F5 SOTA, not worse)**. STOP-if-regresses (recall Phase 1A regressed here). + +**Gates:** RA-G5, RA-G2. **STOP-if:** RA-G5 regresses → diagnose with fleet-fraction diag before cluster. + +--- + +## Phase D — Cluster verify (RA-G6) — CONDITIONAL on Phase C + +Structured checklist: +- **D1:** confirm local HEAD == origin (push), apply Argo template (`feedback_argo_template_must_apply`). +- **D2:** submit Tier 2 — `./scripts/argo-alpha-rl.sh --n-steps 20000 --n-eval-steps 5000 --n-folds 3 --fold-idx 1 --n-backtests 1024 --band-enabled 1` with `FOXHUNT_PIN_SURFER_SCAFFOLD=1 FOXHUNT_USE_ROLLOUT=1` threaded into the workflow env (verify the Argo template forwards these env vars; add them if not). +- **D3:** wait ~70 min; analyze `eval_summary.json`. +- **Gate RA-G6:** `total_pnl_usd` > F2+F5 sk2z9 cluster baseline. +- **Branch (spec §5):** RA-G6 pass → reward alignment was the lever; commit. RA-G6 fail with RA-G4 pass → encoder/generalization branch legitimately isolated (NOT another reward reshape; consider IDAAC decoupling / overfit-rejection ensemble, or — only now — a properly-done warm-start). + +--- + +## Self-review + +**Spec coverage:** +- D1 force-pin → Tasks A1-A3 ✓ | RA-G1 → A6 ✓ | RA-G2 → A7, B4 ✓ | RA-G3 → A5, A6 ✓ +- §3.4 re-measure harness → A5 ✓ | §3.2 RA-G4 diagnostic → Phase B ✓ | RA-G4 → B3 ✓ +- §3.3 1B-C (D2) → Phase C ✓ | RA-G5 → C4 ✓ | RA-G6 → D3 ✓ +- §7 risk 1 (RA-G3 tautology) called out in A6 Step 3 ✓ | §7 risk 3 (magnitude clamp) called out in A6 Step 3 ✓ | §7 risk 4 (1B-C cost) → Phase C kept conditional + audit-first ✓ +- Phase 1A landmine (§1.3) → RA-G4 gates the D2 spend; RA-G5 watches for the regression ✓ + +**Placeholder scan:** Phase A and B-task-B1/B3/B4 contain complete code/commands. Phase B Tasks B2/B2-alt and all of Phase C/D are *intentionally* structured outlines (not placeholders) because their exact code is gated behind RA-G4 and a scaffold audit — pre-coding them would violate the investigation-first methodology and likely be wrong. Each carries its gate and STOP rule. + +**Type/name consistency:** `RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX` (824), `FOXHUNT_PIN_SURFER_SCAFFOLD` env flag, `surfer_scaffold_force_pin` diag key — used consistently across A1/A2/A3/A4/A6. `RL_SLOTS_END=825` consistent A1. Harness path `scripts/measure_reward_alignment.py` consistent A5/A6. + +--- + +## Execution note + +Phase A is the fully-coded, low-risk unit (force-pin + harness + 3 gates, ~0.5-1 day) and is independently valuable: even if Phase B STOPs, Phase A leaves slot 753 actually controllable and a reusable alignment gate in CI. Recommend executing Phase A end-to-end, reviewing RA-G1/G2/G3, then deciding Phase B with the RA-G4 verdict in hand. diff --git a/docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md b/docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md new file mode 100644 index 000000000..c21b8617c --- /dev/null +++ b/docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md @@ -0,0 +1,1004 @@ +# Bellman-Target Foundation Reshape — Breaking the Structural Inaction Bias + +**Date:** 2026-06-04 +**Status:** Design — pre-implementation, candidate-comparison; NOT a plan +**Branch target:** TBD (off Phase 5 `0463e44e0` once the chosen option is selected) +**Estimated effort (recommended option only):** see §6 +**Scope discipline:** This spec analyses FIVE candidate interventions, recommends a disciplined STAGED pair (Phase 7a = F2 alone; Phase 7b = F5 conditional on F2 outcome), and defines distinguishable gates. It does NOT prescribe LOC, code, or implementation order. A separate `plans/` document is required before any coder dispatch. + +**Revision log:** + +- **Revision 1 (2026-06-04, "a").** Original spec analysed F1-F4 and recommended F2-alone. +- **Revision 2 (2026-06-04, "b").** User critique identified a category error in F3 (conflated global vs state-conditional action removal). Spec added **F5 — state-conditional action availability mask**, and recommended *F5 first, F2 conditional* on the argument that F5 has the highest pre-criterion score and addresses the choice-set at the structural level. +- **Revision 3 (2026-06-04, "c", THIS REVISION).** User critique identified that F5 violates the documented **surfer trading principle** (`pearl_dd049d9a4_surfer_baseline_verified`, `pearl_two_alpha_modes_surfer_vs_trend`): F5's hard mask forces opens when flat, destroying patience-while-waiting-for-setup, which is the structural ability the surfer alpha mode depends on. F2 preserves this principle (its hinge zeroes Open mass when `E_Q(Open) ≤ E_Q(Hold)`, so the agent retains the *learned* discretion to wait). The recommendation flips: **Phase 7a = F2 alone; Phase 7b = F5 conditional**, with F5 explicitly framed as a fallback that accepts surfer→trend regression if F2's gradient redirection is insufficient. + +**Executive summary (read before §1).** Foxhunt has demonstrated two profitable trading modes in cluster: + +- **Surfer** (`dd049d9a4` baseline): wr ≈ 0.56, avg_hold ≈ 30 steps, +$10.6M @ step 18k, low-frequency patient entries with high win-rate. +- **Trend-follower** (`6d33f18b7` baseline): wr ≈ 0.36, avg_hold ≈ 17 steps, +$41.5M peak, higher-frequency directional commits with lower win-rate. + +Per `pearl_two_alpha_modes_surfer_vs_trend`, choosing between them is a philosophy decision, and per `feedback_surfer_philosophy_trading` the user has explicitly chosen the surfer mode. The surfer's load-bearing structural property is **patience-while-waiting-for-setup** — the agent must be able to abstain from low-confidence setups (Hold-from-flat) and only commit when its model predicts edge. + +F5 (state-conditional action mask) makes Hold structurally unavailable from flat → forces an Open every flat-state step regardless of belief → **destroys patience-while-waiting** by construction → forces surfer → trend regression. This violates the surfer principle even though F5 mechanically eliminates the h6btt all-Hold attractor. + +F2 (Q-distill advantage target with `max(0, E_Q − baseline)` hinge) preserves the surfer principle by *mathematical derivation*: when `E_Q(Open) ≤ E_Q(Hold)` (no edge detected), the hinge zeroes Open's distill mass and the policy learns "Hold is better, wait"; when `E_Q(Open) > E_Q(Hold)` (edge detected), Open receives positive mass and the agent learns to commit. F2 IS the surfer principle expressed as a loss function. + +This spec therefore recommends **Phase 7a = F2 alone**, with F5 staged as conditional Phase 7b only if F2's outcome gate (eval pnl ≥ h6btt baseline + $5M / session) fails AND F2's mechanism gates pass AND we are prepared to consciously accept surfer → trend regression as the trade-off for breaking the h6btt-class structural collapse. F5 remains a valid fallback; the staging order encodes the surfer-first preference. + +**Linked specs (orthogonal context):** + +- `2026-06-01-reward-policy-alignment-investigation.md` — the precursor investigation that proved the Pearson-misalignment problem and recommended pure-pnl shaping (Phase 3B-Y, then Phase 5) +- `2026-06-02-trainer-rollout-buffer-gae.md` — Phase 1B GAE pivot (still in-flight, not load-bearing here) +- `2026-06-02-multi-head-policy-with-r-multiple.md` — Phase 2A multi-head (the architecture mq2pc deployed) +- `2026-06-03-no-transaction-band-architecture.md` — Phase 4-A band foundation (the architecture h6btt/mq2pc deployed) + +**Linked pearls (load-bearing):** + +- `pearl_foxhunt_pi_trained_by_q_distillation_not_ppo` — π's only gradient source is Q-distill +- `pearl_advantage_kernel_is_done_gated_td_not_gae` — V regression targets are done-gated 1-step TD; T_eff = 1 +- `pearl_foxhunt_trainer_is_genuinely_single_step` — `[B]`-only buffer shape forecloses true trajectory algorithms without multi-week refactor +- `pearl_phase5_term_4_is_almost_potential` — Ng-Harada-Russell verdict on all four Phase 5 shaping terms +- `feedback_going_in_circles_pattern` — the stop-rule that triggered this spec +- `feedback_investigation_first_falsification_methodology` — the methodology this spec is written under + +--- + +## §1. Empirical evidence — three runs, one failure mode + +Three consecutive cluster runs each surfaced a *different* degenerate equilibrium. They are not three bugs. They are three masks over the same architectural pathology: **the Bellman target structurally rewards costless inaction over costly action.** + +### §1.1 The trajectory + +| Cluster | Phase | Failure mode | Cluster bleed | Run-class fix attempted | +|---|---|---|---|---| +| `4hq7c` | Phase 4-C (band foundation) | TrailLoosen unbounded risk; trail grew 376 → 2959 ticks (~$37k/position) | comparable magnitude | Per-unit trail-max ceiling (Phase 5 `0463e44e0`) | +| `h6btt` (killed) | Phase 5 single-head | Hold-collapse to 98%; positions never opened | $4.18/batch/step | Multi-head policy + r-multiple gating (Phase 2A) | +| `mq2pc` (killed) | Phase 5 + multi-head | TrailLoosen-collapse to 90%, then Hold/TrailLoosen oscillation | **$19/batch/step** (4.5× worse than h6btt) | (this spec) | + +The `mq2pc` step-by-step shows the system migrating between degenerate attractors without ever escaping the *class* of degenerate attractors: + +``` +step 100 : 93% Hold, 2% TrailLoosen (attractor A: do-nothing-from-flat) +step 1000 : 14% Hold, 81% TrailLoosen (attractor B: do-nothing-from-position via trail widening) +step 2000 : 4% Hold, 90% TrailLoosen (attractor B saturated; peak bleed $14M / 500 steps) +step 3000 : 40% Hold, 56% TrailLoosen (oscillating between A and B; $1M / 500 steps) +cum pnl @ step 3000: −$58M +linear projection to step 20k: −$386M +``` + +### §1.2 What the trajectory proves + +Three independent claims: + +1. **The attractor is not "Hold" specifically.** Multi-head + r-multiple gating successfully *broke* the Hold attractor — and the system immediately found a new degenerate attractor (TrailLoosen) that is functionally identical. Any intervention that targets *one specific zero-cost action* will be defeated by Q-distill discovering the next zero-cost action. + +2. **The attractor is the *class* of zero-cost actions.** Hold and TrailLoosen share one structural property: they emit no market order, incur no transaction cost, and produce no realized PnL delta. Both have `E[r | a] = 0 + γ·V(s')` while every action that opens or closes a position has `E[r | a] = −cost + γ·V(s')`. Q-distill converges to whichever zero-cost action has the highest E[V(s')] under the current π. + +3. **Goodhart-Skalse 2024 generalizes here.** Their NeurIPS result on proxy-reward fix attenuation in deep RL applies to the *V → Q-Bellman target → softmax(Q/τ) → π distill* chain documented in `pearl_foxhunt_pi_trained_by_q_distillation_not_ppo`. Each link is a learned approximator; each one attenuates the corrective signal of upstream reward fixes. The MTM reward (Phase 5 `0463e44e0`) attempted to *add* a negative reward to underwater positions — and `mq2pc` shows the system simply migrated to the *other* zero-cost action that MTM does not penalize as harshly when trails widen. + +### §1.3 The local Phase 6 SAC α experiment + +A separate local experiment raised SAC α (entropy regularization strength) to MAX and observed that the policy entropy did not increase enough to escape the attractor — when `E_Q[Hold] >> E_Q[Open]`, no amount of entropy push at the policy head can overcome Q-distill's preference for the zero-cost action. This forecloses entropy-side interventions as a foundation fix. **The intervention point must be Q-distill or the Bellman target itself.** + +--- + +## §2. Root cause — why the Bellman target structurally favors inaction + +### §2.1 The math + +For a zero-cost action `a_0` (Hold, TrailLoosen-at-cap, TrailTighten-no-fire, …): + +``` +Q_target(s, a_0) = r(s, a_0) + γ · V(s') + = 0 + γ · V(s') (a) +``` + +For a costly action `a_c` (any open / close / size-change): + +``` +Q_target(s, a_c) = r(s, a_c) + γ · V(s') + = −entry_cost + (realized_pnl_delta) + γ · V(s') (b) +``` + +The expected realized PnL delta is ≈ 0 in a Pearson-misaligned reward regime (see `pearl_reward_signal_anti_aligned_with_pnl`: Pearson 0.11–0.45 vs random baseline 0.5). The entry cost is *deterministic*. So in expectation: + +``` +E[ Q_target(s, a_c) − Q_target(s, a_0) ] = E[realized_delta] − entry_cost + ≈ 0 − entry_cost + < 0 (c) +``` + +**The bias is permanent.** It is independent of training signal, exploration noise, encoder quality, optimizer choice, or any controller adaptation. As long as (i) zero-cost actions exist in the action space and (ii) the Q-distill softmax target sees the un-corrected Bellman-target difference, the policy will prefer them. + +### §2.2 Why Q-distill makes it worse than standard PPO + +In a standard PPO trainer the policy gradient is `∇L = E[ A_t · ∇ log π(a|s) ]` with `A_t = R̂_t − V(s)`. A consistently-negative entry cost would propagate through advantage normalization and the policy would learn a *bounded* preference for inaction. + +In foxhunt's architecture (`pearl_foxhunt_pi_trained_by_q_distillation_not_ppo`): + +- `pi_target = softmax(E_Q / τ)` (kernel `rl_q_pi_distill_grad.cu` line 86) +- The distill gradient is `∇L = λ · (π_θ − π_target)` per-action +- Softmax is *translation-invariant in its input*, but NOT in its target ordering. The target's max action receives the largest gradient share. + +So if `E_Q[a_0] = γV(s')` and `E_Q[a_c] = γV(s') − entry_cost`, then `pi_target` peaks at `a_0` with magnitude `softmax(γV/τ) / sum_a softmax((γV − cost·1[a=c])/τ)`. The gap between the two is exactly the softmax slope at the offset point — **which the agent cannot reduce by training** because the cost is a property of the environment, not the policy. + +### §2.3 Why GAE (Phase 1B) does not fix this + +GAE in `pearl_foxhunt_pi_trained_by_q_distillation_not_ppo` ADDENDUM 2026-06-02e: GAE benefits V regression quality, which propagates downstream through Q → Q-distill → π. **But GAE does not change the cost asymmetry in equation (c).** Better V estimates make the asymmetry *more accurate*, not smaller. GAE is orthogonal to this spec. + +### §2.4 Why T_eff = 1 makes it intractable today + +Per `pearl_advantage_kernel_is_done_gated_td_not_gae`, V regression's advantage is done-gated 1-step TD: `advantages[b] = is_done ? (ret − vt) : 0.0f`. So V only ever sees non-zero learning signal on close events. The path `reward → V → Q-target → π-distill` therefore has its first link operating at T_eff=1 (close events only), and the cost asymmetry in (c) accumulates undistorted into the Q targets at every held step. + +Combined: there is no controller, no shaping term, and no entropy weight that can structurally invert (c). The fix must change either: + +- **F1.** What `r(s, a_0)` is — i.e., make inaction non-zero so (a) and (b) become symmetric +- **F2.** What `softmax target` Q-distill consumes — i.e., remove the absolute Q level so the distill gradient only sees Q-differences from the best no-op +- **F3.** What the action space contains — i.e., eliminate the zero-cost actions +- **F4.** What `Q_target` is initialised to — i.e., warm-start Q with knowledge that Hold is not always optimal +- **F5.** What `A(s)` is — the *available action set* — i.e., keep all actions in the space but hard-mask actions whose availability is incoherent with the current position / trail state. Different from F3: F3 removes actions globally; F5 conditions availability on state. This is the standard Constrained-MDP `A(s) ⊂ A` reformulation (Altman 1999) rather than action-space reduction. + +--- + +## §3. F1-F5 theoretical analysis + +### §3.1 F1 — Bellman-target reshape: add per-step inaction cost + +**Mechanism.** Modify the reward kernel so every zero-cost action incurs a small per-step cost ε: + +``` +r[t] = realized_pnl_delta(t) + w_mtm · unrealized_delta(t) − ε · inaction_indicator(a[t]) +``` + +where `inaction_indicator(a) = 1` iff `a ∈ {Hold, TrailLoosen-no-effect, TrailTighten-no-effect}`. The set of "no-effect" trail actions is determined post-hoc by the trail-mutate kernel detecting the clamp-at-cap branch. + +**Theoretical foundation.** Direct intervention on the Markov reward function. Equation (c) becomes: + +``` +E[Q_target(s, a_c) − Q_target(s, a_0)] = E[realized_delta] − entry_cost − (−ε) + = E[realized_delta] − entry_cost + ε (c') +``` + +With `ε = entry_cost`, the asymmetry vanishes in expectation. The agent's choice between `a_c` and `a_0` is then a function of `E[realized_delta]` — which is the alignment signal the agent is *supposed* to learn from. + +**Citation theorem (foundation).** Ng, Harada, Russell 1999 ("Policy Invariance Under Reward Transformations", ICML): a shaping function `F(s, a, s')` preserves the optimal policy *iff* it has the potential form `F(s,a,s') = γΦ(s') − Φ(s)` for some Φ: S → ℝ. **Pure inaction cost is NOT a potential function** — it depends on `a`, not just `s`. Therefore it CAN (and is designed to) change the optimal policy. This is a feature, not a bug — but it must be acknowledged. + +**Can ε be made potential-conserving?** Define a state potential `Φ(s, a) = −ε · 1[a ∈ no-op]`. Ng-Harada-Russell explicitly require `Φ: S → ℝ` (not `S × A`); the Φ-dependence on action is precisely the case excluded by the theorem. **No.** F1 is provably non-potential. Whether the *intended* policy change (escape from no-op attractors) is the only policy change requires empirical verification (see §5 gates). + +**Cross-reference to `pearl_phase5_term_4_is_almost_potential`.** Per the math agent's audit of Phase 5 shaping, three of four shaping terms were *also* non-potential and *did* destabilise the optimal policy in non-trivial ways. F1 inherits that risk. The mitigation is keeping ε small enough that the *bound on policy distortion* (≤ ε / (1 − γ) in the discounted-return formulation) is dominated by the intended escape benefit. With γ = 0.975 and ε on order of entry_cost ≈ 15 reward-units, the steady-state V-distortion bound is ≈ 600 reward-units per held trajectory. For comparison, average trade PnL magnitude is ~50–200 reward-units. **F1 is borderline non-trivial in distortion.** + +**Expected effect on inaction-bias.** Strong and direct. The reward source itself becomes negative on zero-cost actions; Q learns this within a few epochs; the attractor is removed by construction. + +**Expected failure modes.** + +1. *Overshooting.* If ε is too large, the agent learns to over-trade just to avoid the inaction cost. Goodhart-Skalse style: the agent optimises the proxy (avoid the −ε flag) without optimising pnl. +2. *Definition fragility.* The "inaction" indicator must be precisely defined. TrailLoosen at a non-cap state IS a no-op for pnl but emits a state change (trail distance). The detector must classify it correctly; classification errors are training-data-distribution-dependent. +3. *Q-distill amplification.* Because the cost is in `r`, it propagates through V → Q-Bellman → π-distill at every step. This is *more* signal density than a one-shot fix at the distill target. Could be helpful (more learning signal) or harmful (amplifies misclassification of inaction events). +4. *Interaction with MTM reward (Phase 5).* MTM already gives negative reward to underwater positions. F1 adds negative reward to inaction. Both stack on TrailLoosen-on-underwater (the `mq2pc` attractor). Combined this could create an over-correction toward early-close-at-loss — the classic "loss aversion" pathology. + +**LOC estimate.** Small. ~80–150 LOC: add inaction-indicator computation to `rl_fused_reward_pipeline.cu` Phase 5 section, two new ISV slots (ε_value, enabled-gate), trainer-side ISV bootstrap, controller for ε auto-tune. + +**Predetermined falsification gates.** + +| Gate | Local b=128 (Tier 1.5) | Cluster b=1024 (Tier 2) | +|---|---|---| +| G1 (inaction-share) | Hold% in [10%, 40%] across step 0–2000; TrailLoosen% < 10% | Hold% + TrailLoosen% < 50% at step 5000 | +| G2 (overshoot) | total_trades / 2000 ∈ [50, 400] (vs `mq2pc` baseline ≈ 4 trades/2000 steps) | trades_per_session_per_1000_steps ∈ [10, 300] | +| G3 (eval pnl) | (not measured at Tier 1.5; Tier 2 only) | eval_pnl > 0 OR eval_pnl > h6btt baseline by ≥ $5M / session | + +**Risk vs reward.** Medium-high reward (directly addresses root cause). High risk (proven non-potential; precedent from Phase 5 shows non-potential terms repeatedly destabilise the learned policy). Net: speculative but tractable. + +--- + +### §3.2 F2 — Q-relative-to-best-inaction distill target (advantage-style distill) + +**Mechanism.** Modify `rl_q_pi_distill_grad.cu` so the per-batch softmax target uses `(E_Q − baseline)` instead of `E_Q`: + +``` +baseline(s) = max over a ∈ no-op-set { E_Q(s, a) } or mean E_Q(s, a) over no-ops +π_target(s, ·) = softmax( (E_Q(s, ·) − baseline(s)) / τ ) +``` + +Softmax is mathematically translation-invariant in its input, so `π_target` itself is unchanged after subtracting `baseline`. **The change is in the gradient flowing into π_θ.** + +Wait — this requires careful analysis. Let me redo it. The distill gradient is `grad_distill[a] = λ · (π_θ(a) − π_target(a))`. Subtracting a constant from the *target's pre-softmax input* leaves `π_target` invariant, and the gradient invariant. + +The correct F2 formulation is different: replace `π_target` with one *derived from advantages*. Define: + +``` +adv_Q(s, a) = E_Q(s, a) − baseline(s) +π_target(s, a) = softmax( adv_Q · 1[adv_Q > 0] / τ ) +``` + +i.e., **mask out actions whose Q is no better than the best no-op**. Only super-no-op actions retain mass in the distill target. This is the dueling-value-decomposition idea applied to a distill target. + +Or equivalently (smoother): use a hinge-style target: + +``` +π_target(s, a) ∝ softmax( max(0, E_Q(s, a) − baseline(s)) / τ ) +``` + +**Theoretical foundation.** Wang et al. 2016 ("Dueling Network Architectures for Deep Reinforcement Learning", ICML, arXiv:1511.06581) — the dueling-Q insight: decomposing `Q(s, a) = V(s) + A(s, a)` with `Σ_a A(s, a) = 0` (or `max_a A = 0`) gives stable learning because the absolute Q-level (which is shared across actions and reflects the value of the state) does not contaminate the action-ranking signal. Foxhunt already has dueling Q (Phase 4, per `pearl_phase4_dueling_head_plus_advantage_norm_is_new_sota`), but **the dueling decomposition is internal to the Q head**. The distill target consumes `E_Q` (V + A) and re-introduces the absolute Q level into the policy update. + +F2 brings the dueling insight to the distill target itself. The policy update gradient sees only `A(s, a) − A(s, best-no-op)`, eliminating the cost-asymmetry artifact of equation (c) at the distill stage. + +**Citation chain.** +- Wang et al. 2016, §3 (advantage decomposition is policy-invariant when target is action-ranking). +- Schulman et al. 2015 (GAE), §3 (advantage-based policy updates are unbiased estimators of policy gradient with reduced variance). +- Mnih et al. 2016 (A3C) — advantage-actor-critic for the analogous insight in policy gradient. + +There is no published precedent specifically on "advantage-style Q-distill targets" — distill is normally framed as imitation, not policy improvement. F2 is a small original step from Wang 2016, not a re-discovery. + +**Does F2 provably break the inaction-attractor convergence?** + +Yes, conditional on the no-op-set being correctly defined. Proof sketch: the F2 target has `π_target(a) = 0` for any action whose `E_Q ≤ baseline`. Therefore the distill gradient `λ · (π_θ(a) − 0) = λ · π_θ(a)` is *always* pushing π_θ(a) toward 0 for any no-op-equivalent action — regardless of its absolute Q level. The only way an action survives in π is `E_Q(s, a) > baseline(s)`, which is exactly the condition the agent is *supposed* to learn. + +Caveat: the SAC entropy term still pulls π toward uniform. So `π_θ` cannot collapse to 0 on no-ops as long as α > 0. The equilibrium is `π_θ(no-op) = small but positive`, with magnitude determined by the SAC α / distill λ ratio. This is the desired behavior: exploration on no-ops is preserved, but exploitation pressure is removed. + +**Theoretical concern.** "Which actions to include in baseline set?" is the load-bearing question. + +Three options: + +| Sub-option | Definition of no-op set | Pros / Cons | +|---|---|---| +| F2a | Static set: `{Hold, TrailTighten, TrailLoosen}` | Easy to implement; brittle to action-space changes; ignores that TrailLoosen at non-cap IS effectful | +| F2b | Dynamic: any action with `Δposition_lots == 0` post-execution | Principled (matches the structural property — no realized pnl delta possible without lots change); requires running fill simulator to detect | +| F2c | Implicit: `baseline(s) = top-K mean of E_Q(s, ·)` for some K | Avoids hardcoding the set; gradient direction is preserved if the no-ops cluster at the top | + +F2c is the most elegant but the least falsifiable — its behavior depends on the Q-value landscape, which the agent itself is learning. F2a is operationally cleanest. F2b is theoretically cleanest but architecturally invasive (requires fill simulator output to be visible at the distill kernel, currently it is only visible at the action-execution kernel). + +**Recommendation within F2:** start with F2a as a stub; if cluster shows it works, refactor to F2b for robustness. + +**Expected effect on inaction-bias.** Mechanically removes the inaction attractor from the distill target. The policy can only learn to assign mass to super-no-op actions. Whether the *encoder* produces enough super-no-op Q-signal for the agent to learn at all is the empirical question. + +**F2 preserves the surfer trading principle (load-bearing).** The surfer alpha mode (`pearl_dd049d9a4_surfer_baseline_verified`: wr=0.56, hold=30, +$10.6M @ step 18k) requires *patience-while-waiting-for-setup* — the agent must retain discretion to abstain when its model does not predict edge. F2's hinge target preserves this structurally: + +- When `E_Q(Open) ≤ baseline = E_Q(Hold)` (no edge detected): `max(0, E_Q(Open) − baseline) = 0` → Open receives zero distill mass → the gradient `λ · (π_θ(Open) − 0) = λ · π_θ(Open)` pushes π toward zero on Open → the agent learns "Hold is the higher-Q action; wait." +- When `E_Q(Open) > baseline = E_Q(Hold)` (edge detected): `max(0, E_Q(Open) − baseline) > 0` → Open receives positive mass → the agent learns to commit. + +The decision to act vs wait remains a *learned* property of the policy, conditioned on the encoder's edge signal. The agent retains the surfer's signature patience while still escaping the *baseline-equal* inaction attractor (because the hinge zeroes Hold's contribution too when other actions exceed it). This is the load-bearing reason F2 is now Phase 7a rather than Phase 7b — see §4.1.4 for the full surfer-principle decision criterion. + +**Expected failure modes.** + +1. *Mass-zero distill target.* If no action's `E_Q` exceeds `baseline`, the target is uniform-zero. Then the distill gradient pushes π toward uniform — equivalent to SAC entropy max. The agent becomes a random walker. This is a real risk during early training (cold-start Q gives noisy E_Q). +2. *Q-instability amplification.* The hinge `max(0, A)` introduces a non-differentiable point at the baseline. Stochastic gradient noise around the baseline could cause oscillation in `π_target` between "include" and "exclude" for marginal actions. Smoothing via softmax-with-temperature partly addresses this. +3. *Distill λ recalibration.* The existing controller (slot 486) is tuned for the current `π_target`. F2 reshapes the target distribution → the controller's setpoint will be wrong → must be re-bootstrapped. +4. *Interaction with Phase 4 band.* The band masks actions to Hold *after* the policy emits them. If F2 makes the policy avoid Hold, but the band forces Hold anyway, the gradient signal from F2 never observes the executed action — the entire Q-distill pipeline still trains as if Hold was chosen. **F2 is partially short-circuited by Phase 4-A.** Mitigation: either disable the band during F2 evaluation, OR add a "selected-before-mask" action signal to the Q-distill kernel. + +**LOC estimate.** Small. ~50–100 LOC: modify the `s_pi_target` computation in `rl_q_pi_distill_grad.cu` (lines 80–91), add one or two ISV slots (no-op-baseline-mode, no-op-set-definition), trainer-side dispatch, controller for distill-λ recalibration. + +**Predetermined falsification gates.** + +| Gate | Local b=128 (Tier 1.5) | Cluster b=1024 (Tier 2) | +|---|---|---| +| G1 (target-entropy effect) | `RL_Q_DISTILL_TARGET_ENTROPY_MEAN_INDEX` (slot 733) drops by ≥ 0.5 nats from baseline at step 1000 (proves the target is now action-discriminating, not uniform) | same metric < 1.5 nats by step 5000 | +| G2 (action distribution) | Hold% + TrailLoosen% < 60% across step 100–2000; entropy_action ∈ [1.5, 2.2] nats | Hold% + TrailLoosen% < 50% at step 5000; eval-time action_entropy > 1.0 nats | +| G3 (V quality preserved) | l_v < 4× baseline; no NaN/diverging loss | l_v stable through step 20k | +| G4 (eval pnl) | n/a | eval_pnl > h6btt baseline by ≥ $5M / session | + +**Risk vs reward.** Medium-high reward (mechanism is rigorous; well-grounded in Wang 2016; preserves surfer principle by construction). Low-to-medium risk (small kernel diff; reversible via ISV gate; well-defined gates). Net (per Revision 3 scoring with surfer-principle criterion added): **best risk-adjusted return overall (score 30); preserves the surfer trading principle while attacking the inaction attractor at the gradient axis. Recommended as Phase 7a primary intervention. F5 is staged as a conditional fallback (see §3.5 and §4.1.5/§4.2).** + +--- + +### §3.3 F3 — Action space reduction (REJECTED; see also F5 §3.5) + +**Disambiguation vs F5 (load-bearing).** F3 *removes* actions from the global action space (the enum `crate::rl::common::Action` shrinks from 11 to ~4 entries). F5 *keeps* all 11 actions in the enum but conditionally restricts which ones are *available* in each state via a hard mask applied before the policy softmax. The semantic difference is the same as `A` vs `A(s)` in Altman 1999's Constrained-MDP formulation. The user's critique that surfaced F5 identified that this spec's original F3 framing conflated these two — "the action space is wrongly used or at least the agent doesn't understand it should get out of the no-ops when trading is active, it should not be able to enter a no-op when it's doing nothing." F3 (this section) reads that critique as "shrink the global space"; F5 reads it as "shrink the state-conditional space." F3 stays REJECTED below; F5 is the option F3 should have been. + +**Mechanism.** Reduce the action space from 11 actions to ~4: `{Long+, Short+, Flat, Hold}`. Drop: pyramid variants (LongSmall, LongLarge, ShortSmall, ShortLarge split → one Long, one Short), TrailTighten (a7), TrailLoosen (a8), HalfFlatLong / HalfFlatShort (a9, a10). + +Concretely: +- a0 ← Short (combine ShortSmall + ShortLarge) +- a1 ← Long (combine LongSmall + LongLarge) +- a2 ← Hold +- a3 ← Flat (combine FlatFromLong + FlatFromShort by position-context routing) + +**Theoretical foundation.** No specific theorem — this is a *simplification* argument. Smaller action spaces are easier for Q to estimate well at fixed sample-complexity (cf. Sutton & Barto §6.5 on tabular Q-learning convergence; the rate depends on |A|). Empirically, action-space reduction has been the default move for many RL trading papers (e.g., Deng et al. 2016 used `{Buy, Hold, Sell}` ternary). **It does not address the root cause.** + +The root cause (§2.1 equation c) requires zero-cost-action existence. Hold remains a zero-cost action in F3. The attractor in (c) still exists; it has just been compressed from {Hold, TrailLoosen, TrailTighten} (3 zero-cost actions) to {Hold} (1 zero-cost action). + +**Does F3 fix the failure mode?** Partially. The `mq2pc` oscillation between Hold and TrailLoosen ceases because TrailLoosen does not exist. But the Hold-collapse (h6btt) re-emerges. F3 takes the system *back to the h6btt regime*. The bleed-rate would likely return to $4.18/batch/step (h6btt) rather than $19/batch/step (mq2pc). **F3 reduces severity but does not solve.** + +**Theoretical concern (cross-reference).** Per `git show 20c835713`, TrailTighten and TrailLoosen were added to close `pearl_dead_trail_stop_actions_a7_a8` — actions that existed as enum members in `crate::rl::common::Action` with no kernel consumer. Removing them re-opens the dead-wiring pattern. Worse: the trail-stop mechanism (`rl_trail_stop_check.cu`) currently overrides actions to FlatFrom* when the per-unit trail-distance is breached. Without TrailTighten/Loosen, the trail-distance becomes static (init only, no agent control). This re-introduces a risk-pathology that the SP20 P5 work (`20c835713`) was specifically designed to address: + +> "rl_trail_stop_check.cu — per-batch per-unit breach check. … On breach, OVERRIDE actions[b] to FlatFromLong (a3) or FlatFromShort (a4)." + +A static trail distance means every trade has a fixed-width stop-loss. With volatility shifts (regime changes), this is provably suboptimal vs adaptive trails (cf. Wilder 1978, Chande 1992 on volatility-adaptive stops). **F3 reintroduces a known-suboptimal risk model that earlier work specifically addressed.** + +Additionally, the pyramid variants (`LongSmall` vs `LongLarge`) encode position-size differentiation. Removing them collapses the agent's ability to express conviction differential. Cross-reference to `pearl_position_sizing_missing_adaptation_layer`: foxhunt explicitly identified the missing position-sizing controller as a gap. F3 *removes* the action-level expressiveness needed to address that gap. + +**Expected effect on inaction-bias.** Reduces but does not eliminate. Takes h6btt regime as the bound. NOT a foundation fix. + +**Expected failure modes.** + +1. *Hold attractor re-emerges in pure form.* h6btt bleed re-appears; system collapses to all-Hold. +2. *Static trail risk pathology.* On regime shifts, fixed-width trails either close prematurely (volatility spike) or too late (volatility crash). Either is a known failure mode. +3. *Loss of position-sizing expressiveness.* Without Long/Short small/large differentiation, the agent cannot express conviction differential. Combined with `pearl_position_sizing_missing_adaptation_layer`, this is a regression. +4. *Pyramid pathway gone.* Without small/large position openings, multi-unit pyramiding (Turtle 2N style, per Eckhardt-Dennis) is impossible. This removes a documented profitable pattern from the agent's reach. + +**LOC estimate.** Medium-large. Touches: enum `crate::rl::common::Action`, all 11 `ACTION_*` defines across ≥ 5 kernel files, `actions_to_market_targets.cu` (140+ LOC of action-to-market mapping), `multi_head_policy_forward.cu` (N_ACTIONS = 11 hardcoded), `multi_head_policy_aux_prior.cu`, `rl_band_mask.cu`, `rl_q_pi_distill_grad.cu` (N_ACTIONS = 11), audit-wiring infrastructure. ~500–1000 LOC of changes, plus regression-testing every ISV controller that touches action distribution. + +**Predetermined falsification gates.** + +| Gate | Local b=128 | Cluster b=1024 | +|---|---|---| +| G1 (Hold-collapse re-emerges?) | Hold% > 80% at step 2000 → F3 fails identically to h6btt | Hold% > 80% at step 10000 → F3 fails | +| G2 (eval pnl) | n/a | eval_pnl ≥ h6btt baseline (lower bar than F1/F2) | + +**Risk vs reward.** Low-to-medium reward (does not address root cause; only mitigates a symptom). High implementation cost. **Functionality removal — see `feedback_no_functionality_removal`.** Net: dispreferred unless F1/F2 also fail catastrophically and the policy needs simplification before any further work can be done. + +--- + +### §3.4 F4 — Oracle pretraining (offline Q warm-start) + +**Mechanism.** For each historical state `s_t` in the training data, compute an oracle Q-value `Q_oracle(s_t, a)` using perfect-foresight knowledge of future outcomes. Supervised-train Q to match `Q_oracle` for one or more epochs, THEN RL fine-tune. The agent inherits a Q function that *already knows* Hold is not always optimal because the supervised loss reflects realised returns from each action with perfect foresight. + +The minimal-scope version: compute `Q_oracle(s_t, a) = realized_return(s_t, a, oracle-π-from-t+1)` for a single action `a = argmax` and `Q_oracle(s_t, other) = realized_return(s_t, other) − safety-margin`. + +The full version: compute `Q_oracle` via dynamic-programming-with-oracle on every action × every state, then supervised loss on the full distribution. + +**Theoretical foundation.** Offline-to-online RL. +- Levine et al. 2020 ("Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems", arXiv:2005.01643): survey establishing offline pretraining as a stable initial-policy generator. +- Kumar et al. 2020 ("Conservative Q-Learning for Offline Reinforcement Learning", NeurIPS, arXiv:2006.04779): CQL — pessimistic Q-learning that avoids overestimation on out-of-distribution actions during the offline phase. +- Nair et al. 2020 ("Accelerating Online Reinforcement Learning with Offline Datasets", arXiv:2006.09359): AWAC — advantage-weighted regression for offline-to-online warm-start. +- Kostrikov et al. 2021 ("Offline Reinforcement Learning with Implicit Q-Learning", arXiv:2110.06169): IQL — expectile regression that sidesteps explicit pessimism. + +**Theoretical concern (load-bearing).** `pearl_foxhunt_trainer_is_genuinely_single_step` documents the trainer is `[B]`-only. The Q-learning loop is *embedded* in `step_with_lobsim_gpu_body`. Oracle pretraining would require: + +1. A separate offline training loop (not the current step-wise loop). +2. Persistence of an offline data buffer (`[N_states × N_actions × N_features]` — much larger than `[B]`-only). +3. Oracle generation infrastructure (perfect-foresight lookups across the full training dataset). +4. A loss path that uses MSE-against-oracle, NOT the existing Bellman target — but writes to the *same* Q head weights. +5. A protocol for switching the Q head from offline to online (cf. Lee et al. 2022 on "Offline-to-online RL with implicit policy constraint"). + +**Scope reality check.** Foxhunt has no offline-RL infrastructure. The minimal-scope version of F4 (one epoch of MSE-against-oracle before RL) requires building: +- A standalone CLI binary or training stage +- A perfect-foresight oracle (compute realized return from each action × each state) +- A storage format for oracle Q values (`[N_states × N_actions]` MBP-10 trace, ~10⁸ entries for 24 months × ES → ~1.5GB) +- A loss function consuming this storage +- A protocol for switching from supervised → RL + +The full version (dynamic-programming on every action) is computationally expensive (`O(T × N_actions²)` per dataset). + +**Per the math agent's `pearl_foxhunt_trainer_is_genuinely_single_step` STOP-rule discipline: F4 is a multi-week refactor, not a bounded edit.** + +**Does F4 fix the inaction-bias?** Yes, structurally and elegantly — the agent inherits a Q that has already discovered when each action is optimal. The Q-distill target then naturally puts mass on the correct action. **This is the most principled fix.** It also addresses several adjacent problems (train→eval generalization via supervised regularization, sample efficiency, exploration in cold-start). + +**Why it's wrong to dispatch this now.** Foxhunt's investigation methodology (`feedback_investigation_first_falsification_methodology`) requires bounded scope per dispatch. F4 is unbounded. The prior 64-commit anti-pattern is exactly the trap of dispatching ambitious fixes without falsifiable gates. F4 must be staged: first build offline-RL infrastructure (a separate spec); then evaluate F4 within that infrastructure (another spec); only then deploy. + +**LOC estimate.** Large. ~2000–5000 LOC of new infrastructure plus existing-code refactor. + +**Predetermined falsification gates.** Cannot be defined yet — the infrastructure to evaluate F4 does not exist. Specifying gates pre-infrastructure would be vacuous. + +**Risk vs reward.** High reward (most principled fix; addresses multiple adjacent problems). Very high implementation cost. **OUT OF SCOPE for this dispatch.** Worth a separate "F4 infrastructure" spec once F1 or F2 are deployed. + +--- + +### §3.5 F5 — State-conditional action availability mask (CMDP shielding) + +**Mechanism.** Insert a hard-mask kernel between the policy logits emission and the softmax sampling, structurally identical to the existing `rl_band_mask.cu` (Phase 4-A) but with broader scope. The mask is keyed on `position_lots` (read from the existing `pos_state` buffer at `pos_state + b·pos_bytes + 0..4`) and on the per-batch per-unit trail-stop distance vs the trail-max cap. + +The mask sets logits for unavailable actions to `−∞` (or a large negative sentinel ≤ −10⁶) BEFORE the softmax — equivalent to `softmax(z_a) = 0` for masked actions and uniform redistribution to available ones. The mask is applied identically to (i) the action sampled / `argmax` from `π_θ` and (ii) the `pi_target` distribution Q-distill consumes — both must see the same `A(s)`, otherwise the target-vs-output gradient is internally inconsistent. + +Pseudocode (matches `rl_band_mask.cu` lattice — Grid=(B), single thread per block, ISV-gated): + +``` +extern "C" __global__ void rl_state_action_mask( + float* logits, // [B, N_ACTIONS] — IN/OUT, set to −∞ for unavailable + const unsigned char* pos_state, + const float* trail_distances, // [B, max_units] per-unit trail distances (ticks) + const float* isv, + int b_size, int pos_bytes +) { + const int b = blockIdx.x; + if (b >= b_size) return; + + // Master gate (default OFF; bootstrap = 0.0). Symmetry with band mask + // slot 799 ensures reversibility. + if (isv[RL_STATE_ACTION_MASK_ENABLED_INDEX] <= 0.5f) return; + + const int position_lots = + *reinterpret_cast(pos_state + b * pos_bytes); + + // Branch on position state. + if (position_lots == 0) { + // FLAT: agent MUST commit to a direction (or take an explicit + // forced action — e.g., reset / force-flat sentinel if one + // exists). Mask: {Hold, FlatFromLong, FlatFromShort, HalfFlatLong, + // HalfFlatShort, TrailTighten, TrailLoosen, PyramidLong, + // PyramidShort}. Available: {Open Long (any size), Open Short + // (any size)}. + mask_action(logits, b, ACTION_HOLD); + mask_action(logits, b, ACTION_FLAT_FROM_LONG); + mask_action(logits, b, ACTION_FLAT_FROM_SHORT); + mask_action(logits, b, ACTION_HALF_FLAT_LONG); + mask_action(logits, b, ACTION_HALF_FLAT_SHORT); + mask_action(logits, b, ACTION_TRAIL_TIGHTEN); + mask_action(logits, b, ACTION_TRAIL_LOOSEN); + // Pyramid variants are sized opens from a non-flat base; + // unavailable from flat by construction. + mask_action(logits, b, ACTION_PYRAMID_LONG); + mask_action(logits, b, ACTION_PYRAMID_SHORT); + } else { + // IN POSITION: no re-opens of a new directional commitment from + // an already-committed state. Pyramid actions remain available + // (they extend the existing position). + if (position_lots > 0) { + // Long: mask short opens, mask flat-from-short, mask + // half-flat-from-short. + mask_action(logits, b, ACTION_OPEN_SHORT_SMALL); + mask_action(logits, b, ACTION_OPEN_SHORT_LARGE); + mask_action(logits, b, ACTION_FLAT_FROM_SHORT); + mask_action(logits, b, ACTION_HALF_FLAT_SHORT); + mask_action(logits, b, ACTION_PYRAMID_SHORT); + } else { + // Short: symmetric. + mask_action(logits, b, ACTION_OPEN_LONG_SMALL); + mask_action(logits, b, ACTION_OPEN_LONG_LARGE); + mask_action(logits, b, ACTION_FLAT_FROM_LONG); + mask_action(logits, b, ACTION_HALF_FLAT_LONG); + mask_action(logits, b, ACTION_PYRAMID_LONG); + } + + // CRITICAL trail-at-cap branch (mq2pc-specific): if the per-unit + // trail distance is at the cap, TrailLoosen is a mechanical + // no-op — the kernel `rl_trail_mutate.cu` clamps and emits no + // state change. Mask TrailLoosen in this branch. + const float trail_max = isv[RL_TRAIL_MAX_INDEX]; + const float eps = 1.0e-3f; + // Use max across units to catch the cap-saturated case for any + // pyramid layer. + float max_trail = 0.0f; + for (int u = 0; u < MAX_PYRAMID_UNITS; ++u) { + const float t = trail_distances[b * MAX_PYRAMID_UNITS + u]; + if (t > max_trail) max_trail = t; + } + if (max_trail >= trail_max - eps) { + mask_action(logits, b, ACTION_TRAIL_LOOSEN); + } + } +} +``` + +The masked `logits` then feed both the policy softmax (sampling / argmax) and the Q-distill `pi_target` computation (which must apply the SAME mask before its softmax — see §3.5 "F5 + F2 composition" below). Action indices use the canonical `crate::rl::common::Action` enum to avoid hardcoded duplication. + +**Theoretical foundation.** + +1. **Constrained-MDP formulation.** Altman 1999 ("Constrained Markov Decision Processes", Chapman & Hall) formalises an MDP with state-dependent action set `A(s) ⊂ A`. The Bellman optimality equation becomes `V*(s) = max_{a ∈ A(s)} [r(s, a) + γ Σ_{s'} P(s'|s, a) V*(s')]`. The optimal policy is still well-defined; the action-availability function is a property of the environment, not the policy. **Foxhunt's environment already has implicit availability constraints** (e.g., the trail-stop kernel overrides actions on breach; band-mask forces Hold inside the band; `rl_trail_mutate.cu` clamps TrailLoosen at the cap). F5 makes those constraints explicit at the *policy* level rather than expressing them via post-hoc action overrides. + +2. **Safe-RL / shielding.** Alshiekh et al. 2018 ("Safe Reinforcement Learning via Shielding", AAAI, arXiv:1708.08611). A shield is a reactive system that monitors the agent's chosen action and overrides it when it violates a specification. Foxhunt's current architecture *is* a shield system (band mask, trail-stop override, confidence gate, CMDP cooldown) but the shields run *after* the policy emits an action — the policy never sees the constraint, and Q-distill targets contain mass on actions that the environment will reject. Alshiekh §3 ("Preemptive vs Post-Posed Shielding") argues preemptive shielding (mask logits before sampling) is strictly more sample-efficient than post-hoc override because the agent's gradient signal contains the constraint information. F5 is the canonical preemptive-shielding refactor. + +3. **AlphaZero-style legal-move masking.** Silver et al. 2017 ("Mastering the Game of Go without Human Knowledge", Nature, DOI:10.1038/nature24270) §"Methods → Self-play training pipeline": at every position the network's policy output is masked by the set of legal moves before softmax; illegal moves receive zero probability. The MCTS PUCT computation also respects the legal-move mask. This is the foundational design pattern for game RL with state-conditional action sets. Trading is a game with state-conditional legality: you cannot close a position you do not hold, you cannot widen a trail at its cap. + +4. **Action elimination / pruning literature.** Zahavy et al. 2018 ("Learn What Not to Learn: Action Elimination and Stochastic Shielding in DQN", NeurIPS, arXiv:1809.02121) — pruning provably-suboptimal actions improves Q-learning sample complexity by reducing the effective `|A|` per state. F5 is the structural prior version of Zahavy 2018; foxhunt knows by construction which actions are state-incoherent without needing to learn the action-elimination classifier. + +**Why F5 attacks the inaction-bias at a different point than F1-F4.** + +Equation (c) in §2.1 says: + +``` +E[Q_target(s, a_c) − Q_target(s, a_0)] ≈ −entry_cost < 0 +``` + +F5 does not modify either side of (c). Instead F5 modifies *which actions are in the max-over-actions Q evaluation in the next-step bootstrap*. When the agent is flat and Hold is masked, the policy cannot select Hold; the Q-distill softmax target re-normalises over remaining actions; the policy never observes a flat→Hold trajectory in its on-policy data and the Bellman target's `V(s')` for the flat state becomes the max-Q over `A(s_flat) = {OpenLong, OpenShort}` instead of `A = {Hold, OpenLong, OpenShort, ...}`. + +Critically: F5's effect on equation (c) is to make the comparison vacuous for unavailable actions — there is no `a_0 ∈ A(s)` for the flat state, so the asymmetry has no choice-set on which to manifest. This is structurally different from F1 (modifies reward), F2 (modifies distill target shape), F3 (removes from global enum), and F4 (warm-starts Q). + +**F5 + F2 composition (load-bearing).** F5 alone leaves the in-position attractor (Hold + TrailLoosen-when-not-at-cap) intact: when the agent is long and the trail is not yet at the cap, Hold and TrailLoosen are both *available* and both *zero-cost*. F5 has shrunk `A(s)` but the residual `A(s)` still contains a zero-cost subset whose Q advantages over costly closes match equation (c). F2 (Q-distill advantage target) then operates *on the masked `A(s)`* — the `baseline(s)` is computed only over `a ∈ A(s) ∩ no-op-set`, and the distill target's `max(0, A)` hinge zeroes mass for any action whose Q does not exceed that masked baseline. Composition order: + +``` +Step 1 (F5): raw_logits → masked_logits (unavailable actions → −∞) +Step 2: masked_logits → π_θ (softmax over A(s)) +Step 3 (Q): E_Q(s, ·) computed over all actions, but only a ∈ A(s) used in distill target +Step 4 (F2): baseline(s) = max_{a ∈ A(s) ∩ no-op-set} E_Q(s, a) + π_target(s, a) ∝ softmax( max(0, E_Q(s, a) − baseline(s)) / τ ) · 1[a ∈ A(s)] +Step 5: distill grad = λ · (π_θ − π_target) +``` + +The two interventions are mechanically commutative in the sense that they touch disjoint state in the pipeline: F5 changes the *support* of the policy distribution, F2 changes the *gradient on the support*. F5 cannot break F2 (F2 operates on whatever support F5 leaves). F2 cannot break F5 (F2 does not introduce or remove mask entries). + +**F5 violates the surfer trading principle (load-bearing trade-off, be honest about it).** The surfer alpha mode (`pearl_dd049d9a4_surfer_baseline_verified`: wr=0.56, hold=30, +$10.6M @ step 18k) is defined operationally by long, patient holds with high win-rate — the agent waits for high-confidence setups and only commits when it predicts edge. The structural ability that enables this is *Hold-from-flat as a valid policy choice*: the agent abstains until its encoder signals an edge. + +F5's flat-state mask removes `{Hold, all trail variants, all close/half-flat variants, all pyramid variants}` from `A(flat)`, leaving `A(flat) = {OpenLongSmall, OpenLongLarge, OpenShortSmall, OpenShortLarge}`. Every step the agent is flat, it is *forced* to open a position regardless of whether its encoder predicts edge. This: + +- **Destroys patience-while-waiting-for-setup** — the foundational surfer behavior — by construction. The agent cannot abstain from low-confidence setups; it can only choose between Long and Short. +- **Forces random / low-confidence opens** → win-rate drops below the surfer's 0.56 → the policy regresses toward the trend-follower profile (wr=0.36, hold=17 per `pearl_two_alpha_modes_surfer_vs_trend`). +- **Removes agent discretion to wait**, even when the encoder produces low-confidence Q-signal across all opens. The agent has no expressive way to communicate "no edge detected here." +- **Cannot be untrained.** Unlike F1/F2 (which the agent can *learn around* if it discovers patience pays off), F5 is a hard environmental constraint — it eliminates the patience strategy from the policy support entirely. + +This is a *real* trade-off, not a bug. F5 may still be valuable as a fallback because: + +1. The h6btt all-Hold collapse is unambiguously worse than even the most aggressive trend-follower regression (h6btt bled while taking no trades; the trend-follower at least makes money). +2. F2 may fail to break the h6btt-class flat-state collapse if the encoder's Q-signal is too weak to make any Open's `E_Q` exceed `E_Q(Hold)` — in which case F2 degenerates to uniform random and F5's forced commit is strictly better than uniform random. +3. The user's stated philosophical preference is surfer-style trading, but the surfer baseline's own eval pnl was negative (`pearl_surfer_baseline_was_train_only_never_eval`: −$13.5M wr=0.219 in eval) — preserving surfer mode does not guarantee profitability either. + +The honest position: F5 is the structural fallback if F2's gradient redirection cannot escape the encoder-weakness scenario, and we choose to accept surfer→trend regression in exchange for breaking the all-Hold attractor. The staging in §4 puts F2 first so we test the surfer-preserving option before accepting the surfer-sacrificing fallback. See §4.1.4 for the full surfer-principle decision criterion. + +**Expected effect on inaction-bias.** + +- **From flat (h6btt-class):** F5 directly removes the choice. The agent cannot collapse to all-Hold because Hold is masked from flat. The action distribution from flat states is forced to be `{OpenLong, OpenShort}` with shares determined by the policy's directional preference. This is the strongest possible structural fix for the h6btt regime: not a gradient nudge but a hard constraint. Expected: Hold% from flat-only states → 0%. + +- **From in-position (mq2pc-class):** F5 partially helps. TrailLoosen-at-cap is masked, breaking the specific mq2pc oscillation. But TrailLoosen-not-at-cap and Hold-while-in-position remain available and zero-cost. The residual attractor must be addressed by F2's gradient mechanism. + +- **Combined F5 + F2:** F5 eliminates the flat-state attractor structurally; F2 eliminates the in-position attractor via gradient. The composition is the minimal disciplined set covering both regimes. + +**Expected failure modes.** + +1. **Forced loss-making trades in zero-edge regimes.** If the market is in a regime with no exploitable alpha, F5 forces the agent to open trades anyway (Hold is unavailable). The agent's only choice from flat is `{OpenLong, OpenShort}`, and a random pick has E[pnl] ≈ −commission. In adversarial regimes (high noise, low edge), F5 could *worsen* eval pnl vs the do-nothing baseline. **Mitigation possibilities (NOT specified at this stage):** add a "synthetic Hold" via an extremely-small-size open that the band mask immediately routes to Hold; OR allow Hold from flat only when a confidence gate condition is met. Both are out of scope for the v1 F5 design — see §6. + +2. **Breaks encoder-learned pseudoinaction patterns.** The encoder has been training with the current action distribution. If it has learned to detect specific pseudoinaction signals (e.g., "the agent will Hold when feature X is high"), masking Hold from flat invalidates those learned features. The encoder needs to relearn — gradient should flow but the transient could be lossy. Estimated training-step recovery: 500-2000 steps based on observed reaction-times in prior controller-floor refactors. + +3. **F5 mask + Q-distill target inconsistency.** If the mask is applied to the policy logits but NOT to the `pi_target` softmax input, the distill gradient is misaligned: π_θ has zero mass on masked actions (after the mask) while `pi_target` has non-zero mass (because the unmasked Q values include the masked action's Q). The gradient on the policy weights for the masked action would be `λ · (0 − π_target(a)) = −λ π_target(a)`, which pushes the policy weights toward producing non-zero logits on the masked action — wasted gradient, and a potential numerical instability if the masked logits are pushed back up enough that the `−∞` mask becomes a sentinel comparison rather than a soft constraint. **The mask MUST be applied to both `π_θ` and `pi_target`. The kernel diff must propagate the masked action set into `rl_q_pi_distill_grad.cu`.** + +4. **Trail-at-cap detection edge cases.** Floating-point comparison `max_trail >= trail_max - eps` is sensitive to the exact value of `eps`. If `eps` is too tight, the mask fails to engage at the cap (the very mq2pc pathology persists). If `eps` is too loose, the mask engages near-cap, removing TrailLoosen when there is still some action-level effect. The MQ2PC trace shows TrailLoosen reaching the cap and saturating; a generous `eps ≈ 1e-2 · trail_max` is the safest first cut. Tunable via ISV. + +5. **Interaction with confidence gate.** Per `pearl_confidence_gate_pos_state_unwired` the confidence gate already declares `pos_state` as a parameter but does not read it. F5 is conceptually similar — both gate actions based on position state — but operates earlier in the pipeline (on logits, before softmax). The two should compose: F5 mask first, then policy emits an action over the surviving support, then confidence gate post-overrides if confidence is low. The confidence-gate Case 3 failure documented in that pearl is NOT introduced by F5 and is orthogonal; F5's correctness does not depend on the confidence gate being correctly position-aware. + +6. **Phase 4-A band mask interaction.** The Phase 4-A band mask forces `actions[b] = Hold` when in-position-and-inside-band. F5 from in-position does NOT mask Hold. So the composition is: F5 leaves Hold available from in-position → policy may emit OpenLong / etc. → band mask post-overrides to Hold if inside band. Consistent ordering. Verify in implementation plan. + +7. **Single-direction over-commitment.** With Hold removed from flat, the agent's choice is binary `{Long, Short}`. If the encoder has a directional bias (even small), the policy could collapse to one direction (e.g., 95% Long). F2's `max(0, A)` then only retains mass on the directionally-biased side — and the agent has no expressive way to abstain. Detection: action distribution in F5-G5 / F5-G9 detects > 80% single-action share. Mitigation possibilities (also out of scope for v1): a "synthetic abstain" via a band-narrow-open, or relaxing the flat-Hold mask under high-uncertainty regimes. + +**LOC estimate.** Small-to-medium. ~50-100 LOC total: + +- New kernel `rl_state_action_mask.cu` (modeled after `rl_band_mask.cu`'s 91-line lattice): ~60-80 LOC including the trail-at-cap detection branch. +- Trainer-side dispatch (one `cuLaunchKernel` invocation between policy logits emission and softmax sampling): ~10-15 LOC in `crates/ml-alpha/src/trainer/integrated.rs`. +- Identical mask propagation into `rl_q_pi_distill_grad.cu` for `pi_target` computation: ~20-30 LOC (read the same mask buffer or recompute identically from the same `pos_state`). +- New ISV slots: `RL_STATE_ACTION_MASK_ENABLED_INDEX` (binary master gate, bootstrap 0.0); `RL_STATE_ACTION_MASK_TRAIL_EPS_INDEX` (the cap-tolerance). Two new slots. +- Diag fields: per-batch fraction-masked-from-flat, fraction-masked-trail-at-cap (cf. `pearl_fleet_fraction_not_aggregate`). Wire into existing diag emission. +- Test fixtures: extend `crates/ml-alpha/tests/trade_management_kernels.rs` with mask correctness assertions analogous to the band-mask tests already there. + +Total estimated diff: ~150 LOC across 4 files, comparable to the band-mask Phase 4-A foundation commit `e41a73208`. + +**Pearl references (load-bearing wiring breadcrumbs).** + +- `pearl_confidence_gate_pos_state_unwired` — `rl_confidence_gate.cu` already accepts `pos_state` and `pos_bytes` as kernel parameters but does not read them. The plumbing exists upstream (trainer-side dispatch passes `pos_state` correctly); only the kernel-side read is missing. F5 inherits the same plumbing pattern. +- `pearl_dd049d9a4_surfer_baseline_verified` — the cluster-proven surfer baseline at `dd049d9a4` ran *without* any state-conditional action mask. So F5 is not "restoring a removed surfer feature"; it is "adding a new structural constraint that the surfer baseline never had." The risk that F5 deviates from the surfer regime is genuine — see falsification gate G2.5 below. +- `pearl_surfer_baseline_was_train_only_never_eval` — the surfer baseline's eval pnl was −$13.5M; train profitability is not eval profitability. F5's outcome gates must be evaluated against eval pnl, not train pnl. +- Task list entries 49, 65, 67-68 (Phase 2.3 prior state-mandatory action mask) — earlier work attempted a state-mandatory action mask that evolved into a SOFT mask. F5 proposes returning to a HARD mask but with broader scope and explicit `−∞`-logit semantics. The prior soft mask's lessons (which actions were too constraining, how the policy migrated) are valuable input for the implementation plan but are not load-bearing for the design decision here. +- `pearl_band_mask` (the existing `rl_band_mask.cu` Phase 4-A architecture) — F5 is the band mask's structural twin: same Grid=(B), single-thread per block, ISV-gated, reads `pos_state`, default OFF for bit-equality preservation. Implementation should be parallel. +- `feedback_wire_everything_up` — the F5 mask MUST propagate into the Q-distill `pi_target` computation, not just the policy sampling path. See expected failure mode #3. + +**Predetermined falsification gates.** + +These gates are designed to distinguish F5's effect from F2's effect — see §5 for the canonical combined gate set. **NOTE (Revision 3):** the §5.2 Phase-7b canonical gate IDs renumber some of the F5 outcome gates (F5-G7 here = F5-G8 in §5.2; F5-G8 determinism here = part of F5-G6 in §5.2). Where the two disagree, §5.2 is authoritative. The legacy table below remains as the F5-only design summary from Revision 2: + +| Gate | Local b=128 (Tier 1.5) | Cluster b=1024 (Tier 2) | What it detects | +|---|---|---|---| +| F5-G1 (mask correctness — flat) | Among batches where `position_lots == 0`, action_hist mass on `{Hold, TrailLoosen, TrailTighten, FlatFromLong, FlatFromShort, HalfFlatLong, HalfFlatShort, PyramidLong, PyramidShort}` is exactly 0 | Same | Verifies the hard mask works mechanically; ZERO mass, not "low" mass | +| F5-G2 (mask correctness — in-position) | Among batches where `position_lots > 0`, mass on `{OpenShortSmall, OpenShortLarge, FlatFromShort, HalfFlatShort, PyramidShort}` is exactly 0; symmetric for short | Same | Position-direction coherence | +| F5-G3 (mask correctness — trail at cap) | Among batches where any per-unit trail = trail_max, mass on TrailLoosen is exactly 0 | Same | mq2pc-specific mechanism gate | +| F5-G4 (flat → in-position transition rate) | Fraction of steps with `position_lots = 0 → position_lots ≠ 0` transition ≥ 5× the `mq2pc` baseline (mq2pc had ~4 trades / 2000 steps from flat) | Total opens per 1000 steps ≥ 50 | Direct positive evidence of choice-set forcing | +| F5-G5 (no single-direction collapse) | If F5-G4 passes, no single direction has > 80% of opens (else the agent has merely migrated to a one-sided over-commit attractor) | Same | Detects expected failure mode #7 | +| F5-G6 (training stability) | No NaN; l_pi, l_v, l_q magnitudes within 10× of `mq2pc` baseline | l_pi etc. stable through step 20k | Standard substrate gate | +| F5-G7 (eval pnl) | (not measured at Tier 1.5) | eval_pnl ≥ h6btt baseline + $5M / session | Outcome gate | +| F5-G8 (determinism preserved) | `determinism-check.sh --quick` exits 0 at seeds 42/43/44 | n/a | Substrate per `pearl_determinism_achieved` | + +**Risk vs reward.** Medium reward (directly addresses the choice-set problem; structurally guarantees no flat→Hold collapse; reuses proven band-mask architecture) BUT explicit surfer-principle violation reduces post-Revision-3 score to 28 (vs F2's 30). Low-to-medium implementation risk (small kernel diff; reversible via ISV master gate; verified failure modes have explicit detection gates). Specific outstanding risks: forcing trades in zero-edge regimes (failure mode #1); surfer → trend regression by construction (load-bearing trade-off above). Net (per Revision 3): **valid as a fallback intervention on the choice-set axis when F2's gradient redirection fails from encoder-weakness. Staged as Phase 7b conditional on F2 outcome gate failure AND explicit user acceptance of the surfer→trend trade-off.** + +--- + +## §4. Comparative recommendation + +### §4.1 Score matrix + +Scoring scale: 1 (low / poor) to 5 (high / excellent). Cost is inverted (1 = high cost = bad). F4's total rests on infrastructure that does not exist; treat as a fork. + +Criterion 7 (**"Respects surfer trading principle"**) was added in Revision 3. It asks: does the intervention preserve the agent's discretion to wait patiently for high-confidence setups (the surfer alpha mode per `pearl_dd049d9a4_surfer_baseline_verified`), or does it force structural action regardless of belief? See §4.1.4 for the criterion's full definition. + +| Criterion | F1 (inaction cost) | F2 (Q-distill advantage target) | F3 (action reduction) | F4 (oracle pretraining) | F5 (state-conditional avail. mask) | +|---|:-:|:-:|:-:|:-:|:-:| +| Theoretical soundness | 3 (non-potential per Ng-Harada-Russell) | 4 (Wang 2016 advantage decomposition) | 2 (no specific theorem) | 5 (offline RL canonical) | 5 (Altman 1999 CMDP + Alshiekh 2018 shielding + AlphaZero legality) | +| Addresses root cause | 4 (corrects reward asymmetry directly) | 5 (removes asymmetry at distill stage; mechanism-level fix) | 2 (reduces but does not eliminate) | 5 (warm-starts Q correctly) | 4 (eliminates flat-state choice; partial for in-position) | +| Implementation cost | 2 (medium; controller risk) | 1 (small; well-bounded kernel diff) | 4 (large; touches enum + all action kernels) | 5 (very large; new infra) | 2 (small; band-mask twin pattern; new kernel + Q-distill mask propagation) | +| Falsifiability | 4 (clear local + cluster gates) | 5 (mechanism gates + outcome gates) | 3 (only outcome gates) | 1 (gates require non-existent infra) | 5 (mask correctness directly observable in action_hist; ZERO-mass gates) | +| Foundation impact (resists going-in-circles) | 4 (changes the data) | 5 (changes the gradient) | 1 (just removes options) | 5 (changes the initial conditions) | 5 (changes the choice-set; structurally removes the attractor) | +| Reversibility (ISV gate, no code-rip) | 5 (ISV gate) | 5 (ISV gate) | 1 (enum change is one-way) | 3 (separate offline phase) | 5 (ISV master gate, bootstrap OFF) | +| **Respects surfer trading principle** (Rev 3) | 2 (adds friction to inaction; per-step holding cost discourages patience but does not force action) | 5 (provably preserves principle via hinge: zero Open mass when E_Q(Open) ≤ E_Q(Hold)) | 1 (removes Hold globally; total violation) | 5 (oracle targets reflect optimal patient behavior; preserves principle) | 2 (forces opens when flat; structural violation of patience-while-waiting) | +| **Total (higher = better)** | **24** | **30** | **14** | **29 (gated on F4-infra)** | **28** | + +**Verdict shift from Revision 2.** Pre-criterion-7 ranking was F5 (26) > F2 (25) > F4 (24, gated) > F1 (22) > F3 (13). Post-criterion-7 ranking is **F2 (30) > F4 (29, gated) > F5 (28) > F1 (24) > F3 (14)**. F2 now leads. F5's lead disappears because F5 sacrifices the surfer principle that F2 preserves by mathematical construction. F4 remains gated on infrastructure that does not exist. + +### §4.1.4 Surfer trading principle as a decision criterion (Revision 3 addition) + +The surfer principle is load-bearing for foxhunt's stated trading philosophy; this section makes it explicit so future revisions cannot accidentally violate it without acknowledging the trade-off. + +**What the surfer alpha mode is.** Per `pearl_dd049d9a4_surfer_baseline_verified` (cluster-proven at SHA `dd049d9a4`, b=1024, May 2026): + +| Metric | Surfer baseline value | +|---|---| +| Stable win-rate (wr_ema) | 0.56 | +| Average hold duration | 30 steps | +| Cluster pnl at step 18k | +$10.6M | +| Clip rate ema | 1.0 (saturated; normal for this regime) | +| C51 v_max | 1.0 (frozen) | +| Q-distill KL ema | ≈ 0 (Q and π agree) | + +The trend-follower counter-mode (`6d33f18b7`, per `pearl_two_alpha_modes_surfer_vs_trend`): wr ≈ 0.36, hold ≈ 17 steps, +$41.5M peak. The data supports both modes; the architecture supports both modes; the user has explicitly chosen the surfer per `feedback_surfer_philosophy_trading`. + +**Why the surfer requires patience-while-waiting-for-setup as a structural ability.** The surfer's high win-rate (0.56 vs 0.36) is the alpha signal. It comes from the agent being *selective* about entries — only committing when its encoder signals an edge. The behavioral expression of this selectivity is *long stretches of Hold-from-flat* while the agent observes the market and waits for a setup. If the agent is forced to open positions on every flat-state step, it cannot be selective; its entries become random or low-confidence; its win-rate collapses toward the random baseline. + +The surfer principle, stated precisely: **act when your model believes acting beats waiting; wait otherwise**. The agent's patience-when-no-edge IS the alpha signal. Any intervention that breaks this property breaks the surfer mode by construction. + +**How F5's mask structurally prevents this patience.** F5 sets `A(flat) = {OpenLongSmall, OpenLongLarge, OpenShortSmall, OpenShortLarge}` — Hold is masked out. The policy can sample only from `{Long, Short}` × `{Small, Large}`. The agent has no expressive way to communicate "no edge detected here; wait." Every flat-state turn produces an open regardless of encoder signal. The surfer's wr=0.56 cannot survive this — the new wr will reflect random-pick-when-no-edge, pulling it toward 0.5 minus commission drag, plus whatever genuine edge moments the encoder still detects when forced commits land on them. Net effect: the agent migrates toward the trend-follower profile (higher activity, lower win-rate). + +This is not a *bug* in F5 — it is F5 doing exactly what it was designed to do. F5's purpose is to eliminate the h6btt-class flat-state Hold attractor by making the attractor mechanically impossible. The cost of this design is the surfer principle. + +**How F2's gradient redirection preserves this patience.** F2 sets the Q-distill target to `π_target(a) ∝ softmax(max(0, E_Q(a) − baseline) / τ)` where `baseline = max over a ∈ no-op-set { E_Q(a) }`. The Hold-from-flat case: + +- If the encoder produces `E_Q(Open) ≤ E_Q(Hold)` (no edge detected at this state): `max(0, E_Q(Open) − E_Q(Hold)) = 0`. Open's distill target mass is zero. The gradient `λ · (π_θ(Open) − 0)` pushes π_θ(Open) toward zero, leaving Hold as the surviving high-Q action. The agent learns to wait. **The surfer principle holds.** +- If the encoder produces `E_Q(Open) > E_Q(Hold)` (edge detected): `max(0, E_Q(Open) − E_Q(Hold)) > 0`. Open's distill target mass is positive. The gradient pushes π_θ(Open) toward this target. The agent learns to commit on detected edge. **The surfer's patience-selective entries property holds.** + +The key contrast: F5 makes the wait/act choice for the agent (and always picks "act"); F2 lets the agent *learn* the wait/act choice from the encoder's edge signal. F2 is the surfer principle expressed as a loss function. + +**Caveat.** F2 only preserves the surfer principle as long as the encoder produces an `E_Q` landscape where some actions exceed the no-op baseline at edge-bearing states and no actions exceed it at no-edge states. If the encoder is too noisy or too weak, F2 degenerates to uniform random (the "mass-zero distill target" failure mode in §3.2). In that scenario the surfer principle is moot because there is no edge signal to be selective about. F5 then becomes a meaningful fallback: forced commits at the random-baseline floor are still better than the do-nothing baseline of h6btt's all-Hold collapse, even at the cost of the surfer mode. + +**Citation.** Charpentier, Elie, and Remlinger 2020 ("Reinforcement Learning in Economics and Finance", arXiv:2003.10014), §"Equity trading via RL", document the surfer-vs-trend dichotomy as canonical in financial RL — the same two operating regimes appear across multiple RL trading systems, distinguished by win-rate vs frequency trade-off. The surfer principle (patient selective entries) is the empirically-validated higher-Sharpe regime in much of the literature; the trend-follower is the higher-absolute-return / higher-drawdown counterpart. Foxhunt's cluster bisect at `dd049d9a4` and `6d33f18b7` reproduces this dichotomy. + +**Decision rule for this spec.** Recommend the intervention that *preserves* the surfer principle as Phase 7a. Stage the *principle-sacrificing* intervention only as a conditional Phase 7b fallback, and explicitly acknowledge the trade-off in the gate evaluation. This is the discipline encoded in the revised §4.2 staging recommendation below. + +### §4.1.5 Combined F2 + F5 analysis (revised ordering per Revision 3) + +F2 and F5 attack *mechanically disjoint* aspects of the inaction attractor: + +- **F2 attacks the gradient** (no-op actions have zero advantage gradient flowing into the policy when `E_Q(no-op) ≥ max(other E_Q)`). Soft pressure that depends on Q quality but PRESERVES the surfer principle by construction (agent retains learned discretion to wait when no edge is detected). +- **F5 attacks the choice-set** (when flat, agent CANNOT select Hold; when at trail cap, agent CANNOT select TrailLoosen). Hard constraint on `A(s)`. Independent of gradient signal — works even if Q is uninformative — but VIOLATES the surfer principle (forces commits regardless of belief). + +Per `feedback_going_in_circles_pattern`, multi-fix stacking is suspect. But F2 and F5 are NOT "two feels-right fixes stacked" — they are interventions on disjoint pipeline stages with independent falsification gates. F2's gates (F2-G1, F2-G2) verify mechanism correctness on the gradient distribution; F5's gates (F5-G1, F5-G2, F5-G3) verify mechanism correctness on the action support. Post-cluster attribution is possible because: + +- If F2 gates pass and eval pnl is positive (Phase 7a) → F2 alone carried the run, ship F2 alone, surfer principle preserved. +- If F2 gates pass and eval pnl is negative (Phase 7a) → F2 mechanism is operational but insufficient; evaluate per §4.4. If user accepts surfer→trend trade-off, escalate to Phase 7b (F5 added). +- If F2 + F5 both pass mechanism gates and eval pnl is positive (Phase 7b) → both interventions necessary; ship both with explicit trade-off documented. +- If F2 + F5 both pass mechanism gates and eval pnl is negative (Phase 7b) → both mechanisms operational but insufficient; escalate to F4 or alignment work per §5.3. +- If F2 gates fail (mass-zero target) → fix the kernel (warmup / temperature schedule); this is not an architectural verdict. + +This is the criterion `feedback_investigation_first_falsification_methodology` requires: each intervention has its own falsifiable mechanism gate, and the outcome gate is shared but post-hoc attribution is possible. + +**Marginal benefit analysis:** + +- *Marginal benefit of F5 given F2:* F2 alone redirects gradient away from no-op actions whose E_Q does not exceed `baseline = E_Q(no-op)`. If the encoder produces strong enough edge signal that opens' E_Q exceeds Hold's E_Q at edge-bearing states, F2 alone solves the inaction attractor *while preserving the surfer principle*. F5 then provides no additional benefit and would actively damage the surfer mode. **If the encoder is producing real alpha (the assumption behind the user's surfer preference), F5 adds nothing and subtracts patience.** If the encoder is weak, F2 may fail to escape the attractor and F5's forced commits become a fallback. F5's marginal value is conditional on F2-failure-from-encoder-weakness. + +- *Marginal benefit of F2 given F5:* F5 alone eliminates the flat-state Hold attractor but leaves the in-position attractor (Hold-while-in-position-not-in-band; TrailLoosen-not-at-cap) intact. F2 redirects gradient away from these. In the F5-first staging this was the load-bearing reason for F2 conditional. In the revised F2-first staging, F2's in-position effect is captured in Phase 7a and the question becomes whether F5's flat-state mask adds residual benefit. + +- *Scenario where F2 alone is sufficient:* the encoder produces alpha signal strong enough that E_Q(Open) > E_Q(Hold) at edge-bearing flat states. F2's hinge then pushes π toward Open at those states (action when edge detected) and toward Hold elsewhere (wait when no edge). The agent learns the surfer policy by gradient. h6btt's all-Hold collapse cannot persist because the flat→edge-detected→Open transition gets positive gradient signal; mq2pc's TrailLoosen-collapse cannot persist because TrailLoosen's E_Q does not exceed E_Q(other in-position no-ops) at edge-bearing states. This is the optimistic case and is also the *correct* outcome for the surfer philosophy. + +- *Scenario where F2 alone is insufficient and F5 fallback is needed:* the encoder is too weak. E_Q(Open) ≤ E_Q(Hold) everywhere, including at edge-bearing states. F2's hinge produces a mass-zero target for Open; the policy reverts to SAC entropy-driven uniform with Hold dominant by the prior. The system collapses back into h6btt regime. Then F5's hard mask becomes meaningful: forced commits at a random-baseline floor are strictly better than the do-nothing baseline. The cost is acceptance of surfer→trend regression. + +- *Disciplined staging (revised).* Per `feedback_going_in_circles_pattern` and `feedback_investigation_first_falsification_methodology`, the most disciplined sequence in Revision 3 is: + - **Phase 7a:** F2 alone. Cluster verdict. If F2-G8 (eval pnl) passes AND mechanism gates pass AND no degenerate attractor → STOP; F2 alone is sufficient and the surfer principle is preserved. + - **Phase 7b (conditional):** If F2-G8 fails AND F2 mechanism gates pass AND we accept the surfer→trend trade-off, add F5 on top. New cluster verdict. + +The previous Revision 2 staging (F5 first, F2 conditional) was reversed in Revision 3 because: + - F2 preserves the surfer principle by mathematical construction; F5 violates it by structural constraint. The surfer principle is the user's stated philosophical preference. + - F2 has higher post-criterion-7 score (30 vs 28). + - F2's cluster verdict tells us whether the surfer mode can be reached at all from the current Phase 5 substrate. If yes, ship F2. If no, the question shifts to "do we accept the trade-off" rather than "is the choice-set the load-bearing problem." + +This staging extracts the same information per cluster run as the previous order, but pays the surfer-principle preservation cost first and the surfer-principle-sacrificing fallback second. Going-in-circles discipline is preserved because each phase tests exactly one mechanism with its own gates. + +### §4.2 Recommended option: **F2 (Phase 7a) staged with F5 (Phase 7b)** — surfer-principle-preserving gradient first, structural mask only as fallback + +**Revised in Revision 3 — load-bearing reason.** F2 addresses the root cause via the *gradient axis* (Q-distill target shape) while preserving the surfer trading principle by mathematical construction (see §4.1.4). The hinge `max(0, E_Q(a) − baseline)` zeroes Open's distill mass when no edge is detected and zeroes Hold's contribution when other actions exceed it — the agent learns to wait or act based on the encoder's edge signal, not a structural constraint. F2 is the surfer principle expressed as a loss function. + +F5 is staged as the conditional fallback because: + +1. F5 *violates* the surfer principle by removing Hold-from-flat from `A(s)` entirely. The agent is forced to commit on every flat-state step regardless of belief, destroying patience-while-waiting-for-setup, which is the load-bearing structural property of the surfer alpha mode (`pearl_dd049d9a4_surfer_baseline_verified`). +2. F5 may still be valuable as a fallback if F2 fails from encoder-weakness — if the encoder's E_Q landscape cannot make any Open's Q exceed Hold's Q at edge-bearing states, F2 degenerates to uniform random and F5's forced commits become strictly better than the do-nothing baseline. But this is a conscious trade-off, not an oversight. +3. Per `feedback_investigation_first_falsification_methodology` §"sequential phases with each gate informing the next," the most information-extracting sequence is to test the surfer-preserving option first, then accept the surfer-sacrificing fallback only if necessary. + +Intervention-point comparison: + +``` +F1 (reward kernel) → V regression → Q Bellman target → softmax target → π + ^ corrective signal attenuates 4× through learned approximators (Goodhart-Skalse) + +F2 (distill kernel) → π + ^ corrective signal is direct; one approximator step + ^ PRESERVES surfer principle by construction (hinge zeroes mass on E_Q ≤ baseline) + +F5 (policy logits mask) → π + ^ NOT a corrective signal — a structural constraint on the support of π; + does not attenuate; cannot be "untrained" by gradient + ^ VIOLATES surfer principle by removing Hold from A(flat) + +F2 + F5 (in fallback ordering): F2 reshapes the gradient first; F5 adds support constraint only if F2 alone insufficient. +``` + +**Why F2 first, F5 conditional rather than the Revision-2 order:** + +- *Surfer principle preservation.* The user has explicitly chosen surfer-mode trading. F2 preserves this by construction; F5 violates this by construction. Test the preserving option first. +- *Information per cluster run.* If F2 alone improves eval pnl, we have proven both (i) gradient redirection was the load-bearing intervention and (ii) the encoder is strong enough to produce alpha signal that the surfer mode can exploit. Both are meaningful results. +- *Discipline against accidental philosophy shift.* Choosing F5 first commits the system to the trend-follower regime even before testing whether the surfer regime is reachable. Revision 2's order encoded an implicit "we don't believe the surfer is reachable" assumption that the user has explicitly rejected. +- *Falsifiability of the encoder-weakness hypothesis.* If F2 fails, we have indirect evidence that the encoder's alpha-extraction is the bottleneck (since F2's mechanism gates depend on E_Q(Open) > E_Q(Hold) at edge-bearing states). That evidence is more diagnostic than F5's outcome (which would only tell us "forced trading is better than no trading"). +- *Reversibility.* F2 alone is one ISV master gate. F2+F5 is two. Smaller surface area is preferred for the foundational phase. + +Comparison to non-recommended options: + +- **F5 alone (Revision 2 recommendation):** mechanically eliminates the flat-state Hold attractor but at the cost of the surfer principle. Valid only as a fallback once F2 has been tested and failed. +- **F1 (inaction cost):** more direct on the data but more disruptive to the Q-distill controller chain (which is tuned for the current target distribution), introduces a known-suboptimal non-potential shaping term, and *also* violates the surfer principle by adding friction to waiting (criterion-7 score 2). F1 ranks below F4 even with the surfer criterion added. +- **F3 (action reduction):** regresses to the h6btt regime, removes documented architectural progress, is not reversible, and total surfer-principle violation (criterion-7 score 1). +- **F4 (oracle pretraining):** the right *eventual* answer (criterion-7 score 5 — oracle targets reflect optimal patient behavior) but requires infrastructure that does not exist. + +### §4.3 Predicted outcomes confirming F2 (Phase 7a, 3 specific predictions) + +These are the highest-priority predicted-outcome gates for the revised Phase 7a: + +1. **Q-distill target entropy drops measurably (mechanism gate).** Slot 733 (`RL_Q_DISTILL_TARGET_ENTROPY_MEAN_INDEX`) drops by ≥ 0.5 nats within the first 1000 training steps after F2 is turned ON. This verifies the hinge is non-vacuous: the target distribution is now action-discriminating rather than near-uniform. *Decoupled from outcome — verifies the kernel does what the math says.* + +2. **In-position action distribution shifts away from {Hold, TrailLoosen-not-at-cap} AND surfer-style flat-state patience is preserved (behavioral gate).** Two conjunct conditions must hold: + - Conditional on `position_lots > 0` AND `max_unit_trail < cap`: mass on Hold + TrailLoosen falls below 60% by step 5000. + - Conditional on `position_lots == 0`: Hold% does NOT collapse to either 0% (would indicate the agent is forcibly opening every step, surfer principle lost) or > 95% (would indicate h6btt-class re-collapse). Target band: Hold% ∈ [30%, 85%] from flat states at step 5000. + + The second condition is what makes F2's verdict surfer-aware: F2 must show *learned patience* (some Hold-from-flat, not zero) alongside *learned action* (Open mass at edge-bearing states). + +3. **Eval pnl improves vs h6btt baseline by at least $5M / session (outcome gate).** This is the primary outcome verdict. F2 must demonstrate eval pnl ≥ h6btt baseline + $5M / session. If passed alongside conditions 1 and 2, we have strong evidence that the surfer mode is reachable from the current Phase 5 substrate with F2's gradient redirection alone, and F5 is not needed. + +### §4.3.5 Predicted outcomes confirming F5 (Phase 7b, only run if Phase 7a's F2-G8 fails) + +1. **Action distribution from flat states shifts to {OpenLong, OpenShort} only (mechanism gate).** Action_hist conditioned on `position_lots == 0` should have ZERO mass on `{Hold, TrailLoosen, TrailTighten, FlatFromLong, FlatFromShort, HalfFlatLong, HalfFlatShort, PyramidLong, PyramidShort}` and 100% mass on `{OpenLongSmall, OpenLongLarge, OpenShortSmall, OpenShortLarge}` by step 100. *Decoupled from outcome — verifies the kernel works.* + +2. **Trade frequency increases dramatically (behavioral gate).** Per F5-G4, opens per 1000 steps ≥ 50. The agent is forced to commit; whether it commits *profitably* is the outcome question. **Acknowledged surfer-principle violation:** this gate is *expected* to show the agent abandoning patience-while-waiting; that is what F5 does by construction. + +3. **Eval pnl improves vs F2-alone baseline by at least $5M / session.** F5 must demonstrate *marginal* benefit over F2 alone to justify accepting the surfer→trend regression. If F5 + F2 produces only marginal improvement over F2 alone, revert to F2 alone (preserve the surfer principle); if F5 + F2 produces substantial improvement, commit both with explicit acknowledgement of the trade-off. + +### §4.4 Predicted outcomes refuting the staged intervention (i.e., we are still going in circles) + +**Phase 7a F2-alone decision summary (4-outcome attribution table):** + +| F2-alone cluster result | Next action | Interpretation | +|---|---|---| +| Eval pnl ≥ +$5M improvement over h6btt baseline AND F2-G5 (surfer patience) passes AND no degenerate attractor | **SHIP F2 ALONE.** F5 not needed. Surfer principle preserved. | Q-distill bias was the load-bearing problem; the agent recovered surfer-style patient entries on the gradient axis alone. | +| Eval pnl improves but a new degenerate attractor surfaces (single-direction collapse, open-then-flat oscillation) | STOP — do NOT add F5 reflexively. Consider F5 Phase 7b only with explicit user acceptance that F5 may worsen the directional collapse. Most likely escalation: encoder investigation or F4 infrastructure. | Gradient redirection alone is operational but surfaced a different pathology; forcing trades (F5) on top would likely amplify the migration. | +| Eval pnl marginal (≥ 0 but < +$5M), all mechanism gates including F2-G5 pass | Consider F5 Phase 7b conditional, with explicit user decision to sacrifice some patience for outcome lift. The trade-off is now small (some surfer patience for some additional eval pnl). | F2 partial; F5 may close the gap at the cost of some surfer-mode regression. | +| Eval pnl regressed below h6btt baseline OR F2-G5 collapsed (Hold% from flat < 5% or > 95%) | STOP, audit foundation. Do NOT add F5 (F5 cannot fix a surfer-patience collapse caused by F2's own hinge dynamics or by encoder saturation). Escalate to encoder-alpha signal investigation or F4 infrastructure. | F2 introduced a different pathology; the foundation needs investigation upstream of the Bellman target. | + +**Detailed failure-mode predictions:** + +- **F2 mechanism gates pass (target entropy drops, in-position no-op mass falls) but eval pnl stays flat at h6btt baseline.** Two interpretations: (a) the encoder is too weak to produce alpha signal that the surfer mode can exploit — F5 fallback is then the disciplined next step, with explicit surfer→trend trade-off; (b) the inaction attractor was a *symptom* of a deeper alignment issue, not the load-bearing problem — escalate to reward-pnl alignment work (`pearl_reward_signal_anti_aligned_with_pnl`) or F4. The discriminator is whether F5 then improves eval pnl: if yes, encoder-weakness was the bottleneck; if no, alignment-or-encoder is the deeper problem. + +- **F2 mechanism gates pass, surfer-patience condition pass (Hold% in [30%, 85%]), but action distribution single-direction-dominated.** E.g., the policy commits to Long-only at edge-bearing states. This is a directional collapse, not an inaction attractor. F5 will not help (F5 forces direction commits, not direction diversity). Escalate to encoder investigation or F4. + +- **F2 mechanism gates pass but F5 follow-up (Phase 7b) shows no marginal benefit.** F2 alone is sufficient on the inaction axis; F5's structural mask provides no additional eval pnl. Revert F5, ship F2 alone, surfer principle preserved. + +- **F2 mass-zero target failure (failure mode in §3.2 expected mode #1).** During cold-start, the encoder's E_Q is too noisy and no action exceeds `baseline`. The target is uniform-zero; distill gradient pushes π toward uniform; SAC entropy dominates; agent random-walks. Detection: F2-G1 fails (target entropy does NOT drop) OR target entropy spikes above 1.5 nats. Mitigation in implementation plan: warmup phase with F2 OFF, then enable; OR temperature schedule on the hinge. + +- **F2 + F5 fallback BOTH improve mechanism but eval pnl still flat.** Both interventions operational, no outcome. The inaction attractor was masked-but-not-the-problem. Escalate per §5.4 — reward-pnl alignment or encoder are the next investigation targets, NOT another Bellman-target reshape. + +- **Eval pnl WORSENS in Phase 7b F5 vs Phase 7a F2.** F5's forced commits in a zero-edge data subset produce more loss than F2's learned patience. This is a *valid* finding — it would prove F5 is genuinely harmful in low-edge regimes. Revert F5, document the regime-dependence, consider a Phase 7c "F5 with edge gate" (out of scope for this spec). + +- **Local Tier 1.5 shows the policy diverging — l_pi blows up or NaN.** Indicates a kernel bug (most likely in F2's hinge handling of empty-no-op-set when combined with the Phase 4-A band mask). Fix the kernel; if pathology persists, F2 does not compose with the current Phase 5 substrate; reconsider F1 or wait for F4 infrastructure. + +### §4.5 What we should DO before Phase 7a (F2) cluster + +1. Build F2a (static no-op set: Hold, TrailTighten, TrailLoosen) with an ISV master gate (slot TBD, default 0.0). +2. Verify the F2 kernel modification respects the Phase 4-A band-mask composition: when the band forces Hold post-policy, F2 must still see the *pre-band* action distribution for distill purposes (open uncertainty #2 in §7 — load-bearing). +3. Local Tier 1 (correctness, ~6 sec): confirm `pi_target` correctly applies `max(0, E_Q − baseline)` hinge; confirm no NaN; confirm bit-equality with master gate OFF. +4. Local Tier 1.5 (behavior, ~10 min): confirm F2-G1 (target entropy drop), F2-G2 (no-op mass collapse in target), F2-G3 (training stability), F2-G4 (in-position action redistribution), surfer-patience condition (Hold% from flat in [30%, 85%]) — and determinism per `determinism-check.sh --quick` exits 0 at seeds 42/43/44. +5. **If Tier 1.5 passes:** submit cluster Tier 2 (~70 min). If outcome F2-G8 (eval pnl ≥ h6btt + $5M) passes, COMMIT F2 ALONE, do NOT proceed to F5. The gradient axis was the load-bearing problem AND the surfer principle is preserved. +6. **If Tier 1.5 passes mechanism gates but cluster F2-G8 fails:** evaluate per §4.4 — most likely encoder-weakness scenario. If the user accepts the surfer→trend trade-off, proceed to Phase 7b (add F5 on top of F2). If not, escalate to encoder / alignment / F4 infrastructure work. +7. **If Tier 1.5 fails on F2-G1 (target entropy doesn't drop):** the hinge is vacuous (no action exceeds baseline). This is the mass-zero-target failure mode. Fix via warmup or temperature schedule per implementation plan. +8. **If Tier 1.5 passes F2-G1/G2 but Hold% from flat collapses to 0%** (forced opens): the hinge is also pushing Hold mass below the random-floor, contradicting the surfer-principle preservation argument in §4.1.4. Investigate before cluster — likely a baseline-set definition bug. + +### §4.6 What we should DO before Phase 7b (F5 on top of F2), conditional on §4.5 step 6 AND explicit user acceptance of surfer→trend trade-off + +**Phase 7b is gated on TWO conditions:** (a) F2 outcome gate failure, AND (b) explicit user decision to accept the surfer→trend regression in exchange for breaking the h6btt-class structural collapse. Phase 7b should NOT auto-trigger on F2 failure. + +1. Build F5 kernel `rl_state_action_mask.cu` modeled on `rl_band_mask.cu`, with ISV master gate (slot TBD, default 0.0; separate from F2's gate). +2. Wire the mask into BOTH the policy-sampling path AND the Q-distill `pi_target` computation. Verify identical mask in test fixture. Verify F2's hinge respects F5's masked support: `baseline(s) = max_{a ∈ A(s) ∩ no-op-set} E_Q(s, a)` over the F5-masked set; empty-baseline edge case handled per uncertainty #7 in §7. +3. Local Tier 1 (correctness, ~6 sec): confirm mask kernel sets logits to `−∞` for unavailable actions; confirm ZERO mass post-softmax on unavailable actions; confirm F2 hinge handles F5-masked no-op set without NaN. +4. Local Tier 1.5 (behavior, ~10 min): confirm F5-G1 through F5-G6 + F5-G8 (determinism) plus all F2 mechanism gates still pass. +5. **If Tier 1.5 passes:** submit cluster Tier 2. If F5's outcome gate (F5-G8: eval pnl ≥ F2-alone + $5M) passes, commit F2 + F5 with explicit surfer→trend trade-off documented. If not, see §4.4. +6. **If Tier 1.5 fails on F5 mechanism gates:** fix the kernel; this is not an architectural verdict. +7. **If Tier 1.5 passes F5 mechanism gates but fails F5-G5 (single-direction collapse):** F5 has merely migrated the attractor; STOP and escalate to F4. F2 + F5 together will not help with directional collapse. + +--- + +## §5. Falsification gates for F2 + F5 (recommended staged combination, surfer-principle ordering) + +Gates are partitioned into three categories: **F2-specific** (verify the gradient redirection works, observable independent of F5), **F5-specific** (verify the mask works, observable independent of F2; only used in Phase 7b), and **Combined** (require both to be active; outcome-focused). Each F2-specific gate verifies a property F5 cannot affect; each F5-specific gate verifies a property F2 cannot affect. This is the disciplined-combined-intervention discipline required by `feedback_investigation_first_falsification_methodology`. + +### §5.1 Phase 7a — F2-alone gates (the surfer-principle-preserving primary phase) + +#### §5.1.1 Local b=128 Tier 1.5 — F2 mechanism + +| Gate ID | Metric | Pass condition | Distinguishes | +|---|---|---|---| +| F2-G1 (target entropy effect) | Slot 733 `RL_Q_DISTILL_TARGET_ENTROPY_MEAN_INDEX` at step 1000 | Drops by ≥ 0.5 nats vs Phase 5 baseline | F2's hinge is non-vacuous | +| F2-G2 (no-op mass collapse in target) | Sum of `π_target[a]` for `a ∈ no-op-set` | Below 30% by step 1000 (across all states) | F2's `max(0, A)` zeroes no-op mass | +| F2-G3 (training stability) | l_pi, l_v, l_q | No NaN; ≤ 10× Phase 5 baseline | Standard substrate | +| F2-G4 (in-position action distribution) | When `position_lots > 0 AND max_unit_trail < cap`, sum of Hold + TrailLoosen | < 60% by step 2000 | F2 redirects in-position gradient | +| F2-G5 (surfer-patience preserved) | Hold% conditional on `position_lots == 0` at step 5000 | ∈ [30%, 85%] (NOT 0% — forced opens; NOT > 95% — h6btt collapse) | **Surfer principle preservation per §4.1.4** | +| F2-G6 (training stability, full-loss) | All loss magnitudes | No NaN through step 5000 | Standard substrate | +| F2-G7 (determinism) | `determinism-check.sh --quick` exits 0 | seeds 42/43/44 bit-equal | `pearl_determinism_achieved` | + +**Note on F2-G5 (load-bearing).** This is the critical surfer-principle preservation gate added in Revision 3. F2's mathematical argument is that the hinge zeroes Open mass when `E_Q(Open) ≤ E_Q(Hold)`, so the agent should retain *some* Hold mass at no-edge flat states. If Hold% from flat collapses to ≈ 0%, the hinge is forcing opens (contradicting the §4.1.4 argument) — possible causes: baseline-set definition bug, SAC entropy overwhelming the hinge, or the encoder's E_Q is so saturated that every Open exceeds Hold at every flat state. Investigate before cluster. If Hold% from flat is > 95%, the hinge is not breaking the h6btt-class attractor — distill gradient is insufficient to compete with the SAC prior; F5 fallback is the disciplined next step (Phase 7b). + +#### §5.1.2 Cluster b=1024 Tier 2 — F2 outcome + +| Gate ID | Metric | Pass condition | Distinguishes | +|---|---|---|---| +| F2-G8 (eval pnl primary) | `eval_summary.total_pnl_usd` | ≥ h6btt baseline + $5M/session | F2 alone is sufficient → STOP, do not run F5 | +| F2-G9 (no new attractor in cluster) | Max single-action share at eval step 5000 + step 15000 | < 80% | Detects mq2pc-style migration to a new degenerate | +| F2-G10 (eval action entropy) | `eval_summary.eval_action_entropy_mean` | ≥ 1.0 nats | Confirms diversity preserved | +| F2-G11 (surfer-mode fingerprint in eval) | eval `wr_ema` AND `avg_hold_steps` | wr_ema ≥ 0.45 AND avg_hold_steps ≥ 15 | Positive evidence agent recovered surfer-style patient entries | + +**Decision rule:** if F2-G8 passes → commit F2 alone; F5 is not needed; the gradient axis was the load-bearing problem AND the surfer principle is preserved. If F2-G8 fails AND F2-G1/G2/G3 pass (mechanism works) AND F2-G5 passes (surfer patience preserved) AND F2-G9 passes (no degenerate migration) → evaluate per §4.4. If F2-G8 fails AND F2-G5 collapsed (no surfer patience because every Open exceeds Hold by saturation, or because the hinge zeroed Hold from flat too aggressively), investigate before any F5 fallback — F5 will worsen this pathology. + +### §5.2 Phase 7b — F5-on-top-of-F2 gates (conditional, surfer→trend trade-off) + +These gates run only if Phase 7a's F2-G8 failed under the conditions above AND the user has explicitly accepted the surfer→trend trade-off. F2 remains ON; F5 is added. + +#### §5.2.1 Local b=128 Tier 1.5 — F5 mechanism (with F2 active) + +| Gate ID | Metric | Pass condition | Distinguishes | +|---|---|---|---| +| F5-G1 (flat-state mask correctness) | Among `position_lots == 0` batches, action_hist mass on `{Hold, TrailLoosen, TrailTighten, FlatFromLong, FlatFromShort, HalfFlatLong, HalfFlatShort, PyramidLong, PyramidShort}` | EXACTLY 0 (not "low"; zero) | F5 mask works mechanically | +| F5-G2 (in-position direction mask correctness) | When long, mass on short-side opens / closes / pyramids | EXACTLY 0; symmetric for short | F5 mask works for in-position branch | +| F5-G3 (trail-at-cap mask correctness) | When `max_unit_trail ≥ trail_max − eps`, mass on TrailLoosen | EXACTLY 0 | mq2pc-specific mechanism gate | +| F5-G4 (flat→in-position rate) | Opens per 1000 steps from flat | ≥ 5× mq2pc baseline (≈ 50/1000) | Positive evidence of choice-set forcing (and acknowledged surfer-principle violation) | +| F5-G5 (no single-direction collapse) | Max single-action share across all positions | < 80% | Detects expected failure mode #7 | +| F5-G6 (training stability with F2) | l_pi, l_v, l_q | No NaN; ≤ 10× F2-alone baseline | F2 + F5 do not pathologically interact | +| F5-G7 (empty-baseline guard) | Fraction of batches where `A(s) ∩ no-op-set = ∅` (e.g., F5 removed all no-ops from flat) | Handled without NaN; documented per-batch | F2 + F5 composition edge case | + +#### §5.2.2 Cluster b=1024 Tier 2 — F5 outcome (marginal vs F2-alone) + +| Gate ID | Metric | Pass condition | Distinguishes | +|---|---|---|---| +| F5-G8 (eval pnl marginal benefit) | `eval_summary.total_pnl_usd` | ≥ F2-alone cluster eval pnl + $5M/session | F5 provides marginal value; if no marginal → revert to F2-alone commit | +| F5-G9 (Q-distill controller stability) | Slot 488 `RL_Q_DISTILL_KL_EMA_INDEX` | No monotonic growth through training | Distill controller accepted F5-masked target shape | +| F5-G10 (in-position action entropy in eval) | Action entropy conditional on `position_lots > 0` | ≥ 1.0 nats | In-position diversity preserved despite mask | + +**Decision rule:** if F5-G8 passes → commit F2 + F5 with explicit documentation of the surfer→trend trade-off. If F5-G8 fails (no marginal benefit over F2-alone) → revert F5, commit F2-alone, and escalate the residual to a separate spec (likely encoder-alpha or F4 infrastructure). + +### §5.3 Distinguishing F2 effect from F5 effect (the load-bearing discipline) + +A reader of the cluster diag must be able to attribute eval-pnl change to F2 or F5 specifically. The gate design enables the following distinguishable cluster outcomes (Phase 7a outcomes first, Phase 7b outcomes second): + +| Phase | F2 mech gates (G1,G2,G3) | F2-G5 (surfer patience) | F5 mech gates | Outcome | Attribution | +|---|---|---|---|---|---| +| 7a | PASS | PASS (Hold% from flat ∈ [30%, 85%]) | n/a | F2-G8 PASS | **SHIP F2 ALONE.** Gradient redirection was the load-bearing problem; surfer principle preserved. F5 not needed. | +| 7a | PASS | PASS | n/a | F2-G8 FAIL, no degenerate migration | F2 necessary but insufficient; evaluate per §4.4. If user accepts surfer→trend trade-off, proceed to 7b. Otherwise escalate to encoder / alignment / F4 work. | +| 7a | PASS | FAIL (Hold% from flat < 5% or > 95%) | n/a | any | Surfer principle broke in F2 alone (either over-action or under-action). Investigate before F5. F5 will worsen the over-action case. | +| 7a | PASS | PASS | n/a | F2-G8 FAIL, F2-G9 FAIL (new degenerate) | F2 surfaced a different pathology; STOP, escalate to F4 (do NOT add F5 on top of a degenerate). | +| 7a | FAIL | n/a | n/a | any | F2 mechanism broken (mass-zero target). Fix kernel (warmup / temperature schedule). | +| 7b | PASS | n/a | PASS | F5-G8 PASS | F5 marginal benefit confirmed; commit F2 + F5 WITH explicit surfer→trend trade-off documented. | +| 7b | PASS | n/a | PASS | F5-G8 FAIL | F5 has no marginal benefit; revert F5, commit F2-alone, escalate residual to encoder / alignment work. | +| 7b | PASS | n/a | FAIL | any | F5 mechanism broken; fix or abandon F5. | +| 7b | FAIL | n/a | n/a | any | F2 broke when F5 turned on; F2 + F5 composition is the bug (likely empty-baseline edge case). | + +Per `feedback_investigation_first_falsification_methodology` §"falsifiable verdicts", every cluster run produces an attribution row in this table. Going in circles is structurally prevented because the row identifies the specific next action. + +### §5.4 The "if F2 + F5 are both wrong" prediction + +Per `feedback_going_in_circles_pattern`, the spec must predict what failure looks like if both intervention layers fail — so we don't tile the next fix on top of these. + +**If F2 + F5 both pass mechanism gates but eval pnl stays at h6btt baseline:** the inaction attractor was a *symptom* of a deeper problem (most likely reward-pnl alignment per `pearl_reward_signal_anti_aligned_with_pnl`, or encoder alpha extraction, or train→eval generalisation per `pearl_surfer_baseline_was_train_only_never_eval`). The gradient is correctly redirected away from no-ops (F2), the choice-set is constrained from collapsing to inaction (F5), and the agent still cannot make eval-profitable trades — that isolates the failure to either (a) the reward signal is not pnl-aligned, or (b) the encoder is not extracting alpha. Neither is solvable by another Bellman-target reshape. + +**Specifically watch for these signals:** + +- Action distribution diverse (action_entropy ≥ 1.5 nats), no degenerate attractor, eval pnl negative or zero → reward-pnl misalignment is dominant; escalate to a reward-signal investigation per `pearl_reward_signal_anti_aligned_with_pnl`. +- Action distribution diverse, eval pnl negative, train pnl significantly positive (≥ +$5M) → train→eval generalisation failure per `pearl_surfer_baseline_was_train_only_never_eval`; escalate to a regularisation / sim2real spec. +- Action distribution single-direction-dominated (e.g., 90% Long open, then immediate close), eval pnl noise-floor → encoder is not producing directional alpha; escalate to encoder investigation. + +**If any of these occur, STOP per `feedback_going_in_circles_pattern`. The next intervention is NOT a further reshape of the Bellman target — it is a foundation-level intervention upstream. F4 (oracle pretraining) becomes the right next dispatch if the encoder hypothesis dominates; reward-signal investigation if alignment dominates.** + +--- + +## §6. Out of scope + +This spec contains TWO interventions (F2 and F5) but they are disciplined as a single combined Phase 7 with a staged sub-sequence (7a = F2-alone, 7b = F5-conditional). This is NOT four-fix stacking; it is a single decision tree with one branch point. The discipline rule from `feedback_going_in_circles_pattern` — "one fix at a time with predetermined gates" — is honored because (a) Phase 7a tests F2 in isolation (surfer-principle-preserving primary phase), and (b) Phase 7b is only entered if Phase 7a's outcome gate identifies the residual AND the user accepts the surfer→trend trade-off encoded in F5. + +This spec deliberately does NOT cover: + +1. **F4 oracle-pretraining infrastructure.** Belongs in a separate spec (~`2026-06-05-offline-rl-infrastructure.md`). Estimated 3–4 weeks. Should be drafted after Phase 7a/7b cluster verdict. +2. **F1 (per-step inaction cost) implementation details.** Backup option only; specified at theoretical level only here. +3. **F3 action-space reduction.** Disfavored; not specified at implementation level. F5 is the option F3 should have been. +4. **Reward-policy alignment investigation.** Already covered by `2026-06-01-reward-policy-alignment-investigation.md`. F5 + F2 are orthogonal — the reward signal and the choice-set / distill target are independent mechanisms. +5. **GAE / Phase 1B rollout buffer.** Orthogonal; can ship independently. GAE benefits V quality; F5 benefits the policy support; F2 benefits π update locality — none blocks the other. +6. **Multi-head policy (Phase 2A) interaction tuning.** F5 and F2 should work on both single-head and multi-head architectures. The mask in F5 is per-batch per-action, independent of head; F2 modifies the distill target shape per-head. Multi-head verification is part of cluster Tier 2. +7. **Encoder alpha investigation.** If F5 + F2 both pass mechanism gates but eval pnl stays at baseline (§5.4), the encoder may be implicated. That escalation path is a separate spec. +8. **Controller recalibration after F2.** The Q-distill controller (slot 486) is tuned for the current target shape. F2 reshapes the target → the controller may need re-bootstrapping. This is *implementation work* for the F2 plan document, not design work for this spec. +9. **CUDA Graph A/B capture impact.** F5 and F2 kernel diffs are small and should preserve graph capture. Verification belongs in the implementation plan. +10. **Synthetic-Hold fallback for zero-edge regimes.** If F5 forces unprofitable trades in no-edge market regimes (failure mode #1), a "synthetic abstain" via micro-open / band-narrow-open is a candidate Phase 7c intervention. Not specified at this stage; depends on Phase 7a outcome. +11. **Confidence-gate position-aware wiring.** Per `pearl_confidence_gate_pos_state_unwired`, `rl_confidence_gate.cu` accepts `pos_state` but doesn't read it. F5 does not depend on fixing this — F5 operates at a different pipeline stage (pre-softmax logit mask, not post-action override). The confidence-gate wiring is a separate fix and out of scope. + +--- + +## §7. Honest uncertainty flags + +This section explicitly flags claims the spec makes that are speculative or unverified, per `feedback_investigation_first_falsification_methodology`: + +1. **§3.2 mass-zero distill target risk.** The risk that "no action's E_Q exceeds the baseline" during early training is real but unquantified. Without a Tier 1.5 run, the probability is unknown. The mitigation (SAC entropy preserves uniform exploration when target is zero) is intuitive but not verified. +2. **§3.2 F2-band interaction.** Phase 4-A band masking short-circuits the policy's chosen action *after* the policy emits it but *before* the action is executed. The Q-distill kernel reads the policy's emitted action and the Q-targets — it does NOT directly read the executed action. Whether F2 still works correctly when the band overrides actions to Hold is an open question requiring code verification in the implementation plan. **This is a load-bearing uncertainty.** F5 partially supersedes this by operating at the *logit* stage (before sampling) rather than the *action* stage (post-sampling) — F5 + F2 composition through the band is cleaner because F5 itself is the same lattice as the band and runs in series with it. +3. **§3.3 F3's reintroduction of static trail risk.** The argument that removing TrailTighten/TrailLoosen reintroduces a known-suboptimal risk model is sound, but the *magnitude* of the regression depends on volatility regime. In a low-volatility data subset, static trails might be fine; in high-volatility, they're disastrous. Unquantified. (Moot since F3 is rejected; F5 does not remove these actions, only conditionally masks them.) +4. **§3.4 F4 scope estimate.** "2000–5000 LOC" is an order-of-magnitude guess based on the size of comparable offline-RL implementations elsewhere. Foxhunt's specific scope could be larger if the existing dataset infrastructure does not support per-state oracle lookups, or smaller if a minimal one-epoch version is acceptable. +5. **§3.5 F5 force-trade failure mode.** Failure mode #1 (forcing trades in zero-edge regimes) is the biggest substantive risk in F5. There is no quantitative estimate of how often the ES MBP-10 dataset contains zero-edge regimes vs alpha-bearing regimes. If a significant fraction of trading sessions has no exploitable alpha, F5's forced trading from flat could *worsen* eval pnl vs the do-nothing baseline. The h6btt eval pnl baseline is itself negative — if F5 makes it *more* negative, that is a Phase 7a refutation per §4.4. +6. **§3.5 F5 mask propagation into Q-distill correctness.** Failure mode #3 — applying the mask asymmetrically to π_θ and `pi_target` — is the biggest *implementation* risk. The mask must propagate identically into the Q-distill kernel. The original `rl_confidence_gate.cu` failure (per `pearl_confidence_gate_pos_state_unwired`) is precedent that "this kernel reads `pos_state`" can be silently false. Tier 1 correctness tests must explicitly verify that masked actions have zero mass in BOTH the policy distribution AND the `pi_target` distribution. +7. **§3.5 F5 + F2 baseline-empty edge case.** When F5 masks all no-ops from a given state (e.g., from flat, all of {Hold, TrailLoosen, TrailTighten} are unavailable), F2's `baseline(s) = max_{a ∈ A(s) ∩ no-op-set} E_Q(s, a)` is over an empty set. The natural definition is `baseline(s) = −∞` in this case (any action's `E_Q > −∞` → hinge passes for all), but this requires explicit handling in `rl_q_pi_distill_grad.cu`. F5-G7 (in §5.2.1) catches this; the Phase 7b implementation plan must specify the behavior. +8. **§4.1 score matrix weighting.** The implicit weighting (equal across criteria) is arbitrary. A different weighting (e.g., "implementation cost matters 3× as much as theoretical soundness") would push the recommendation toward F5/F2 even more strongly. A pro-research weighting (theoretical soundness 3×) would push toward F4. The spec author judges roughly-equal weighting as most appropriate. +9. **§4.1.5 "F2 alone may be sufficient" claim (revised in Revision 3).** This is a structured guess, not a verified prediction. The argument (the surfer alpha mode is reachable from the current Phase 5 substrate; F2's hinge expresses the surfer principle as a loss function; the encoder produces sufficient edge signal for E_Q(Open) > E_Q(Hold) at edge-bearing states) is plausible but unverified. The opposing failure scenario (encoder too weak; F2 degenerates to uniform random) is the load-bearing risk to monitor. Phase 7a's cluster verdict is the only way to know. +10. **§5.1 / §5.2 thresholds (0.5 nats target-entropy drop, 5× open-rate increase, $5M eval pnl improvement, [30%, 85%] Hold% band for surfer-patience).** All threshold values are guesses motivated by mq2pc / h6btt / dd049d9a4 baselines but not pre-calibrated. The Tier 1.5 implementation should report *actual* deltas; the plan document can tighten thresholds based on smoke data. +11. **§5.2 F5-G8 marginal benefit threshold ($5M / session BEYOND F2-alone).** If F2 alone produces a $4M improvement and F5 produces an additional $4M, the combined $8M passes F2-G8 but fails F5-G8 (which requires $5M *marginal*). The threshold is conservative — it prefers "F2 alone is enough" over "F2+F5 is slightly better." This may understate F5's true value; consider lowering to $2-3M in the Phase 7b plan document. The bias toward F2-alone is intentional in Revision 3 because preserving the surfer principle is preferred when the marginal benefit of F5 is small. +12. **§4.1.4 surfer-patience band [30%, 85%] (Revision 3 addition).** The [30%, 85%] Hold% range for F2-G5 is calibrated against the dd049d9a4 surfer baseline (which had avg_hold ≈ 30 with frequent positions; back-of-envelope this implies Hold-from-flat in roughly the 40-70% range) but not directly measured at the per-state-conditional level. The exact target band should be re-derived from per-state action histograms in the surfer baseline diag.jsonl during plan development. + +--- + +## §8. Document discipline notes + +- **No LOC, no code, no implementation order in this spec.** Per `feedback_investigation_first_falsification_methodology` §6: spec is WHY + design + falsification; plan is HOW. This is the spec. (LOC estimates in §3.1, §3.2, §3.5 are *scoping* notes for the comparative analysis; they are not implementation prescriptions.) +- **Staged combined intervention with single-fix-per-phase discipline (revised in Revision 3).** Per `feedback_going_in_circles_pattern`: one fix at a time with predetermined gates. The combined F2 + F5 recommendation is realized as Phase 7a (F2-alone, surfer-principle-preserving) followed by conditional Phase 7b (F5 added, surfer→trend trade-off accepted). At each phase boundary, exactly one new mechanism is being tested, with its own mechanism gates. This is NOT two-fix stacking; it is a decision tree with one branch point informed by a falsifiable outcome gate. +- **Both mechanism gates and outcome gates.** Mechanism gates (F2-G1/G2 for the gradient axis; F5-G1/G2/G3 for the choice-set axis if Phase 7b is invoked) tell us if the kernel does what the math says it does. Outcome gates (F2-G8 for Phase 7a; F5-G8 for Phase 7b) tell us if the math is right about the policy effect. The Revision 3 surfer-patience gate F2-G5 is a *behavioral* gate that fits between mechanism and outcome — it verifies the policy expresses the surfer principle in the action distribution, even before the cluster outcome arrives. All three categories are required to extract maximum information from each cluster run. +- **Distinguishable attribution.** §5.3's 4-outcome table makes every cluster verdict map to a specific next action. Going in circles is structurally prevented. +- **Pre-registered failure prediction.** §5.4 commits in advance what failure looks like if both F2 and F5 pass mechanism gates but eval pnl stays flat, so we cannot rationalize ambiguous results. + +--- + +## §9. Appendix — Plan handover surface + +### §9.1 Phase 7a (F2-alone) plan handover — the next deliverable + +If F2 is selected and proceeds to implementation (the Revision 3 recommendation), the plan document at `docs/superpowers/plans/2026-06-04-bellman-target-foundation-reshape-F2-plan.md` should cover: + +- New ISV slots: F2 master gate (binary, bootstrap 0.0); no-op-set mode enum (F2a static vs F2b dynamic); optional λ-controller-bootstrap slot. +- Kernel diff for `rl_q_pi_distill_grad.cu` lines 80–91 (the `s_pi_target` computation): add the baseline subtraction + `max(0, ·)` hinge over the no-op-set, then softmax. +- **Definition of the no-op set.** Start with F2a (static set: Hold, TrailTighten, TrailLoosen). If cluster shows F2a works, defer F2b (dynamic, lots-change-based) to a follow-up plan. +- **Mass-zero target failure mode handling (load-bearing).** During cold-start, the encoder's E_Q may be too noisy and no action exceeds `baseline`. The target is uniform-zero; SAC entropy dominates; agent random-walks. Mitigation: warmup phase with F2 OFF (first 500-1000 steps), then enable; OR temperature schedule on the hinge. +- **Phase 4-A band-mask interaction (load-bearing per §7 uncertainty #2).** The band-mask post-overrides the policy's chosen action *after* the policy emits it. F2's distill target reads E_Q and produces `π_target` over all actions; the policy's emitted action goes through the band. Whether F2's gradient remains correct when the band overrides actions is an open question. Verify in the implementation plan via code reading + correctness test. +- **Surfer-principle preservation verification (F2-G5).** The plan must include a Tier 1.5 test that explicitly measures Hold% from flat states and gates on the [30%, 85%] band. If the local smoke produces Hold% from flat outside this band, investigate the baseline-set definition before cluster. +- Trainer-side dispatch (kernel call sequence unchanged; input distribution unchanged; ISV-driven branch is new). +- Diag emit changes — slot 733 already exists, but new diag fields for "action mass in no-op-set" and "fraction of batches where target is zero-mass" per `pearl_fleet_fraction_not_aggregate`. Add diag for Hold% conditional on `position_lots == 0` (the F2-G5 metric). +- Controller re-bootstrap — the distill-λ controller (slot 486) and SAC-α (slot 581) may need re-bootstrap because their setpoints assume the previous target shape. +- Tier 1 (correctness, ~6 sec): `pi_target` no-op mass = 0 for any state where any `a_c` has `E_Q > baseline`; no NaN; bit-equality with master gate OFF. +- Tier 1.5 (behavior, ~10 min): F2-G1 through F2-G7 (including the F2-G5 surfer-patience gate). +- Tier 2 (eval verdict, ~70 min): F2-G8, F2-G9, F2-G10, F2-G11 (the surfer-fingerprint gate). +- Cluster commit boundary: one commit per phase per `feedback_commit_per_phase_for_long_coder_tasks`. + +### §9.2 Phase 7b (F5-on-top-of-F2) plan handover, conditional on §9.1 outcome AND user acceptance of surfer→trend trade-off + +If F2-G8 fails per §4.5 step 6 AND the user explicitly accepts the surfer→trend trade-off, the plan document at `docs/superpowers/plans/2026-06-04-bellman-target-foundation-reshape-F5-plan.md` should cover: + +- New ISV slots: F5 master gate (binary, bootstrap 0.0; separate from F2's gate); trail-eps tolerance (float, bootstrap ≈ 1e-2 · trail_max). +- New kernel `crates/ml-alpha/cuda/rl_state_action_mask.cu` modeled on `rl_band_mask.cu` lattice (Grid=(B), Block=(1,1,1)), reading `pos_state[b*pos_bytes+0..4]` for `position_lots` and `trail_distances[b*MAX_UNITS+u]` for per-unit trail distances. Action mask via `logits[b*N_ACTIONS+a] = −∞` sentinel. +- Kernel launch site in `crates/ml-alpha/src/trainer/integrated.rs` — between policy logits emission and softmax sampling. Determine the precise call site by reading the existing dispatch chain. +- **Mask propagation into Q-distill `pi_target` computation (load-bearing).** Either pass the same masked logits to `rl_q_pi_distill_grad.cu`, or pass the `pos_state` + ISV and recompute the mask identically inside the distill kernel. The chosen approach must be verified by a correctness test that the masked-action mass is zero in BOTH the policy distribution AND the distill target. +- **Empty-baseline edge case handling (load-bearing).** When F5 masks all no-ops from a given state (e.g., from flat, all of {Hold, TrailLoosen, TrailTighten} become unavailable), F2's `baseline(s) = max_{a ∈ A(s) ∩ no-op-set} E_Q(s, a)` is over an empty set. Use `−∞` sentinel so the hinge passes for all actions; document the per-batch fraction of empty-baseline states. +- Diag emit changes: per-batch fraction-masked-from-flat, fraction-masked-trail-at-cap (cf. `pearl_fleet_fraction_not_aggregate`). Wire into existing diag emission. +- Test fixtures in `crates/ml-alpha/tests/trade_management_kernels.rs` (parallel to existing band-mask tests): flat-state forces opens; in-position forbids opposite-direction opens; trail-at-cap forbids TrailLoosen; bit-equality with master gate OFF. +- **Explicit documentation of surfer→trend trade-off in commit message and plan summary.** F5 is being shipped because F2 alone was insufficient and the user has consciously decided that breaking the h6btt-class structural collapse is worth the surfer-principle regression. +- Tier 1 (correctness, ~6 sec): kernel correctness + Q-distill mask correctness + F2 hinge handles F5-masked no-op set + no NaN. +- Tier 1.5 (behavior, ~10 min): F5-G1 through F5-G7 + all F2 mechanism gates still pass. +- Tier 2 (eval verdict, ~70 min): F5-G8 + F5-G9 + F5-G10. +- Cluster commit boundary: one commit per phase per `feedback_commit_per_phase_for_long_coder_tasks`. + +--- + +## §10. Linked work + +- Phase 4-A band (`2026-06-03-no-transaction-band-architecture.md`): the band mask post-overrides the policy's chosen action when inside the band. F2 modifies the Q-distill target shape, which reads E_Q over all actions; the policy's emitted action subsequently goes through the band. F2 must verify it composes with the band mask — see §7 uncertainty #2 (load-bearing). F5 (if Phase 7b is triggered) is the band mask's structural twin and runs in series with it: F5 applies first (pre-softmax logit mask); band mask applies second (post-action override to Hold when inside band). +- Phase 5 MTM reward (`0463e44e0`): F2 + F5 leave the reward signal untouched. MTM continues to operate. All interventions can compose. +- Phase 2A multi-head (`2026-06-02-multi-head-policy-with-r-multiple.md`): F2 modifies the Q-distill kernel which is shared across heads. F5 mask (if invoked) is per-batch per-action, head-independent. The modifications are per-action, not per-head, so multi-head architecture is unaffected by F2 or F5 mechanism. +- Phase 1B GAE (`2026-06-02-trainer-rollout-buffer-gae.md`): orthogonal; can ship independently. + +--- + +## §11. Decision log entry + +If F2 (Phase 7a) is approved for implementation, log the decision per `feedback_v7_gem_methodology` §decision-log: + +``` +Decision: Adopt F2 (Q-distill advantage target with max(0, E_Q − baseline) hinge) as Phase 7a, with F5 (state-conditional action availability mask) staged as conditional Phase 7b ONLY IF F2 outcome gate fails AND user explicitly accepts surfer→trend regression trade-off. +Rejected: F1 (inaction-cost reward shaping), F3 (action-space reduction), F4 (oracle pretraining — gated on infrastructure that does not exist). +Rationale: F2 addresses the structural inaction-bias problem via the gradient axis (Q-distill target shape) while PRESERVING the surfer trading principle by mathematical construction. The hinge `max(0, E_Q(a) − baseline)` zeroes Open mass when no edge is detected and zeroes Hold's contribution when other actions exceed it — the agent learns to wait or act based on the encoder's edge signal, not a structural constraint. F2 IS the surfer principle expressed as a loss function (per pearl_dd049d9a4_surfer_baseline_verified, the surfer alpha mode wr=0.56, hold=30, +$10.6M @ step 18k is the cluster-proven target). + +F2 is theoretically grounded in Wang et al. 2016 (Dueling Network Architectures, advantage decomposition), Schulman et al. 2015 (GAE, advantage-based policy updates), and Mnih et al. 2016 (A3C, advantage-actor-critic). F2 is implementable with bounded scope (~50-100 LOC across the rl_q_pi_distill_grad.cu kernel) and is falsifiable with deterministic mechanism gates (target entropy drop, no-op mass collapse) AND a surfer-principle preservation gate (Hold% from flat ∈ [30%, 85%]). + +F5 is staged as conditional Phase 7b because (a) F5 violates the surfer trading principle by removing Hold-from-flat from A(s) — destroying patience-while-waiting-for-setup, the load-bearing structural property of the surfer alpha mode — and (b) per `feedback_going_in_circles_pattern` the most disciplined sequence tests the surfer-preserving option first. If F2 alone passes F2-G8 (eval pnl ≥ h6btt + $5M) with surfer-patience preserved (F2-G5), F5 is deferred. If F2-G8 fails AND F2 mechanism gates pass AND no new degenerate attractor surfaces AND the user explicitly accepts the surfer→trend trade-off, F5 is added on top. + +Revision history: Revision 1 (2026-06-04) recommended F2 alone. Revision 2 (2026-06-04b) added F5 and recommended F5 first / F2 conditional based on pre-surfer-criterion scoring. Revision 3 (2026-06-04c, THIS revision) added the surfer trading principle as a 7th scoring criterion and flipped to F2 first / F5 conditional. Score totals: F2=30, F4=29 (gated), F5=28, F1=24, F3=14. + +F4 is the longer-term right answer but requires infrastructure that does not exist. + +Date: 2026-06-04 (Revision 3 — surfer-principle-aware staging) +Falsification trigger (Phase 7a): F2-G8 per §5.1.2 + F2-G5 per §5.1.1 +Falsification trigger (Phase 7b, conditional + user-acceptance-gated): F5-G8 per §5.2.2 +Going-in-circles trigger: §5.4 (both passed mechanism, eval still flat → escalate to F4 or reward-alignment) +Distinguishable-attribution discipline: §5.3 (outcome attribution table) +Surfer-principle decision criterion: §4.1.4 +``` diff --git a/docs/superpowers/specs/2026-06-05-f4-oracle-pretraining-infrastructure.md b/docs/superpowers/specs/2026-06-05-f4-oracle-pretraining-infrastructure.md new file mode 100644 index 000000000..2ac297951 --- /dev/null +++ b/docs/superpowers/specs/2026-06-05-f4-oracle-pretraining-infrastructure.md @@ -0,0 +1,733 @@ +# F4 — Oracle Pretraining Infrastructure Spec + +> **For agentic workers:** This spec defines the INFRASTRUCTURE for F4 oracle pretraining; it does NOT itself implement F4. Implementation follows in a separate plan after this spec is approved. + +**Status:** Draft 1 (2026-06-05) +**Predecessor:** `docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md` (F4 §3.4) +**Phase 7a/7b empirical context:** F2 + F5 + OptA committed (HEAD `7376b1c67` on `ml-alpha-regime-observer`); cluster bleed −$6.4M @ step 519 → projected −$247M @ step 20k (sk2z9). F2 alone hinge_zero_rate saturated to 1.0 (mq2pc descendant); F5 alone surfaced 6:1 Short:Long bias; F2+F5 produced trade rate but 3× faster bleed than F2-alone. + +**Branch target:** TBD (off post-Phase-7b commit `7376b1c67` once this spec is approved). + +**Estimated wall-clock to evaluable F4-MVP cluster verdict:** ~7-10 working days (see §6). Honest range: 7 days if every phase passes first attempt; 14+ days if Phase 7c.0 oracle ground-truth fails or Phase 7c.5 cluster regresses. + +**Scope discipline:** This is an INFRASTRUCTURE spec. It defines what must be BUILT to make F4 evaluable. It does NOT specify F4 hyperparameters, prescribe LOC counts at the kernel level, or replace the implementation plan that must follow. F4-Full (multi-epoch, full-action distribution, encoder-freeze variants) is explicitly OUT OF SCOPE; only F4-MVP is in scope. + +**Linked specs (load-bearing context):** + +- `docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md` — F4 §3.4 (predecessor), F2 §3.2 (Phase 7a, committed), F5 §3.5 (Phase 7b, committed) +- `docs/superpowers/specs/2026-06-01-reward-policy-alignment-investigation.md` — Pearson-misalignment foundation, established the reward function shape that F4 oracle must match +- `docs/superpowers/specs/2026-06-02-fast-dev-cycle.md` — Tier 1 / 1.5 / 2 funnel that all phase gates use +- `docs/superpowers/specs/2026-06-02-determinism-foundation.md` — bit-equality guarantee F4 must preserve + +**Linked pearls (load-bearing):** + +- `pearl_foxhunt_pi_trained_by_q_distillation_not_ppo` — π's only gradient source is Q-distill; F4's pretrained Q is what π distills from +- `pearl_foxhunt_trainer_is_genuinely_single_step` — `[B]`-only buffer shape; F4-MVP intentionally preserves this constraint +- `pearl_advantage_kernel_is_done_gated_td_not_gae` — V regression operates at T_eff=1; F4 oracle returns are γ-discounted not bootstrap-1-step +- `pearl_diag_pnl_fields_are_shaped_reward_not_usd` — load-bearing reward-function-identity caveat; F4 oracle MUST consume the same shaped reward function the trainer consumes (NOT raw USD pnl) +- `pearl_phase5_term_4_is_almost_potential` — Ng-Harada-Russell verdict on Phase 5 shaping terms; F4 oracle inherits the same non-potential terms (entry cost, MTM, short-hold penalty) +- `pearl_determinism_achieved` — same-seed bit-equality, F4 pretrain phase must preserve +- `pearl_fleet_fraction_not_aggregate` — at b=1024+ aggregate diag hides per-batch reality; F4 oracle quality gates must report fleet fractions +- `pearl_reward_signal_anti_aligned_with_pnl` — the load-bearing motivation for any oracle-style supervised warm-start +- `pearl_surfer_baseline_was_train_only_never_eval` — F4 gates must measure eval pnl, not train; train→eval transfer is itself a gate + +--- + +## §1. Why F4, why now + +### §1.1 The Phase 7 empirical trajectory + +Three cluster runs after Phase 5 substrate, each with a distinct mechanism activation profile: + +| Cluster | Phase | What was enabled | Mechanism observed | Eval pnl (cluster) | +|---|---|---|---|---| +| h6btt | Phase 5 single-head | (baseline; pre-Phase-7) | Hold-collapse to 98%; no opens | bleed ≈ $4.18/batch/step | +| mq2pc | Phase 5 + multi-head | (baseline) | TrailLoosen-collapse to 90%; oscillating attractors | bleed ≈ $19/batch/step | +| sk2z9 | Phase 7a F2 + Phase 7b F5 + OptA Hold-suppression | F2 hinged distill target + F5 flat-state mask + Gate-aware Hold suppression | F2 hinge active (hinge_zero_rate 1.0→0.91 across training; advantage_max=1.86); F5 forced opens (6:1 Short:Long); trade rate 7× higher than mq2pc | bleed −$6.4M @ step 519, linear projection −$247M @ step 20k (3× faster than F2-alone) | + +### §1.2 What this proves about Phase 7's foundation + +F2 + F5 mechanism gates BOTH pass (verifiable from cluster diag: hinge_zero_rate dropped, advantage_max > 0, flat-state mask correctness 100%, action distribution from flat ∈ {Long, Short}). The Phase 7 spec §5.3 attribution table maps this row exactly: + +``` +Phase 7b: F2 mech PASS | F2-G5 surfer (n/a, F5 active) | F5 mech PASS | F5-G8 FAIL + → "F5 has no marginal benefit OR introduces new pathology; revert F5, commit F2-alone, + escalate residual to encoder / alignment work." +``` + +But F2-alone also failed (sk2z9 sibling run with F5=0 showed hinge_zero_rate 1.0 saturation — Q never discovers Open advantage from cold start; random exploration cannot produce enough above-baseline samples to gradient-train the Q-head out of the inaction attractor). The empirical reading is: + +- **F5 alone (forced opens)** loses money because random forced opens cannot beat commission drag in the no-edge regime fraction (predicted by Phase 7 spec §3.5 expected failure mode #1). +- **F2 alone (gradient redirection)** zeroes everything because cold-start Q has no above-baseline samples; hinge_zero_rate saturates at 1.0 (predicted by Phase 7 spec §3.2 expected failure mode #1 "mass-zero distill target"). +- **F2 + F5 (gradient + forced data)** activates F2's hinge but feeds it the noise distribution of random forced opens; F2 learns "no Open action has positive advantage in expectation" because the data shows it, faster than the encoder can extract real edge signal. + +Per spec §5.3 attribution outcome 3 / §5.4 escalation path: + +> "If F2 + F5 both pass mechanism gates but eval pnl stays at baseline OR worsens: the inaction attractor was a *symptom* of a deeper problem. Neither is solvable by another Bellman-target reshape… F4 (oracle pretraining) becomes the right next dispatch if the encoder hypothesis dominates." + +The encoder hypothesis is now favored: the agent's Q-head cannot bootstrap discriminating-action-quality estimates from cold-start random-policy-rollout-collected data because the signal-to-noise ratio is too low. The cluster data shows the agent CAN find some directional signal (F5-only directional bias 6:1 is non-random; F2's advantage_max=1.86 is non-zero) — but cannot find the *Q-quality* of specific actions in specific states fast enough to break the inaction attractor. + +### §1.3 The disciplined response per `feedback_going_in_circles_pattern` + +The previous spec's §5.3 attribution row identifies F4 as the next dispatch when F2+F5 mechanism gates pass and outcome fails. That row also requires that no further Bellman-target reshape (F1, F3, more F5 variants) be attempted. F4 attacks a structurally different axis — not the gradient, not the choice-set, but the *initial conditions* of Q. F4 either fixes the problem (cluster eval improves) or refutes the "encoder cannot bootstrap" hypothesis (cluster eval still fails) — and that refutation would itself be progress, since it would isolate the problem to genuine reward-pnl misalignment or train→eval distribution shift. + +F4 is therefore the textbook next step under the spec's own escalation logic. This spec defines the infrastructure to make F4 evaluable. + +--- + +## §2. F4 mechanism + +### §2.1 Restated definition + +Per the predecessor spec §3.4: + +> "For each historical state `s_t` in the training data, compute an oracle Q-value `Q_oracle(s_t, a)` using perfect-foresight knowledge of future outcomes. Supervised-train Q to match `Q_oracle` for one or more epochs, THEN RL fine-tune. The agent inherits a Q function that *already knows* Hold is not always optimal because the supervised loss reflects realised returns from each action with perfect foresight." + +In formal terms: + +``` +Q_oracle(s_t, a) = E_{trajectory | a_t = a, π* thereafter} [ Σ_{k=0..∞} γ^k · r_shaped(s_{t+k}, a_{t+k}) ] +``` + +where: +- `r_shaped` is the trainer's reward function (Phase 5 MTM + Phase 5 trail-max ceiling + base pnl + Phase 7a hinge if applicable). Per `pearl_diag_pnl_fields_are_shaped_reward_not_usd`, this is NOT raw USD pnl. +- `π*` is the per-state-after-t optimal policy. "Optimal" is defined by the same `r_shaped` function (the F4-MVP simplification is recursive — oracle's notion of optimal is bootstrapped from greedy-on-oracle-Q one step lookahead; see §2.3 below). +- `γ = 0.975` (foxhunt's current C51 / IQN discount factor). + +### §2.2 Two scope tiers + +#### F4-MVP (this spec's scope) + +- Oracle Q for the **argmax action only**. Specifically: for each state `s_t`, compute one `(s_t, a_argmax, Q_argmax)` triple where `a_argmax` is the highest-realized-return action under a chosen oracle policy. +- Supervised MSE regression for **ONE epoch over the training fold** BEFORE the RL loop begins. +- Q head + encoder both training during the supervised phase (no freeze). +- Single fold (fold 1) only. +- Pretrain writes to the SAME Q head weights that the Bellman target writes to during RL fine-tune. + +#### F4-Full (out of scope, eventual) + +- Oracle Q for all 11 actions per state. +- Multi-epoch supervised regression (curriculum, annealing). +- Per-fold oracle generation across walk-forward folds. +- Encoder-only pretrain phase, Q-head-only pretrain phase, joint variants. +- Distillation regularization to prevent catastrophic Q drift during RL fine-tune. +- Online oracle update during RL (Dyna-style). + +### §2.3 Mathematical relationship to the existing Bellman target + +The trainer's existing Bellman update is: + +``` +Q_target(s, a) = r(s, a) + γ · max_{a'} Q_target_net(s', a') +L_q(s, a) = (Q_θ(s, a) − Q_target(s, a))² +``` + +F4-MVP inserts a supervised-regression phase BEFORE the RL phase: + +``` +Phase F4-pretrain (epochs=1): + for each (s_t, a_argmax, Q_argmax) in oracle dataset: + Q_θ(s_t, a_argmax) ← trained via MSE against Q_argmax + # All other actions Q_θ(s_t, a≠argmax) receive no supervised loss (F4-MVP). + # Encoder gradient flows through Q head as in standard supervised training. + +Phase RL fine-tune (existing trainer): + Q_θ is now initialized to "the oracle-argmax-q-shape Q" + Standard Bellman fine-tune proceeds; Q_θ learns the off-argmax actions via the Bellman target; + Q_θ for argmax actions starts near-correct rather than random. +``` + +The relationship to the inaction-bias problem (Phase 7 spec §2.1 equation c): + +``` +Pre-F4: Q_init(s, a) ≈ random → Q_θ(s, no-op) > Q_θ(s, open) by entry_cost in expectation + → F2 hinge zeroes Open mass → no learning signal → collapse + +Post-F4-MVP: Q_init(s, a_argmax) ≈ Q_oracle(s, a_argmax) + → for edge-bearing states where a_argmax = Open, Q_θ(s, Open) > Q_θ(s, Hold) at init + → F2 hinge passes for Open at edge-bearing states → distill mass on Open → learning proceeds + → for no-edge states where a_argmax = Hold, Q_θ(s, Open) < Q_θ(s, Hold) at init + → F2 hinge zeroes Open mass → distill mass on Hold → surfer-style patience preserved +``` + +F4-MVP + F2 + (optionally) F5 is the conjecture this spec puts under test. Empirical falsification of that conjecture is encoded in the F4-G7/G8/G9 cluster outcome gates in §5. + +### §2.4 The "optimally thereafter" simplification (load-bearing) + +A truly optimal oracle would require knowing `Q*_oracle` for every state — which is the very thing F4 is trying to bootstrap. F4-MVP uses a *greedy-1-step-lookahead-on-realized-return* approximation: + +``` +For each (s_t): + For each action a ∈ A: + # Simulate the action through the trainer's LobSim for K steps (γ-discounted). + # K is chosen large enough that γ^K is negligible (e.g., γ=0.975 → K=400 → γ^K ≈ 5e-5). + # The first action is `a`; subsequent actions follow a heuristic policy: + # - if in-position-and-pnl > 2·entry_cost: close + # - if in-position-and-trail-breached: close + # - else: Hold + # This is the "frictionless trail-stop heuristic policy" π_oracle. + rollout_K = simulate_lobsim(s_t, a, π_oracle, K_steps=400) + Q_oracle(s_t, a) = Σ_k γ^k · r_shaped(rollout_K[k]) + a_argmax = argmax_a Q_oracle(s_t, a) +``` + +This is a *heuristic* oracle (not a fixed-point oracle). The justification is that under any reasonable trading regime: + +- The heuristic captures the dominant pnl driver (taking the trade and exiting on edge depletion). +- Heuristic-optimal Q values bound the random-policy Q values from above by a large margin. +- F4-MVP's job is to give Q a "good direction to start," not to compute the unreachable theoretical optimum. The RL fine-tune phase refines around the bootstrapped baseline. + +Two-step honest framing for the implementation plan: + +1. The oracle is NOT optimal in the game-theoretic sense; it is a *strong heuristic baseline* under perfect foresight. +2. F4-G1 (oracle-ground-truth gate, §5) tests whether the heuristic correlates with realized pnl on held-out states. If F4-G1 fails, the heuristic itself is broken (a tighter heuristic policy is required) — this is BEFORE we go anywhere near the trainer. + +### §2.5 Citation chain + +Oracle pretraining for offline-to-online RL is a well-established direction: + +- Levine et al. 2020 ("Offline RL: Tutorial, Review, and Perspectives," arXiv:2005.01643) — survey establishing the framework. +- Kumar et al. 2020 ("Conservative Q-Learning," arXiv:2006.04779, NeurIPS) — CQL pessimism for offline-to-online warm-start. +- Nair et al. 2020 ("AWAC: Accelerating Online RL with Offline Datasets," arXiv:2006.09359) — advantage-weighted regression bridging. +- Kostrikov et al. 2021 ("Offline RL with Implicit Q-Learning," arXiv:2110.06169) — expectile regression sidestepping explicit pessimism. + +F4-MVP is a simplified Nair-2020 (AWAC-style) warm-start: supervised regression on perfect-foresight return labels, then RL fine-tune. The trade-offs vs CQL pessimism / IQL expectile are out of scope for F4-MVP; revisit in F4-Full if MVP shows benefit. + +For the heuristic oracle specifically: closest precedent is Schaul et al. 2015 ("Prioritized Experience Replay," arXiv:1511.05952) which uses TD-error-rank as a heuristic priority; F4-MVP uses heuristic-realized-return as a label rather than as a sampling weight. The mechanism difference is the level of supervision (PER reweights existing TD targets; F4 supplies new MSE targets). + +--- + +## §3. What infrastructure is needed + +This is the load-bearing section. F4-MVP requires building four pieces of infrastructure foxhunt does not currently have. + +### §3.1 Oracle computation pipeline + +**Purpose:** Given an MBP-10 trading session trace, produce a per-state oracle Q-value `Q_oracle(s_t, a_argmax)` and identify `a_argmax`. + +**Where it runs.** Two design options: + +| Option | Runs in | Pros | Cons | +|---|---|---|---| +| §3.1.A — Python preprocess (offline) | Python script over MBP-10 fixtures; outputs binary | Fast iteration on oracle policy logic; no CUDA-kernel changes; reuses `crates/ml-alpha/cuda/...` reward kernel definitions via JSON-mirrored math | Risk of math drift between Python and CUDA reward kernel; requires regeneration on every reward-shape change | +| §3.1.B — Rust + CUDA on-the-fly | A new binary `bin/foxhunt-oracle-gen` invoking the same `rl_fused_reward_pipeline.cu` and LobSim CUDA kernels the trainer uses | No math-drift risk (uses exact trainer reward); reusable for ablation studies | Higher infra cost; longer iteration cycle; CUDA dev environment required | + +**Recommendation:** §3.1.B (Rust + CUDA on-the-fly). The math-drift risk is load-bearing: per `pearl_diag_pnl_fields_are_shaped_reward_not_usd`, the diag's `_usd` fields were misleading because the reward function was shaped differently than supposed. F4 oracle must be COMPUTED with the trainer's reward function, not a Python re-derivation, or F4 will fail F4-G1 ground-truth gate due to reward-function-identity bugs that are silent. + +**Cost analysis (back-of-envelope, must be verified in Phase 7c.0).** + +- 24 months × ES × MBP-10 events ≈ ~10⁸ states (per `pearl_mbp10_data_is_structurally_broken`, cluster PVC has 148GB coherent 24-month ES MBP-10; foxhunt's effective training-event throughput is ~10⁷-10⁸ per fold) +- F4-MVP needs only `(s_t, a_argmax, Q_argmax)` per state. Per state: ~16 bytes (state-id i64 + a_argmax i32 + Q_argmax f32). Per fold: ~1.6GB at 10⁸ states. +- However: producing `a_argmax` requires evaluating all 11 actions per state (per §2.4 algorithm), each requiring K=400-step LobSim rollout. At GPU-batched throughput ~10⁵ states/sec (extrapolated from existing LobSim benchmarks), full-fold oracle generation = 10⁸ / 10⁵ = ~10³ seconds = ~17 minutes per fold (highly speculative; Phase 7c.0 must measure). +- Per-action rollout is embarrassingly parallel across states and across actions. GPU implementation should batch them efficiently. + +**Storage volume.** F4-MVP per-fold ≈ 1.6GB. Acceptable on local 4GB RTX 3050 (oracle generation is one-shot, not held in GPU memory during the trainer step). PVC storage on cluster is ample (Scaleway Kapsule has terabytes available). + +**Frictionless vs realistic LobSim.** Two sub-options: + +| Sub-option | LobSim accounting | Pros | Cons | +|---|---|---|---| +| §3.1.B.1 — Frictionless oracle | Slippage = 0; fills at mid; commission = 0 | Simpler; faster compute; tighter upper bound on achievable pnl | Oracle Q values overestimate real-world achievable; risk of Q being miscalibrated when RL fine-tune introduces friction | +| §3.1.B.2 — Realistic oracle | Use the SAME LobSim friction model the trainer uses (slippage, commission, MAS) | Math-identity with trainer; F4-pretrained Q is correctly calibrated | Slower compute; oracle complexity bound by LobSim complexity | + +**Recommendation:** §3.1.B.2 (realistic oracle). Same reasoning as §3.1.B vs §3.1.A: math-identity with trainer is load-bearing. Better to spend the compute than to debug calibration mismatches later. + +**Action-state coupling (load-bearing).** Several of foxhunt's 11 actions depend on `pos_state`, `unit_state`, and `trail_distance`: + +- `ACTION_HOLD` is universally available but its effect depends on whether the trail is breached. +- `ACTION_TRAIL_LOOSEN` is a no-op at trail-max cap. +- `ACTION_PYRAMID_LONG/SHORT` is unavailable from flat (per F5 mask), and effect depends on `unit_state` (how many units are already open). +- `ACTION_FLAT_FROM_LONG/SHORT` is a no-op when flat. +- `ACTION_HALF_FLAT_LONG/SHORT` reduces position size by 50%. + +The oracle simulator must track `pos_state` and `unit_state` per (s_t, a) branch through the K-step rollout. This means each (state, action) branch is a separate stateful simulation. Per-state cost is therefore O(K × |A|) LobSim steps = 400 × 11 = ~4400 simulator transitions per state. Over 10⁸ states → 4.4 × 10¹¹ simulator transitions. This is large; Phase 7c.0 cost estimate may need to recalibrate by: + +- Pruning: pre-mask actions that are no-ops per state (saves ~50% of work for most flat states). +- Stride sampling: rather than oracle-label every state, label every Nth state (N=10-100) and let RL handle the rest. F4-MVP must specify N. + +**Time horizon truncation.** The oracle integral is over an infinite-horizon discounted return. Truncate at K such that `γ^K · max_possible_reward < ε`. For γ=0.975, K=400 → γ^K ≈ 5e-5 — adequate if max reward magnitude is bounded (it is, via PopArt and trail-max). Use K=400 unless Phase 7c.0 finds the truncation introduces meaningful bias. + +**Per-action MTM accounting.** The oracle MUST consume the same reward function as the trainer: + +``` +r_shaped(s, a, s_next) = + r_base(s, a, s_next) # base pnl-delta or popart-whitened + + RL_MTM_REWARD_WEIGHT_INDEX · MTM_delta(s, s_next) # Phase 5 MTM + + Phase_5_short_hold_penalty(s, a, s_next) # Phase 5 term 1 (non-potential) + + Phase_5_long_ride_bonus(s, a, s_next) # Phase 5 term 3 (non-potential) + + Phase_5_hold_bonus(s, a, s_next) # Phase 5 term 4 (gate-zeroed, almost-potential) + + Phase_5_entry_cost # Phase 5 term 2 (non-potential) + + Phase_7a_F2_hinge_signal # if F2 enabled — but F2 is the consumer not a producer +``` + +Per `pearl_phase5_term_4_is_almost_potential`, three of Phase 5's four terms are explicitly non-potential. F4 oracle MUST include them (otherwise Q is calibrated for a different reward function than the trainer uses → F4-G3 will fail). The oracle reward function is the SAME function the trainer reads from `rl_fused_reward_pipeline.cu`. + +**Versioning.** When the reward function changes (e.g., Phase 5 MTM weight tuned, new shaping term added), the oracle must be regenerated. Implementation plan must: + +- Compute a hash of the reward-function configuration (Phase 5 weights + Phase 7a F2 mode + γ + K). +- Embed the hash in the oracle file name. +- Trainer reads the hash and refuses to load oracle if hash mismatches current ISV state. + +### §3.2 Storage format and access path + +**Format options.** + +| Option | Format | Pros | Cons | +|---|---|---|---| +| §3.2.A — Predecoded binary (foxhunt convention) | `.predecoded.bin` with same MMap pattern as MBP-10 trace | Reuses existing infra (`crates/ml-alpha/src/data/loader.rs:85-98`); zero-copy I/O | Custom serialization needed | +| §3.2.B — Parquet | Columnar Arrow-compatible | Industry standard; introspectable | Adds Arrow dep to ml-alpha; per-row decode cost | +| §3.2.C — TFRecord / Numpy | Cross-tool compatibility | Easy to inspect from Python | Foreign to foxhunt's Rust toolchain | + +**Recommendation:** §3.2.A (predecoded binary). Reuses foxhunt's existing MMap convention. Naming pattern: + +``` +/ES.FUT_.dbn.zst.oracle_q_argmax_v.predecoded.bin +``` + +Where the file body is a sequence of `(state_global_id: u64, a_argmax: u8, q_argmax: f32)` records. Each record is 13 bytes; aligned to 16 for fast vectorized load. + +The trainer's existing data loader (`crates/ml-alpha/src/data/loader.rs`) already MMaps MBP-10 traces — extending it to MMap a parallel oracle file is straightforward. + +**Indexing.** The trainer references states by `(file_idx, byte_offset)` already (per existing MBP-10 loader). The oracle file uses the same indexing: each state in the MBP-10 trace has a corresponding entry in the oracle file at the same record offset. State-by-state alignment is therefore zero-cost (no hash table, no lookup, just record-index arithmetic). + +**Cluster delivery.** PVC-mounted, same as MBP-10 training data: + +``` +/mnt/training-data/futures-baseline/ES.FUT/ES.FUT_.dbn.zst.oracle_q_argmax_v.predecoded.bin +``` + +The Argo template `argo-alpha-rl.sh` would need to mount the new file (no template change required since PVC already exposes the directory). + +**Versioning protocol.** + +1. Reward-function hash computed at compile time from `crates/ml-alpha/src/rl/isv_slots.rs` Phase 5 + Phase 7a bootstrap values + γ. +2. Oracle generation binary embeds the hash in the filename. +3. Trainer at load time reads the hash and: (a) if it matches → load; (b) if it mismatches → log a warning AND refuse to load (force regeneration). +4. CI / cluster automation regenerates oracle when the hash changes (Argo workflow step). + +### §3.3 Supervised pretraining phase + +**Where in the trainer it runs.** Before the existing `step_with_lobsim_gpu` loop's first call: + +```rust +pub fn step_with_lobsim_gpu(...) -> Result<...> { + if self.f4_pretrain_enabled() && !self.f4_pretrain_done { + self.pretrain_from_oracle()?; // ← NEW + self.f4_pretrain_done = true; + } + // Existing logic from here. + ... +} +``` + +**New training loop body (F4-MVP).** + +```rust +fn pretrain_from_oracle(&mut self) -> Result<()> { + let n_batches = oracle_dataset.len() / BATCH_SIZE; // BATCH_SIZE = 128 local, 1024 cluster + for batch_idx in 0..n_batches { + // 1. Sample BATCH_SIZE (state_id, a_argmax, q_argmax) records. + let (state_batch, a_argmax_batch, q_argmax_batch) = oracle_dataset.sample(BATCH_SIZE, prng_seed); + + // 2. Run encoder forward on state_batch (existing forward kernel chain). + self.encoder_forward(&state_batch)?; + + // 3. Q head forward to produce Q_θ(s, a_argmax) — read only at a_argmax positions. + self.q_head_forward(&state_batch)?; + let q_pred = self.gather_q_at_action(&a_argmax_batch); // [B] — Q_θ(s, a_argmax) + + // 4. MSE loss: (q_pred - q_argmax_batch).pow(2).mean() + let loss = (q_pred - q_argmax_batch).pow(2).mean(); + + // 5. Backward + Adam through Q head AND encoder. + loss.backward(); + self.optimizer.step(); + + // 6. Diag emit: pretrain step + MSE + max |Δ| + self.emit_pretrain_diag(batch_idx, loss); + } + Ok(()) +} +``` + +**Encoder freeze (F4-MVP decision).** Both Q head AND encoder train during pretraining (encoder unfrozen). Rationale: foxhunt's encoder (mamba2-based) shares weight with the policy head; freezing the encoder would also freeze the policy's perceptual capacity. F4-Full may revisit encoder-only or Q-only freeze variants once F4-MVP results are in. + +**Number of pretrain epochs.** ONE epoch. Adaptive stopping (e.g., until MSE plateaus) is F4-Full scope. + +**Gradient flow specifics (load-bearing).** + +- The MSE loss writes to Q head weights (the same weights the Bellman target writes to during RL fine-tune). NO separate "pretrain head" — F4-MVP intentionally uses the production Q head. +- The encoder gradient flows from the Q head's L2 loss back through the trunk. This means encoder weights at the start of RL fine-tune are a *function of the oracle pretrain*. If the oracle is biased (e.g., heuristic policy is bad), the encoder is biased in that direction. +- Gradient clipping: use the existing trainer's per-batch gradient norm clip (slot TBD; verify in Phase 7c.0). +- Learning rate: use the existing Q-head Adam learning rate. NO new LR controller for pretrain (F4-MVP simplification). If MSE diverges, this is a STOP signal per F4-G2. + +**New ISV slots.** + +| Slot ID | Name | Bootstrap | Effect when bootstrap | +|---|---|---|---| +| 824 (next free) | `RL_F4_PRETRAIN_MODE_INDEX` | 0.0 (OFF) | 0=skip, 1=MVP one-epoch, 2=full multi-epoch (reserved) | +| 825 | `RL_F4_PRETRAIN_BATCH_COUNT_INDEX` | 0.0 (read-only counter) | Filled by trainer with `n_batches` at start of pretrain; diag exposes for verification | +| 826 | `RL_F4_PRETRAIN_FINAL_MSE_INDEX` | 0.0 (read-only) | Filled by trainer at end of pretrain; F4-G2 reads this | +| 827 | `RL_F4_PRETRAIN_DONE_INDEX` | 0.0 | Set to 1.0 once pretrain completes; trainer reads to gate "skip if already done" | + +Slot ID ranges are illustrative; Phase 7c.0 must verify current `RL_SLOTS_END` and allocate from there. + +**Determinism.** The pretrain phase must preserve `pearl_determinism_achieved`. Oracle reads use the established seeded-PRNG pattern (per the IQN race fix + OutcomeHead seed fix in commit `c75c45a93`). The pretrain epoch's sample order is deterministic for a given seed. + +### §3.4 Trainer integration points + +Per `pearl_foxhunt_trainer_is_genuinely_single_step`, the trainer is `[B]`-only. F4-MVP intentionally preserves this: + +- Pretrain samples are `[B]`-shaped (one batch = B independent states, one action each). +- No rollout buffer needed — the supervised loss is computed over individual transitions. +- The pretrain loop is a separate code path in `integrated.rs`; it does not interfere with the trainer's main step body. + +**New struct fields on `IntegratedTrainer`.** + +```rust +pub struct IntegratedTrainer { + // ... existing fields ... + + /// F4 oracle storage handle (memory-mapped predecoded binary). + pub oracle_storage: Option, + + /// F4 pretrain step counter (for diag emit). + pub f4_pretrain_step: u64, + + /// F4 pretrain done flag (idempotent skip if already done). + pub f4_pretrain_done: bool, +} +``` + +**New code paths.** + +1. `OracleStorage::load_from_predecoded(path) -> Self` — MMap and validate hash. +2. `IntegratedTrainer::pretrain_from_oracle(&mut self) -> Result<()>` — the supervised loop body. +3. `IntegratedTrainer::oracle_batch_sample(&mut self, batch_size, prng_seed) -> (StateIds, ActionIds, QValues)` — deterministic batch sampler. +4. `IntegratedTrainer::gather_q_at_action(actions: &[u8]) -> CudaSlice` — utility to gather Q_θ values at specific action indices (similar to the existing `gather` pattern in `rl_q_pi_distill_grad.cu`). + +**Determinism wiring.** + +- The oracle sample PRNG seeds from the existing trainer seed. +- The kernel reads from `oracle_q_d` (a new CudaSlice) and `oracle_a_argmax_d` (a new CudaSlice) are race-free (single thread per batch reads its row). +- The MSE gradient kernel uses the existing Adam optimizer step. + +**Code structure.** A new module `crates/ml-alpha/src/data/oracle_loader.rs` for the predecoded-binary MMap; a new module `crates/ml-alpha/src/trainer/f4_pretrain.rs` for the pretrain loop body. Both files are new but the surface area is small (~200-400 LOC each, mostly trivial). + +--- + +## §4. F4-MVP scope boundary + +The discipline boundary of F4-MVP is the single most important section of this spec for preventing scope creep into F4-Full. + +### §4.1 What F4-MVP IS + +- ONE supervised pretrain epoch (no curriculum, no annealing, no early-stopping on MSE). +- argmax-only oracle: one `(s_t, a_argmax, Q_argmax)` triple per state. +- Q-head + encoder both training during pretrain (no freeze). +- Single fold (fold 1) only. +- Pretrain writes to SAME Q head weights that Bellman target writes to during RL fine-tune. +- NO separate "pretrain head." +- NO distillation regularization to prevent post-pretrain Q drift. +- NO online oracle update during RL. + +### §4.2 What F4-MVP is NOT + +| Out-of-scope feature | Why excluded from MVP | When to revisit | +|---|---|---| +| Multi-epoch supervised | F4-Full; MVP tests whether one-epoch warm-start moves the needle | F4-MVP cluster passes F4-G4; user wants to explore further pretrain | +| Full-distribution oracle (all 11 actions) | F4-Full; 11× compute cost; MVP-only-argmax is the cheapest meaningful test | F4-MVP fails F4-G3 because non-argmax actions need supervision | +| Encoder freeze | F4-Full; risks overfitting trade-off | F4-MVP cluster shows encoder catastrophic-shift post-pretrain | +| Per-fold oracle generation (walk-forward) | F4-Full; MVP tests fold-1 only | F4-MVP fold-1 passes and we need to verify the win generalizes across folds | +| Action-curriculum (oracle gradually reveals actions) | F4-Full | F4-MVP fails F4-G6 because Q drifts too far from oracle during RL | +| Reward-shape-curriculum (oracle gradually reveals shaping) | F4-Full | Phase 5 shaping terms cause oracle to over-recommend wait | +| Adversarial / smarter-than-heuristic oracle | F4-Full | F4-G1 ground-truth gate suggests heuristic ceiling is too low | +| Behavioral cloning from human demonstrations | Out of scope entirely | No human demonstrations exist in foxhunt | +| Imitation learning from a learned-policy expert | Out of scope entirely | A learned expert policy must itself be trained — recursive | +| Online oracle update (Dyna-style) | F4-Full; major arch change | F4-MVP cluster passes and we want to tighten oracle as policy improves | +| Hindsight relabeling (HER) | Out of scope; orthogonal | Separate spec if pursued | +| F4 + F5 combined (Phase 7c.6 if needed) | Sequenced after F4-MVP-only result | F4-MVP fails F4-G7 outcome and Phase 7c.6 is invoked per §6 | + +--- + +## §5. Empirical falsification gates + +Three categories: **Pretrain-phase gates** (verify oracle pipeline is correct), **RL fine-tune phase gates** (verify F4 changes downstream training), **Cluster outcome gates** (verify F4 actually helps). + +### §5.1 Pretrain-phase gates — verify oracle pipeline correctness + +Run during Phase 7c.0 / Phase 7c.1 / Phase 7c.2 / Phase 7c.3 of the implementation plan. + +| Gate ID | Metric | Pass condition | When measured | What it falsifies | +|---|---|---|---|---| +| F4-G1 (oracle ground-truth) | Pearson correlation between `Q_oracle(s_t, a_argmax)` and the actually-realized discounted return when an agent forced to take `a_argmax` is run forward K=400 steps on held-out data | ≥ 0.7 | Phase 7c.0 (offline) | If fails: oracle policy heuristic is wrong (re-design heuristic or escalate to F4-Full smarter oracle) | +| F4-G2 (pretrain loss decreases) | Final MSE after 1 epoch / initial MSE at epoch start | ≤ 1e-2 (i.e., MSE drops ≥ 100×) | Phase 7c.2 (local Tier 1.5) | If fails: supervised loss path is broken (kernel bug, wrong gradient flow, oracle hash mismatch) | +| F4-G3 (Q-head outputs match oracle) | On held-out state subset: `argmax_a Q_θ(s_t, a) == a_argmax` agreement rate | ≥ 70% | Phase 7c.3 (local Tier 1.5) | If fails: encoder/Q-head capacity insufficient OR oracle is degenerate (most a_argmax are Hold) | + +**Honest caveat.** F4-G1's 0.7 Pearson threshold is calibrated against `pearl_reward_signal_anti_aligned_with_pnl` baseline (0.11-0.45 random-policy). 0.7 is a *substantial* improvement target; if F4-G1 produces 0.5-0.7, this may still be worthwhile but is a STOP-and-evaluate signal. If F4-G1 produces < 0.5, the oracle heuristic is broken and Phase 7c.1+ MUST NOT proceed (the pipeline would be wired to wrong labels). + +### §5.2 RL fine-tune phase gates — verify F4 changes downstream training + +Run during Phase 7c.4 of the implementation plan, with F4 pretrain ON and standard RL fine-tune proceeding. + +| Gate ID | Metric | Pass condition | When measured | What it falsifies | +|---|---|---|---|---| +| F4-G4 (RL improves vs cold-start) | Steps until `l_q` reaches the cold-start cluster `l_q` at step N | ≤ 0.3 × N | Phase 7c.4 (local Tier 1.5) | If fails: pretrained Q drifts away from oracle too fast during RL (need distillation regularization, F4-Full) | +| F4-G5 (early policy quality) | Action entropy at step 100 of RL | ≥ 1.5 nats | Phase 7c.4 | If fails: pretrained policy collapsed during pretrain (encoder overfit or oracle too argmax-degenerate) | +| F4-G6 (no surfer-attractor migration) | F2's `hinge_zero_rate` (existing diag from F2 commit) at step 1000 | ≤ 0.5 | Phase 7c.4 | If fails: F4's pretrained Q does not prevent F2's hinge from saturating (Q still has Hold > Open everywhere) | + +### §5.3 Cluster outcome gates — verify F4 actually helps + +Run during Phase 7c.5 of the implementation plan, at cluster Tier 2 (b=1024, 20k train + 5k eval, fold 1, ~70 min wallclock). + +| Gate ID | Metric | Pass condition | When measured | What it falsifies | +|---|---|---|---|---| +| F4-G7 (cluster eval pnl) | `eval_summary.total_pnl_usd` | ≥ F2+F5 (sk2z9 baseline) cluster eval pnl + $10M / session | Phase 7c.5 | If fails: F4-MVP does not solve the eval-fail problem; consider F4-Full or escalate to model-architecture | +| F4-G8 (no degenerate attractor) | Max single-action share at eval steps 5k and 15k | < 80% | Phase 7c.5 | If fails: F4 has migrated to a different degenerate (e.g., always-Open-Long); diagnose before committing | +| F4-G9 (train→eval transfer) | `eval_pnl / train_pnl` | ≥ 0.7 | Phase 7c.5 | If fails: F4 does not solve the train→eval generalization gap; problem is upstream of Q-init (encoder generalization, regime shift) | + +**F4-G9 is the most consequential gate.** Per `pearl_surfer_baseline_was_train_only_never_eval`, the train→eval gap has been the long-standing failure mode (eval ≈ 5% of train at b=128 currently). F4 explicitly aims to address this by giving Q a starting point that already encodes "what works on held-out data." If F4 still produces ~5% eval-to-train ratio, F4-MVP has not solved the problem and we have refuted the encoder-Q-bootstrap hypothesis. F4-G9 failure is a *progress signal* — it isolates the next investigation direction. + +### §5.4 Stop-on-fail rules + +- **F4-G1 fails (oracle ground-truth Pearson < 0.7):** STOP at Phase 7c.0; redesign oracle heuristic or escalate to F4-Full smarter oracle. DO NOT proceed to Phase 7c.1+. +- **F4-G2 fails (MSE does not decrease):** STOP at Phase 7c.2; the supervised loss path is broken (kernel bug, hash mismatch, or gradient-flow bug). Debug before continuing. +- **F4-G3 fails (Q-head does not match oracle on held-out):** STOP at Phase 7c.3; either encoder capacity insufficient, or oracle is degenerate (overwhelmingly Hold). Either escalate to F4-Full (full-distribution oracle) or accept and proceed to Phase 7c.4 with degraded expectation. +- **F4-G4, F4-G5, or F4-G6 fail at smoke:** Do not cluster. Diagnose with `pearl_fleet_fraction_not_aggregate`-discipline diag; consider F4-Full distillation regularization. +- **F4-G7 fails at cluster but G8 + G9 are healthy:** F4-MVP insufficient; consider Phase 7c.6 (F4 + F5) or escalate to F4-Full. +- **F4-G9 fails at cluster regardless of G7:** F4-MVP has refuted the encoder-Q-bootstrap hypothesis. STOP; escalate to model-architecture work (e.g., different encoder, attention modifications) or train→eval regime-shift work. + +### §5.5 The "going in circles" guard + +Per `feedback_going_in_circles_pattern` and `feedback_investigation_first_falsification_methodology`: F4 is the third independent intervention axis tested in Phase 7 (gradient axis = F2; choice-set axis = F5; initial-conditions axis = F4). If F4-MVP fails on F4-G7 AND F4-G9, the predetermined next action is NOT another Bellman-target reshape (per Phase 7 spec §5.4 → all the easy variants are now tested). The next direction is upstream: encoder, reward function, or train/eval regime structure. This guard prevents the empirical loop from spawning a Phase 8 / 9 / 10 series of more-Bellman-reshapes. + +--- + +## §6. Implementation phases + +Each phase has a predetermined gate and a STOP-if-fail rule. Phases run sequentially. + +### Phase 7c.0 — Oracle pipeline (build + verify) — ~2-3 days + +Build the offline oracle generator. Verify ground-truth correlation. + +- Build `bin/foxhunt-oracle-gen` (a new binary). +- Implement realistic LobSim rollout for each (s_t, a) over K=400 steps. +- Implement heuristic-policy continuation per §2.4. +- Output `(state_global_id, a_argmax, q_argmax)` triples in predecoded.bin format. +- Test against hand-computed ground truth on 10 sample states (verify by inspection). +- Generate oracle for fold-1 training data (one quarter of ES MBP-10). +- Compute F4-G1: Pearson correlation between oracle `Q_argmax` and forced-rollout realized return on held-out fold-1-eval states. + +**Gates:** F4-G1 only. + +**STOP if:** F4-G1 < 0.7. Redesign heuristic or accept F4-Full as required prerequisite. + +### Phase 7c.1 — Storage + access path (build + verify) — ~1-2 days + +Wire oracle reads into the trainer. + +- Add `crates/ml-alpha/src/data/oracle_loader.rs` (MMap predecoded.bin). +- Add hash validation + version check. +- Add `OracleStorage::sample(batch_size, prng_seed)` with bit-deterministic behavior. +- Write a Rust unit test that two same-seed sample calls produce identical (state_id, a, q) batches. +- Add minimal Tier 1 correctness test (load + sample + validate hash). + +**Gates:** Determinism unit test only (no behavioral gate yet). + +**STOP if:** Determinism test fails. Per `pearl_determinism_achieved`, this MUST hold. + +### Phase 7c.2 — Pretraining MSE loss (build + verify) — ~2-3 days + +Add `pretrain_from_oracle()` method. + +- Add new ISV slots 824-827 per §3.3. +- Implement `IntegratedTrainer::pretrain_from_oracle()` body (MSE loss, backprop, Adam step). +- Default `RL_F4_PRETRAIN_MODE_INDEX = 0.0` (OFF) — pretrain skipped by default. +- Add diag emit for `pretrain_mse_initial`, `pretrain_mse_final`, `pretrain_steps_completed`. +- Run local Tier 1.5 smoke with `RL_F4_PRETRAIN_MODE_INDEX = 1.0` and verify MSE decreases (F4-G2). + +**Gates:** F4-G2 (MSE decreases ≥ 100×). + +**STOP if:** F4-G2 fails. Debug gradient flow / hash mismatch. + +### Phase 7c.3 — Verify pretrained Q matches oracle — ~1 day + +Confirm F4-G3 on held-out subset. + +- Add a Rust integration test that runs the full pretrain on local fold-1 train fixtures, then evaluates `argmax_a Q_θ` vs `a_argmax` on held-out states. +- Emit per-state agreement rate. +- Report aggregate F4-G3 metric. + +**Gates:** F4-G3 (≥ 70% argmax agreement). + +**STOP if:** F4-G3 < 50%. Likely either degenerate oracle (mostly Hold) or insufficient encoder capacity; consider F4-Full full-distribution oracle before proceeding. + +### Phase 7c.4 — RL fine-tune from pretrained Q (smoke + verify) — ~1 day + +Run Tier 1.5 smoke with full F4 pipeline. + +- Configure: F2 ON, F5 OFF (clean F4 isolation), F4 pretrain ON, Band ON (for surfer principle preservation). +- Run mid-smoke (~10 min wallclock). +- Verify F4-G4, F4-G5, F4-G6 against post-pretrain diag fields. +- Run `determinism-check.sh --quick` to verify pretrain phase preserves bit-equality. + +**Gates:** F4-G4, F4-G5, F4-G6, determinism preserved. + +**STOP if:** Any RL fine-tune gate fails. Diagnose before cluster. + +### Phase 7c.5 — Cluster verify — ~80 min wallclock + analysis time + +Submit Argo run. + +- Configure: single-head, Band ON, F2 ON, F5 OFF, F4 ON. +- Submit cluster Tier 2 (b=1024, 20k train + 5k eval, fold 1). +- Wait ~70 min for completion. +- Analyze `eval_summary.json` for F4-G7, F4-G8, F4-G9. + +**Gates:** F4-G7 (eval pnl improvement), F4-G8 (no new attractor), F4-G9 (train→eval transfer). + +**STOP if:** F4-G7 fails AND F4-G9 fails → escalate per §5.4. F4-G7 alone fails AND F4-G9 passes → consider Phase 7c.6 (F4 + F5 combined). + +### Phase 7c.6 (CONDITIONAL) — Re-engage F5 with F4 pretrain — ~80 min cluster + +ONLY if Phase 7c.5 shows F4 alone insufficient on F4-G7 BUT F4-G9 healthy (train→eval transfer improved). + +- Configure: single-head, Band ON, F2 ON, F5 ON, F4 ON. +- Submit cluster Tier 2. +- Compare F4 + F5 vs F4-alone vs sk2z9 (F2 + F5 + OptA without F4). + +**Gates:** F4-G7 vs F4-alone (must show marginal improvement from F5). + +**STOP if:** Phase 7c.6 fails. F4-Full or model-architecture work is the next direction. + +### Total time estimate + +| Phase | Duration | Cumulative | +|---|---|---| +| 7c.0 | 2-3 days | 2-3 days | +| 7c.1 | 1-2 days | 3-5 days | +| 7c.2 | 2-3 days | 5-8 days | +| 7c.3 | 1 day | 6-9 days | +| 7c.4 | 1 day | 7-10 days | +| 7c.5 | 1-2 days (incl. analysis) | 8-12 days | +| 7c.6 (conditional) | 1-2 days | 9-14 days | + +**Honest range: 7-14 working days for F4-MVP cluster verdict.** Faster if every phase passes first attempt; slower if Phase 7c.0 oracle heuristic needs iteration (likely) or Phase 7c.5 cluster surfaces new pathology (also likely given Phase 7 history). + +--- + +## §7. Surprises, risks, and out-of-scope + +### §7.1 Honest risks + +1. **Oracle ground-truth requires non-trivial LobSim simulation.** §2.4's heuristic policy is itself a hypothesis. The Phase 7c.0 ground-truth gate (F4-G1) is non-trivial to pass. If heuristic-realized-return correlates poorly with achievable return, the oracle is broken before we get to the trainer. Mitigation: spend the time on the heuristic; consider hybrid heuristic + RL-trained-on-small-fold for the oracle policy. + +2. **Q drift during RL fine-tune may undo the warm-start.** Without distillation regularization (KL between current Q and pretrained Q, à la AWAC), the RL Bellman target may move Q rapidly away from oracle. F4-MVP intentionally excludes this regularizer for scope reasons, but F4-G4 (RL improvement rate) and F4-G6 (hinge_zero_rate) will detect rapid drift. If drift is severe, F4-Full distillation is required. + +3. **Encoder may overfit during pretraining.** One epoch with no dropout / weight decay could push the encoder to memorize the oracle's training-fold features. Mitigation: F4-G5 (action entropy at step 100 of RL) detects overfit-collapse. If encoder overfit is detected, F4-Full encoder-freeze variant is needed. + +4. **F4 does not solve regime shift.** If the test data has different action-quality structure than the train data, F4 trained on train will give a Q that is correctly bootstrapped for train but wrong for eval. This is a fundamental limitation. F4-G9 (train→eval transfer ratio) measures the impact. If F4-G9 fails, the problem is regime-shift, not Q-init — F4 does not solve it. + +5. **Reward function identity is load-bearing.** Per `pearl_diag_pnl_fields_are_shaped_reward_not_usd`, the trainer's `r_shaped` is non-trivial (Phase 5 MTM + entry cost + short-hold penalty + long-ride bonus). The oracle MUST use the EXACT SAME function. Re-derivation in Python (§3.1.A rejected option) was rejected precisely because of this. If §3.1.B math-identity is not maintained, F4 will appear to fail for the wrong reasons. + +6. **Determinism requirement is non-trivial.** Per `pearl_determinism_achieved`, the bit-equality discipline took ~170 LOC of fixes across multiple subsystems. The pretrain phase introduces new kernel paths (gather Q at action, MSE loss, etc.) — each must be carefully implemented to preserve same-seed bit-equality. F4-MVP's Phase 7c.4 includes a `determinism-check.sh` gate; do not skip. + +7. **Cluster vs local divergence.** Per `pearl_cluster_log_is_ground_truth`, local b=128 PPO diverges even when cluster b=1024 trains stable. F4-G4/G5/G6 are local-Tier-1.5 gates; F4-G7/G8/G9 are cluster gates. They MAY produce different verdicts. The discipline is: do not skip local gates (catch obvious bugs early), but do not over-weight local results (cluster is ground truth). + +### §7.2 Out of scope (explicit) + +- F4-Full (multi-epoch, full-distribution oracle) — separate spec eventually. +- Encoder-only pretraining (Q frozen, encoder trained on auxiliary task) — separate spec. +- Adversarial / RL-trained oracle (smarter than argmax heuristic) — separate spec. +- Online oracle update during RL (Dyna-style) — separate spec, major arch change. +- Hindsight relabeling (HER) — orthogonal direction. +- F4 + F5 combined (Phase 7c.6 if needed) — conditional within this spec. +- Multi-fold walk-forward oracle generation — F4-Full. +- Distillation regularization to prevent Q drift — F4-Full. +- Confidence-gate position-aware wiring (`pearl_confidence_gate_pos_state_unwired`) — pre-existing, orthogonal. +- GAE / rollout buffer — orthogonal; per `pearl_foxhunt_trainer_is_genuinely_single_step` not blocked by F4. + +--- + +## §8. Decision-relevant alternatives surveyed + +Briefly note alternatives ruled out at this stage. Full analysis is in the predecessor spec §3. + +- **Architectural changes (e.g., transformer policy, attention modifications).** Out of scope for this dispatch; would require new encoder + retrain everything. Becomes the next direction if F4 fails F4-G9. +- **Reward shaping F1 (per-step inaction cost).** Per Phase 7 spec §3.1: non-potential per Ng-Harada-Russell 1999, violates surfer principle, criterion-7 score 2. F4 has criterion-7 score 5 (preserves surfer by encoding patient behavior in oracle Q labels). +- **F3 action-space reduction.** Per Phase 7 spec §3.3: criterion-7 score 1 (total surfer violation), removes documented architectural progress, not reversible. Rejected. +- **Behavioral cloning from human expert.** No human expert demonstrations exist for foxhunt; out of scope. +- **Imitation learning from a heuristic policy.** A heuristic policy would itself need to be designed; oracle is a stronger signal (perfect-foresight return vs heuristic action sequence). + +**Explicit statement:** F4-MVP is the most disciplined next step because: + +(a) F2+F5 mechanism gates pass but outcome gates fail at cluster (per `sk2z9` empirical data + spec §5.3 attribution outcome 3). + +(b) F4 has the strongest theoretical foundation among unimplemented options (Levine 2020 / Nair 2020 / Kumar 2020 / Kostrikov 2021 — offline RL is a mature direction). + +(c) F4-MVP scope is bounded to ~7-14 days with predetermined gates at each phase. + +(d) The methodology `feedback_investigation_first_falsification_methodology` requires spec → plan → implement (not direct dive). This is the spec. + +(e) F4 attacks a distinct mechanistic axis (initial conditions) from F2 (gradient) and F5 (choice-set). The Phase 7 axis-completion logic is preserved. + +--- + +## §9. Open questions for the implementer + +The implementation plan that follows this spec must resolve at least these questions: + +1. **Trade-boundary marking in MBP-10.** Where in MBP-10 data layout do we identify trade boundaries for oracle return computation? The existing predecoded format includes mid-price events — does the oracle use mid-prices or top-of-book bid/ask for fill simulation? Verify in Phase 7c.0. + +2. **Reward function versioning protocol.** Exact hash function for reward-function-state. Include: Phase 5 weight slots (MTM, entry cost, etc.), Phase 7a F2 slots, Phase 7b F5 slots, γ. Spec the hash format in the plan. + +3. **Slippage / market impact accounting.** Should the oracle account for slippage? Phase 7 spec §3.5 LobSim trail-stop kernel includes a slippage model. Use the same? Or assume frictionless? Recommendation: §3.1.B.2 (realistic). Confirm in Phase 7c.0. + +4. **Per-state action-availability tracking.** How is oracle action-availability tracked through the K-step rollout? Specifically, `unit_state` (current pyramid layer count) affects PYRAMID action effects. Spec the per-(s_t, a) state propagation. + +5. **Oracle for train fold only or train+eval folds?** F4-MVP says train only (no oracle for eval). But F4-G3 (held-out match rate) requires SOME eval-state oracle to compute. Resolve: use a strict train-fold sub-split for in-pretrain held-out (e.g., last 10% of fold-1-train), OR generate oracle for fold-1-eval purely for the gate (and discard for training). Recommendation: latter — gate-only oracle on eval is acceptable. + +6. **ISV slot range.** Per current state (`grep RL_SLOTS_END` shows 824), allocate 824-827 (or shift if collisions). Confirm in Phase 7c.2 step 1. + +7. **Encoder LR for pretraining.** Use same LR as RL fine-tune? Or a separate (typically larger) supervised LR? F4-MVP simplification says: same LR. If F4-G2 shows MSE plateau too high, revisit. + +8. **Diag emission discipline.** Per `pearl_fleet_fraction_not_aggregate`, F4 pretrain diag emissions MUST include per-batch fractions (e.g., "fraction of pretrain batches where MSE > threshold") not just aggregate `worst_X`. Specify diag schema in Phase 7c.2. + +9. **Batch size for pretrain.** Use same B as RL (128 local / 1024 cluster)? Or different? F4-MVP simplification: same B. Recommend confirming compute headroom in Phase 7c.0. + +10. **Resumption semantics.** What happens if pretrain is interrupted (kernel crash, OOM)? F4-MVP simplification: ALL or NOTHING — pretrain restarts from scratch on resume. `RL_F4_PRETRAIN_DONE_INDEX` is the idempotent gate. F4-Full would add checkpoint/resume. + +--- + +## §10. Document discipline notes + +- **No implementation code in this spec.** Pseudocode in §3.3 is design clarification, not implementation prescription. +- **No new files outside this one spec.** The §3.1 / §3.4 references to `bin/foxhunt-oracle-gen` and `crates/ml-alpha/src/data/oracle_loader.rs` are forward references for the implementation plan, not files this spec creates. +- **Honest scope.** F4 is multi-week even at MVP scope. The 7-14 day range in §6 is honest about variance. Do not commit to F4-MVP if there is < 2 weeks of focused capacity available; better to defer. +- **Theoretical claims cited.** Every claim in §2.5, §7, §8 has a citation to predecessor spec, pearl, or external paper. +- **Predetermined falsification at every phase.** §5.4 and §6's "STOP if" rules are not optional; they are the methodology's load-bearing discipline per `feedback_investigation_first_falsification_methodology`. +- **No dispatch of further agents from this spec.** The output is the spec file. The implementation plan is a separate dispatch. + +--- + +## §11. Decision log entry + +If F4-MVP is approved for implementation, log the decision per `feedback_v7_gem_methodology` §decision-log: + +``` +Decision: Adopt F4-MVP (oracle pretraining with argmax-only realistic-LobSim heuristic oracle) as Phase 7c, sequenced after Phase 7a (F2 committed) and Phase 7b (F5 committed). F4-MVP is the disciplined response per Phase 7 spec §5.3 attribution outcome 3 + §5.4 escalation path when F2 + F5 mechanism gates pass but cluster outcome regresses. +Rejected: F4-Full (multi-epoch, full-distribution, encoder-freeze variants — deferred until F4-MVP cluster verdict). +Rationale: F2 + F5 + OptA (sk2z9) demonstrated that gradient-axis and choice-set-axis interventions are mechanically operational but cannot bootstrap Q-quality estimates from cold-start random-policy data. F4 attacks the distinct initial-conditions axis: warm-start Q with perfect-foresight realized-return labels BEFORE RL fine-tune begins. This addresses the "encoder cannot bootstrap" hypothesis directly. + +F4 is theoretically grounded in Levine 2020 (Offline RL Survey), Nair 2020 (AWAC), Kumar 2020 (CQL), Kostrikov 2021 (IQL). F4-MVP is a simplified Nair-2020-style warm-start. F4-MVP scope is bounded by §4 (one epoch, argmax-only oracle, fold-1 only, no distillation regularizer, no encoder freeze). + +F4-MVP is implementable with bounded scope (~7-14 days) across 6 sequential phases (7c.0 — 7c.5) with predetermined gates F4-G1 through F4-G9, three of which (F4-G1, F4-G2, F4-G3) gate progression of the pipeline itself. Phase 7c.6 is conditional re-engagement of F5. + +Date: 2026-06-05 +Falsification trigger (oracle pipeline): F4-G1 per §5.1 +Falsification trigger (RL fine-tune): F4-G4 + F4-G5 + F4-G6 per §5.2 +Falsification trigger (outcome): F4-G7 + F4-G9 per §5.3 +Going-in-circles trigger: §5.4 F4-G9 failure → STOP, no further Bellman-target reshape +Pre-registered failure prediction: §5.4 "F4-G9 failure refutes encoder-Q-bootstrap hypothesis" +``` diff --git a/docs/superpowers/specs/2026-06-05-reward-pnl-alignment-restoration.md b/docs/superpowers/specs/2026-06-05-reward-pnl-alignment-restoration.md new file mode 100644 index 000000000..ab2eb3b52 --- /dev/null +++ b/docs/superpowers/specs/2026-06-05-reward-pnl-alignment-restoration.md @@ -0,0 +1,158 @@ +# Reward–PnL Alignment Restoration Spec + +> **For agentic workers:** This spec defines WHY and WHAT for restoring reward↔pnl alignment as the next intervention, displacing F4 oracle pretraining. It does NOT itself implement the fix; an implementation plan follows after approval. + +**Status:** Draft 1 (2026-06-05) +**Displaces (sequencing):** `docs/superpowers/specs/2026-06-05-f4-oracle-pretraining-infrastructure.md` — F4 is GATED behind this spec per the foundation-reshape spec's own §5.4 fork (alignment-branch must be ruled out before encoder-branch). +**Predecessors:** `docs/superpowers/specs/2026-06-01-reward-policy-alignment-investigation.md` (diagnosis), `docs/superpowers/specs/2026-06-02-trainer-rollout-buffer-gae.md` (the rollout/GAE scaffold this spec completes), `docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md` (§5.4 escalation fork). +**Branch target:** off `7376b1c67` on `ml-alpha-regime-observer` (or a fresh branch once approved). +**Estimated wall-clock:** Phase A+B (force-pin + cheap diagnostic gate) ~1-2 days; Phase C (1B-C credit-assignment completion) ~3-6 days IF Phase B passes. Honest range 1-2 days (if Phase B STOPs) to ~8 days (full build + cluster). Less than F4's 7-14 days, and on the **confirmed** lever. + +**Linked pearls (load-bearing):** +- `pearl_reward_misalign_blocks_f4_slot753_override` — the measurement + mechanism this spec acts on +- `pearl_reward_signal_anti_aligned_with_pnl` — the original diagnosis (4 ADDENDA) +- `pearl_advantage_kernel_is_done_gated_td_not_gae` — why aligned-but-sparse reward starves at T_eff=1 +- `pearl_foxhunt_trainer_is_genuinely_single_step` — why the credit-assignment fix is real work, not a flag flip +- `pearl_phase5_term_4_is_almost_potential` — the non-potential shaping terms slot 753 re-enables +- `pearl_diag_pnl_fields_are_shaped_reward_not_usd` — `realized_pnl_usd_delta` is the authoritative USD signal the gates use +- `pearl_pure_pnl_mode_starves_b16_controllers` — pure-pnl at b<1024 starves controllers (use b=1024 for reward-distribution changes) +- `feedback_investigation_first_falsification_methodology` — the discipline; cheapest-informative-step-first + +--- + +## §1. Why this, why now + +### §1.1 Three measured findings (2026-06-05) + +1. **Reward is misaligned, measured.** `Pearson(rewards.sum, trading.realized_pnl_usd_delta) = 0.28` on the mid-smoke diag (`/tmp/foxhunt-mid-smoke/diag.jsonl`, b=128 seed=42, 2000 rows). Gate for trustworthy Q learning is ≥ 0.70. Sign-agreement = 0.655 (improved from the original 0.23-0.32; the dominant failure has shifted from sign-flips to **magnitude decoupling** — among worst-5% pnl steps losing $42k-$155k, reward range was [-1509, +591], i.e. some catastrophic-loss steps received positive reward). + +2. **The "pure-pnl" bootstrap is cosmetic — verified against live code.** `integrated.rs:4293` sets `RL_SURFER_SCAFFOLD_WEIGHT_INDEX` (753) bootstrap = 0.0, but `rl_surfer_scaffold_controller.cu:101` writes `isv[753] = w` **unconditionally** with **no force-pin gate** (grep-confirmed absent). Runtime diag shows `surfer_scaffold_weight` pinned at **0.966 → 0.974** across all 2000 rows. All four Phase-5 non-potential shaping terms (`rl_fused_reward_pipeline.cu:344-373`, each gated `w*…`) are therefore live at ~97% strength regardless of the bootstrap. + +3. **Falsification: the value function is NOT the fault — the reward target is.** value-error-vs-policy-error, eval/train median ratios: `loss.v` = 0.91×, `loss.q` = 0.83×, `loss.pi` = 1.05×. The value function generalizes fine (eval error ≈ train, even lower), the policy agrees with Q (`q_pi_agree_ema=0.95`, `q_value_abs_max=4.92` by end of train) — yet eval pnl = -$4.96M, wr 0.317, PF 0.64. *Low value error + high Q-π agreement + negative pnl on both folds* ⇒ the agent **faithfully optimizes a misaligned objective**. This simultaneously falsifies (a) F4 (Q-init is not the fault; Q learns its target fine) and (b) generalization machinery / IDAAC decoupling (value does not overfit). It confirms the **reward target** is the lever. + +### §1.2 Why not F4 + +Per `pearl_reward_misalign_blocks_f4_slot753_override` and four independent omnisearch research agents (2026-06-05): F4 would label its oracle Q against this same `r_shaped` (math-identity is load-bearing in the F4 spec §3.1.B), warm-starting Q onto a 0.28-aligned target. F4-G1 cannot detect this (both sides of its Pearson are in shaped units — tautology). External literature is decisive: value-pretraining is an optimization lever, not a generalization lever (IDAAC, Raileanu & Fergus 2021); one-epoch-MSE-then-finetune triggers the WSRL (ICLR 2025) recalibration collapse; argmax-on-realized-path is look-ahead leakage. **Fix the target before warm-starting onto it.** + +### §1.3 The Phase 1A landmine (why this is not a 10-LOC fix) + +Force-pinning slot 753 to 0 gives reward ≈ pnl-delta → Pearson → ~1.0 **trivially** (the metric becomes near-tautological). But pure-pnl ALONE was already tried (Phase 1A, `pearl_reward_signal_anti_aligned_with_pnl` ADDENDUM 2026-06-02d) and **regressed eval −15.5%**, because aligned reward is *sparse* (pnl realizes at close events only) and the advantage kernel is done-gated 1-step TD (`compute_advantage_return.cu:43`, `advantages[b] = is_done ? (ret-vt) : 0`, T_eff=1) — only ~6.75 non-zero advantages per b=128 step. Aligned-but-sparse reward without multi-step credit assignment starves the gradient. **Alignment and credit-assignment are co-required.** This spec sequences a cheap diagnostic gate (Phase B) to confirm the co-fix will work *before* committing to the credit-assignment build (Phase C). + +--- + +## §2. The fault mechanism + +``` +slot 753 bootstrap = 0.0 (cosmetic) + ↓ rl_surfer_scaffold_controller.cu:101 writes isv[753] = w UNCONDITIONALLY (no pin gate) + ↓ w = max(1 − competence, w_decay); cold-start n_trades=0 → competence≈0 → w≈0.97 every step + ↓ rl_fused_reward_pipeline.cu:344-373: 4 non-potential Phase-5 shaping terms live at 0.97 + ↓ r_shaped only 0.28-Pearson-aligned with realized USD pnl (magnitude decoupled) + ↓ V regresses r_shaped (loss.v fits fine on train AND eval); Q distills from V/Bellman + ↓ π distills from Q (q_pi_agree 0.95) → π faithfully optimizes the misaligned objective + ↓ eval pnl negative DESPITE low value error and high Q-π agreement +``` + +The two independent defects: +- **D1 (alignment):** the controller silently re-enables anti-aligned shaping; the 0.0 bootstrap never takes effect at runtime. +- **D2 (credit assignment):** even with aligned (sparse) reward, the done-gated 1-step TD advantage kernel cannot propagate close-event pnl back to the opening decision. The GAE rollout scaffold exists (`rollout_buffer.rs`, `rollout.rs`, `rollout_collection.rs`, GAE kernel, commits `5cd2f8703`+`779c03b9d`) but is **collection-only** (Phase 1B-B): training still runs per-step TD; the multi-epoch PPO update over the buffer (Phase 1B-C) was deferred per STOP-on-unexpected-finding. + +--- + +## §3. What to build + +### §3.1 Force-pin for slot 753 (D1 fix) — bounded + +A pin gate so the bootstrap value is honored at runtime. Two design options (resolve in the plan): + +| Option | Mechanism | Pros | Cons | +|---|---|---|---| +| §3.1.A New ISV pin slot | `RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX` (next free slot); controller reads it and `if (pin > 0.5) return;` before writing `isv[753]` | ISV-native; A/B-toggleable; determinism-preserving | one new slot | +| §3.1.B Reuse bootstrap-as-pin | controller skips the write when an existing "design constant locked" flag is set | no new slot | conflates two concerns; less explicit | + +**Recommendation:** §3.1.A. Mirror the F2/F5 slot-allocation pattern (`isv_slots.rs`, bootstrap table `integrated.rs:4382-4417`). Bootstrap pin = 0.0 (OFF → bit-equal to HEAD); set to 1.0 only in the alignment experiment config. The pinned *value* of slot 753 is then whatever its bootstrap says (0.0 = pure-pnl, or a chosen partial-shaping weight for ablation). + +### §3.2 Cheap diagnostic gate (decides whether D2 build is justified) — bounded + +With slot 753 pinned (aligned reward) AND `FOXHUNT_USE_ROLLOUT=1` (GAE computed as side-effect, training still TD), add a diagnostic that measures **`Pearson(GAE_advantage, realized_pnl_at_close)`** in the rollout buffer. This answers: *does multi-step GAE credit assignment on the aligned reward produce advantages that align with realized pnl?* — WITHOUT the expensive 1B-C training rearchitecture. This is the load-bearing investigation-first gate. + +### §3.3 Complete Phase 1B-C (D2 fix) — substantive, CONDITIONAL on §3.2 passing + +Multi-epoch PPO update over the GAE rollout buffer, replacing the per-step done-gated TD update path. Per `rollout_collection.rs:24-37` this is where per-step training is split. Scope this in the plan against the existing scaffold; do NOT rebuild the buffer/GAE/collection (already done in 1B-B). + +### §3.4 Re-measure harness + +A reusable script (`scripts/measure_reward_alignment.py`, NOT in root) that computes `Pearson(rewards.sum, realized_pnl_usd_delta)` and sign-agreement from any diag.jsonl, so every smoke reports the alignment number as a first-class gate. + +--- + +## §4. Scope boundary + +### §4.1 In scope (MVP) +- Force-pin slot 753 (D1). +- Diagnostic: GAE-advantage↔pnl Pearson (the gate). +- IF gate passes: complete 1B-C multi-epoch PPO over rollout buffer (D2). +- Re-measure harness + falsification gates at smoke and cluster. + +### §4.2 Out of scope +- F4 oracle pretraining (gated behind this spec; revisit only if aligned reward + GAE still fails eval). +- Magnitude-clamp / popart redesign (the secondary finding; separate spec if Pearson stalls after D1+D2). +- New reward shaping terms (this spec REMOVES misaligned shaping, adds none). +- Generalization machinery (IDAAC decoupling, ensembles, augmentation) — only if D1+D2 succeed and a *residual* train→eval gap remains. +- A reward redesign beyond pinning existing weights (e.g. potential-based reshaping) — F4-Full-style; separate spec. + +--- + +## §5. Falsification gates + +| Gate | Metric | Pass | When | Falsifies | +|---|---|---|---|---| +| RA-G1 (pin works) | runtime `surfer_scaffold_weight` with pin=1.0, bootstrap=0.0 | == 0.0 across all rows | Phase A smoke | If fails: pin gate not wired to the controller write | +| RA-G2 (determinism) | `determinism-check.sh --quick` seeds 42/43/44 | exit 0, eval_summary bit-equal | Phase A | If fails: pin/diag introduced nondeterminism | +| RA-G3 (alignment restored) | `Pearson(rewards.sum, realized_pnl_usd_delta)` with pin=1.0 bootstrap=0.0 | ≥ 0.70 | Phase A smoke | (near-tautological with pure-pnl; confirms the pin removed the shaping) | +| **RA-G4 (credit-assignment viability — the load-bearing gate)** | `Pearson(GAE_advantage, realized_pnl_at_close)` in rollout buffer, aligned reward + USE_ROLLOUT=1 | ≥ 0.70 | Phase B smoke | **If fails: GAE on aligned reward still does not align advantages with pnl → STOP; the fault is deeper than credit assignment (reward shape itself / regime). Do NOT build 1B-C.** | +| RA-G5 (no eval regression) | smoke eval pnl with D1+D2 | ≥ F2+F5 SOTA −$415,237 (not worse than current) | Phase C smoke | If fails: D2 build did not deliver; diagnose before cluster (recall Phase 1A regressed here) | +| RA-G6 (cluster eval) | `eval_summary.total_pnl_usd` at Tier 2 b=1024 20k+5k fold 1 | > F2+F5 sk2z9 cluster baseline | Phase D cluster | If fails AND RA-G4 passed: aligned reward+GAE insufficient; THEN encoder/generalization or F4 is the next fork | + +**Stop-on-fail discipline:** +- RA-G4 is the spec's economic linchpin: it costs ~1 day (no training rearchitecture) and decides whether the ~3-6 day Phase C build is justified. **If RA-G4 < 0.70, STOP — do not build 1B-C.** A failure here means even perfect multi-step credit assignment on the aligned reward does not recover pnl-aligned advantages, which would point at deeper reward-shape or regime-shift causes (and would, notably, also predict F4 failure — so it is decisive either way). +- RA-G6 failure with RA-G4 pass cleanly isolates the residual to encoder/generalization or genuine OOS regime shift — the foundation-reshape §5.4 "encoder branch," now legitimately reached. + +--- + +## §6. Implementation phases + +### Phase A — Force-pin slot 753 (~0.5-1 day) +Add pin slot + controller gate + bootstrap-table entry. Smoke with pin=1.0 bootstrap=0.0. **Gates:** RA-G1, RA-G2, RA-G3. **STOP if** RA-G1/G2 fail. + +### Phase B — Credit-assignment viability diagnostic (~0.5-1 day) +Add `Pearson(GAE_advantage, realized_pnl_at_close)` diag to the rollout-buffer path. Smoke with pin=1.0 + `FOXHUNT_USE_ROLLOUT=1` (training unchanged). **Gate:** RA-G4. **STOP if** RA-G4 < 0.70 — escalate to reward-shape/regime investigation, NOT 1B-C, NOT F4. + +### Phase C — Complete 1B-C multi-epoch PPO over rollout buffer (~3-6 days) — CONDITIONAL on RA-G4 +Replace per-step done-gated TD update with multi-epoch PPO over the GAE buffer. Mid-smoke. **Gates:** RA-G5, RA-G2 (determinism preserved). **STOP if** RA-G5 regresses. + +### Phase D — Cluster verify (~80 min + analysis) +Tier 2 b=1024, 20k+5k, fold 1, pin=1.0 + USE_ROLLOUT=1 + 1B-C. **Gate:** RA-G6. Branch per §5. + +--- + +## §7. Risks + +1. **RA-G3 is near-tautological** — pinning to pure-pnl makes reward≈pnl by construction. That is why RA-G4 (advantage↔pnl, NOT reward↔pnl) is the real gate. Do not over-read RA-G3. +2. **Phase 1A precedent** — pure-pnl regressed eval −15.5%. This spec's whole bet is that the regression was a credit-assignment (D2) artifact, not an alignment (D1) artifact. RA-G4 tests that bet cheaply before the Phase C spend. If RA-G4 fails, the bet is wrong and we stop early — cheaply. +3. **Magnitude decoupling may persist after D1** — if removing shaping still leaves Pearson < 0.70 because of clamp/popart magnitude crushing (`pearl_popart_blind_to_session_signals`), a clamp/popart fix becomes the next sub-spec. RA-G3 catches this. +4. **1B-C cost is the dominant uncertainty** — `pearl_foxhunt_trainer_is_genuinely_single_step` warns multi-step training is real work. The buffer+GAE+collection exist; the multi-epoch PPO update path does not. Plan must scope 1B-C against the existing scaffold and STOP if it balloons beyond the estimate. +5. **b<1024 controller starvation** — per `pearl_pure_pnl_mode_starves_b16_controllers`, validate reward-distribution changes at b=1024; treat local b=128 smoke as directional only (`pearl_cluster_log_is_ground_truth`). + +--- + +## §8. Decision log entry + +``` +Decision: Adopt Reward–PnL Alignment Restoration as the next intervention, GATING F4 oracle pretraining behind it. Sequenced: force-pin slot 753 (D1) → cheap GAE-advantage↔pnl diagnostic (RA-G4) → conditional 1B-C credit-assignment completion (D2) → cluster. +Rejected (for now): F4 oracle pretraining — falsified as the lever (value generalizes; Q learns its target fine; the fault is the misaligned reward TARGET, not Q-init). External lit (Cal-QL/WSRL/IDAAC, 2023-2025) concurs value-pretrain is the wrong tool here. +Rationale: measured Pearson(reward,pnl)=0.28 (gate 0.70); slot-753 controller override makes the pure-pnl bootstrap cosmetic; value-error-vs-policy-error falsification (0.91×/0.83×/1.05× eval-vs-train) shows value generalizes while the agent loses money on both folds — the objective is misaligned, not the estimator. +Date: 2026-06-05 +Falsification trigger (viability): RA-G4 < 0.70 → STOP before 1B-C build. +Going-in-circles guard: this is the alignment branch of foundation-reshape §5.4; if D1+D2 pass RA-G4/G5 but RA-G6 fails at cluster, the encoder/generalization branch is legitimately isolated — NOT another reward reshape. +```