# F4 — Oracle Pretraining Infrastructure Spec

> **For agentic workers:** This spec defines the INFRASTRUCTURE for F4 oracle pretraining; it does NOT itself implement F4. Implementation follows in a separate plan after this spec is approved.

**Status:** Draft 1 (2026-06-05)
**Predecessor:** `docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md` (F4 §3.4)
**Phase 7a/7b empirical context:** F2 + F5 + OptA committed (HEAD `7376b1c67` on `ml-alpha-regime-observer`); cluster bleed −$6.4M @ step 519 → projected −$247M @ step 20k (sk2z9). F2 alone hinge_zero_rate saturated to 1.0 (mq2pc descendant); F5 alone surfaced 6:1 Short:Long bias; F2+F5 produced trade rate but 3× faster bleed than F2-alone.

**Branch target:** TBD (off post-Phase-7b commit `7376b1c67` once this spec is approved).

**Estimated wall-clock to evaluable F4-MVP cluster verdict:** ~7-10 working days (see §6). Honest range: 7 days if every phase passes first attempt; 14+ days if Phase 7c.0 oracle ground-truth fails or Phase 7c.5 cluster regresses.

**Scope discipline:** This is an INFRASTRUCTURE spec. It defines what must be BUILT to make F4 evaluable. It does NOT specify F4 hyperparameters, prescribe LOC counts at the kernel level, or replace the implementation plan that must follow. F4-Full (multi-epoch, full-action distribution, encoder-freeze variants) is explicitly OUT OF SCOPE; only F4-MVP is in scope.

**Linked specs (load-bearing context):**

- `docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md` — F4 §3.4 (predecessor), F2 §3.2 (Phase 7a, committed), F5 §3.5 (Phase 7b, committed)
- `docs/superpowers/specs/2026-06-01-reward-policy-alignment-investigation.md` — Pearson-misalignment foundation, established the reward function shape that F4 oracle must match
- `docs/superpowers/specs/2026-06-02-fast-dev-cycle.md` — Tier 1 / 1.5 / 2 funnel that all phase gates use
- `docs/superpowers/specs/2026-06-02-determinism-foundation.md` — bit-equality guarantee F4 must preserve

**Linked pearls (load-bearing):**

- `pearl_foxhunt_pi_trained_by_q_distillation_not_ppo` — π's only gradient source is Q-distill; F4's pretrained Q is what π distills from
- `pearl_foxhunt_trainer_is_genuinely_single_step` — `[B]`-only buffer shape; F4-MVP intentionally preserves this constraint
- `pearl_advantage_kernel_is_done_gated_td_not_gae` — V regression operates at T_eff=1; F4 oracle returns are γ-discounted not bootstrap-1-step
- `pearl_diag_pnl_fields_are_shaped_reward_not_usd` — load-bearing reward-function-identity caveat; F4 oracle MUST consume the same shaped reward function the trainer consumes (NOT raw USD pnl)
- `pearl_phase5_term_4_is_almost_potential` — Ng-Harada-Russell verdict on Phase 5 shaping terms; F4 oracle inherits the same non-potential terms (entry cost, MTM, short-hold penalty)
- `pearl_determinism_achieved` — same-seed bit-equality, F4 pretrain phase must preserve
- `pearl_fleet_fraction_not_aggregate` — at b=1024+ aggregate diag hides per-batch reality; F4 oracle quality gates must report fleet fractions
- `pearl_reward_signal_anti_aligned_with_pnl` — the load-bearing motivation for any oracle-style supervised warm-start
- `pearl_surfer_baseline_was_train_only_never_eval` — F4 gates must measure eval pnl, not train; train→eval transfer is itself a gate

---

## §1. Why F4, why now

### §1.1 The Phase 7 empirical trajectory

Three cluster runs after Phase 5 substrate, each with a distinct mechanism activation profile:

| Cluster | Phase | What was enabled | Mechanism observed | Eval pnl (cluster) |
|---|---|---|---|---|
| h6btt | Phase 5 single-head | (baseline; pre-Phase-7) | Hold-collapse to 98%; no opens | bleed ≈ $4.18/batch/step |
| mq2pc | Phase 5 + multi-head | (baseline) | TrailLoosen-collapse to 90%; oscillating attractors | bleed ≈ $19/batch/step |
| sk2z9 | Phase 7a F2 + Phase 7b F5 + OptA Hold-suppression | F2 hinged distill target + F5 flat-state mask + Gate-aware Hold suppression | F2 hinge active (hinge_zero_rate 1.0→0.91 across training; advantage_max=1.86); F5 forced opens (6:1 Short:Long); trade rate 7× higher than mq2pc | bleed −$6.4M @ step 519, linear projection −$247M @ step 20k (3× faster than F2-alone) |

### §1.2 What this proves about Phase 7's foundation

F2 + F5 mechanism gates BOTH pass (verifiable from cluster diag: hinge_zero_rate dropped, advantage_max > 0, flat-state mask correctness 100%, action distribution from flat ∈ {Long, Short}). The Phase 7 spec §5.3 attribution table maps this row exactly:

```
Phase 7b: F2 mech PASS | F2-G5 surfer (n/a, F5 active) | F5 mech PASS | F5-G8 FAIL
  → "F5 has no marginal benefit OR introduces new pathology; revert F5, commit F2-alone,
     escalate residual to encoder / alignment work."
```

But F2-alone also failed (sk2z9 sibling run with F5=0 showed hinge_zero_rate 1.0 saturation — Q never discovers Open advantage from cold start; random exploration cannot produce enough above-baseline samples to gradient-train the Q-head out of the inaction attractor). The empirical reading is:

- **F5 alone (forced opens)** loses money because random forced opens cannot beat commission drag in the no-edge regime fraction (predicted by Phase 7 spec §3.5 expected failure mode #1).
- **F2 alone (gradient redirection)** zeroes everything because cold-start Q has no above-baseline samples; hinge_zero_rate saturates at 1.0 (predicted by Phase 7 spec §3.2 expected failure mode #1 "mass-zero distill target").
- **F2 + F5 (gradient + forced data)** activates F2's hinge but feeds it the noise distribution of random forced opens; F2 learns "no Open action has positive advantage in expectation" because the data shows it, faster than the encoder can extract real edge signal.

Per spec §5.3 attribution outcome 3 / §5.4 escalation path:

> "If F2 + F5 both pass mechanism gates but eval pnl stays at baseline OR worsens: the inaction attractor was a *symptom* of a deeper problem. Neither is solvable by another Bellman-target reshape… F4 (oracle pretraining) becomes the right next dispatch if the encoder hypothesis dominates."

The encoder hypothesis is now favored: the agent's Q-head cannot bootstrap discriminating-action-quality estimates from cold-start random-policy-rollout-collected data because the signal-to-noise ratio is too low. The cluster data shows the agent CAN find some directional signal (F5-only directional bias 6:1 is non-random; F2's advantage_max=1.86 is non-zero) — but cannot find the *Q-quality* of specific actions in specific states fast enough to break the inaction attractor.

### §1.3 The disciplined response per `feedback_going_in_circles_pattern`

The previous spec's §5.3 attribution row identifies F4 as the next dispatch when F2+F5 mechanism gates pass and outcome fails. That row also requires that no further Bellman-target reshape (F1, F3, more F5 variants) be attempted. F4 attacks a structurally different axis — not the gradient, not the choice-set, but the *initial conditions* of Q. F4 either fixes the problem (cluster eval improves) or refutes the "encoder cannot bootstrap" hypothesis (cluster eval still fails) — and that refutation would itself be progress, since it would isolate the problem to genuine reward-pnl misalignment or train→eval distribution shift.

F4 is therefore the textbook next step under the spec's own escalation logic. This spec defines the infrastructure to make F4 evaluable.

---

## §2. F4 mechanism

### §2.1 Restated definition

Per the predecessor spec §3.4:

> "For each historical state `s_t` in the training data, compute an oracle Q-value `Q_oracle(s_t, a)` using perfect-foresight knowledge of future outcomes. Supervised-train Q to match `Q_oracle` for one or more epochs, THEN RL fine-tune. The agent inherits a Q function that *already knows* Hold is not always optimal because the supervised loss reflects realised returns from each action with perfect foresight."

In formal terms:

```
Q_oracle(s_t, a) = E_{trajectory | a_t = a, π* thereafter} [ Σ_{k=0..∞} γ^k · r_shaped(s_{t+k}, a_{t+k}) ]
```

where:
- `r_shaped` is the trainer's reward function (Phase 5 MTM + Phase 5 trail-max ceiling + base pnl + Phase 7a hinge if applicable). Per `pearl_diag_pnl_fields_are_shaped_reward_not_usd`, this is NOT raw USD pnl.
- `π*` is the per-state-after-t optimal policy. "Optimal" is defined by the same `r_shaped` function (the F4-MVP simplification is recursive — oracle's notion of optimal is bootstrapped from greedy-on-oracle-Q one step lookahead; see §2.3 below).
- `γ = 0.975` (foxhunt's current C51 / IQN discount factor).

### §2.2 Two scope tiers

#### F4-MVP (this spec's scope)

- Oracle Q for the **argmax action only**. Specifically: for each state `s_t`, compute one `(s_t, a_argmax, Q_argmax)` triple where `a_argmax` is the highest-realized-return action under a chosen oracle policy.
- Supervised MSE regression for **ONE epoch over the training fold** BEFORE the RL loop begins.
- Q head + encoder both training during the supervised phase (no freeze).
- Single fold (fold 1) only.
- Pretrain writes to the SAME Q head weights that the Bellman target writes to during RL fine-tune.

#### F4-Full (out of scope, eventual)

- Oracle Q for all 11 actions per state.
- Multi-epoch supervised regression (curriculum, annealing).
- Per-fold oracle generation across walk-forward folds.
- Encoder-only pretrain phase, Q-head-only pretrain phase, joint variants.
- Distillation regularization to prevent catastrophic Q drift during RL fine-tune.
- Online oracle update during RL (Dyna-style).

### §2.3 Mathematical relationship to the existing Bellman target

The trainer's existing Bellman update is:

```
Q_target(s, a) = r(s, a) + γ · max_{a'} Q_target_net(s', a')
L_q(s, a) = (Q_θ(s, a) − Q_target(s, a))²
```

F4-MVP inserts a supervised-regression phase BEFORE the RL phase:

```
Phase F4-pretrain (epochs=1):
    for each (s_t, a_argmax, Q_argmax) in oracle dataset:
        Q_θ(s_t, a_argmax)  ←  trained via MSE against Q_argmax
        # All other actions Q_θ(s_t, a≠argmax) receive no supervised loss (F4-MVP).
        # Encoder gradient flows through Q head as in standard supervised training.

Phase RL fine-tune (existing trainer):
    Q_θ is now initialized to "the oracle-argmax-q-shape Q"
    Standard Bellman fine-tune proceeds; Q_θ learns the off-argmax actions via the Bellman target;
    Q_θ for argmax actions starts near-correct rather than random.
```

The relationship to the inaction-bias problem (Phase 7 spec §2.1 equation c):

```
Pre-F4:    Q_init(s, a) ≈ random  →  Q_θ(s, no-op) > Q_θ(s, open) by entry_cost in expectation
                                  → F2 hinge zeroes Open mass → no learning signal → collapse

Post-F4-MVP: Q_init(s, a_argmax) ≈ Q_oracle(s, a_argmax)
                                  → for edge-bearing states where a_argmax = Open, Q_θ(s, Open) > Q_θ(s, Hold) at init
                                  → F2 hinge passes for Open at edge-bearing states → distill mass on Open → learning proceeds
                                  → for no-edge states where a_argmax = Hold, Q_θ(s, Open) < Q_θ(s, Hold) at init
                                  → F2 hinge zeroes Open mass → distill mass on Hold → surfer-style patience preserved
```

F4-MVP + F2 + (optionally) F5 is the conjecture this spec puts under test. Empirical falsification of that conjecture is encoded in the F4-G7/G8/G9 cluster outcome gates in §5.

### §2.4 The "optimally thereafter" simplification (load-bearing)

A truly optimal oracle would require knowing `Q*_oracle` for every state — which is the very thing F4 is trying to bootstrap. F4-MVP uses a *greedy-1-step-lookahead-on-realized-return* approximation:

```
For each (s_t):
    For each action a ∈ A:
        # Simulate the action through the trainer's LobSim for K steps (γ-discounted).
        # K is chosen large enough that γ^K is negligible (e.g., γ=0.975 → K=400 → γ^K ≈ 5e-5).
        # The first action is `a`; subsequent actions follow a heuristic policy:
        #   - if in-position-and-pnl > 2·entry_cost: close
        #   - if in-position-and-trail-breached: close
        #   - else: Hold
        # This is the "frictionless trail-stop heuristic policy" π_oracle.
        rollout_K = simulate_lobsim(s_t, a, π_oracle, K_steps=400)
        Q_oracle(s_t, a) = Σ_k γ^k · r_shaped(rollout_K[k])
    a_argmax = argmax_a Q_oracle(s_t, a)
```

This is a *heuristic* oracle (not a fixed-point oracle). The justification is that under any reasonable trading regime:

- The heuristic captures the dominant pnl driver (taking the trade and exiting on edge depletion).
- Heuristic-optimal Q values bound the random-policy Q values from above by a large margin.
- F4-MVP's job is to give Q a "good direction to start," not to compute the unreachable theoretical optimum. The RL fine-tune phase refines around the bootstrapped baseline.

Two-step honest framing for the implementation plan:

1. The oracle is NOT optimal in the game-theoretic sense; it is a *strong heuristic baseline* under perfect foresight.
2. F4-G1 (oracle-ground-truth gate, §5) tests whether the heuristic correlates with realized pnl on held-out states. If F4-G1 fails, the heuristic itself is broken (a tighter heuristic policy is required) — this is BEFORE we go anywhere near the trainer.

### §2.5 Citation chain

Oracle pretraining for offline-to-online RL is a well-established direction:

- Levine et al. 2020 ("Offline RL: Tutorial, Review, and Perspectives," arXiv:2005.01643) — survey establishing the framework.
- Kumar et al. 2020 ("Conservative Q-Learning," arXiv:2006.04779, NeurIPS) — CQL pessimism for offline-to-online warm-start.
- Nair et al. 2020 ("AWAC: Accelerating Online RL with Offline Datasets," arXiv:2006.09359) — advantage-weighted regression bridging.
- Kostrikov et al. 2021 ("Offline RL with Implicit Q-Learning," arXiv:2110.06169) — expectile regression sidestepping explicit pessimism.

F4-MVP is a simplified Nair-2020 (AWAC-style) warm-start: supervised regression on perfect-foresight return labels, then RL fine-tune. The trade-offs vs CQL pessimism / IQL expectile are out of scope for F4-MVP; revisit in F4-Full if MVP shows benefit.

For the heuristic oracle specifically: closest precedent is Schaul et al. 2015 ("Prioritized Experience Replay," arXiv:1511.05952) which uses TD-error-rank as a heuristic priority; F4-MVP uses heuristic-realized-return as a label rather than as a sampling weight. The mechanism difference is the level of supervision (PER reweights existing TD targets; F4 supplies new MSE targets).

---

## §3. What infrastructure is needed

This is the load-bearing section. F4-MVP requires building four pieces of infrastructure foxhunt does not currently have.

### §3.1 Oracle computation pipeline

**Purpose:** Given an MBP-10 trading session trace, produce a per-state oracle Q-value `Q_oracle(s_t, a_argmax)` and identify `a_argmax`.

**Where it runs.** Two design options:

| Option | Runs in | Pros | Cons |
|---|---|---|---|
| §3.1.A — Python preprocess (offline) | Python script over MBP-10 fixtures; outputs binary | Fast iteration on oracle policy logic; no CUDA-kernel changes; reuses `crates/ml-alpha/cuda/...` reward kernel definitions via JSON-mirrored math | Risk of math drift between Python and CUDA reward kernel; requires regeneration on every reward-shape change |
| §3.1.B — Rust + CUDA on-the-fly | A new binary `bin/foxhunt-oracle-gen` invoking the same `rl_fused_reward_pipeline.cu` and LobSim CUDA kernels the trainer uses | No math-drift risk (uses exact trainer reward); reusable for ablation studies | Higher infra cost; longer iteration cycle; CUDA dev environment required |

**Recommendation:** §3.1.B (Rust + CUDA on-the-fly). The math-drift risk is load-bearing: per `pearl_diag_pnl_fields_are_shaped_reward_not_usd`, the diag's `_usd` fields were misleading because the reward function was shaped differently than supposed. F4 oracle must be COMPUTED with the trainer's reward function, not a Python re-derivation, or F4 will fail F4-G1 ground-truth gate due to reward-function-identity bugs that are silent.

**Cost analysis (back-of-envelope, must be verified in Phase 7c.0).**

- 24 months × ES × MBP-10 events ≈ ~10⁸ states (per `pearl_mbp10_data_is_structurally_broken`, cluster PVC has 148GB coherent 24-month ES MBP-10; foxhunt's effective training-event throughput is ~10⁷-10⁸ per fold)
- F4-MVP needs only `(s_t, a_argmax, Q_argmax)` per state. Per state: ~16 bytes (state-id i64 + a_argmax i32 + Q_argmax f32). Per fold: ~1.6GB at 10⁸ states.
- However: producing `a_argmax` requires evaluating all 11 actions per state (per §2.4 algorithm), each requiring K=400-step LobSim rollout. At GPU-batched throughput ~10⁵ states/sec (extrapolated from existing LobSim benchmarks), full-fold oracle generation = 10⁸ / 10⁵ = ~10³ seconds = ~17 minutes per fold (highly speculative; Phase 7c.0 must measure).
- Per-action rollout is embarrassingly parallel across states and across actions. GPU implementation should batch them efficiently.

**Storage volume.** F4-MVP per-fold ≈ 1.6GB. Acceptable on local 4GB RTX 3050 (oracle generation is one-shot, not held in GPU memory during the trainer step). PVC storage on cluster is ample (Scaleway Kapsule has terabytes available).

**Frictionless vs realistic LobSim.** Two sub-options:

| Sub-option | LobSim accounting | Pros | Cons |
|---|---|---|---|
| §3.1.B.1 — Frictionless oracle | Slippage = 0; fills at mid; commission = 0 | Simpler; faster compute; tighter upper bound on achievable pnl | Oracle Q values overestimate real-world achievable; risk of Q being miscalibrated when RL fine-tune introduces friction |
| §3.1.B.2 — Realistic oracle | Use the SAME LobSim friction model the trainer uses (slippage, commission, MAS) | Math-identity with trainer; F4-pretrained Q is correctly calibrated | Slower compute; oracle complexity bound by LobSim complexity |

**Recommendation:** §3.1.B.2 (realistic oracle). Same reasoning as §3.1.B vs §3.1.A: math-identity with trainer is load-bearing. Better to spend the compute than to debug calibration mismatches later.

**Action-state coupling (load-bearing).** Several of foxhunt's 11 actions depend on `pos_state`, `unit_state`, and `trail_distance`:

- `ACTION_HOLD` is universally available but its effect depends on whether the trail is breached.
- `ACTION_TRAIL_LOOSEN` is a no-op at trail-max cap.
- `ACTION_PYRAMID_LONG/SHORT` is unavailable from flat (per F5 mask), and effect depends on `unit_state` (how many units are already open).
- `ACTION_FLAT_FROM_LONG/SHORT` is a no-op when flat.
- `ACTION_HALF_FLAT_LONG/SHORT` reduces position size by 50%.

The oracle simulator must track `pos_state` and `unit_state` per (s_t, a) branch through the K-step rollout. This means each (state, action) branch is a separate stateful simulation. Per-state cost is therefore O(K × |A|) LobSim steps = 400 × 11 = ~4400 simulator transitions per state. Over 10⁸ states → 4.4 × 10¹¹ simulator transitions. This is large; Phase 7c.0 cost estimate may need to recalibrate by:

- Pruning: pre-mask actions that are no-ops per state (saves ~50% of work for most flat states).
- Stride sampling: rather than oracle-label every state, label every Nth state (N=10-100) and let RL handle the rest. F4-MVP must specify N.

**Time horizon truncation.** The oracle integral is over an infinite-horizon discounted return. Truncate at K such that `γ^K · max_possible_reward < ε`. For γ=0.975, K=400 → γ^K ≈ 5e-5 — adequate if max reward magnitude is bounded (it is, via PopArt and trail-max). Use K=400 unless Phase 7c.0 finds the truncation introduces meaningful bias.

**Per-action MTM accounting.** The oracle MUST consume the same reward function as the trainer:

```
r_shaped(s, a, s_next) =
  r_base(s, a, s_next)                              # base pnl-delta or popart-whitened
  + RL_MTM_REWARD_WEIGHT_INDEX · MTM_delta(s, s_next)  # Phase 5 MTM
  + Phase_5_short_hold_penalty(s, a, s_next)        # Phase 5 term 1 (non-potential)
  + Phase_5_long_ride_bonus(s, a, s_next)           # Phase 5 term 3 (non-potential)
  + Phase_5_hold_bonus(s, a, s_next)                # Phase 5 term 4 (gate-zeroed, almost-potential)
  + Phase_5_entry_cost                              # Phase 5 term 2 (non-potential)
  + Phase_7a_F2_hinge_signal                        # if F2 enabled — but F2 is the consumer not a producer
```

Per `pearl_phase5_term_4_is_almost_potential`, three of Phase 5's four terms are explicitly non-potential. F4 oracle MUST include them (otherwise Q is calibrated for a different reward function than the trainer uses → F4-G3 will fail). The oracle reward function is the SAME function the trainer reads from `rl_fused_reward_pipeline.cu`.

**Versioning.** When the reward function changes (e.g., Phase 5 MTM weight tuned, new shaping term added), the oracle must be regenerated. Implementation plan must:

- Compute a hash of the reward-function configuration (Phase 5 weights + Phase 7a F2 mode + γ + K).
- Embed the hash in the oracle file name.
- Trainer reads the hash and refuses to load oracle if hash mismatches current ISV state.

### §3.2 Storage format and access path

**Format options.**

| Option | Format | Pros | Cons |
|---|---|---|---|
| §3.2.A — Predecoded binary (foxhunt convention) | `.predecoded.bin` with same MMap pattern as MBP-10 trace | Reuses existing infra (`crates/ml-alpha/src/data/loader.rs:85-98`); zero-copy I/O | Custom serialization needed |
| §3.2.B — Parquet | Columnar Arrow-compatible | Industry standard; introspectable | Adds Arrow dep to ml-alpha; per-row decode cost |
| §3.2.C — TFRecord / Numpy | Cross-tool compatibility | Easy to inspect from Python | Foreign to foxhunt's Rust toolchain |

**Recommendation:** §3.2.A (predecoded binary). Reuses foxhunt's existing MMap convention. Naming pattern:

```
<predecoded_dir>/ES.FUT_<quarter>.dbn.zst.oracle_q_argmax_v<reward_hash>.predecoded.bin
```

Where the file body is a sequence of `(state_global_id: u64, a_argmax: u8, q_argmax: f32)` records. Each record is 13 bytes; aligned to 16 for fast vectorized load.

The trainer's existing data loader (`crates/ml-alpha/src/data/loader.rs`) already MMaps MBP-10 traces — extending it to MMap a parallel oracle file is straightforward.

**Indexing.** The trainer references states by `(file_idx, byte_offset)` already (per existing MBP-10 loader). The oracle file uses the same indexing: each state in the MBP-10 trace has a corresponding entry in the oracle file at the same record offset. State-by-state alignment is therefore zero-cost (no hash table, no lookup, just record-index arithmetic).

**Cluster delivery.** PVC-mounted, same as MBP-10 training data:

```
/mnt/training-data/futures-baseline/ES.FUT/ES.FUT_<quarter>.dbn.zst.oracle_q_argmax_v<hash>.predecoded.bin
```

The Argo template `argo-alpha-rl.sh` would need to mount the new file (no template change required since PVC already exposes the directory).

**Versioning protocol.**

1. Reward-function hash computed at compile time from `crates/ml-alpha/src/rl/isv_slots.rs` Phase 5 + Phase 7a bootstrap values + γ.
2. Oracle generation binary embeds the hash in the filename.
3. Trainer at load time reads the hash and: (a) if it matches → load; (b) if it mismatches → log a warning AND refuse to load (force regeneration).
4. CI / cluster automation regenerates oracle when the hash changes (Argo workflow step).

### §3.3 Supervised pretraining phase

**Where in the trainer it runs.** Before the existing `step_with_lobsim_gpu` loop's first call:

```rust
pub fn step_with_lobsim_gpu(...) -> Result<...> {
    if self.f4_pretrain_enabled() && !self.f4_pretrain_done {
        self.pretrain_from_oracle()?;   // ← NEW
        self.f4_pretrain_done = true;
    }
    // Existing logic from here.
    ...
}
```

**New training loop body (F4-MVP).**

```rust
fn pretrain_from_oracle(&mut self) -> Result<()> {
    let n_batches = oracle_dataset.len() / BATCH_SIZE;  // BATCH_SIZE = 128 local, 1024 cluster
    for batch_idx in 0..n_batches {
        // 1. Sample BATCH_SIZE (state_id, a_argmax, q_argmax) records.
        let (state_batch, a_argmax_batch, q_argmax_batch) = oracle_dataset.sample(BATCH_SIZE, prng_seed);

        // 2. Run encoder forward on state_batch (existing forward kernel chain).
        self.encoder_forward(&state_batch)?;

        // 3. Q head forward to produce Q_θ(s, a_argmax) — read only at a_argmax positions.
        self.q_head_forward(&state_batch)?;
        let q_pred = self.gather_q_at_action(&a_argmax_batch);   // [B] — Q_θ(s, a_argmax)

        // 4. MSE loss: (q_pred - q_argmax_batch).pow(2).mean()
        let loss = (q_pred - q_argmax_batch).pow(2).mean();

        // 5. Backward + Adam through Q head AND encoder.
        loss.backward();
        self.optimizer.step();

        // 6. Diag emit: pretrain step + MSE + max |Δ|
        self.emit_pretrain_diag(batch_idx, loss);
    }
    Ok(())
}
```

**Encoder freeze (F4-MVP decision).** Both Q head AND encoder train during pretraining (encoder unfrozen). Rationale: foxhunt's encoder (mamba2-based) shares weight with the policy head; freezing the encoder would also freeze the policy's perceptual capacity. F4-Full may revisit encoder-only or Q-only freeze variants once F4-MVP results are in.

**Number of pretrain epochs.** ONE epoch. Adaptive stopping (e.g., until MSE plateaus) is F4-Full scope.

**Gradient flow specifics (load-bearing).**

- The MSE loss writes to Q head weights (the same weights the Bellman target writes to during RL fine-tune). NO separate "pretrain head" — F4-MVP intentionally uses the production Q head.
- The encoder gradient flows from the Q head's L2 loss back through the trunk. This means encoder weights at the start of RL fine-tune are a *function of the oracle pretrain*. If the oracle is biased (e.g., heuristic policy is bad), the encoder is biased in that direction.
- Gradient clipping: use the existing trainer's per-batch gradient norm clip (slot TBD; verify in Phase 7c.0).
- Learning rate: use the existing Q-head Adam learning rate. NO new LR controller for pretrain (F4-MVP simplification). If MSE diverges, this is a STOP signal per F4-G2.

**New ISV slots.**

| Slot ID | Name | Bootstrap | Effect when bootstrap |
|---|---|---|---|
| 824 (next free) | `RL_F4_PRETRAIN_MODE_INDEX` | 0.0 (OFF) | 0=skip, 1=MVP one-epoch, 2=full multi-epoch (reserved) |
| 825 | `RL_F4_PRETRAIN_BATCH_COUNT_INDEX` | 0.0 (read-only counter) | Filled by trainer with `n_batches` at start of pretrain; diag exposes for verification |
| 826 | `RL_F4_PRETRAIN_FINAL_MSE_INDEX` | 0.0 (read-only) | Filled by trainer at end of pretrain; F4-G2 reads this |
| 827 | `RL_F4_PRETRAIN_DONE_INDEX` | 0.0 | Set to 1.0 once pretrain completes; trainer reads to gate "skip if already done" |

Slot ID ranges are illustrative; Phase 7c.0 must verify current `RL_SLOTS_END` and allocate from there.

**Determinism.** The pretrain phase must preserve `pearl_determinism_achieved`. Oracle reads use the established seeded-PRNG pattern (per the IQN race fix + OutcomeHead seed fix in commit `c75c45a93`). The pretrain epoch's sample order is deterministic for a given seed.

### §3.4 Trainer integration points

Per `pearl_foxhunt_trainer_is_genuinely_single_step`, the trainer is `[B]`-only. F4-MVP intentionally preserves this:

- Pretrain samples are `[B]`-shaped (one batch = B independent states, one action each).
- No rollout buffer needed — the supervised loss is computed over individual transitions.
- The pretrain loop is a separate code path in `integrated.rs`; it does not interfere with the trainer's main step body.

**New struct fields on `IntegratedTrainer`.**

```rust
pub struct IntegratedTrainer {
    // ... existing fields ...

    /// F4 oracle storage handle (memory-mapped predecoded binary).
    pub oracle_storage: Option<crate::data::oracle_loader::OracleStorage>,

    /// F4 pretrain step counter (for diag emit).
    pub f4_pretrain_step: u64,

    /// F4 pretrain done flag (idempotent skip if already done).
    pub f4_pretrain_done: bool,
}
```

**New code paths.**

1. `OracleStorage::load_from_predecoded(path) -> Self` — MMap and validate hash.
2. `IntegratedTrainer::pretrain_from_oracle(&mut self) -> Result<()>` — the supervised loop body.
3. `IntegratedTrainer::oracle_batch_sample(&mut self, batch_size, prng_seed) -> (StateIds, ActionIds, QValues)` — deterministic batch sampler.
4. `IntegratedTrainer::gather_q_at_action(actions: &[u8]) -> CudaSlice<f32>` — utility to gather Q_θ values at specific action indices (similar to the existing `gather` pattern in `rl_q_pi_distill_grad.cu`).

**Determinism wiring.**

- The oracle sample PRNG seeds from the existing trainer seed.
- The kernel reads from `oracle_q_d` (a new CudaSlice<f32>) and `oracle_a_argmax_d` (a new CudaSlice<u8>) are race-free (single thread per batch reads its row).
- The MSE gradient kernel uses the existing Adam optimizer step.

**Code structure.** A new module `crates/ml-alpha/src/data/oracle_loader.rs` for the predecoded-binary MMap; a new module `crates/ml-alpha/src/trainer/f4_pretrain.rs` for the pretrain loop body. Both files are new but the surface area is small (~200-400 LOC each, mostly trivial).

---

## §4. F4-MVP scope boundary

The discipline boundary of F4-MVP is the single most important section of this spec for preventing scope creep into F4-Full.

### §4.1 What F4-MVP IS

- ONE supervised pretrain epoch (no curriculum, no annealing, no early-stopping on MSE).
- argmax-only oracle: one `(s_t, a_argmax, Q_argmax)` triple per state.
- Q-head + encoder both training during pretrain (no freeze).
- Single fold (fold 1) only.
- Pretrain writes to SAME Q head weights that Bellman target writes to during RL fine-tune.
- NO separate "pretrain head."
- NO distillation regularization to prevent post-pretrain Q drift.
- NO online oracle update during RL.

### §4.2 What F4-MVP is NOT

| Out-of-scope feature | Why excluded from MVP | When to revisit |
|---|---|---|
| Multi-epoch supervised | F4-Full; MVP tests whether one-epoch warm-start moves the needle | F4-MVP cluster passes F4-G4; user wants to explore further pretrain |
| Full-distribution oracle (all 11 actions) | F4-Full; 11× compute cost; MVP-only-argmax is the cheapest meaningful test | F4-MVP fails F4-G3 because non-argmax actions need supervision |
| Encoder freeze | F4-Full; risks overfitting trade-off | F4-MVP cluster shows encoder catastrophic-shift post-pretrain |
| Per-fold oracle generation (walk-forward) | F4-Full; MVP tests fold-1 only | F4-MVP fold-1 passes and we need to verify the win generalizes across folds |
| Action-curriculum (oracle gradually reveals actions) | F4-Full | F4-MVP fails F4-G6 because Q drifts too far from oracle during RL |
| Reward-shape-curriculum (oracle gradually reveals shaping) | F4-Full | Phase 5 shaping terms cause oracle to over-recommend wait |
| Adversarial / smarter-than-heuristic oracle | F4-Full | F4-G1 ground-truth gate suggests heuristic ceiling is too low |
| Behavioral cloning from human demonstrations | Out of scope entirely | No human demonstrations exist in foxhunt |
| Imitation learning from a learned-policy expert | Out of scope entirely | A learned expert policy must itself be trained — recursive |
| Online oracle update (Dyna-style) | F4-Full; major arch change | F4-MVP cluster passes and we want to tighten oracle as policy improves |
| Hindsight relabeling (HER) | Out of scope; orthogonal | Separate spec if pursued |
| F4 + F5 combined (Phase 7c.6 if needed) | Sequenced after F4-MVP-only result | F4-MVP fails F4-G7 outcome and Phase 7c.6 is invoked per §6 |

---

## §5. Empirical falsification gates

Three categories: **Pretrain-phase gates** (verify oracle pipeline is correct), **RL fine-tune phase gates** (verify F4 changes downstream training), **Cluster outcome gates** (verify F4 actually helps).

### §5.1 Pretrain-phase gates — verify oracle pipeline correctness

Run during Phase 7c.0 / Phase 7c.1 / Phase 7c.2 / Phase 7c.3 of the implementation plan.

| Gate ID | Metric | Pass condition | When measured | What it falsifies |
|---|---|---|---|---|
| F4-G1 (oracle ground-truth) | Pearson correlation between `Q_oracle(s_t, a_argmax)` and the actually-realized discounted return when an agent forced to take `a_argmax` is run forward K=400 steps on held-out data | ≥ 0.7 | Phase 7c.0 (offline) | If fails: oracle policy heuristic is wrong (re-design heuristic or escalate to F4-Full smarter oracle) |
| F4-G2 (pretrain loss decreases) | Final MSE after 1 epoch / initial MSE at epoch start | ≤ 1e-2 (i.e., MSE drops ≥ 100×) | Phase 7c.2 (local Tier 1.5) | If fails: supervised loss path is broken (kernel bug, wrong gradient flow, oracle hash mismatch) |
| F4-G3 (Q-head outputs match oracle) | On held-out state subset: `argmax_a Q_θ(s_t, a) == a_argmax` agreement rate | ≥ 70% | Phase 7c.3 (local Tier 1.5) | If fails: encoder/Q-head capacity insufficient OR oracle is degenerate (most a_argmax are Hold) |

**Honest caveat.** F4-G1's 0.7 Pearson threshold is calibrated against `pearl_reward_signal_anti_aligned_with_pnl` baseline (0.11-0.45 random-policy). 0.7 is a *substantial* improvement target; if F4-G1 produces 0.5-0.7, this may still be worthwhile but is a STOP-and-evaluate signal. If F4-G1 produces < 0.5, the oracle heuristic is broken and Phase 7c.1+ MUST NOT proceed (the pipeline would be wired to wrong labels).

### §5.2 RL fine-tune phase gates — verify F4 changes downstream training

Run during Phase 7c.4 of the implementation plan, with F4 pretrain ON and standard RL fine-tune proceeding.

| Gate ID | Metric | Pass condition | When measured | What it falsifies |
|---|---|---|---|---|
| F4-G4 (RL improves vs cold-start) | Steps until `l_q` reaches the cold-start cluster `l_q` at step N | ≤ 0.3 × N | Phase 7c.4 (local Tier 1.5) | If fails: pretrained Q drifts away from oracle too fast during RL (need distillation regularization, F4-Full) |
| F4-G5 (early policy quality) | Action entropy at step 100 of RL | ≥ 1.5 nats | Phase 7c.4 | If fails: pretrained policy collapsed during pretrain (encoder overfit or oracle too argmax-degenerate) |
| F4-G6 (no surfer-attractor migration) | F2's `hinge_zero_rate` (existing diag from F2 commit) at step 1000 | ≤ 0.5 | Phase 7c.4 | If fails: F4's pretrained Q does not prevent F2's hinge from saturating (Q still has Hold > Open everywhere) |

### §5.3 Cluster outcome gates — verify F4 actually helps

Run during Phase 7c.5 of the implementation plan, at cluster Tier 2 (b=1024, 20k train + 5k eval, fold 1, ~70 min wallclock).

| Gate ID | Metric | Pass condition | When measured | What it falsifies |
|---|---|---|---|---|
| F4-G7 (cluster eval pnl) | `eval_summary.total_pnl_usd` | ≥ F2+F5 (sk2z9 baseline) cluster eval pnl + $10M / session | Phase 7c.5 | If fails: F4-MVP does not solve the eval-fail problem; consider F4-Full or escalate to model-architecture |
| F4-G8 (no degenerate attractor) | Max single-action share at eval steps 5k and 15k | < 80% | Phase 7c.5 | If fails: F4 has migrated to a different degenerate (e.g., always-Open-Long); diagnose before committing |
| F4-G9 (train→eval transfer) | `eval_pnl / train_pnl` | ≥ 0.7 | Phase 7c.5 | If fails: F4 does not solve the train→eval generalization gap; problem is upstream of Q-init (encoder generalization, regime shift) |

**F4-G9 is the most consequential gate.** Per `pearl_surfer_baseline_was_train_only_never_eval`, the train→eval gap has been the long-standing failure mode (eval ≈ 5% of train at b=128 currently). F4 explicitly aims to address this by giving Q a starting point that already encodes "what works on held-out data." If F4 still produces ~5% eval-to-train ratio, F4-MVP has not solved the problem and we have refuted the encoder-Q-bootstrap hypothesis. F4-G9 failure is a *progress signal* — it isolates the next investigation direction.

### §5.4 Stop-on-fail rules

- **F4-G1 fails (oracle ground-truth Pearson < 0.7):** STOP at Phase 7c.0; redesign oracle heuristic or escalate to F4-Full smarter oracle. DO NOT proceed to Phase 7c.1+.
- **F4-G2 fails (MSE does not decrease):** STOP at Phase 7c.2; the supervised loss path is broken (kernel bug, hash mismatch, or gradient-flow bug). Debug before continuing.
- **F4-G3 fails (Q-head does not match oracle on held-out):** STOP at Phase 7c.3; either encoder capacity insufficient, or oracle is degenerate (overwhelmingly Hold). Either escalate to F4-Full (full-distribution oracle) or accept and proceed to Phase 7c.4 with degraded expectation.
- **F4-G4, F4-G5, or F4-G6 fail at smoke:** Do not cluster. Diagnose with `pearl_fleet_fraction_not_aggregate`-discipline diag; consider F4-Full distillation regularization.
- **F4-G7 fails at cluster but G8 + G9 are healthy:** F4-MVP insufficient; consider Phase 7c.6 (F4 + F5) or escalate to F4-Full.
- **F4-G9 fails at cluster regardless of G7:** F4-MVP has refuted the encoder-Q-bootstrap hypothesis. STOP; escalate to model-architecture work (e.g., different encoder, attention modifications) or train→eval regime-shift work.

### §5.5 The "going in circles" guard

Per `feedback_going_in_circles_pattern` and `feedback_investigation_first_falsification_methodology`: F4 is the third independent intervention axis tested in Phase 7 (gradient axis = F2; choice-set axis = F5; initial-conditions axis = F4). If F4-MVP fails on F4-G7 AND F4-G9, the predetermined next action is NOT another Bellman-target reshape (per Phase 7 spec §5.4 → all the easy variants are now tested). The next direction is upstream: encoder, reward function, or train/eval regime structure. This guard prevents the empirical loop from spawning a Phase 8 / 9 / 10 series of more-Bellman-reshapes.

---

## §6. Implementation phases

Each phase has a predetermined gate and a STOP-if-fail rule. Phases run sequentially.

### Phase 7c.0 — Oracle pipeline (build + verify) — ~2-3 days

Build the offline oracle generator. Verify ground-truth correlation.

- Build `bin/foxhunt-oracle-gen` (a new binary).
- Implement realistic LobSim rollout for each (s_t, a) over K=400 steps.
- Implement heuristic-policy continuation per §2.4.
- Output `(state_global_id, a_argmax, q_argmax)` triples in predecoded.bin format.
- Test against hand-computed ground truth on 10 sample states (verify by inspection).
- Generate oracle for fold-1 training data (one quarter of ES MBP-10).
- Compute F4-G1: Pearson correlation between oracle `Q_argmax` and forced-rollout realized return on held-out fold-1-eval states.

**Gates:** F4-G1 only.

**STOP if:** F4-G1 < 0.7. Redesign heuristic or accept F4-Full as required prerequisite.

### Phase 7c.1 — Storage + access path (build + verify) — ~1-2 days

Wire oracle reads into the trainer.

- Add `crates/ml-alpha/src/data/oracle_loader.rs` (MMap predecoded.bin).
- Add hash validation + version check.
- Add `OracleStorage::sample(batch_size, prng_seed)` with bit-deterministic behavior.
- Write a Rust unit test that two same-seed sample calls produce identical (state_id, a, q) batches.
- Add minimal Tier 1 correctness test (load + sample + validate hash).

**Gates:** Determinism unit test only (no behavioral gate yet).

**STOP if:** Determinism test fails. Per `pearl_determinism_achieved`, this MUST hold.

### Phase 7c.2 — Pretraining MSE loss (build + verify) — ~2-3 days

Add `pretrain_from_oracle()` method.

- Add new ISV slots 824-827 per §3.3.
- Implement `IntegratedTrainer::pretrain_from_oracle()` body (MSE loss, backprop, Adam step).
- Default `RL_F4_PRETRAIN_MODE_INDEX = 0.0` (OFF) — pretrain skipped by default.
- Add diag emit for `pretrain_mse_initial`, `pretrain_mse_final`, `pretrain_steps_completed`.
- Run local Tier 1.5 smoke with `RL_F4_PRETRAIN_MODE_INDEX = 1.0` and verify MSE decreases (F4-G2).

**Gates:** F4-G2 (MSE decreases ≥ 100×).

**STOP if:** F4-G2 fails. Debug gradient flow / hash mismatch.

### Phase 7c.3 — Verify pretrained Q matches oracle — ~1 day

Confirm F4-G3 on held-out subset.

- Add a Rust integration test that runs the full pretrain on local fold-1 train fixtures, then evaluates `argmax_a Q_θ` vs `a_argmax` on held-out states.
- Emit per-state agreement rate.
- Report aggregate F4-G3 metric.

**Gates:** F4-G3 (≥ 70% argmax agreement).

**STOP if:** F4-G3 < 50%. Likely either degenerate oracle (mostly Hold) or insufficient encoder capacity; consider F4-Full full-distribution oracle before proceeding.

### Phase 7c.4 — RL fine-tune from pretrained Q (smoke + verify) — ~1 day

Run Tier 1.5 smoke with full F4 pipeline.

- Configure: F2 ON, F5 OFF (clean F4 isolation), F4 pretrain ON, Band ON (for surfer principle preservation).
- Run mid-smoke (~10 min wallclock).
- Verify F4-G4, F4-G5, F4-G6 against post-pretrain diag fields.
- Run `determinism-check.sh --quick` to verify pretrain phase preserves bit-equality.

**Gates:** F4-G4, F4-G5, F4-G6, determinism preserved.

**STOP if:** Any RL fine-tune gate fails. Diagnose before cluster.

### Phase 7c.5 — Cluster verify — ~80 min wallclock + analysis time

Submit Argo run.

- Configure: single-head, Band ON, F2 ON, F5 OFF, F4 ON.
- Submit cluster Tier 2 (b=1024, 20k train + 5k eval, fold 1).
- Wait ~70 min for completion.
- Analyze `eval_summary.json` for F4-G7, F4-G8, F4-G9.

**Gates:** F4-G7 (eval pnl improvement), F4-G8 (no new attractor), F4-G9 (train→eval transfer).

**STOP if:** F4-G7 fails AND F4-G9 fails → escalate per §5.4. F4-G7 alone fails AND F4-G9 passes → consider Phase 7c.6 (F4 + F5 combined).

### Phase 7c.6 (CONDITIONAL) — Re-engage F5 with F4 pretrain — ~80 min cluster

ONLY if Phase 7c.5 shows F4 alone insufficient on F4-G7 BUT F4-G9 healthy (train→eval transfer improved).

- Configure: single-head, Band ON, F2 ON, F5 ON, F4 ON.
- Submit cluster Tier 2.
- Compare F4 + F5 vs F4-alone vs sk2z9 (F2 + F5 + OptA without F4).

**Gates:** F4-G7 vs F4-alone (must show marginal improvement from F5).

**STOP if:** Phase 7c.6 fails. F4-Full or model-architecture work is the next direction.

### Total time estimate

| Phase | Duration | Cumulative |
|---|---|---|
| 7c.0 | 2-3 days | 2-3 days |
| 7c.1 | 1-2 days | 3-5 days |
| 7c.2 | 2-3 days | 5-8 days |
| 7c.3 | 1 day | 6-9 days |
| 7c.4 | 1 day | 7-10 days |
| 7c.5 | 1-2 days (incl. analysis) | 8-12 days |
| 7c.6 (conditional) | 1-2 days | 9-14 days |

**Honest range: 7-14 working days for F4-MVP cluster verdict.** Faster if every phase passes first attempt; slower if Phase 7c.0 oracle heuristic needs iteration (likely) or Phase 7c.5 cluster surfaces new pathology (also likely given Phase 7 history).

---

## §7. Surprises, risks, and out-of-scope

### §7.1 Honest risks

1. **Oracle ground-truth requires non-trivial LobSim simulation.** §2.4's heuristic policy is itself a hypothesis. The Phase 7c.0 ground-truth gate (F4-G1) is non-trivial to pass. If heuristic-realized-return correlates poorly with achievable return, the oracle is broken before we get to the trainer. Mitigation: spend the time on the heuristic; consider hybrid heuristic + RL-trained-on-small-fold for the oracle policy.

2. **Q drift during RL fine-tune may undo the warm-start.** Without distillation regularization (KL between current Q and pretrained Q, à la AWAC), the RL Bellman target may move Q rapidly away from oracle. F4-MVP intentionally excludes this regularizer for scope reasons, but F4-G4 (RL improvement rate) and F4-G6 (hinge_zero_rate) will detect rapid drift. If drift is severe, F4-Full distillation is required.

3. **Encoder may overfit during pretraining.** One epoch with no dropout / weight decay could push the encoder to memorize the oracle's training-fold features. Mitigation: F4-G5 (action entropy at step 100 of RL) detects overfit-collapse. If encoder overfit is detected, F4-Full encoder-freeze variant is needed.

4. **F4 does not solve regime shift.** If the test data has different action-quality structure than the train data, F4 trained on train will give a Q that is correctly bootstrapped for train but wrong for eval. This is a fundamental limitation. F4-G9 (train→eval transfer ratio) measures the impact. If F4-G9 fails, the problem is regime-shift, not Q-init — F4 does not solve it.

5. **Reward function identity is load-bearing.** Per `pearl_diag_pnl_fields_are_shaped_reward_not_usd`, the trainer's `r_shaped` is non-trivial (Phase 5 MTM + entry cost + short-hold penalty + long-ride bonus). The oracle MUST use the EXACT SAME function. Re-derivation in Python (§3.1.A rejected option) was rejected precisely because of this. If §3.1.B math-identity is not maintained, F4 will appear to fail for the wrong reasons.

6. **Determinism requirement is non-trivial.** Per `pearl_determinism_achieved`, the bit-equality discipline took ~170 LOC of fixes across multiple subsystems. The pretrain phase introduces new kernel paths (gather Q at action, MSE loss, etc.) — each must be carefully implemented to preserve same-seed bit-equality. F4-MVP's Phase 7c.4 includes a `determinism-check.sh` gate; do not skip.

7. **Cluster vs local divergence.** Per `pearl_cluster_log_is_ground_truth`, local b=128 PPO diverges even when cluster b=1024 trains stable. F4-G4/G5/G6 are local-Tier-1.5 gates; F4-G7/G8/G9 are cluster gates. They MAY produce different verdicts. The discipline is: do not skip local gates (catch obvious bugs early), but do not over-weight local results (cluster is ground truth).

### §7.2 Out of scope (explicit)

- F4-Full (multi-epoch, full-distribution oracle) — separate spec eventually.
- Encoder-only pretraining (Q frozen, encoder trained on auxiliary task) — separate spec.
- Adversarial / RL-trained oracle (smarter than argmax heuristic) — separate spec.
- Online oracle update during RL (Dyna-style) — separate spec, major arch change.
- Hindsight relabeling (HER) — orthogonal direction.
- F4 + F5 combined (Phase 7c.6 if needed) — conditional within this spec.
- Multi-fold walk-forward oracle generation — F4-Full.
- Distillation regularization to prevent Q drift — F4-Full.
- Confidence-gate position-aware wiring (`pearl_confidence_gate_pos_state_unwired`) — pre-existing, orthogonal.
- GAE / rollout buffer — orthogonal; per `pearl_foxhunt_trainer_is_genuinely_single_step` not blocked by F4.

---

## §8. Decision-relevant alternatives surveyed

Briefly note alternatives ruled out at this stage. Full analysis is in the predecessor spec §3.

- **Architectural changes (e.g., transformer policy, attention modifications).** Out of scope for this dispatch; would require new encoder + retrain everything. Becomes the next direction if F4 fails F4-G9.
- **Reward shaping F1 (per-step inaction cost).** Per Phase 7 spec §3.1: non-potential per Ng-Harada-Russell 1999, violates surfer principle, criterion-7 score 2. F4 has criterion-7 score 5 (preserves surfer by encoding patient behavior in oracle Q labels).
- **F3 action-space reduction.** Per Phase 7 spec §3.3: criterion-7 score 1 (total surfer violation), removes documented architectural progress, not reversible. Rejected.
- **Behavioral cloning from human expert.** No human expert demonstrations exist for foxhunt; out of scope.
- **Imitation learning from a heuristic policy.** A heuristic policy would itself need to be designed; oracle is a stronger signal (perfect-foresight return vs heuristic action sequence).

**Explicit statement:** F4-MVP is the most disciplined next step because:

(a) F2+F5 mechanism gates pass but outcome gates fail at cluster (per `sk2z9` empirical data + spec §5.3 attribution outcome 3).

(b) F4 has the strongest theoretical foundation among unimplemented options (Levine 2020 / Nair 2020 / Kumar 2020 / Kostrikov 2021 — offline RL is a mature direction).

(c) F4-MVP scope is bounded to ~7-14 days with predetermined gates at each phase.

(d) The methodology `feedback_investigation_first_falsification_methodology` requires spec → plan → implement (not direct dive). This is the spec.

(e) F4 attacks a distinct mechanistic axis (initial conditions) from F2 (gradient) and F5 (choice-set). The Phase 7 axis-completion logic is preserved.

---

## §9. Open questions for the implementer

The implementation plan that follows this spec must resolve at least these questions:

1. **Trade-boundary marking in MBP-10.** Where in MBP-10 data layout do we identify trade boundaries for oracle return computation? The existing predecoded format includes mid-price events — does the oracle use mid-prices or top-of-book bid/ask for fill simulation? Verify in Phase 7c.0.

2. **Reward function versioning protocol.** Exact hash function for reward-function-state. Include: Phase 5 weight slots (MTM, entry cost, etc.), Phase 7a F2 slots, Phase 7b F5 slots, γ. Spec the hash format in the plan.

3. **Slippage / market impact accounting.** Should the oracle account for slippage? Phase 7 spec §3.5 LobSim trail-stop kernel includes a slippage model. Use the same? Or assume frictionless? Recommendation: §3.1.B.2 (realistic). Confirm in Phase 7c.0.

4. **Per-state action-availability tracking.** How is oracle action-availability tracked through the K-step rollout? Specifically, `unit_state` (current pyramid layer count) affects PYRAMID action effects. Spec the per-(s_t, a) state propagation.

5. **Oracle for train fold only or train+eval folds?** F4-MVP says train only (no oracle for eval). But F4-G3 (held-out match rate) requires SOME eval-state oracle to compute. Resolve: use a strict train-fold sub-split for in-pretrain held-out (e.g., last 10% of fold-1-train), OR generate oracle for fold-1-eval purely for the gate (and discard for training). Recommendation: latter — gate-only oracle on eval is acceptable.

6. **ISV slot range.** Per current state (`grep RL_SLOTS_END` shows 824), allocate 824-827 (or shift if collisions). Confirm in Phase 7c.2 step 1.

7. **Encoder LR for pretraining.** Use same LR as RL fine-tune? Or a separate (typically larger) supervised LR? F4-MVP simplification says: same LR. If F4-G2 shows MSE plateau too high, revisit.

8. **Diag emission discipline.** Per `pearl_fleet_fraction_not_aggregate`, F4 pretrain diag emissions MUST include per-batch fractions (e.g., "fraction of pretrain batches where MSE > threshold") not just aggregate `worst_X`. Specify diag schema in Phase 7c.2.

9. **Batch size for pretrain.** Use same B as RL (128 local / 1024 cluster)? Or different? F4-MVP simplification: same B. Recommend confirming compute headroom in Phase 7c.0.

10. **Resumption semantics.** What happens if pretrain is interrupted (kernel crash, OOM)? F4-MVP simplification: ALL or NOTHING — pretrain restarts from scratch on resume. `RL_F4_PRETRAIN_DONE_INDEX` is the idempotent gate. F4-Full would add checkpoint/resume.

---

## §10. Document discipline notes

- **No implementation code in this spec.** Pseudocode in §3.3 is design clarification, not implementation prescription.
- **No new files outside this one spec.** The §3.1 / §3.4 references to `bin/foxhunt-oracle-gen` and `crates/ml-alpha/src/data/oracle_loader.rs` are forward references for the implementation plan, not files this spec creates.
- **Honest scope.** F4 is multi-week even at MVP scope. The 7-14 day range in §6 is honest about variance. Do not commit to F4-MVP if there is < 2 weeks of focused capacity available; better to defer.
- **Theoretical claims cited.** Every claim in §2.5, §7, §8 has a citation to predecessor spec, pearl, or external paper.
- **Predetermined falsification at every phase.** §5.4 and §6's "STOP if" rules are not optional; they are the methodology's load-bearing discipline per `feedback_investigation_first_falsification_methodology`.
- **No dispatch of further agents from this spec.** The output is the spec file. The implementation plan is a separate dispatch.

---

## §11. Decision log entry

If F4-MVP is approved for implementation, log the decision per `feedback_v7_gem_methodology` §decision-log:

```
Decision: Adopt F4-MVP (oracle pretraining with argmax-only realistic-LobSim heuristic oracle) as Phase 7c, sequenced after Phase 7a (F2 committed) and Phase 7b (F5 committed). F4-MVP is the disciplined response per Phase 7 spec §5.3 attribution outcome 3 + §5.4 escalation path when F2 + F5 mechanism gates pass but cluster outcome regresses.
Rejected: F4-Full (multi-epoch, full-distribution, encoder-freeze variants — deferred until F4-MVP cluster verdict).
Rationale: F2 + F5 + OptA (sk2z9) demonstrated that gradient-axis and choice-set-axis interventions are mechanically operational but cannot bootstrap Q-quality estimates from cold-start random-policy data. F4 attacks the distinct initial-conditions axis: warm-start Q with perfect-foresight realized-return labels BEFORE RL fine-tune begins. This addresses the "encoder cannot bootstrap" hypothesis directly.

F4 is theoretically grounded in Levine 2020 (Offline RL Survey), Nair 2020 (AWAC), Kumar 2020 (CQL), Kostrikov 2021 (IQL). F4-MVP is a simplified Nair-2020-style warm-start. F4-MVP scope is bounded by §4 (one epoch, argmax-only oracle, fold-1 only, no distillation regularizer, no encoder freeze).

F4-MVP is implementable with bounded scope (~7-14 days) across 6 sequential phases (7c.0 — 7c.5) with predetermined gates F4-G1 through F4-G9, three of which (F4-G1, F4-G2, F4-G3) gate progression of the pipeline itself. Phase 7c.6 is conditional re-engagement of F5.

Date: 2026-06-05
Falsification trigger (oracle pipeline): F4-G1 per §5.1
Falsification trigger (RL fine-tune): F4-G4 + F4-G5 + F4-G6 per §5.2
Falsification trigger (outcome): F4-G7 + F4-G9 per §5.3
Going-in-circles trigger: §5.4 F4-G9 failure → STOP, no further Bellman-target reshape
Pre-registered failure prediction: §5.4 "F4-G9 failure refutes encoder-Q-bootstrap hypothesis"
```
