# Bellman-Target Foundation Reshape — Breaking the Structural Inaction Bias

**Date:** 2026-06-04
**Status:** Design — pre-implementation, candidate-comparison; NOT a plan
**Branch target:** TBD (off Phase 5 `0463e44e0` once the chosen option is selected)
**Estimated effort (recommended option only):** see §6
**Scope discipline:** This spec analyses FIVE candidate interventions, recommends a disciplined STAGED pair (Phase 7a = F2 alone; Phase 7b = F5 conditional on F2 outcome), and defines distinguishable gates. It does NOT prescribe LOC, code, or implementation order. A separate `plans/` document is required before any coder dispatch.

**Revision log:**

- **Revision 1 (2026-06-04, "a").** Original spec analysed F1-F4 and recommended F2-alone.
- **Revision 2 (2026-06-04, "b").** User critique identified a category error in F3 (conflated global vs state-conditional action removal). Spec added **F5 — state-conditional action availability mask**, and recommended *F5 first, F2 conditional* on the argument that F5 has the highest pre-criterion score and addresses the choice-set at the structural level.
- **Revision 3 (2026-06-04, "c", THIS REVISION).** User critique identified that F5 violates the documented **surfer trading principle** (`pearl_dd049d9a4_surfer_baseline_verified`, `pearl_two_alpha_modes_surfer_vs_trend`): F5's hard mask forces opens when flat, destroying patience-while-waiting-for-setup, which is the structural ability the surfer alpha mode depends on. F2 preserves this principle (its hinge zeroes Open mass when `E_Q(Open) ≤ E_Q(Hold)`, so the agent retains the *learned* discretion to wait). The recommendation flips: **Phase 7a = F2 alone; Phase 7b = F5 conditional**, with F5 explicitly framed as a fallback that accepts surfer→trend regression if F2's gradient redirection is insufficient.

**Executive summary (read before §1).** Foxhunt has demonstrated two profitable trading modes in cluster:

- **Surfer** (`dd049d9a4` baseline): wr ≈ 0.56, avg_hold ≈ 30 steps, +$10.6M @ step 18k, low-frequency patient entries with high win-rate.
- **Trend-follower** (`6d33f18b7` baseline): wr ≈ 0.36, avg_hold ≈ 17 steps, +$41.5M peak, higher-frequency directional commits with lower win-rate.

Per `pearl_two_alpha_modes_surfer_vs_trend`, choosing between them is a philosophy decision, and per `feedback_surfer_philosophy_trading` the user has explicitly chosen the surfer mode. The surfer's load-bearing structural property is **patience-while-waiting-for-setup** — the agent must be able to abstain from low-confidence setups (Hold-from-flat) and only commit when its model predicts edge.

F5 (state-conditional action mask) makes Hold structurally unavailable from flat → forces an Open every flat-state step regardless of belief → **destroys patience-while-waiting** by construction → forces surfer → trend regression. This violates the surfer principle even though F5 mechanically eliminates the h6btt all-Hold attractor.

F2 (Q-distill advantage target with `max(0, E_Q − baseline)` hinge) preserves the surfer principle by *mathematical derivation*: when `E_Q(Open) ≤ E_Q(Hold)` (no edge detected), the hinge zeroes Open's distill mass and the policy learns "Hold is better, wait"; when `E_Q(Open) > E_Q(Hold)` (edge detected), Open receives positive mass and the agent learns to commit. F2 IS the surfer principle expressed as a loss function.

This spec therefore recommends **Phase 7a = F2 alone**, with F5 staged as conditional Phase 7b only if F2's outcome gate (eval pnl ≥ h6btt baseline + $5M / session) fails AND F2's mechanism gates pass AND we are prepared to consciously accept surfer → trend regression as the trade-off for breaking the h6btt-class structural collapse. F5 remains a valid fallback; the staging order encodes the surfer-first preference.

**Linked specs (orthogonal context):**

- `2026-06-01-reward-policy-alignment-investigation.md` — the precursor investigation that proved the Pearson-misalignment problem and recommended pure-pnl shaping (Phase 3B-Y, then Phase 5)
- `2026-06-02-trainer-rollout-buffer-gae.md` — Phase 1B GAE pivot (still in-flight, not load-bearing here)
- `2026-06-02-multi-head-policy-with-r-multiple.md` — Phase 2A multi-head (the architecture mq2pc deployed)
- `2026-06-03-no-transaction-band-architecture.md` — Phase 4-A band foundation (the architecture h6btt/mq2pc deployed)

**Linked pearls (load-bearing):**

- `pearl_foxhunt_pi_trained_by_q_distillation_not_ppo` — π's only gradient source is Q-distill
- `pearl_advantage_kernel_is_done_gated_td_not_gae` — V regression targets are done-gated 1-step TD; T_eff = 1
- `pearl_foxhunt_trainer_is_genuinely_single_step` — `[B]`-only buffer shape forecloses true trajectory algorithms without multi-week refactor
- `pearl_phase5_term_4_is_almost_potential` — Ng-Harada-Russell verdict on all four Phase 5 shaping terms
- `feedback_going_in_circles_pattern` — the stop-rule that triggered this spec
- `feedback_investigation_first_falsification_methodology` — the methodology this spec is written under

---

## §1. Empirical evidence — three runs, one failure mode

Three consecutive cluster runs each surfaced a *different* degenerate equilibrium. They are not three bugs. They are three masks over the same architectural pathology: **the Bellman target structurally rewards costless inaction over costly action.**

### §1.1 The trajectory

| Cluster | Phase | Failure mode | Cluster bleed | Run-class fix attempted |
|---|---|---|---|---|
| `4hq7c` | Phase 4-C (band foundation) | TrailLoosen unbounded risk; trail grew 376 → 2959 ticks (~$37k/position) | comparable magnitude | Per-unit trail-max ceiling (Phase 5 `0463e44e0`) |
| `h6btt` (killed) | Phase 5 single-head | Hold-collapse to 98%; positions never opened | $4.18/batch/step | Multi-head policy + r-multiple gating (Phase 2A) |
| `mq2pc` (killed) | Phase 5 + multi-head | TrailLoosen-collapse to 90%, then Hold/TrailLoosen oscillation | **$19/batch/step** (4.5× worse than h6btt) | (this spec) |

The `mq2pc` step-by-step shows the system migrating between degenerate attractors without ever escaping the *class* of degenerate attractors:

```
step  100 :  93% Hold,  2% TrailLoosen          (attractor A: do-nothing-from-flat)
step 1000 :  14% Hold, 81% TrailLoosen          (attractor B: do-nothing-from-position via trail widening)
step 2000 :   4% Hold, 90% TrailLoosen          (attractor B saturated; peak bleed $14M / 500 steps)
step 3000 :  40% Hold, 56% TrailLoosen          (oscillating between A and B; $1M / 500 steps)
cum pnl @ step 3000:  −$58M
linear projection to step 20k:  −$386M
```

### §1.2 What the trajectory proves

Three independent claims:

1. **The attractor is not "Hold" specifically.** Multi-head + r-multiple gating successfully *broke* the Hold attractor — and the system immediately found a new degenerate attractor (TrailLoosen) that is functionally identical. Any intervention that targets *one specific zero-cost action* will be defeated by Q-distill discovering the next zero-cost action.

2. **The attractor is the *class* of zero-cost actions.** Hold and TrailLoosen share one structural property: they emit no market order, incur no transaction cost, and produce no realized PnL delta. Both have `E[r | a] = 0 + γ·V(s')` while every action that opens or closes a position has `E[r | a] = −cost + γ·V(s')`. Q-distill converges to whichever zero-cost action has the highest E[V(s')] under the current π.

3. **Goodhart-Skalse 2024 generalizes here.** Their NeurIPS result on proxy-reward fix attenuation in deep RL applies to the *V → Q-Bellman target → softmax(Q/τ) → π distill* chain documented in `pearl_foxhunt_pi_trained_by_q_distillation_not_ppo`. Each link is a learned approximator; each one attenuates the corrective signal of upstream reward fixes. The MTM reward (Phase 5 `0463e44e0`) attempted to *add* a negative reward to underwater positions — and `mq2pc` shows the system simply migrated to the *other* zero-cost action that MTM does not penalize as harshly when trails widen.

### §1.3 The local Phase 6 SAC α experiment

A separate local experiment raised SAC α (entropy regularization strength) to MAX and observed that the policy entropy did not increase enough to escape the attractor — when `E_Q[Hold] >> E_Q[Open]`, no amount of entropy push at the policy head can overcome Q-distill's preference for the zero-cost action. This forecloses entropy-side interventions as a foundation fix. **The intervention point must be Q-distill or the Bellman target itself.**

---

## §2. Root cause — why the Bellman target structurally favors inaction

### §2.1 The math

For a zero-cost action `a_0` (Hold, TrailLoosen-at-cap, TrailTighten-no-fire, …):

```
Q_target(s, a_0) = r(s, a_0) + γ · V(s')
                 = 0 + γ · V(s')                                (a)
```

For a costly action `a_c` (any open / close / size-change):

```
Q_target(s, a_c) = r(s, a_c) + γ · V(s')
                 = −entry_cost + (realized_pnl_delta) + γ · V(s')  (b)
```

The expected realized PnL delta is ≈ 0 in a Pearson-misaligned reward regime (see `pearl_reward_signal_anti_aligned_with_pnl`: Pearson 0.11–0.45 vs random baseline 0.5). The entry cost is *deterministic*. So in expectation:

```
E[ Q_target(s, a_c) − Q_target(s, a_0) ] = E[realized_delta] − entry_cost
                                         ≈ 0 − entry_cost
                                         < 0                       (c)
```

**The bias is permanent.** It is independent of training signal, exploration noise, encoder quality, optimizer choice, or any controller adaptation. As long as (i) zero-cost actions exist in the action space and (ii) the Q-distill softmax target sees the un-corrected Bellman-target difference, the policy will prefer them.

### §2.2 Why Q-distill makes it worse than standard PPO

In a standard PPO trainer the policy gradient is `∇L = E[ A_t · ∇ log π(a|s) ]` with `A_t = R̂_t − V(s)`. A consistently-negative entry cost would propagate through advantage normalization and the policy would learn a *bounded* preference for inaction.

In foxhunt's architecture (`pearl_foxhunt_pi_trained_by_q_distillation_not_ppo`):

- `pi_target = softmax(E_Q / τ)` (kernel `rl_q_pi_distill_grad.cu` line 86)
- The distill gradient is `∇L = λ · (π_θ − π_target)` per-action
- Softmax is *translation-invariant in its input*, but NOT in its target ordering. The target's max action receives the largest gradient share.

So if `E_Q[a_0] = γV(s')` and `E_Q[a_c] = γV(s') − entry_cost`, then `pi_target` peaks at `a_0` with magnitude `softmax(γV/τ) / sum_a softmax((γV − cost·1[a=c])/τ)`. The gap between the two is exactly the softmax slope at the offset point — **which the agent cannot reduce by training** because the cost is a property of the environment, not the policy.

### §2.3 Why GAE (Phase 1B) does not fix this

GAE in `pearl_foxhunt_pi_trained_by_q_distillation_not_ppo` ADDENDUM 2026-06-02e: GAE benefits V regression quality, which propagates downstream through Q → Q-distill → π. **But GAE does not change the cost asymmetry in equation (c).** Better V estimates make the asymmetry *more accurate*, not smaller. GAE is orthogonal to this spec.

### §2.4 Why T_eff = 1 makes it intractable today

Per `pearl_advantage_kernel_is_done_gated_td_not_gae`, V regression's advantage is done-gated 1-step TD: `advantages[b] = is_done ? (ret − vt) : 0.0f`. So V only ever sees non-zero learning signal on close events. The path `reward → V → Q-target → π-distill` therefore has its first link operating at T_eff=1 (close events only), and the cost asymmetry in (c) accumulates undistorted into the Q targets at every held step.

Combined: there is no controller, no shaping term, and no entropy weight that can structurally invert (c). The fix must change either:

- **F1.** What `r(s, a_0)` is — i.e., make inaction non-zero so (a) and (b) become symmetric
- **F2.** What `softmax target` Q-distill consumes — i.e., remove the absolute Q level so the distill gradient only sees Q-differences from the best no-op
- **F3.** What the action space contains — i.e., eliminate the zero-cost actions
- **F4.** What `Q_target` is initialised to — i.e., warm-start Q with knowledge that Hold is not always optimal
- **F5.** What `A(s)` is — the *available action set* — i.e., keep all actions in the space but hard-mask actions whose availability is incoherent with the current position / trail state. Different from F3: F3 removes actions globally; F5 conditions availability on state. This is the standard Constrained-MDP `A(s) ⊂ A` reformulation (Altman 1999) rather than action-space reduction.

---

## §3. F1-F5 theoretical analysis

### §3.1 F1 — Bellman-target reshape: add per-step inaction cost

**Mechanism.** Modify the reward kernel so every zero-cost action incurs a small per-step cost ε:

```
r[t] = realized_pnl_delta(t) + w_mtm · unrealized_delta(t) − ε · inaction_indicator(a[t])
```

where `inaction_indicator(a) = 1` iff `a ∈ {Hold, TrailLoosen-no-effect, TrailTighten-no-effect}`. The set of "no-effect" trail actions is determined post-hoc by the trail-mutate kernel detecting the clamp-at-cap branch.

**Theoretical foundation.** Direct intervention on the Markov reward function. Equation (c) becomes:

```
E[Q_target(s, a_c) − Q_target(s, a_0)] = E[realized_delta] − entry_cost − (−ε)
                                       = E[realized_delta] − entry_cost + ε   (c')
```

With `ε = entry_cost`, the asymmetry vanishes in expectation. The agent's choice between `a_c` and `a_0` is then a function of `E[realized_delta]` — which is the alignment signal the agent is *supposed* to learn from.

**Citation theorem (foundation).**  Ng, Harada, Russell 1999 ("Policy Invariance Under Reward Transformations", ICML): a shaping function `F(s, a, s')` preserves the optimal policy *iff* it has the potential form `F(s,a,s') = γΦ(s') − Φ(s)` for some Φ: S → ℝ. **Pure inaction cost is NOT a potential function** — it depends on `a`, not just `s`. Therefore it CAN (and is designed to) change the optimal policy. This is a feature, not a bug — but it must be acknowledged.

**Can ε be made potential-conserving?** Define a state potential `Φ(s, a) = −ε · 1[a ∈ no-op]`. Ng-Harada-Russell explicitly require `Φ: S → ℝ` (not `S × A`); the Φ-dependence on action is precisely the case excluded by the theorem. **No.** F1 is provably non-potential. Whether the *intended* policy change (escape from no-op attractors) is the only policy change requires empirical verification (see §5 gates).

**Cross-reference to `pearl_phase5_term_4_is_almost_potential`.** Per the math agent's audit of Phase 5 shaping, three of four shaping terms were *also* non-potential and *did* destabilise the optimal policy in non-trivial ways. F1 inherits that risk. The mitigation is keeping ε small enough that the *bound on policy distortion* (≤ ε / (1 − γ) in the discounted-return formulation) is dominated by the intended escape benefit. With γ = 0.975 and ε on order of entry_cost ≈ 15 reward-units, the steady-state V-distortion bound is ≈ 600 reward-units per held trajectory. For comparison, average trade PnL magnitude is ~50–200 reward-units. **F1 is borderline non-trivial in distortion.**

**Expected effect on inaction-bias.** Strong and direct. The reward source itself becomes negative on zero-cost actions; Q learns this within a few epochs; the attractor is removed by construction.

**Expected failure modes.**

1. *Overshooting.* If ε is too large, the agent learns to over-trade just to avoid the inaction cost. Goodhart-Skalse style: the agent optimises the proxy (avoid the −ε flag) without optimising pnl.
2. *Definition fragility.* The "inaction" indicator must be precisely defined. TrailLoosen at a non-cap state IS a no-op for pnl but emits a state change (trail distance). The detector must classify it correctly; classification errors are training-data-distribution-dependent.
3. *Q-distill amplification.* Because the cost is in `r`, it propagates through V → Q-Bellman → π-distill at every step. This is *more* signal density than a one-shot fix at the distill target. Could be helpful (more learning signal) or harmful (amplifies misclassification of inaction events).
4. *Interaction with MTM reward (Phase 5).* MTM already gives negative reward to underwater positions. F1 adds negative reward to inaction. Both stack on TrailLoosen-on-underwater (the `mq2pc` attractor). Combined this could create an over-correction toward early-close-at-loss — the classic "loss aversion" pathology.

**LOC estimate.** Small. ~80–150 LOC: add inaction-indicator computation to `rl_fused_reward_pipeline.cu` Phase 5 section, two new ISV slots (ε_value, enabled-gate), trainer-side ISV bootstrap, controller for ε auto-tune.

**Predetermined falsification gates.**

| Gate | Local b=128 (Tier 1.5) | Cluster b=1024 (Tier 2) |
|---|---|---|
| G1 (inaction-share) | Hold% in [10%, 40%] across step 0–2000; TrailLoosen% < 10% | Hold% + TrailLoosen% < 50% at step 5000 |
| G2 (overshoot) | total_trades / 2000 ∈ [50, 400] (vs `mq2pc` baseline ≈ 4 trades/2000 steps) | trades_per_session_per_1000_steps ∈ [10, 300] |
| G3 (eval pnl) | (not measured at Tier 1.5; Tier 2 only) | eval_pnl > 0 OR eval_pnl > h6btt baseline by ≥ $5M / session |

**Risk vs reward.** Medium-high reward (directly addresses root cause). High risk (proven non-potential; precedent from Phase 5 shows non-potential terms repeatedly destabilise the learned policy). Net: speculative but tractable.

---

### §3.2 F2 — Q-relative-to-best-inaction distill target (advantage-style distill)

**Mechanism.** Modify `rl_q_pi_distill_grad.cu` so the per-batch softmax target uses `(E_Q − baseline)` instead of `E_Q`:

```
baseline(s) = max over a ∈ no-op-set { E_Q(s, a) }    or    mean E_Q(s, a) over no-ops
π_target(s, ·) = softmax( (E_Q(s, ·) − baseline(s)) / τ )
```

Softmax is mathematically translation-invariant in its input, so `π_target` itself is unchanged after subtracting `baseline`. **The change is in the gradient flowing into π_θ.**

Wait — this requires careful analysis. Let me redo it. The distill gradient is `grad_distill[a] = λ · (π_θ(a) − π_target(a))`. Subtracting a constant from the *target's pre-softmax input* leaves `π_target` invariant, and the gradient invariant.

The correct F2 formulation is different: replace `π_target` with one *derived from advantages*. Define:

```
adv_Q(s, a) = E_Q(s, a) − baseline(s)
π_target(s, a) = softmax( adv_Q · 1[adv_Q > 0] / τ )
```

i.e., **mask out actions whose Q is no better than the best no-op**. Only super-no-op actions retain mass in the distill target. This is the dueling-value-decomposition idea applied to a distill target.

Or equivalently (smoother): use a hinge-style target:

```
π_target(s, a) ∝ softmax( max(0, E_Q(s, a) − baseline(s)) / τ )
```

**Theoretical foundation.** Wang et al. 2016 ("Dueling Network Architectures for Deep Reinforcement Learning", ICML, arXiv:1511.06581) — the dueling-Q insight: decomposing `Q(s, a) = V(s) + A(s, a)` with `Σ_a A(s, a) = 0` (or `max_a A = 0`) gives stable learning because the absolute Q-level (which is shared across actions and reflects the value of the state) does not contaminate the action-ranking signal. Foxhunt already has dueling Q (Phase 4, per `pearl_phase4_dueling_head_plus_advantage_norm_is_new_sota`), but **the dueling decomposition is internal to the Q head**. The distill target consumes `E_Q` (V + A) and re-introduces the absolute Q level into the policy update.

F2 brings the dueling insight to the distill target itself. The policy update gradient sees only `A(s, a) − A(s, best-no-op)`, eliminating the cost-asymmetry artifact of equation (c) at the distill stage.

**Citation chain.**
- Wang et al. 2016, §3 (advantage decomposition is policy-invariant when target is action-ranking).
- Schulman et al. 2015 (GAE), §3 (advantage-based policy updates are unbiased estimators of policy gradient with reduced variance).
- Mnih et al. 2016 (A3C) — advantage-actor-critic for the analogous insight in policy gradient.

There is no published precedent specifically on "advantage-style Q-distill targets" — distill is normally framed as imitation, not policy improvement. F2 is a small original step from Wang 2016, not a re-discovery.

**Does F2 provably break the inaction-attractor convergence?**

Yes, conditional on the no-op-set being correctly defined. Proof sketch: the F2 target has `π_target(a) = 0` for any action whose `E_Q ≤ baseline`. Therefore the distill gradient `λ · (π_θ(a) − 0) = λ · π_θ(a)` is *always* pushing π_θ(a) toward 0 for any no-op-equivalent action — regardless of its absolute Q level. The only way an action survives in π is `E_Q(s, a) > baseline(s)`, which is exactly the condition the agent is *supposed* to learn.

Caveat: the SAC entropy term still pulls π toward uniform. So `π_θ` cannot collapse to 0 on no-ops as long as α > 0. The equilibrium is `π_θ(no-op) = small but positive`, with magnitude determined by the SAC α / distill λ ratio. This is the desired behavior: exploration on no-ops is preserved, but exploitation pressure is removed.

**Theoretical concern.** "Which actions to include in baseline set?" is the load-bearing question.

Three options:

| Sub-option | Definition of no-op set | Pros / Cons |
|---|---|---|
| F2a | Static set: `{Hold, TrailTighten, TrailLoosen}` | Easy to implement; brittle to action-space changes; ignores that TrailLoosen at non-cap IS effectful |
| F2b | Dynamic: any action with `Δposition_lots == 0` post-execution | Principled (matches the structural property — no realized pnl delta possible without lots change); requires running fill simulator to detect |
| F2c | Implicit: `baseline(s) = top-K mean of E_Q(s, ·)` for some K | Avoids hardcoding the set; gradient direction is preserved if the no-ops cluster at the top |

F2c is the most elegant but the least falsifiable — its behavior depends on the Q-value landscape, which the agent itself is learning. F2a is operationally cleanest. F2b is theoretically cleanest but architecturally invasive (requires fill simulator output to be visible at the distill kernel, currently it is only visible at the action-execution kernel).

**Recommendation within F2:** start with F2a as a stub; if cluster shows it works, refactor to F2b for robustness.

**Expected effect on inaction-bias.** Mechanically removes the inaction attractor from the distill target. The policy can only learn to assign mass to super-no-op actions. Whether the *encoder* produces enough super-no-op Q-signal for the agent to learn at all is the empirical question.

**F2 preserves the surfer trading principle (load-bearing).** The surfer alpha mode (`pearl_dd049d9a4_surfer_baseline_verified`: wr=0.56, hold=30, +$10.6M @ step 18k) requires *patience-while-waiting-for-setup* — the agent must retain discretion to abstain when its model does not predict edge. F2's hinge target preserves this structurally:

- When `E_Q(Open) ≤ baseline = E_Q(Hold)` (no edge detected): `max(0, E_Q(Open) − baseline) = 0` → Open receives zero distill mass → the gradient `λ · (π_θ(Open) − 0) = λ · π_θ(Open)` pushes π toward zero on Open → the agent learns "Hold is the higher-Q action; wait."
- When `E_Q(Open) > baseline = E_Q(Hold)` (edge detected): `max(0, E_Q(Open) − baseline) > 0` → Open receives positive mass → the agent learns to commit.

The decision to act vs wait remains a *learned* property of the policy, conditioned on the encoder's edge signal. The agent retains the surfer's signature patience while still escaping the *baseline-equal* inaction attractor (because the hinge zeroes Hold's contribution too when other actions exceed it). This is the load-bearing reason F2 is now Phase 7a rather than Phase 7b — see §4.1.4 for the full surfer-principle decision criterion.

**Expected failure modes.**

1. *Mass-zero distill target.* If no action's `E_Q` exceeds `baseline`, the target is uniform-zero. Then the distill gradient pushes π toward uniform — equivalent to SAC entropy max. The agent becomes a random walker. This is a real risk during early training (cold-start Q gives noisy E_Q).
2. *Q-instability amplification.* The hinge `max(0, A)` introduces a non-differentiable point at the baseline. Stochastic gradient noise around the baseline could cause oscillation in `π_target` between "include" and "exclude" for marginal actions. Smoothing via softmax-with-temperature partly addresses this.
3. *Distill λ recalibration.* The existing controller (slot 486) is tuned for the current `π_target`. F2 reshapes the target distribution → the controller's setpoint will be wrong → must be re-bootstrapped.
4. *Interaction with Phase 4 band.* The band masks actions to Hold *after* the policy emits them. If F2 makes the policy avoid Hold, but the band forces Hold anyway, the gradient signal from F2 never observes the executed action — the entire Q-distill pipeline still trains as if Hold was chosen. **F2 is partially short-circuited by Phase 4-A.** Mitigation: either disable the band during F2 evaluation, OR add a "selected-before-mask" action signal to the Q-distill kernel.

**LOC estimate.** Small. ~50–100 LOC: modify the `s_pi_target` computation in `rl_q_pi_distill_grad.cu` (lines 80–91), add one or two ISV slots (no-op-baseline-mode, no-op-set-definition), trainer-side dispatch, controller for distill-λ recalibration.

**Predetermined falsification gates.**

| Gate | Local b=128 (Tier 1.5) | Cluster b=1024 (Tier 2) |
|---|---|---|
| G1 (target-entropy effect) | `RL_Q_DISTILL_TARGET_ENTROPY_MEAN_INDEX` (slot 733) drops by ≥ 0.5 nats from baseline at step 1000 (proves the target is now action-discriminating, not uniform) | same metric < 1.5 nats by step 5000 |
| G2 (action distribution) | Hold% + TrailLoosen% < 60% across step 100–2000; entropy_action ∈ [1.5, 2.2] nats | Hold% + TrailLoosen% < 50% at step 5000; eval-time action_entropy > 1.0 nats |
| G3 (V quality preserved) | l_v < 4× baseline; no NaN/diverging loss | l_v stable through step 20k |
| G4 (eval pnl) | n/a | eval_pnl > h6btt baseline by ≥ $5M / session |

**Risk vs reward.** Medium-high reward (mechanism is rigorous; well-grounded in Wang 2016; preserves surfer principle by construction). Low-to-medium risk (small kernel diff; reversible via ISV gate; well-defined gates). Net (per Revision 3 scoring with surfer-principle criterion added): **best risk-adjusted return overall (score 30); preserves the surfer trading principle while attacking the inaction attractor at the gradient axis. Recommended as Phase 7a primary intervention. F5 is staged as a conditional fallback (see §3.5 and §4.1.5/§4.2).**

---

### §3.3 F3 — Action space reduction (REJECTED; see also F5 §3.5)

**Disambiguation vs F5 (load-bearing).** F3 *removes* actions from the global action space (the enum `crate::rl::common::Action` shrinks from 11 to ~4 entries). F5 *keeps* all 11 actions in the enum but conditionally restricts which ones are *available* in each state via a hard mask applied before the policy softmax. The semantic difference is the same as `A` vs `A(s)` in Altman 1999's Constrained-MDP formulation. The user's critique that surfaced F5 identified that this spec's original F3 framing conflated these two — "the action space is wrongly used or at least the agent doesn't understand it should get out of the no-ops when trading is active, it should not be able to enter a no-op when it's doing nothing." F3 (this section) reads that critique as "shrink the global space"; F5 reads it as "shrink the state-conditional space." F3 stays REJECTED below; F5 is the option F3 should have been.

**Mechanism.** Reduce the action space from 11 actions to ~4: `{Long+, Short+, Flat, Hold}`. Drop: pyramid variants (LongSmall, LongLarge, ShortSmall, ShortLarge split → one Long, one Short), TrailTighten (a7), TrailLoosen (a8), HalfFlatLong / HalfFlatShort (a9, a10).

Concretely:
- a0 ← Short (combine ShortSmall + ShortLarge)
- a1 ← Long (combine LongSmall + LongLarge)
- a2 ← Hold
- a3 ← Flat (combine FlatFromLong + FlatFromShort by position-context routing)

**Theoretical foundation.** No specific theorem — this is a *simplification* argument. Smaller action spaces are easier for Q to estimate well at fixed sample-complexity (cf. Sutton & Barto §6.5 on tabular Q-learning convergence; the rate depends on |A|). Empirically, action-space reduction has been the default move for many RL trading papers (e.g., Deng et al. 2016 used `{Buy, Hold, Sell}` ternary). **It does not address the root cause.**

The root cause (§2.1 equation c) requires zero-cost-action existence. Hold remains a zero-cost action in F3. The attractor in (c) still exists; it has just been compressed from {Hold, TrailLoosen, TrailTighten} (3 zero-cost actions) to {Hold} (1 zero-cost action).

**Does F3 fix the failure mode?** Partially. The `mq2pc` oscillation between Hold and TrailLoosen ceases because TrailLoosen does not exist. But the Hold-collapse (h6btt) re-emerges. F3 takes the system *back to the h6btt regime*. The bleed-rate would likely return to $4.18/batch/step (h6btt) rather than $19/batch/step (mq2pc). **F3 reduces severity but does not solve.**

**Theoretical concern (cross-reference).** Per `git show 20c835713`, TrailTighten and TrailLoosen were added to close `pearl_dead_trail_stop_actions_a7_a8` — actions that existed as enum members in `crate::rl::common::Action` with no kernel consumer. Removing them re-opens the dead-wiring pattern. Worse: the trail-stop mechanism (`rl_trail_stop_check.cu`) currently overrides actions to FlatFrom* when the per-unit trail-distance is breached. Without TrailTighten/Loosen, the trail-distance becomes static (init only, no agent control). This re-introduces a risk-pathology that the SP20 P5 work (`20c835713`) was specifically designed to address:

> "rl_trail_stop_check.cu — per-batch per-unit breach check. … On breach, OVERRIDE actions[b] to FlatFromLong (a3) or FlatFromShort (a4)."

A static trail distance means every trade has a fixed-width stop-loss. With volatility shifts (regime changes), this is provably suboptimal vs adaptive trails (cf. Wilder 1978, Chande 1992 on volatility-adaptive stops). **F3 reintroduces a known-suboptimal risk model that earlier work specifically addressed.**

Additionally, the pyramid variants (`LongSmall` vs `LongLarge`) encode position-size differentiation. Removing them collapses the agent's ability to express conviction differential. Cross-reference to `pearl_position_sizing_missing_adaptation_layer`: foxhunt explicitly identified the missing position-sizing controller as a gap. F3 *removes* the action-level expressiveness needed to address that gap.

**Expected effect on inaction-bias.** Reduces but does not eliminate. Takes h6btt regime as the bound. NOT a foundation fix.

**Expected failure modes.**

1. *Hold attractor re-emerges in pure form.* h6btt bleed re-appears; system collapses to all-Hold.
2. *Static trail risk pathology.* On regime shifts, fixed-width trails either close prematurely (volatility spike) or too late (volatility crash). Either is a known failure mode.
3. *Loss of position-sizing expressiveness.* Without Long/Short small/large differentiation, the agent cannot express conviction differential. Combined with `pearl_position_sizing_missing_adaptation_layer`, this is a regression.
4. *Pyramid pathway gone.* Without small/large position openings, multi-unit pyramiding (Turtle 2N style, per Eckhardt-Dennis) is impossible. This removes a documented profitable pattern from the agent's reach.

**LOC estimate.** Medium-large. Touches: enum `crate::rl::common::Action`, all 11 `ACTION_*` defines across ≥ 5 kernel files, `actions_to_market_targets.cu` (140+ LOC of action-to-market mapping), `multi_head_policy_forward.cu` (N_ACTIONS = 11 hardcoded), `multi_head_policy_aux_prior.cu`, `rl_band_mask.cu`, `rl_q_pi_distill_grad.cu` (N_ACTIONS = 11), audit-wiring infrastructure. ~500–1000 LOC of changes, plus regression-testing every ISV controller that touches action distribution.

**Predetermined falsification gates.**

| Gate | Local b=128 | Cluster b=1024 |
|---|---|---|
| G1 (Hold-collapse re-emerges?) | Hold% > 80% at step 2000 → F3 fails identically to h6btt | Hold% > 80% at step 10000 → F3 fails |
| G2 (eval pnl) | n/a | eval_pnl ≥ h6btt baseline (lower bar than F1/F2) |

**Risk vs reward.** Low-to-medium reward (does not address root cause; only mitigates a symptom). High implementation cost. **Functionality removal — see `feedback_no_functionality_removal`.** Net: dispreferred unless F1/F2 also fail catastrophically and the policy needs simplification before any further work can be done.

---

### §3.4 F4 — Oracle pretraining (offline Q warm-start)

**Mechanism.** For each historical state `s_t` in the training data, compute an oracle Q-value `Q_oracle(s_t, a)` using perfect-foresight knowledge of future outcomes. Supervised-train Q to match `Q_oracle` for one or more epochs, THEN RL fine-tune. The agent inherits a Q function that *already knows* Hold is not always optimal because the supervised loss reflects realised returns from each action with perfect foresight.

The minimal-scope version: compute `Q_oracle(s_t, a) = realized_return(s_t, a, oracle-π-from-t+1)` for a single action `a = argmax` and `Q_oracle(s_t, other) = realized_return(s_t, other) − safety-margin`.

The full version: compute `Q_oracle` via dynamic-programming-with-oracle on every action × every state, then supervised loss on the full distribution.

**Theoretical foundation.** Offline-to-online RL.
- Levine et al. 2020 ("Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems", arXiv:2005.01643): survey establishing offline pretraining as a stable initial-policy generator.
- Kumar et al. 2020 ("Conservative Q-Learning for Offline Reinforcement Learning", NeurIPS, arXiv:2006.04779): CQL — pessimistic Q-learning that avoids overestimation on out-of-distribution actions during the offline phase.
- Nair et al. 2020 ("Accelerating Online Reinforcement Learning with Offline Datasets", arXiv:2006.09359): AWAC — advantage-weighted regression for offline-to-online warm-start.
- Kostrikov et al. 2021 ("Offline Reinforcement Learning with Implicit Q-Learning", arXiv:2110.06169): IQL — expectile regression that sidesteps explicit pessimism.

**Theoretical concern (load-bearing).** `pearl_foxhunt_trainer_is_genuinely_single_step` documents the trainer is `[B]`-only. The Q-learning loop is *embedded* in `step_with_lobsim_gpu_body`. Oracle pretraining would require:

1. A separate offline training loop (not the current step-wise loop).
2. Persistence of an offline data buffer (`[N_states × N_actions × N_features]` — much larger than `[B]`-only).
3. Oracle generation infrastructure (perfect-foresight lookups across the full training dataset).
4. A loss path that uses MSE-against-oracle, NOT the existing Bellman target — but writes to the *same* Q head weights.
5. A protocol for switching the Q head from offline to online (cf. Lee et al. 2022 on "Offline-to-online RL with implicit policy constraint").

**Scope reality check.** Foxhunt has no offline-RL infrastructure. The minimal-scope version of F4 (one epoch of MSE-against-oracle before RL) requires building:
- A standalone CLI binary or training stage
- A perfect-foresight oracle (compute realized return from each action × each state)
- A storage format for oracle Q values (`[N_states × N_actions]` MBP-10 trace, ~10⁸ entries for 24 months × ES → ~1.5GB)
- A loss function consuming this storage
- A protocol for switching from supervised → RL

The full version (dynamic-programming on every action) is computationally expensive (`O(T × N_actions²)` per dataset).

**Per the math agent's `pearl_foxhunt_trainer_is_genuinely_single_step` STOP-rule discipline: F4 is a multi-week refactor, not a bounded edit.**

**Does F4 fix the inaction-bias?** Yes, structurally and elegantly — the agent inherits a Q that has already discovered when each action is optimal. The Q-distill target then naturally puts mass on the correct action. **This is the most principled fix.** It also addresses several adjacent problems (train→eval generalization via supervised regularization, sample efficiency, exploration in cold-start).

**Why it's wrong to dispatch this now.** Foxhunt's investigation methodology (`feedback_investigation_first_falsification_methodology`) requires bounded scope per dispatch. F4 is unbounded. The prior 64-commit anti-pattern is exactly the trap of dispatching ambitious fixes without falsifiable gates. F4 must be staged: first build offline-RL infrastructure (a separate spec); then evaluate F4 within that infrastructure (another spec); only then deploy.

**LOC estimate.** Large. ~2000–5000 LOC of new infrastructure plus existing-code refactor.

**Predetermined falsification gates.** Cannot be defined yet — the infrastructure to evaluate F4 does not exist. Specifying gates pre-infrastructure would be vacuous.

**Risk vs reward.** High reward (most principled fix; addresses multiple adjacent problems). Very high implementation cost. **OUT OF SCOPE for this dispatch.** Worth a separate "F4 infrastructure" spec once F1 or F2 are deployed.

---

### §3.5 F5 — State-conditional action availability mask (CMDP shielding)

**Mechanism.** Insert a hard-mask kernel between the policy logits emission and the softmax sampling, structurally identical to the existing `rl_band_mask.cu` (Phase 4-A) but with broader scope. The mask is keyed on `position_lots` (read from the existing `pos_state` buffer at `pos_state + b·pos_bytes + 0..4`) and on the per-batch per-unit trail-stop distance vs the trail-max cap.

The mask sets logits for unavailable actions to `−∞` (or a large negative sentinel ≤ −10⁶) BEFORE the softmax — equivalent to `softmax(z_a) = 0` for masked actions and uniform redistribution to available ones. The mask is applied identically to (i) the action sampled / `argmax` from `π_θ` and (ii) the `pi_target` distribution Q-distill consumes — both must see the same `A(s)`, otherwise the target-vs-output gradient is internally inconsistent.

Pseudocode (matches `rl_band_mask.cu` lattice — Grid=(B), single thread per block, ISV-gated):

```
extern "C" __global__ void rl_state_action_mask(
    float* logits,             // [B, N_ACTIONS] — IN/OUT, set to −∞ for unavailable
    const unsigned char* pos_state,
    const float* trail_distances,   // [B, max_units]  per-unit trail distances (ticks)
    const float* isv,
    int b_size, int pos_bytes
) {
    const int b = blockIdx.x;
    if (b >= b_size) return;

    // Master gate (default OFF; bootstrap = 0.0). Symmetry with band mask
    // slot 799 ensures reversibility.
    if (isv[RL_STATE_ACTION_MASK_ENABLED_INDEX] <= 0.5f) return;

    const int position_lots =
        *reinterpret_cast<const int*>(pos_state + b * pos_bytes);

    // Branch on position state.
    if (position_lots == 0) {
        // FLAT: agent MUST commit to a direction (or take an explicit
        // forced action — e.g., reset / force-flat sentinel if one
        // exists). Mask: {Hold, FlatFromLong, FlatFromShort, HalfFlatLong,
        // HalfFlatShort, TrailTighten, TrailLoosen, PyramidLong,
        // PyramidShort}. Available: {Open Long (any size), Open Short
        // (any size)}.
        mask_action(logits, b, ACTION_HOLD);
        mask_action(logits, b, ACTION_FLAT_FROM_LONG);
        mask_action(logits, b, ACTION_FLAT_FROM_SHORT);
        mask_action(logits, b, ACTION_HALF_FLAT_LONG);
        mask_action(logits, b, ACTION_HALF_FLAT_SHORT);
        mask_action(logits, b, ACTION_TRAIL_TIGHTEN);
        mask_action(logits, b, ACTION_TRAIL_LOOSEN);
        // Pyramid variants are sized opens from a non-flat base;
        // unavailable from flat by construction.
        mask_action(logits, b, ACTION_PYRAMID_LONG);
        mask_action(logits, b, ACTION_PYRAMID_SHORT);
    } else {
        // IN POSITION: no re-opens of a new directional commitment from
        // an already-committed state. Pyramid actions remain available
        // (they extend the existing position).
        if (position_lots > 0) {
            // Long: mask short opens, mask flat-from-short, mask
            // half-flat-from-short.
            mask_action(logits, b, ACTION_OPEN_SHORT_SMALL);
            mask_action(logits, b, ACTION_OPEN_SHORT_LARGE);
            mask_action(logits, b, ACTION_FLAT_FROM_SHORT);
            mask_action(logits, b, ACTION_HALF_FLAT_SHORT);
            mask_action(logits, b, ACTION_PYRAMID_SHORT);
        } else {
            // Short: symmetric.
            mask_action(logits, b, ACTION_OPEN_LONG_SMALL);
            mask_action(logits, b, ACTION_OPEN_LONG_LARGE);
            mask_action(logits, b, ACTION_FLAT_FROM_LONG);
            mask_action(logits, b, ACTION_HALF_FLAT_LONG);
            mask_action(logits, b, ACTION_PYRAMID_LONG);
        }

        // CRITICAL trail-at-cap branch (mq2pc-specific): if the per-unit
        // trail distance is at the cap, TrailLoosen is a mechanical
        // no-op — the kernel `rl_trail_mutate.cu` clamps and emits no
        // state change. Mask TrailLoosen in this branch.
        const float trail_max = isv[RL_TRAIL_MAX_INDEX];
        const float eps = 1.0e-3f;
        // Use max across units to catch the cap-saturated case for any
        // pyramid layer.
        float max_trail = 0.0f;
        for (int u = 0; u < MAX_PYRAMID_UNITS; ++u) {
            const float t = trail_distances[b * MAX_PYRAMID_UNITS + u];
            if (t > max_trail) max_trail = t;
        }
        if (max_trail >= trail_max - eps) {
            mask_action(logits, b, ACTION_TRAIL_LOOSEN);
        }
    }
}
```

The masked `logits` then feed both the policy softmax (sampling / argmax) and the Q-distill `pi_target` computation (which must apply the SAME mask before its softmax — see §3.5 "F5 + F2 composition" below). Action indices use the canonical `crate::rl::common::Action` enum to avoid hardcoded duplication.

**Theoretical foundation.**

1. **Constrained-MDP formulation.** Altman 1999 ("Constrained Markov Decision Processes", Chapman & Hall) formalises an MDP with state-dependent action set `A(s) ⊂ A`. The Bellman optimality equation becomes `V*(s) = max_{a ∈ A(s)} [r(s, a) + γ Σ_{s'} P(s'|s, a) V*(s')]`. The optimal policy is still well-defined; the action-availability function is a property of the environment, not the policy. **Foxhunt's environment already has implicit availability constraints** (e.g., the trail-stop kernel overrides actions on breach; band-mask forces Hold inside the band; `rl_trail_mutate.cu` clamps TrailLoosen at the cap). F5 makes those constraints explicit at the *policy* level rather than expressing them via post-hoc action overrides.

2. **Safe-RL / shielding.** Alshiekh et al. 2018 ("Safe Reinforcement Learning via Shielding", AAAI, arXiv:1708.08611). A shield is a reactive system that monitors the agent's chosen action and overrides it when it violates a specification. Foxhunt's current architecture *is* a shield system (band mask, trail-stop override, confidence gate, CMDP cooldown) but the shields run *after* the policy emits an action — the policy never sees the constraint, and Q-distill targets contain mass on actions that the environment will reject. Alshiekh §3 ("Preemptive vs Post-Posed Shielding") argues preemptive shielding (mask logits before sampling) is strictly more sample-efficient than post-hoc override because the agent's gradient signal contains the constraint information. F5 is the canonical preemptive-shielding refactor.

3. **AlphaZero-style legal-move masking.** Silver et al. 2017 ("Mastering the Game of Go without Human Knowledge", Nature, DOI:10.1038/nature24270) §"Methods → Self-play training pipeline": at every position the network's policy output is masked by the set of legal moves before softmax; illegal moves receive zero probability. The MCTS PUCT computation also respects the legal-move mask. This is the foundational design pattern for game RL with state-conditional action sets. Trading is a game with state-conditional legality: you cannot close a position you do not hold, you cannot widen a trail at its cap.

4. **Action elimination / pruning literature.** Zahavy et al. 2018 ("Learn What Not to Learn: Action Elimination and Stochastic Shielding in DQN", NeurIPS, arXiv:1809.02121) — pruning provably-suboptimal actions improves Q-learning sample complexity by reducing the effective `|A|` per state. F5 is the structural prior version of Zahavy 2018; foxhunt knows by construction which actions are state-incoherent without needing to learn the action-elimination classifier.

**Why F5 attacks the inaction-bias at a different point than F1-F4.**

Equation (c) in §2.1 says:

```
E[Q_target(s, a_c) − Q_target(s, a_0)] ≈ −entry_cost  < 0
```

F5 does not modify either side of (c). Instead F5 modifies *which actions are in the max-over-actions Q evaluation in the next-step bootstrap*. When the agent is flat and Hold is masked, the policy cannot select Hold; the Q-distill softmax target re-normalises over remaining actions; the policy never observes a flat→Hold trajectory in its on-policy data and the Bellman target's `V(s')` for the flat state becomes the max-Q over `A(s_flat) = {OpenLong, OpenShort}` instead of `A = {Hold, OpenLong, OpenShort, ...}`.

Critically: F5's effect on equation (c) is to make the comparison vacuous for unavailable actions — there is no `a_0 ∈ A(s)` for the flat state, so the asymmetry has no choice-set on which to manifest. This is structurally different from F1 (modifies reward), F2 (modifies distill target shape), F3 (removes from global enum), and F4 (warm-starts Q).

**F5 + F2 composition (load-bearing).** F5 alone leaves the in-position attractor (Hold + TrailLoosen-when-not-at-cap) intact: when the agent is long and the trail is not yet at the cap, Hold and TrailLoosen are both *available* and both *zero-cost*. F5 has shrunk `A(s)` but the residual `A(s)` still contains a zero-cost subset whose Q advantages over costly closes match equation (c). F2 (Q-distill advantage target) then operates *on the masked `A(s)`* — the `baseline(s)` is computed only over `a ∈ A(s) ∩ no-op-set`, and the distill target's `max(0, A)` hinge zeroes mass for any action whose Q does not exceed that masked baseline. Composition order:

```
Step 1 (F5):  raw_logits → masked_logits (unavailable actions → −∞)
Step 2:       masked_logits → π_θ (softmax over A(s))
Step 3 (Q):   E_Q(s, ·) computed over all actions, but only a ∈ A(s) used in distill target
Step 4 (F2):  baseline(s) = max_{a ∈ A(s) ∩ no-op-set} E_Q(s, a)
              π_target(s, a) ∝ softmax( max(0, E_Q(s, a) − baseline(s)) / τ ) · 1[a ∈ A(s)]
Step 5:       distill grad = λ · (π_θ − π_target)
```

The two interventions are mechanically commutative in the sense that they touch disjoint state in the pipeline: F5 changes the *support* of the policy distribution, F2 changes the *gradient on the support*. F5 cannot break F2 (F2 operates on whatever support F5 leaves). F2 cannot break F5 (F2 does not introduce or remove mask entries).

**F5 violates the surfer trading principle (load-bearing trade-off, be honest about it).** The surfer alpha mode (`pearl_dd049d9a4_surfer_baseline_verified`: wr=0.56, hold=30, +$10.6M @ step 18k) is defined operationally by long, patient holds with high win-rate — the agent waits for high-confidence setups and only commits when it predicts edge. The structural ability that enables this is *Hold-from-flat as a valid policy choice*: the agent abstains until its encoder signals an edge.

F5's flat-state mask removes `{Hold, all trail variants, all close/half-flat variants, all pyramid variants}` from `A(flat)`, leaving `A(flat) = {OpenLongSmall, OpenLongLarge, OpenShortSmall, OpenShortLarge}`. Every step the agent is flat, it is *forced* to open a position regardless of whether its encoder predicts edge. This:

- **Destroys patience-while-waiting-for-setup** — the foundational surfer behavior — by construction. The agent cannot abstain from low-confidence setups; it can only choose between Long and Short.
- **Forces random / low-confidence opens** → win-rate drops below the surfer's 0.56 → the policy regresses toward the trend-follower profile (wr=0.36, hold=17 per `pearl_two_alpha_modes_surfer_vs_trend`).
- **Removes agent discretion to wait**, even when the encoder produces low-confidence Q-signal across all opens. The agent has no expressive way to communicate "no edge detected here."
- **Cannot be untrained.** Unlike F1/F2 (which the agent can *learn around* if it discovers patience pays off), F5 is a hard environmental constraint — it eliminates the patience strategy from the policy support entirely.

This is a *real* trade-off, not a bug. F5 may still be valuable as a fallback because:

1. The h6btt all-Hold collapse is unambiguously worse than even the most aggressive trend-follower regression (h6btt bled while taking no trades; the trend-follower at least makes money).
2. F2 may fail to break the h6btt-class flat-state collapse if the encoder's Q-signal is too weak to make any Open's `E_Q` exceed `E_Q(Hold)` — in which case F2 degenerates to uniform random and F5's forced commit is strictly better than uniform random.
3. The user's stated philosophical preference is surfer-style trading, but the surfer baseline's own eval pnl was negative (`pearl_surfer_baseline_was_train_only_never_eval`: −$13.5M wr=0.219 in eval) — preserving surfer mode does not guarantee profitability either.

The honest position: F5 is the structural fallback if F2's gradient redirection cannot escape the encoder-weakness scenario, and we choose to accept surfer→trend regression in exchange for breaking the all-Hold attractor. The staging in §4 puts F2 first so we test the surfer-preserving option before accepting the surfer-sacrificing fallback. See §4.1.4 for the full surfer-principle decision criterion.

**Expected effect on inaction-bias.**

- **From flat (h6btt-class):** F5 directly removes the choice. The agent cannot collapse to all-Hold because Hold is masked from flat. The action distribution from flat states is forced to be `{OpenLong, OpenShort}` with shares determined by the policy's directional preference. This is the strongest possible structural fix for the h6btt regime: not a gradient nudge but a hard constraint. Expected: Hold% from flat-only states → 0%.

- **From in-position (mq2pc-class):** F5 partially helps. TrailLoosen-at-cap is masked, breaking the specific mq2pc oscillation. But TrailLoosen-not-at-cap and Hold-while-in-position remain available and zero-cost. The residual attractor must be addressed by F2's gradient mechanism.

- **Combined F5 + F2:** F5 eliminates the flat-state attractor structurally; F2 eliminates the in-position attractor via gradient. The composition is the minimal disciplined set covering both regimes.

**Expected failure modes.**

1. **Forced loss-making trades in zero-edge regimes.** If the market is in a regime with no exploitable alpha, F5 forces the agent to open trades anyway (Hold is unavailable). The agent's only choice from flat is `{OpenLong, OpenShort}`, and a random pick has E[pnl] ≈ −commission. In adversarial regimes (high noise, low edge), F5 could *worsen* eval pnl vs the do-nothing baseline. **Mitigation possibilities (NOT specified at this stage):** add a "synthetic Hold" via an extremely-small-size open that the band mask immediately routes to Hold; OR allow Hold from flat only when a confidence gate condition is met. Both are out of scope for the v1 F5 design — see §6.

2. **Breaks encoder-learned pseudoinaction patterns.** The encoder has been training with the current action distribution. If it has learned to detect specific pseudoinaction signals (e.g., "the agent will Hold when feature X is high"), masking Hold from flat invalidates those learned features. The encoder needs to relearn — gradient should flow but the transient could be lossy. Estimated training-step recovery: 500-2000 steps based on observed reaction-times in prior controller-floor refactors.

3. **F5 mask + Q-distill target inconsistency.** If the mask is applied to the policy logits but NOT to the `pi_target` softmax input, the distill gradient is misaligned: π_θ has zero mass on masked actions (after the mask) while `pi_target` has non-zero mass (because the unmasked Q values include the masked action's Q). The gradient on the policy weights for the masked action would be `λ · (0 − π_target(a)) = −λ π_target(a)`, which pushes the policy weights toward producing non-zero logits on the masked action — wasted gradient, and a potential numerical instability if the masked logits are pushed back up enough that the `−∞` mask becomes a sentinel comparison rather than a soft constraint. **The mask MUST be applied to both `π_θ` and `pi_target`. The kernel diff must propagate the masked action set into `rl_q_pi_distill_grad.cu`.**

4. **Trail-at-cap detection edge cases.** Floating-point comparison `max_trail >= trail_max - eps` is sensitive to the exact value of `eps`. If `eps` is too tight, the mask fails to engage at the cap (the very mq2pc pathology persists). If `eps` is too loose, the mask engages near-cap, removing TrailLoosen when there is still some action-level effect. The MQ2PC trace shows TrailLoosen reaching the cap and saturating; a generous `eps ≈ 1e-2 · trail_max` is the safest first cut. Tunable via ISV.

5. **Interaction with confidence gate.** Per `pearl_confidence_gate_pos_state_unwired` the confidence gate already declares `pos_state` as a parameter but does not read it. F5 is conceptually similar — both gate actions based on position state — but operates earlier in the pipeline (on logits, before softmax). The two should compose: F5 mask first, then policy emits an action over the surviving support, then confidence gate post-overrides if confidence is low. The confidence-gate Case 3 failure documented in that pearl is NOT introduced by F5 and is orthogonal; F5's correctness does not depend on the confidence gate being correctly position-aware.

6. **Phase 4-A band mask interaction.** The Phase 4-A band mask forces `actions[b] = Hold` when in-position-and-inside-band. F5 from in-position does NOT mask Hold. So the composition is: F5 leaves Hold available from in-position → policy may emit OpenLong / etc. → band mask post-overrides to Hold if inside band. Consistent ordering. Verify in implementation plan.

7. **Single-direction over-commitment.** With Hold removed from flat, the agent's choice is binary `{Long, Short}`. If the encoder has a directional bias (even small), the policy could collapse to one direction (e.g., 95% Long). F2's `max(0, A)` then only retains mass on the directionally-biased side — and the agent has no expressive way to abstain. Detection: action distribution in F5-G5 / F5-G9 detects > 80% single-action share. Mitigation possibilities (also out of scope for v1): a "synthetic abstain" via a band-narrow-open, or relaxing the flat-Hold mask under high-uncertainty regimes.

**LOC estimate.** Small-to-medium. ~50-100 LOC total:

- New kernel `rl_state_action_mask.cu` (modeled after `rl_band_mask.cu`'s 91-line lattice): ~60-80 LOC including the trail-at-cap detection branch.
- Trainer-side dispatch (one `cuLaunchKernel` invocation between policy logits emission and softmax sampling): ~10-15 LOC in `crates/ml-alpha/src/trainer/integrated.rs`.
- Identical mask propagation into `rl_q_pi_distill_grad.cu` for `pi_target` computation: ~20-30 LOC (read the same mask buffer or recompute identically from the same `pos_state`).
- New ISV slots: `RL_STATE_ACTION_MASK_ENABLED_INDEX` (binary master gate, bootstrap 0.0); `RL_STATE_ACTION_MASK_TRAIL_EPS_INDEX` (the cap-tolerance). Two new slots.
- Diag fields: per-batch fraction-masked-from-flat, fraction-masked-trail-at-cap (cf. `pearl_fleet_fraction_not_aggregate`). Wire into existing diag emission.
- Test fixtures: extend `crates/ml-alpha/tests/trade_management_kernels.rs` with mask correctness assertions analogous to the band-mask tests already there.

Total estimated diff: ~150 LOC across 4 files, comparable to the band-mask Phase 4-A foundation commit `e41a73208`.

**Pearl references (load-bearing wiring breadcrumbs).**

- `pearl_confidence_gate_pos_state_unwired` — `rl_confidence_gate.cu` already accepts `pos_state` and `pos_bytes` as kernel parameters but does not read them. The plumbing exists upstream (trainer-side dispatch passes `pos_state` correctly); only the kernel-side read is missing. F5 inherits the same plumbing pattern.
- `pearl_dd049d9a4_surfer_baseline_verified` — the cluster-proven surfer baseline at `dd049d9a4` ran *without* any state-conditional action mask. So F5 is not "restoring a removed surfer feature"; it is "adding a new structural constraint that the surfer baseline never had." The risk that F5 deviates from the surfer regime is genuine — see falsification gate G2.5 below.
- `pearl_surfer_baseline_was_train_only_never_eval` — the surfer baseline's eval pnl was −$13.5M; train profitability is not eval profitability. F5's outcome gates must be evaluated against eval pnl, not train pnl.
- Task list entries 49, 65, 67-68 (Phase 2.3 prior state-mandatory action mask) — earlier work attempted a state-mandatory action mask that evolved into a SOFT mask. F5 proposes returning to a HARD mask but with broader scope and explicit `−∞`-logit semantics. The prior soft mask's lessons (which actions were too constraining, how the policy migrated) are valuable input for the implementation plan but are not load-bearing for the design decision here.
- `pearl_band_mask` (the existing `rl_band_mask.cu` Phase 4-A architecture) — F5 is the band mask's structural twin: same Grid=(B), single-thread per block, ISV-gated, reads `pos_state`, default OFF for bit-equality preservation. Implementation should be parallel.
- `feedback_wire_everything_up` — the F5 mask MUST propagate into the Q-distill `pi_target` computation, not just the policy sampling path. See expected failure mode #3.

**Predetermined falsification gates.**

These gates are designed to distinguish F5's effect from F2's effect — see §5 for the canonical combined gate set. **NOTE (Revision 3):** the §5.2 Phase-7b canonical gate IDs renumber some of the F5 outcome gates (F5-G7 here = F5-G8 in §5.2; F5-G8 determinism here = part of F5-G6 in §5.2). Where the two disagree, §5.2 is authoritative. The legacy table below remains as the F5-only design summary from Revision 2:

| Gate | Local b=128 (Tier 1.5) | Cluster b=1024 (Tier 2) | What it detects |
|---|---|---|---|
| F5-G1 (mask correctness — flat) | Among batches where `position_lots == 0`, action_hist mass on `{Hold, TrailLoosen, TrailTighten, FlatFromLong, FlatFromShort, HalfFlatLong, HalfFlatShort, PyramidLong, PyramidShort}` is exactly 0 | Same | Verifies the hard mask works mechanically; ZERO mass, not "low" mass |
| F5-G2 (mask correctness — in-position) | Among batches where `position_lots > 0`, mass on `{OpenShortSmall, OpenShortLarge, FlatFromShort, HalfFlatShort, PyramidShort}` is exactly 0; symmetric for short | Same | Position-direction coherence |
| F5-G3 (mask correctness — trail at cap) | Among batches where any per-unit trail = trail_max, mass on TrailLoosen is exactly 0 | Same | mq2pc-specific mechanism gate |
| F5-G4 (flat → in-position transition rate) | Fraction of steps with `position_lots = 0 → position_lots ≠ 0` transition ≥ 5× the `mq2pc` baseline (mq2pc had ~4 trades / 2000 steps from flat) | Total opens per 1000 steps ≥ 50 | Direct positive evidence of choice-set forcing |
| F5-G5 (no single-direction collapse) | If F5-G4 passes, no single direction has > 80% of opens (else the agent has merely migrated to a one-sided over-commit attractor) | Same | Detects expected failure mode #7 |
| F5-G6 (training stability) | No NaN; l_pi, l_v, l_q magnitudes within 10× of `mq2pc` baseline | l_pi etc. stable through step 20k | Standard substrate gate |
| F5-G7 (eval pnl) | (not measured at Tier 1.5) | eval_pnl ≥ h6btt baseline + $5M / session | Outcome gate |
| F5-G8 (determinism preserved) | `determinism-check.sh --quick` exits 0 at seeds 42/43/44 | n/a | Substrate per `pearl_determinism_achieved` |

**Risk vs reward.** Medium reward (directly addresses the choice-set problem; structurally guarantees no flat→Hold collapse; reuses proven band-mask architecture) BUT explicit surfer-principle violation reduces post-Revision-3 score to 28 (vs F2's 30). Low-to-medium implementation risk (small kernel diff; reversible via ISV master gate; verified failure modes have explicit detection gates). Specific outstanding risks: forcing trades in zero-edge regimes (failure mode #1); surfer → trend regression by construction (load-bearing trade-off above). Net (per Revision 3): **valid as a fallback intervention on the choice-set axis when F2's gradient redirection fails from encoder-weakness. Staged as Phase 7b conditional on F2 outcome gate failure AND explicit user acceptance of the surfer→trend trade-off.**

---

## §4. Comparative recommendation

### §4.1 Score matrix

Scoring scale: 1 (low / poor) to 5 (high / excellent). Cost is inverted (1 = high cost = bad). F4's total rests on infrastructure that does not exist; treat as a fork.

Criterion 7 (**"Respects surfer trading principle"**) was added in Revision 3. It asks: does the intervention preserve the agent's discretion to wait patiently for high-confidence setups (the surfer alpha mode per `pearl_dd049d9a4_surfer_baseline_verified`), or does it force structural action regardless of belief? See §4.1.4 for the criterion's full definition.

| Criterion | F1 (inaction cost) | F2 (Q-distill advantage target) | F3 (action reduction) | F4 (oracle pretraining) | F5 (state-conditional avail. mask) |
|---|:-:|:-:|:-:|:-:|:-:|
| Theoretical soundness | 3 (non-potential per Ng-Harada-Russell) | 4 (Wang 2016 advantage decomposition) | 2 (no specific theorem) | 5 (offline RL canonical) | 5 (Altman 1999 CMDP + Alshiekh 2018 shielding + AlphaZero legality) |
| Addresses root cause | 4 (corrects reward asymmetry directly) | 5 (removes asymmetry at distill stage; mechanism-level fix) | 2 (reduces but does not eliminate) | 5 (warm-starts Q correctly) | 4 (eliminates flat-state choice; partial for in-position) |
| Implementation cost | 2 (medium; controller risk) | 1 (small; well-bounded kernel diff) | 4 (large; touches enum + all action kernels) | 5 (very large; new infra) | 2 (small; band-mask twin pattern; new kernel + Q-distill mask propagation) |
| Falsifiability | 4 (clear local + cluster gates) | 5 (mechanism gates + outcome gates) | 3 (only outcome gates) | 1 (gates require non-existent infra) | 5 (mask correctness directly observable in action_hist; ZERO-mass gates) |
| Foundation impact (resists going-in-circles) | 4 (changes the data) | 5 (changes the gradient) | 1 (just removes options) | 5 (changes the initial conditions) | 5 (changes the choice-set; structurally removes the attractor) |
| Reversibility (ISV gate, no code-rip) | 5 (ISV gate) | 5 (ISV gate) | 1 (enum change is one-way) | 3 (separate offline phase) | 5 (ISV master gate, bootstrap OFF) |
| **Respects surfer trading principle** (Rev 3) | 2 (adds friction to inaction; per-step holding cost discourages patience but does not force action) | 5 (provably preserves principle via hinge: zero Open mass when E_Q(Open) ≤ E_Q(Hold)) | 1 (removes Hold globally; total violation) | 5 (oracle targets reflect optimal patient behavior; preserves principle) | 2 (forces opens when flat; structural violation of patience-while-waiting) |
| **Total (higher = better)** | **24** | **30** | **14** | **29 (gated on F4-infra)** | **28** |

**Verdict shift from Revision 2.** Pre-criterion-7 ranking was F5 (26) > F2 (25) > F4 (24, gated) > F1 (22) > F3 (13). Post-criterion-7 ranking is **F2 (30) > F4 (29, gated) > F5 (28) > F1 (24) > F3 (14)**. F2 now leads. F5's lead disappears because F5 sacrifices the surfer principle that F2 preserves by mathematical construction. F4 remains gated on infrastructure that does not exist.

### §4.1.4 Surfer trading principle as a decision criterion (Revision 3 addition)

The surfer principle is load-bearing for foxhunt's stated trading philosophy; this section makes it explicit so future revisions cannot accidentally violate it without acknowledging the trade-off.

**What the surfer alpha mode is.** Per `pearl_dd049d9a4_surfer_baseline_verified` (cluster-proven at SHA `dd049d9a4`, b=1024, May 2026):

| Metric | Surfer baseline value |
|---|---|
| Stable win-rate (wr_ema) | 0.56 |
| Average hold duration | 30 steps |
| Cluster pnl at step 18k | +$10.6M |
| Clip rate ema | 1.0 (saturated; normal for this regime) |
| C51 v_max | 1.0 (frozen) |
| Q-distill KL ema | ≈ 0 (Q and π agree) |

The trend-follower counter-mode (`6d33f18b7`, per `pearl_two_alpha_modes_surfer_vs_trend`): wr ≈ 0.36, hold ≈ 17 steps, +$41.5M peak. The data supports both modes; the architecture supports both modes; the user has explicitly chosen the surfer per `feedback_surfer_philosophy_trading`.

**Why the surfer requires patience-while-waiting-for-setup as a structural ability.** The surfer's high win-rate (0.56 vs 0.36) is the alpha signal. It comes from the agent being *selective* about entries — only committing when its encoder signals an edge. The behavioral expression of this selectivity is *long stretches of Hold-from-flat* while the agent observes the market and waits for a setup. If the agent is forced to open positions on every flat-state step, it cannot be selective; its entries become random or low-confidence; its win-rate collapses toward the random baseline.

The surfer principle, stated precisely: **act when your model believes acting beats waiting; wait otherwise**. The agent's patience-when-no-edge IS the alpha signal. Any intervention that breaks this property breaks the surfer mode by construction.

**How F5's mask structurally prevents this patience.** F5 sets `A(flat) = {OpenLongSmall, OpenLongLarge, OpenShortSmall, OpenShortLarge}` — Hold is masked out. The policy can sample only from `{Long, Short}` × `{Small, Large}`. The agent has no expressive way to communicate "no edge detected here; wait." Every flat-state turn produces an open regardless of encoder signal. The surfer's wr=0.56 cannot survive this — the new wr will reflect random-pick-when-no-edge, pulling it toward 0.5 minus commission drag, plus whatever genuine edge moments the encoder still detects when forced commits land on them. Net effect: the agent migrates toward the trend-follower profile (higher activity, lower win-rate).

This is not a *bug* in F5 — it is F5 doing exactly what it was designed to do. F5's purpose is to eliminate the h6btt-class flat-state Hold attractor by making the attractor mechanically impossible. The cost of this design is the surfer principle.

**How F2's gradient redirection preserves this patience.** F2 sets the Q-distill target to `π_target(a) ∝ softmax(max(0, E_Q(a) − baseline) / τ)` where `baseline = max over a ∈ no-op-set { E_Q(a) }`. The Hold-from-flat case:

- If the encoder produces `E_Q(Open) ≤ E_Q(Hold)` (no edge detected at this state): `max(0, E_Q(Open) − E_Q(Hold)) = 0`. Open's distill target mass is zero. The gradient `λ · (π_θ(Open) − 0)` pushes π_θ(Open) toward zero, leaving Hold as the surviving high-Q action. The agent learns to wait. **The surfer principle holds.**
- If the encoder produces `E_Q(Open) > E_Q(Hold)` (edge detected): `max(0, E_Q(Open) − E_Q(Hold)) > 0`. Open's distill target mass is positive. The gradient pushes π_θ(Open) toward this target. The agent learns to commit on detected edge. **The surfer's patience-selective entries property holds.**

The key contrast: F5 makes the wait/act choice for the agent (and always picks "act"); F2 lets the agent *learn* the wait/act choice from the encoder's edge signal. F2 is the surfer principle expressed as a loss function.

**Caveat.** F2 only preserves the surfer principle as long as the encoder produces an `E_Q` landscape where some actions exceed the no-op baseline at edge-bearing states and no actions exceed it at no-edge states. If the encoder is too noisy or too weak, F2 degenerates to uniform random (the "mass-zero distill target" failure mode in §3.2). In that scenario the surfer principle is moot because there is no edge signal to be selective about. F5 then becomes a meaningful fallback: forced commits at the random-baseline floor are still better than the do-nothing baseline of h6btt's all-Hold collapse, even at the cost of the surfer mode.

**Citation.** Charpentier, Elie, and Remlinger 2020 ("Reinforcement Learning in Economics and Finance", arXiv:2003.10014), §"Equity trading via RL", document the surfer-vs-trend dichotomy as canonical in financial RL — the same two operating regimes appear across multiple RL trading systems, distinguished by win-rate vs frequency trade-off. The surfer principle (patient selective entries) is the empirically-validated higher-Sharpe regime in much of the literature; the trend-follower is the higher-absolute-return / higher-drawdown counterpart. Foxhunt's cluster bisect at `dd049d9a4` and `6d33f18b7` reproduces this dichotomy.

**Decision rule for this spec.** Recommend the intervention that *preserves* the surfer principle as Phase 7a. Stage the *principle-sacrificing* intervention only as a conditional Phase 7b fallback, and explicitly acknowledge the trade-off in the gate evaluation. This is the discipline encoded in the revised §4.2 staging recommendation below.

### §4.1.5 Combined F2 + F5 analysis (revised ordering per Revision 3)

F2 and F5 attack *mechanically disjoint* aspects of the inaction attractor:

- **F2 attacks the gradient** (no-op actions have zero advantage gradient flowing into the policy when `E_Q(no-op) ≥ max(other E_Q)`). Soft pressure that depends on Q quality but PRESERVES the surfer principle by construction (agent retains learned discretion to wait when no edge is detected).
- **F5 attacks the choice-set** (when flat, agent CANNOT select Hold; when at trail cap, agent CANNOT select TrailLoosen). Hard constraint on `A(s)`. Independent of gradient signal — works even if Q is uninformative — but VIOLATES the surfer principle (forces commits regardless of belief).

Per `feedback_going_in_circles_pattern`, multi-fix stacking is suspect. But F2 and F5 are NOT "two feels-right fixes stacked" — they are interventions on disjoint pipeline stages with independent falsification gates. F2's gates (F2-G1, F2-G2) verify mechanism correctness on the gradient distribution; F5's gates (F5-G1, F5-G2, F5-G3) verify mechanism correctness on the action support. Post-cluster attribution is possible because:

- If F2 gates pass and eval pnl is positive (Phase 7a) → F2 alone carried the run, ship F2 alone, surfer principle preserved.
- If F2 gates pass and eval pnl is negative (Phase 7a) → F2 mechanism is operational but insufficient; evaluate per §4.4. If user accepts surfer→trend trade-off, escalate to Phase 7b (F5 added).
- If F2 + F5 both pass mechanism gates and eval pnl is positive (Phase 7b) → both interventions necessary; ship both with explicit trade-off documented.
- If F2 + F5 both pass mechanism gates and eval pnl is negative (Phase 7b) → both mechanisms operational but insufficient; escalate to F4 or alignment work per §5.3.
- If F2 gates fail (mass-zero target) → fix the kernel (warmup / temperature schedule); this is not an architectural verdict.

This is the criterion `feedback_investigation_first_falsification_methodology` requires: each intervention has its own falsifiable mechanism gate, and the outcome gate is shared but post-hoc attribution is possible.

**Marginal benefit analysis:**

- *Marginal benefit of F5 given F2:* F2 alone redirects gradient away from no-op actions whose E_Q does not exceed `baseline = E_Q(no-op)`. If the encoder produces strong enough edge signal that opens' E_Q exceeds Hold's E_Q at edge-bearing states, F2 alone solves the inaction attractor *while preserving the surfer principle*. F5 then provides no additional benefit and would actively damage the surfer mode. **If the encoder is producing real alpha (the assumption behind the user's surfer preference), F5 adds nothing and subtracts patience.** If the encoder is weak, F2 may fail to escape the attractor and F5's forced commits become a fallback. F5's marginal value is conditional on F2-failure-from-encoder-weakness.

- *Marginal benefit of F2 given F5:* F5 alone eliminates the flat-state Hold attractor but leaves the in-position attractor (Hold-while-in-position-not-in-band; TrailLoosen-not-at-cap) intact. F2 redirects gradient away from these. In the F5-first staging this was the load-bearing reason for F2 conditional. In the revised F2-first staging, F2's in-position effect is captured in Phase 7a and the question becomes whether F5's flat-state mask adds residual benefit.

- *Scenario where F2 alone is sufficient:* the encoder produces alpha signal strong enough that E_Q(Open) > E_Q(Hold) at edge-bearing flat states. F2's hinge then pushes π toward Open at those states (action when edge detected) and toward Hold elsewhere (wait when no edge). The agent learns the surfer policy by gradient. h6btt's all-Hold collapse cannot persist because the flat→edge-detected→Open transition gets positive gradient signal; mq2pc's TrailLoosen-collapse cannot persist because TrailLoosen's E_Q does not exceed E_Q(other in-position no-ops) at edge-bearing states. This is the optimistic case and is also the *correct* outcome for the surfer philosophy.

- *Scenario where F2 alone is insufficient and F5 fallback is needed:* the encoder is too weak. E_Q(Open) ≤ E_Q(Hold) everywhere, including at edge-bearing states. F2's hinge produces a mass-zero target for Open; the policy reverts to SAC entropy-driven uniform with Hold dominant by the prior. The system collapses back into h6btt regime. Then F5's hard mask becomes meaningful: forced commits at a random-baseline floor are strictly better than the do-nothing baseline. The cost is acceptance of surfer→trend regression.

- *Disciplined staging (revised).* Per `feedback_going_in_circles_pattern` and `feedback_investigation_first_falsification_methodology`, the most disciplined sequence in Revision 3 is:
  - **Phase 7a:** F2 alone. Cluster verdict. If F2-G8 (eval pnl) passes AND mechanism gates pass AND no degenerate attractor → STOP; F2 alone is sufficient and the surfer principle is preserved.
  - **Phase 7b (conditional):** If F2-G8 fails AND F2 mechanism gates pass AND we accept the surfer→trend trade-off, add F5 on top. New cluster verdict.

The previous Revision 2 staging (F5 first, F2 conditional) was reversed in Revision 3 because:
  - F2 preserves the surfer principle by mathematical construction; F5 violates it by structural constraint. The surfer principle is the user's stated philosophical preference.
  - F2 has higher post-criterion-7 score (30 vs 28).
  - F2's cluster verdict tells us whether the surfer mode can be reached at all from the current Phase 5 substrate. If yes, ship F2. If no, the question shifts to "do we accept the trade-off" rather than "is the choice-set the load-bearing problem."

This staging extracts the same information per cluster run as the previous order, but pays the surfer-principle preservation cost first and the surfer-principle-sacrificing fallback second. Going-in-circles discipline is preserved because each phase tests exactly one mechanism with its own gates.

### §4.2 Recommended option: **F2 (Phase 7a) staged with F5 (Phase 7b)** — surfer-principle-preserving gradient first, structural mask only as fallback

**Revised in Revision 3 — load-bearing reason.** F2 addresses the root cause via the *gradient axis* (Q-distill target shape) while preserving the surfer trading principle by mathematical construction (see §4.1.4). The hinge `max(0, E_Q(a) − baseline)` zeroes Open's distill mass when no edge is detected and zeroes Hold's contribution when other actions exceed it — the agent learns to wait or act based on the encoder's edge signal, not a structural constraint. F2 is the surfer principle expressed as a loss function.

F5 is staged as the conditional fallback because:

1. F5 *violates* the surfer principle by removing Hold-from-flat from `A(s)` entirely. The agent is forced to commit on every flat-state step regardless of belief, destroying patience-while-waiting-for-setup, which is the load-bearing structural property of the surfer alpha mode (`pearl_dd049d9a4_surfer_baseline_verified`).
2. F5 may still be valuable as a fallback if F2 fails from encoder-weakness — if the encoder's E_Q landscape cannot make any Open's Q exceed Hold's Q at edge-bearing states, F2 degenerates to uniform random and F5's forced commits become strictly better than the do-nothing baseline. But this is a conscious trade-off, not an oversight.
3. Per `feedback_investigation_first_falsification_methodology` §"sequential phases with each gate informing the next," the most information-extracting sequence is to test the surfer-preserving option first, then accept the surfer-sacrificing fallback only if necessary.

Intervention-point comparison:

```
F1 (reward kernel)  →  V regression  →  Q Bellman target  →  softmax target  →  π
   ^ corrective signal attenuates 4× through learned approximators (Goodhart-Skalse)

F2 (distill kernel) →  π
   ^ corrective signal is direct; one approximator step
   ^ PRESERVES surfer principle by construction (hinge zeroes mass on E_Q ≤ baseline)

F5 (policy logits mask)  →  π
   ^ NOT a corrective signal — a structural constraint on the support of π;
     does not attenuate; cannot be "untrained" by gradient
   ^ VIOLATES surfer principle by removing Hold from A(flat)

F2 + F5 (in fallback ordering): F2 reshapes the gradient first; F5 adds support constraint only if F2 alone insufficient.
```

**Why F2 first, F5 conditional rather than the Revision-2 order:**

- *Surfer principle preservation.* The user has explicitly chosen surfer-mode trading. F2 preserves this by construction; F5 violates this by construction. Test the preserving option first.
- *Information per cluster run.* If F2 alone improves eval pnl, we have proven both (i) gradient redirection was the load-bearing intervention and (ii) the encoder is strong enough to produce alpha signal that the surfer mode can exploit. Both are meaningful results.
- *Discipline against accidental philosophy shift.* Choosing F5 first commits the system to the trend-follower regime even before testing whether the surfer regime is reachable. Revision 2's order encoded an implicit "we don't believe the surfer is reachable" assumption that the user has explicitly rejected.
- *Falsifiability of the encoder-weakness hypothesis.* If F2 fails, we have indirect evidence that the encoder's alpha-extraction is the bottleneck (since F2's mechanism gates depend on E_Q(Open) > E_Q(Hold) at edge-bearing states). That evidence is more diagnostic than F5's outcome (which would only tell us "forced trading is better than no trading").
- *Reversibility.* F2 alone is one ISV master gate. F2+F5 is two. Smaller surface area is preferred for the foundational phase.

Comparison to non-recommended options:

- **F5 alone (Revision 2 recommendation):** mechanically eliminates the flat-state Hold attractor but at the cost of the surfer principle. Valid only as a fallback once F2 has been tested and failed.
- **F1 (inaction cost):** more direct on the data but more disruptive to the Q-distill controller chain (which is tuned for the current target distribution), introduces a known-suboptimal non-potential shaping term, and *also* violates the surfer principle by adding friction to waiting (criterion-7 score 2). F1 ranks below F4 even with the surfer criterion added.
- **F3 (action reduction):** regresses to the h6btt regime, removes documented architectural progress, is not reversible, and total surfer-principle violation (criterion-7 score 1).
- **F4 (oracle pretraining):** the right *eventual* answer (criterion-7 score 5 — oracle targets reflect optimal patient behavior) but requires infrastructure that does not exist.

### §4.3 Predicted outcomes confirming F2 (Phase 7a, 3 specific predictions)

These are the highest-priority predicted-outcome gates for the revised Phase 7a:

1. **Q-distill target entropy drops measurably (mechanism gate).** Slot 733 (`RL_Q_DISTILL_TARGET_ENTROPY_MEAN_INDEX`) drops by ≥ 0.5 nats within the first 1000 training steps after F2 is turned ON. This verifies the hinge is non-vacuous: the target distribution is now action-discriminating rather than near-uniform. *Decoupled from outcome — verifies the kernel does what the math says.*

2. **In-position action distribution shifts away from {Hold, TrailLoosen-not-at-cap} AND surfer-style flat-state patience is preserved (behavioral gate).** Two conjunct conditions must hold:
   - Conditional on `position_lots > 0` AND `max_unit_trail < cap`: mass on Hold + TrailLoosen falls below 60% by step 5000.
   - Conditional on `position_lots == 0`: Hold% does NOT collapse to either 0% (would indicate the agent is forcibly opening every step, surfer principle lost) or > 95% (would indicate h6btt-class re-collapse). Target band: Hold% ∈ [30%, 85%] from flat states at step 5000.

   The second condition is what makes F2's verdict surfer-aware: F2 must show *learned patience* (some Hold-from-flat, not zero) alongside *learned action* (Open mass at edge-bearing states).

3. **Eval pnl improves vs h6btt baseline by at least $5M / session (outcome gate).** This is the primary outcome verdict. F2 must demonstrate eval pnl ≥ h6btt baseline + $5M / session. If passed alongside conditions 1 and 2, we have strong evidence that the surfer mode is reachable from the current Phase 5 substrate with F2's gradient redirection alone, and F5 is not needed.

### §4.3.5 Predicted outcomes confirming F5 (Phase 7b, only run if Phase 7a's F2-G8 fails)

1. **Action distribution from flat states shifts to {OpenLong, OpenShort} only (mechanism gate).** Action_hist conditioned on `position_lots == 0` should have ZERO mass on `{Hold, TrailLoosen, TrailTighten, FlatFromLong, FlatFromShort, HalfFlatLong, HalfFlatShort, PyramidLong, PyramidShort}` and 100% mass on `{OpenLongSmall, OpenLongLarge, OpenShortSmall, OpenShortLarge}` by step 100. *Decoupled from outcome — verifies the kernel works.*

2. **Trade frequency increases dramatically (behavioral gate).** Per F5-G4, opens per 1000 steps ≥ 50. The agent is forced to commit; whether it commits *profitably* is the outcome question. **Acknowledged surfer-principle violation:** this gate is *expected* to show the agent abandoning patience-while-waiting; that is what F5 does by construction.

3. **Eval pnl improves vs F2-alone baseline by at least $5M / session.** F5 must demonstrate *marginal* benefit over F2 alone to justify accepting the surfer→trend regression. If F5 + F2 produces only marginal improvement over F2 alone, revert to F2 alone (preserve the surfer principle); if F5 + F2 produces substantial improvement, commit both with explicit acknowledgement of the trade-off.

### §4.4 Predicted outcomes refuting the staged intervention (i.e., we are still going in circles)

**Phase 7a F2-alone decision summary (4-outcome attribution table):**

| F2-alone cluster result | Next action | Interpretation |
|---|---|---|
| Eval pnl ≥ +$5M improvement over h6btt baseline AND F2-G5 (surfer patience) passes AND no degenerate attractor | **SHIP F2 ALONE.** F5 not needed. Surfer principle preserved. | Q-distill bias was the load-bearing problem; the agent recovered surfer-style patient entries on the gradient axis alone. |
| Eval pnl improves but a new degenerate attractor surfaces (single-direction collapse, open-then-flat oscillation) | STOP — do NOT add F5 reflexively. Consider F5 Phase 7b only with explicit user acceptance that F5 may worsen the directional collapse. Most likely escalation: encoder investigation or F4 infrastructure. | Gradient redirection alone is operational but surfaced a different pathology; forcing trades (F5) on top would likely amplify the migration. |
| Eval pnl marginal (≥ 0 but < +$5M), all mechanism gates including F2-G5 pass | Consider F5 Phase 7b conditional, with explicit user decision to sacrifice some patience for outcome lift. The trade-off is now small (some surfer patience for some additional eval pnl). | F2 partial; F5 may close the gap at the cost of some surfer-mode regression. |
| Eval pnl regressed below h6btt baseline OR F2-G5 collapsed (Hold% from flat < 5% or > 95%) | STOP, audit foundation. Do NOT add F5 (F5 cannot fix a surfer-patience collapse caused by F2's own hinge dynamics or by encoder saturation). Escalate to encoder-alpha signal investigation or F4 infrastructure. | F2 introduced a different pathology; the foundation needs investigation upstream of the Bellman target. |

**Detailed failure-mode predictions:**

- **F2 mechanism gates pass (target entropy drops, in-position no-op mass falls) but eval pnl stays flat at h6btt baseline.** Two interpretations: (a) the encoder is too weak to produce alpha signal that the surfer mode can exploit — F5 fallback is then the disciplined next step, with explicit surfer→trend trade-off; (b) the inaction attractor was a *symptom* of a deeper alignment issue, not the load-bearing problem — escalate to reward-pnl alignment work (`pearl_reward_signal_anti_aligned_with_pnl`) or F4. The discriminator is whether F5 then improves eval pnl: if yes, encoder-weakness was the bottleneck; if no, alignment-or-encoder is the deeper problem.

- **F2 mechanism gates pass, surfer-patience condition pass (Hold% in [30%, 85%]), but action distribution single-direction-dominated.** E.g., the policy commits to Long-only at edge-bearing states. This is a directional collapse, not an inaction attractor. F5 will not help (F5 forces direction commits, not direction diversity). Escalate to encoder investigation or F4.

- **F2 mechanism gates pass but F5 follow-up (Phase 7b) shows no marginal benefit.** F2 alone is sufficient on the inaction axis; F5's structural mask provides no additional eval pnl. Revert F5, ship F2 alone, surfer principle preserved.

- **F2 mass-zero target failure (failure mode in §3.2 expected mode #1).** During cold-start, the encoder's E_Q is too noisy and no action exceeds `baseline`. The target is uniform-zero; distill gradient pushes π toward uniform; SAC entropy dominates; agent random-walks. Detection: F2-G1 fails (target entropy does NOT drop) OR target entropy spikes above 1.5 nats. Mitigation in implementation plan: warmup phase with F2 OFF, then enable; OR temperature schedule on the hinge.

- **F2 + F5 fallback BOTH improve mechanism but eval pnl still flat.** Both interventions operational, no outcome. The inaction attractor was masked-but-not-the-problem. Escalate per §5.4 — reward-pnl alignment or encoder are the next investigation targets, NOT another Bellman-target reshape.

- **Eval pnl WORSENS in Phase 7b F5 vs Phase 7a F2.** F5's forced commits in a zero-edge data subset produce more loss than F2's learned patience. This is a *valid* finding — it would prove F5 is genuinely harmful in low-edge regimes. Revert F5, document the regime-dependence, consider a Phase 7c "F5 with edge gate" (out of scope for this spec).

- **Local Tier 1.5 shows the policy diverging — l_pi blows up or NaN.** Indicates a kernel bug (most likely in F2's hinge handling of empty-no-op-set when combined with the Phase 4-A band mask). Fix the kernel; if pathology persists, F2 does not compose with the current Phase 5 substrate; reconsider F1 or wait for F4 infrastructure.

### §4.5 What we should DO before Phase 7a (F2) cluster

1. Build F2a (static no-op set: Hold, TrailTighten, TrailLoosen) with an ISV master gate (slot TBD, default 0.0).
2. Verify the F2 kernel modification respects the Phase 4-A band-mask composition: when the band forces Hold post-policy, F2 must still see the *pre-band* action distribution for distill purposes (open uncertainty #2 in §7 — load-bearing).
3. Local Tier 1 (correctness, ~6 sec): confirm `pi_target` correctly applies `max(0, E_Q − baseline)` hinge; confirm no NaN; confirm bit-equality with master gate OFF.
4. Local Tier 1.5 (behavior, ~10 min): confirm F2-G1 (target entropy drop), F2-G2 (no-op mass collapse in target), F2-G3 (training stability), F2-G4 (in-position action redistribution), surfer-patience condition (Hold% from flat in [30%, 85%]) — and determinism per `determinism-check.sh --quick` exits 0 at seeds 42/43/44.
5. **If Tier 1.5 passes:** submit cluster Tier 2 (~70 min). If outcome F2-G8 (eval pnl ≥ h6btt + $5M) passes, COMMIT F2 ALONE, do NOT proceed to F5. The gradient axis was the load-bearing problem AND the surfer principle is preserved.
6. **If Tier 1.5 passes mechanism gates but cluster F2-G8 fails:** evaluate per §4.4 — most likely encoder-weakness scenario. If the user accepts the surfer→trend trade-off, proceed to Phase 7b (add F5 on top of F2). If not, escalate to encoder / alignment / F4 infrastructure work.
7. **If Tier 1.5 fails on F2-G1 (target entropy doesn't drop):** the hinge is vacuous (no action exceeds baseline). This is the mass-zero-target failure mode. Fix via warmup or temperature schedule per implementation plan.
8. **If Tier 1.5 passes F2-G1/G2 but Hold% from flat collapses to 0%** (forced opens): the hinge is also pushing Hold mass below the random-floor, contradicting the surfer-principle preservation argument in §4.1.4. Investigate before cluster — likely a baseline-set definition bug.

### §4.6 What we should DO before Phase 7b (F5 on top of F2), conditional on §4.5 step 6 AND explicit user acceptance of surfer→trend trade-off

**Phase 7b is gated on TWO conditions:** (a) F2 outcome gate failure, AND (b) explicit user decision to accept the surfer→trend regression in exchange for breaking the h6btt-class structural collapse. Phase 7b should NOT auto-trigger on F2 failure.

1. Build F5 kernel `rl_state_action_mask.cu` modeled on `rl_band_mask.cu`, with ISV master gate (slot TBD, default 0.0; separate from F2's gate).
2. Wire the mask into BOTH the policy-sampling path AND the Q-distill `pi_target` computation. Verify identical mask in test fixture. Verify F2's hinge respects F5's masked support: `baseline(s) = max_{a ∈ A(s) ∩ no-op-set} E_Q(s, a)` over the F5-masked set; empty-baseline edge case handled per uncertainty #7 in §7.
3. Local Tier 1 (correctness, ~6 sec): confirm mask kernel sets logits to `−∞` for unavailable actions; confirm ZERO mass post-softmax on unavailable actions; confirm F2 hinge handles F5-masked no-op set without NaN.
4. Local Tier 1.5 (behavior, ~10 min): confirm F5-G1 through F5-G6 + F5-G8 (determinism) plus all F2 mechanism gates still pass.
5. **If Tier 1.5 passes:** submit cluster Tier 2. If F5's outcome gate (F5-G8: eval pnl ≥ F2-alone + $5M) passes, commit F2 + F5 with explicit surfer→trend trade-off documented. If not, see §4.4.
6. **If Tier 1.5 fails on F5 mechanism gates:** fix the kernel; this is not an architectural verdict.
7. **If Tier 1.5 passes F5 mechanism gates but fails F5-G5 (single-direction collapse):** F5 has merely migrated the attractor; STOP and escalate to F4. F2 + F5 together will not help with directional collapse.

---

## §5. Falsification gates for F2 + F5 (recommended staged combination, surfer-principle ordering)

Gates are partitioned into three categories: **F2-specific** (verify the gradient redirection works, observable independent of F5), **F5-specific** (verify the mask works, observable independent of F2; only used in Phase 7b), and **Combined** (require both to be active; outcome-focused). Each F2-specific gate verifies a property F5 cannot affect; each F5-specific gate verifies a property F2 cannot affect. This is the disciplined-combined-intervention discipline required by `feedback_investigation_first_falsification_methodology`.

### §5.1 Phase 7a — F2-alone gates (the surfer-principle-preserving primary phase)

#### §5.1.1 Local b=128 Tier 1.5 — F2 mechanism

| Gate ID | Metric | Pass condition | Distinguishes |
|---|---|---|---|
| F2-G1 (target entropy effect) | Slot 733 `RL_Q_DISTILL_TARGET_ENTROPY_MEAN_INDEX` at step 1000 | Drops by ≥ 0.5 nats vs Phase 5 baseline | F2's hinge is non-vacuous |
| F2-G2 (no-op mass collapse in target) | Sum of `π_target[a]` for `a ∈ no-op-set` | Below 30% by step 1000 (across all states) | F2's `max(0, A)` zeroes no-op mass |
| F2-G3 (training stability) | l_pi, l_v, l_q | No NaN; ≤ 10× Phase 5 baseline | Standard substrate |
| F2-G4 (in-position action distribution) | When `position_lots > 0 AND max_unit_trail < cap`, sum of Hold + TrailLoosen | < 60% by step 2000 | F2 redirects in-position gradient |
| F2-G5 (surfer-patience preserved) | Hold% conditional on `position_lots == 0` at step 5000 | ∈ [30%, 85%] (NOT 0% — forced opens; NOT > 95% — h6btt collapse) | **Surfer principle preservation per §4.1.4** |
| F2-G6 (training stability, full-loss) | All loss magnitudes | No NaN through step 5000 | Standard substrate |
| F2-G7 (determinism) | `determinism-check.sh --quick` exits 0 | seeds 42/43/44 bit-equal | `pearl_determinism_achieved` |

**Note on F2-G5 (load-bearing).** This is the critical surfer-principle preservation gate added in Revision 3. F2's mathematical argument is that the hinge zeroes Open mass when `E_Q(Open) ≤ E_Q(Hold)`, so the agent should retain *some* Hold mass at no-edge flat states. If Hold% from flat collapses to ≈ 0%, the hinge is forcing opens (contradicting the §4.1.4 argument) — possible causes: baseline-set definition bug, SAC entropy overwhelming the hinge, or the encoder's E_Q is so saturated that every Open exceeds Hold at every flat state. Investigate before cluster. If Hold% from flat is > 95%, the hinge is not breaking the h6btt-class attractor — distill gradient is insufficient to compete with the SAC prior; F5 fallback is the disciplined next step (Phase 7b).

#### §5.1.2 Cluster b=1024 Tier 2 — F2 outcome

| Gate ID | Metric | Pass condition | Distinguishes |
|---|---|---|---|
| F2-G8 (eval pnl primary) | `eval_summary.total_pnl_usd` | ≥ h6btt baseline + $5M/session | F2 alone is sufficient → STOP, do not run F5 |
| F2-G9 (no new attractor in cluster) | Max single-action share at eval step 5000 + step 15000 | < 80% | Detects mq2pc-style migration to a new degenerate |
| F2-G10 (eval action entropy) | `eval_summary.eval_action_entropy_mean` | ≥ 1.0 nats | Confirms diversity preserved |
| F2-G11 (surfer-mode fingerprint in eval) | eval `wr_ema` AND `avg_hold_steps` | wr_ema ≥ 0.45 AND avg_hold_steps ≥ 15 | Positive evidence agent recovered surfer-style patient entries |

**Decision rule:** if F2-G8 passes → commit F2 alone; F5 is not needed; the gradient axis was the load-bearing problem AND the surfer principle is preserved. If F2-G8 fails AND F2-G1/G2/G3 pass (mechanism works) AND F2-G5 passes (surfer patience preserved) AND F2-G9 passes (no degenerate migration) → evaluate per §4.4. If F2-G8 fails AND F2-G5 collapsed (no surfer patience because every Open exceeds Hold by saturation, or because the hinge zeroed Hold from flat too aggressively), investigate before any F5 fallback — F5 will worsen this pathology.

### §5.2 Phase 7b — F5-on-top-of-F2 gates (conditional, surfer→trend trade-off)

These gates run only if Phase 7a's F2-G8 failed under the conditions above AND the user has explicitly accepted the surfer→trend trade-off. F2 remains ON; F5 is added.

#### §5.2.1 Local b=128 Tier 1.5 — F5 mechanism (with F2 active)

| Gate ID | Metric | Pass condition | Distinguishes |
|---|---|---|---|
| F5-G1 (flat-state mask correctness) | Among `position_lots == 0` batches, action_hist mass on `{Hold, TrailLoosen, TrailTighten, FlatFromLong, FlatFromShort, HalfFlatLong, HalfFlatShort, PyramidLong, PyramidShort}` | EXACTLY 0 (not "low"; zero) | F5 mask works mechanically |
| F5-G2 (in-position direction mask correctness) | When long, mass on short-side opens / closes / pyramids | EXACTLY 0; symmetric for short | F5 mask works for in-position branch |
| F5-G3 (trail-at-cap mask correctness) | When `max_unit_trail ≥ trail_max − eps`, mass on TrailLoosen | EXACTLY 0 | mq2pc-specific mechanism gate |
| F5-G4 (flat→in-position rate) | Opens per 1000 steps from flat | ≥ 5× mq2pc baseline (≈ 50/1000) | Positive evidence of choice-set forcing (and acknowledged surfer-principle violation) |
| F5-G5 (no single-direction collapse) | Max single-action share across all positions | < 80% | Detects expected failure mode #7 |
| F5-G6 (training stability with F2) | l_pi, l_v, l_q | No NaN; ≤ 10× F2-alone baseline | F2 + F5 do not pathologically interact |
| F5-G7 (empty-baseline guard) | Fraction of batches where `A(s) ∩ no-op-set = ∅` (e.g., F5 removed all no-ops from flat) | Handled without NaN; documented per-batch | F2 + F5 composition edge case |

#### §5.2.2 Cluster b=1024 Tier 2 — F5 outcome (marginal vs F2-alone)

| Gate ID | Metric | Pass condition | Distinguishes |
|---|---|---|---|
| F5-G8 (eval pnl marginal benefit) | `eval_summary.total_pnl_usd` | ≥ F2-alone cluster eval pnl + $5M/session | F5 provides marginal value; if no marginal → revert to F2-alone commit |
| F5-G9 (Q-distill controller stability) | Slot 488 `RL_Q_DISTILL_KL_EMA_INDEX` | No monotonic growth through training | Distill controller accepted F5-masked target shape |
| F5-G10 (in-position action entropy in eval) | Action entropy conditional on `position_lots > 0` | ≥ 1.0 nats | In-position diversity preserved despite mask |

**Decision rule:** if F5-G8 passes → commit F2 + F5 with explicit documentation of the surfer→trend trade-off. If F5-G8 fails (no marginal benefit over F2-alone) → revert F5, commit F2-alone, and escalate the residual to a separate spec (likely encoder-alpha or F4 infrastructure).

### §5.3 Distinguishing F2 effect from F5 effect (the load-bearing discipline)

A reader of the cluster diag must be able to attribute eval-pnl change to F2 or F5 specifically. The gate design enables the following distinguishable cluster outcomes (Phase 7a outcomes first, Phase 7b outcomes second):

| Phase | F2 mech gates (G1,G2,G3) | F2-G5 (surfer patience) | F5 mech gates | Outcome | Attribution |
|---|---|---|---|---|---|
| 7a | PASS | PASS (Hold% from flat ∈ [30%, 85%]) | n/a | F2-G8 PASS | **SHIP F2 ALONE.** Gradient redirection was the load-bearing problem; surfer principle preserved. F5 not needed. |
| 7a | PASS | PASS | n/a | F2-G8 FAIL, no degenerate migration | F2 necessary but insufficient; evaluate per §4.4. If user accepts surfer→trend trade-off, proceed to 7b. Otherwise escalate to encoder / alignment / F4 work. |
| 7a | PASS | FAIL (Hold% from flat < 5% or > 95%) | n/a | any | Surfer principle broke in F2 alone (either over-action or under-action). Investigate before F5. F5 will worsen the over-action case. |
| 7a | PASS | PASS | n/a | F2-G8 FAIL, F2-G9 FAIL (new degenerate) | F2 surfaced a different pathology; STOP, escalate to F4 (do NOT add F5 on top of a degenerate). |
| 7a | FAIL | n/a | n/a | any | F2 mechanism broken (mass-zero target). Fix kernel (warmup / temperature schedule). |
| 7b | PASS | n/a | PASS | F5-G8 PASS | F5 marginal benefit confirmed; commit F2 + F5 WITH explicit surfer→trend trade-off documented. |
| 7b | PASS | n/a | PASS | F5-G8 FAIL | F5 has no marginal benefit; revert F5, commit F2-alone, escalate residual to encoder / alignment work. |
| 7b | PASS | n/a | FAIL | any | F5 mechanism broken; fix or abandon F5. |
| 7b | FAIL | n/a | n/a | any | F2 broke when F5 turned on; F2 + F5 composition is the bug (likely empty-baseline edge case). |

Per `feedback_investigation_first_falsification_methodology` §"falsifiable verdicts", every cluster run produces an attribution row in this table. Going in circles is structurally prevented because the row identifies the specific next action.

### §5.4 The "if F2 + F5 are both wrong" prediction

Per `feedback_going_in_circles_pattern`, the spec must predict what failure looks like if both intervention layers fail — so we don't tile the next fix on top of these.

**If F2 + F5 both pass mechanism gates but eval pnl stays at h6btt baseline:** the inaction attractor was a *symptom* of a deeper problem (most likely reward-pnl alignment per `pearl_reward_signal_anti_aligned_with_pnl`, or encoder alpha extraction, or train→eval generalisation per `pearl_surfer_baseline_was_train_only_never_eval`). The gradient is correctly redirected away from no-ops (F2), the choice-set is constrained from collapsing to inaction (F5), and the agent still cannot make eval-profitable trades — that isolates the failure to either (a) the reward signal is not pnl-aligned, or (b) the encoder is not extracting alpha. Neither is solvable by another Bellman-target reshape.

**Specifically watch for these signals:**

- Action distribution diverse (action_entropy ≥ 1.5 nats), no degenerate attractor, eval pnl negative or zero → reward-pnl misalignment is dominant; escalate to a reward-signal investigation per `pearl_reward_signal_anti_aligned_with_pnl`.
- Action distribution diverse, eval pnl negative, train pnl significantly positive (≥ +$5M) → train→eval generalisation failure per `pearl_surfer_baseline_was_train_only_never_eval`; escalate to a regularisation / sim2real spec.
- Action distribution single-direction-dominated (e.g., 90% Long open, then immediate close), eval pnl noise-floor → encoder is not producing directional alpha; escalate to encoder investigation.

**If any of these occur, STOP per `feedback_going_in_circles_pattern`. The next intervention is NOT a further reshape of the Bellman target — it is a foundation-level intervention upstream. F4 (oracle pretraining) becomes the right next dispatch if the encoder hypothesis dominates; reward-signal investigation if alignment dominates.**

---

## §6. Out of scope

This spec contains TWO interventions (F2 and F5) but they are disciplined as a single combined Phase 7 with a staged sub-sequence (7a = F2-alone, 7b = F5-conditional). This is NOT four-fix stacking; it is a single decision tree with one branch point. The discipline rule from `feedback_going_in_circles_pattern` — "one fix at a time with predetermined gates" — is honored because (a) Phase 7a tests F2 in isolation (surfer-principle-preserving primary phase), and (b) Phase 7b is only entered if Phase 7a's outcome gate identifies the residual AND the user accepts the surfer→trend trade-off encoded in F5.

This spec deliberately does NOT cover:

1. **F4 oracle-pretraining infrastructure.** Belongs in a separate spec (~`2026-06-05-offline-rl-infrastructure.md`). Estimated 3–4 weeks. Should be drafted after Phase 7a/7b cluster verdict.
2. **F1 (per-step inaction cost) implementation details.** Backup option only; specified at theoretical level only here.
3. **F3 action-space reduction.** Disfavored; not specified at implementation level. F5 is the option F3 should have been.
4. **Reward-policy alignment investigation.** Already covered by `2026-06-01-reward-policy-alignment-investigation.md`. F5 + F2 are orthogonal — the reward signal and the choice-set / distill target are independent mechanisms.
5. **GAE / Phase 1B rollout buffer.** Orthogonal; can ship independently. GAE benefits V quality; F5 benefits the policy support; F2 benefits π update locality — none blocks the other.
6. **Multi-head policy (Phase 2A) interaction tuning.** F5 and F2 should work on both single-head and multi-head architectures. The mask in F5 is per-batch per-action, independent of head; F2 modifies the distill target shape per-head. Multi-head verification is part of cluster Tier 2.
7. **Encoder alpha investigation.** If F5 + F2 both pass mechanism gates but eval pnl stays at baseline (§5.4), the encoder may be implicated. That escalation path is a separate spec.
8. **Controller recalibration after F2.** The Q-distill controller (slot 486) is tuned for the current target shape. F2 reshapes the target → the controller may need re-bootstrapping. This is *implementation work* for the F2 plan document, not design work for this spec.
9. **CUDA Graph A/B capture impact.** F5 and F2 kernel diffs are small and should preserve graph capture. Verification belongs in the implementation plan.
10. **Synthetic-Hold fallback for zero-edge regimes.** If F5 forces unprofitable trades in no-edge market regimes (failure mode #1), a "synthetic abstain" via micro-open / band-narrow-open is a candidate Phase 7c intervention. Not specified at this stage; depends on Phase 7a outcome.
11. **Confidence-gate position-aware wiring.** Per `pearl_confidence_gate_pos_state_unwired`, `rl_confidence_gate.cu` accepts `pos_state` but doesn't read it. F5 does not depend on fixing this — F5 operates at a different pipeline stage (pre-softmax logit mask, not post-action override). The confidence-gate wiring is a separate fix and out of scope.

---

## §7. Honest uncertainty flags

This section explicitly flags claims the spec makes that are speculative or unverified, per `feedback_investigation_first_falsification_methodology`:

1. **§3.2 mass-zero distill target risk.** The risk that "no action's E_Q exceeds the baseline" during early training is real but unquantified. Without a Tier 1.5 run, the probability is unknown. The mitigation (SAC entropy preserves uniform exploration when target is zero) is intuitive but not verified.
2. **§3.2 F2-band interaction.** Phase 4-A band masking short-circuits the policy's chosen action *after* the policy emits it but *before* the action is executed. The Q-distill kernel reads the policy's emitted action and the Q-targets — it does NOT directly read the executed action. Whether F2 still works correctly when the band overrides actions to Hold is an open question requiring code verification in the implementation plan. **This is a load-bearing uncertainty.** F5 partially supersedes this by operating at the *logit* stage (before sampling) rather than the *action* stage (post-sampling) — F5 + F2 composition through the band is cleaner because F5 itself is the same lattice as the band and runs in series with it.
3. **§3.3 F3's reintroduction of static trail risk.** The argument that removing TrailTighten/TrailLoosen reintroduces a known-suboptimal risk model is sound, but the *magnitude* of the regression depends on volatility regime. In a low-volatility data subset, static trails might be fine; in high-volatility, they're disastrous. Unquantified. (Moot since F3 is rejected; F5 does not remove these actions, only conditionally masks them.)
4. **§3.4 F4 scope estimate.** "2000–5000 LOC" is an order-of-magnitude guess based on the size of comparable offline-RL implementations elsewhere. Foxhunt's specific scope could be larger if the existing dataset infrastructure does not support per-state oracle lookups, or smaller if a minimal one-epoch version is acceptable.
5. **§3.5 F5 force-trade failure mode.** Failure mode #1 (forcing trades in zero-edge regimes) is the biggest substantive risk in F5. There is no quantitative estimate of how often the ES MBP-10 dataset contains zero-edge regimes vs alpha-bearing regimes. If a significant fraction of trading sessions has no exploitable alpha, F5's forced trading from flat could *worsen* eval pnl vs the do-nothing baseline. The h6btt eval pnl baseline is itself negative — if F5 makes it *more* negative, that is a Phase 7a refutation per §4.4.
6. **§3.5 F5 mask propagation into Q-distill correctness.** Failure mode #3 — applying the mask asymmetrically to π_θ and `pi_target` — is the biggest *implementation* risk. The mask must propagate identically into the Q-distill kernel. The original `rl_confidence_gate.cu` failure (per `pearl_confidence_gate_pos_state_unwired`) is precedent that "this kernel reads `pos_state`" can be silently false. Tier 1 correctness tests must explicitly verify that masked actions have zero mass in BOTH the policy distribution AND the `pi_target` distribution.
7. **§3.5 F5 + F2 baseline-empty edge case.** When F5 masks all no-ops from a given state (e.g., from flat, all of {Hold, TrailLoosen, TrailTighten} are unavailable), F2's `baseline(s) = max_{a ∈ A(s) ∩ no-op-set} E_Q(s, a)` is over an empty set. The natural definition is `baseline(s) = −∞` in this case (any action's `E_Q > −∞` → hinge passes for all), but this requires explicit handling in `rl_q_pi_distill_grad.cu`. F5-G7 (in §5.2.1) catches this; the Phase 7b implementation plan must specify the behavior.
8. **§4.1 score matrix weighting.** The implicit weighting (equal across criteria) is arbitrary. A different weighting (e.g., "implementation cost matters 3× as much as theoretical soundness") would push the recommendation toward F5/F2 even more strongly. A pro-research weighting (theoretical soundness 3×) would push toward F4. The spec author judges roughly-equal weighting as most appropriate.
9. **§4.1.5 "F2 alone may be sufficient" claim (revised in Revision 3).** This is a structured guess, not a verified prediction. The argument (the surfer alpha mode is reachable from the current Phase 5 substrate; F2's hinge expresses the surfer principle as a loss function; the encoder produces sufficient edge signal for E_Q(Open) > E_Q(Hold) at edge-bearing states) is plausible but unverified. The opposing failure scenario (encoder too weak; F2 degenerates to uniform random) is the load-bearing risk to monitor. Phase 7a's cluster verdict is the only way to know.
10. **§5.1 / §5.2 thresholds (0.5 nats target-entropy drop, 5× open-rate increase, $5M eval pnl improvement, [30%, 85%] Hold% band for surfer-patience).** All threshold values are guesses motivated by mq2pc / h6btt / dd049d9a4 baselines but not pre-calibrated. The Tier 1.5 implementation should report *actual* deltas; the plan document can tighten thresholds based on smoke data.
11. **§5.2 F5-G8 marginal benefit threshold ($5M / session BEYOND F2-alone).** If F2 alone produces a $4M improvement and F5 produces an additional $4M, the combined $8M passes F2-G8 but fails F5-G8 (which requires $5M *marginal*). The threshold is conservative — it prefers "F2 alone is enough" over "F2+F5 is slightly better." This may understate F5's true value; consider lowering to $2-3M in the Phase 7b plan document. The bias toward F2-alone is intentional in Revision 3 because preserving the surfer principle is preferred when the marginal benefit of F5 is small.
12. **§4.1.4 surfer-patience band [30%, 85%] (Revision 3 addition).** The [30%, 85%] Hold% range for F2-G5 is calibrated against the dd049d9a4 surfer baseline (which had avg_hold ≈ 30 with frequent positions; back-of-envelope this implies Hold-from-flat in roughly the 40-70% range) but not directly measured at the per-state-conditional level. The exact target band should be re-derived from per-state action histograms in the surfer baseline diag.jsonl during plan development.

---

## §8. Document discipline notes

- **No LOC, no code, no implementation order in this spec.** Per `feedback_investigation_first_falsification_methodology` §6: spec is WHY + design + falsification; plan is HOW. This is the spec. (LOC estimates in §3.1, §3.2, §3.5 are *scoping* notes for the comparative analysis; they are not implementation prescriptions.)
- **Staged combined intervention with single-fix-per-phase discipline (revised in Revision 3).** Per `feedback_going_in_circles_pattern`: one fix at a time with predetermined gates. The combined F2 + F5 recommendation is realized as Phase 7a (F2-alone, surfer-principle-preserving) followed by conditional Phase 7b (F5 added, surfer→trend trade-off accepted). At each phase boundary, exactly one new mechanism is being tested, with its own mechanism gates. This is NOT two-fix stacking; it is a decision tree with one branch point informed by a falsifiable outcome gate.
- **Both mechanism gates and outcome gates.** Mechanism gates (F2-G1/G2 for the gradient axis; F5-G1/G2/G3 for the choice-set axis if Phase 7b is invoked) tell us if the kernel does what the math says it does. Outcome gates (F2-G8 for Phase 7a; F5-G8 for Phase 7b) tell us if the math is right about the policy effect. The Revision 3 surfer-patience gate F2-G5 is a *behavioral* gate that fits between mechanism and outcome — it verifies the policy expresses the surfer principle in the action distribution, even before the cluster outcome arrives. All three categories are required to extract maximum information from each cluster run.
- **Distinguishable attribution.** §5.3's 4-outcome table makes every cluster verdict map to a specific next action. Going in circles is structurally prevented.
- **Pre-registered failure prediction.** §5.4 commits in advance what failure looks like if both F2 and F5 pass mechanism gates but eval pnl stays flat, so we cannot rationalize ambiguous results.

---

## §9. Appendix — Plan handover surface

### §9.1 Phase 7a (F2-alone) plan handover — the next deliverable

If F2 is selected and proceeds to implementation (the Revision 3 recommendation), the plan document at `docs/superpowers/plans/2026-06-04-bellman-target-foundation-reshape-F2-plan.md` should cover:

- New ISV slots: F2 master gate (binary, bootstrap 0.0); no-op-set mode enum (F2a static vs F2b dynamic); optional λ-controller-bootstrap slot.
- Kernel diff for `rl_q_pi_distill_grad.cu` lines 80–91 (the `s_pi_target` computation): add the baseline subtraction + `max(0, ·)` hinge over the no-op-set, then softmax.
- **Definition of the no-op set.** Start with F2a (static set: Hold, TrailTighten, TrailLoosen). If cluster shows F2a works, defer F2b (dynamic, lots-change-based) to a follow-up plan.
- **Mass-zero target failure mode handling (load-bearing).** During cold-start, the encoder's E_Q may be too noisy and no action exceeds `baseline`. The target is uniform-zero; SAC entropy dominates; agent random-walks. Mitigation: warmup phase with F2 OFF (first 500-1000 steps), then enable; OR temperature schedule on the hinge.
- **Phase 4-A band-mask interaction (load-bearing per §7 uncertainty #2).** The band-mask post-overrides the policy's chosen action *after* the policy emits it. F2's distill target reads E_Q and produces `π_target` over all actions; the policy's emitted action goes through the band. Whether F2's gradient remains correct when the band overrides actions is an open question. Verify in the implementation plan via code reading + correctness test.
- **Surfer-principle preservation verification (F2-G5).** The plan must include a Tier 1.5 test that explicitly measures Hold% from flat states and gates on the [30%, 85%] band. If the local smoke produces Hold% from flat outside this band, investigate the baseline-set definition before cluster.
- Trainer-side dispatch (kernel call sequence unchanged; input distribution unchanged; ISV-driven branch is new).
- Diag emit changes — slot 733 already exists, but new diag fields for "action mass in no-op-set" and "fraction of batches where target is zero-mass" per `pearl_fleet_fraction_not_aggregate`. Add diag for Hold% conditional on `position_lots == 0` (the F2-G5 metric).
- Controller re-bootstrap — the distill-λ controller (slot 486) and SAC-α (slot 581) may need re-bootstrap because their setpoints assume the previous target shape.
- Tier 1 (correctness, ~6 sec): `pi_target` no-op mass = 0 for any state where any `a_c` has `E_Q > baseline`; no NaN; bit-equality with master gate OFF.
- Tier 1.5 (behavior, ~10 min): F2-G1 through F2-G7 (including the F2-G5 surfer-patience gate).
- Tier 2 (eval verdict, ~70 min): F2-G8, F2-G9, F2-G10, F2-G11 (the surfer-fingerprint gate).
- Cluster commit boundary: one commit per phase per `feedback_commit_per_phase_for_long_coder_tasks`.

### §9.2 Phase 7b (F5-on-top-of-F2) plan handover, conditional on §9.1 outcome AND user acceptance of surfer→trend trade-off

If F2-G8 fails per §4.5 step 6 AND the user explicitly accepts the surfer→trend trade-off, the plan document at `docs/superpowers/plans/2026-06-04-bellman-target-foundation-reshape-F5-plan.md` should cover:

- New ISV slots: F5 master gate (binary, bootstrap 0.0; separate from F2's gate); trail-eps tolerance (float, bootstrap ≈ 1e-2 · trail_max).
- New kernel `crates/ml-alpha/cuda/rl_state_action_mask.cu` modeled on `rl_band_mask.cu` lattice (Grid=(B), Block=(1,1,1)), reading `pos_state[b*pos_bytes+0..4]` for `position_lots` and `trail_distances[b*MAX_UNITS+u]` for per-unit trail distances. Action mask via `logits[b*N_ACTIONS+a] = −∞` sentinel.
- Kernel launch site in `crates/ml-alpha/src/trainer/integrated.rs` — between policy logits emission and softmax sampling. Determine the precise call site by reading the existing dispatch chain.
- **Mask propagation into Q-distill `pi_target` computation (load-bearing).** Either pass the same masked logits to `rl_q_pi_distill_grad.cu`, or pass the `pos_state` + ISV and recompute the mask identically inside the distill kernel. The chosen approach must be verified by a correctness test that the masked-action mass is zero in BOTH the policy distribution AND the distill target.
- **Empty-baseline edge case handling (load-bearing).** When F5 masks all no-ops from a given state (e.g., from flat, all of {Hold, TrailLoosen, TrailTighten} become unavailable), F2's `baseline(s) = max_{a ∈ A(s) ∩ no-op-set} E_Q(s, a)` is over an empty set. Use `−∞` sentinel so the hinge passes for all actions; document the per-batch fraction of empty-baseline states.
- Diag emit changes: per-batch fraction-masked-from-flat, fraction-masked-trail-at-cap (cf. `pearl_fleet_fraction_not_aggregate`). Wire into existing diag emission.
- Test fixtures in `crates/ml-alpha/tests/trade_management_kernels.rs` (parallel to existing band-mask tests): flat-state forces opens; in-position forbids opposite-direction opens; trail-at-cap forbids TrailLoosen; bit-equality with master gate OFF.
- **Explicit documentation of surfer→trend trade-off in commit message and plan summary.** F5 is being shipped because F2 alone was insufficient and the user has consciously decided that breaking the h6btt-class structural collapse is worth the surfer-principle regression.
- Tier 1 (correctness, ~6 sec): kernel correctness + Q-distill mask correctness + F2 hinge handles F5-masked no-op set + no NaN.
- Tier 1.5 (behavior, ~10 min): F5-G1 through F5-G7 + all F2 mechanism gates still pass.
- Tier 2 (eval verdict, ~70 min): F5-G8 + F5-G9 + F5-G10.
- Cluster commit boundary: one commit per phase per `feedback_commit_per_phase_for_long_coder_tasks`.

---

## §10. Linked work

- Phase 4-A band (`2026-06-03-no-transaction-band-architecture.md`): the band mask post-overrides the policy's chosen action when inside the band. F2 modifies the Q-distill target shape, which reads E_Q over all actions; the policy's emitted action subsequently goes through the band. F2 must verify it composes with the band mask — see §7 uncertainty #2 (load-bearing). F5 (if Phase 7b is triggered) is the band mask's structural twin and runs in series with it: F5 applies first (pre-softmax logit mask); band mask applies second (post-action override to Hold when inside band).
- Phase 5 MTM reward (`0463e44e0`): F2 + F5 leave the reward signal untouched. MTM continues to operate. All interventions can compose.
- Phase 2A multi-head (`2026-06-02-multi-head-policy-with-r-multiple.md`): F2 modifies the Q-distill kernel which is shared across heads. F5 mask (if invoked) is per-batch per-action, head-independent. The modifications are per-action, not per-head, so multi-head architecture is unaffected by F2 or F5 mechanism.
- Phase 1B GAE (`2026-06-02-trainer-rollout-buffer-gae.md`): orthogonal; can ship independently.

---

## §11. Decision log entry

If F2 (Phase 7a) is approved for implementation, log the decision per `feedback_v7_gem_methodology` §decision-log:

```
Decision: Adopt F2 (Q-distill advantage target with max(0, E_Q − baseline) hinge) as Phase 7a, with F5 (state-conditional action availability mask) staged as conditional Phase 7b ONLY IF F2 outcome gate fails AND user explicitly accepts surfer→trend regression trade-off.
Rejected: F1 (inaction-cost reward shaping), F3 (action-space reduction), F4 (oracle pretraining — gated on infrastructure that does not exist).
Rationale: F2 addresses the structural inaction-bias problem via the gradient axis (Q-distill target shape) while PRESERVING the surfer trading principle by mathematical construction. The hinge `max(0, E_Q(a) − baseline)` zeroes Open mass when no edge is detected and zeroes Hold's contribution when other actions exceed it — the agent learns to wait or act based on the encoder's edge signal, not a structural constraint. F2 IS the surfer principle expressed as a loss function (per pearl_dd049d9a4_surfer_baseline_verified, the surfer alpha mode wr=0.56, hold=30, +$10.6M @ step 18k is the cluster-proven target).

F2 is theoretically grounded in Wang et al. 2016 (Dueling Network Architectures, advantage decomposition), Schulman et al. 2015 (GAE, advantage-based policy updates), and Mnih et al. 2016 (A3C, advantage-actor-critic). F2 is implementable with bounded scope (~50-100 LOC across the rl_q_pi_distill_grad.cu kernel) and is falsifiable with deterministic mechanism gates (target entropy drop, no-op mass collapse) AND a surfer-principle preservation gate (Hold% from flat ∈ [30%, 85%]).

F5 is staged as conditional Phase 7b because (a) F5 violates the surfer trading principle by removing Hold-from-flat from A(s) — destroying patience-while-waiting-for-setup, the load-bearing structural property of the surfer alpha mode — and (b) per `feedback_going_in_circles_pattern` the most disciplined sequence tests the surfer-preserving option first. If F2 alone passes F2-G8 (eval pnl ≥ h6btt + $5M) with surfer-patience preserved (F2-G5), F5 is deferred. If F2-G8 fails AND F2 mechanism gates pass AND no new degenerate attractor surfaces AND the user explicitly accepts the surfer→trend trade-off, F5 is added on top.

Revision history: Revision 1 (2026-06-04) recommended F2 alone. Revision 2 (2026-06-04b) added F5 and recommended F5 first / F2 conditional based on pre-surfer-criterion scoring. Revision 3 (2026-06-04c, THIS revision) added the surfer trading principle as a 7th scoring criterion and flipped to F2 first / F5 conditional. Score totals: F2=30, F4=29 (gated), F5=28, F1=24, F3=14.

F4 is the longer-term right answer but requires infrastructure that does not exist.

Date: 2026-06-04 (Revision 3 — surfer-principle-aware staging)
Falsification trigger (Phase 7a): F2-G8 per §5.1.2 + F2-G5 per §5.1.1
Falsification trigger (Phase 7b, conditional + user-acceptance-gated): F5-G8 per §5.2.2
Going-in-circles trigger: §5.4 (both passed mechanism, eval still flat → escalate to F4 or reward-alignment)
Distinguishable-attribution discipline: §5.3 (outcome attribution table)
Surfer-principle decision criterion: §4.1.4
```
