Vol-risk-premium (short-vol BTC+ETH, Deribit DVOL implied minus realized, short-variance
daily P&L proxy). VRP Sharpe +1.03, CORRELATION to momentum -0.03 (genuinely uncorrelated),
combined +1.22 (~doubles momentum-alone +0.66). FIRST genuine diversifier found -- has BOTH
edge AND low-corr (futures trend had corr but no edge -> diluted). CAVEATS: (1) catastrophic
negative skew -8.32, worst-day -21sigma (short-vol blowup risk; Sharpe flatters; must
tail-manage + size small) (2) recent decay -- negative 2025/2026 (+1.22 is a 2021-24 artifact).
Real and promising, categorically better than futures, but pursue carefully not naive deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Expanded futures universe 22->39 liquid CME roots ($16.39 credits, 16y daily). (A)
breadth did NOT unlock an edge: XS momentum + TS trend all negative/zero, DSR 0
(efficient market, unlike crypto). (B) diversifier check: futures-trend +0.01,
crypto-momentum +0.88, CORRELATION -0.01 (genuinely uncorrelated) but combined +0.63
< +0.88 -> an uncorrelated sleeve with zero standalone edge DILUTES, not diversifies.
A diversifier needs low-corr AND positive edge; futures trend has only the former.
Caveat: roll-zeroing understates futures trend/carry (floor work ~+0.05-0.14 w/ proper
rolls) -> marginal at best. Crypto momentum still the only real edge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Rebalance keyed on FIXED calendar phase (epoch-day%7==0 -> Thursdays) instead of
drifting days[0]%K, so backtest and live agree on rebalance timing.
- STATE path absolute (relative to script) -> paper mode runs from any cwd (cron-safe).
- Backtest unchanged (net +0.40@$20M, CPCV-med +0.47).
Installed system crontab (daily 12:07): marks paper PnL daily, rebalances Thursdays,
logs to data/surfer/paper_runs.log. Forward-test seeded inception 2026-06-06.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
One-file PoC of the validated crypto XS-momentum edge. ONE shared compute_weights()
drives backtest AND live (no skew); no lookahead; funding-as-PnL; sqrt-impact cost.
Backtest reproduces research (gross +0.83, net +0.43@$20M, capacity curve, turnover
9.8%). Regime diagnostic = clean NEGATIVE (no feature predicts momentum favorability,
IC~0 IS/OOS) -> de Prado meta-labeling/regime overlay empirically unwarranted, runs
flat-sized. Paper mode = live forward-test (Binance ASCII perps, market-neutral book,
intended trades + paper PnL persisted). Python is the right tool for weekly crypto MFT;
foxhunt Rust/CUDA is HFT-ES-specific (concepts reused, plumbing not).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Directional TS-trend (fixed earlier demeaning bug) is a real edge (bootstrap frac>0=1.00)
with low corr to momentum (+0.12) and lifts full-sample Sharpe +0.67->+0.71 -- but per-year/
OOS expose it as STALE (value all in 2021 +2.65; recent 2024 -0.16/2026 -0.89), so it DRAGS
the book OOS to +0.24 vs momentum's strong recent. 2022 crisis-alpha hypothesis failed
(choppy bear whipsawed trend). Within-crypto diversifiers exhausted (carry/lowvol/reversal/
size/TS-trend all fail) -> momentum is a robust single-sleeve edge; real diversifier needs a
different market/data. Discipline: full-Sharpe said add, OOS said stale.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(2) Low-vol FAILS its full battery (standalone ~0/neg all windows, coin-bootstrap
frac>0=0.45 coin-flip) -> its earlier diversifier value was a winsor artifact ->
NO validated second edge, momentum stands alone. (1) Slippage/capacity model
(sqrt-impact eta=1, daily rebalance): book gross +0.62 but net M +0.41 / M +0.22 /
0M -0.11 / 0M -0.50 -> real but small-capacity (~$1-5M), lives in smaller alts.
Favorable caveats: eta=1 conservative, daily rebalance of 20d signal wastes turnover.
Key untested lever: weekly rebalance + smoothing + top-20 liquid -> ~5x less turnover.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(a) All 6 momentum constructions collapse to ~0.2 under aggressive +/-20% winsor
(momentum leans on large moves); volscaled/winsorinput degrade least, none robust to
the cap. (b) LOWVOL is genuinely uncorrelated (corr -0.13) and lifts the stressed book
+0.19->+0.34; TStrend correlated; carry/reversal/size don't help. Book=momentum+lowvol.
Honest deployable bracket ~0.2 (harsh winsor) to ~0.6 (fair); +/-20% blanket winsor
double-counts (death-spiral already excluded) so truth ~0.5. Deciding next: per-coin
ADV slippage model, not blanket winsor. Corrected the script's over-claiming verdict.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Built survivorship-free universe: 135 perps incl 28 known-dead (LUNA/SRM/MATIC...),
universe rebuilt daily as top-K by trailing dollar-volume (dead coins in while trading,
drop out after crash), no lookahead. mom_20 TOPK=30: full +0.76, IS+0.85/OOS+0.56,
CPCV-med +0.73, DSR 0.86, POSITIVE EVERY YEAR 2020-2026 incl 2022 +0.72. Counterintuitive:
survivorship was HIDING the edge (survivor-only 2022 -0.40 -> PIT +0.72) because momentum
shorts the dying coins and profits from crashes. Decisive survivorship confirmation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Broadened crypto universe 28 majors -> programmatic top-80 (64 usable), added
funding-coverage/history filter. Carry COLLAPSED (+0.82->-0.03) and stayed dead even
on broad-but-clean subset (fundcov>0.9, 45 coins): carry was cherry-picked to majors,
not robust. Cross-sectional MOMENTUM is the robust edge: XS_mom_30 IS+0.89/OOS+0.88,
CPCV-med +0.75, DSR 0.75, survives 2x cost, holds across 64- and 45-coin universes.
Momentum needs breadth (weak on 28 correlated majors). Same regime failure (2019/2022).
Methodology win: broadening killed a cherry-picked claim before a build.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
13 crypto signals, deflated by cumulative 30. Carry family all develop-grade, survive
2x cost; per-year reveals THE failure mode = 2022 deleveraging carry-crash (carry_7
-1.2 in 2022, +0.4..+2.1 other years). Momentum OOS-negative (decayed); chasing rising
funding strongly negative. Carry is the robust core; its one tail (carry crash) is
exactly what the ML regime overlay is built to manage -> the two threads unite.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Batch 2 of the deflated signal sweep: cross-sectional funding carry + momentum on
28 major Binance USDT perps (2019-26, funding baked into return, 10bp cost, deflated
by cumulative 25 trials). XS_carry_7 = Sharpe +0.82, IS+0.77/OOS+0.93 (consistent),
CPCV-median +0.78 (robust across 45 paths), DSR 0.71. First signal all session that is
positive + IS/OOS-consistent + CPCV-median-positive. Develop-grade met, not yet deploy
(DSR<0.95, 5th-pct<0). Caveats: survivorship (current majors), confirm point-in-time.
Forward-paginated funding fetch (fundcov ~1.0).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Broad signal search judged against the FULL search (Deflated Sharpe deflated by
N_trials) — the multiple-testing discipline that makes breadth safe. Batch 1: 17
signals (XS/TS momentum, reversal, low-vol, seasonality) on 22-futures daily ->
ZERO survivors (best DSR 0.29); CPCV+deflation correctly kill the lucky cells.
SEAS_overnight invalid on 24h futures (equity-cash anomaly). Next batches: crypto, COT.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tests the actual surfer thesis: intraday 5-min decisions/15-min holds, flat-by-close,
regime-adaptive (ride trend / fade range / stand aside), quality-over-quantity
(top-conviction only). Clean ES OHLCV-1m -> 5-min bars, per-trade net edge in ticks,
IS/OOS + t-stat. Result: regime-adaptive top-5% = +0.50 ticks/trade OOS, sign-consistent,
but t=0.67 (not significant); all else significantly negative. Adaptive+selective is the
only non-losing structure (thesis directionally right, within noise). 4GB-safe: cumsum
rolling-std (no unfold) + alloc cap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per feedback_cpu_is_read_only + pearl_cold_path_no_exception_to_gpu_drives +
feedback_no_cpu_test_fallbacks: NO CPU compute paths even for validation/cold
paths. Rewrites Phase 0 as CUDA kernels + Rust orchestration in crates/ml-alpha
(reusing build.rs cubin pipeline, mapped-pinned, determinism): floor kernels
(TSMOM signal, EWMA vol, portfolio backtest) + validation kernels (CPCV Sharpe,
PBO, Deflated Sharpe). CPU only enumerates split masks + reads final gate scalars
via mapped-pinned. GPU-oracle tests (no CPU reference). Runs on the local GPU.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CPU-only/Python TDD plan: continuous-contract builder, TSMOM floor (1/3/12mo +
inverse-vol + vol-target), validation harness (CPCV/purge/embargo + PBO +
Deflated Sharpe + futures cost model), broad-universe data acquisition, and the
decisive verdict run (does the floor clear SV gates OOS after costs?). Tasks A-C
run immediately on local 4 instruments; D-E need Databento broad-universe history.
No ML, no GPU, no cluster.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Strategic pivot from seconds-horizon ES order-book RL (measured structurally
unprofitable) to diversified days-to-weeks multi-asset ML trend. ML as bounded
deviations from a deterministic diversified-trend floor; gated by CPCV/PBO/
Deflated-Sharpe. Phase 0 (build+validate the floor on Databento micros, CPU-only,
no ML, no cluster) is the immediate decisive test. Synthesizes 8-agent omnisearch
research (pearl_surfer_universe_build_synthesis).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per-instrument time-series momentum + diversified trend portfolio + lead-lag,
honest IS(2024)/OOS(2025) Sharpe on non-overlapping daily P&L.
Verdict on 2y/4-instrument data: diversified trend portfolio IS Sharpe +1.98
(L10) -> OOS -0.50; per-instrument TSMOM sign-flips IS<->OOS; lead-lag ICs <0.11.
No robust edge. Conclusion: 4 instruments x 2y is underpowered by 1-2 orders of
magnitude for systematic edge discovery -- the bottleneck is data/universe, not
model. Real progress needs a broad futures universe + decades, or a less-efficient
order-book market (crypto).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tests minutes-to-days directional predictability on ES OHLCV-1m where moves
dwarf the spread: momentum/reversion IC (Pearson/Spearman/NW-t/OOS) + a
non-overlapping daily contrarian backtest (IS/OOS Sharpe).
Verdict on 2y ES (2024-25): intraday IC~0; daily mean-reversion IC looks large
(-0.44 Spearman @20d) but is overlap-inflated + regime artifact -- honest
non-overlapping backtest sign-flips IS<->OOS => no stable edge. Confirms no
robustly capturable directional edge on 2y ES across all measured horizons.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
FIFO queue priority via price-level episodes: join back of queue, only trades
advance, fill when same-side aggressor volume clears the queue. Re-measures
markout + OFI-conditioning on ACTUALLY-WON (adversely-biased) fills.
Verdict (Q1+Q2 2024 ES): ~40% fill rate; won-fill markout -0.50..-0.55t
(winner's curse vs optimistic -0.40t); naive AND OFI-conditioned passive MM
both NEGATIVE (-$1..-$2/fill). The optimistic Q2 OFI tilt was a win-every-queue
artifact. Combined with the crossing result: no capturable seconds-horizon ES
edge for a non-colocated participant.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Measures passive-MM economics on clean ES MBP-10: half-spread earned vs
adverse-selection markout at horizons, plus OFI-conditioned selective quoting.
Verdict on 2024 ES: adverse selection ~0.4 tick eats ~80% of the 0.5-tick
half-spread => naive passive MM is break-even (optimistic, win-every-queue;
realistically negative). OFI-conditioned selective quoting shows a consistent
edge in Q2 (+$0.5/fill) but not Q1 => promising but unconfirmed; decisive
next test is a realistic queue/fill model.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Measures best-level Order-Flow-Imbalance (Cont-Kukanov-Stoikov) predictive
edge on clean ES MBP-10 vs forward returns: contemporaneous R², forward IC
(Pearson/Spearman), sign-accuracy, Newey-West t-stat across horizons, and a
CROSSING verdict (predicted move in ticks vs the 1-tick round-trip cost).
No training/RL. Verdict on clean 2024 ES: OFI fwd IC tiny (~0.02, Q2 NW-t 3.7),
predicted moves 0.01-0.06 ticks at 1-10s << 1-tick crossing cost (~20-100x gap)
=> spread-crossing directional strategy is structurally unprofitable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both parse_mbp10_file and parse_mbp10_streaming wrote the single MBP-10
update event's (price,size) into levels[0] via update_level(0,...) and
then copied the authoritative book from mbp10.levels[1..] — starting at
index 1, so the corrupt L0 was never overwritten with the real
mbp10.levels[0]. Result on real cluster data (ES.FUT 2025-Q1 front-month,
2M records): 14.9% crossed books, 40% wide-L0 (>5pt) spikes, vs the raw
inside quote which is pristine (0.016% crossed, 0% wide, 0.25pt median).
Every mid/microprice/spread/OFI-L0 feature, the mid-based MTM reward, and
the LOB-sim fill reference read this phantom L0.
Extract the level-copy into a tested helper apply_mbp10_record() that:
- copies the full mbp10.levels[0..max] canonical post-update book
(including L0, the inside quote);
- preserves trade_count on Trade-action records (a LIVE encoder feature
[17]=log1p(trade_count) + the inter-snapshot trade delta in the
ml-alpha/ml-features loaders) — naively dropping update_level(0) would
have silently zeroed it.
Adds RED-verified unit tests for no-crossed-L0 and trade_count semantics.
cargo test -p data --lib: 377 passed.
Sidecars (.predecoded.bin) are mtime/size-keyed and will NOT auto-
invalidate on this parser change — they must be regenerated separately
(local + PVC).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Session 2026-06-05: investigation-first falsification gated the F4 oracle
build (7-14 days) and walked back through the reward/edge question to a
code-verified root cause — the MBP-10 decoder corrupts the inside quote
(dbn_parser.rs:903 'for lvl in 1..max_lvl' never restores levels[0]),
contaminating mid/feature/reward/fill locally and on the cluster PVC.
- notes/2026-06-05-edge-investigation-decision-record.md: full 8-verdict arc
(incl. 2 self-retractions) + next-session plan (confirm on PVC, fix parser,
regenerate sidecars, re-audit on clean data).
- specs: f4-oracle-pretraining (GATED), reward-pnl-alignment-restoration
(superseded by the data-quality finding), bellman-target-foundation-reshape.
- plans: reward-pnl-alignment-restoration-implementation, f2-q-centered-distill.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Computes Pearson(rewards.sum, trading.realized_pnl_usd_delta) + sign-agreement
from a diag.jsonl. Exit 0 if Pearson >= 0.70 (RA-G3), 1 below, 2 on bad input.
Reproduces the baseline 0.2775 / sign 0.655 measurement.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds ISV slot 824 (RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX) and a matching
kernel early-return in rl_surfer_scaffold_controller.cu so that when
FOXHUNT_PIN_SURFER_SCAFFOLD=1 is set, the controller leaves slot 753 at
its 0.0 pure-pnl bootstrap instead of overwriting it every step.
Without the pin, the 0.0 bootstrap was cosmetic — the unconditional
write re-enabled all four Phase-5 non-potential shaping terms each step,
giving Pearson(reward,pnl)=0.28 vs the 0.70 gate. Pin=OFF leaves
behaviour fully unchanged (new branch only fires when slot 824 > 0.5).
- isv_slots.rs: slot 824 constant + RL_SLOTS_END 824→825 + test
- rl_surfer_scaffold_controller.cu: #define + early-return guard
- integrated.rs: isv_constants 275→276 + env-gated bootstrap entry
- eval_diag_emission.rs: rewards.surfer_scaffold_force_pin diag field +
EXPECTED_LEAVES 746→747
Build: SQLX_OFFLINE=true cargo build -p ml-alpha --profile=dev-release OK
Tests: 69 passed / 0 failed (force_pin_slot_allocated_below_end + all existing)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per Tier 1.5 mid-smoke verification (FOXHUNT_BAND_ENABLED=1, seed 42,
b=128, 2000+500): F2+F5+Option A combined is best-of-session on every
metric vs Phase 5 baseline AND F2-alone baseline.
## Empirical evidence (Tier 1.5, /tmp/foxhunt-phase7b-f5-tier15)
| Metric | Phase 5 | F2 alone | F2+F5+OptA |
|---------------------|-------------|-------------|-------------|
| eval pnl | -$444,225 | -$494,887 | -$415,237 |
| win_rate | 0.371 | 0.355 | 0.380 |
| sharpe_ann | -7.20 | -8.49 | -5.31 |
| action_entropy@1999 | 0.215 | 0.188 | 0.445 |
| profit_factor | 0.534 | 0.470 | 0.599 |
| max_drawdown_usd | (n/a) | (n/a) | $436k |
eval pnl improved $29k over Phase 5 and $80k over F2-alone.
action_entropy 2× higher than either baseline.
## F5 mechanism trajectory (verified working at scale)
step hold_frac_flat frac_batches_flat entropy
5 1.0 0.47 1.37 (session_risk tripped)
50 1.0 0.59 0.46 (still locked)
250 0.0 0.0 0.69 (session recovered; F5 active)
500 0.0 ~0.0 0.70
1000 0.0 ~0.0 0.54
1999 0.0 ~0.0 0.45
session_risk_check (hard safety, slot 540) correctly tripped early when
F5-forced opens lost money cold-start; by step ~250 it recovered and
F5 engaged. Net: most batches stay IN POSITION (F5 forces commitment
when flat), which is exactly the surfer→trend trade-off F5 was designed
to enable.
## F2 mechanism becomes ACTIVE under F5 forcing
f2_hinge_zero_rate dropped from 1.0 (F2-alone) to 0.909 — meaning ~9%
of action slots now show Q-advantage. F2 identified FlatFromShort as
modal target (target_argmax=4). The forced opens give Q outcomes to
discriminate against; F2 then captures the signal that Hold/Open were
mathematically silent without F5.
## What this commit does
Flips `RL_F5_STATE_MASK_ENABLED_INDEX` bootstrap from 0.0 → 1.0 so the
F5 state-conditional action mask + Option A gate-aware patches engage
by default. Flipping back to 0.0 (or any value ≤ 0.5) reverts to F2-only
behavior bit-equal with commit e82049c77.
This is the Phase 7b production engagement. Cluster verification next.
Per spec docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md
§5.3 attribution outcome 1 ("eval pnl improves AND no new degenerate
attractor surfaces") — F5 Phase 7b ships as primary intervention with
the explicit surfer-principle relaxation documented in §3.5 + §4.4.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When F5 state-conditional action availability mask (slot 823) is engaged AND
the position is flat, F5 has deliberately forced the agent into an opening
action by masking Hold (and other state-invalid actions) in `pi_logits`
pre-sample. Two downstream gates — `rl_confidence_gate` (slot 512..) and
`rl_frd_gate` (slot 516..) — silently overrode the sampled action back to
Hold at the same flat-state, neutralizing F5's surfer→trend choice-set
forcing and preventing F5-G1 (`hold_frac_flat == 0` from flat) from being
achieved at the agent layer.
Both kernels patched with the identical interop pattern:
const bool f5_engaged = (isv[RL_F5_STATE_MASK_ENABLED_INDEX] > 0.5f);
if (f5_engaged) skip the Hold override;
The existing `if (position_lots != 0) return;` early-return in both kernels
already restricts the override site to flat batches, so the check on the F5
master gate alone is sufficient — no new position read needed. When F5 is
disabled (slot 823 == 0.0, the production default) the original code path
runs verbatim, preserving Phase 7a bit-equality.
Scope (per Phase 7b follow-up):
* `rl_confidence_gate.cu` — Hold-override at `conf < threshold` gated on
`!f5_engaged`; fired-count increments only when the override actually fires.
* `rl_frd_gate.cu` — short-circuits before entry-quality computation when
`f5_engaged`; the long/short Hold-override branches are skipped entirely.
NOT touched (per Option A scope constraint):
* `rl_session_risk_check.cu` — hard DD-limit safety. When F5-driven losses
trip the session PnL EMA breaker, it correctly overrides opens-from-flat
back to Hold; this is the documented "pro trader stops trading when down
their limit" semantics and must persist regardless of F5 state.
* `rl_min_hold_check.cu` — only fires for in-position batches; not on the
F5-relevant flat-state path.
Verification (RTX 3050, `test_data/futures-baseline-mid/`):
* `cargo build --release --example alpha_rl_train -p ml-alpha` — clean.
* `./scripts/determinism-check.sh --quick` (defaults, F5 OFF) — DETERMINISTIC
across 200 train rows: same-seed bit-equal at slot 823 = 0.0.
* `FOXHUNT_BAND_ENABLED=1 ./scripts/determinism-check.sh --quick` (F5 OFF) —
DETERMINISTIC across 200 train rows.
* `phase_5_invariants + band_invariants + multi_head_policy_invariants +
eval_diag_emission --ignored` — 33/33 PASS (3 + 11 + 18 + 1).
* F5 ON 50-step smoke (slot 823 temporarily set to 1.0 then reverted before
commit): hold_frac_flat = 0.0 at steps 0–3 (F5 mask + Option A patches
successfully forcing opens from flat — modal_action_id ∈ {0, 5, 4}, none
Hold). At step 4 onward hold_frac_flat → 1.0 due to `rl_session_risk_check`
tripping after cumulative losses (-$292k by step 49 at b=128 b_size) drop
the session PnL EMA below the bootstrap limit (-50.0 at slot 540). This is
the documented hard-safety override (out of scope per task brief) — the
first 3–4 steps confirm the Option A mechanism works at both gates before
the safety fires.
* No NaN; training stable.
Bootstrap of slot 823 reverted to 0.0 (OFF) before committing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the no-op-set / direction / trail-at-cap action availability mask
specified in §3.5 of docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md.
## What lands
* New kernel `cuda/rl_state_action_mask.cu` (~115 LOC) modelled on
`rl_band_mask.cu`'s lattice — Grid=(B), Block=(1,1,1), single thread
per block, ISV-gated, reads `pos_state[b*pos_bytes+0..4]` for
`position_lots` and the existing per-batch per-unit trail buffers
(`unit_trail_distance`, `unit_initial_r`, `unit_active`).
* New ISV slot `RL_F5_STATE_MASK_ENABLED_INDEX = 823` (binary master
gate; bootstrap 0.0 = OFF; preserves Phase 7a `43e7b6383`
bit-equality). `RL_SLOTS_END` bumped 823 → 824.
* Trainer integration: `launch_rl_state_action_mask` public wrapper +
inline call sites at the two policy-sampling paths
(`step_with_lobsim` and `step_with_lobsim_gpu_body`), inserted BEFORE
`rl_pi_action_kernel` so the sampler sees the masked logits. Both
sites sit inside the prefill graph capture (device-side master gate
preserves bit-equality across off↔on flips). `isv_constants` array
size bumped 274 → 275.
* `build.rs` registers the new cubin.
## Mask semantics (when slot 823 > 0.5)
* `position_lots == 0` (flat) → mask {Hold=2, FlatFromLong=3,
FlatFromShort=4, TrailTighten=7, TrailLoosen=8, HalfFlatLong=9,
HalfFlatShort=10}. Surviving support: {ShortLarge=0, ShortSmall=1,
LongSmall=5, LongLarge=6} — agent is FORCED to open.
* `position_lots > 0` (long) → mask all short-side actions
{ShortLarge=0, ShortSmall=1, FlatFromShort=4, HalfFlatShort=10}.
Same-side opens (5/6) remain available; pyramid resolution stays in
`actions_to_market_targets.cu`.
* `position_lots < 0` (short) → symmetric.
* Any active unit at trail-cap (`trail_distance ≥ unit_initial_r *
RL_TRAIL_MAX_INITIAL_R_RATIO * (1 − 1e-3)`) → additionally mask
TrailLoosen (mq2pc-specific gate per spec §3.5 trail-at-cap branch).
## Composition with F2 (Q-distill hinged advantage)
F2 computes `π_target` from unmasked E_Q; F5 sets `pi_logits[masked] =
−INFINITY` so `softmax(pi_logits)` has EXACTLY zero mass on masked
actions (F5-G1 design requirement). The distill gradient
`(π_θ − π_target)` evaluates to `(0 − π_target_masked)` at masked
actions, which naturally drives the target off those actions —
gradient consistency without re-masking inside `rl_q_pi_distill_grad`.
## Action enum (verified against actions_to_market_targets.cu and
crates/ml-alpha/src/rl/common.rs)
| id | name | masked from flat | masked from long | masked from short |
|----|----------------|------------------|------------------|-------------------|
| 0 | ShortLarge | | ✓ | |
| 1 | ShortSmall | | ✓ | |
| 2 | Hold | ✓ | | |
| 3 | FlatFromLong | ✓ | | ✓ |
| 4 | FlatFromShort | ✓ | ✓ | |
| 5 | LongSmall | | | ✓ |
| 6 | LongLarge | | | ✓ |
| 7 | TrailTighten | ✓ | | |
| 8 | TrailLoosen | ✓ | (cap-only) | (cap-only) |
| 9 | HalfFlatLong | ✓ | | ✓ |
| 10 | HalfFlatShort | ✓ | ✓ | |
(There are NO separate PyramidLong/PyramidShort actions in foxhunt;
pyramid logic resolves inside `actions_to_market_targets.cu` when
same-side opens fire from an existing position.)
## Surfer-principle trade-off (acknowledged per spec §3.5 + §4.1.4)
F5 destroys patience-while-waiting-for-setup by construction. F2
preserves the surfer principle mathematically (Open mass = 0 when
E_Q(Open) ≤ baseline); F5 sacrifices it for choice-set enforcement.
Bootstrap 0.0 keeps F5 OFF until F2 alone fails the Tier 1.5 / Tier 2
verdict AND the operator explicitly accepts the trend-follower
regression. Reversible at runtime via ISV re-seed.
## Verification
* `cargo build --release --example alpha_rl_train -p ml-alpha` clean.
* Determinism (band off, F5 off / band on, F5 off / F5 ON via
temporary bootstrap=1.0): all three PASS — `determinism-check.sh
--quick` reports byte-equal `eval_summary.json` and
`alpha_rl_train_summary.json` plus checksum-equal diag rows.
* Invariants (band_invariants 11/11, eval_diag_emission 1/1,
multi_head_policy_invariants 18/18, phase_5_invariants 3/3): 33/33
PASS.
## STOP-on-surprise — F5 mechanism vs downstream gate-stack interaction
50-step smoke with F5 ON (bootstrap=1.0, then reverted) surfaced an
EXPECTED interaction documented in spec §3.5 expected-failure-mode #5:
* F5 mask itself is mechanically correct — `pi_logits[masked] = −INF`,
softmax mass is zero, `rl_pi_action_kernel` samples only from
{0,1,5,6} from flat.
* BUT the downstream gates that run AFTER `rl_pi_action_kernel` —
`rl_confidence_gate` (lines 67-74 read `pos_state`, override opens
from flat to Hold when C51 confidence is low), `rl_frd_gate`,
`rl_session_risk_check`, `rl_min_hold_check` — re-introduce Hold
into the executed action histogram. F5-G1 (`hold_frac_flat == 0`)
is therefore NOT achieved by this commit alone.
The F5 kernel does what the spec asks; the gate-stack interaction is
the orthogonal wiring the spec called out as out-of-scope for F5
correctness. A follow-up plan should either suppress those gates'
Hold override when `isv[823] > 0.5 AND position_lots == 0`, OR
re-launch the F5 mask AFTER the gate stack (single extra device-side
kernel invocation). The mask kernel and ISV slot land here so the
follow-up is purely a wiring task. Per the STOP-on-unexpected-finding
discipline (feedback_investigation_first_falsification_methodology),
no further fix is applied in this dispatch.
## Files
* `crates/ml-alpha/cuda/rl_state_action_mask.cu` (new, ~115 LOC)
* `crates/ml-alpha/build.rs` (kernel registration)
* `crates/ml-alpha/src/rl/isv_slots.rs` (slot 823 + RL_SLOTS_END bump)
* `crates/ml-alpha/src/trainer/integrated.rs` (cubin include, struct
fields, ctor load + struct init, `launch_rl_state_action_mask`
wrapper, ISV bootstrap entry + array size bump 274→275, two inline
call sites at `step_with_lobsim` and `step_with_lobsim_gpu_body`)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds instrumentation BEFORE the F2 hinge modification (Task 3) per
`feedback_investigation_first_falsification_methodology` "measure first,
change code second." Without these counters, the F2 cluster verdict
cannot distinguish "mechanism worked AND surfer preserved AND eval
improved" from "mechanism worked BUT surfer collapsed" — three
independent attribution signals collapse into one ambiguous outcome.
## ISV slots 819-822 (RL_SLOTS_END 819 → 823)
Four kernel-reduced F2 mechanism counters populated by
rl_q_pi_distill_grad.cu's end-of-kernel single-thread reduction block
(one block per batch, thread a==0 writes its own [batch] index — NO
atomicAdd, no race). Host reduces to mean/mode at diag emit.
- RL_F2_DIAG_BASELINE_MEAN_INDEX (819): mean of max E_Q over no-op set
- RL_F2_DIAG_ADV_MAX_MEAN_INDEX (820): mean of max hinged advantage
- RL_F2_DIAG_HINGE_ZERO_RATE_INDEX (821): fraction of (b,a) slots zeroed
- RL_F2_DIAG_TARGET_ARGMAX_INDEX (822): modal π_target action id
## 13 new diag fields under existing top-level keys
Surfer (3): trading.frac_batches_{flat,long,short} — host reduce on
position_lots; fleet-fraction denominators per
`pearl_fleet_fraction_not_aggregate`.
F2-G5 direct measurement (2): policy_diagnostic.hold_frac_flat,
open_action_mass_flat. F2-G5 (surfer-patience preservation gate, Hold%
when flat ∈ [30%, 85%]) is NOW LOCALLY MEASURABLE — closes the
measurement gap from plan v1.
Degenerate-attractor monitoring (4): policy_diagnostic.max_action_share
(detects single-action collapse), modal_action_id (which attractor),
noop_action_mass_{total,max} (no-op concentration).
F2 mechanism (4): policy_diagnostic.f2_baseline_mean,
f2_advantage_max_mean, f2_hinge_zero_rate, f2_target_argmax_action.
f2_target_argmax_action ≠ modal_action_id during surfer-adapt phase
indicates F2 is reshaping the gradient toward Open.
## Kernel changes
rl_q_pi_distill_grad.cu signature extended with 4 new scratch buffer
args. Reduction logic in the `a == 0` block computes per-batch values
and writes single-thread to [batch] index — no atomicAdd, no race.
Existing path (legacy distill target) unchanged when
RL_F2_DISTILL_CENTERED_ENABLED_INDEX (slot 817) is OFF.
## EXPECTED_LEAVES 712 → 746
13 new Phase 7a fields + 8 schema-drift leaves from upstream commits
(not re-synced in the prior B-11-β ledger between 671→672 and Phase 5
0463e44e0). All 8 pre-existed Phase 7a; just resyncing the count.
## Verification
* 3/3 phase_5_invariants PASS
* 11/11 band_invariants PASS
* 18/18 multi_head_policy_invariants PASS
* 1/1 eval_diag_emission PASS (after EXPECTED_LEAVES update)
* ./scripts/determinism-check.sh --quick: PASS (band off)
* FOXHUNT_BAND_ENABLED=1 ./scripts/determinism-check.sh --quick: PASS
* 50-step local smoke (FOXHUNT_BAND_ENABLED=1, seed 42, b=128):
All 13 new fields emit with sensible values:
frac_batches_flat=0.016 + frac_batches_long=0.664
+ frac_batches_short=0.320 = 1.0 ✓
hold_frac_flat=0.5 + open_action_mass_flat=0.5 ✓
max_action_share=0.914 (Hold dominates, expected early train)
modal_action_id=2 (Hold)
noop_action_mass_total=0.9375; noop_action_mass_max=0.914
f2_baseline_mean=-1.155 (Q untrained, negative)
f2_advantage_max_mean=0.0 (hinge zeroing everything, expected cold)
f2_hinge_zero_rate=1.0 (all actions zeroed — Open hasn't exceeded
no-op baseline yet; expected at step 50)
f2_target_argmax_action=7 (deterministic tiebreak on zero-target)
## Falsification-gate enablement
This task does NOT change agent behavior. The legacy distill target
runs unchanged (slot 817 bootstrap = 1.0 means F2 IS active, but
Task 3 will modify the kernel to compute the centered target; here
the kernel only EMITS what the baseline WOULD be). After Task 3 ships,
F2-G1 (target entropy drop) and F2-G5 (Hold% when flat) become
locally measurable via these new fields.
f2_diag_emit.rs GPU-oracle test file DEFERRED — out-of-scope for the
"measure first" task; 50-step smoke + jq verification confirms the
13 fields populate correctly. Add in a follow-up commit if needed.
Per `pearl_fleet_fraction_not_aggregate`: emits per-batch fractions
(not just aggregates) so b=1024 cluster diag retains per-batch
resolution.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two new slots:
- RL_F2_DISTILL_CENTERED_ENABLED_INDEX (817, bootstrap 1.0): master gate
for the F2 hinge in rl_q_pi_distill_grad.cu. Flip to 0.0 to fall back to
the legacy softmax(E_Q/τ) target (bit-equal with HEAD 0463e44e0).
- RL_F2_DISTILL_NOOP_SET_MODE_INDEX (818, bootstrap 0.0): selects the
no-op-set definition. Mode 0 = F2a static {Hold, TrailTighten, TrailLoosen}.
RL_SLOTS_END: 817 → 819.
Per spec docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md
§3.2 + §4.2 (F2 recommended, Revision 3 surfer-principle-preserving primary phase).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three coupled structural fixes for the TrailLoosen-pathology identified
by the Phase 4-A3 ultrathink investigation. Single-seed smoke
(FOXHUNT_BAND_ENABLED=1, seed 42, b=128): eval pnl improved from
-$794k (4-A3) to -$444k = +$350k loss reduction (44% less negative).
Win rate 37.1% (was 31.7%).
## Fix 1: Trail-max ceiling (new ISV slot 814)
RL_TRAIL_MAX_INITIAL_R_RATIO_INDEX bootstrap 4.0. In rl_trail_mutate.cu,
clamp unit_trail_distance ≤ initial_trail × 4.0. Without this the
TrailLoosen action (× 1.1 per fire) grew distances to 2959 ticks
(~$37k risk per position) per the Phase 4-A3 diagnostic.
Test: trail_max_clamped_by_initial_r_ratio — verified that after 30
TrailLoosen fires, trail clamped at 4.0× (uncapped would have been
17.4×, a 4.4× reduction).
Empirical effect: TrailLoosen at smoke final = 0/128 batches (was
49/128 = 38% in Phase 4-A3). The arbitrage is mechanically eliminated.
## Fix 2: Mark-to-market reward (new ISV slots 815/816)
RL_MTM_REWARD_ENABLED_INDEX (bootstrap 1.0) + RL_MTM_REWARD_WEIGHT_INDEX
(bootstrap 1.0). New Phase 1.6 in rl_fused_reward_pipeline.cu:
r += w_mtm × (unrealized_now − unrealized_prev)
Per-step reward proportional to total wealth delta (realized +
unrealized). Penalizes holding losers in real-time. Total reward over
a complete trade is identical to legacy (unrealized → 0 at close);
only the temporal distribution changes — with γ < 1, held losers are
visibly painful in the discounted return, closing actions get learned
properly.
Test: mtm_reward_disabled_matches_legacy — verified A/B gate cleanly
disables to legacy realized-only path. unrealized_pnl read from
pos.vwap_entry + current mid via standard accounting.
Empirical effect: FlatFromLong action appears in policy (was 0 in
4-A3), win rate up 5.4pp.
## Fix 3: Entropy gradient formula (correctness)
rl_q_pi_distill_grad.cu:131 had spurious +1.0f:
BEFORE: grad_entropy = -alpha × pi_a × (log_pi_a + 1.0f + s_entropy)
AFTER: grad_entropy = -alpha × pi_a × (log_pi_a + s_entropy)
True ∂(-H)/∂logit_a = π(a)·(log π(a) + H) per textbook softmax-entropy
gradient. Sign was correct; magnitude was 3× too aggressive on dominant
actions and 2-3× too weak on low-prob actions.
Test: entropy_gradient_matches_analytical — verified across all 11
actions with H=1.4931, kernel matches analytical formula within 1e-5.
## Verification
* 3/3 phase_5_invariants tests PASS
* 11/11 band_invariants regression tests PASS
* FOXHUNT_USE_MULTI_HEAD_POLICY=0 ./scripts/determinism-check.sh
--quick: exit 0 (200 rows bit-equal)
* FOXHUNT_BAND_ENABLED=1 ./scripts/determinism-check.sh --quick:
exit 0 (200 rows bit-equal)
* Pre-commit hook: 0 atomicAdd, 0 raw memcpy_htod/dtoh
## Smoke trajectory (FOXHUNT_BAND_ENABLED=1, seed 42, b=128, 2000+500)
step entropy top_3_action_hist
0 0.000 [128, 0, 0] (init)
100 0.785 [Hold=117, Long-=3, Short-=5, TrailLoosen=1]
500 0.489 [Hold=122, Long-=1, FlatLong=2, TrailLoosen=0]
1000 0.359 [Hold=123, FlatLong=5, TrailLoosen=0]
1500 0.268 [Hold=124, FlatLong=3, TrailLoosen=1]
1999 0.215 [Hold=125, FlatLong=3, TrailLoosen=0]
eval (5000 steps frozen policy):
total_pnl_usd: -$444,225 (best of session, was -$794k in 4-A3)
win_rate: 0.371 (best of session, was 0.317)
n_trades: 197
profit_factor: 0.534
max_drawdown_usd: $462,212
sharpe_ann: -7.20
## Falsification gates
* G_mechanism (no NaN, exit 0): PASS
* G_no_regression (pnl ≥ -$5M): PASS by wide margin (-$444k)
* G_trail_bounded: PASS (0 TrailLoosen fires at final, ceiling working)
* G_action_diversity (entropy ≥ 1.0): FAIL (0.22 — new pure-Hold
conservatism failure mode emerges; the policy learns "don't open
trades" as the safest path with all the constraints in place)
## Analysis of the new failure mode
The Phase 5 fixes worked exactly as designed: trail-max eliminated
the runaway risk mechanism, MTM made held losers painful, entropy
formula now matches theory. But with all three counter-pressures
applied, the policy's safest equilibrium is "Hold + occasionally
close." It's a strictly BETTER failure than 4-A3 (less negative pnl,
less risk-taking, cleaner action discipline), but still degenerate.
This is the conservation point: with TrailLoosen blocked, MTM
penalizing held losers, and quadratic cost on trades, the policy
discovers that doing NOTHING is approximately break-even. Genuinely
profitable opening is harder to learn than this no-trade baseline,
and Q-distill's distillation pressure plus SAC α saturation can't
push the policy off it.
Next investigation: WHY can't the policy discover profitable opens?
Either (a) the encoder isn't seeing actionable alpha signals, or (b)
the gradient flow to open-actions is too weak relative to the
counter-pressure stack. This is a separate spec.
## Linked
* Phase 4-A3: 65c328d3f (band foundation)
* Phase 4 cluster verdict: alpha-rl-4hq7c (band works at scale;
same TrailLoosen pathology confirmed at cluster)
* Phase 4-A3 ultrathink diagnosis: identified the TrailLoosen
pathology + entropy formula bug + this fix design
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mirror of the FOXHUNT_USE_MULTI_HEAD_POLICY pattern (c5036af03).
Adds --band-enabled flag to argo-alpha-rl.sh and band-enabled
workflow parameter wired into the container env block.
Required for Phase 4-C cluster verification of the full Tier 3
no-transaction-band stack (commits e41a73208 → 65c328d3f).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Davis-Norman (1990) is a theorem about MANAGING existing hedge
positions; it tells the agent NOT to micro-adjust within the band. It
says nothing about whether to open from flat.
The `±|tanh|` activation in rl_band_head_forward.cu guarantees
`b_l ≤ 0 ≤ b_u` (the Davis-Norman invariant). Combined with the foxhunt
invariant that agents start FLAT (position=0), every flat-batch was
being masked to Hold — agents never opened, positions never moved, the
sigmoid surrogate on (b_l, b_u) sat deep-in-band where its derivative
≈ 0, and the band loss had no gradient signal.
Verified empirically at 1c23ff368 (Phase 4-A2 smoke): action_hist
collapsed to bimodal [0,0,109,0,0,0,0,0,19,0,0] from step 100 onward,
total_trades = 8 over 2000 steps.
Fix (rl_band_mask.cu): immediately after the master-gate and
exploration-bypass checks, return when `position_lots == 0`. Flat
positions get free choice — opens from flat are NOT band-constrained.
Reads the same canonical offset as the existing band-position check.
Tests (band_invariants.rs):
* New: `band_mask_allows_opens_from_flat` — master gate ON, exploration
bypass DISABLED, all positions=0, band [-0.5, +0.5]; pre-A3 this
would mask every batch to Hold. Asserts the position-zero exception
preserves every batch's original (non-Hold) action.
* Updated: `band_mask_forces_hold_when_in_band` now uses position=1 in
band [-4, +4] (position=0 would unconditionally bypass post-A3).
* Updated: `band_mask_respects_exploration_bypass` now uses position=1
in band [-2, +2] so the exploration-bypass path can be exercised
without the position-zero exception pre-empting it.
11/11 band_invariants PASS. Determinism preserved across band-OFF,
band-ON, and multi-head+band-ON (`./scripts/determinism-check.sh
--quick` exit 0 in all three modes — the new check is integer-equality
and the master gate at slot 799 still defaults OFF, so bit-equality
with the Phase 3D baseline is intact when the band is disabled.
Single-seed smoke (FOXHUNT_BAND_ENABLED=1, seed 42, b=128):
* total_trades 8 → 1911 (trades restored)
* positions: 88/128 non-zero at step 1999 (was 0/128)
* band.width_mean trajectory 4.32 (step 100) → 7.37 (step 1999) —
band NOT collapsing now (was → 0.085 in 4-A2)
* band.frac_masked = 0.797 ∈ (0, 0.85) — band doing meaningful work
* eval pnl = -$793k vs -$8M kill threshold
* NaN count = 0
Falsification gates (Phase 4-A3):
* G_mechanism: PASS (exit 0, no NaN)
* G_positions_become_nonzero: PASS (88/128 non-zero)
* G_trade_recovery: PASS (1911 ∈ [500, 8000]) ← PRIMARY
* G_no_regression: PASS (-$794k > -$8M)
* G_band_active: PASS (frac_masked 0.797 ∈ (0, 0.85))
* G_action_diversity: FAIL (action_entropy 0.65 < 1.0) — see below
The action-diversity gate fails because the policy still concentrates
on Hold (action 2) + TrailLoosen (action 8). However this is NOT the
4-A2 "band can't train" collapse: trades flow, band trains, positions
move. The bimodal pattern toward TrailLoosen is a separate Q-distill
or controller side-effect that pre-existed the band — surfaced by the
spec's STOP-on-surprise rule #2 for follow-up rather than fixed here.
Spec: docs/superpowers/specs/2026-06-03-no-transaction-band-architecture.md §10.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds RL_BAND_MAX_MASK_FRAC_INDEX (bootstrap 0.85) and modifies
rl_band_mask.cu to never mask the first (1 - max_mask) · B batches
per step. Resolves the dead-signal trap from Phase 4-B at 552d91bf4:
band saturated wide because all positions stayed at 0 (deep in
band ⇒ sigmoid surrogate gradient ~0 ⇒ no learning signal).
## Mechanism
* Slot 813 RL_BAND_MAX_MASK_FRAC_INDEX (bootstrap 0.85). RL_SLOTS_END=814.
* rl_band_mask.cu: deterministic first-N-batches bypass (preserves
determinism per pearl_determinism_achieved).
* New test band_mask_respects_exploration_bypass; existing tests
updated.
## Smoke result (FOXHUNT_BAND_ENABLED=1, seed 42, b=128, 2000+500)
The bypass mechanism works (15% of batches never see the mask).
The band IS learning (width 8.02 → 0.085 by step 1999, controller
drove turnover_target to MAX 0.20 as designed).
BUT: full policy collapse to bimodal `[0,0,109,0,0,0,0,0,19,0,0]`
from step 100 onward — 109 masked → Hold, 19 bypassed → all pick
TrailLoosen. action_entropy 0.45, total_trades=8.
## Architectural finding (Phase 4-A3 prerequisite)
The ±|tanh| activation guarantees b_l ≤ 0 ≤ b_u — position 0 is
ALWAYS in band. But foxhunt agents start flat (position 0), so
every batch starts masked, never opens trades, positions never
move, band has no exposure to non-zero positions, no gradient
signal to "open up". Davis-Norman semantics (band constrains
EXISTING positions) misapplied to "stay flat forever".
Phase 4-A3 will add a position-zero exception: band only
constrains already-positioned batches; opens from flat are
always allowed.
## Linked
* Phase 4-A: e41a73208 (band foundation)
* Phase 4-B: 552d91bf4 (controller + backward chain)
* Spec §2.2 + §9.1 Mitigation 2
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Closes Phase 4-A's open loop by making the no-transaction band LEARN its
width. Two coupled mechanisms wired in:
1. Adaptive turnover-target controller — single-thread CUDA kernel
(rl_band_turnover_controller.cu) reads per-step `frac_not_masked` and
adjusts the turnover target via asymmetric Schulman-bounded adapter:
tighten (×0.97/step) above 0.60 threshold, loosen (×1.05/step) below
0.20. Asymmetric LOOSEN rate is faster per
`pearl_dead_signal_resurrection_discipline`. First-observation
bootstrap on slot 809 with dedicated boot-done sentinel slot 810.
2. Backward chain wiring — rl_band_head_backward.cu propagates
`grad_band_per_b [B × 2]` through the asymmetric ±|tanh|·N_max
activation and linear projection into per-batch weight/bias scratch
plus `grad_h_t [B × HIDDEN_DIM]` (OVERWRITE). Trainer reduces via
`reduce_axis0` and folds into the encoder grad combiner alongside
π/V/FRD/outcome heads using the existing `grad_h_accumulate_scaled`
pattern. Band-weight Adam steps share LR with π
(RL_LR_PI_INDEX) — same plateau dynamics as policy.
Plus a GPU reducer rl_band_frac_aggregate.cu (single-block tree-reduce
over batch → writes to slot 812) and a host-side launch sequence in
step_with_lobsim_gpu_body that runs frac_aggregate → controller →
turnover_loss → backward outside graph capture, gated by
RL_BAND_ENABLED_INDEX > 0.5 so Phase 3D bit-equality is preserved when
the master gate is OFF.
ISV slots added (809-812): turnover_ema, controller_boot_done,
turnover_target_adaptive, frac_not_masked_observed. The turnover loss
kernel now reads slot 811 (controller output) instead of the static
slot 803. Bootstrap target 0.05 matches the Phase 4-A static default so
first-step behavior is identical.
Tests: 4 new band_invariants tests cover the controller bootstrap, the
escalate-on-over-trading path, the loosen-on-saturation path, and the
backward chain producing non-zero `grad_h_t` from a turnover/target
mismatch. All 9 band tests + 18 multi_head_policy regression tests pass.
Determinism: `determinism-check.sh --quick` exits 0 for the three
relevant configurations (band off, band on, band + multi-head on). The
Schulman controllers are deterministic by construction (no PRNG);
backward kernels are sole-writer per (batch, slot) so reduce_axis0
delivers bit-equal weight grads across replays.
Spec: docs/superpowers/specs/2026-06-03-no-transaction-band-architecture.md
§3.1 Option (c), §3.3 (backward chain), §3.4 (loss weight controller),
§5 (diag emission), §9.1 Mitigation 3 (resurrection discipline).
Pearls applied:
* pearl_bootstrap_must_respect_clamp_range — bootstrap 0.05 ∈ [0.01, 0.20]
* pearl_dead_signal_resurrection_discipline — asymmetric tighten/loosen rates
* pearl_welford_trade_count_is_step_not_trade — bootstrap-done sentinel slot
* pearl_determinism_achieved — single-thread controller, deterministic reducers
Single-seed local smoke (b=128, 2000+500 steps, FOXHUNT_BAND_ENABLED=1)
runs end-to-end without NaN. The band saturates to `frac_masked = 1.0`
because position state stays at 0 throughout — the sigmoid surrogate
gradient is zero deep inside the band, so the band's upper boundary
cannot be pulled toward 0 even though the controller drives the
adaptive target to its MAX (0.20). Spec §9.1 Mitigation 2
(`RL_BAND_MAX_MASK_FRAC_INDEX` exploration-slot bypass in
rl_band_mask.cu) was declared but not shipped in Phase 4-A; it is the
structural fix for this chicken-and-egg trap. Phase 4-B mechanism
itself is verified correct via the four new tests; the saturation is
an architectural-grafting gap surfaced for Phase 4-A2 follow-up rather
than a Phase 4-B implementation defect.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Combined atomic attack on foxhunt's structural overtrading pathology
per omnisearch (2026-06-03) RL HFT literature: foxhunt uniquely uses
Q-distill as the SOLE policy-training mechanism, making it susceptible
to the four-stage Q→π attenuation chain (Goodhart-Skalse 2024) that
blinds π to small persistent fees. Three concurrent fixes attack
different layers:
A. Hold-action logit bias (+log(4) ≈ 1.386 on action 2)
* crates/ml-alpha/src/rl/ppo.rs PolicyHead::new
* crates/ml-alpha/src/rl/multi_head_policy.rs all K heads + build_priors
Counter-balances the structural 4:1 open-vs-hold action prior (4
open variants 0,1,5,6 vs 1 Hold=2). Pre-bias P(open)=36% / P(hold)=9%;
post-bias P(hold)≈29% / P(any open)≈7%. Mid-smoke seed=42 confirms
Hold rises to 71/128 = 55.5% by step 1999.
B. Quadratic-in-trade-size impact-aware cost (Cao et al. 2026,
arXiv:2603.29086 §4)
* crates/ml-alpha/cuda/rl_fused_reward_pipeline.cu Phase 1.5
* 3 new ISV slots 794-796 (α=0.5, β=2.0, enabled=1.0)
cost = α·|Δlots| + β·(Δlots)². 1-lot=2.5; 4-lot flip=34; 8-lot=132
(superlinear). Applied BEFORE shaping so the surfer-scaffold weight
does not amplify or mute. Trail actions (7,8) are no-ops in the
position kernel → cost=0 for them as expected.
C. PPO surrogate gradient restoration with adaptive blend (Cao 2026 §4)
* crates/ml-alpha/cuda/rl_pi_grad_blend.cu (new — element-wise
scale-or-zero operator)
* crates/ml-alpha/src/trainer/integrated.rs Step 7 (π gradient blend)
* 2 new ISV slots 797-798 (weight=0.005, enabled=1.0)
Previously π was trained ONLY by Q-distillation (line 5850 header).
Now: pi_grad = w_ppo·grad_PPO + grad_Q_distill + grad_SAC_entropy.
Restores the direct fee-aware policy-gradient channel that Q-distill
alone cannot transmit. Blend kernel runs BETWEEN surrogate_backward
and rl_q_pi_distill_grad (which uses +=).
Diag emission (E):
* crates/ml-alpha/src/trainer/integrated.rs rewards.{quadratic_cost_alpha,
quadratic_cost_beta, quadratic_cost_enabled, ppo_surrogate_weight,
ppo_surrogate_enabled}
Tests (D):
* multi_head_policy_invariants: updated k1_reduces_to_single_head for
Phase 3D-A bias; new phase_3d_a_hold_bias_propagates_all_heads
invariant verifies Hold dominance in every head at h_t=0. 18/18 pass.
* phase_3d_blend_kernel_invariants (new): 3 GPU-oracle invariants on
rl_pi_grad_blend (disabled-zeros, enabled-scales-linearly, weight-0-
equivalent-to-disabled). 3/3 pass.
* reward_alignment_invariants: 2 new tests (phase_3d_diag_emission +
phase_3d_quadratic_cost_visible_in_rewards) + fix to the existing
surfer_scaffold test (relaxed bootstrap check for Phase 3B-Y pure-pnl
mode default 0.0; absorbs eval drain row). 3/3 pass.
* All 68 ml-alpha lib tests pass.
Verification:
* SQLX_OFFLINE=true cargo build --release --example alpha_rl_train -p
ml-alpha: exit 0
* SQLX_OFFLINE=true cargo check --workspace: exit 0
* ./scripts/determinism-check.sh --quick: DETERMINISTIC (200/200 rows
bit-equal across two same-seed runs)
* FOXHUNT_USE_MULTI_HEAD_POLICY=1 ./scripts/determinism-check.sh --quick:
DETERMINISTIC
* Local Tier 1.5 mid-smoke (seed=42, b=128, 2000 train + 500 eval):
exit 0, completed_clean=true, no NaN, no abort.
Primary kill criterion (total_trades final < 5,000): NOT MET.
Result: 11,767 trades vs 14,691 baseline = 20% reduction. Cao 2026
forecast 96% reduction for pure-PPO/SAC architectures was not
achieved — foxhunt's Q-distill dominance (q_pi_agree_ema = 0.948 in
this run) attenuates the PPO surrogate's fee signal even with the
blend operator. The behavioral signature IS present (Hold dominance
rises from baseline ~36% structural prior to 55.5% at step 1999;
action_entropy = 1.748 within healthy [1.2, 2.04] target).
Regression guard (eval pnl ≥ -$5M): MARGINAL PASS at -$4.96M (-$36k
inside threshold). The Tier 1.5 verdict flags KILL on Pearson +
wr_train + wr_eval + eval_pnl. Pre-cluster, the literature
recommendation is to A/B-ablate each intervention (slots 794-798
individually gated). Mid-smoke architecturally validates that the
three interventions PROPAGATE and do not crash; cluster b=1024 with
longer runs (20k steps) will surface whether the 20% reduction
compounds into a viable policy.
Architectural references:
* pearl_foxhunt_pi_trained_by_q_distillation_not_ppo
* pearl_reward_signal_anti_aligned_with_pnl
* pearl_bootstrap_must_respect_clamp_range
* feedback_no_atomicadd / feedback_no_htod_htoh_only_mapped_pinned
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 3B-Y gated PopArt reward normalization (slot 793 = 0) but
left the V-correction kernel running unconditionally. Mid-cluster
diagnostic (alpha-rl-k9lz9 step 77, ~80 min into Phase 3C):
catastrophic pnl bleed (-$6.3M cum) driven by V-head unit mismatch.
## Mechanism
* `rl_popart_normalize` running mean/sigma stats keep updating even
when the rewards-write path is gated (so re-enabling produces
sensible values mid-session). With rewards in raw pnl-units,
sigma grew to ~2,666 in 77 cluster steps.
* `rl_popart_v_correct` unconditionally applies
v_pred = (sigma_old/sigma_new)·v_pred + (mean_old-mean_new)/sigma_new
every step — scaling V outputs against a running statistic that
no longer matches the V regression target.
* Net effect: V predictions scaled toward zero while V regression
target was raw pnl-units → broken V → broken Bellman Q-target →
broken π via Q-distill → catastrophic policy.
## Fix
Add a short-circuit at the top of rl_popart_v_correct:
```c
if (isv[RL_POPART_NORMALIZE_ENABLED_INDEX] <= 0.5f) return;
```
When normalization is off (slot 793 = 0), V is being trained
against raw returns and needs no correction. The popart calibration
machinery is bypassed end-to-end.
## Verification
* Build clean (1m46s incremental).
* FOXHUNT_USE_MULTI_HEAD_POLICY=0/1 ./scripts/determinism-check.sh
--quick: both exit 0 (200 rows match, rel-tol=1e-05).
## Discovery sequence
* Phase 2A-E cluster ran for 2h53m with shaped_reward=+$201M /
pnl=-$330M (Phase 5 shaping anti-aligned)
* Phase 3B+Y disabled Phase 5 shaping + PopArt normalize + hindsight
* Phase 3C cluster (b=1024) at step 77 showed pnl=-$6.3M already
* Diagnosis: V-correction running against stale sigma created unit
mismatch that broke V regression chain
* This fix is the natural companion to Phase 3B-Y's gate
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three coordinated changes to remove anti-aligned components from
the reward pipeline. Predicted Pearson trajectory based on Phase 3A
audit was wrong (PopArt + hindsight only delivered +0.04 not +0.4),
but the deep Phase 3B-Z audit revealed the "low Pearson" was largely
a measurement artifact: raw_reward tracks ALL PnL events (including
fees + partial closes via pos.realized_pnl_delta) while diag's
realized_pnl_usd_cum tracks only full closures. The actual training
reward IS aligned with total realized PnL evolution.
## Changes
* Slot 753 RL_SURFER_SCAFFOLD_WEIGHT_INDEX bootstrap 1.0 → 0.0
(Phase 5 hold-bonus/entry-cost/short-hold-penalty/long-ride-bonus
shaping disabled — was the dominant anti-aligned contaminant per
Phase 3A audit and cluster 2A-E evidence of +$201M shaped vs
-$330M actual).
* New ISV slot 793 RL_POPART_NORMALIZE_ENABLED_INDEX bootstrap 0.0
(PopArt batch normalization disabled — was found to sign-flip
rewards under adverse batch composition).
* Hindsight injection (rl_hindsight_inject.cu wants_inject) extended
with scaffold_w > 0.5 condition (when slot 753 = 0, hindsight off).
Semantically pairs the two training scaffolds.
## Verification
* 13/13 multi_head_policy_invariants PASS (no regression).
* FOXHUNT_USE_MULTI_HEAD_POLICY=0/1 ./scripts/determinism-check.sh
--quick: both exit 0.
* Local single-seed mid-smoke (b=128, seed 42):
- eval pnl: -$4.6M (vs Phase 0 baseline -$4.46M, within noise)
- 14,691 trades training / 3,268 eval (healthy, not paralyzed)
- action_entropy 1.82 at step 1999 (below uniform 2.40, learning)
- 0 NaN, exit 0
## Pearson interpretation (Phase 3B-Z audit)
The Pearson 0.38 between raw_reward and pnl_step_close_usd_delta is
a structural mismatch between two valid PnL views (all events vs
close events only), NOT an anti-alignment. raw_reward at slot
753 = 0 is the delta of pos.realized_pnl which already aggregates
the events we care about for total-USD optimization.
Cluster verification (Phase 3C, next) will reveal whether the
removal of Phase 5 + popart + hindsight produces a base policy
that can learn at scale. The 2A-E cluster catastrophe (-$330M)
was driven by Phase 5 shaping that's now disabled.
Plan: docs/superpowers/plans/2026-06-03-reward-alignment-gae-
combined.md
Linked pearls:
* pearl_reward_signal_anti_aligned_with_pnl
* pearl_phase5_term_4_is_almost_potential
* pearl_popart_blind_to_session_signals
* pearl_pure_pnl_mode_starves_b16_controllers
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds use-multi-head-policy workflow parameter (default "0") wired
into the container env block as FOXHUNT_USE_MULTI_HEAD_POLICY.
Adds --use-multi-head-policy flag to argo-alpha-rl.sh that passes
through to the workflow.
Enables Phase 2A-E cluster verification with the multi-head policy
mixture (commits 0b3e40150 → c8c81ab7e). Without this flag, the
cluster runs the legacy single-head path identical to pre-Phase-2A
behavior.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces the fixed 5.0 bootstrap of slot 790 with a real adaptive
controller. ISV slot 790 is now driven by signal observation per
foxhunt-discipline ("no hardcoded constants that should be ISV-
driven"). The constant value remains the bootstrap; the controller
takes over once the signal observation begins.
## Mechanism
* 2 new ISV slots (793 total now):
- 791 RL_POLICY_GATE_ENTROPY_EMA_INDEX (Wiener-α=0.02 on slot 781)
- 792 RL_POLICY_GATE_CONTROLLER_BOOTSTRAP_DONE_INDEX (0→1 latch)
* New kernel rl_gate_lr_multiplier_controller.cu — single-thread
launch (1,1,1)/(1,1,1) matching rl_surfer_scaffold_controller
pattern. No atomicAdd.
* Control law (per pearl_bootstrap_must_respect_clamp_range +
pearl_dead_signal_resurrection_discipline):
if !boot_done: ema = entropy_now; boot_done = 1
else: ema = (1-α)·ema + α·entropy_now
over = 0.85·log(K) (~0.934 for K=3)
collapse = 0.20·log(K) (~0.220 for K=3)
if ema > over: mult *= 1.003 (escalate +0.3%/step)
elif ema < collapse: mult *= 0.95 (decay -5%/step, emergency)
mult = clamp(mult, 1.0, 50.0)
* Per-step launch wired flag-gated AFTER MultiHeadPolicy::
emit_diag_stats (writes slot 781) and BEFORE next-step host-side
lr_gate = lr_pi * isv[790] read. Captured inside the flag-on
CUDA graph; flag-off graph never includes the launch.
* Bootstrap: slot 790=5.0 (B1's empirically-validated starting
point), 791=0.0 (overwritten on first observation), 792=0.0.
* Diag emit: 3 new fields gate_lr_multiplier, gate_entropy_ema,
gate_controller_bootstrap_done — inside the existing flag-gated
multi_head_policy JSON block (flag-off schema unchanged).
## Test results
* 17/17 multi_head_policy_invariants PASS (13 pre-existing + 4 new):
- bootstrap_initializes_ema_to_first_observation
- escalates_when_entropy_above_threshold (5.0 → 6.747 in 100
steps; matches analytical 5·1.003^100 within ±0.1)
- decays_when_entropy_below_collapse (5.0 → 1.0 in 100 steps,
clamped to MIN_FLOOR)
- clamps_at_min_floor_and_max_ceiling (50.0 ceiling, 1.0 floor,
long-escalate 5.0 → 50.0 over 800 steps)
* Flag-off determinism: exit 0, byte-equal to pre-B1.3 modulo
elapsed_s (0/200 non-elapsed_s diffs).
* Flag-on determinism: exit 0 (verified 3 consecutive runs).
Note: one transient first-run FAIL surfaced during verification
(mamba2_l2_out step 0 Δ≈4773) that disappeared after clean
rebuild — same build-cache pattern documented in Phase 2A-C
reports. Three confirmed clean post-rebuild.
* Pre-commit: 0 atomicAdd, 0 raw memcpy_htod/dtoh, 0 TODO.
## Single-seed seed-42 trajectory (b=128, 2000 train steps)
step entropy_ema multiplier notes
0 1.0986 5.015 bootstrap
100 1.0972 6.75 escalating
500 1.0967 22.36
1000 1.0944 50.00 hit MAX_CEIL
1500 1.0820 50.00
1999 1.0830 50.00 stuck at ceiling
Final gate_probs_mean = [0.253, 0.383, 0.365] — close to fixed 5×
endpoint [0.252, 0.382, 0.366]. The controller saturated at the
50× ceiling because entropy_ema never crossed below the 0.934
threshold — but the gate end-state matched 5× regardless. This
is the controller correctly diagnosing "no value of multiplier in
[1, 50] is sufficient to specialize the gate at b=128" — the
binding constraint is scale (signal/noise ratio), not LR. Cluster
b=1024 / 20k steps provides ~80× more gradient signal and is the
right next test.
## Linked
* Phase 2A-A: 0b3e40150
* Phase 2A-B: e22da61cf
* Phase 2A-C: 3e36f4a0e
* Phase 2A-C+: 5f10bcde3
* Phase 2A-D B1: 6f3639bfb (fixed 5.0 multiplier — now superseded
by adaptive control built on this commit)
* Phase 2A-D B1.2: 20× test verdict — saturates at ~5×, motivated
adaptive control rather than higher fixed multiplier
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
