Daily no-capital forward test. Pulls live Binance USDT-perp funding (public API, no key),
applies validated filter (liquid + trailing-30d mean daily funding > 5bp, hysteresis), computes
intended equal-weight delta-neutral book, tracks realized funding on prior book -> forward track
record on unseen data. Modes: paper|status. cwd-independent, state-persisted, cron-ready.
Day-1 live run: 19 of 247 liquid perps qualify (regime filter working in real time). Cron added
at 01:17 UTC daily (after 00:00 funding settlement). Validates the one open question: does
positive funding carry persist forward? Honest scope: tracks gross carry + turnover; does NOT
model basis vol (needs live fills) or counterparty risk (un-simulable).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Investigated crypto funding carry (researchers' flagged retail path to >1 Sharpe). DELTA-NEUTRAL
cash-and-carry (short perp+long spot, collect funding, price cancels) -- different from the
directional carry trap. Funding positive 75pct of time. Backtest net of cost: APR 9.9pct, raw
Sharpe +5.68, maxDD -7.4pct, survives to 20bp cost, worst month -1.5pct (delta-neutral held).
Discipline finds: (1) real; (2) 5.68 inflated (funding-only model, 1.8pct vol; real basis vol
-> honest ~2-3 Sharpe = literature's 1.5-2.5); (3) regime-dependent + DECAYING (negative 2022,
2025, 2026). Caveats: counterparty/exchange tail (FTX -100pct, not in backtest = real killer),
decay, operationally real. The ONE genuine path past the 0.7 ceiling -- but it's crypto +
counterparty tail is the price of admission. The journey converges: higher Sharpe exists,
reachable, but lives where the user didn't want to go with a tail the Sharpe doesn't show.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The strategy class we never tried: harvest structural premia + risk system, not predict.
Diversified futures + risk-parity/vol-target/trend; clean 5-asset-class version (ES/ZN/GC/CL/BTC).
Result: plain 60/40 (Sharpe +0.72, 2010-2026) BEATS risk-parity+vol-target (+0.46) and RP+trend
(+0.33, trend hurts); 5-asset+crypto RP only ties 60/40 and loses to buy-hold equity. Meta-pattern
now complete in BOTH games: simple beats/equals sophisticated in prediction AND harvesting.
Constructive deliverable: a simple premium harvest (60/40 / risk-parity) IS a real deployable
robust strategy (~0.5-0.72 Sharpe, low DD, no prediction, minimal complexity). The engine's
sophistication was never the return-generator -- deployable path is light (harvest + risk overlay),
engine's value is infra/discipline/product. (Also: CAISO intraday gate blocked by OASIS plumbing.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First repurpose target with real legs. DE day-ahead battery arbitrage (free Energy-Charts,
2019-2025, 2464 days). 4h/1MW/85pct: perfect-foresight e93k/yr/MW (>> e50-80k viability),
naive only e10k -> forecasting/RL gap = 89pct of value. Per-year GROWING (e28k 2019 ->
e141k 2025) = not competed away (batteries slow to build = operational moat). Structurally
opposite of trading: real economic value (not zero-sum), growing (not arbitraged), and 89pct
of value is forecasting/RL = exactly the engine's job. Caveats: perfect-foresight is upper
bound (realistic ~60-85pct capture -> ~e55-80k/yr/MW); needs a battery (capital) or sell the
dispatch software to operators (engine-as-product) or aggregator. NEXT: forecaster + RL
dispatch, measure realized-capture vs foresight.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ran the quant-fund method (gradboost+ridge+IC+equal combining 13 weak cross-sectional
signals) on liquid US equities. Single-split gradboost looked amazing (OOS +1.14, DSR 0.62)
but leak-free WALK-FORWARD diagnostic: gradboost OOS predictive IC = 0.0041 (statistically
ZERO; no leak; successful equity ML is 0.02-0.05). Single-split was overfit; WF +32 Sharpe
was a variance-degeneracy; equal/ridge/IC all fail OOS. The ML combination does NOT work on
efficient equities -- not because the ML is bad (works perfectly) but because there's no
signal (IC 0.004) to combine. DEFINITIVE answer to 'millions of LOC of ML, why nothing?':
the ML is not the missing piece, MARKET ACCESS is. Pointed the actual RenTech/TwoSigma
method at liquid equities -> IC 0.004 = noise. ML amplifies signal, cannot create it;
efficient markets have none. Crypto (less-efficient) is the one place the same machinery
finds robust signal. Sophistication was never the bottleneck.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bought XSP (mini-SPX) options ohlcv-1d 2013-2026 = $66.86 (hard-capped $70, get_cost-gated,
year-chunked vs 504, 2025 backfilled) + SPY $0.01. CORRECTION: index VRP is NOT $900 (that was
SPY root); clean instrument XSP $66.86/13y or SPX $143/13y. THREE measurement attempts all gave
spurious NEGATIVE VRP (RV 28-33% vs implied ~16%) = MEASUREMENT ERROR not finding (contradicts
decades of SPX VRP evidence). Bugs: noisy parity underlying; ~469/2800 days survive -> multi-day
gaps inflate RV; SPY!=XSP divergence corrupts ATM. Root: XSP EOD ohlcv too sparse to reconstruct
clean underlying+ATM-IV. Implied (16.7%) reads right. Frontier real+affordable+in-hand but proper
extraction is a real options-quant build (denser SPX +$76 or IV-surface w/ quotes), not a gate.
Did NOT record -2.7 as a result (artifact). Crypto momentum+VRP remains only deploy-grade edge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Decomposed the equity low-vol +0.74. Deployable half DEAD: long leg (low-vol, borrow-free)
alpha +0.04 / DSR 0.02 = just cohort beta. ALL edge is shorting high-vol decile (needs
expensive borrow). Episodic: top-10 days = 133pct of P&L (a few crash events). Just broke:
2026 = -2.72 (anomaly reversed). Beta-neutral residual +0.98 = genuine betting-against-beta
anomaly but lumpy/short-only/recently-inverted. Turnover low (cost was fair). Verdict: real
anomaly, un-harvestable by a small retail borrow-constrained book. Detail killed the lead
honestly. Crypto momentum+VRP remains the only deploy-grade edge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Full gauntlet on the lottery-aversion lead (liquid high-vol cohort). BASE L/S +0.74/OOS
+0.72 but: name-bootstrap frac>0 only 0.77 (fragile vs momentum's 1.00); edge lives at
$5-20M floor (smaller names, real spreads > the 10bp assumed), dies at $50M; the L/S edge
is mostly shorting hard-to-borrow hype names (30%/yr borrow -> +0.04 gone); long-only
realistic = +0.33, DSR 0.07 (not significant). corr to crypto book -0.05 (uncorrelated ->
would diversify IF real). DSR 0.22/0.07 -- neither clears 0.5. Closest non-crypto market,
theoretically sound, uncorrelated -- but marginal after realistic borrow+small-name cost,
NOT deploy-grade. A lead on the shelf, not a strategy. Crypto momentum+VRP remains the only
deploy-grade edge. Re-confirms the boundary.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reasoned about WHERE our edge structure fits (reflexivity+breadth+low-cost overlap) ->
picked the liquid high-vol equity cohort (>$10M/day liquid AND high realized vol =
reflexive story-stocks, ~87 names) = equity analog of crypto's reflexive cohort, avoiding
both walls. Momentum/reversal still negative (regime+efficiency), but LOW-VOL (lottery-
aversion) FITS: L/S net +0.74, OOS +0.72, CPCVmed +0.73, positive 3/4 years, LongOnly +0.33.
First equity signal to survive net cost (broad test had lowvol +0.02). Theoretically sound
(fade overpriced hype/lottery names in the reflexive cohort). LEAD not deploy-grade: DSR
0.28<0.5, short 3.2y data, thin cohort, short-side borrow cost unmodeled. Non-crypto,
low-cost, Databento. Next: full gauntlet (bootstrap/cohort-robustness/borrow/deflation).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DBEQ.BASIC all US equities daily 2023-2026 ($67.85, 17305 instruments). Small/mid-cap band
(excl top-50 mega, $2M floor, ~576 names/day), weekly rebal+smooth, illiquidity-scaled cost
(30-150bp). NO factor survives net: momentum negative even gross (2023-24 momentum-crash);
reversal gross +0.15 eaten by cost -> NET -0.86 (Amihud paradox); low-vol best but NET ~0
(DSR 0.03). Didn't even charge short-borrow. Confirms Databento tradeable universe (equities+
futures) is efficient+cost-walled -> no retail edge. Honest tension: cheap-to-trade markets too
efficient; the inefficient market with real edge (crypto) is the disliked one. Caveat: 3.2y only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fetched Bybit funding (OKX fetch failed, so Binance+Bybit) for 18 majors. Cross-venue
DISPERSION & Binance-premium signals all negative. Only positive = carry_multivenue +0.84
(OOS+0.80, corr_mom+0.01, marg_t+2.58) -- BUT Binance-only control on the SAME 18 majors =
+0.69, proving the edge is MAJOR-COIN SELECTION not the cross-venue angle (multi-venue adds
only +0.15 noise-reduction). This is the known cherry-picked carry trap (majors +0.8 ->
broad-clean -0.03). Almost re-fooled by the seductive number; the Binance-only control
disconfirmed it. Orthogonality doesn't rescue universe-fragility. Databento has no crypto/
on-chain data either. Both orthogonal frontiers closed. Product = residual-momentum + VRP.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Before building deep sequence models, tested whether cost-surviving signal exists at
the minute horizon for them to extract. 12 perps, 60d free 1m klines (OFI proxy from
takerBuyQuote/quoteVol), features -> forward 1/5/15min returns net of 5bp taker. ALL
features negative net (best gross ~0.2bp vs 5bp cost = 25x gap); reversal IC -0.027 and
vol-continuation +0.009 are real but ~25x too small; OFI/taker-imbalance ~zero predictive
power. -> No minute-horizon signal -> Mamba2/CfC/TLOB have nothing to extract -> don't
build (no model lifts IC 25x). Crossing-cost wall holds in crypto (sec-ES 100x, min-crypto
25x); maker=adverse-selection wall, full-LOB=colocation territory. Gate saved a multi-week
GPU build for $0/10min. Product remains the daily two-sleeve book.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase-1 upgrade applied to the PoC base sleeve: rank residual momentum (each coin's
return stripped of its rolling beta to the equal-weight market) instead of raw momentum.
Removes high-beta coins dominating the sort by market co-movement. Backtest: gross
+0.82->+0.99, net @$20M +0.40->+0.57, capacity $5M +0.60->+0.78, CPCV-med +0.47->+0.59,
combined two-sleeve book +1.56->+1.66. Same edge, cleaner construction. Live panel depth
bumped to cover the 60d beta window. No lookahead (beta + cumsum use closed bars).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tested 5 theoretically-distinct new crypto signals (residual-momentum, MAX/lottery,
idiosyncratic-vol, Amihud-illiquidity, acceleration) on PIT top-50 with full gauntlet
(OOS/CPCV/deflation/coin-bootstrap) + correlation to momentum. NO new DISTINCT validated
signal: survivors = momentum + residual-momentum (corr 0.95 = same edge). Others fail
(lottery/accel negative; ivol/amihud fail bootstrap-robustness, amihud closest miss
corr+0.20/boot0.75). DECISIVE: validated signal set is thin (one alpha + VRP diversifier)
-> RL/PPO has no rich interacting set to combine -> would not beat the linear baseline OOS
-> don't build the RL (the falsifiable gate returns NO; avoids the 65th-commit trap).
SMALL WIN: residual (beta-stripped) momentum +0.72 > raw +0.57 -- better base construction.
Product = simple two-sleeve book. RL needs non-price signals to ever justify itself.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
backtest now reports momentum + VRP (tail-managed, corr -0.04) + COMBINED book
(momentum 70% / VRP 30% risk) Sharpe +1.56, all years positive. paper marks BOTH
sleeves + combined equity each run; status shows all three + live Sharpes. VRP gate
('don't sell vol when DVOL rising/elevated') runs live -- currently 'flat (gated)'.
cwd-independence fixes: pit_sweep.load + _vrp_calib now use abs paths so the cron paper
run calibrates VRP correctly from $HOME (was silently falling back to zero-VRP). Live
panel depth bumped to cover the 60d VRP gate window. Honest: VRP +2.85 is the no-Greeks
proxy (real ~1-1.5) so combined is optimistic (~0.8-1.0 realistic) -- still > momentum alone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Vol-risk-premium (short-vol BTC+ETH, Deribit DVOL implied minus realized, short-variance
daily P&L proxy). VRP Sharpe +1.03, CORRELATION to momentum -0.03 (genuinely uncorrelated),
combined +1.22 (~doubles momentum-alone +0.66). FIRST genuine diversifier found -- has BOTH
edge AND low-corr (futures trend had corr but no edge -> diluted). CAVEATS: (1) catastrophic
negative skew -8.32, worst-day -21sigma (short-vol blowup risk; Sharpe flatters; must
tail-manage + size small) (2) recent decay -- negative 2025/2026 (+1.22 is a 2021-24 artifact).
Real and promising, categorically better than futures, but pursue carefully not naive deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Expanded futures universe 22->39 liquid CME roots ($16.39 credits, 16y daily). (A)
breadth did NOT unlock an edge: XS momentum + TS trend all negative/zero, DSR 0
(efficient market, unlike crypto). (B) diversifier check: futures-trend +0.01,
crypto-momentum +0.88, CORRELATION -0.01 (genuinely uncorrelated) but combined +0.63
< +0.88 -> an uncorrelated sleeve with zero standalone edge DILUTES, not diversifies.
A diversifier needs low-corr AND positive edge; futures trend has only the former.
Caveat: roll-zeroing understates futures trend/carry (floor work ~+0.05-0.14 w/ proper
rolls) -> marginal at best. Crypto momentum still the only real edge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Rebalance keyed on FIXED calendar phase (epoch-day%7==0 -> Thursdays) instead of
drifting days[0]%K, so backtest and live agree on rebalance timing.
- STATE path absolute (relative to script) -> paper mode runs from any cwd (cron-safe).
- Backtest unchanged (net +0.40@$20M, CPCV-med +0.47).
Installed system crontab (daily 12:07): marks paper PnL daily, rebalances Thursdays,
logs to data/surfer/paper_runs.log. Forward-test seeded inception 2026-06-06.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
One-file PoC of the validated crypto XS-momentum edge. ONE shared compute_weights()
drives backtest AND live (no skew); no lookahead; funding-as-PnL; sqrt-impact cost.
Backtest reproduces research (gross +0.83, net +0.43@$20M, capacity curve, turnover
9.8%). Regime diagnostic = clean NEGATIVE (no feature predicts momentum favorability,
IC~0 IS/OOS) -> de Prado meta-labeling/regime overlay empirically unwarranted, runs
flat-sized. Paper mode = live forward-test (Binance ASCII perps, market-neutral book,
intended trades + paper PnL persisted). Python is the right tool for weekly crypto MFT;
foxhunt Rust/CUDA is HFT-ES-specific (concepts reused, plumbing not).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Directional TS-trend (fixed earlier demeaning bug) is a real edge (bootstrap frac>0=1.00)
with low corr to momentum (+0.12) and lifts full-sample Sharpe +0.67->+0.71 -- but per-year/
OOS expose it as STALE (value all in 2021 +2.65; recent 2024 -0.16/2026 -0.89), so it DRAGS
the book OOS to +0.24 vs momentum's strong recent. 2022 crisis-alpha hypothesis failed
(choppy bear whipsawed trend). Within-crypto diversifiers exhausted (carry/lowvol/reversal/
size/TS-trend all fail) -> momentum is a robust single-sleeve edge; real diversifier needs a
different market/data. Discipline: full-Sharpe said add, OOS said stale.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(2) Low-vol FAILS its full battery (standalone ~0/neg all windows, coin-bootstrap
frac>0=0.45 coin-flip) -> its earlier diversifier value was a winsor artifact ->
NO validated second edge, momentum stands alone. (1) Slippage/capacity model
(sqrt-impact eta=1, daily rebalance): book gross +0.62 but net M +0.41 / M +0.22 /
0M -0.11 / 0M -0.50 -> real but small-capacity (~$1-5M), lives in smaller alts.
Favorable caveats: eta=1 conservative, daily rebalance of 20d signal wastes turnover.
Key untested lever: weekly rebalance + smoothing + top-20 liquid -> ~5x less turnover.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(a) All 6 momentum constructions collapse to ~0.2 under aggressive +/-20% winsor
(momentum leans on large moves); volscaled/winsorinput degrade least, none robust to
the cap. (b) LOWVOL is genuinely uncorrelated (corr -0.13) and lifts the stressed book
+0.19->+0.34; TStrend correlated; carry/reversal/size don't help. Book=momentum+lowvol.
Honest deployable bracket ~0.2 (harsh winsor) to ~0.6 (fair); +/-20% blanket winsor
double-counts (death-spiral already excluded) so truth ~0.5. Deciding next: per-coin
ADV slippage model, not blanket winsor. Corrected the script's over-claiming verdict.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Built survivorship-free universe: 135 perps incl 28 known-dead (LUNA/SRM/MATIC...),
universe rebuilt daily as top-K by trailing dollar-volume (dead coins in while trading,
drop out after crash), no lookahead. mom_20 TOPK=30: full +0.76, IS+0.85/OOS+0.56,
CPCV-med +0.73, DSR 0.86, POSITIVE EVERY YEAR 2020-2026 incl 2022 +0.72. Counterintuitive:
survivorship was HIDING the edge (survivor-only 2022 -0.40 -> PIT +0.72) because momentum
shorts the dying coins and profits from crashes. Decisive survivorship confirmation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Broadened crypto universe 28 majors -> programmatic top-80 (64 usable), added
funding-coverage/history filter. Carry COLLAPSED (+0.82->-0.03) and stayed dead even
on broad-but-clean subset (fundcov>0.9, 45 coins): carry was cherry-picked to majors,
not robust. Cross-sectional MOMENTUM is the robust edge: XS_mom_30 IS+0.89/OOS+0.88,
CPCV-med +0.75, DSR 0.75, survives 2x cost, holds across 64- and 45-coin universes.
Momentum needs breadth (weak on 28 correlated majors). Same regime failure (2019/2022).
Methodology win: broadening killed a cherry-picked claim before a build.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
13 crypto signals, deflated by cumulative 30. Carry family all develop-grade, survive
2x cost; per-year reveals THE failure mode = 2022 deleveraging carry-crash (carry_7
-1.2 in 2022, +0.4..+2.1 other years). Momentum OOS-negative (decayed); chasing rising
funding strongly negative. Carry is the robust core; its one tail (carry crash) is
exactly what the ML regime overlay is built to manage -> the two threads unite.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Batch 2 of the deflated signal sweep: cross-sectional funding carry + momentum on
28 major Binance USDT perps (2019-26, funding baked into return, 10bp cost, deflated
by cumulative 25 trials). XS_carry_7 = Sharpe +0.82, IS+0.77/OOS+0.93 (consistent),
CPCV-median +0.78 (robust across 45 paths), DSR 0.71. First signal all session that is
positive + IS/OOS-consistent + CPCV-median-positive. Develop-grade met, not yet deploy
(DSR<0.95, 5th-pct<0). Caveats: survivorship (current majors), confirm point-in-time.
Forward-paginated funding fetch (fundcov ~1.0).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Broad signal search judged against the FULL search (Deflated Sharpe deflated by
N_trials) — the multiple-testing discipline that makes breadth safe. Batch 1: 17
signals (XS/TS momentum, reversal, low-vol, seasonality) on 22-futures daily ->
ZERO survivors (best DSR 0.29); CPCV+deflation correctly kill the lucky cells.
SEAS_overnight invalid on 24h futures (equity-cash anomaly). Next batches: crypto, COT.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tests the actual surfer thesis: intraday 5-min decisions/15-min holds, flat-by-close,
regime-adaptive (ride trend / fade range / stand aside), quality-over-quantity
(top-conviction only). Clean ES OHLCV-1m -> 5-min bars, per-trade net edge in ticks,
IS/OOS + t-stat. Result: regime-adaptive top-5% = +0.50 ticks/trade OOS, sign-consistent,
but t=0.67 (not significant); all else significantly negative. Adaptive+selective is the
only non-losing structure (thesis directionally right, within noise). 4GB-safe: cumsum
rolling-std (no unfold) + alloc cap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per-instrument time-series momentum + diversified trend portfolio + lead-lag,
honest IS(2024)/OOS(2025) Sharpe on non-overlapping daily P&L.
Verdict on 2y/4-instrument data: diversified trend portfolio IS Sharpe +1.98
(L10) -> OOS -0.50; per-instrument TSMOM sign-flips IS<->OOS; lead-lag ICs <0.11.
No robust edge. Conclusion: 4 instruments x 2y is underpowered by 1-2 orders of
magnitude for systematic edge discovery -- the bottleneck is data/universe, not
model. Real progress needs a broad futures universe + decades, or a less-efficient
order-book market (crypto).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tests minutes-to-days directional predictability on ES OHLCV-1m where moves
dwarf the spread: momentum/reversion IC (Pearson/Spearman/NW-t/OOS) + a
non-overlapping daily contrarian backtest (IS/OOS Sharpe).
Verdict on 2y ES (2024-25): intraday IC~0; daily mean-reversion IC looks large
(-0.44 Spearman @20d) but is overlap-inflated + regime artifact -- honest
non-overlapping backtest sign-flips IS<->OOS => no stable edge. Confirms no
robustly capturable directional edge on 2y ES across all measured horizons.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
FIFO queue priority via price-level episodes: join back of queue, only trades
advance, fill when same-side aggressor volume clears the queue. Re-measures
markout + OFI-conditioning on ACTUALLY-WON (adversely-biased) fills.
Verdict (Q1+Q2 2024 ES): ~40% fill rate; won-fill markout -0.50..-0.55t
(winner's curse vs optimistic -0.40t); naive AND OFI-conditioned passive MM
both NEGATIVE (-$1..-$2/fill). The optimistic Q2 OFI tilt was a win-every-queue
artifact. Combined with the crossing result: no capturable seconds-horizon ES
edge for a non-colocated participant.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Measures passive-MM economics on clean ES MBP-10: half-spread earned vs
adverse-selection markout at horizons, plus OFI-conditioned selective quoting.
Verdict on 2024 ES: adverse selection ~0.4 tick eats ~80% of the 0.5-tick
half-spread => naive passive MM is break-even (optimistic, win-every-queue;
realistically negative). OFI-conditioned selective quoting shows a consistent
edge in Q2 (+$0.5/fill) but not Q1 => promising but unconfirmed; decisive
next test is a realistic queue/fill model.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Measures best-level Order-Flow-Imbalance (Cont-Kukanov-Stoikov) predictive
edge on clean ES MBP-10 vs forward returns: contemporaneous R², forward IC
(Pearson/Spearman), sign-accuracy, Newey-West t-stat across horizons, and a
CROSSING verdict (predicted move in ticks vs the 1-tick round-trip cost).
No training/RL. Verdict on clean 2024 ES: OFI fwd IC tiny (~0.02, Q2 NW-t 3.7),
predicted moves 0.01-0.06 ticks at 1-10s << 1-tick crossing cost (~20-100x gap)
=> spread-crossing directional strategy is structurally unprofitable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Computes Pearson(rewards.sum, trading.realized_pnl_usd_delta) + sign-agreement
from a diag.jsonl. Exit 0 if Pearson >= 0.70 (RA-G3), 1 below, 2 on bad input.
Reproduces the baseline 0.2775 / sign 0.655 measurement.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror of the FOXHUNT_USE_MULTI_HEAD_POLICY pattern (c5036af03).
Adds --band-enabled flag to argo-alpha-rl.sh and band-enabled
workflow parameter wired into the container env block.
Required for Phase 4-C cluster verification of the full Tier 3
no-transaction-band stack (commits e41a73208 → 65c328d3f).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds use-multi-head-policy workflow parameter (default "0") wired
into the container env block as FOXHUNT_USE_MULTI_HEAD_POLICY.
Adds --use-multi-head-policy flag to argo-alpha-rl.sh that passes
through to the workflow.
Enables Phase 2A-E cluster verification with the multi-head policy
mixture (commits 0b3e40150 → c8c81ab7e). Without this flag, the
cluster runs the legacy single-head path identical to pre-Phase-2A
behavior.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The diag's `trading.*_pnl_*_usd` fields were systematically mislabeled, making
the eval verdict signal untrustworthy. Three quantities shared the `_usd`
suffix but were not in USD:
- `eval_summary.total_pnl_usd` (USD ground truth, $50/pt ES applied)
- `trading.realized_pnl_cum_usd` (cumulative SHAPED reward at close events;
pts × lots × Phase-5 shaping; NOT USD despite the suffix)
- `trading.pnl_cum_usd` (mathematically nonsense — reads rewards_d after
apply_reward_scale AND rl_popart_normalize whiten it in-place; dividing
by current_scale un-does only the first transform)
At baseline mid-smoke (b=128, 2000+500, seed=42): eval_summary.total_pnl_usd
= -$4,457,625 while diag.trading.realized_pnl_cum_usd = +$469k and
diag.trading.pnl_cum_usd = +$12.9k. 346x ratio, sign-opposite. Same
mechanism as `pearl_grwwh_eval_catastrophic_collapse` ($234M cluster
discrepancy). Every Pearson(reward, Δpnl_usd) computation against these
fields was shaped-vs-shaped tautology, not pnl alignment.
This commit:
DELETE `trading.pnl_cum_usd` (mathematically nonsense post-popart).
RENAME `trading.realized_pnl_cum_usd` -> `trading.shaped_reward_close_event_cum`.
Truthful name; the underlying values are post-Phase-5 shaped reward
summed at close events, useful for gradient-signal diagnostics but never
to be confused with USD pnl.
ADD `trading.realized_pnl_usd_delta` and `trading.realized_pnl_usd_cum` in
diag.jsonl / eval_diag.jsonl. Source: new per-batch float buffer
pnl_step_close_usd_d on LobSimCuda, zeroed via raw_memset_d8_zero before
each pnl_track_step launch, written by pnl_track.cu's close branch as
realised_pnl_usd_fp / 100.0 (same arithmetic as eval_summary's
TradeRecord aggregator, applying ES $50/pt). Single-writer per block —
no atomicAdd per feedback_no_atomicadd.
Direct readback via read_slice_d_into<f32> from sim.pnl_step_close_usd_d()
after step_with_lobsim_gpu returns (main stream already synchronized via
self.stream.synchronize() at pnl_track_step exit). NOT routed through
diag_staging double-buffer — a prior implementation tried this and
broke determinism (cross-stream race between main-stream
raw_memset_d8_zero + pnl_track_step write and diag-stream
raw_memcpy_dtod_async read). Direct same-stream read avoids the race
entirely and stays mega-graph compatible (the readback runs AFTER
per-step pipeline finishes, outside any captured CUDA graph).
Cumulative tracked Rust-side in alpha_rl_train.rs; reset at train->eval
boundary so realized_pnl_usd_cum tracks the eval window only.
scripts/tier1_5_verdict.py upgrade:
- signal_reward_alignment reads trading.realized_pnl_usd_cum
- signal_eval_pnl falls back to realized_pnl_usd_cum
- signal_consistency upgraded from WARN-at-5% to KILL-at-$50 with a new
consistency_kill_usd threshold key; cross-source disagreement is now
a hard kill not a warning
tests/eval_diag_emission.rs: bumped EXPECTED_LEAVES 711 -> 712 (-1 deleted,
-0 renamed, +2 added); added invariant that cum == running_sum(delta) and
regression check that legacy pnl_cum_usd / realized_pnl_cum_usd fields are
absent from the schema; allowed n_eval_steps + 1 row count for the
drain-row pattern.
Falsification gate (load-bearing, must hold every smoke from this commit
forward): `diag.last_eval_row.trading.realized_pnl_usd_cum` matches
`eval_summary.total_pnl_usd` within $50.
Validation (mid-smoke b=128, 2000 train + 500 eval, seed=42, RTX 3050):
- cargo build --release --example alpha_rl_train -p ml-alpha: exit 0
- cargo test -p ml-alpha --test eval_diag_emission: PASS (712 leaves)
- local-mid-smoke.sh: exit 0, drain row at step 2500
- cross-source consistency: eval_summary=-$4,457,625.00 vs
diag.realized_pnl_usd_cum=-$4,457,625.18 -> delta=$0.18 (exactly fp/100
f32 rounding noise; well within $50 tolerance)
- ./scripts/determinism-check.sh --quick: PASS — all checksums.* leaves
match across all 200 rows (rel-tol 1e-5, abs-tol 1e-7)
- Pre-commit hook on staged diff: PASS
Memory pearls created in this session document the diagnostic chain:
- pearl_diag_pnl_fields_are_shaped_reward_not_usd (the mislabeling root)
- pearl_reward_signal_anti_aligned_with_pnl ADDENDUM 2026-06-02b
(correction to the 2026-06-02 ADDENDUM — the "+0.93 Pearson at HEAD"
finding was a measurement artifact because both sides were in shaped
pts x lots units, not USD)
- pearl_advantage_kernel_is_done_gated_td_not_gae (related signal-density
gap — Phase 1.5 follow-up)
- pearl_phase5_term_4_is_almost_potential (Ng-Harada-Russell analysis of
Phase 5 shaping terms — Phase 1 follow-up)
Unlocks: TRUE Pearson(rewards.sum, Δpnl_usd) can now be measured per-step
at HEAD. The eval-collapse investigation (next phases: Ng-Harada-Russell
shaping cleanup + GAE upgrade) is now falsifiable with bit-deterministic
verdicts grounded in authoritative USD pnl.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two same-seed runs now produce bit-equal eval_summary.json, alpha_rl_train_summary.json,
and diag.jsonl (modulo wall-clock elapsed_s). The 5-phase falsification chain landed:
Phase 2 PER tree-rebuild: __threadfence is NOT a grid-wide barrier; multiple blocks
raced across sum-tree levels. Fix: Grid=(1) Block=(1024) + __syncthreads
in rl_per_tree_rebuild.cu.
Phase 2.3 cuBLAS GEMM_DFALT + TF32 default-math allowed split-K non-deterministic
accumulation at 3 sites. New crates/ml-alpha/src/cublas_determinism.rs
applies CUBLAS_PEDANTIC_MATH via FOXHUNT_DETERMINISTIC env toggle
(0=TF32 prod, 1=PEDANTIC dev default, 2=DEFAULT_MATH control).
Phase 2.6 Two bugs surfaced sequentially in the backward kernel chain:
(1) rl_iqn_tau_cos_features had a multi-block r/w race on prng_state[batch]
— all N_TAU=32 blocks read seed; only tau_idx==0 wrote back; no
inter-block barrier. Fix: split into READ-ONLY rl_iqn_tau_cos_features
+ new sibling rl_iqn_advance_prng_state launched on same stream
(kernel-launch ordering = grid-wide barrier).
(2) OutcomeHead::new called near_zero_xavier without scoped_init_seed,
falling back to time+thread-id RNG. Stayed dormant until first done
event activated non-sentinel labels and divergent weights flowed via
grad_h_t_outcome into encoder gradient. Fix: add seed param + install
scoped_init_seed(dqn_seed.wrapping_add(0x0CE0)) guard.
Validation (./scripts/determinism-check.sh --quick, RTX 3050, b=128, 200+50 steps):
- All 200 rows of checksums.* leaves match (rel-tol 1e-5, abs-tol 1e-7)
- eval_summary.json, alpha_rl_train_summary.json byte-equal between runs
- diag.jsonl byte-equal modulo elapsed_s
- Eval pnl identical run-A vs run-B at seed 42
Pre-fix baseline (Phase 2.5 measurement): same-seed eval pnl spread $450k
($187k vs -$261k). Post-fix: $0 spread.
Speed cost: ~1.5ms/step amortised; ~10-15% slower than TF32 production
(PEDANTIC tax — acceptable in dev, toggle to FOXHUNT_DETERMINISTIC=0 for prod).
Mapped-pinned discipline: all 11 NEW memcpy_dtoh sites in diagnostic dump methods
+ per-step checksum readback use a new pub(crate) helper
read_slice_d_into<T: Copy>(stream, src, dst) — MappedRecordBuffer + raw
memcpy_dtod_async + raw_stream_sync + volatile read. Generic over T (f32, f64,
i32, u32, u8). Satisfies feedback_no_htod_htoh_only_mapped_pinned + hook guard.
Bundled Tier 1.5 fast-dev-cycle infrastructure (spec
docs/superpowers/specs/2026-06-02-fast-dev-cycle.md):
- scripts/local-mid-smoke.sh b=128, 2000+500, ~10min on RTX 3050
- scripts/determinism-check.sh runs mid-smoke twice, diffs checksums
- scripts/tier1_5_verdict.py behavioral kill verdict
- AdamW checkpoint save/load (crates/ml-alpha/src/trainer/optim.rs)
- IntegratedTrainer checkpoint save/load (resume from checkpoint)
- 15 Phase 1 checksum leaves in build_diag_value
- Env-gated dump methods (FOXHUNT_DETERMINISM_DEBUG_PER/MAMBA2/RL/BACKWARD)
for future divergence-chasing — never run in production
Documentation:
- docs/superpowers/specs/2026-06-02-determinism-foundation.md
- docs/superpowers/specs/2026-06-02-fast-dev-cycle.md
- docs/superpowers/plans/2026-06-02-determinism-foundation-implementation.md
- docs/superpowers/notes/2026-06-02-determinism-phase{1,2,2.2,2.5,2.6}-*.md
- Adjacent specs/plans/notes from the analytical chain that surfaced determinism
as the load-bearing blocker (eval-summary, eval-boundary, regime-observer,
multi-head policy, regime-invariance, Phase 3 IQN-complement post-mortem)
Unlocks: every controller / architecture / reward-shaping A/B from this commit
onward attributes outcome differences to the change, not random-init kernel-race
drift cascading through training x eval LOB-sim trajectories. The eval-collapse
investigation (pearl_reward_signal_anti_aligned_with_pnl, multi-head spec,
regime-invariance spec) is now testable with trustworthy verdicts.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
