diff --git a/docs/superpowers/specs/2026-06-05-surfer-uncorrelated-universe-design.md b/docs/superpowers/specs/2026-06-05-surfer-uncorrelated-universe-design.md new file mode 100644 index 000000000..de97cbdef --- /dev/null +++ b/docs/superpowers/specs/2026-06-05-surfer-uncorrelated-universe-design.md @@ -0,0 +1,137 @@ +# The Surfer over an Uncorrelated Universe — Design Spec + +> **For agentic workers:** This spec defines WHY and WHAT for foxhunt's strategic pivot from +> single-instrument seconds-horizon order-book RL to a diversified, days-to-weeks, multi-asset +> ML trend system. It does NOT implement; an implementation plan follows (Phase 0 first). + +**Status:** Draft 1 (2026-06-05) +**Supersedes (direction):** the seconds-horizon ES order-book RL thesis (measured structurally unprofitable). +**Branch target:** new branch `surfer-universe` off the current branch (keeps the session's validation harnesses). +**Capital stance:** $35k is a PILOT to prove the pipeline + edge (paper / tiny live); real book targets ~$100k. + +**Linked pearls (load-bearing):** +- `pearl_surfer_thesis_is_uncorrelated_trend_recognition` — the thesis +- `pearl_surfer_universe_build_synthesis` — the 8-agent research synthesis behind every choice here +- `pearl_ofi_edge_uncapturable_by_crossing` / `pearl_passive_mm_knife_edge_ofi_conditioning_promising` — why seconds-horizon ES is dead +- `pearl_lowfreq_no_robust_edge_on_2y_es` — why breadth/universe is the bottleneck; validation discipline +- `pearl_omnisearch_synthesis_signal_first_and_mtm_reward` — dense-MtM → portfolio Differential Sharpe reward + +--- + +## §1. Why this, why now + +Measured this session (zero cluster runs, ~5 cheap harnesses): on ES, seconds-horizon crossing edge is ~100× smaller +than the spread; passive MM is adversely-selected negative (winner's curse); intraday has no IC; daily/cross-asset +edges sign-flip IS↔OOS (overfit). **The bottleneck was the configuration — single instrument × seconds-horizon × +order-book microstructure — not the model.** That is the worst possible config for a non-colocated ML participant: +most-efficient market, most speed-dependent horizon, zero diversification. + +The fix is to invert all three: **many uncorrelated instruments × days-to-weeks horizon × trend/cross-sectional +recognition.** This is also foxhunt's actual "surfer" thesis ("no wave is the same") finally pointed at an ocean +with enough waves to ride. Breadth = opportunity sourcing: across many uncorrelated markets, some subset always +has a rideable trend, and uncorrelated rides don't sink together. + +## §2. The thesis + +A fixed trend rule applies identical logic to every wave. The ML "surfer" reads each unique wave — is it a real +trend or chop? building or exhausting? what's the sea-state (regime)? — and adjusts. That recognition is the +legitimate ML edge ON TOP of the trend risk premium. **Breadth gives the floor; ML earns its keep above it.** + +## §3. Architecture + +**Core design principle (the eval-collapse cure):** the ML surfer outputs **bounded DEVIATIONS from a deterministic +diversified-trend floor**: `w_i = floor_i + clamp(ML_delta_i, ±δ)`. The model can only add value above a proven +baseline and degrades gracefully to the floor when it has no edge. It structurally cannot run off a cliff. + +### §3.1 The floor (deterministic) +Diversified time-series momentum: `signal_i = mean over h∈{1,3,12}mo of sign(return_{i,h})` (12mo skips last ~21d); +inverse-vol size with 63-day EWMA vol; portfolio vol-target 10% annual (rescale weekly on realized portfolio vol); +weekly rebalance + ≥10%-change no-trade band. Realistic net Sharpe 0.4–0.5 (small universe), DD 15–25%. + +### §3.2 The ML surfer (bounded deltas), in confidence order +1. **Regime / change-point overlay (build FIRST)** — recognizes trend-vs-chop, throttles gross, cuts drawdowns. + ML's most defensible edge (DD reduction, NOT return prediction). Lowest overfit surface. +2. **Cross-sectional tilt (panel)** — ONE shared encoder over all instruments (N×T samples) + per-instrument + embedding → continuous dollar-neutral tilt on the floor; gated on rank-IC / ICIR. NOT discrete ranks (a <20-name + universe is statistically starved; breadth is recovered through time + pooling). +3. **Direct return prediction → DEFERRED** (the 64-commit trap; cost-cliff at 2–3 bps = turnover noise). + +### §3.3 Reward / objective +Portfolio **Differential Sharpe Ratio** (Moody-Saffell) + drawdown penalty + turnover/cost penalty — the +portfolio-level, risk-adjusted generalization of the dense per-step mark-to-market reward (Φ=unrealized-PnL). + +### §3.4 Risk / sizing +Vol-target 10%; inverse-vol within asset class + ERC across classes; fractional Kelly (half/quarter, reuse the +existing Kelly controller clamped [0,0.5]); drawdown throttle θ; reuse the CMDP overlay as the tail kill-switch. +Discreteness is binding at micro size: target <0.5 contract → hold 0, carry the residual; $35k sustains ~3–5 +simultaneous 1-lot positions. + +### §3.5 Reuse of existing foxhunt assets +CfC/Mamba2 encoder (shared across instruments + instrument embedding + cross-sectional attention head); RL stack; +Kelly + CMDP controllers; the determinism foundation; the session's validation harnesses (`scripts/measure_*.py`). + +## §4. Universe + data pipeline + +**Universe (~8 micros, maximal asset-class diversification):** MES, M2K (equity); 10Y micro-yield (rates — note +YIELD-based = sign inversion vs price); M6E, M6A (FX); MGC (metals); MCL (energy); MBT (crypto). +**Data:** Databento `GLBX.MDP3`, `ohlcv-1d` (+ `ohlcv-1h`). **Continuous contracts built in-house:** volume-roll +(validate vs OI) + **backward RATIO adjustment for the encoder's returns**; keep an **UNADJUSTED** series for +actual fills + USD reward. NEVER Panama-adjust for a trend learner; NEVER compute reward on adjusted prices. + +## §5. Validation gates (the guardrail — non-negotiable) + +| Gate | Metric | Pass | +|---|---|---| +| SV-G0 | Leakage audit: purge+embargo covers full hold + feature lookback | verified | +| SV-G1 | CPCV (≥50 paths): 5th-percentile OOS pnl after costs | > 0 | +| SV-G2 | PBO (CSCV) | < 0.20 | +| SV-G3 | Deflated Sharpe, deflated by N = ALL trials ever (64-commit + session + this build) | > 0.95 | +| SV-G4 | IS↔OOS config rank-consistency (Spearman) — the sign-flip detector | > 0 | +| SV-G5 | Edge survives 2× cost + roll cost | net OOS pnl > 0 | +| SV-G6 | Capacity/turnover: cost × turnover < fixed fraction of gross | pass | + +Every config / seed / hyperparameter is a logged trial feeding N. The 64 failed commits prove N is large. + +## §6. Phases (signal-first, cheapest-first, zero cluster until validated) + +- **Phase 0 — Floor + validation harness (CPU-only, no ML, no cluster). FULLY SPECIFIED in the plan.** + Build the deterministic floor on real Databento micro data; build the CPCV/PBO/DSR harness; measure whether the + floor itself clears the gates OOS after costs. **STOP-if:** floor fails SV-G1/G3/G5 → the universe has no + capturable trend premium at this scale; reassess (more instruments / more history) before any ML. +- **Phase 1 — Regime overlay.** Must beat the floor OOS (DD reduction first). Gates SV-G1..G6 vs floor. +- **Phase 2 — Cross-sectional panel tilt** (shared encoder). Must beat floor+regime OOS via rank-IC/ICIR. +- **Phase 3 — Full RL surfer** (floor + bounded delta, DSR reward). Cluster only here, gated by all above. + +## §7. Scope boundary + +**In (this spec):** the floor, the bounded-delta architecture, the 3 ML layers (regime → cross-sectional → RL), +the universe + continuous-contract pipeline, the validation gates, the phased build. **Phase 0 is the immediate +deliverable.** +**Out:** seconds-horizon / order-book microstructure (measured dead); direct return-prediction sequence models +(deferred until floor+regime+cross-sectional prove out); live-capital deployment beyond a pilot (until ~$100k and +gates pass); broker/execution automation (separate spec once a validated strategy exists). + +## §8. Risks + +1. **Capital ($35k below viable minimum)** — under-diversified → trend's drawdowns without its smoothing; micro-cost + drag eats Sharpe. MITIGATION: treat as a pilot to prove pipeline/edge; scale to ~$100k for the real book. +2. **ML adds nothing over the floor** — entirely possible; the bounded-delta design makes that a graceful + no-op (you keep the floor), and the gates catch it before any spend. +3. **Floor itself has no edge at this scale/universe** — Phase 0 STOP catches this cheaply. +4. **Re-overfitting (commit #65)** — the SV gates + deflation-by-all-trials are the explicit defense; honor them. +5. **Roll / continuous-contract artifacts** — ratio-adjust + volume-roll + reward-on-unadjusted; validated in Phase 0. + +## §9. Decision log + +``` +Decision: Pivot foxhunt to "the surfer over an uncorrelated universe" — diversified days-to-weeks ML trend across +~8 uncorrelated micro futures, ML as bounded deviations from a deterministic trend floor, gated by CPCV/PBO/Deflated- +Sharpe. Phase 0 (build+validate the floor, CPU-only) is the immediate, decisive, near-free test. +Rejected: continuing seconds-horizon ES order-book RL (measured structurally unprofitable for a non-colocated ML +participant — crossing edge 100× < spread; passive MM adversely-selected; daily/cross-asset edges sign-flip). +Capital: $35k = pilot to prove the approach; real book ~$100k (Carver retail floor; under-diversification = main risk). +ML scope: regime overlay first (DD reduction = ML's defensible edge); cross-sectional tilt second; direct return +prediction deferred (the 64-commit trap). +Date: 2026-06-05 +Falsification (Phase 0): floor fails SV-G1/G3/G5 OOS → no capturable trend premium at this scale; reassess before ML. +```