docs(surfer): design spec — the surfer over an uncorrelated universe
Strategic pivot from seconds-horizon ES order-book RL (measured structurally unprofitable) to diversified days-to-weeks multi-asset ML trend. ML as bounded deviations from a deterministic diversified-trend floor; gated by CPCV/PBO/ Deflated-Sharpe. Phase 0 (build+validate the floor on Databento micros, CPU-only, no ML, no cluster) is the immediate decisive test. Synthesizes 8-agent omnisearch research (pearl_surfer_universe_build_synthesis). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -0,0 +1,137 @@
|
||||
# The Surfer over an Uncorrelated Universe — Design Spec
|
||||
|
||||
> **For agentic workers:** This spec defines WHY and WHAT for foxhunt's strategic pivot from
|
||||
> single-instrument seconds-horizon order-book RL to a diversified, days-to-weeks, multi-asset
|
||||
> ML trend system. It does NOT implement; an implementation plan follows (Phase 0 first).
|
||||
|
||||
**Status:** Draft 1 (2026-06-05)
|
||||
**Supersedes (direction):** the seconds-horizon ES order-book RL thesis (measured structurally unprofitable).
|
||||
**Branch target:** new branch `surfer-universe` off the current branch (keeps the session's validation harnesses).
|
||||
**Capital stance:** $35k is a PILOT to prove the pipeline + edge (paper / tiny live); real book targets ~$100k.
|
||||
|
||||
**Linked pearls (load-bearing):**
|
||||
- `pearl_surfer_thesis_is_uncorrelated_trend_recognition` — the thesis
|
||||
- `pearl_surfer_universe_build_synthesis` — the 8-agent research synthesis behind every choice here
|
||||
- `pearl_ofi_edge_uncapturable_by_crossing` / `pearl_passive_mm_knife_edge_ofi_conditioning_promising` — why seconds-horizon ES is dead
|
||||
- `pearl_lowfreq_no_robust_edge_on_2y_es` — why breadth/universe is the bottleneck; validation discipline
|
||||
- `pearl_omnisearch_synthesis_signal_first_and_mtm_reward` — dense-MtM → portfolio Differential Sharpe reward
|
||||
|
||||
---
|
||||
|
||||
## §1. Why this, why now
|
||||
|
||||
Measured this session (zero cluster runs, ~5 cheap harnesses): on ES, seconds-horizon crossing edge is ~100× smaller
|
||||
than the spread; passive MM is adversely-selected negative (winner's curse); intraday has no IC; daily/cross-asset
|
||||
edges sign-flip IS↔OOS (overfit). **The bottleneck was the configuration — single instrument × seconds-horizon ×
|
||||
order-book microstructure — not the model.** That is the worst possible config for a non-colocated ML participant:
|
||||
most-efficient market, most speed-dependent horizon, zero diversification.
|
||||
|
||||
The fix is to invert all three: **many uncorrelated instruments × days-to-weeks horizon × trend/cross-sectional
|
||||
recognition.** This is also foxhunt's actual "surfer" thesis ("no wave is the same") finally pointed at an ocean
|
||||
with enough waves to ride. Breadth = opportunity sourcing: across many uncorrelated markets, some subset always
|
||||
has a rideable trend, and uncorrelated rides don't sink together.
|
||||
|
||||
## §2. The thesis
|
||||
|
||||
A fixed trend rule applies identical logic to every wave. The ML "surfer" reads each unique wave — is it a real
|
||||
trend or chop? building or exhausting? what's the sea-state (regime)? — and adjusts. That recognition is the
|
||||
legitimate ML edge ON TOP of the trend risk premium. **Breadth gives the floor; ML earns its keep above it.**
|
||||
|
||||
## §3. Architecture
|
||||
|
||||
**Core design principle (the eval-collapse cure):** the ML surfer outputs **bounded DEVIATIONS from a deterministic
|
||||
diversified-trend floor**: `w_i = floor_i + clamp(ML_delta_i, ±δ)`. The model can only add value above a proven
|
||||
baseline and degrades gracefully to the floor when it has no edge. It structurally cannot run off a cliff.
|
||||
|
||||
### §3.1 The floor (deterministic)
|
||||
Diversified time-series momentum: `signal_i = mean over h∈{1,3,12}mo of sign(return_{i,h})` (12mo skips last ~21d);
|
||||
inverse-vol size with 63-day EWMA vol; portfolio vol-target 10% annual (rescale weekly on realized portfolio vol);
|
||||
weekly rebalance + ≥10%-change no-trade band. Realistic net Sharpe 0.4–0.5 (small universe), DD 15–25%.
|
||||
|
||||
### §3.2 The ML surfer (bounded deltas), in confidence order
|
||||
1. **Regime / change-point overlay (build FIRST)** — recognizes trend-vs-chop, throttles gross, cuts drawdowns.
|
||||
ML's most defensible edge (DD reduction, NOT return prediction). Lowest overfit surface.
|
||||
2. **Cross-sectional tilt (panel)** — ONE shared encoder over all instruments (N×T samples) + per-instrument
|
||||
embedding → continuous dollar-neutral tilt on the floor; gated on rank-IC / ICIR. NOT discrete ranks (a <20-name
|
||||
universe is statistically starved; breadth is recovered through time + pooling).
|
||||
3. **Direct return prediction → DEFERRED** (the 64-commit trap; cost-cliff at 2–3 bps = turnover noise).
|
||||
|
||||
### §3.3 Reward / objective
|
||||
Portfolio **Differential Sharpe Ratio** (Moody-Saffell) + drawdown penalty + turnover/cost penalty — the
|
||||
portfolio-level, risk-adjusted generalization of the dense per-step mark-to-market reward (Φ=unrealized-PnL).
|
||||
|
||||
### §3.4 Risk / sizing
|
||||
Vol-target 10%; inverse-vol within asset class + ERC across classes; fractional Kelly (half/quarter, reuse the
|
||||
existing Kelly controller clamped [0,0.5]); drawdown throttle θ; reuse the CMDP overlay as the tail kill-switch.
|
||||
Discreteness is binding at micro size: target <0.5 contract → hold 0, carry the residual; $35k sustains ~3–5
|
||||
simultaneous 1-lot positions.
|
||||
|
||||
### §3.5 Reuse of existing foxhunt assets
|
||||
CfC/Mamba2 encoder (shared across instruments + instrument embedding + cross-sectional attention head); RL stack;
|
||||
Kelly + CMDP controllers; the determinism foundation; the session's validation harnesses (`scripts/measure_*.py`).
|
||||
|
||||
## §4. Universe + data pipeline
|
||||
|
||||
**Universe (~8 micros, maximal asset-class diversification):** MES, M2K (equity); 10Y micro-yield (rates — note
|
||||
YIELD-based = sign inversion vs price); M6E, M6A (FX); MGC (metals); MCL (energy); MBT (crypto).
|
||||
**Data:** Databento `GLBX.MDP3`, `ohlcv-1d` (+ `ohlcv-1h`). **Continuous contracts built in-house:** volume-roll
|
||||
(validate vs OI) + **backward RATIO adjustment for the encoder's returns**; keep an **UNADJUSTED** series for
|
||||
actual fills + USD reward. NEVER Panama-adjust for a trend learner; NEVER compute reward on adjusted prices.
|
||||
|
||||
## §5. Validation gates (the guardrail — non-negotiable)
|
||||
|
||||
| Gate | Metric | Pass |
|
||||
|---|---|---|
|
||||
| SV-G0 | Leakage audit: purge+embargo covers full hold + feature lookback | verified |
|
||||
| SV-G1 | CPCV (≥50 paths): 5th-percentile OOS pnl after costs | > 0 |
|
||||
| SV-G2 | PBO (CSCV) | < 0.20 |
|
||||
| SV-G3 | Deflated Sharpe, deflated by N = ALL trials ever (64-commit + session + this build) | > 0.95 |
|
||||
| SV-G4 | IS↔OOS config rank-consistency (Spearman) — the sign-flip detector | > 0 |
|
||||
| SV-G5 | Edge survives 2× cost + roll cost | net OOS pnl > 0 |
|
||||
| SV-G6 | Capacity/turnover: cost × turnover < fixed fraction of gross | pass |
|
||||
|
||||
Every config / seed / hyperparameter is a logged trial feeding N. The 64 failed commits prove N is large.
|
||||
|
||||
## §6. Phases (signal-first, cheapest-first, zero cluster until validated)
|
||||
|
||||
- **Phase 0 — Floor + validation harness (CPU-only, no ML, no cluster). FULLY SPECIFIED in the plan.**
|
||||
Build the deterministic floor on real Databento micro data; build the CPCV/PBO/DSR harness; measure whether the
|
||||
floor itself clears the gates OOS after costs. **STOP-if:** floor fails SV-G1/G3/G5 → the universe has no
|
||||
capturable trend premium at this scale; reassess (more instruments / more history) before any ML.
|
||||
- **Phase 1 — Regime overlay.** Must beat the floor OOS (DD reduction first). Gates SV-G1..G6 vs floor.
|
||||
- **Phase 2 — Cross-sectional panel tilt** (shared encoder). Must beat floor+regime OOS via rank-IC/ICIR.
|
||||
- **Phase 3 — Full RL surfer** (floor + bounded delta, DSR reward). Cluster only here, gated by all above.
|
||||
|
||||
## §7. Scope boundary
|
||||
|
||||
**In (this spec):** the floor, the bounded-delta architecture, the 3 ML layers (regime → cross-sectional → RL),
|
||||
the universe + continuous-contract pipeline, the validation gates, the phased build. **Phase 0 is the immediate
|
||||
deliverable.**
|
||||
**Out:** seconds-horizon / order-book microstructure (measured dead); direct return-prediction sequence models
|
||||
(deferred until floor+regime+cross-sectional prove out); live-capital deployment beyond a pilot (until ~$100k and
|
||||
gates pass); broker/execution automation (separate spec once a validated strategy exists).
|
||||
|
||||
## §8. Risks
|
||||
|
||||
1. **Capital ($35k below viable minimum)** — under-diversified → trend's drawdowns without its smoothing; micro-cost
|
||||
drag eats Sharpe. MITIGATION: treat as a pilot to prove pipeline/edge; scale to ~$100k for the real book.
|
||||
2. **ML adds nothing over the floor** — entirely possible; the bounded-delta design makes that a graceful
|
||||
no-op (you keep the floor), and the gates catch it before any spend.
|
||||
3. **Floor itself has no edge at this scale/universe** — Phase 0 STOP catches this cheaply.
|
||||
4. **Re-overfitting (commit #65)** — the SV gates + deflation-by-all-trials are the explicit defense; honor them.
|
||||
5. **Roll / continuous-contract artifacts** — ratio-adjust + volume-roll + reward-on-unadjusted; validated in Phase 0.
|
||||
|
||||
## §9. Decision log
|
||||
|
||||
```
|
||||
Decision: Pivot foxhunt to "the surfer over an uncorrelated universe" — diversified days-to-weeks ML trend across
|
||||
~8 uncorrelated micro futures, ML as bounded deviations from a deterministic trend floor, gated by CPCV/PBO/Deflated-
|
||||
Sharpe. Phase 0 (build+validate the floor, CPU-only) is the immediate, decisive, near-free test.
|
||||
Rejected: continuing seconds-horizon ES order-book RL (measured structurally unprofitable for a non-colocated ML
|
||||
participant — crossing edge 100× < spread; passive MM adversely-selected; daily/cross-asset edges sign-flip).
|
||||
Capital: $35k = pilot to prove the approach; real book ~$100k (Carver retail floor; under-diversification = main risk).
|
||||
ML scope: regime overlay first (DD reduction = ML's defensible edge); cross-sectional tilt second; direct return
|
||||
prediction deferred (the 64-commit trap).
|
||||
Date: 2026-06-05
|
||||
Falsification (Phase 0): floor fails SV-G1/G3/G5 OOS → no capturable trend premium at this scale; reassess before ML.
|
||||
```
|
||||
Reference in New Issue
Block a user