gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(surfer): full-16y MFT settle — regime-adaptive surfer significantly negative

Browse Source 
Pulled 16y ES ohlcv-1m continuous (year-chunked, $20 credits, no 504) + light
to_ndarray loader (to_df OOM'd at 20GB). On 1.1M 5-min bars the 1.3y +0.50/t=0.67
top-5% hint collapsed: regime-adaptive top-5% = -0.52 ticks/trade t=-2.51
(significantly negative); ALL cells/signals/convictions significantly negative.
Decisive: no capturable intraday directional edge for crossing/non-colocated setup.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-06-06 12:15:37 +02:00
parent fa700c7b5f
commit 1cf38232e8
2 changed files with 113 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

46
scripts/surfer/mft_quality_test.py
View File

@@ -35,22 +35,40 @@ DAY_NS = 86_400 * 10**9
def load_es_5min():
import databento as db
# Prefer the full-history continuous front-month pull (data/surfer/es1m/), else fall back
# to the local ~2y parent OHLCV-1m (futures-baseline/ES, per-quarter front-month pick).
es1m = sorted(glob.glob("data/surfer/es1m/*.dbn"))
ts_all, c_all = [], []
for p in sorted(glob.glob("test_data/futures-baseline/ES.FUT/*.dbn.zst")):
try:
df = db.DBNStore.from_file(p).to_df().reset_index()
except Exception:
continue
if df.empty or "close" not in df.columns:
continue
df = df[df["close"] > 0]
if df.empty:
continue
dom = df["instrument_id"].value_counts().idxmax() # front month for the quarter
df = df[df["instrument_id"] == dom].sort_values("ts_event")
ts_all.append(df["ts_event"].astype("int64").to_numpy())
c_all.append(df["close"].to_numpy(np.float64))
if es1m:
for p in es1m: # to_ndarray = light (to_df OOMs at 20GB)
try:
arr = db.DBNStore.from_file(p).to_ndarray()
except Exception:
continue
if len(arr) == 0 or "close" not in arr.dtype.names:
continue
ts = arr["ts_event"].astype(np.int64)
c = arr["close"].astype(np.float64) / 1e9 # raw 1e9 fixed-point → price
m = c > 0
ts_all.append(ts[m]); c_all.append(c[m])
else:
for p in sorted(glob.glob("test_data/futures-baseline/ES.FUT/*.dbn.zst")):
try:
df = db.DBNStore.from_file(p).to_df().reset_index()
except Exception:
continue
if df.empty or "close" not in df.columns:
continue
df = df[df["close"] > 0]
if df.empty:
continue
dom = df["instrument_id"].value_counts().idxmax() # front month for the quarter
df = df[df["instrument_id"] == dom].sort_values("ts_event")
ts_all.append(df["ts_event"].astype("int64").to_numpy())
c_all.append(df["close"].to_numpy(np.float64))
ts = np.concatenate(ts_all); c = np.concatenate(c_all)
_u, _ui = np.unique(ts, return_index=True) # dedup any overlapping ts at chunk seams
ts, c = ts[_ui], c[_ui]
o = np.argsort(ts, kind="stable"); ts, c = ts[o], c[o]
b5 = ts // BAR_NS # 5-min bin id
_, first = np.unique(b5, return_index=True)