Initial single-cost backtest at τ=0.25 cost=0.25 revealed the binding
constraint: mean_ret = -0.176, pre-cost EV ≈ +0.074, so the model has
a real directional edge but K=6000 price moves are too small to clear
1-tick round-trip cost. The cost-vs-edge balance is the real Phase 1d.4
verdict question, not whether the model has signal.
Two enhancements per the insight block in the previous run:
1. **Cost sweep**: GpuBacktest::run now takes `costs: &[f32]` and
produces (C × T) rows instead of T. The smoke runs at five costs:
- 0.0 frictionless upper bound (theoretical max Sharpe)
- 0.0625 quarter-tick (very aggressive execution)
- 0.125 half-tick (professional desk)
- 0.25 1 tick = $12.50/contract (retail / pessimistic)
- 0.50 2 ticks (very pessimistic)
Tells us the break-even cost where Sharpe crosses zero.
2. **Annualised Sharpe**: per-trade Sharpe × sqrt(trades_per_year).
trades_per_year = n_trades × (seconds_per_year / test_time_span_seconds).
test_time_span_seconds derived from first/last test sequence end-bar
timestamps via FxCacheReader::record_timestamp. Standard Sharpe-time-
scaling assumption (trades roughly i.i.d.); imperfect when signals
cluster in correlated regimes, but the right ballpark for comparison
with industry benchmarks.
Output adds per-cost-band "best operating point" tables plus a clear
"REALISTIC VERDICT" line at cost=0.125 (half-tick — what a professional
desk would actually pay) with three gates:
- Sharpe_ann > 2.0 → deployable
- 0.5 < Sharpe_ann ≤ 2.0 → marginal
- Sharpe_ann ≤ 0.5 → fail at realistic cost
The "FRICTIONLESS UPPER BOUND" line reports the intrinsic edge — what
the model could theoretically achieve at zero cost. Even if realistic
Sharpe fails, this number tells us whether the model has anything to
optimise toward at deployment.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Four new kernels in mamba2_alpha_kernel.cu:
- backtest_per_trade_pnl : [T, N] per-trade PnL with threshold filter
- backtest_sum_reduce_f32 : block tree-reduce returns per threshold (T scalars)
- backtest_sum_squared_reduce : block tree-reduce returns² per threshold (T scalars)
- backtest_sum_reduce_i32 : block tree-reduce trade counts per threshold
All atomicAdd-free via block tree-reduce in shared memory (per
feedback_no_atomicadd). Single kernel launch handles the full
threshold sweep across all sequences via grid_x=T, grid_y=ceil(N/256).
New module crates/ml-alpha/src/backtest.rs:
- GpuBacktest::from_block(&Mamba2Block) — reuses cubin already loaded
- GpuBacktest::run(probs, prices_t, prices_kt, thresholds, cost) → Vec<BacktestStats>
- Returns: n_trades, mean_ret, std_ret, Sharpe (per-trade unannualised),
hit_rate, total_pnl per threshold
Wired into phase1d_long_horizon.rs after the stacker eval:
- Convert stacker_logits → probs via sigmoid
- Upload probs + end-bar prices + (end-bar + horizon) prices to GPU
- Sweep thresholds [0.00, 0.02, 0.05, 0.10, 0.15, 0.20, 0.25]
- Print per-threshold table + best Sharpe operating point
- GATE: per-trade Sharpe > 1.5 = deployable, 0.5-1.5 = marginal, < 0.5 = fail
Cost model: 0.25 price units round-trip = 1 ES.FUT tick = $12.50/contract.
Tunable via --cost-per-trade. Realistic for retail flow; brokers can
trade at half-tick or better.
GPU-pure on the hot path: kernels do per-trade math + reductions;
host only receives T (= 7 here) scalars per metric for final Sharpe
arithmetic. No GPU↔CPU roundtrip per trade.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bool flag with `default_value_t = true` doesn't accept `--stacker true`
on the command line in clap — it expects either the flag alone (which
inverts) or a custom action. Cleanest fix: drop the flag entirely;
the stacker block always runs when cal_frac is in (0, 1).
Phase 1d.3 stacker delivered the key result:
- Stacker test AUC: 0.7078 (raw Mamba: 0.6619, +4.6pts)
- Stacker test accuracy: 0.6683 (raw Mamba: 0.6187, +5.0pts)
- Stacker test Brier: 0.2088 (raw Mamba: 0.2299, well below chance 0.25)
- Stacker spread-Q4 accuracy: 0.8164 (raw Mamba spread-Q4: 0.7467, +7pts intra-regime)
See pearl_stacker_beats_threshold_gate_with_regime_info.md for full
write-up and design implications for Phase 1d.4 backtest.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Builds a second-stage MLP that takes [mamba_logit, 6 Block-S features]
as input (7 dims) and learns the joint alpha-and-regime score in one
calibrated output. Trains on the cal half of val (same 50/50 split as
Platt/isotonic so all comparisons are on the same held-out test bars).
Architecture: 7 → hidden_dim → 1 sigmoid, GELU activation, BCE-with-logits
loss, AdamW. Uses the existing GPU-native `MlpModel` from
crates/ml-alpha/src/mlp.rs — same primitives used for the Phase 1c MLP
baseline. No CPU compute on the hot path (per feedback_cpu_is_read_only);
all weights, activations, gradients, optimizer state on GPU; host writes
the input matrix to a pinned buffer once per batch via GpuTensor::from_host.
The Block-S columns are z-score normalised using cal-half statistics
(then applied to the full val matrix) before training; mamba_logit is
left raw since it's already close to standard-normal scale via the
Mamba's natural calibration (see pearl_mamba_sss_state_yields_native_calibration).
After training, reports stacker held-out accuracy + AUC + Brier +
log-loss, plus stratified accuracy by Block-S feature so we can see
whether the stacker absorbed the regime conditioning (uniform accuracy
across quintiles) or just sharpened the Q4-gate (still elevated in Q4).
Why this matters for production deployment per pearl_mamba_inherits_regime_structure:
- Single calibrated score for conformal coverage gating downstream
- Retrainable when market regimes drift
- Captures interactions between regime features that a static threshold
AND can't (e.g., spread-Q4 only when book is balanced)
- Replaces the planned Phase 1d.3 dual-head architecture with a smaller
stacked-generalisation approach (no separate regime classifier)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
After calibration, also stratify val accuracy across the 6 Block-S
features (time_since_trade, time_since_snap, book_event_rate, spread_bps,
L1_imbalance, micro_mid_drift) by sampling each val sequence's END BAR
feature value, then running `metrics_detail::stratified_accuracy` per
column with 5 quintile bins.
Tells us whether the K=6000 Mamba alpha concentrates in specific book
regimes (justifying an explicit regime head per Phase 1d.3) or is
uniform across regimes (allowing direct backtest in Phase 1d.4). The
Phase 1c stateless MLP showed strong stratification (spread-Q4 hit
0.752 acc on 76K samples while middle quintiles fell below 0.50);
this run tests whether the Mamba inherits or transcends that pattern.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Extends phase1d_long_horizon with a 50/50 val split (cal/test halves):
fit Platt and Isotonic on cal, evaluate on held-out test. Reports both
uncalibrated AND calibrated metrics (accuracy, AUC, Brier, log-loss).
Hypothesis: the K=6000 Mamba result (AUC=0.66 / acc=0.62 from the
uncalibrated 4cf9499b5 smoke) has a 4-point AUC-accuracy gap which
mirrors the Phase 1c pattern; Platt should compress that gap and lift
accuracy further without retraining.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The DECISIVE gate for FoxhuntQ-Δ's two-head architecture. The K-sweep
(commit db874b184) showed stateless single-snapshot alpha decays from
K=50 peak to gone by K=500. The two-head design exists to amplify
short-horizon evidence into long-horizon prediction via SSM state
accumulation; this smoke is the actual test of that hypothesis.
Gate per the implementation plan:
- AUC > 0.55 at K=6000 → multi-minute alpha confirmed, design validated
- AUC < 0.52 → decisive FAIL, design dead in current form
- 0.52 ≤ AUC ≤ 0.55 → marginal, tune or extend seq_len
New module `multi_horizon_labels.rs` generates labels at arbitrary K
with tie-drops + NaN guards (mirror of `purged_split::binary_direction_label`
semantics but bypassing Phase1aConfig's hardcoded K=100). 5 unit tests
covering: strict-ramp all-ones, constant-series all-tied, K-too-large
edge case, index alignment with mixed up/down/tied, non-finite drops.
New example `phase1d_long_horizon.rs` loads snapshot fxcache, generates
K=6000 labels, splits 80/20 with horizon-sized embargo (so train sequences
ending near boundary don't share forward-window prices with val), trains
Mamba2 (seq_len=32, hidden=64, state=16), reports val AUC.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Trains the from-scratch GPU-pure Mamba2 block against the snapshot fxcache,
gathers sequence batches via end-bar lookup into train/val labels, runs
AdamW for N epochs, computes val AUC.
First-shot result (epochs=3, stride=8, lr=1e-3, hidden=64, state=16, seq_len=32):
- Train BCE: 2.338 → 1.164 → 0.957 (monotone, still dropping)
- Val accuracy: 0.5645 (beats MLP 0.5241)
- Val AUC: 0.5684 (below MLP 0.6849)
Interpretation: undertrained (loss curve still descending steeply; stride=8
sees only 1/8 of data; lr=1e-3 conservative given the training-loop unit
test converged at lr=1e-2). Not yet a clean GATE FAIL — needs a retry with
stride=2, lr=3e-3, epochs=10-20 before declaring the model class has a
ceiling below the stateless MLP baseline.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Platt scaling drops held-out Brier from 0.346 → 0.221 (chance=0.250);
log-loss 1.096 → 0.632. Both Platt and isotonic land below chance baseline.
AUC-accuracy gap was pure miscalibration, not fundamental misexpression.
Learned: Platt a=0.32 (raw logits too extreme), b=0.89 (positive offset
needed). Trained MLP underconfidence on positives compounds with negative
prior. Proceed to 1d.1.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three things landing atomically because they're load-bearing for each other:
1. **Trend-scanning leakage fix** — trend_scanning.rs was emitting OLS slope+t-stat
over a *forward* window [t, t+L]. With the Phase 1a label = sign(price[t+60]
− price[t]), the forward feature window overlaps the label window, contaminating
it. Purged walk-forward only sterilizes forward-looking *labels* that cross
the train/val split, not forward-looking *features* that peek inside the same
horizon the label measures. The leak inflated MLP accuracy from 0.49
(legacy 74-dim baseline) to 0.75 — vanished to 0.50 after switching to a
trailing window. Bounded the perfect-fit t-stat sentinel from ±1e6 → ±20
(p<1e-30 is already meaningless); eliminated the 16k corruption-cap drops.
2. **Variable-dim alpha column** — fxcache schema now carries the alpha-feature
width via metadata (`alpha_feature_dim`), not a compile-time constant. Same
on-disk format hosts the 134-dim bar-level stack OR the 81-dim snapshot stack.
Reader + auto-detect honor the metadata-declared dim; downstream MLP auto-sizes
`in_dim`. Single schema, no forks.
3. **Snapshot pipeline (Phase 1c falsification)** — `snapshot_pipeline.rs`: 81-dim
per-MBP10-snapshot extractor reusing 10 snapshot-native alpha blocks + 6 new
snapshot-specific features (time-since-trade, time-since-snap, event-rate,
spread-bps, L1-imbalance, microprice-mid drift). `precompute_features` gets
`--row-unit snapshot` flag; emits one fxcache row per LOB update (1.97M rows
from MBP-10 data vs 206K for bar mode).
**Smoke verdict on real data** (ES.FUT, 1.97M snapshots, 384K val):
- Bar-level honest alpha: accuracy=0.5005, AUC=0.5043 (no signal)
- **Snapshot-level alpha**: accuracy=0.5241, AUC=0.6849 (real signal, 384K val)
- GBM corroboration: accuracy=0.5401 (non-linear partitioning sees more)
- Horizon decay: alpha peaks at K=20-50 snapshots (~5-25ms), gone by K=500
- Regime-conditional: spread-Q4 quintile hits 0.752 accuracy on 76k samples
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
