Commit Graph

13 Commits

Author SHA1 Message Date
jgrusewski
691dec144e fix(ml-backtesting): root-cause NaN/Inf book + duplicate close emissions
Two orthogonal bugs that compounded to produce the exit_px=±21474836
sentinel in cluster smokes (315 trades baseline, 500 trades pearl):

Bug A — NaN/Inf book propagation:
- apply_snapshot_kernel passes through NaN/Inf prices from MBP-10
  predecoded data (session boundaries, gap-fills).
- walk_ask_for_buy / walk_bid_for_sell accumulate cost += take * NaN.
- apply_fill_to_pos: avg_px = NaN; realized_pnl += NaN → permanent NaN.
- Subsequent close: (int)(NaN-derived * 5000) → i32::MAX sentinel.
- Fix: zero-out non-finite levels in apply_snapshot_kernel; add
  isfinite() guards in walk helpers as defense-in-depth; ATR update
  guards against non-finite mid.

Bug B — pnl_track close branch doesn't reset scratch:
- Scratch reset (full 24-byte memset to 0) was already present in the
  current code; no source change required for Bug B.

Tests:
- book_nan_inf_prices_dont_corrupt_realized_pnl: NaN at ask[3] + Inf
  at bid[5] survives the fill pipeline without making realized_pnl
  non-finite.
- pnl_track_resets_scratch_on_close: open+close+5 flat events emits
  exactly 1 record; second open+close emits exactly 1 more (=2 total).

14/14 stop_controller tests pass; all other ml-backtesting test suites
unaffected.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 08:28:12 +02:00
jgrusewski
bf619a2e7b feat(ml-backtesting): max-hold + exit_px defensive + spec criteria revision
Three follow-ups from cluster smoke gp74n (trade_vol pearl validation):

1. Max-hold force-close: max_hold_ns added as per-backtest config
   (default 0 = disabled). Fires force-flat (3, 0) when current_ts -
   entry_ts >= max_hold_ns, BEFORE SL/trail check. Tested via
   max_hold_forces_close. Sweep YAML sets 60s cap to bound the long
   tail observed in gp74n (263985s pathological hold).

2. exit_px defensive sanity check: 500/1024 gp74n trades reported
   exit_px = ±i32::MAX/100 (float→int saturation sentinel from likely
   NaN segment_realized). Defensive fix in pnl_track_step: if exit_px
   is non-finite or diverges from entry_px by >10%, fall back to
   entry_px with zero realised_pnl. Root cause to be traced separately.

3. Spec §9.2 criteria revision: mean_hold<30s replaced with p95<600s
   + max<=max_hold_ns. The 30s threshold reflected the pre-pearl
   sub-cost churning bug, not a real feature criterion. p95 catches
   long-tail pathology while letting alpha-driven exit timing breathe.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 08:10:00 +02:00
jgrusewski
9b29f9fd0a feat(ml-backtesting): trade-vol floor replaces 2×cost literal in stop_check_isv
Replaces the Task 12 `2.0f * cost` floor (hardcoded multiplier) with
trade_vol = sqrt(realised_return_var) bootstrapped from cost². Per
pearl_trade_level_vol_for_stop_distance.md: microstructure ATR is the
wrong time scale for trade-level stop decisions; per-horizon
realised_return_var is the right one, with cost² as a structural cold-
start sentinel.

cost now appears exactly once — inside the sqrt as a bootstrap sentinel,
never as a distance multiplier. The 2.0f literal is eliminated;
controller is fully ISV-driven.

var_avg accumulates realised_return_var in the same single-pass horizon
loop as ema_loss/ema_win. Cold-start (var_avg=0): trade_vol = cost.
Post-bootstrap: sqrt(var_avg) dominates.

Test retargeted: cost_floor_prevents_sub_cost_stops →
trade_vol_floor_prevents_sub_cost_stops, with boundaries straddling
trade_vol=cost=0.125 instead of the prior 2*cost=0.25 (no-fire Δ=0.08,
fire Δ=0.20).

Spec §5, §10, §11, §12 amended.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 01:45:37 +02:00
jgrusewski
da6e887174 feat(ml-backtesting): cost-floor in stop_check_isv prevents sub-cost churning
Cluster smoke w7b4p showed 44864 closed trades in 1M events at 200ms
latency — physically impossible without sub-cost churning. Root cause:
sl_distance = max(pnl_ema_loss, atr) ≈ 0.05 at cold-start, but round-trip
cost = 2 × 0.125 = 0.25. Every stop-out guaranteed loss > 5× distance;
EMA converges sub-cost.

Amendment: triple-max sl_distance = max(pnl_ema_loss, atr, 2*cost);
trail_distance = max(pnl_ema_win, atr, 2*cost). cost is an ISV-discipline
strategy anchor (already per-backtest from P4 — no new state slot, no
tuned constants). Added cost_per_lot_per_side_per_b param to both decision
kernel signatures and threaded cost_per_lot_per_side_d through both launch
sites in step_decision_with_latency.

Test: cost_floor_prevents_sub_cost_stops validates unrealized in (ATR, 0.25)
does NOT fire SL; unrealized > 0.25 DOES fire. Spec amended (§10, §12).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 01:27:28 +02:00
jgrusewski
4101796ad9 refactor(ml-backtesting): delete StopRules + use_cold_start_stopgap atomically
- StopRules struct + sl_tp_rules field + all literals deleted; the ISV
  stop controller (Tasks 2-9) replaces this dead data path.
- use_cold_start_stopgap propagation deleted across harness,
  batched_config, fxt-backtest, sweep_smoke.yaml, and 5 test files.
- Q1 stopgap branch in harness.rs deleted; replaced with the original
  simple strategy-upload loop.
- decision_floor_coldstart test retargeted: cold_start_persistent_
  bullish_with_default_stops_never_closes -> ..._now_closes, asserts
  trades > 0 AND |pos| <= max_lots (99 trades, pos=1 on local GPU).
- CBSW spec + plan prefixed with SUPERSEDED headers.

Per feedback_single_source_of_truth_no_duplicates +
feedback_no_partial_refactor: contract change migrates every consumer
in one commit. No legacy wrappers; no version suffixes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 00:40:14 +02:00
jgrusewski
0c4b106394 feat(ml-backtesting): pnl_track_step resets trail_hwm on close transition
Two-instruction addition to the existing close-emission branch
(prev != 0 && now == 0). Without the reset, the next entry inherits
a stale HWM that could arm the trail spuriously on event 0 of the
new trade.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 00:31:05 +02:00
jgrusewski
2e0434a84c feat(ml-backtesting): target-delta semantics in seed_inflight_limits_batched
market_targets[b] now interpreted as target absolute position (side=0
long, side=1 short, side=2 no-op preserved, side=3 force-flat).
seed_inflight_limits_batched computes order_lots = target_signed -
effective_position where effective_position sums pos.position_lots
plus all unfilled signed slot sizes (active in {1, 2}). Fixes both
the additive accumulation bug AND the worse latency-bypass variant
(naive delta would have made things 200x worse under 200ms latency).

3 new tests added (all passing):
- position_target_not_additive: latency=0, 10 repeated target events stay <= max_lots=5
- position_target_not_additive_with_latency: 200ms latency, 500 events, in-flight summation prevents runaway
- alpha_noop_side_2_preserved: side=2 events leave filled position unchanged

All 9 stop_controller tests pass; parallel_sim and fuzz regression clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 00:28:01 +02:00
jgrusewski
1d852a994d feat(ml-backtesting): wire stop_check_isv into decision_policy_program
Both decision kernels now call the same __device__ helper at the top
of per-backtest dispatch (after the program_lens early-return). Single
source of truth for stop logic across default and bytecode-VM paths.
Parity test ensures they produce identical market_target output for
matched scenarios.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 00:20:40 +02:00
jgrusewski
dcffb11f5d test(ml-backtesting): multi-horizon mask averaging invariant test
Boundary test that fails on bit-pick-first or bit-pick-max regressions
of the open_horizon_masks averaging in stop_check_isv. Per-horizon
pnl_ema_loss values (2, 4, 3, 3, 3) yield mean=3.0 across the 0x1F
mask; snapshots placed relative to vwap_entry for ask-spread safety.
Tests Δ_entry=2.5 (no-fire) and Δ_entry=4.5 (fire).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 00:15:26 +02:00
jgrusewski
7e1995d0ad feat(ml-backtesting): trail-TP trigger with HWM ratchet in stop_check_isv
trail_distance = max(pnl_ema_win, atr_mid_ema). HWM ratchets up via
fmaxf each event while position open; trail fires when HWM clears
the arming threshold AND unrealized drops by trail_distance from
HWM. Same force-flat (3, 0) write as SL.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 00:08:48 +02:00
jgrusewski
4aca6330f0 feat(ml-backtesting): hard-SL trigger in stop_check_isv
sl_distance = max(pnl_ema_loss, atr_mid_ema) per
pearl_blend_formulas_must_have_permanent_floor. Fires force-flat
(3, 0) when unrealized_pl_per_lot drops below -sl_distance.
Multi-horizon mask averaging + trail trigger land in Tasks 5-6.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 00:04:07 +02:00
jgrusewski
2cb4dfed28 feat(ml-backtesting): stop_check_isv __device__ helper + gate-on-flat
Skeleton helper called from decision_policy_default at top of
per-backtest dispatch. Returns 0 (no-op) on flat positions; trigger
logic for open positions added in Tasks 4-6. Single source of truth
for stop logic across decision_policy_{default,program}.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 23:54:48 +02:00
jgrusewski
dc8c37e11e feat(ml-backtesting): ATR-EMA on mid-price in book_update_apply_snapshot
Per-event Wiener-α=0.4 EMA on |Δmid| with first-observation bootstrap.
Floor source for the SL/trail-distance controller (spec §6). Thread 0
of the broadcast-snapshot kernel handles the update per backtest;
threads 1..9 still handle the 10 level writes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 23:49:02 +02:00