gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
5,588 Commits 13 Branches 0 Tags
4bed8f2dbfb46387e263e06ef40a4c1948de35bf
Commit Graph

46 Commits

Author SHA1 Message Date
jgrusewski
da2ad438ca feat(rl): R2 — fee model + loader pair API 
Ports two scope-complete fixes from the ml-alpha-phase-f-g-flawed
reference branch:

A7 fee model:
- order_match.cu::submit_market_immediate gains 2 new kernel args
  (cost_per_lot_per_side, total_fees_per_b) mirroring the per-fill
  fee deduction in resting_orders.cu::apply_fill_to_pos:213-219.
  Fee deducts from pos.realized_pnl on EVERY fill (open, scale-in,
  counter); total_fees_per_b accumulates for telemetry.
  Single-writer-per-block plain += is safe — line 79 has
  `threadIdx.x != 0 → return` (feedback_no_atomicadd).
- LobSimCuda::submit_market launch updated to pass the 2 new args.
- LobSimCuda::upload_cost_per_lot_per_side host API lets callers
  configure ES-realistic fees (≈$1.25/contract/side). Default
  alloc_zeros = $0; production decision-policy path is unchanged
  (uploads its own cost via step_decision_with_latency).

A8 loader pair API:
- MultiHorizonLoader::next_sequence_pair returns
  (LabeledSequence, LabeledSequence) at adjacent anchors in the
  same source file. anchor_t sampled from [min_anchor, max_anchor−1)
  so anchor+1 also fits the upper-bound. Counts as ONE yielded
  sequence against n_max_sequences.
- next_sequence_random and next_sequence_pair share a new private
  helper build_sequence_at(lf, anchor) -> LabeledSequence that
  contains the multi-resolution windowing logic. Single source of
  truth for the build (feedback_single_source_of_truth_no_duplicates).
- next_sequence's caller-facing contract (random anchor, one
  sequence per call) is unchanged — alpha_train.rs supervised
  pipeline keeps working as-is.

No new local tests this phase per the rebuild plan (R2): the fee
deduction with default cost=0 is a no-op for existing callers, and
G7 in R6 covers the with-fees path via a rebuilt reward_calibration
test driving LobSimCuda directly (no LobEnv adapter).

cargo check -p ml-alpha -p ml-backtesting + cargo build --tests on
ml-backtesting both green; baseline test suite unaffected.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-23 09:48:38 +02:00
jgrusewski
0c8b4b5608 refactor(per-horizon): N_HORIZONS 5→3 — ml-backtesting full propagation 
Source migration:
- crates/ml-backtesting/src/lob/mod.rs:5 + policy/mod.rs:18: local
  const N_HORIZONS 5→3 via re-export from ml_alpha::heads::N_HORIZONS
  (single source of truth across crates)
- crates/ml-backtesting/src/sim/mod.rs:1751,1773,1784: [IsvKellyStateHost; 5]
  array literals → ; N_HORIZONS]
- crates/ml-backtesting/cuda/lob_state.cuh:9: #define N_HORIZONS 5→3
  (this triggers cubin rebuild of decision_policy + 4 other ml-backtesting
  kernels that #include this header)

Test migration (8 test files + 2 JSON fixtures):
- threshold_and_cost.rs, decision_floor_coldstart.rs, parallel_sim_
  correctness.rs, stop_controller.rs (37+10+1 broadcast_alpha calls),
  lob_sim_integrated_fuzz.rs, lob_sim_fixtures.rs: hardcoded [f32; 5]
  alpha-probs and [IsvKellyStateHost; 5] arrays → N_HORIZONS-sized via
  std::array::from_fn or [v; N_HORIZONS] literals
- trainer_parity.rs:34 + ring3_replay.rs:47: horizons literal
  [30,100,300,1000,6000] → ml_alpha::heads::HORIZONS
- fixtures/decision_alpha_buy_close.json + decision_program_h4_only.json:
  5-element warm_start_isv_kelly / alpha_probs / expected_isv_kelly_after
  trimmed to 3 elements; active horizon relocated to N_HORIZONS-1

Library lib-test rewrite (per pearl_tests_must_prove_not_lock_observations):
- crates/ml-backtesting/src/policy/mod.rs:197-233: lib tests
  default_strategy_has_5_horizon_leaves + ..._flattens_to_5_emits...
  renamed to N_HORIZONS-parametric form (observed-value 5 and 7
  were bug-locks).

cargo check -p ml-backtesting --all-targets: clean.
cargo test -p ml-backtesting --lib: 33 passed.
cargo test -p ml-backtesting --tests (non-CUDA, non-fixture-data): 2 passed,
53 ignored (CUDA-gated or FOXHUNT_TEST_DATA-gated).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 01:37:23 +02:00
jgrusewski
0384f76743 fix(decision-policy): use signed-conviction EMA, not magnitude-only EMA 
Anti-calibration evidence (smoke ckpts across architecture variants
showed higher reported conviction → MORE negative PnL, -15.6 → -145.2)
traced to a magnitude/direction mismatch in the sizing formula:

  conv_ema = EMA(|conviction_signed|)         # magnitude smoothing
  target_lots = sign(conviction_signed)       # INSTANTANEOUS sign
              × conv_ema × max_lots           # × SMOOTHED magnitude

When per-horizon directions disagree event-to-event, the magnitude EMA
stays high (|x| EMA can't cancel) but the sign flips on noise. Result:
big trades in random direction whenever the 5 horizons disagree.

Replace with a single signed-conviction EMA:

  conv_signed_ema = EMA(conviction_signed)
  target_lots = sign(conv_signed_ema)
              × |conv_signed_ema| × max_lots

Now BOTH sign AND magnitude come from the same smoothed signal. When
directions disagree, signed EMA → 0 collapses size to zero naturally.

Atomic migration across decision_policy.cu (2 kernels), pnl_track.cu
(open-branch reads the SIGNED buffer and takes fabsf for the magnitude-
semantics open_trade_state offsets that composite_exit_check forms ratios
from), and src/sim/mod.rs (renamed device buffers + accessor + 4 launch
sites). No legacy aliases per feedback_no_legacy_aliases; no parallel
buffers per feedback_single_source_of_truth_no_duplicates; α floor 0.4
preserved per pearl_wiener_alpha_floor_for_nonstationary; first-
observation bootstrap preserved per pearl_first_observation_bootstrap.

ml-backtesting lib: 33 passed.
GPU oracle: signed_conviction_ema_collapses_on_sign_flips passes —
observed steady-state shows |signed_ema| ≈ 0.205 (post-fix) producing
|target_lots| = 2 every tail event, vs the pre-fix bug's predicted
|target_lots| = 7 every tail event. Existing
multi_horizon_conviction_cancels_on_disagreement still passes (zero-
cancellation regime is even cleaner under signed EMA).
ml-alpha lib: 33 passed (no regression).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 00:26:22 +02:00
jgrusewski
0e17fd4f2d diag(crt-2): per-horizon alpha-input EMA test — hypothesis investigation 
Driven by ffr59 (commit b44a97ff9) findings: all 5 horizons flip
direction every 2.5 events; win rate flat 24% across conviction; mean
PnL anti-correlated with conviction. Hypothesis: per-event alpha output
is high-frequency noise on top of a slower signal. If true, smoothing
input alpha BEFORE the conviction formula should reduce direction flips
and recover signal.

Adds Wiener-α adaptive EMA on raw alpha_probs[h] for each horizon,
applied BEFORE the multi-horizon conviction formula. Floor at 0.1
(stronger than the 0.4 floor on the output-side conviction EMA — this
tests whether INPUT smoothing has different impact than OUTPUT smoothing).

Three new device slots:
  - alpha_ema_per_b_per_h (per-horizon EMA state)
  - alpha_diff_var_per_b_per_h (variance of changes)
  - alpha_sample_var_per_b_per_h (variance of value)

The CRT.diag Group A direction-flip counter still reads RAW alpha_probs
so we have a head-to-head comparison: raw flip rate vs smoothed flip rate.
Group E adds the smoothed-direction counter + mean run length.

End-of-run log adds one line per horizon:
  crt_diag h<X> smoothed: flips=Y mean_run_len=Z events (vs raw F / M)

If smoothed mean_run_len >> raw mean_run_len: hypothesis is RIGHT, the
input had signal under the noise. Next step would be to make this an
operational EMA in the controller.

If smoothed and raw are similar: hypothesis is WRONG, per-event output
is genuinely noisy. Next step would be to investigate model training
(horizon collapse) OR the AUC=0.66 measurement definition.

Either way, definitive result from one smoke run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 22:34:35 +02:00
jgrusewski
b44a97ff94 diag(crt): empirical measurement battery for signal × controller × cost analysis 
Pure-instrumentation commit. Zero behavior change. The Gate CRT.1 smoke
p9cnk (commit 1e656948b) produced 222k trades and 29688% drawdown,
worse than the failed CRT.A. The structural fixes (multi-horizon
conviction, no-trade band, composite exit) all work as designed — but
the trade cadence is fundamentally too fast for the signal/cost ratio.
Before another round of threshold tuning, measure the structural truth.

Adds device-side counters dumped at smoke close (single end-of-run
memcpy_dtoh batch in read_diagnostics(), NOT per-event):

Group A — per-horizon signal persistence:
  - flip_count[h]:        total direction-sign flips
  - sum_run_length[h]:    cumulative events between flips (u64)
  - run_length_hist[h]:   5-bucket log-scale histogram (1-9, 10-99,
                          100-999, 1k-9.9k, 10k+)
  Output: mean_run_len per horizon = signal half-life proxy

Group B — conviction smoothed-EMA distribution:
  - conv_hist[10]:        10-bucket histogram over [0, 1)
  Output: shape of conviction distribution at decision time

Group C — hold-time distribution:
  - hold_hist[6]:         <1s, 1-10s, 10-60s, 1-10m, 10-60m, >1h
  Output: empirical distribution of trade hold times

Group D — outcome by entry-conviction:
  - outcome_n[10]:        trades per conviction bucket
  - outcome_sum_pnl[10]:  cumulative realized PnL per bucket (price-units)
  - outcome_n_wins[10]:   wins per bucket
  Output: validates "high conviction = better trades" hypothesis OR
          surfaces signal miscalibration

Per-backtest memory: ~330 bytes. Negligible vs existing state.

All counters single-writer-per-block (threadIdx.x == 0 convention; no
atomicAdd per feedback_no_atomicadd). All updates kernel-internal (no
host roundtrip per event). The end-of-run dump uses memcpy_dtoh in
read_diagnostics() — called once at harness termination from the
post-stop_ctrl_counters block, never per-event. Per
feedback_no_htod_htoh_only_mapped_pinned, the hot path is untouched.

N_HORIZONS = 5 (heads.rs canonical {30, 100, 300, 1000, 6000}); the
plan text says 4 but the workspace constant is 5 — used the code value.

Output at end of cluster smoke as a `crt_diag` log block:
  crt_diag h30: flips=X mean_run_len=Y events
  crt_diag h30 run_length_hist: 1-9:N 10-99:N 100-999:N ...
  ...
  crt_diag conv_ema_hist: [0.0-0.1]:N [0.1-0.2]:N ...
  crt_diag hold_time_hist: <1s:N 1-10s:N ...
  crt_diag outcome_by_entry_conv[0.0-0.1]: n=N win_rate=X.X% mean_pnl_pu=Y.YYY
  ...

The output drives the next round of CRT.1 threshold design (delta_floor,
composite exit factors, min-hold cooldown) from data instead of guesses.

Per pearl_adaptive_not_tuned and pearl_controller_anchors_isv_driven:
thresholds will become ISV-derived in CRT.2; CRT.1 must first establish
defensible defaults from the empirical signal characteristics.

Verified: stop_controller (22/22 pass), decision_floor_coldstart (3/3
pass), threshold_and_cost (3/3 pass), parallel_sim_correctness (1/1
pass), lob_sim_integrated_fuzz (3/3 pass). Two lob_sim_fixtures
failures (fix_decision_alpha_buy_close, fix_decision_program_h4_only)
were pre-existing at HEAD 1e656948b — not caused by this commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 21:56:07 +02:00
jgrusewski
1e656948b3 arch(crt-1): composite exit_signal safety circuit-breaker (C1.4) 
Per spec §4.3 + plan C1.4. Safety backup for the primary exit path
(target → 0 from collapsed multi-horizon conviction, C1.2). Catches
the edge case where conviction stays bounded above zero but the trade
is deep in unrealized loss, or short/long horizons disagree for many
events while threshold-gate logic still permits sizing.

Three terms, max-of-three composite:
  degradation_term  = (1 − conv_ema_now / conviction_at_entry) / 2.0
  disagreement_term = disagree_consecutive_events / 32.0
  pnl_decay_term    = max(−pnl_adjusted_conv_ema, 0) / 1.0
  exit_signal       = max(deg, dis, pnl_decay)

Fires force-flat (market_targets = (3, 0)) when exit_signal >= 1.0.

State-update flow:
  - pnl_track open branch: write conviction_at_entry (offset 20),
    conviction_ema (offset 24, initialised to entry value),
    pnl_adjusted_conviction_ema (offset 44, = 0),
    peak_unrealized_pnl (offset 48, = 0). Takes new conviction_ema_per_b
    read-only arg to snapshot at open.
  - decision kernel intra-event (when pos open): update offsets
    24/44/48/56 (disagreement counter) via update_open_trade_trajectory.
  - composite_exit_check device helper: reads offsets 20/24/44/56,
    writes (3, 0) on fire. Sibling of stop_check_isv with same return
    semantics; called after trajectory update so all four reads see
    the freshly-written values.

Fields not read in CRT.1 (offsets 28-43 per-horizon EMA, 52 degradation
counter, 60 horizon_idx_dominant) are NOT written by the open branch —
alloc_zeros covers init. Per feedback_no_hiding: don't populate unused
fields. Also fixes a latent C1.1 bug: pnl_track close branch read
horizon_idx from offset 20 (now conviction_at_entry, f32); moved the
read to offset 60 per the documented layout.

Threshold-bootstrap defaults (will be ISV-derived in CRT.2 alongside
the rest of the controller anchors per pearl_controller_anchors_isv_driven):
  degradation_factor  = 2.0  (50% conviction decay vs entry → ≤0.5 term)
  disagreement_factor = 32.0 (32 events of short/long disagreement)
  pnl_decay_factor    = 1.0  (pnl_adj_conv structurally ∈ [-1, +1])

Under these defaults disagreement_term is the only term that can fire
on its own — degradation_term caps at 0.5 (conv_ema/conviction_at_entry
≥ 0) and pnl_decay_term caps at 1.0 in the limit. The three terms still
add evidence in superposition; CRT.2 will retune factors against ISV
percentiles. Unit-test coverage validates the disagreement path
(composite_exit_signal_fires_on_disagreement).

Per pearl_no_host_branches_in_captured_graph — all composite logic
lives inside the decision kernel; no host roundtrip in the per-event
hot path. Per feedback_no_htod_htoh_only_mapped_pinned — no new
memcpy or stream.synchronize() in step_pnl_track or
dispatch_latent_market_orders.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 21:15:40 +02:00
jgrusewski
7c850a6c02 arch(crt-1): no-trade band in seed_inflight — sparse action at signal cadence (C1.3) 
Per spec §4.0 + plan C1.3. Without this gate, every fractional change
in target_lots seeds an order via target-delta semantics; combined with
the continuous controller invocation (every event after A1) this
generates hyperactivity even with the multi-horizon §4.4 conviction
formula (C1.2). v2 Gate-1 failures confirmed: scalar EMA + amplification
clamp + per-event controller = 155k trades on a 2M-event smoke.

The band absorbs fractional target adjustments so most events produce
no order. When the signal genuinely shifts and target moves enough to
cross the band, the kernel seeds. Continuous evaluation, sparse action.

Greenfields atomic refactor — single config knob threaded through every
layer in one commit:
  SweepBase.delta_floor: f32  (YAML, default 1.0 = 1 lot)
  SimVariant.delta_floor: Option<f32>  (per-variant override)
  ResolvedSimVariant.delta_floor: f32
  UniformSimParams.delta_floor: f32
  BatchedSimConfig.delta_floor: Vec<f32>
  LobSimCuda.delta_floor_d: CudaSlice<f32>
  seed_inflight_limits_batched kernel param + skip-if-below-floor logic

New accessor: LobSimCuda::read_inflight_count(b) — counts active != 0
limit slots for backtest b. Not cfg(test); follows existing read_limit_slot
pattern.

Test: no_trade_band_blocks_micro_delta verifies that a second decision
with the same strong-bullish alpha (same target, effective = in-flight
lots) produces delta=0 and does not re-seed. max_lots=1 keeps the
arithmetic unambiguous.

Existing tests: all UniformSimParams struct literals updated with
delta_floor=0.0 (band disabled) so existing behaviour is preserved.
harness.rs from_uniform path uses delta_floor=1.0 (production default).

Hot-path discipline: the delta_floor_d upload happens once per run in
the existing config-upload block (alongside threshold, cost, max_hold_ns);
the kernel reads the device slot per-event without any host roundtrip. No
memcpy_htod / dtoh / dtov / synchronize introduced on the per-event path.

Per pearl_controller_anchors_isv_driven, this floor is currently a
config constant; CRT.2 (Phase 2) makes it ISV-derived from rolling
spread cost / signal volatility.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 20:59:50 +02:00
jgrusewski
632296021c arch(crt-1): multi-horizon ISV-weighted conviction replaces scalar EMA + rescale (C1.2) 
Per spec §4.4 + plan C1.2. Replaces v2 A2 (1d889d2de) + A2.1
(fe2498769) scalar approach that failed Gate 1 catastrophically on
smokes vjmwc and lkrdf (155k trades, 9100% drawdown, Sharpe -15.7).

The structural fix: direction comes from per-horizon SIGNED sum, so
disagreeing horizons cancel. The scalar EMA on max(|p-0.5|) could only
smooth magnitude — it could not smooth direction jitter. Only this
multi-horizon weighted formula can.

Formula (spec §4.4):
  weight_h           = max(isv[h].net_edge, ε) / (isv[h].var + cost²)
  weighted_h         = magnitude_h × weight_h × direction_h
  conviction_signed  = sum(weighted_h) / sum(|weight_h|)
  conviction_ema     = Wiener-α EMA(|conviction_signed|)   (reuses A2 slots)
  target_lots        = round(sign(conviction_signed) × conviction_ema × max_lots)

Pearl conformance:
  - pearl_controller_anchors_isv_driven: weights from ISV state
  - pearl_one_unbounded_signal_per_reward: net_edge/(var+cost²) is the
    single unbounded factor; magnitude × direction is bounded in [-1,1]
  - pearl_zscore_normalization_for_magnitude_asymmetric_signals: divide
    by total_abs_weight for scale invariance
  - pearl_trade_level_vol_for_stop_distance: cost² is variance bootstrap
  - pearl_blend_formulas_must_have_permanent_floor: ε = cost · 0.01 on
    net_edge — no cold-start regime switch
  - pearl_audit_unboundedness_for_implicit_asymmetry: net_edge is
    one-sided positive (losing-edge horizons drop out, not flip sign)
  - pearl_wiener_alpha_floor_for_nonstationary: α floor at 0.4 (unchanged
    EMA mechanism, repointed at the multi-horizon scalar)

DELETED:
  - The `final_size *= (conv_ema/raw_max_conv)` aggregate rescale step
    (the v2 bug that amplified weak signals)
  - The per-horizon `signed_sizes[h]` Kelly + Sharpe-weight aggregation
    loop in decision_policy_default (replaced by direct §4.4 formula)
  - Three obsolete tests asserting on the old scalar path:
    * conviction_ema_smooths_micro_oscillations
    * conviction_ema_does_not_lag_reversals
    * conviction_ema_rescale_never_amplifies_weak_signal
    * cfg_high_kelly_floor helper (only used by deleted tests)
  - cold_start_with_zero_floor_reproduces_old_bug (tested OLD kelly_floor
    semantics that v3 doesn't expose)
  - cold_start_persistent_bullish_now_closes (was passing under v2 via
    target oscillation from integer rounding; v3 produces stable target
    on stable signal — closes need price movement)
  - alpha_noop_side_2_preserved (anchored on the v2 kelly_frac_floor
    integer-truncation path that doesn't exist under v3)

Applied to BOTH decision_policy_default and decision_policy_program
(bytecode VM at OP_WRITE_ORDER). Per feedback_no_partial_refactor —
both consumers migrate atomically. The VM's per-horizon emit + aggregate
opcodes still execute but their stack `final_size` is unused at
OP_WRITE_ORDER (the §4.4 conviction-driven target replaces it). The
stack's attribution mask is still consumed for entry crediting.

The three conviction-EMA device slots (conviction_ema_d, _diff_var,
_sample_var) STAY in LobSimCuda. The Wiener-α EMA mechanism is reused
on the multi-horizon conviction scalar; the device-helper signature
changed from taking alpha_probs[] to taking the precomputed scalar
|conviction_signed|.

Kernel ABI change: decision_policy_default no longer takes
target_annual_vol_units / annualisation_factor / kelly_frac_floor /
sharpe_weight_floor (the §4.4 formula doesn't use them). Removed from
both the CUDA signature and the launch builder in sim/mod.rs. The
bytecode VM (decision_policy_program) still consumes those params for
its OP_EMIT_PER_HORIZON_SIZE / OP_AGG_WEIGHTED_SHARPE opcodes — their
device slots remain allocated.

Test: multi_horizon_conviction_cancels_on_disagreement verifies the
structural fix — bullish [0.7,0.3,0.7,0.3,0.5] horizons cancel to
no-trade output (side=2/3, size=0, conv_ema≈0).

Existing tests rebased: cfgs that used cost_per_lot_per_side=0.0 now
use 1.0 (the §4.4 formula's eps_edge floor requires cost > 0). Each
affected test file documents the rebase inline. Tests that were
anchored on observed v2-kernel values (per
pearl_tests_must_prove_not_lock_observations) are deleted; tests with
genuine invariants (boundedness, direction-sanity, stop-controller
behavior) are preserved.

Hot-path discipline: no memcpy_htod/dtoh/dtov/synchronize introduced.
Only kernel-internal computation; one launch arg removed from the
default decision kernel.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 20:50:43 +02:00
jgrusewski
e27fc90b9b arch(crt-1): open_trade_state 24→64 byte expansion — atomic refactor (C1.1) 
Per spec §7 (v3 architecture reset, promoted from v2 Phase B into CRT.1).
Single-cache-line layout. Every reader and writer of open_trade_state
migrates in this commit per feedback_no_partial_refactor.

New 64-byte layout (existing 24-byte fields preserved at original offsets):
  Offset  Size  Field
    0     8     entry_ts_ns                              (u64)
    8     4     entry_px_x100                            (i32)
   12     4     entry_size                               (i32)
   16     4     realized_at_open                         (f32)
   20     4     conviction_at_entry                      (f32)  ← NEW
   24     4     conviction_ema                           (f32)  ← NEW
   28    16     conviction_per_horizon_ema [4 × f32]    ← NEW
   44     4     pnl_adjusted_conviction_ema              (f32)  ← NEW
   48     4     peak_unrealized_pnl                      (f32)  ← NEW
   52     4     degradation_consecutive_events           (u32)  ← NEW
   56     4     disagreement_consecutive_events          (u32)  ← NEW
   60     1     horizon_idx_dominant_at_entry            (u8)   ← NEW
   61     3     pad

The 24-byte prefix is unchanged so downstream readers (stop_check_isv in
decision_policy.cu, max_hold event-rate check in resting_orders.cu) need
only update the stride constant. The new fields at offsets 20..63 are
populated by subsequent CRT.1 tasks (C1.2 multi-horizon conviction, C1.4
composite exit signal).

Open branch in pnl_track.cu zeros the new fields implicitly because
alloc_zeros on the device slot zero-initialises everything; subsequent
writes only touch the 0-23 byte range (existing fields). Close branch
reset-loop iterates OPEN_TRADE_STATE_BYTES which now zeros all 64.

Files changed:
  crates/ml-backtesting/src/lob/mod.rs  — pub const OPEN_TRADE_STATE_BYTES: usize = 64
  crates/ml-backtesting/cuda/pnl_track.cu  — #define OPEN_TRADE_STATE_BYTES 64
  crates/ml-backtesting/cuda/resting_orders.cu — comment + stride literal 24→64
  crates/ml-backtesting/tests/stop_controller.rs — open_trade_state_64_byte_layout test

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 20:25:05 +02:00
jgrusewski
fe24987690 fix(crt-a2.1): clamp conviction-EMA rescale to [0, 1] — never amplify weak signals 
Gate 1 catastrophic failure (smoke vjmwc, commit 3d8f12deb):
  - 62× hyperactivity (157,470 trades vs 2,511 baseline)
  - 9,347% max-drawdown ($327M loss on $3.5M base)
  - Sharpe -15.63 (2.4× worse than baseline)

Root cause: A2's formula
    final_size *= (conv_ema / raw_max_conv)
amplified weak signals because EMA tracks the MEAN of raw_max_conv, NOT a
running max as the author's comment claimed. When raw_max_conv < conv_ema
(normal during quiet periods between strong signals), the multiplier was
> 1, blowing up small-signal positions:
  raw=0.05, ema=0.3 → 6× amplification
  raw=0.01, ema=0.3 → 30× amplification

Fix: clamp scale ∈ [0, 1] in BOTH decision_policy_default and
decision_policy_program (OP_WRITE_ORDER). Only damp when raw is stronger
than EMA (scale < 1); never amplify when raw is weaker (scale capped at 1).

Math:
  scale = conv_ema / raw_max_conv     // can be 0..∞
  scale_clamped = min(scale, 1.0)     // can only be 0..1
  final_size *= scale_clamped

Test: conviction_ema_rescale_never_amplifies_weak_signal validates
target_lots stays ≤ 1 when alpha=0.51 (raw_conv=0.02) after EMA warmup
at alpha=0.7 (ema~=0.4). Without the clamp the broken multiplier is 20×.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-20 19:38:50 +02:00
jgrusewski
1d889d2de9 arch(crt-a): Wiener-α conviction-EMA smoothing in decision_policy 
After A1 (commit 045850e8f) the controller fires every event. Without
smoothing, target lots would oscillate as alpha probabilities jitter
event-to-event, generating hyperactive spread-bleed from back-and-forth
trades.

Ships per spec §4.2 and §3.7: Wiener-α adaptive EMA on max-conviction-
across-horizons. Formula:
  α_raw    = diff_var / (diff_var + sample_var + ε)
  α_active = max(α_raw, 0.4)        per pearl_wiener_alpha_floor_for_nonstationary
  ema      = α_active × new + (1 − α_active) × prev
First-observation bootstrap: prev_ema == 0 → replace directly per
pearl_first_observation_bootstrap.

Three new device slots (per-backtest):
  - conviction_ema_d:             smoothed conviction (used for final-aggregate rescale)
  - conviction_diff_var_ema_d:    second-order EMA, drives adaptive α
  - conviction_sample_var_ema_d:  second-order EMA, drives adaptive α

Architectural choice: rescale at the final aggregate (final_size *= conv_ema /
raw_max_conv), not at per-horizon sig_mag. Per-horizon sizing keeps raw
|alpha-0.5|*2 so horizons with no signal (alpha=0.5) contribute zero —
the spec §4.2 "smoothed conviction" is a unified gain over the aggregate,
not a substitute for per-horizon signal strength. At bootstrap the scale
is exactly 1.0 (raw_max_conv == conv_ema) so the very first decision is
bit-identical to the pre-A2 kernel. Direction recovery (sign of
alpha-0.5) remains per-horizon — genuine reversals respond at event rate.

Threaded through both decision_policy_default and decision_policy_program
kernels' signatures and launches in sim/mod.rs.

Full multi-horizon conviction *aggregation* (spec §4.4) remains Phase B;
A2 ships ONLY the smoothing operator on the existing scalar conviction.

Pure on-device: no memcpy_htod / dtoh / dtov / synchronize in hot path.
Single test-only DtoH accessor (read_conviction_ema) for unit-test
inspection of the smoothed value.

Tests:
- conviction_ema_smooths_micro_oscillations — sentinel→bootstrap→bounded
  EMA across 10 alternating high/low all-bullish alpha drives; direction
  stable.
- conviction_ema_does_not_lag_reversals — sign flip in alpha → target
  side flips next event.
- All 22 stop_controller + 5 decision_floor_coldstart + 3 threshold_and_cost
  + parallel_sim + ring3_replay + trainer_parity + 6 fuzz tests pass.
- 2 lob_sim_fixtures tests (fix_decision_alpha_buy_close, fix_decision_program_h4_only)
  were already failing on baseline 045850e8f pre-A2; not a regression from
  this commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 19:01:09 +02:00
jgrusewski
045850e8f3 arch(crt-a): delete decision_stride field — greenfields atomic refactor 
Per spec 2026-05-20-continuous-reasoning-trader-design.md §3.3 and §8:
decision_stride is REMOVED, not deprecated. No backwards-compat shim,
no fallback. Every consumer migrates in this commit per
feedback_no_partial_refactor.

Removed from:
  - bin/fxt-backtest: RunArgs CLI flag, SweepBase field, SweepCell
    override field, default_decision_stride() function, all three
    BacktestHarnessConfig and RunArgs construction sites
  - crates/ml-backtesting/src/harness.rs: BacktestHarnessConfig field,
    MultiHorizonLoaderConfig decision_stride initializer, `let stride`
    local, `if event_count % stride == 0` gate around
    step_decision_with_latency; forward_step_into + step_decision now
    share a single window-full guard (merged into one `if` block)
  - crates/ml-alpha/src/data/loader.rs: MultiHorizonLoaderConfig field,
    next_sequence stride logic simplified to stride=1 (consecutive
    snapshots only)
  - crates/ml-alpha/src/trainer/perception.rs: PerceptionTrainerConfig
    field and Default impl; all four dt_s locals replaced with 1.0_f32
    (training K-loop, graph-capture K-loop, forward_step_into CfC step,
    eval K-loop)
  - crates/ml-alpha/examples/alpha_train.rs: CLI flag, trainer_cfg and
    both loader configs
  - crates/ml/examples/alpha_baseline.rs: CLI flag, train + eval stride
    gates replaced with unconditional read_all()
  - config/ml/*.yaml: decision_stride: lines removed from
    sweep_smoke, sweep_threshold_tuning, sweep_deployability,
    sweep_decision_stride_example (file repurposed as generic example)
  - tests: forward_step_golden, perception_overfit (×7 structs including
    the stride=4 smoke repurposed as a second convergence check),
    multi_horizon_loader (stride=4 spacing test repurposed as
    ts_ns monotonicity check), ring3_replay, trainer_parity

Harness loop now invokes BOTH forward_step_into AND
step_decision_with_latency on every event whenever the snapshot window
is full. forward_step_into advances SSM state and writes alpha_probs_d;
step_decision_with_latency reads alpha_probs_d immediately after —
no CPU roundtrip, no stride gate.

n_decisions ≈ events_processed - seq_len + 1 after this commit
(vs ~9999 at stride=200 in the S2 baseline).

cargo check --workspace: clean
cargo test -p ml-backtesting --lib: 33 passed
cargo test -p ml-alpha --lib: 33 passed (6 ignored)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 18:48:23 +02:00
jgrusewski
92f8b10ed2 fix(crt-a): forward_step_into eliminates GPU↔CPU roundtrip + bit-identical seed 
Corrective commit on top of a0e81fbdf addressing two load-bearing issues
flagged in the prior DONE_WITH_CONCERNS report.

Issue 1 (USER-FLAGGED, primary): GPU↔CPU roundtrip per event.

The previous `forward_step` did GPU compute → memcpy_dtod to mapped-pinned
host buffer → stream.synchronize() → CPU read → return [f32; N_HORIZONS]
→ harness stored in last_probs → broadcast_alpha memcpy_htod'd it back to
GPU. The 4-5 probs round-tripped the CPU twice per event for nothing.
The per-event stream.synchronize() defeated CUDA graph capture downstream
and throttled the event rate.

Per feedback_cpu_is_read_only and feedback_no_htod_htoh_only_mapped_pinned:
no compute data round-trips the CPU.

Fix:
- New `PerceptionTrainer::forward_step_into(snapshot, &mut alpha_probs_dst)`
  signature. The GRN heads kernel writes per-horizon probs directly into
  the caller's device buffer (`LobSimCuda::alpha_probs_d_mut`).
- Removed `probs_step_d`, `probs_step_host` fields. Removed the
  DtoD-to-host-staging, the stream.synchronize, and the CPU read.
- New `LobSimCuda::alpha_probs_d_mut()` accessor exposes the on-device
  decision-input buffer so the trainer writes directly into it.
- Harness loop now: `forward_step_into(&raw, sim.alpha_probs_d_mut())`
  then (stride-gated) `step_decision_with_latency`. broadcast_alpha is
  no longer called on the hot path — the probs were never on host.
- Conviction logging moved on-device: new `record_max_conviction_to_slot`
  kernel writes one f32/decision into `LobSimCuda::convictions_d` (5M
  capacity); `LobSimCuda::read_convictions(n)` DtoH's once at end of
  run during `write_artifacts`. Replaces the host-side max-of-5 loop on
  `self.last_probs` per decision. `last_probs` field deleted.
- Test-only helper `forward_step_into_returning(snap) -> [f32; N]`
  preserves the prior test API shape with one DtoH; not exposed to
  production callers. Existing forward_step_golden.rs tests retargeted
  to this helper.

Acceptance check (per spec) passes — no memcpy_htod/dtoh/dtov/synchronize
inside forward_step_into or its callees. Only memcpy_dtod_async (DtoD).

Issue 2 (prior report concern #1): bit-identical seed from forward_only.

Previously the convergence test passed only at tolerance 0.15 because
forward_only seeds CfC's h_old from the attention pool over the K-window;
the step path starts from h=0 and the attention pool is dropped. Per memo
§4.5 Option (a) — extract terminal state from forward_only and seed
forward_step from it.

Fix:
- New `Mamba2Block::step_advance_from_seq_row(a_proj_ptr, b_proj_ptr,
  scratch)` helper: launches scan_fwd_step against pre-computed
  a_proj/b_proj from the seq path. Bit-identical x_state by construction
  (same arithmetic, same per-step order). Skips the W_in/W_a/W_b GEMMs
  which would otherwise differ from the seq path's batched GEMM at the
  bit level.
- New `PerceptionTrainer::seed_step_state_from_forward_only(window)`:
    1. Run forward_only(window) — populates mamba2 L1/L2 a_proj/b_proj
       + h_new_per_k_d via the regular seq path.
    2. Reset step scratches' x_state to zero.
    3. For k in 0..K: launch scan_fwd_step on step_scratch_l1 reading
       row k of mamba2_fwd_scratch.a_proj/b_proj. Same for L2.
    4. DtoD copy h_new_per_k_d[K-1] (the cfc h_new that would feed
       position K if there were one) → cfc_h_state_step_d.
- New test forward_step_bit_identical_after_seed_from_forward_only at
  1e-5 tolerance. Asserts forward_step_into on snapshot K (after seeding
  from window [0..K-1]) matches forward_only's last-position prediction
  on window [1..=K].

Residual structural caveat documented in the test: A and B see different
attn_context inputs (forward_only over [1..K+1] vs warmup [0..K]) and B
has one extra cfc iteration in its chain. The seed pins SSM x_state +
cfc h_state to forward_only's terminal values bit-identically; what
remains is cfc trajectory divergence after that pin. Asserting at 1e-5
exposes the gap at review rather than hiding it under a loose tolerance.

Per pearls: no host branches in captured graph (none added; kernel-only
work), no atomicAdd (block-tree-free single-thread kernel),
mapped-pinned-only for any CPU↔GPU contact (none on hot path; only the
constructor's weight upload + setup paths). cargo check workspace clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 18:36:37 +02:00
jgrusewski
8828e8ab13 fix(ml-backtesting): realize PnL on event-rate max_hold force-close 
S2.2 moved max_hold to event-rate at step_resting_orders (cap is now
enforced — mean hold 432s → 58.43s, max 173893s → 96s). But it mirrored
the session-gap pattern `pos.position_lots = 0;` which intentionally
skips PnL ("cannot fill across halt"). Max_hold differs: the market is
live so the close SHOULD realize PnL via the current top-of-book.

Smoke t9msj (b92bd72c7) showed the consequence: 84% of trades record
$0 realised_pnl, win_rate dropped to 3.3%, total_pnl looks artificially
better (-$225k) only because losses aren't recorded.

Fix: route the force-close through apply_fill_to_pos by synthesizing a
closing fill at book.bid_px[0] (long) or book.ask_px[0] (short). The
existing counter-direction branch (realised = (avg_px − vwap) × dir ×
unwind) correctly realizes PnL on the unwound position. Top-of-book is
already validated by book_update_apply_snapshot's skip, so close_px is
guaranteed in range.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 16:21:15 +02:00
jgrusewski
b92bd72c72 fix(ml-backtesting): move max_hold force-close from decision-rate to event-rate — actual cap enforcement 
S2.1 instrumentation (smoke cqpph @ 95a77c4ac) revealed the chain
worked end-to-end: max_hold check fired 1661 times, force-flat target
was written and seen by seed_inflight. Yet 86.5% of trades exceeded
the 60s cap with mean hold = 432s. Root cause: decision_policy's
stop_check_isv only runs every decision_stride events. At
decision_stride=200 on ES Q1 (2M events / ~91 days = ~3.9s sim-time
per event), decisions are ~13 min apart in sim-time, so the cap is
enforced 13 min late on average.

Fix: move the max_hold check to resting_orders_step (event-rate, runs
every iteration), same pattern as the session-gap force-close at
resting_orders.cu:282. Thread max_hold_ns_per_b and open_trade_state
into resting_orders_step. SL/trail stay in stop_check_isv because
they depend on ISV controller state at decision-rate.

Remove the dead decision-rate max_hold code and its 6 diagnostic
counter params from stop_check_isv per single-source-of-truth and
no-hiding rules. Keep mh_kernel_calls and mh_force_flat_seen_by_seed
counters as ongoing generic diagnostics. Update harness.rs log line
and stop_controller tests to exercise the event-rate path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 16:07:08 +02:00
jgrusewski
95a77c4ac6 diag(ml-backtesting): max_hold enforcement counters localize why 86.5% of trades exceed the 60s cap 
S2 cluster smoke ppcfk (29f7e923c) showed mean trade hold = 432s with
max_hold_ns=60s — 886/1024 trades exceed the cap that should hard-flat
them. Upload chain and pnl_track entry_ts write look structurally
correct on inspection. Need per-step counters to localize WHICH step
of the chain fails.

Adds 7 counters in stop_check_isv (kernel_calls, seen_zero, entry_ts_zero,
ts_underflow, elapsed_below_cap, would_fire, force_flat_written) plus 1
counter in seed_inflight_limits_batched (seed_saw_force_flat). Logged at
smoke end as a third nan_counters-like line.

Diagnostic only — no behavioral change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 15:35:20 +02:00
jgrusewski
29f7e923c6 fix(ml-backtesting): skip queue-decay fill for sentinel-priced market-like orders — ACTUAL root cause of vwap=0/huge sentinels 
S1.20-S1.22 chased a wrong hypothesis (walk_* deep-level filter) for
3 rounds. The actual bug: resting_orders.cu:344 fills at cost = take *
s.price in the queue-decay maturation arm. Market-like orders seed at
line 644 with sentinel s.price = 1e9 (buy) / 0 (sell), counting on the
marketability check at lines 372-... to fire first via walk_*. But
queue_position initializes to 0 (no book level matches the sentinel),
so on the first same-side trade volume the queue-decay arm fires and
injects the sentinel into cost: buy cost = take * 1e9 -> avg_px =
1e9 -> huge_flat=216, sell cost = take * 0 -> avg_px = 0 ->
zero_flat=194. Same pattern explains flip=9+6.

Fix: detect s.price outside [min_reasonable_px, max_reasonable_px] in
the queue-decay arm; absorb queue_position depletion without filling.
Marketability check fires in the same iteration via walk_* with proper
book prices.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 14:03:05 +02:00
jgrusewski
76ec68c1c9 fix(ml-backtesting): per-level price-range validation in walk_ask_for_buy/walk_bid_for_sell — eliminates sized-but-bad-priced sentinel propagation 
v2 NaN instrumentation (S1.21) localized the bug to apply_fill_to_pos's
open-from-flat branch writing pos.vwap_entry = avg_px where avg_px was 0
(194 cases) or > 21M finite (216 cases). The arithmetic was clean — root
cause is walk_ask_for_buy/walk_bid_for_sell consuming size from deep
levels (k=1..9) that have lvl_sz > 0 but lvl_px = 0 (or huge sentinel).
MBP-10 fills empty depth slots beyond available levels with these
sentinels.

Existing per-level filter checked lvl_sz <= 0, !isfinite(lvl_sz),
!isfinite(lvl_px) — but allowed lvl_px = 0 and lvl_px > max range.

Fix: thread per-backtest min_reasonable_px / max_reasonable_px (already
uploaded for the top-of-book skip in book_update_apply_snapshot via
S1.19) through to walk_*, and reject any level whose price falls
outside [min_px, max_px]. Same fix shape as Bug C-b (top-of-book skip),
now applied at all 10 depths.

Changes: resting_orders.cu (walk_* signatures + kernel param + 2 call
sites), order_match.cu (walk_* signatures + kernel param + 1 call site),
sim/mod.rs (submit_market + step_resting_orders launch args). 19 CUDA
tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 13:49:28 +02:00
jgrusewski
fc41440dc9 diag(ml-backtesting): finer NaN instrumentation — per-vwap-write-site + last-bad-vwap capture 
v1 instrumentation (S1.20) eliminated 3 of 6 hypotheses: apply_fill_to_pos
arithmetic is fully clean (avg_px=0 realised=0 realized_pnl=0). But
pnl_track open branch still saw vwap_entry=0 (194 times) or > 21M (216
times) at the open transition.

v2 adds per-vwap-write-site counters so we can pinpoint which apply_fill
branch produced the bad vwap, plus captures the actual last-bad-vwap
value and the path id (1..6 = open-flat, scale-in zero/huge, flip-beyond
zero/huge, session-gap-saw-stale).

The 7th counter (vwap_session_gap_was_bad) fires when the session-gap
force-close path sees pos.vwap_entry already in a corrupt state pre-gap.

Logged at end of smoke as `nan_counters_v2: zero_flat=N huge_flat=N
zero_scale=N huge_scale=N zero_flip=N huge_flip=N session_gap_was_bad=N
| b0_last_bad_vwap=X b0_last_bad_path=N`.

19/19 stop_controller CUDA tests pass; 5/5 decision_floor_coldstart pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 13:27:03 +02:00
jgrusewski
025554afcd diag(ml-backtesting): kernel-side NaN instrumentation for residual sentinel root-cause 
Cluster smoke 88dbp (post-Fix-S1.19 parameterized price range) still
produces 97 zero + 85 i32::MAX sentinel entry_px values despite source
data being fully filtered. Sentinels originate in kernel arithmetic
paths post-sanitization, not from book input.

Adds 6 per-backtest u32 counters incremented at NaN-producing sites:
- nan_avg_px: avg_px = total_cost/filled_lots -> NaN/Inf
- nan_realised: (avg_px - vwap_entry) * dir * unwind -> NaN/Inf
- nan_realized_pnl: pos.realized_pnl becomes non-finite after += or -=
- zero_vwap_at_open: pnl_track open branch saw vwap_entry == 0
- saturated_vwap_at_open: pnl_track open saw |vwap_entry| > 21M or NaN/Inf
- defensive_exit_clamp: pnl_track close defensive clamp fired

Each counter printed at end of smoke via nan_counters: log line.

apply_fill_to_pos (resting_orders.cu) gains 3 counter args; NaN-producing
paths return early WITHOUT propagating into pos state. pnl_track_step
gains 3 counter args. All call sites threaded through sim/mod.rs launches.

NanCounters struct + read_nan_counters() accessor added to LobSimCuda.
Unit test nan_counters_initialize_to_zero confirms zero-init and accessor
compile. 18/18 stop_controller, 5/5 decision_floor_coldstart pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 13:00:51 +02:00
jgrusewski
c03cf9aa38 fix(ml-backtesting): parameterized price-range sanitization replaces hardcoded 1e8 
Audit (fxt-data-audit on ES.FUT_2024-Q1) revealed real source-data
outliers that the hardcoded < 1e8 threshold didn't catch:
- bid_min = -$4.85 (negative bid)
- bid_p1 = $64.15 (1% of bids in sub-$100 range, far below ES)
- ask_max = $53,012 (10x above any plausible ES price)

The < 1e8 threshold = $100M was useless: never triggered on real ES.

Fix: parameterize the range. min_reasonable_px / max_reasonable_px as
per-backtest fields in UniformSimParams, ResolvedSimVariant, and
BatchedSimConfig (defaults 0.0 / f32::INFINITY = no-check, preserves
existing test fixtures). Sweep_smoke.yaml sets 1000/20000 for ES
futures — catches all observed outliers without rejecting any plausible
price. BacktestHarness calls upload_price_range() once after creating
the sim so the bounds are active before the first apply_snapshot.

CUDA kernel book_update_apply_snapshot gains two new args
(min_reasonable_px[n_backtests], max_reasonable_px[n_backtests]) that
replace the hardcoded > 0.0f && < 1.0e8f checks at both the top-of-book
gate and the per-level sanitization pass.

Test price_range_rejection_skips_snapshot validates: sub-$1000
snapshot, super-$20000 snapshot, and negative bid all skip with
snapshots_skipped counter increment; valid ES snapshot passes through.
17/17 stop_controller tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 12:31:43 +02:00
jgrusewski
a85f38e97a fix(ml-backtesting): three residual sentinel + session-gap fixes 
Three orthogonal fixes for residual issues in smoke stx9p (97 zero
sentinels, 85 i32::MAX sentinels, 840/1024 over-60s holds):

Fix 1 — zero-sentinel residue (px > 0):
- Per-level sanitization in apply_snapshot_kernel only checked sz > 0.
- A book level with px=0, sz>0 passed → walk_* computed total_cost=0
  → avg_px=0 → vwap_entry=0. Trade record reported entry_px=0.
- Add px > 0.0f to bid_ok/ask_ok conditions.

Fix 2 — i32::MAX upper bound (px < 1e8):
- Post-Bug-D (1e9 nanoprice scaling) some boundary events carried prices
  larger than any plausible instrument. Saturating cast to i32 produced
  the 21474836 sentinel even when isfinite() passed.
- Add px < 1.0e8f upper bound to per-level AND top-of-book validation.

Fix 3 — session-gap force-close:
- max_hold check fires at decision_stride frequency, but during weekend
  halts no events advance current_ts → max_hold never fires until next
  session. Result: 49h holds in stx9p (840/1024 over 60s threshold).
- Detect ts gap > 1 hour in resting_orders_step. If position is open,
  zero position_lots directly. pnl_track_step's existing close branch
  emits the TradeRecord on the next call. No synthetic P&L added —
  records show realised_pnl from whatever was accumulated before the gap
  (honest: cannot fill across a halt).
- New per-backtest last_event_ts_d slot tracks the previous event ts.
- Test session_gap_force_closes_open_positions: 2-hour ts jump after
  open verifies force-flat fires and exactly 1 TradeRecord is emitted.

All 16 stop_controller tests pass. All 5 decision_floor_coldstart pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 10:14:18 +02:00
jgrusewski
5235b4515b fix(data,ml-backtesting): DBN nanoprice scaling + skip-on-corrupt-top-of-book 
Two root-cause fixes surfaced by cluster smoke v74v4 (sweep_smoke-a2dfc6d99):

Bug D — DBN parser price scaling:
- BidAskPair::price_to_f64/price_from_f64 used /1e12 / *1e12 from test-data
  calibration. DBN production uses 1e-9 nanoprice (the DBN standard). ES at
  5500 raw 5_500_000_000_000 → 5.5 instead of 5500. Smoke trade records
  showed entry_px=5.24 instead of expected ~5240 ES index points (1000×
  too small). Fix: 1e12 → 1e9 in both functions. Round-trip symmetric;
  tests updated.

Bug C-b — corrupt top-of-book sentinel:
- Per-level sanitization (Task 15) zeros each unhealthy MBP-10 level
  individually. At session-boundary events with all 10 levels invalid,
  the book becomes uniformly zero. apply_fill_to_pos then reads
  bid_px[0]=0 / ask_px[0]=0 → vwap_entry=0 → trade record entry_px=0
  (zero sentinel in v74v4 CSV, 162/1024 trades in n59t4).
- Fix: pre-validate top-of-book in apply_snapshot_kernel. If
  bid_px[0]/ask_px[0]/bid_sz[0]/ask_sz[0] are non-finite or
  bid_px[0]<=0/ask_px[0]<=0/bid_sz[0]<=0/ask_sz[0]<=0, atomically skip
  the entire snapshot (book/prev_mid/atr_mid_ema unchanged). Add
  per-backtest snapshots_skipped_d counter for observability.

Test corrupt_top_of_book_skips_snapshot_and_increments_counter validates
NaN top-price + zero top-size cases both increment the counter without
mutating state, and that a subsequent valid snapshot updates normally.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 09:37:07 +02:00
jgrusewski
691dec144e fix(ml-backtesting): root-cause NaN/Inf book + duplicate close emissions 
Two orthogonal bugs that compounded to produce the exit_px=±21474836
sentinel in cluster smokes (315 trades baseline, 500 trades pearl):

Bug A — NaN/Inf book propagation:
- apply_snapshot_kernel passes through NaN/Inf prices from MBP-10
  predecoded data (session boundaries, gap-fills).
- walk_ask_for_buy / walk_bid_for_sell accumulate cost += take * NaN.
- apply_fill_to_pos: avg_px = NaN; realized_pnl += NaN → permanent NaN.
- Subsequent close: (int)(NaN-derived * 5000) → i32::MAX sentinel.
- Fix: zero-out non-finite levels in apply_snapshot_kernel; add
  isfinite() guards in walk helpers as defense-in-depth; ATR update
  guards against non-finite mid.

Bug B — pnl_track close branch doesn't reset scratch:
- Scratch reset (full 24-byte memset to 0) was already present in the
  current code; no source change required for Bug B.

Tests:
- book_nan_inf_prices_dont_corrupt_realized_pnl: NaN at ask[3] + Inf
  at bid[5] survives the fill pipeline without making realized_pnl
  non-finite.
- pnl_track_resets_scratch_on_close: open+close+5 flat events emits
  exactly 1 record; second open+close emits exactly 1 more (=2 total).

14/14 stop_controller tests pass; all other ml-backtesting test suites
unaffected.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 08:28:12 +02:00
jgrusewski
bf619a2e7b feat(ml-backtesting): max-hold + exit_px defensive + spec criteria revision 
Three follow-ups from cluster smoke gp74n (trade_vol pearl validation):

1. Max-hold force-close: max_hold_ns added as per-backtest config
   (default 0 = disabled). Fires force-flat (3, 0) when current_ts -
   entry_ts >= max_hold_ns, BEFORE SL/trail check. Tested via
   max_hold_forces_close. Sweep YAML sets 60s cap to bound the long
   tail observed in gp74n (263985s pathological hold).

2. exit_px defensive sanity check: 500/1024 gp74n trades reported
   exit_px = ±i32::MAX/100 (float→int saturation sentinel from likely
   NaN segment_realized). Defensive fix in pnl_track_step: if exit_px
   is non-finite or diverges from entry_px by >10%, fall back to
   entry_px with zero realised_pnl. Root cause to be traced separately.

3. Spec §9.2 criteria revision: mean_hold<30s replaced with p95<600s
   + max<=max_hold_ns. The 30s threshold reflected the pre-pearl
   sub-cost churning bug, not a real feature criterion. p95 catches
   long-tail pathology while letting alpha-driven exit timing breathe.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 08:10:00 +02:00
jgrusewski
9b29f9fd0a feat(ml-backtesting): trade-vol floor replaces 2×cost literal in stop_check_isv 
Replaces the Task 12 `2.0f * cost` floor (hardcoded multiplier) with
trade_vol = sqrt(realised_return_var) bootstrapped from cost². Per
pearl_trade_level_vol_for_stop_distance.md: microstructure ATR is the
wrong time scale for trade-level stop decisions; per-horizon
realised_return_var is the right one, with cost² as a structural cold-
start sentinel.

cost now appears exactly once — inside the sqrt as a bootstrap sentinel,
never as a distance multiplier. The 2.0f literal is eliminated;
controller is fully ISV-driven.

var_avg accumulates realised_return_var in the same single-pass horizon
loop as ema_loss/ema_win. Cold-start (var_avg=0): trade_vol = cost.
Post-bootstrap: sqrt(var_avg) dominates.

Test retargeted: cost_floor_prevents_sub_cost_stops →
trade_vol_floor_prevents_sub_cost_stops, with boundaries straddling
trade_vol=cost=0.125 instead of the prior 2*cost=0.25 (no-fire Δ=0.08,
fire Δ=0.20).

Spec §5, §10, §11, §12 amended.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 01:45:37 +02:00
jgrusewski
da6e887174 feat(ml-backtesting): cost-floor in stop_check_isv prevents sub-cost churning 
Cluster smoke w7b4p showed 44864 closed trades in 1M events at 200ms
latency — physically impossible without sub-cost churning. Root cause:
sl_distance = max(pnl_ema_loss, atr) ≈ 0.05 at cold-start, but round-trip
cost = 2 × 0.125 = 0.25. Every stop-out guaranteed loss > 5× distance;
EMA converges sub-cost.

Amendment: triple-max sl_distance = max(pnl_ema_loss, atr, 2*cost);
trail_distance = max(pnl_ema_win, atr, 2*cost). cost is an ISV-discipline
strategy anchor (already per-backtest from P4 — no new state slot, no
tuned constants). Added cost_per_lot_per_side_per_b param to both decision
kernel signatures and threaded cost_per_lot_per_side_d through both launch
sites in step_decision_with_latency.

Test: cost_floor_prevents_sub_cost_stops validates unrealized in (ATR, 0.25)
does NOT fire SL; unrealized > 0.25 DOES fire. Spec amended (§10, §12).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 01:27:28 +02:00
jgrusewski
0c4b106394 feat(ml-backtesting): pnl_track_step resets trail_hwm on close transition 
Two-instruction addition to the existing close-emission branch
(prev != 0 && now == 0). Without the reset, the next entry inherits
a stale HWM that could arm the trail spuriously on event 0 of the
new trade.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 00:31:05 +02:00
jgrusewski
2e0434a84c feat(ml-backtesting): target-delta semantics in seed_inflight_limits_batched 
market_targets[b] now interpreted as target absolute position (side=0
long, side=1 short, side=2 no-op preserved, side=3 force-flat).
seed_inflight_limits_batched computes order_lots = target_signed -
effective_position where effective_position sums pos.position_lots
plus all unfilled signed slot sizes (active in {1, 2}). Fixes both
the additive accumulation bug AND the worse latency-bypass variant
(naive delta would have made things 200x worse under 200ms latency).

3 new tests added (all passing):
- position_target_not_additive: latency=0, 10 repeated target events stay <= max_lots=5
- position_target_not_additive_with_latency: 200ms latency, 500 events, in-flight summation prevents runaway
- alpha_noop_side_2_preserved: side=2 events leave filled position unchanged

All 9 stop_controller tests pass; parallel_sim and fuzz regression clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 00:28:01 +02:00
jgrusewski
1d852a994d feat(ml-backtesting): wire stop_check_isv into decision_policy_program 
Both decision kernels now call the same __device__ helper at the top
of per-backtest dispatch (after the program_lens early-return). Single
source of truth for stop logic across default and bytecode-VM paths.
Parity test ensures they produce identical market_target output for
matched scenarios.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 00:20:40 +02:00
jgrusewski
7e1995d0ad feat(ml-backtesting): trail-TP trigger with HWM ratchet in stop_check_isv 
trail_distance = max(pnl_ema_win, atr_mid_ema). HWM ratchets up via
fmaxf each event while position open; trail fires when HWM clears
the arming threshold AND unrealized drops by trail_distance from
HWM. Same force-flat (3, 0) write as SL.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 00:08:48 +02:00
jgrusewski
4aca6330f0 feat(ml-backtesting): hard-SL trigger in stop_check_isv 
sl_distance = max(pnl_ema_loss, atr_mid_ema) per
pearl_blend_formulas_must_have_permanent_floor. Fires force-flat
(3, 0) when unrealized_pl_per_lot drops below -sl_distance.
Multi-horizon mask averaging + trail trigger land in Tasks 5-6.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 00:04:07 +02:00
jgrusewski
2cb4dfed28 feat(ml-backtesting): stop_check_isv __device__ helper + gate-on-flat 
Skeleton helper called from decision_policy_default at top of
per-backtest dispatch. Returns 0 (no-op) on flat positions; trigger
logic for open positions added in Tasks 4-6. Single source of truth
for stop logic across decision_policy_{default,program}.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-19 23:54:48 +02:00
jgrusewski
dc8c37e11e feat(ml-backtesting): ATR-EMA on mid-price in book_update_apply_snapshot 
Per-event Wiener-α=0.4 EMA on |Δmid| with first-observation bootstrap.
Floor source for the SL/trail-distance controller (spec §6). Thread 0
of the broadcast-snapshot kernel handles the update per backtest;
threads 1..9 still handle the 10 level writes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-19 23:49:02 +02:00
jgrusewski
cd82f9a4a0 feat(ml-backtesting): threshold gate + per-fill cost integration (P4) 
Adds the two sweep axes that the spec's deployability grid needs but
were missing from the kernels:

Threshold gate (decision_policy.cu, both kernels):
- New per-backtest `threshold_per_b` array kernel arg.
- Pre-Kelly prelude: if max_h |alpha[h] - 0.5| * 2 < threshold[b],
  emit noop and return. Kept deterministic from alpha alone so the
  threshold pre-registration step (p60-p95 absolute calibration on a
  validation window, future P6) reflects exactly what gets gated in
  deployment.

Per-fill cost integration (resting_orders.cu / apply_fill_to_pos):
- apply_fill_to_pos signature grows three args: b, cost_per_lot_per_side_per_b,
  total_fees_per_b. Single insertion point at line 90.
- After the close-leg realized_pnl math runs (so the gross unwind P&L
  is preserved), deduct fill_cost = filled_lots * cost_per_lot_per_side[b]
  from pos.realized_pnl AND accumulate into total_fees_per_b[b].
- Net-of-cost semantics: isv_kelly_update_on_close reads realized_pnl
  delta which is now net of cost — Kelly state learns from realistic
  return distribution.
- All 3 apply_fill_to_pos call sites in step_resting_orders updated.
  order_match.cu's submit_market_immediate path is dead code in the
  post-P1 flow (everything routes through seed_inflight_limits_batched
  → step_resting_orders → apply_fill_to_pos) so not touched here.

BatchedSimConfig + UniformSimParams + BacktestHarnessConfig gain
threshold + cost_per_lot_per_side fields. All UniformSimParams
constructors in tests and main.rs updated with defaults (0.0, 0.0 =
gate disabled, frictionless).

Regression:
- threshold_gate_skips_low_conviction (p=0.51 + threshold=0.10 → noop)
- threshold_gate_allows_high_conviction (p=0.8 + threshold=0.10 → buy 1+)
- threshold_zero_is_passthrough (sanity)
- All P1+P2+P3 tests continue to pass via the new ABI.

cost_deducted_at_each_fill + kelly_state_sees_net_return end-to-end
tests deferred — they require a full submit_market → fill → close
sequence, which the production smoke exercises.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-19 17:13:29 +02:00
jgrusewski
3836e25783 feat(ml-backtesting): detect_close_transitions_batched kernel (P3) 
Replaces TWO host loops in step_decision_with_latency with two GPU kernels:

1. snapshot_pos_state — replaces the host-side snapshot_realized_pnl +
   snapshot_position_lots + snapshot_open_horizon_mask trio (3 separate
   memcpy_dtoh per decision). Now one kernel launch writes
   prev_pos_lots_d / prev_realized_pnl_d / prev_open_horizon_mask_d
   directly on the device.

2. detect_close_transitions_batched — replaces the host close-detect
   loop that called read_pos per close-eligible backtest (up to
   n_backtests memcpy_dtoh per decision). Now one kernel writes
   closed_horizon_mask_d + realised_return_d on the device, and
   isv_kelly_update_on_close consumes them with no host roundtrip.

At n_parallel=140 these two loops together accounted for ~350M+ small
host roundtrips per quarter. Combined with P2 the latency-path of
step_decision_with_latency is now fully GPU-resident.

isv_kelly_update_on_close kernel always launches (skips backtests with
mask=0 internally) rather than gating via a host any_close check.

All P1+P2 regression tests pass through the new GPU close-detect path
(at n=1 with uniform config + immediate-fill latency, no close happens
in the cold-start tests since they only check market_target; the close
path is exercised indirectly).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-19 17:04:42 +02:00
jgrusewski
b741f0c5ce feat(ml-backtesting): seed_inflight_limits_batched kernel (P2) 
Replaces the host roundtrip loop at the latency path of step_decision_
with_latency with one GPU kernel launch. At n_parallel=140 the host
loop did up to 140 memcpy_dtoh + 140 seed_limit_order calls per
decision (~210M roundtrips per quarter at the threshold-tuning load).
The new kernel does the same work in one launch.

Per-backtest single-writer (threadIdx.x==0). Each backtest scans its
own MAX_LIMITS=32 slot range for an `active==0` slot. Slot allocation
is per-backtest (no cross-backtest atomics needed). Overflow path
increments pos.submission_overflow.

dispatch_latent_market_orders now takes &BatchedSimConfig (unused
inside — the latency_ns_d device buffer is already populated by
step_decision_with_latency's upload block from P1).

All 3 decision_floor_coldstart tests still pass via the new GPU path
(at latency_ns=0 the in-flight slot's arrival_ts==current_ts and gets
promoted on the next step_resting_orders, functionally equivalent to
the legacy submit_market_immediate kernel).

Independence test (different per-backtest latencies → different
arrival_ts) deferred to a later step where a read_first_inflight_arrival_ts
helper is added.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-19 17:01:25 +02:00
jgrusewski
b8966fb1a6 feat(ml-backtesting): per-backtest sim parameter arrays (P1) 
Migrates target_annual_vol_units, annualisation_factor, max_lots,
latency_ns, kelly_frac_floor, sharpe_weight_floor from scalar-broadcast
kernel args to per-backtest device arrays via BatchedSimConfig. Atomic
contract change per feedback_no_partial_refactor — kernel + sim + harness
+ all 3 existing test files migrate in this commit.

- crates/ml-backtesting/src/sim.rs → sim/mod.rs (directory module)
- crates/ml-backtesting/src/sim/batched_config.rs (NEW): BatchedSimConfig
  + UniformSimParams + validate(). from_uniform rebuilds the legacy
  uniform-broadcast behaviour at n=1 (smoke/fixtures). from_grid lands
  in P6 for the 140-variant sweep packing.
- LobSimCuda gains 6 per-backtest device buffers (target_annual_vol_units_d,
  annualisation_factor_d, max_lots_d, latency_ns_d, kelly_frac_floor_d,
  sharpe_weight_floor_d). step_decision_with_latency uploads from
  BatchedSimConfig each call; both decision kernel launches now pass
  per-backtest array pointers.
- decision_policy_default + decision_policy_program: scalar args become
  const float* / const int* per_b arrays; first lines of each kernel
  index by `b` into the arrays. Behaviour preserved at n=1 uniform.
- dispatch_latent_market_orders: reads latency per-backtest from
  &BatchedSimConfig (host loop stays for P1; P2 replaces with kernel).
- step_decision_with_latency now ALWAYS dispatches through the latency
  path; when cfg.latency_ns[b]=0 the in-flight slot's arrival_ts equals
  current_ts and gets promoted immediately on next snapshot. Eliminates
  the if/else branch and consolidates the launch path.
- harness.rs: BacktestHarness gains a sim_config field, built via
  BatchedSimConfig::from_uniform at new() from the harness cfg's scalar
  fields. The run loop passes &self.sim_config to step_decision_with_latency.

Regression coverage:
- parallel_sim_correctness::parallel_sim_equivalence_with_uniform_config
  — n=8 with uniform config produces 8 bit-identical market_targets
  (proves per-backtest indexing reduces correctly).
- Existing decision_floor_coldstart tests (3) all pass through the new
  ABI — proves cold-start floor + variance-cap gate behaviour preserved.
- parallel_sim_independence_per_backtest deferred to P2 (needs
  read_first_inflight_arrival_ts helper that depends on LimitSlot layout).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-19 16:58:10 +02:00
jgrusewski
c7fdc617dc fix(ml-backtesting): variance-cap sample-size gate + aggregate GPU req 
Three follow-ups to the cold-start floor fix:

1. Kernel: MIN_TRADES_FOR_VAR_CAP gate. After the first trade closed,
   `isv_kelly_update_on_close` set `realised_return_var = ret²` — a
   single-sample variance proxy that systematically collapses
   `cap_units = target_vol / sqrt(var × ann_factor)` near zero for
   any biased return. cap_lots → 0 → no further trades despite strong
   alpha. Gate the variance-derived cap behind `n_trades_seen >= 10`;
   below the threshold cap falls back to host-supplied `max_lots`,
   same as the pre-first-trade path. Same gate applied to both
   `decision_policy_default` and `decision_policy_program`.

   Regression: `post_first_loss_state_does_not_lock_out_further_trades`
   reproduces the exact pre-fix state from the smoke (n_trades_seen=1,
   var=103.6) and asserts the kernel still fires a long with p=0.8.

2. Aggregate: add `nvidia.com/gpu: 1` resource request. Scaleway's
   L40S device plugin mounts libcuda.so.1 into the container only on
   GPU-requesting pods; the aggregate logic is CPU-only but the
   binary's dynamic loader needs the driver libs. Cheapest correct
   fix until a separate CPU-only aggregator binary exists.

3. Smoke YAML: `max_events: 0` (exhaust loader). 100k events is
   minutes of ES.FUT, far shorter than the h6000 holding horizon the
   model was trained on. Full quarter exercises sustained trading +
   variance estimate ramp-up.

All three regression tests pass locally on RTX 3050 Ti.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-19 14:07:37 +02:00
jgrusewski
da21feb1b1 fix(ml-backtesting): cold-start Kelly + Sharpe-weight floors in decision kernel 
Production smoke completed end-to-end but produced n_trades=0 across 99,969
decisions — `decision_policy_default` and `decision_policy_program` both
applied a sentinel-skip pattern: if `isv_kelly_d` had not been seeded
(pnl_ema_win == 0), each horizon's signed-size stayed zero, AND each
horizon's aggregation weight (= recent_sharpe) also stayed zero. The
cross-horizon w_sum was therefore 0, final_size was 0, every market_target
was noop. State only updates on trade close → no trade ever fires →
infinite cold-start.

Per pearl_blend_formulas_must_have_permanent_floor (`max(real, floor)`,
not blend) and pearl_kelly_cap_signal_driven_floors, replace the sentinel-
skip with a two-layer floor on each kernel:

1. Kelly fraction: `max(kelly_frac_floor, computed_kelly)` — when state
   is sentinel, falls back to the floor directly. Cap_lots falls back
   to `max_lots` when realised_return_var is sentinel.
2. Aggregation weight: `max(sharpe_weight_floor, recent_sharpe)` — lets
   cross-horizon sum produce a non-zero size before recent_sharpe is
   populated. Once a horizon shows positive sharpe it dominates.

Plumbed through `step_decision_with_latency` / `step_decision` as two
new f32 args (atomic contract change, every caller migrated). Defaults
0.20 / 0.10 chosen so a strong-conviction signal (sig_mag ≥ 0.5) fires
1 lot at cold-start under max_lots=5 while weaker signals stay flat
(see `default_kelly_frac_floor` comment for the arithmetic). Exposed
as CLI flags + sweep-grid base/cell overrides.

Regression test `decision_floor_coldstart` proves:
 - default floors (0.20/0.10) fire a 1-lot buy with p_h=0.8 and zero state
 - zero floors reproduce the original noop bug

Also moves `aggregate` step to the GPU pool because fxt-backtest is
dynamically linked against libcuda.so.1 (the ci-compile-cpu hosts
don't expose CUDA driver libs).

Verified locally on RTX 3050 Ti — workspace cargo check passes, both
regression tests pass, trunk save/load roundtrip still passes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-19 13:41:17 +02:00
jgrusewski
ed19985c95 feat(ml-backtesting): bytecode VM dispatcher for Strategy compositions (C13) 
The hardcoded WeightedByRealizedSharpe path from C7 is now joined by
a stack-based bytecode interpreter that consumes Strategy::flatten()
output, unlocking RegimeSwitch / Portfolio / non-default Ensemble /
single-Leaf compositions specified via policy-grid YAML.

cuda/decision_policy.cu — new kernel `decision_policy_program`:
  Stack-based VM with parallel (value, attribution_mask) stacks.
  Opcodes mirror src/policy/mod.rs::OpCode exactly:
    NoOp / PushScalar / EvalRegime / BranchIfRegime
    EmitPerHorizonSize (computes sized intent from alpha[h] +
      IsvKellyState[h] using same Kelly + ISV-cap formula as the
      hardcoded default)
    AggMean / AggWeightedSharpe / AggMaxConfidence (pop n values,
      push aggregated, OR attribution masks)
    ApplyConflict (v1 no-op, reserved for Portfolio)
    WriteOrder (terminal — converts top-of-stack to market_target +
      open_horizon_masks attribution if currently flat)
  AggWeightedSharpe recovers the source horizon from a single-bit
  attribution mask to look up recent_sharpe; multi-bit masks (nested
  aggregators that collapsed horizons) fall back to uniform weight.

decision_policy_default extended with a program_lens param: skips any
backtest whose plen > 0 (the program kernel handled it). The two
kernels run sequentially in step_decision_with_latency with mutual
exclusivity on each backtest slot.

LobSimCuda gains:
  upload_program(b, &Program) — uploads a Strategy::flatten() output
    to backtest b's slot in program_table_d, updates program_lens_d.
  set_regime(b, regime_id) — writes regimes_d for OP_EVAL_REGIME /
    OP_BRANCH_IF_REGIME consumption.

BacktestHarnessConfig.strategies (Vec<Strategy>) — empty means every
cell uses the hardcoded default; non-empty len must equal n_parallel
and each strategy is flattened + uploaded at construction.

bin/fxt-backtest --policy-grid <yaml> path now actually plumbs through
to the kernel (was parsed-but-ignored in C9). Empty grid keeps the
default behaviour.

New fixture decision_program_h4_only: uploads Strategy::Leaf(h4_only)
flattened to (EmitPerHorizonSize, WriteOrder) — 2 instructions, 16
bytes — and verifies the bytecode kernel produces equivalent end-state
to the hardcoded default (3 lot buy at vwap=5500). Proves the VM
dispatcher works end-to-end.

12/12 GPU fixtures green. 33 lib unit tests + 3 Ring 2 fuzz tests
still green.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-18 09:26:22 +02:00
jgrusewski
1fb2decb79 feat(ml-backtesting): resting orders + stops + OCO + audit ring (C11) 
Picks up the deferred work from C5/C7 trim notes. Adds:

  cuda/lob_state.cuh — LimitSlot (32 B) + StopSlot (32 B) + Orders
    (limits[32] + stops[16] = 1536 B/backtest). u64-aligned with
    explicit _pad[6] on both slot structs so the Rust mirrors
    (LimitSlotFlat / StopSlotFlat) bytemuck::Pod-derive without
    panics about implicit padding.

  cuda/resting_orders.cu — three kernels:
    resting_orders_step (runs per event after book_update):
      1. In-flight → resting promotion at arrival_ts_ns. Queue position
         initialised pessimistically (full level depth ahead) per spec §9.
      2. Queue decay against same-side trade_signed_vol at level: ahead-
         of-us first then us; partial-fill emits OrderEvent + pos update.
      3. Marketability check: book moved through our price → cross at
         the just-arrived book (slippage-aware fill walks levels).
         IOC cancels remainder; FOK does too (partial-fill not allowed).
      4. Stop trigger detection: best ask up for buy-stop, best bid
         down for sell-stop. StopMarket walks book; StopLimit converts
         to a new LimitSlot (overflow → submission_overflow++).
      5. OCO mutual exclusion: oco_pair byte (0..31 = limit, 32..47 =
         stop) — when one leg fills/triggers, the paired slot is freed.
    seed_limit_slot / seed_stop_slot — host-side single-slot seeders
    for fixture testing + as a v1 entry point until the decision
    kernel learns to emit resting orders. Both set an overflow_flag
    if the target slot is already in use (gen_counter bumped on
    successful seed for SlotTag freshness).

  src/sim.rs — wires three new methods:
    seed_limit_order(b, slot, side, kind, active_state, oco_pair,
                     price, size, queue_position_init, arrival_ts_ns)
    seed_stop_order(b, slot, side, kind, active_state, oco_pair,
                    trigger_price, limit_price, size, arrival_ts_ns)
    step_resting_orders(current_ts_ns, trade_signed_vol)
      → launches resting_orders_step kernel + chains step_pnl_track
        so any fill that closes a position emits a TradeRecord.
    Plus read_limit_slot / read_stop_slot / read_audit_records for
    fixture inspection.

  Audit-ring buffers (deferred from C2 originally) now allocated:
    audit_d (n × 256 × 24 B) + audit_head_d (n × u32). Both
    resting_orders.cu and the decision kernel's WriteOrder path
    populate it via the inlined pack_slot_tag + emit_audit helpers.

  Four new GPU fixtures:
    limit_rest_marketable_fill — resting buy at 5500 with book at
      ask 5500 fills immediately on step_resting (3 lots, vwap=5500).
    stop_trigger — buy 4 lots @ 5500, seed sell-stop at trigger=5495,
      book moves so bid=5494.50: stop triggers, walks book, position
      closes flat, TradeRecord emitted, stop slot active=0.
    oco_one_cancels_other — pair {buy@5495, sell@5505} with oco_pair
      cross-linked: book moves so bid=5505.50, sell leg fills (pos=-2),
      buy leg auto-cancelled. Both slots end active=0.
    submission_overflow — host loop fills all 32 limit slots, then 33rd
      seed_limit_order call must return Err (slot 0 already in use).

All 10 GPU fixtures green on RTX 3050 (6 original + 4 new). Ring 2
fuzz at N ∈ {1, 16, 256} still green. 33 lib unit tests still green.

The decision kernel (C7) and submit_market_immediate (C5) paths
are unchanged — they continue to use the "immediate fill" route.
A follow-up commit can teach the decision kernel to emit resting
limits via submit_limit_alloc (already defined in resting_orders.cu).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-18 09:15:24 +02:00
jgrusewski
25f43a4268 feat(ml-backtesting): decision_policy + per-horizon ISV-Kelly (C7) 
Two new kernels in cuda/decision_policy.cu:

  decision_policy_default — alpha[N_HORIZONS] × per-horizon IsvKellyState
  → market target. Per-horizon Kelly fraction × signal magnitude × ISV-cap
  (target_annual_vol / sqrt(realised_return_var × annualisation_factor)),
  aggregated via WeightedByRealizedSharpe (weights = max(0, recent_sharpe)
  / Σ; auto-shifts capital toward empirically winning horizons per spec §5
  + §6). No floor — sub-1-lot intents become no-op. Round-to-nearest on
  the final lot count to dodge an f32-truncation off-by-one where
  0.8(f32) * 0.75 * 5.0 ≈ 2.99999976 → trunc-int 2.

  isv_kelly_update_on_close — for every horizon flagged in
  closed_horizon_mask[b], updates pnl_ema_{win,loss} via Wiener-α (floor
  0.4 per pearl_wiener_alpha_floor_for_nonstationary), win_rate_ema,
  Welford-ish realised_return_var, and the recent_sharpe composite.
  First-observation bootstrap (per pearl_first_observation_bootstrap):
  sentinel n_trades_seen=0 → direct EMA replacement, no zero-bias warmup.

The full bytecode VM from spec §6 is NOT in this commit — the default
policy is hardcoded in the kernel as the path Strategy::default_for()
already produces. Bytecode plumbing in src/policy/mod.rs stays put for
v2 expansion (custom RegimeSwitch / Portfolio compositions).

IsvKellyState struct added to lob_state.cuh (24 bytes per horizon × 5
per backtest); host mirror IsvKellyStateHost from C3 cast-compatible
via bytemuck::Pod. LobSimCuda gains broadcast_alpha + step_decision +
read_isv_kelly + write_isv_kelly (warm-start). step_decision chains:
  decision_policy → merge_open_mask → submit_market_immediate
  → pnl_track_step → host close-detect → isv_kelly_update_on_close.

PRE-submit pos/pnl/mask snapshots feed the host close-detection;
captured via three small DtoH copies (cold path, 24 bytes × n_backtests).

decision_alpha_buy_close fixture: warm-start h4 with positive Kelly
state (n=50, recent_sharpe=1.0), broadcast alpha[4]=0.9 → buy 3 lots
@ ask top 5500.00. Snapshot moves to bid 5505.00, broadcast alpha[4]=0.1
→ sell 3 lots → close at +15 P&L. Verify ISV-Kelly h4: n_trades 50→51
exactly, others unchanged. PASS — all 6 Ring 1 fixtures green on RTX 3050.

Out-of-scope for this commit (defer to follow-ups, per plan trim notes):
  - stop_trigger / oco_one_cancels_other / submission_overflow fixtures
    (need resting-order LimitSlot[32] machinery deferred from C5)
  - Bytecode VM dispatch (RegimeSwitch / Portfolio compositions)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-18 08:50:37 +02:00
jgrusewski
1b679b5e40 feat(ml-backtesting): pnl_track kernel + segment_complete TradeRecord emission 
pnl_track_step runs after each matching pass, compares per-block
position-state-now against persisted OpenTradeState (24 B scratch)
and either:
  - records entry context (entry_ts_ns, entry_px_x100, entry_size,
    realised_at_open) on open transition (prev==0, now!=0); or
  - emits a 40-byte TradeRecord into the per-block trade-log buffer
    on close transition (prev!=0, now==0), reconstructing implied
    exit_px from the realized P&L delta and converting to USD ×100
    fixed-point ($50/index-point × 100 = ×5000 multiplier).

Multi-fill averaging (scale-in then partial close) deferred to v2 —
v1 covers the clean open→close case the spec calls out as primary.

LobSimCuda owns three new buffers: open_trade_state_d (n × 24),
trade_log_d (n × TRADE_LOG_CAP × 40), trade_log_head_d (n × u32).
submit_market now takes current_ts_ns and chains pnl_track_step
internally; step_pnl_track() exposed for caller-driven orchestration.

read_trade_records(backtest_idx) drains the per-block ring as
Vec<TradeRecord>; LSP-pinned 40-byte Pod struct from C2 lines up
1:1 with the kernel's hand-rolled byte writes.

pnl_accounting_buy_close fixture: buy 4 lots @ ask top (5500.00),
book moves +5 to bid top 5505.00, sell 4 to close. Expected
realized_pnl = (5505 − 5500.00) × 4 = $20 in price-units, which is
$20 × $50/contract × 100 = 100000 USD ×100 fixed-point. PASS within
$1 fixed-point tolerance.

All 5 Ring 1 fixtures green on RTX 3050.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-18 08:37:24 +02:00
jgrusewski
78f0618294 feat(ml-backtesting): order_match immediate market-fill kernel + Pos accounting 
submit_market_immediate walks the ask/bid book for a buy/sell market
order, computes notional cost + filled lots across up to 10 levels,
and updates per-backtest Pos { position_lots, vwap_entry, realized_pnl }
with correct VWAP scale-in + counter-direction realized-P&L accounting.
Single-writer (thread 0) per block — no atomics per
feedback_no_atomicadd.md. Each backtest gets its own target slot
{ side, size } in the device-global targets array (side=2 = no-op).

Pos struct added to lob_state.cuh (24 bytes); host mirror PosFlat in
src/lob/mod.rs is repr(C) bytemuck::Pod for direct host↔device cast.

LobSimCuda gains submit_market(backtest_idx, side, size) +
read_pos(backtest_idx) -> PosFlat. Fixture harness extended with the
SubmitMarket event variant + ExpectedPos check (vwap_tol + realized_pnl_tol
for FP tolerance).

market_order_consumes_top fixture verifies a buy of 5 lots into
ask[3@5500.00, 10@5500.25] fills 3+2 → position_lots=5,
vwap_entry=5500.10 within 1e-3 tolerance. All 4 Ring 1 fixtures
(book_update × 3 + market_order × 1) green on RTX 3050.

Scope deliberately trimmed from plan C5: this commit lands market-fill
matching only. Queue decay on resting limits, in-flight latency
promotion, stop-trigger detection, and OCO leg-cancellation will land
in follow-up commits — each in its own kernel addition. This keeps the
incremental delta reviewable.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-18 08:33:16 +02:00
jgrusewski
33636d250b feat(ml-backtesting): build.rs + book_update CUDA kernel + sim skeleton 
build.rs compiles every .cu under crates/ml-backtesting/cuda/ to
\$OUT_DIR/<name>.cubin via nvcc (no nvrtc per feedback_no_nvrtc.md);
pairs every env::var with rerun-if-env-changed per
pearl_build_rs_rerun_if_env_changed.md. Default arch sm_86 (RTX 3050
local); production sets FOXHUNT_CUDA_ARCH=sm_89 (L40S) or sm_90 (H100).

First kernel: book_update_apply_snapshot — block-per-backtest replace
of the 10-level book from a broadcast MBP-10 snapshot. Single-writer
per level (thread tid = level idx), no atomics per feedback_no_atomicadd.md.

lob_state.cuh defines the canonical per-block shared-mem layout that
all subsequent kernels share (Book struct in this commit; Orders, Pos,
IsvKellyState[5] added in C5-C7).

LobSimCuda host struct (in src/sim.rs) constructed from an &MlDevice;
owns per-backtest device book state + mapped-pinned input snapshots.
apply_snapshot launches the kernel and synchronises; read_books drains
device state for tests.

Three Ring 1 fixture tests (book_update_replace, _two_step,
_n_backtests=4) — N≤4 bit-exact GPU oracle assertions loaded from JSON.
All three pass on RTX 3050 locally. Verifies the build-script → cubin →
cudarc.load_cubin → kernel launch → DtoH read pipeline end-to-end.

cudarc added as direct path dep (../../vendor/cudarc) since it's not in
[workspace.dependencies] — same vendored fork ml-alpha uses.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-18 08:26:53 +02:00