Commit Graph

3026 Commits

Author SHA1 Message Date
jgrusewski
dcffb11f5d test(ml-backtesting): multi-horizon mask averaging invariant test
Boundary test that fails on bit-pick-first or bit-pick-max regressions
of the open_horizon_masks averaging in stop_check_isv. Per-horizon
pnl_ema_loss values (2, 4, 3, 3, 3) yield mean=3.0 across the 0x1F
mask; snapshots placed relative to vwap_entry for ask-spread safety.
Tests Δ_entry=2.5 (no-fire) and Δ_entry=4.5 (fire).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 00:15:26 +02:00
jgrusewski
7e1995d0ad feat(ml-backtesting): trail-TP trigger with HWM ratchet in stop_check_isv
trail_distance = max(pnl_ema_win, atr_mid_ema). HWM ratchets up via
fmaxf each event while position open; trail fires when HWM clears
the arming threshold AND unrealized drops by trail_distance from
HWM. Same force-flat (3, 0) write as SL.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 00:08:48 +02:00
jgrusewski
4aca6330f0 feat(ml-backtesting): hard-SL trigger in stop_check_isv
sl_distance = max(pnl_ema_loss, atr_mid_ema) per
pearl_blend_formulas_must_have_permanent_floor. Fires force-flat
(3, 0) when unrealized_pl_per_lot drops below -sl_distance.
Multi-horizon mask averaging + trail trigger land in Tasks 5-6.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 00:04:07 +02:00
jgrusewski
2cb4dfed28 feat(ml-backtesting): stop_check_isv __device__ helper + gate-on-flat
Skeleton helper called from decision_policy_default at top of
per-backtest dispatch. Returns 0 (no-op) on flat positions; trigger
logic for open positions added in Tasks 4-6. Single source of truth
for stop logic across decision_policy_{default,program}.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 23:54:48 +02:00
jgrusewski
24a8b4c621 chore(ml-backtesting): allow(unsafe_code) on sim/mod.rs for cudarc launches
cudarc kernel launches require `unsafe` blocks — the driver API has no
way to type-check kernel args against the cubin signature. The
workspace-wide `-W unsafe-code` lint produces noise on every launch
site (10+ blocks); suppressing at the module level keeps the lint
useful elsewhere in the crate.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 23:50:31 +02:00
jgrusewski
dc8c37e11e feat(ml-backtesting): ATR-EMA on mid-price in book_update_apply_snapshot
Per-event Wiener-α=0.4 EMA on |Δmid| with first-observation bootstrap.
Floor source for the SL/trail-distance controller (spec §6). Thread 0
of the broadcast-snapshot kernel handles the update per backtest;
threads 1..9 still handle the 10 level writes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 23:49:02 +02:00
jgrusewski
5b5292aecd feat(ml-backtesting): add stop-controller state slots + accessors
Three CudaSlice<f32> per-backtest slots (prev_mid_d, atr_mid_ema_d,
trail_hwm_d) plus test-only accessors. Foundation for the
ISV-driven stop controller; no behavioral change until subsequent
tasks wire them into kernels.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 23:45:39 +02:00
jgrusewski
ef831ba598 obs(ml-backtesting): trades=N on progress line for intra-run firing visibility
Adds LobSimCuda::read_total_trade_count (cheap n_backtests*4 byte DtoH of
the trade-log head counters) and emits the sum on each PROGRESS_EVERY
eprintln. Lets future smokes (Q2 CBSW validation, P7 sweep) see trade
firing mid-run instead of waiting for write_artifacts at harness end.

Heads count kernel writes (clamped to ring cap), so the sum is a
monotone trade counter across the sweep — drops back only on harness
reset which doesn't happen mid-run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 22:29:52 +02:00
jgrusewski
fef5939556 feat(ml-backtesting): cold-start stopgap — max-confidence bytecode policy (Q1/Tier1)
The threshold-tuning smoke at 81decf40f produced n_trades=0 despite
74.6% of decisions having max_conv ≥ 0.30 — the linear-weighted-mean
aggregator in decision_policy_default is structurally dilution-bound
at cold-start (per spec §1).

Q1 stopgap: when sim_variants[i].use_cold_start_stopgap = true, the
harness uploads a max-confidence Strategy bytecode program for that
backtest, routing decisions through decision_policy_program with
OP_AGG_MAX_CONFIDENCE. Existing kernel; zero CUDA changes.

Field additions (atomically across BatchedSimConfig + UniformSimParams
+ ResolvedSimVariant + SweepBase.SimVariant) — every UniformSimParams
literal migrated to include use_cold_start_stopgap: false (default).
The sweep YAML's sim_variants entry sets it to true only for the
validation run; production deployability uses Q2's kernel fix instead.

Sweep YAML (config/ml/sweep_smoke.yaml) flipped to use_cold_start_stopgap=true
at threshold=0.0, cost=0.125 — same anchor as the threshold-tuning
smoke that produced n_trades=0, for direct comparison.

This is a VALIDATION step. Cluster smoke at this commit MUST produce
n_trades > 100 + finite metrics. Q2's kernel CBSW immediately follows
and deletes this entire stopgap atomically (field, harness branch,
YAML setting, every literal).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 22:00:52 +02:00
jgrusewski
81decf40f8 feat(ml-backtesting): conviction_log side-channel + threshold-tuning smoke
P4 plumbed the threshold-gate kernel side but deferred the side-channel
that captures observed max_conviction per decision. Wire it now so the
threshold pre-registration step (spec §3.4) can compute the calibrated
p60-p95 absolute threshold values from a real model-on-data run.

Harness changes:
- BacktestHarness gains conviction_log: Vec<f32>. Per decision, computes
  max_h |alpha[h] - 0.5| * 2 from the SAME probs that go into broadcast_
  alpha (same value the threshold gate would compare against), pushes
  to the log. One shared vec — batched cells broadcast the same probs
  to every backtest, so per-backtest is redundant.
- write_artifacts emits convictions.bin (raw little-endian f32) +
  conviction_percentiles.json with pre-computed p10/p25/p50/p60/p70/
  p80/p90/p95/p99 + mean/min/max. Also eprintln-prints the summary
  line for at-a-glance log inspection.

Smoke YAML switched to the threshold-tuning configuration: threshold=0
(no gate, full distribution captured), cost=0.125 (1-tick realistic
anchor so the observed Sharpe is the no-gate net-of-cost floor for
the sweep's deployability story).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 19:21:29 +02:00
jgrusewski
7b96268efc fix(ml-backtesting): aggregator supports P6 batched output layout
Legacy fan-out writes <sweep-dir>/<cell>/summary.json (one cell per
Argo task). P6 batched flow writes <sweep-dir>/<cell>/sim_<variant>/
summary.json (one Argo task → run_batched_cell → harness with
variant_names → sim_<name>/ subdirs per spec §3.3).

The aggregator was looking only at <sweep-dir>/<cell>/summary.json,
so the realistic batched smoke completed the actual backtest fine
(2M events, 500k decisions, real artifacts written) but the
end-of-sweep aggregate step errored with "no cell directories with
summary.json".

Walk both layouts: directories containing summary.json directly are
flat cells (legacy); directories one level deeper that contain
summary.json are batched-cell variants. Cell labels become
"<cell>/<variant>" so the aggregate.parquet rows distinguish.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 19:07:35 +02:00
jgrusewski
b1f8cd4389 chore(ml-backtesting): drop dead snapshot helpers from P3 migration
snapshot_realized_pnl / snapshot_position_lots / snapshot_open_horizon_mask
were the host-side read paths for the close-detection loop that P3
(3836e2578) replaced with snapshot_pos_state + detect_close_transitions_
batched kernels. The helpers stayed dead-code-warned after P3; per
feedback_no_legacy_aliases + feedback_no_hiding, delete them.

submit_market_fn is NOT dead — pub fn submit_market wraps it and has 4
test callers (lob_sim_fixtures, lob_sim_fuzz, ring3_replay × 2). Kept.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 18:11:23 +02:00
jgrusewski
9d4fda36ab feat(ml-backtesting): batched-cell sweep schema + 140-variant runner (P6)
Sweep YAML now supports the batched flow per spec §3.3 + Task 6:
- SweepBase.sim_variants: Vec<SimVariant> — list of (cost, latency,
  threshold, ...) variants. When non-empty, each cell runs ONE harness
  at n_parallel=variants.len() with BatchedSimConfig::from_grid instead
  of the legacy one-harness-per-cell fan-out.
- SweepBase.data_template: Option<String> — when set with `{window}`
  placeholder, each cell's `window` field interpolates the per-cell
  data path. Replaces single scalar `data` for the windowed flow.
- SweepCell.window: Option<String> — window identifier (e.g., "2025-Q2").
- SimVariant: threshold + cost_per_lot_per_side required (the spec's
  primary axes); other fields optional overrides on top of SweepBase
  scalars.

New runner pieces:
- BatchedSimConfig::from_grid(&[ResolvedSimVariant]) in
  crates/ml-backtesting/src/sim/batched_config.rs.
- ResolvedSimVariant — per-variant fully-resolved sim params.
- resolve_sim_variants(&SweepBase) in main.rs — layers per-variant
  overrides over base scalars.
- run_batched_cell() in main.rs — builds the harness with
  sim_config_override + variant_names plumbed through. Writes per-
  backtest artifacts to sim_<variant_name>/ subdirs (spec §3.3).

Harness side:
- BacktestHarnessConfig gains variant_names + sim_config_override
  Option fields. When sim_config_override is Some, harness uses that
  directly instead of building from_uniform off scalar cfg. When
  variant_names is Some, write_artifacts uses sim_<name>/ instead of
  cell_NNNN/ subdirs. Both None preserve legacy single-cell behaviour
  (smoke, fixtures unchanged).

YAML configs:
- config/ml/sweep_threshold_tuning.yaml: 1 cell (W0) × 8 sim_variants
  (p60-p95 in 5pt steps) with cost=0.125 (1-tick anchor). Threshold
  pre-registration pass.
- config/ml/sweep_deployability.yaml: 4 cells (W1-W4) × 140 variants
  each (7 costs × 4 latencies × 5 thresholds). Generated by
  scripts/generate_sweep_variants.py — placeholder threshold values
  (p60-p95) until threshold-tuning publishes calibrated absolutes to
  config/ml/v2_prod_thresholds.json.

Deferred to P7 (operational glue):
- argo-lob-sweep.sh adaptation for the batched flow (cells = windows,
  not sim-variants; one Argo task per window invokes `fxt-backtest sweep`
  end-to-end inside the pod rather than `fxt-backtest run`).

Regression: all 7 existing CUDA tests pass through the new harness
construction path (sim_config_override = None → from_uniform fallback).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 17:41:20 +02:00
jgrusewski
453a22f47f feat(ml-alpha): CUDA Graph capture of forward_only (P5)
X11's original plan said "capture_graph_a covers full v2 forward" but
the X11 commit (4f888abbf) only shipped forward_only + from_checkpoint.
Graph capture is now actually implemented for the inference path.

Mirrors the pattern already in step_batched (perception.rs:1221-1257):
- First call: eager dispatch + set forward_warmed flag.
- Second call: begin_capture -> dispatch_forward_kernels -> end_capture
  -> store CudaGraph.
- Subsequent calls: graph.launch() — captured replay.

forward_only now performs its own staging-fill of the mapped-pinned
host buffers (input data varies per call), then dispatches through
the three-state machine. The captured region is the new private
dispatch_forward_kernels helper: a copy of evaluate_batched's
forward chain (VSN -> Mamba2 x2 -> LN x2 -> attn-pool -> CfC K-loop
-> heads) that omits labels, BCE, and any stream syncs. The final
sync + dtoh of probs_per_k_d happens OUTSIDE capture in forward_only.

Per pearl_no_host_branches_in_captured_graph: no host branches /
scalar-arg-changes / host-mallocs inside the captured region; all
kernel launches use pre-bound device pointers stable across replays.
Per pearl_cudarc_disable_event_tracking_for_graph_capture: event
tracking is already disabled for the trainer's lifetime at
construction (see PerceptionTrainer::new ~line 529), so the captured
region is free of cuStreamWaitEvent / event.record() insertions.

Vestigial loader.rs:272 doc comment referencing the never-shipped
CfcTrunk::capture_graph_a updated to point at the now-real
PerceptionTrainer::forward_only warmup path.

Regression: forward_captured_matches_uncaptured — eager (call 1) vs
captured replay (call 3) agree within 1e-5 relative tolerance per
element. NOT strict bit-identity because CUDA Graph capture can
reorder kernel launches and flip f32 reductions by 1 ULP harmlessly.
Local RTX 3050 Ti run: 160 elements, max rel_err = 0e0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 17:32:28 +02:00
jgrusewski
cd82f9a4a0 feat(ml-backtesting): threshold gate + per-fill cost integration (P4)
Adds the two sweep axes that the spec's deployability grid needs but
were missing from the kernels:

Threshold gate (decision_policy.cu, both kernels):
- New per-backtest `threshold_per_b` array kernel arg.
- Pre-Kelly prelude: if max_h |alpha[h] - 0.5| * 2 < threshold[b],
  emit noop and return. Kept deterministic from alpha alone so the
  threshold pre-registration step (p60-p95 absolute calibration on a
  validation window, future P6) reflects exactly what gets gated in
  deployment.

Per-fill cost integration (resting_orders.cu / apply_fill_to_pos):
- apply_fill_to_pos signature grows three args: b, cost_per_lot_per_side_per_b,
  total_fees_per_b. Single insertion point at line 90.
- After the close-leg realized_pnl math runs (so the gross unwind P&L
  is preserved), deduct fill_cost = filled_lots * cost_per_lot_per_side[b]
  from pos.realized_pnl AND accumulate into total_fees_per_b[b].
- Net-of-cost semantics: isv_kelly_update_on_close reads realized_pnl
  delta which is now net of cost — Kelly state learns from realistic
  return distribution.
- All 3 apply_fill_to_pos call sites in step_resting_orders updated.
  order_match.cu's submit_market_immediate path is dead code in the
  post-P1 flow (everything routes through seed_inflight_limits_batched
  → step_resting_orders → apply_fill_to_pos) so not touched here.

BatchedSimConfig + UniformSimParams + BacktestHarnessConfig gain
threshold + cost_per_lot_per_side fields. All UniformSimParams
constructors in tests and main.rs updated with defaults (0.0, 0.0 =
gate disabled, frictionless).

Regression:
- threshold_gate_skips_low_conviction (p=0.51 + threshold=0.10 → noop)
- threshold_gate_allows_high_conviction (p=0.8 + threshold=0.10 → buy 1+)
- threshold_zero_is_passthrough (sanity)
- All P1+P2+P3 tests continue to pass via the new ABI.

cost_deducted_at_each_fill + kelly_state_sees_net_return end-to-end
tests deferred — they require a full submit_market → fill → close
sequence, which the production smoke exercises.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 17:13:29 +02:00
jgrusewski
3836e25783 feat(ml-backtesting): detect_close_transitions_batched kernel (P3)
Replaces TWO host loops in step_decision_with_latency with two GPU kernels:

1. snapshot_pos_state — replaces the host-side snapshot_realized_pnl +
   snapshot_position_lots + snapshot_open_horizon_mask trio (3 separate
   memcpy_dtoh per decision). Now one kernel launch writes
   prev_pos_lots_d / prev_realized_pnl_d / prev_open_horizon_mask_d
   directly on the device.

2. detect_close_transitions_batched — replaces the host close-detect
   loop that called read_pos per close-eligible backtest (up to
   n_backtests memcpy_dtoh per decision). Now one kernel writes
   closed_horizon_mask_d + realised_return_d on the device, and
   isv_kelly_update_on_close consumes them with no host roundtrip.

At n_parallel=140 these two loops together accounted for ~350M+ small
host roundtrips per quarter. Combined with P2 the latency-path of
step_decision_with_latency is now fully GPU-resident.

isv_kelly_update_on_close kernel always launches (skips backtests with
mask=0 internally) rather than gating via a host any_close check.

All P1+P2 regression tests pass through the new GPU close-detect path
(at n=1 with uniform config + immediate-fill latency, no close happens
in the cold-start tests since they only check market_target; the close
path is exercised indirectly).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 17:04:42 +02:00
jgrusewski
b741f0c5ce feat(ml-backtesting): seed_inflight_limits_batched kernel (P2)
Replaces the host roundtrip loop at the latency path of step_decision_
with_latency with one GPU kernel launch. At n_parallel=140 the host
loop did up to 140 memcpy_dtoh + 140 seed_limit_order calls per
decision (~210M roundtrips per quarter at the threshold-tuning load).
The new kernel does the same work in one launch.

Per-backtest single-writer (threadIdx.x==0). Each backtest scans its
own MAX_LIMITS=32 slot range for an `active==0` slot. Slot allocation
is per-backtest (no cross-backtest atomics needed). Overflow path
increments pos.submission_overflow.

dispatch_latent_market_orders now takes &BatchedSimConfig (unused
inside — the latency_ns_d device buffer is already populated by
step_decision_with_latency's upload block from P1).

All 3 decision_floor_coldstart tests still pass via the new GPU path
(at latency_ns=0 the in-flight slot's arrival_ts==current_ts and gets
promoted on the next step_resting_orders, functionally equivalent to
the legacy submit_market_immediate kernel).

Independence test (different per-backtest latencies → different
arrival_ts) deferred to a later step where a read_first_inflight_arrival_ts
helper is added.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 17:01:25 +02:00
jgrusewski
b8966fb1a6 feat(ml-backtesting): per-backtest sim parameter arrays (P1)
Migrates target_annual_vol_units, annualisation_factor, max_lots,
latency_ns, kelly_frac_floor, sharpe_weight_floor from scalar-broadcast
kernel args to per-backtest device arrays via BatchedSimConfig. Atomic
contract change per feedback_no_partial_refactor — kernel + sim + harness
+ all 3 existing test files migrate in this commit.

- crates/ml-backtesting/src/sim.rs → sim/mod.rs (directory module)
- crates/ml-backtesting/src/sim/batched_config.rs (NEW): BatchedSimConfig
  + UniformSimParams + validate(). from_uniform rebuilds the legacy
  uniform-broadcast behaviour at n=1 (smoke/fixtures). from_grid lands
  in P6 for the 140-variant sweep packing.
- LobSimCuda gains 6 per-backtest device buffers (target_annual_vol_units_d,
  annualisation_factor_d, max_lots_d, latency_ns_d, kelly_frac_floor_d,
  sharpe_weight_floor_d). step_decision_with_latency uploads from
  BatchedSimConfig each call; both decision kernel launches now pass
  per-backtest array pointers.
- decision_policy_default + decision_policy_program: scalar args become
  const float* / const int* per_b arrays; first lines of each kernel
  index by `b` into the arrays. Behaviour preserved at n=1 uniform.
- dispatch_latent_market_orders: reads latency per-backtest from
  &BatchedSimConfig (host loop stays for P1; P2 replaces with kernel).
- step_decision_with_latency now ALWAYS dispatches through the latency
  path; when cfg.latency_ns[b]=0 the in-flight slot's arrival_ts equals
  current_ts and gets promoted immediately on next snapshot. Eliminates
  the if/else branch and consolidates the launch path.
- harness.rs: BacktestHarness gains a sim_config field, built via
  BatchedSimConfig::from_uniform at new() from the harness cfg's scalar
  fields. The run loop passes &self.sim_config to step_decision_with_latency.

Regression coverage:
- parallel_sim_correctness::parallel_sim_equivalence_with_uniform_config
  — n=8 with uniform config produces 8 bit-identical market_targets
  (proves per-backtest indexing reduces correctly).
- Existing decision_floor_coldstart tests (3) all pass through the new
  ABI — proves cold-start floor + variance-cap gate behaviour preserved.
- parallel_sim_independence_per_backtest deferred to P2 (needs
  read_first_inflight_arrival_ts helper that depends on LimitSlot layout).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 16:58:10 +02:00
jgrusewski
28d3ea57b7 fix(ml-backtesting): smoke stride 4 + harness progress log line
After widening the smoke to max_events=0 (full quarter), the inherited
decision_stride=1 became 10M+ forward passes — hours of GPU per cell,
indistinguishable from a deadlock in pod logs.

Two operational fixes:
- sweep_smoke.yaml: decision_stride 1 -> 4 (matches the sweep
  template's own default).
- harness.rs: emit a `progress: events=... decisions=... elapsed=...
  rate=...ev/s` line every 1M events on stderr so multi-million-event
  cells are observable from kubectl logs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 14:41:35 +02:00
jgrusewski
c7fdc617dc fix(ml-backtesting): variance-cap sample-size gate + aggregate GPU req
Three follow-ups to the cold-start floor fix:

1. Kernel: MIN_TRADES_FOR_VAR_CAP gate. After the first trade closed,
   `isv_kelly_update_on_close` set `realised_return_var = ret²` — a
   single-sample variance proxy that systematically collapses
   `cap_units = target_vol / sqrt(var × ann_factor)` near zero for
   any biased return. cap_lots → 0 → no further trades despite strong
   alpha. Gate the variance-derived cap behind `n_trades_seen >= 10`;
   below the threshold cap falls back to host-supplied `max_lots`,
   same as the pre-first-trade path. Same gate applied to both
   `decision_policy_default` and `decision_policy_program`.

   Regression: `post_first_loss_state_does_not_lock_out_further_trades`
   reproduces the exact pre-fix state from the smoke (n_trades_seen=1,
   var=103.6) and asserts the kernel still fires a long with p=0.8.

2. Aggregate: add `nvidia.com/gpu: 1` resource request. Scaleway's
   L40S device plugin mounts libcuda.so.1 into the container only on
   GPU-requesting pods; the aggregate logic is CPU-only but the
   binary's dynamic loader needs the driver libs. Cheapest correct
   fix until a separate CPU-only aggregator binary exists.

3. Smoke YAML: `max_events: 0` (exhaust loader). 100k events is
   minutes of ES.FUT, far shorter than the h6000 holding horizon the
   model was trained on. Full quarter exercises sustained trading +
   variance estimate ramp-up.

All three regression tests pass locally on RTX 3050 Ti.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 14:07:37 +02:00
jgrusewski
da21feb1b1 fix(ml-backtesting): cold-start Kelly + Sharpe-weight floors in decision kernel
Production smoke completed end-to-end but produced n_trades=0 across 99,969
decisions — `decision_policy_default` and `decision_policy_program` both
applied a sentinel-skip pattern: if `isv_kelly_d` had not been seeded
(pnl_ema_win == 0), each horizon's signed-size stayed zero, AND each
horizon's aggregation weight (= recent_sharpe) also stayed zero. The
cross-horizon w_sum was therefore 0, final_size was 0, every market_target
was noop. State only updates on trade close → no trade ever fires →
infinite cold-start.

Per pearl_blend_formulas_must_have_permanent_floor (`max(real, floor)`,
not blend) and pearl_kelly_cap_signal_driven_floors, replace the sentinel-
skip with a two-layer floor on each kernel:

1. Kelly fraction: `max(kelly_frac_floor, computed_kelly)` — when state
   is sentinel, falls back to the floor directly. Cap_lots falls back
   to `max_lots` when realised_return_var is sentinel.
2. Aggregation weight: `max(sharpe_weight_floor, recent_sharpe)` — lets
   cross-horizon sum produce a non-zero size before recent_sharpe is
   populated. Once a horizon shows positive sharpe it dominates.

Plumbed through `step_decision_with_latency` / `step_decision` as two
new f32 args (atomic contract change, every caller migrated). Defaults
0.20 / 0.10 chosen so a strong-conviction signal (sig_mag ≥ 0.5) fires
1 lot at cold-start under max_lots=5 while weaker signals stay flat
(see `default_kelly_frac_floor` comment for the arithmetic). Exposed
as CLI flags + sweep-grid base/cell overrides.

Regression test `decision_floor_coldstart` proves:
 - default floors (0.20/0.10) fire a 1-lot buy with p_h=0.8 and zero state
 - zero floors reproduce the original noop bug

Also moves `aggregate` step to the GPU pool because fxt-backtest is
dynamically linked against libcuda.so.1 (the ci-compile-cpu hosts
don't expose CUDA driver libs).

Verified locally on RTX 3050 Ti — workspace cargo check passes, both
regression tests pass, trunk save/load roundtrip still passes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 13:41:17 +02:00
jgrusewski
e2e3848b86 fix(ml-alpha): construct Mamba2 stacks in CfcTrunk::new_random
Production smoke panicked at trunk.rs:mamba2_l1() during
PerceptionTrainer::from_checkpoint. Root cause: load_checkpoint
called new_random which left mamba2_stack_1/2 = None, then tried
to upload weights into None blocks.

PerceptionTrainer::new was the only caller that populated the
Mamba2 stacks (X3/X5 migrations stopped at the accessor methods
but never moved construction). Inference paths that didn't go
through PerceptionTrainer::new — exactly what fxt-backtest does
via from_checkpoint → load_checkpoint — hit the gap.

Move Mamba2Block::new + state allocation into CfcTrunk::new_random.
PerceptionTrainer::new now sets up its optimizer + scratches against
the trunk-owned blocks via existing accessors (mamba2_l1/mamba2_l2).

Extends the trunk roundtrip test to:
  - assert mamba2 weight slices are non-empty (catches empty-buffer
    regressions that would let bit-equivalence pass trivially)
  - bit-equivalence-check Mamba2 stack 1 + stack 2 weights through
    save -> load, reproducing the exact failure path that hit prod.

Verified locally on RTX 3050 Ti — full workspace cargo check passes,
test passes with non-empty weight slices.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 13:13:02 +02:00
jgrusewski
cecc08a122 chore(ml-alpha): deep cleanup — delete all V1 dead code
CfcTrunk (~250 lines deleted):
- Deleted V1 forward methods: dispatch_perception, capture_graph_a,
  perception_forward_captured, snapshot_hidden, update_input_buffers,
  forward_snapshot, upload_pre_allocated
- Deleted V1 weight fields: heads_w_d, heads_b_d, proj_w_d, proj_b_d,
  proj_g_d, proj_n_d
- Deleted V1 per-step scratches: h_ping, h_pong, bid_px_d, bid_sz_d,
  ask_px_d, ask_sz_d, prev_bid_sz_d, prev_ask_sz_d, regime_d,
  snap_feat_d, probs_d, proj_out_d
- Deleted V1 staging buffers: stg_bid_px / stg_bid_sz / stg_ask_px /
  stg_ask_sz / stg_regime (MappedF32Buffer was only used by V1)
- Deleted graph_a field + _proj_module + V1 fn handles (snap_fn, step_fn,
  heads_fn, proj_fn)
- Deleted V1-only init in new_random (heads_w/b, proj_w/b/g/n, per-step
  scratch allocs)
- Deleted file-level helpers used only by V1: upload(stream, host),
  upload_into, copy_dtod, download
- Deleted PROJ_CUBIN constant
- Updated save_load_roundtrip test to assert on v2 weight tensors
- Stripped unused imports (CUgraphInstantiate_flags, CUstreamCaptureMode,
  CudaGraph, LaunchConfig, PushKernelArg, DevicePtr, DevicePtrMut,
  MappedF32Buffer, ES_TICK_SIZE, Mbp10RawInput, REGIME_DIM, PROJ_DIM)

PerceptionTrainer (~30 lines deleted):
- Deleted duplicate cubin fn fields made dead by X10b: snap_batched_fn,
  step_batched_fn, heads_grn_fwd_fn, transpose_3d_fn, vsn_fwd_fn,
  ln_fwd_fn, attn_fwd_fn (trainer reads these from self.trunk now)
- Deleted their load_function bindings in PerceptionTrainer::new
- Backward kernels (ln_bwd_fn, vsn_bwd_fn, attn_bwd_fn, step_bwd_batched_fn,
  heads_grn_bwd_fn) kept — training-only, not on trunk

Verification:
- perception_forward_golden: PASS (max_diff = 0.000000)
- ml-alpha lib tests: 33 pass
- ml-backtesting + fxt-backtest build clean

Trunk.rs shrunk from 800+ to ~480 lines. Code is now purely the v2
inference graph: weights, kernel handles, save_checkpoint/load_checkpoint,
mamba2_l1/l2 accessors. No V1 surface area left.
2026-05-19 09:19:26 +02:00
jgrusewski
71b467be40 chore(ml-alpha): remove V1-forward test files (broken since X8 reshape)
Both tests exercised CfcTrunk::capture_graph_a + perception_forward_captured
+ snapshot_hidden, which feed V1-shaped CfC weights. After X8 the trunk's
CfC was reshaped to v2 layout (cfc_n_in=HIDDEN_DIM); the V1 forward path
now feeds FEATURE_DIM input into HIDDEN_DIM-shaped CfC — runtime garbage.

The methods themselves are still in trunk.rs (marked dead-code by rustc).
A deeper cleanup pass — deleting the V1 weight fields (heads_w_d, proj_*),
V1 per-step scratches, V1 cubin function handles, and the V1 forward
methods themselves — is a follow-up commit when fresh.

Verification: ml-alpha + ml-backtesting + fxt-backtest all build clean.
2026-05-19 09:08:04 +02:00
jgrusewski
395e0d3000 refactor(ml-backtesting): drive forward via PerceptionTrainer.forward_only
BacktestHarness now owns a PerceptionTrainer (in inference role) instead
of a raw CfcTrunk. The sliding K-window of recent snapshots accumulates
in the harness; at each decision-stride boundary (and only once the
window has reached cfg.seq_len), the harness calls
trainer.forward_only(&window) and broadcasts the last K position's
per-horizon probs to the LobSim.

fxt-backtest's main.rs constructs the trainer via
PerceptionTrainer::from_checkpoint when --checkpoint is supplied (else
random init for noise baseline).

Why this shape: PerceptionTrainer's evaluate_batched already runs the
full inference chain (snap → vsn → mamba2 → ln → mamba2 → ln →
attn_pool → cfc K-loop → grn heads) correctly. Duplicating that 400-line
forward chain on CfcTrunk would double the surface area for the same
result — the trunk's role is weight-source-of-truth (achieved in X1-X9),
not kernel-launch orchestration.

End-to-end status: alpha_train emits Checkpoint files via X14 wiring;
fxt-backtest now loads those Checkpoints via from_checkpoint and drives
forward via forward_only. Phase 2 (Argo runtime: training → smoke →
threshold pre-reg → 560-cell deployability sweep → verdict) is unblocked.

Adds PerceptionTrainer::config() accessor so the harness can read seq_len.

Verification: ml-alpha + ml-backtesting + fxt-backtest all build clean.
2026-05-19 09:06:26 +02:00
jgrusewski
4f888abbf0 feat(ml-alpha): PerceptionTrainer::forward_only + from_checkpoint (X11)
X11 inference interface for the deployability backtester. Two new
public methods on PerceptionTrainer:

  forward_only(snapshots) -> Vec<f32>
      Forward-only pass over a K-snapshot window. Wraps evaluate() with
      dummy zero-valued labels and discards the loss. Returns the same
      [K, B, N_HORIZONS]-shaped probability output as evaluate_batched.

  from_checkpoint(dev, cfg, path) -> Result<Self>
      Build a PerceptionTrainer ready for inference: constructs a fresh
      trainer (with random init) to wire up kernel handles + grad
      buffers + scratches, then overwrites the trunk's weights from the
      Checkpoint file via CfcTrunk::load_checkpoint. The optimizer +
      grad buffers stay allocated — unused at inference, but allocating
      them keeps the struct invariant uniform. A leaner inference-only
      struct can be added later if memory matters.

Architectural note: the trunk is the source of truth for weights (X1-X9
established that). PerceptionTrainer is the kernel-launch adapter that
drives forward + backward over those weights. Inference doesn't need a
separate forward path on the trunk — the trainer's evaluate_batched
already does the full chain correctly. fxt-backtest can now construct a
PerceptionTrainer in inference role via from_checkpoint + call
forward_only per decision.

Verification: ml-alpha + ml-backtesting + fxt-backtest build clean.
ml-alpha lib tests: 33 pass.
2026-05-19 09:03:09 +02:00
jgrusewski
87b8303950 chore(ml-alpha): drop 'v2' naming — greenfield, no backward compat
Per project_ml_alpha_starting_capital greenfield posture: there is no V1
to differentiate from (the V1 trunk forward was dead code, no V1
checkpoint files exist in the wild). The 'v2' prefix on every identifier
was historical baggage from the migration period.

Renames:
- CheckpointV2 -> Checkpoint (also drops the version: u32 field —
  bincode either deserialises a current envelope or errors; no migration
  path needed)
- CheckpointVersionProbe removed (was only for V1 rejection)
- LAYER_NORM_CUBIN_V2 / VARIABLE_SELECTION_CUBIN_V2 / ATTENTION_POOL_CUBIN_V2
  -> LAYER_NORM_CUBIN / VARIABLE_SELECTION_CUBIN / ATTENTION_POOL_CUBIN
- _ln_module_v2 / _vsn_module_v2 / _attn_module_v2 -> drop _v2 suffix
- smoke_load_v2_checkpoint test -> smoke_load_checkpoint
- config/ml/sweep_v2_*.yaml -> config/ml/sweep_*.yaml
- migration-era 'V2 weight skeleton' / 'V2 fields' / etc. comments
  cleaned to remove the v2 prefix

Pre-existing 'v2' references in ml-backtesting CUDA files
(decision_policy.cu, pnl_track.cu) are NOT touched — those refer to
future planned 'v2' refinements (Portfolio mode, multi-fill averaging)
from the C1-C19 commits and reflect aspirational features unrelated to
this session's trunk-grows work.

Verification: ml-alpha + ml-backtesting + fxt-backtest all build clean.
perception_forward_golden bit-exact (max_diff = 0.000000).
2026-05-19 08:58:28 +02:00
jgrusewski
f4ad96bacb feat(ml-alpha): CfcConfig extended with n_batch + seq_len for v2 forward (X11a)
Adds CfcConfig.n_batch (default 1) and CfcConfig.seq_len (default 32).
PerceptionTrainer's trunk construction passes its training-time
n_batch/seq_len. Default values support backtester inference (B=1, K=32).

X11 foundation: subsequent commits add intermediate buffer fields +
forward_v2 method on CfcTrunk sized from these config fields. With X11
complete, fxt-backtest can drive the v2 forward through the trunk
without instantiating a full PerceptionTrainer.

Verification: ml-alpha + ml-backtesting + fxt-backtest all build clean.

Per spec §1.1 (X11 foundation).
2026-05-19 08:52:03 +02:00
jgrusewski
5e8f49bf1c test(ml-backtesting): GPU smoke against real CheckpointV2 (X18)
Adds #[ignore]d integration test that loads a CheckpointV2 file produced
by alpha_train and verifies CfcTrunk::load_checkpoint accepts it.
Activated via FOXHUNT_SMOKE_CKPT env var on a CUDA-capable host.

Also adds a compile-time witness test confirming Summary.max_drawdown_pct
field exists post-X16 (required by emit_deployability_verdict).

Note: full BacktestHarness end-to-end smoke (running one cell against
fixture MBP-10) requires the v2 forward path to live on the trunk
(X11 — deferred). This commit verifies the producer/consumer wire-up.

Per spec §3.4, §4.1 (X18).
2026-05-19 08:45:50 +02:00
jgrusewski
0ddc861708 feat(ml-backtesting): max_drawdown_pct + deployability verdict emitter (X16+X17)
X16: Adds Summary.max_drawdown_pct field as |max_drawdown_usd| /
STARTING_CAPITAL_USD (pinned at $35k per project_ml_alpha_starting_capital
memory — realistic ES single-contract small-account anchor). Used by
the verdict emitter as the capital-deployability hard gate (median
across windows < 20%).

X17: Adds VerdictTier enum (Pass-robust / Pass-nominal / Fail-inconclusive
/ Fail / Fail-degenerate), AnchorSpec / AnchorReport / DeployabilityVerdict
types, classify_verdict (tiered logic per spec §3.5), and
emit_deployability_verdict that reads per-cell summary.json files at
both realistic (1.0 tick, 200 ms) and stress (1.5 tick, 400 ms) anchors,
computes median Sharpe / Sortino / max_dd_pct / profit_factor across
walk-forward windows, and applies the gates.

Per spec §3.2 (X16), §3.5 (X17).
2026-05-19 08:44:46 +02:00
jgrusewski
7545651bce feat(ml-alpha): PerceptionTrainer.save_checkpoint + alpha_train wiring (X13+X14)
X13: Adds PerceptionTrainer::save_checkpoint as a thin delegate to
self.trunk.save_checkpoint. Inference-only serialization — grads + AdamW
state aren't included.

X14: Inside the existing auc_h6000_improved block in alpha_train.rs,
calls trainer.save_checkpoint(out_dir / 'trunk_best_h6000.bin') so the
trained trunk lands alongside alpha_train_summary.json. Extends
AlphaTrainSummary with best_h6000_ckpt_path (Option<String>) so
downstream tooling (fxt-backtest --checkpoint) can locate the file
without re-deriving the path.

After this commit, every alpha-perception Argo workflow run produces
a CheckpointV2 file at every new-best-h6000 epoch, ready for backtest
consumption.

Verification:
- ml-alpha lib tests: 34 pass
- alpha_train example builds clean (release)

Per spec §1.1 (X13+X14).
2026-05-19 08:41:12 +02:00
jgrusewski
47a605e4c5 feat(ml-alpha): CheckpointV2 envelope + save/load for full v2 trunk (X12)
Replaces CheckpointV1 with CheckpointV2 — covers the full v2 inference
graph: VSN, Mamba2 stacks 1+2 (in/a/b/c/out weights + biases), LN_a/LN_b,
attention-pool, CfC (now v2-shaped via cfc_n_in=HIDDEN_DIM), and the
full GRN heads (10 tensors: w1/b1, w2/b2, w_gate/b_gate, w_main/b_main,
w_skip/b_skip).

save_checkpoint reads every trunk weight tensor via memcpy_dtoh and
packs into the CheckpointV2 bincode envelope. load_checkpoint peeks
the version first (CheckpointVersionProbe), rejects non-2 versions,
deserialises into CheckpointV2, validates n_in / n_hid / cfc_n_in /
mamba2_state_dim match the supplied cfg, and uploads each weight
tensor with size-checked memcpy_htod.

V1 envelopes hard-rejected — alpha_train never produced V1 files, so
no migration. The old V1-shaped roundtrip test is removed; new V2
round-trip test will land alongside the alpha_train wiring (X14).

Verification:
- perception_forward_golden: PASS (max_diff = 0.000000)
- ml-alpha lib tests: 34 pass
- fxt-backtest binary builds clean

Per spec §1.2 (X12).
2026-05-19 08:39:28 +02:00
jgrusewski
716e0d0781 refactor(ml-alpha): trainer launches forward kernels via trunk handles (X10b)
PerceptionTrainer's evaluate_batched + step_batched now read forward
kernel handles from self.trunk (snap_batched_fn, vsn_fwd_fn, ln_fwd_fn,
attn_fwd_fn, step_batched_fn, heads_grn_fwd_fn, transpose_3d_fn).
Trainer's duplicate cubin/function loading stays in place for now;
deferred dead-code cleanup.

Backward kernels (vsn_bwd_fn, ln_bwd_fn, etc.) stay on trainer — they
are training-only and don't belong on the inference trunk.

Verification:
- perception_forward_golden: PASS (max_diff = 0.000000)

Per spec §2.2 (X10b).
2026-05-19 08:33:37 +02:00
jgrusewski
c820b669de feat(ml-alpha): CfcTrunk loads v2 forward kernel handles (X10 foundation)
Adds the v2 forward kernel cubins (layer_norm, variable_selection,
attention_pool) and function handles (vsn_fwd_fn, ln_fwd_fn, attn_fwd_fn,
snap_batched_fn, step_batched_fn, heads_grn_fwd_fn, transpose_3d_fn)
onto CfcTrunk. The trunk now owns every kernel handle the v2 forward
chain needs.

PerceptionTrainer still loads its own copies of these cubins (duplicate
loading) and uses its own handles in evaluate_batched / step_batched —
the trainer-side consolidation lands in X10b. This commit is the
foundation: X11 will build capture_graph_a using these trunk-owned
handles, and X12 (CheckpointV2) doesn't depend on the consolidation.

Verification:
- perception_forward_golden: PASS (max_diff = 0.000000)
- ml-alpha lib tests: 34 pass

Per spec §2.2 (X10).
2026-05-19 08:31:17 +02:00
jgrusewski
5bd8897880 refactor(ml-alpha): move GRN heads from PerceptionTrainer to CfcTrunk (X9)
Adds the 4 missing GRN head fields (heads_w1, heads_b1, heads_w2, heads_b2)
that X1's skeleton under-modeled. Trunk now owns all 10 GRN head tensors:
input projection (HIDDEN→HEAD_MID), mid→mid layer, gate/main/skip outputs.

Trainer's 10 head fields removed. Trainer init still draws heads from
its ChaCha8Rng chain at the same call position, then memcpy_htod's them
into the trunk's now-allocated slots. Forward, backward, and AdamW
access sites redirect to self.trunk.heads_*. Gradient buffers + AdamW
state stay at trainer level.

Verification:
- perception_forward_golden: PASS (max_diff = 0.000000)
- ml-alpha lib tests: 34 pass

Per spec §2.2 (X9). After this commit, the trunk owns ALL v2 inference
weights — the source of truth for downstream checkpoint serialization.
2026-05-19 08:24:48 +02:00
jgrusewski
ef29e13c74 refactor(ml-alpha): move CfC weights from PerceptionTrainer to CfcTrunk (X8)
Adds CfcConfig.cfc_n_in field (default HIDDEN_DIM) so the trunk's CfC
layer is sized for v2 usage (CfC input = LN_b output = HIDDEN_DIM) rather
than V1 usage (CfC input = snap features = FEATURE_DIM). The previous
CfcConfig.n_in field stays as "raw snap feature dim" for any V1 callers
still in the tree (fxt-backtest, trunk_forward.rs, graph_a_replay.rs);
their forward paths will be cleaned up in X10/X11 when the v2 forward
graph lives natively on the trunk.

Trainer's w_in_d / w_rec_d / b_d / tau_d fields are removed. Trainer
init still draws CfC weights from its ChaCha8Rng chain at the same
call position, then memcpy_htod's them into the trunk's now-v2-shaped
slots. Forward, backward, and AdamW access sites redirect to
self.trunk.{w_in,w_rec,b,tau}_d.

Verification:
- perception_forward_golden: PASS (max_diff = 0.000000)
- ml-alpha lib tests: 34 pass

Per spec §2.2 (X8).
2026-05-19 08:20:33 +02:00
jgrusewski
2bd774ad14 refactor(ml-alpha): move attention-pool weight to CfcTrunk (X7)
attn_q_d moved from PerceptionTrainer to self.trunk.attn_q_d.

Verification: golden bit-exact; ml-alpha lib green.

Per spec §2.2 (X7).
2026-05-19 01:45:57 +02:00
jgrusewski
3357699431 refactor(ml-alpha): move LN_b weights from PerceptionTrainer to CfcTrunk (X6)
LN_b (formerly ln_gain_d / ln_bias_d on trainer) now lives at
self.trunk.ln_b_gain_d / ln_b_bias_d. LN_b is the LayerNorm after
Mamba2 stack 2.

Verification: golden bit-exact; ml-alpha lib green.

Per spec §2.2 (X6).
2026-05-19 01:44:48 +02:00
jgrusewski
5a534e9972 refactor(ml-alpha): move Mamba2 stack 2 from PerceptionTrainer to CfcTrunk (X5)
Same pattern as X3: trainer constructs mamba2_l2 + Mamba2AdamW (against
&mamba2_l2), then moves the block into trunk.mamba2_stack_2. All
forward/backward/AdamW access sites redirect to self.trunk.mamba2_l2_mut().
Gradient buffers + AdamW state remain at trainer level.

Verification: golden bit-exact; ml-alpha lib green.

Per spec §2.2 (X5).
2026-05-19 01:43:00 +02:00
jgrusewski
868021e818 refactor(ml-alpha): move LN_a weights from PerceptionTrainer to CfcTrunk (X4)
LN_a gain/bias tensors now live at self.trunk.ln_a_gain_d / ln_a_bias_d.
Trainer-side initialization values still drive the upload, preserving
PRNG-driven init values. Gradient buffers + AdamW state remain at
trainer level.

Verification: golden bit-exact (max_diff = 0.000000); ml-alpha lib green.

Per spec 2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md §2.2 (X4).
2026-05-19 01:41:19 +02:00
jgrusewski
2d849dd5e3 refactor(ml-alpha): move Mamba2 stack 1 from PerceptionTrainer to CfcTrunk (X3)
PerceptionTrainer no longer owns the Mamba2 stack-1 block; the trunk's
mamba2_stack_1 Option is filled in after Mamba2Block::new (which still
runs at the same point in PerceptionTrainer::new so the Mamba2 weight
init order is unchanged). Mamba2AdamW was already constructed against
&mamba2 before the move, so optimizer state is preserved.

CfcTrunk gains mamba2_l1_mut() / mamba2_l1() / mamba2_l2_mut() / mamba2_l2()
accessors that unwrap the Options. Forward, backward, and AdamW step
sites in evaluate_batched / step_batched / evaluate redirect through the
new accessors.

Gradient buffers (mamba2_grads_buffers) and AdamW state (mamba2_adamw)
remain at trainer level — training-only state stays with the trainer.

Verification:
- perception_forward_golden: PASS (max_diff = 0.000000)
- ml-alpha lib tests: 34 pass

Per spec 2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md §1.1, §2.2 (X3).
EOF
)
2026-05-19 01:39:47 +02:00
jgrusewski
fdef6efe98 refactor(ml-alpha): move VSN weights from PerceptionTrainer to CfcTrunk
PerceptionTrainer gains a `trunk: CfcTrunk` field constructed at the
top of `new()` (after the determinism seed guard). VSN's
weight tensors (`vsn_w_d`, `vsn_b_d`) now live on `self.trunk`; the
trainer-owned copies are removed. Forward + backward + AdamW access
sites redirect to `self.trunk.vsn_w_d` / `self.trunk.vsn_b_d`.

PRNG-state preservation: the trainer's ChaCha8Rng chain still draws
VSN values at the same call position as before, then memcpy_htod's
them into the trunk's zero-initialised VSN slots from X1. This keeps
every downstream weight (attn_q, ...) bit-identical to the pre-X2
layout — perception_forward_golden continues to pass with
max_diff = 0.000000.

Gradient buffers (`grad_vsn_w_d`, `grad_vsn_b_d`) and AdamW state
(`opt_vsn_w`, `opt_vsn_b`) remain at trainer level — they're
training-only and shouldn't move to the inference trunk.

Verification:
- perception_forward_golden: PASS (max_diff = 0.000000)
- ml-alpha lib tests: 34 pass
- alpha_train example builds clean

Per spec 2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md
§1.1, §2.2 (X2).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 01:30:12 +02:00
jgrusewski
e338000eec feat(ml-alpha): CfcTrunk v2 weight skeleton (no callers yet)
Adds zero-initialised v2 weight tensors (VSN, LN_a/b, attn_q, GRN heads)
and Option<Mamba2Block> slots for stacks 1 and 2 to CfcTrunk. Extends
CfcConfig with mamba2_state_dim (default 16, matches
PerceptionTrainerConfig::default). Imports HEAD_MID_DIM for GRN head
sizing.

No forward path changes — fields allocated, not yet read. Existing
new_random init for V1 fields (CfC + simple heads + projection) is
preserved unchanged so the trunk's forward kernels still produce the
same output.

Skeleton for X2..X9 weight-group migrations.

Verification:
- ml-alpha lib tests: 34 pass
- perception_forward_golden bit-equivalence: PASS (max_diff = 0.000000)

Per spec 2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md
§1.1, §2.2 (X1).
2026-05-19 01:23:05 +02:00
jgrusewski
b47b2fabfb fix(ml-core): deterministic GPU weight init via scoped_init_seed
ml_core::cuda_autograd::init::generate_uniform (backing xavier_uniform,
kaiming_uniform, bias_uniform, near_zero_xavier) defaulted to seeding
from SystemTime::now() + thread_id, producing non-reproducible weights
across processes. Mamba2 stacks initialise via OwnedGpuLinear::xavier,
which routes through this helper — so PerceptionTrainer.evaluate output
diverged 5-30% across fresh-process runs with identical cfg.seed.

Fix: thread-local seedable RNG override. New API:

    let _g = ml_core::cuda_autograd::init::scoped_init_seed(seed);
    // ... all xavier/kaiming/bias/near_zero calls draw from
    //     StdRng::seed_from_u64(seed) chain while _g is alive ...
    // _g dropped here -> restores default time-based seeding

PerceptionTrainer::new now installs the guard before any Mamba2Block
construction, so the trainer is reproducible from cfg.seed end-to-end.
CfC/VSN/heads already used explicit ChaCha8Rng::seed_from_u64 — only
Mamba2 was affected.

Production behavior unchanged when no guard is set. ml-core: 306 tests
pass, ml-alpha: 34 lib tests pass.

Regression test: crates/ml-alpha/tests/perception_forward_golden.rs
captures bit-exact PerceptionTrainer.evaluate output (loss + 160 probs
on a deterministic seed=42 fixture) into a 644-byte golden file.
Three consecutive runs now produce max_abs_diff=0; pre-fix runs varied
by 0.1-0.3 absolute on individual probs.

.gitignore: added exception for crates/ml-alpha/tests/fixtures/*.bin
so deterministic test fixtures land in repo.

Per pearl_scoped_init_seed_for_reproducibility in project memory.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 01:19:13 +02:00
jgrusewski
e004d6c217 Revert "arch(ml-alpha): restore count-delta redundancy at feature slot [26]"
This reverts commit 008f65d894.
2026-05-18 23:16:38 +02:00
jgrusewski
004b662c80 obs(ml-alpha): per-N-step train_loss log line for liveness + intra-epoch monitor
Adds a single tracing::info!("step") emitted every 500 training steps:

  if epoch_train_steps % 500 == 0 {
      tracing::info!(epoch, step = epoch_train_steps, loss = loss, "step");
  }

Consumed by /tmp/alpha_monitor.py (v2: per-epoch trajectories + ISV +
liveness) to render intra-epoch loss trajectory and detect stalls
faster than the once-per-epoch granularity allowed.

At 8000 steps/epoch and ~36s/epoch on L40S → ~16 step lines per epoch
per fold, well below the cluster log volume budget.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-18 22:33:44 +02:00
jgrusewski
008f65d894 arch(ml-alpha): restore count-delta redundancy at feature slot [26]
The C16 tick-rule swap (19986c8d9) replaced the dense `trade_count` delta
at snap_feature_assemble's slot [18] with a signed L1 tick-rule estimate.
That broke a load-bearing redundancy in Phase 1+2+3:

  Phase 1+2+3 feature [17] = log1p(trade_count_delta)
  Phase 1+2+3 feature [18] = signed_log1p(trade_count_delta)
                           = log1p(trade_count_delta) for count ≥ 0

Within a file, `cur.trade_count >= prev.trade_count` (monotonic), so the
delta is non-negative and the two features were *bit-identical* floats.
The encoder's Mamba2 W_in had two random-initialised rows projecting the
same dense signal, giving it effective 2× capacity allocation on
trade-flow.

Post-C16:
  feature [17] = log1p(trade_count_delta)         (unchanged)
  feature [18] = signed_log1p(tick_rule_estimate) (new, uncorrelated)

Empirical (7.8M ES MBP-10 snapshots, Q1+Q2 2024 production data):

  | Stat                   | OLD count_delta | NEW tick_rule   |
  |------------------------|-----------------|-----------------|
  | zero rate              | 10.2%           | 49.1%           |
  | mean ± std             | 4.26 ± 3.82     | -0.18 ± 10.95   |
  | max |value|            | 100             | 3378            |
  | Pearson r vs count     | 1.0000 (id)     | 0.0002          |

  Sign-class breakdown vs Phase 1+2+3 slot [18]:
    44.3% new=0 but old≠0  (44% of true trades MISSED by tick-rule)
     5.5% new≠0 but old=0  (cancel-as-trade false positives)
    22.8% both positive    (agree)
     0.0% both negative    (old never negative)
    22.6% sign disagreement (old saw trades, new says "seller")

The tick-rule heuristic is a strictly different (and noisier) signal,
not a superset. Three mechanisms simultaneously regressed mean_auc:
  A) lost 2× W_in capacity on count signal
  B) noisier signal at slot [18]
  C) extreme outliers (max 33× wider) destabilise LayerNorm at [18]

Fix (Option 2 per the diagnostic):

  out[26] = signed_log1p((float) trade_count)

Restores the duplicate count-delta signal at a previously-reserved slot.
Slot [18] keeps the new tick-rule signal — the 5.5% "signal added" and
the directional info at L1 are still available. FEATURE_DIM (40) is
unchanged; LayerNorm + Mamba2 W_in dimensions are unchanged.

Both the single-snapshot kernel and the batched kernel are updated.
`snap_feature_bit_equiv::reserved_slots_are_zero` updated to assert
the new slot-26 semantics (signed_log1p of synthetic trade_count=7
= log(8) ≈ 2.079).

All 9 perception_overfit tests pass + all 9 snap_feature_bit_equiv
tests pass.

Cluster verification: single-fold smoke + 3-fold validation follow.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-18 22:19:23 +02:00
jgrusewski
17eb825113 arch(ml-alpha): σ becomes sidecar — BCE drops Kendall damping (upgraded path)
Empirical record at this data + this architecture:
  Phase 1+2+3 (no σ in loss):           mean_auc 0.7749 ± 0.024
  v2 (σ + axes B/C/D/E):                mean_auc 0.7541 ± 0.005
  σ-only (kept σ, dropped B/C/D/E):     mean_auc 0.7506 ± 0.008
  Perf-fix (σ-only math):               mean_auc 0.7499 ± 0.010
  ISV-σ (closed-form σ + adaptive λ):   mean_auc ~0.75 (2/3 folds)

Every architecture with σ-in-loss lands at 0.75. Removing σ is the
only thing that hits 0.77. That is a framework mismatch, not a tuning
problem.

Kendall+Gal+Cipolla 2018 frames σ as TASK NOISE level — damp the noisy
task, trust the clean one. Our horizons don't have different label
noise; they have different intrinsic difficulty (longer horizon = more
price-walk uncertainty = lower achievable AUC). σ-Kendall sees "high
BCE on h6000" and interprets it as "h6000 is unreliable, back off" —
precisely the opposite of what we want. h6000 is the deployment
target; damping it is a self-inflicted wound. With mean_bce ~ 0.7,
σ ≈ √0.7 ≈ 0.84 ⇒ w_h ≈ 0.71, uniformly attenuating gradient by
~30% across every horizon. λ's z-score boost (max 2×) can rebalance
relative-per-horizon but cannot recover the absolute magnitude.

Per-horizon prioritization remains via the ISV-driven λ controller
(grad scaler in heads_grn_bwd, per pearl_adam_normalizes_loss_weights).
That controller IS appropriate for our problem: it boosts hard
horizons rather than damping them, and it operates on the gradient
into the trunk rather than on the loss aggregate (Adam-cancellation
safe).

Changes to bce_loss_multi_horizon.cu (six lines):
  w_h    = bw                  (was: 0.5 * bw * exp(-2 * log_sigma_h))
  d_log_sigma_h[h] = 0.0       (was: 1 - 2 * w_h * mean_bce)
  total_loss += bw * mean_bce  (was: + w_h * mean_bce + log_sigma_h)

σ infrastructure preserved unchanged:
  - horizon_ema_and_lambda still computes log_sigma_h closed-form from
    loss_ema (Kendall equilibrium) for telemetry / future label-noise
    estimation use cases
  - log_sigma_h kernel arg still in BCE signature (zero churn at
    callsite); ignored inside

All 9 perception_overfit tests pass — including
horizon_ema_and_lambda_track_after_training which validates the
per-horizon controller end-to-end through 64 K-loop iterations of
capture/replay.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-18 21:12:45 +02:00
jgrusewski
410ab6b0ea arch(ml-alpha): ISV-driven σ + adaptive Z_SCALE — both controllers anchor on loss_ema
Per pearl_controller_anchors_isv_driven, every controller anchor/target/
cap derives from a tracked signal, not hardcoded constants. The σ-only
revert kept Kendall σ as a free Adam-learned scalar — that violated ISV
discipline and fought Adam's m/√v normalization
(pearl_adam_normalizes_loss_weights).

Single source of truth for both per-horizon controllers:

  log_sigma_h[h]  ← max(log(0.5), 0.5 * log(loss_ema[h]))
                    Kendall equilibrium (∂L/∂log σ = 0 ⟹ σ_h² = mean_bce_h)
                    in closed form. Asymmetric floor at log(0.5) prevents
                    collapse. No Adam state, no gradient delay.

  lambda[h]       ← clamp(1.0, 2.0, 1.0 + Z_SCALE_ISV * z_h)
                    Z_SCALE_ISV = (LAMBDA_CEILING - LAMBDA_FLOOR) / z_max_ema
                    Adaptive scale auto-uses the full clamp envelope:
                    the historical-max-z horizon maps exactly to
                    LAMBDA_CEILING. Replaces hardcoded Z_SCALE=0.5 which
                    rarely engaged on real data (max observed λ ~1.04).

Both anchor on the same ISV (loss_ema). z_max_ema is a new single-scalar
EMA state tracking max |z| across horizons, with first-obs bootstrap.

Removes:
  - opt_log_sigma AdamW optimizer (σ no longer learned)
  - grad_log_sigma_h_d memset (BCE kernel writes; output ignored — kept
    only to preserve BCE kernel signature)

Kernel signature change (horizon_ema_and_lambda):
  +z_max_ema [1]       (read+write EMA state)
  +log_sigma_h [5]     (closed-form output, overwrite)

Discipline:
  - First-obs bootstrap (sentinel <= 0) per pearl_first_observation_bootstrap
  - Permanent floor (max(real, floor)) per pearl_blend_formulas_must_have_permanent_floor
  - Asymmetric clamp per pearl_audit_unboundedness_for_implicit_asymmetry
  - Z-score normalisation per pearl_zscore_normalization_for_magnitude_asymmetric_signals
  - No nvrtc, no atomicAdd, no host branches in graph capture

All 9 perception_overfit tests pass — including
horizon_ema_and_lambda_track_after_training which validates the kernel
end-to-end through 64 K-loop iterations of capture/replay.

Submit local smoke; cluster A/B vs σ-only baseline (0.7506/0.7519) and
vs Phase 1+2+3 (0.7749/0.7591) follows once the perf-only 3-fold A/B
confirms no regression at b23f8f2ef.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-18 19:32:03 +02:00
jgrusewski
b23f8f2efa perf(ml-alpha): NVIDIA-grade rewrite of CfC K-loop hot kernels — 2.15× faster
Local L40S profile (perception_overfit smoke) GPU kernel time:
  1589ms → 739ms  (53.5% reduction, 2.15× speedup).
Wall-clock smoke: 9.6s → 4.94s (1.94× faster).

Per-kernel deltas (nsys --cuda-graph-trace=node):

  reduce_axis0:              362ms → 10ms   (36× faster)
    Block layout: per-column (1 block / output) → 32-wide column tile
    (block_dim = 32 × 8). Cross-thread reads were strided by n_tail
    (~40K floats = 160KB stride) — one cache line per thread, 8× HBM
    bandwidth wasted. New tile gives coalesced 128B transactions per
    warp. Block tree-reduce kept (no atomicAdd, per feedback_no_atomicadd).
    +1 shared-mem pad to eliminate 32-way bank conflict on the ty reduce.

  multi_horizon_heads_grn_bwd_batched:  540ms → 113ms  (4.8× faster)
    1. Stage h_row[HIDDEN] and a1[5,HEAD_MID] in shared at block entry.
       Eliminates ~28K redundant DRAM reads/block across Pass 3 + Pass 5.
    2. Pass 5 reorder: k outer / i inner with d_z1[k,m] pinned in
       register; writes to grad_w1_scratch are sequential per-thread.
    3. Block size 64 → 128 threads. Pass 5/6 now partition over i
       (output column): cross-thread writes become COALESCED 128B/warp
       (was stride-128 = 512B). Passes 2/3/4 gate on (tid < HEAD_MID).
    4. Pass 3 thread role: m_out → m_in/n. Same coalescing fix on
       grad_w2 writes AND w2 reads in the d_eta_2 sum.

  cfc_step_backward_batched: 351ms → 271ms  (1.3× faster)
    1. Stage x_b[n_in] and h_old_b[n_hid] in shared (was 128× redundant
       DRAM reads per block; now 1× cooperative load).
    2. Pass 1 thread role: i (output row) → k (output col). For each
       i loop iteration, the warp writes grad_w_in[..., tid] /
       grad_w_rec[..., tid] — COALESCED 128B/warp (was stride-128
       non-coalesced).

  multi_horizon_heads_grn_fwd_batched:  211ms → 204ms
    Stage h_row[HIDDEN] in shared — Pass 1 and Pass 3 both consume.

  cfc_step_batched (fwd):     95ms →  94ms
    Stage x_b and h_old_b in shared.

Shared-mem budgets fit comfortably under the 48KB SM cap (~6KB / ~2KB
respectively). All 9 perception_overfit tests pass — gradient
correctness validated end-to-end (constant-signal overfit, K-loop
capture/replay, stride-4 path, evaluate-only paths).

Discipline:
  - Block tree-reduce only, never atomicAdd
  - No nvrtc; pre-compiled cubins via build.rs
  - Mapped-pinned-only is unaffected (CPU↔GPU contract untouched)
  - Single source of truth: replaced kernels in place, no v2 suffixes

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-18 19:07:29 +02:00