Boundary test that fails on bit-pick-first or bit-pick-max regressions
of the open_horizon_masks averaging in stop_check_isv. Per-horizon
pnl_ema_loss values (2, 4, 3, 3, 3) yield mean=3.0 across the 0x1F
mask; snapshots placed relative to vwap_entry for ask-spread safety.
Tests Δ_entry=2.5 (no-fire) and Δ_entry=4.5 (fire).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
trail_distance = max(pnl_ema_win, atr_mid_ema). HWM ratchets up via
fmaxf each event while position open; trail fires when HWM clears
the arming threshold AND unrealized drops by trail_distance from
HWM. Same force-flat (3, 0) write as SL.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Skeleton helper called from decision_policy_default at top of
per-backtest dispatch. Returns 0 (no-op) on flat positions; trigger
logic for open positions added in Tasks 4-6. Single source of truth
for stop logic across decision_policy_{default,program}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cudarc kernel launches require `unsafe` blocks — the driver API has no
way to type-check kernel args against the cubin signature. The
workspace-wide `-W unsafe-code` lint produces noise on every launch
site (10+ blocks); suppressing at the module level keeps the lint
useful elsewhere in the crate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per-event Wiener-α=0.4 EMA on |Δmid| with first-observation bootstrap.
Floor source for the SL/trail-distance controller (spec §6). Thread 0
of the broadcast-snapshot kernel handles the update per backtest;
threads 1..9 still handle the 10 level writes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three CudaSlice<f32> per-backtest slots (prev_mid_d, atr_mid_ema_d,
trail_hwm_d) plus test-only accessors. Foundation for the
ISV-driven stop controller; no behavioral change until subsequent
tasks wire them into kernels.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
11 tasks, each one commit, TDD-disciplined (test → fail → impl →
pass → commit). Covers all 11 sections of the spec at
docs/superpowers/specs/2026-05-19-isv-driven-stop-controller-design.md:
- Tasks 1-2: state slots + ATR EMA in book_update_apply_snapshot
- Task 3: stop_check_isv shared __device__ helper skeleton
- Tasks 4-6: SL trigger, trail-TP + HWM ratchet, multi-horizon avg
- Task 7: bytecode VM parity (wire helper into decision_policy_program)
- Task 8: seed_inflight_limits_batched target-delta + in-flight summation
- Task 9: pnl_track_step trail_hwm reset on close
- Task 10: atomic deletion of StopRules + use_cold_start_stopgap +
integration-test retarget + CBSW SUPERSEDED markers
- Task 11: cluster smoke validation via argo-lob-sweep.sh
All 10 §9.1 unit tests have explicit failing-test scaffolds before
their implementations, and the retargeted integration test
(cold_start_persistent_bullish_now_closes) flips the assertion to
prove the fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Issues caught in self-review and fixed:
1. Encoding ambiguity: stop-fire writing (2, 0) would have silently
collapsed weak-alpha no-op semantics with force-flat. Introduced
distinct side=3 = force-flat; preserved side=2 = no-op for entry
path. Updated §3 + §5 + §7 truth table.
2. trail_hwm reset on close was implied in §4 but missing from §8's
touched-files list. Added pnl_track.cu modification (§7.1) +
added to §8 Added list.
3. __device__ helper enforcement: §3 now mandates stop_check_isv()
as a shared __device__ function called from both decision kernels,
not duplicated code.
4. Kernel arg additions enumerated explicitly in new §4.1 table —
every kernel-launch site gets named, no "thread through every kernel"
hand-wave.
5. seed_inflight_limits_batched (resting_orders.cu:439–475) named
explicitly in §1 + §7 — the actual kernel that needs the position-
target-semantics fix.
6. In-flight order summation: naive delta = target_signed - pos was
strictly worse than the original additive bug under non-zero
latency (would produce pos = pre + K·delta after fills). Fixed
§7 algorithm to compute effective_position by scanning all
active∈{1, 2} slots and summing their signed sizes. Added
position_target_not_additive_with_latency test.
7. ATR α=0.4 honest justification in §10: not strictly derived from
pearl_wiener_alpha_floor_for_nonstationary (which is about co-
adapting control loops, not passive observation); chosen as a
pragmatic project-wide constant matching isv_kelly_update_on_close.
8. CUDA Graph host-branch-free affirmation added to §3 — explicit
compatibility with P5 graph capture.
9. Cold-start symmetry caveat (§5): at n_trades_seen=0 both EMAs are
0 so sl_distance == trail_distance == atr — asymmetry only
emerges post-bootstrap. Acknowledged honestly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces StopRules::default()-all-zeros (the actual cause of the cluster
smoke's n_trades=0, falsifying the CBSW dilution diagnosis) with an
ISV-driven SL + trail-stop controller folded into the existing decision
kernel(s). Per-backtest ATR EMA on mid-price + per-horizon pnl_ema_*
drive distances under max(real, floor) per
pearl_blend_formulas_must_have_permanent_floor. No new kernel; single
source of truth in step_decision*. Folds in the position-target
semantics fix (200-event local repro showed position_lots=199 from
additive-vs-target order semantics) — same commit.
Removes: StopRules, sl_tp_rules, Q1 stopgap field/branch,
use_cold_start_stopgap propagation. Marks CBSW spec + plan SUPERSEDED.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds LobSimCuda::read_total_trade_count (cheap n_backtests*4 byte DtoH of
the trade-log head counters) and emits the sum on each PROGRESS_EVERY
eprintln. Lets future smokes (Q2 CBSW validation, P7 sweep) see trade
firing mid-run instead of waiting for write_artifacts at harness end.
Heads count kernel writes (clamped to ring cap), so the sum is a
monotone trade counter across the sweep — drops back only on harness
reset which doesn't happen mid-run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The threshold-tuning smoke at 81decf40f produced n_trades=0 despite
74.6% of decisions having max_conv ≥ 0.30 — the linear-weighted-mean
aggregator in decision_policy_default is structurally dilution-bound
at cold-start (per spec §1).
Q1 stopgap: when sim_variants[i].use_cold_start_stopgap = true, the
harness uploads a max-confidence Strategy bytecode program for that
backtest, routing decisions through decision_policy_program with
OP_AGG_MAX_CONFIDENCE. Existing kernel; zero CUDA changes.
Field additions (atomically across BatchedSimConfig + UniformSimParams
+ ResolvedSimVariant + SweepBase.SimVariant) — every UniformSimParams
literal migrated to include use_cold_start_stopgap: false (default).
The sweep YAML's sim_variants entry sets it to true only for the
validation run; production deployability uses Q2's kernel fix instead.
Sweep YAML (config/ml/sweep_smoke.yaml) flipped to use_cold_start_stopgap=true
at threshold=0.0, cost=0.125 — same anchor as the threshold-tuning
smoke that produced n_trades=0, for direct comparison.
This is a VALIDATION step. Cluster smoke at this commit MUST produce
n_trades > 100 + finite metrics. Q2's kernel CBSW immediately follows
and deletes this entire stopgap atomically (field, harness branch,
YAML setting, every literal).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4-task atomic ladder (Q1-Q4) implementing spec 566e8bcb0. Each task
maps to one commit per feedback_no_partial_refactor:
Q1: Cold-start stopgap (bytecode max-confidence policy upload)
Q2: CBSW kernel aggregator + Tier 1 revert (atomic)
Q3: Memory pearl pearl_conviction_bootstrap_for_kelly_aggregation
Q4: Parallelism spec §3.4 cross-reference
Kernel ABI unchanged across all 4 commits — Q2 modifies only
decision_policy_default's body (decision_policy_program left as a
pluggable bytecode-VM experiment surface). Q2 atomically deletes
the Q1 stopgap field/CLI/YAML/harness branch in the same commit
that lands the kernel fix (feedback_no_legacy_aliases compliance:
grep returns zero hits for use_cold_start_stopgap post-Q2).
Validation gates per spec §9:
- Q1: cluster smoke n_trades > 100, total_pnl != 0 (diagnosis
confirmation). If n_trades = 0 still, STOP — downstream bug.
- Q2: pre-existing 11 CUDA tests still pass; 7 new cbsw_*
regression tests pass; cluster smoke n_trades > 100; ev/s
rate ratio Q2/Q1 ≥ 0.95.
Self-review confirms all 11 spec sections covered, no placeholders,
type/signature consistency across tasks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Critical review surfaced 11 actionable issues in the v1 spec; 10 fixed
inline (#11 — legacy-comparison test — dropped per user decision since
empirical legacy behavior is already known: n_trades=0).
Performance:
1. expf → piecewise-linear ramp (~5× cheaper on GPU, no transcendental
in the hot path). Operationally equivalent: monotonic, saturating,
midpoint at K. Side-benefit: pure max-confidence at cold-start
(sq=0 instead of sigmoid's 11.9% leakage).
2. Single fused per-horizon pass replaces the two-loop sketch.
3. Tier 2 scope narrowed to decision_policy_default ONLY. The bytecode
VM (decision_policy_program) stays unchanged — runs only for custom
strategy experiments, never in production policy. Halves Tier 2's
surface area + test cost.
4. __device__ helpers cbsw_signal_quality + cbsw_weight introduced for
single source of truth.
Correctness bugs (silently present in v1 sketch, would have shipped):
5. strong_h and sq_min_active initializers added (were referenced
before init in the per-horizon loop).
6. Attribution mask at cold-start was setting all 5 bits because every
w[h] >= floor > 1e-9; trades would pollute ALL horizons'
recent_sharpe. Fix: binary split — at sq_min_active < 0.5 attribute
only to strong_h; at mature, attribute per weights > floor + epsilon.
7. sq_min was "min over h", which permanently locked the aggregator
into cold-start mode if any horizon never gets attributed (e.g.,
h6000-only-trading regime). Fix: sq_min_active = min over h with
n_trades_seen > 0; cold horizons don't gate maturity.
8. Opposing-horizons case now correctly fires on the strongest single
horizon at cold-start (piecewise-linear sq=0 → pure max-conf), with
explicit design-choice note explaining why this conservatism trade-
off favors firing.
Spec hygiene:
9. Rate validation committed to a measurement gate (Q2 ev/s ≥ 95% of
Q1 ev/s) instead of back-of-envelope estimate.
10. Kernel ABI explicitly stated as unchanged → feedback_no_partial_
refactor compliance is trivial at the kernel boundary.
12. Tier 1's use_cold_start_stopgap field is DROPPED in Q2 atomically
(not orphaned), and DoD checklist includes a grep-zero gate for
feedback_no_legacy_aliases compliance.
Risks updated: removed the sigmoid-narrowing risk (irrelevant now);
added register-pressure risk with the rate-gate mitigation; added
explicit "cold-start may lose on noisy trades" risk with empirical
escalation path (bump kelly_floor 0.20 → 0.40 if Q2 smoke shows large
negative PnL).
Math walk-throughs rewritten for piecewise-linear (different numbers
at cold-start): cold = pure max-conf, mature = pure weighted-sharpe,
clean separation at sq_min_active threshold.
Awaiting review of the revised spec before writing-plans dispatch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The post-trunk-grows threshold-tuning smoke (81decf40f) produced
n_trades=0 despite the model having a HEALTHY max-conviction
distribution (74.6% of decisions ≥ 0.30, 26.7% ≥ 0.70, full spread
across [0,1]). Diagnosis: the linear-weighted-mean aggregator in
decision_policy_default is structurally dilution-bound at cold-start
— single-horizon strong signals get washed out when uniform-floor
weights produce mean-over-horizons aggregation.
Solution: Conviction-Bootstrapped Sharpe Weighting (CBSW). Hybrid
max-confidence × weighted-sharpe with a per-horizon sigmoid transition
keyed on n_trades_seen vs MIN_TRADES_FOR_VAR_CAP. Cold-start: sig_mag
(decision-time conviction) drives weights AND max-confidence aggregator
fires single-horizon trades. Mature: recent_sharpe (historical) drives
weights AND linear-mean aggregator emphasizes strong-Sharpe horizons.
Permanent floor preserved per pearl_blend_formulas_must_have_permanent_floor.
Mirrors DQN bootstrap pattern (pearl_thompson_for_distributional_action_
selection): when historical estimates are uncertain, use available
signal as the bootstrap. Sig_mag is the ISV signal at decision time;
recent_sharpe is the ISV signal at trade-close time. Transition
self-terminates based on data accumulation, not time constants.
3-tier delivery (one spec, atomic commits):
Q1: Bytecode VM stopgap — upload max-confidence 7-instruction
program per backtest. Validates diagnosis; zero kernel work.
Q2: Kernel CBSW — replace weight + aggregator in both decision
kernels. 5 new regression tests covering cold/mature/transition.
Q3: New memory pearl pearl_conviction_bootstrap_for_kelly_aggregation
capturing the lesson.
Q4: Cross-reference from parallelism spec (deployability sweep
depends on CBSW being live to produce meaningful verdict).
The parallelism work (P1-P6) is fully working — confirmed by the
threshold-tuning smoke completing end-to-end (Succeeded status, 500k
decisions, artifacts written, aggregator parquet emitted). What's
blocked is the deployability VERDICT, because the dilution bug means
all variants would show n_trades=0 regardless of cost/latency/threshold.
CBSW unblocks the verdict.
Awaiting review before transitioning to writing-plans for Q1-Q4
implementation plan.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
P4 plumbed the threshold-gate kernel side but deferred the side-channel
that captures observed max_conviction per decision. Wire it now so the
threshold pre-registration step (spec §3.4) can compute the calibrated
p60-p95 absolute threshold values from a real model-on-data run.
Harness changes:
- BacktestHarness gains conviction_log: Vec<f32>. Per decision, computes
max_h |alpha[h] - 0.5| * 2 from the SAME probs that go into broadcast_
alpha (same value the threshold gate would compare against), pushes
to the log. One shared vec — batched cells broadcast the same probs
to every backtest, so per-backtest is redundant.
- write_artifacts emits convictions.bin (raw little-endian f32) +
conviction_percentiles.json with pre-computed p10/p25/p50/p60/p70/
p80/p90/p95/p99 + mean/min/max. Also eprintln-prints the summary
line for at-a-glance log inspection.
Smoke YAML switched to the threshold-tuning configuration: threshold=0
(no gate, full distribution captured), cost=0.125 (1-tick realistic
anchor so the observed Sharpe is the no-gate net-of-cost floor for
the sweep's deployability story).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Legacy fan-out writes <sweep-dir>/<cell>/summary.json (one cell per
Argo task). P6 batched flow writes <sweep-dir>/<cell>/sim_<variant>/
summary.json (one Argo task → run_batched_cell → harness with
variant_names → sim_<name>/ subdirs per spec §3.3).
The aggregator was looking only at <sweep-dir>/<cell>/summary.json,
so the realistic batched smoke completed the actual backtest fine
(2M events, 500k decisions, real artifacts written) but the
end-of-sweep aggregate step errored with "no cell directories with
summary.json".
Walk both layouts: directories containing summary.json directly are
flat cells (legacy); directories one level deeper that contain
summary.json are batched-cell variants. Cell labels become
"<cell>/<variant>" so the aggregate.parquet rows distinguish.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
data_template + cell.window interpolation produces a single-file path,
but MultiHorizonLoader expects a directory (iterates all files inside).
The data_template path is reserved for a future per-quarter loader
filter; for the smoke, fall back to scalar `data` pointing at the
directory + cap max_events at 2M to keep wall under ~20 min.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Frictionless + threshold=0 produces an uninformative number (every
signal trades for free). Swap to realistic anchor:
- latency_ns: 200ms (Scaleway PAR → IBKR baseline)
- cost_per_lot_per_side: 0.125 (1 tick on ES)
- threshold: 0.30 (moderate gate, ~p70-80 of conviction)
Uses the P6 batched flow (sim_variants list with 1 entry) so this run
also exercises the run-sweep Argo template + BatchedSimConfig::from_grid
end-to-end. Output: cell_W0/sim_t30c1l200/{summary.json, trades.csv,
pnl_curve.bin} with REAL net-of-cost Sharpe + max-drawdown.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The cargo-target-cuda PVC persists `$BUILD` across runs. cargo build
mutates Cargo.lock, so the next run's `git checkout $SHA` fails with
"Your local changes to the following files would be overwritten by
checkout: Cargo.lock". Same pattern alpha-perception-template uses
(`git checkout --force $SHA; git clean -fd`) handles this cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the operational gap from P6: when a sweep grid YAML carries
sim_variants, argo-lob-sweep.sh now emits ONE run-sweep-batched task
instead of N fan-out run-cell tasks. The new run-sweep template
invokes `fxt-backtest sweep` against the full grid (base64-encoded
inline as Argo parameter to avoid YAML special-char encoding traps).
The binary's P6 sweep() function handles cell × variant fan-out
internally via BatchedSimConfig::from_grid, so one pod processes all
4 windows × 140 variants sequentially. Trade-off: no inter-window
parallelism in this rev (4 quarters sequential in one pod ≈ firm-bound
2h wall per spec §9). 3-pod scale-out is P7 future work — needs the
script to split the YAML into per-window sub-grids and emit one
run-sweep task per sub-grid.
Backward-compat: legacy (no sim_variants) flow unchanged. The dry-run
of sweep_smoke.yaml continues to emit run-cell-* fan-out tasks; the
dry-run of sweep_deployability.yaml emits one run-sweep-batched task.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
snapshot_realized_pnl / snapshot_position_lots / snapshot_open_horizon_mask
were the host-side read paths for the close-detection loop that P3
(3836e2578) replaced with snapshot_pos_state + detect_close_transitions_
batched kernels. The helpers stayed dead-code-warned after P3; per
feedback_no_legacy_aliases + feedback_no_hiding, delete them.
submit_market_fn is NOT dead — pub fn submit_market wraps it and has 4
test callers (lob_sim_fixtures, lob_sim_fuzz, ring3_replay × 2). Kept.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sweep YAML now supports the batched flow per spec §3.3 + Task 6:
- SweepBase.sim_variants: Vec<SimVariant> — list of (cost, latency,
threshold, ...) variants. When non-empty, each cell runs ONE harness
at n_parallel=variants.len() with BatchedSimConfig::from_grid instead
of the legacy one-harness-per-cell fan-out.
- SweepBase.data_template: Option<String> — when set with `{window}`
placeholder, each cell's `window` field interpolates the per-cell
data path. Replaces single scalar `data` for the windowed flow.
- SweepCell.window: Option<String> — window identifier (e.g., "2025-Q2").
- SimVariant: threshold + cost_per_lot_per_side required (the spec's
primary axes); other fields optional overrides on top of SweepBase
scalars.
New runner pieces:
- BatchedSimConfig::from_grid(&[ResolvedSimVariant]) in
crates/ml-backtesting/src/sim/batched_config.rs.
- ResolvedSimVariant — per-variant fully-resolved sim params.
- resolve_sim_variants(&SweepBase) in main.rs — layers per-variant
overrides over base scalars.
- run_batched_cell() in main.rs — builds the harness with
sim_config_override + variant_names plumbed through. Writes per-
backtest artifacts to sim_<variant_name>/ subdirs (spec §3.3).
Harness side:
- BacktestHarnessConfig gains variant_names + sim_config_override
Option fields. When sim_config_override is Some, harness uses that
directly instead of building from_uniform off scalar cfg. When
variant_names is Some, write_artifacts uses sim_<name>/ instead of
cell_NNNN/ subdirs. Both None preserve legacy single-cell behaviour
(smoke, fixtures unchanged).
YAML configs:
- config/ml/sweep_threshold_tuning.yaml: 1 cell (W0) × 8 sim_variants
(p60-p95 in 5pt steps) with cost=0.125 (1-tick anchor). Threshold
pre-registration pass.
- config/ml/sweep_deployability.yaml: 4 cells (W1-W4) × 140 variants
each (7 costs × 4 latencies × 5 thresholds). Generated by
scripts/generate_sweep_variants.py — placeholder threshold values
(p60-p95) until threshold-tuning publishes calibrated absolutes to
config/ml/v2_prod_thresholds.json.
Deferred to P7 (operational glue):
- argo-lob-sweep.sh adaptation for the batched flow (cells = windows,
not sim-variants; one Argo task per window invokes `fxt-backtest sweep`
end-to-end inside the pod rather than `fxt-backtest run`).
Regression: all 7 existing CUDA tests pass through the new harness
construction path (sim_config_override = None → from_uniform fallback).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
X11's original plan said "capture_graph_a covers full v2 forward" but
the X11 commit (4f888abbf) only shipped forward_only + from_checkpoint.
Graph capture is now actually implemented for the inference path.
Mirrors the pattern already in step_batched (perception.rs:1221-1257):
- First call: eager dispatch + set forward_warmed flag.
- Second call: begin_capture -> dispatch_forward_kernels -> end_capture
-> store CudaGraph.
- Subsequent calls: graph.launch() — captured replay.
forward_only now performs its own staging-fill of the mapped-pinned
host buffers (input data varies per call), then dispatches through
the three-state machine. The captured region is the new private
dispatch_forward_kernels helper: a copy of evaluate_batched's
forward chain (VSN -> Mamba2 x2 -> LN x2 -> attn-pool -> CfC K-loop
-> heads) that omits labels, BCE, and any stream syncs. The final
sync + dtoh of probs_per_k_d happens OUTSIDE capture in forward_only.
Per pearl_no_host_branches_in_captured_graph: no host branches /
scalar-arg-changes / host-mallocs inside the captured region; all
kernel launches use pre-bound device pointers stable across replays.
Per pearl_cudarc_disable_event_tracking_for_graph_capture: event
tracking is already disabled for the trainer's lifetime at
construction (see PerceptionTrainer::new ~line 529), so the captured
region is free of cuStreamWaitEvent / event.record() insertions.
Vestigial loader.rs:272 doc comment referencing the never-shipped
CfcTrunk::capture_graph_a updated to point at the now-real
PerceptionTrainer::forward_only warmup path.
Regression: forward_captured_matches_uncaptured — eager (call 1) vs
captured replay (call 3) agree within 1e-5 relative tolerance per
element. NOT strict bit-identity because CUDA Graph capture can
reorder kernel launches and flip f32 reductions by 1 ULP harmlessly.
Local RTX 3050 Ti run: 160 elements, max rel_err = 0e0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the two sweep axes that the spec's deployability grid needs but
were missing from the kernels:
Threshold gate (decision_policy.cu, both kernels):
- New per-backtest `threshold_per_b` array kernel arg.
- Pre-Kelly prelude: if max_h |alpha[h] - 0.5| * 2 < threshold[b],
emit noop and return. Kept deterministic from alpha alone so the
threshold pre-registration step (p60-p95 absolute calibration on a
validation window, future P6) reflects exactly what gets gated in
deployment.
Per-fill cost integration (resting_orders.cu / apply_fill_to_pos):
- apply_fill_to_pos signature grows three args: b, cost_per_lot_per_side_per_b,
total_fees_per_b. Single insertion point at line 90.
- After the close-leg realized_pnl math runs (so the gross unwind P&L
is preserved), deduct fill_cost = filled_lots * cost_per_lot_per_side[b]
from pos.realized_pnl AND accumulate into total_fees_per_b[b].
- Net-of-cost semantics: isv_kelly_update_on_close reads realized_pnl
delta which is now net of cost — Kelly state learns from realistic
return distribution.
- All 3 apply_fill_to_pos call sites in step_resting_orders updated.
order_match.cu's submit_market_immediate path is dead code in the
post-P1 flow (everything routes through seed_inflight_limits_batched
→ step_resting_orders → apply_fill_to_pos) so not touched here.
BatchedSimConfig + UniformSimParams + BacktestHarnessConfig gain
threshold + cost_per_lot_per_side fields. All UniformSimParams
constructors in tests and main.rs updated with defaults (0.0, 0.0 =
gate disabled, frictionless).
Regression:
- threshold_gate_skips_low_conviction (p=0.51 + threshold=0.10 → noop)
- threshold_gate_allows_high_conviction (p=0.8 + threshold=0.10 → buy 1+)
- threshold_zero_is_passthrough (sanity)
- All P1+P2+P3 tests continue to pass via the new ABI.
cost_deducted_at_each_fill + kelly_state_sees_net_return end-to-end
tests deferred — they require a full submit_market → fill → close
sequence, which the production smoke exercises.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces TWO host loops in step_decision_with_latency with two GPU kernels:
1. snapshot_pos_state — replaces the host-side snapshot_realized_pnl +
snapshot_position_lots + snapshot_open_horizon_mask trio (3 separate
memcpy_dtoh per decision). Now one kernel launch writes
prev_pos_lots_d / prev_realized_pnl_d / prev_open_horizon_mask_d
directly on the device.
2. detect_close_transitions_batched — replaces the host close-detect
loop that called read_pos per close-eligible backtest (up to
n_backtests memcpy_dtoh per decision). Now one kernel writes
closed_horizon_mask_d + realised_return_d on the device, and
isv_kelly_update_on_close consumes them with no host roundtrip.
At n_parallel=140 these two loops together accounted for ~350M+ small
host roundtrips per quarter. Combined with P2 the latency-path of
step_decision_with_latency is now fully GPU-resident.
isv_kelly_update_on_close kernel always launches (skips backtests with
mask=0 internally) rather than gating via a host any_close check.
All P1+P2 regression tests pass through the new GPU close-detect path
(at n=1 with uniform config + immediate-fill latency, no close happens
in the cold-start tests since they only check market_target; the close
path is exercised indirectly).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the host roundtrip loop at the latency path of step_decision_
with_latency with one GPU kernel launch. At n_parallel=140 the host
loop did up to 140 memcpy_dtoh + 140 seed_limit_order calls per
decision (~210M roundtrips per quarter at the threshold-tuning load).
The new kernel does the same work in one launch.
Per-backtest single-writer (threadIdx.x==0). Each backtest scans its
own MAX_LIMITS=32 slot range for an `active==0` slot. Slot allocation
is per-backtest (no cross-backtest atomics needed). Overflow path
increments pos.submission_overflow.
dispatch_latent_market_orders now takes &BatchedSimConfig (unused
inside — the latency_ns_d device buffer is already populated by
step_decision_with_latency's upload block from P1).
All 3 decision_floor_coldstart tests still pass via the new GPU path
(at latency_ns=0 the in-flight slot's arrival_ts==current_ts and gets
promoted on the next step_resting_orders, functionally equivalent to
the legacy submit_market_immediate kernel).
Independence test (different per-backtest latencies → different
arrival_ts) deferred to a later step where a read_first_inflight_arrival_ts
helper is added.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Migrates target_annual_vol_units, annualisation_factor, max_lots,
latency_ns, kelly_frac_floor, sharpe_weight_floor from scalar-broadcast
kernel args to per-backtest device arrays via BatchedSimConfig. Atomic
contract change per feedback_no_partial_refactor — kernel + sim + harness
+ all 3 existing test files migrate in this commit.
- crates/ml-backtesting/src/sim.rs → sim/mod.rs (directory module)
- crates/ml-backtesting/src/sim/batched_config.rs (NEW): BatchedSimConfig
+ UniformSimParams + validate(). from_uniform rebuilds the legacy
uniform-broadcast behaviour at n=1 (smoke/fixtures). from_grid lands
in P6 for the 140-variant sweep packing.
- LobSimCuda gains 6 per-backtest device buffers (target_annual_vol_units_d,
annualisation_factor_d, max_lots_d, latency_ns_d, kelly_frac_floor_d,
sharpe_weight_floor_d). step_decision_with_latency uploads from
BatchedSimConfig each call; both decision kernel launches now pass
per-backtest array pointers.
- decision_policy_default + decision_policy_program: scalar args become
const float* / const int* per_b arrays; first lines of each kernel
index by `b` into the arrays. Behaviour preserved at n=1 uniform.
- dispatch_latent_market_orders: reads latency per-backtest from
&BatchedSimConfig (host loop stays for P1; P2 replaces with kernel).
- step_decision_with_latency now ALWAYS dispatches through the latency
path; when cfg.latency_ns[b]=0 the in-flight slot's arrival_ts equals
current_ts and gets promoted immediately on next snapshot. Eliminates
the if/else branch and consolidates the launch path.
- harness.rs: BacktestHarness gains a sim_config field, built via
BatchedSimConfig::from_uniform at new() from the harness cfg's scalar
fields. The run loop passes &self.sim_config to step_decision_with_latency.
Regression coverage:
- parallel_sim_correctness::parallel_sim_equivalence_with_uniform_config
— n=8 with uniform config produces 8 bit-identical market_targets
(proves per-backtest indexing reduces correctly).
- Existing decision_floor_coldstart tests (3) all pass through the new
ABI — proves cold-start floor + variance-cap gate behaviour preserved.
- parallel_sim_independence_per_backtest deferred to P2 (needs
read_first_inflight_arrival_ts helper that depends on LimitSlot layout).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Critical review surfaced 9 issues; all fixed inline:
1. §1 — Reframed "140× amortization" as "amortization on forward; sim
runs parallel". True win is on the ~2ms forward shared across 140
cells, not on sim work which scales linearly with n_backtests.
2. §2 — Made the 1h target vs firm bound explicit (≤1h target, ≤2h
firm). Acknowledges Graph capture realistic speedup is 1.2-1.5×,
not 2×.
3. §5 — Dropped atomicAdd. Plain `+=` under the single-writer-per-block
convention (existing pattern across sim kernels). No race.
4. §4 — Documented max-over-horizons threshold rationale (vs per-
horizon or aggregate-conviction). Flagged per-horizon as a follow-up
tweak if dilution pathway matters in the verdict.
5. §7 P5 + §10 risk — Captured-vs-uncaptured tolerance is 1e-5 relative,
NOT strict bit-identity. CUDA Graph capture can reorder reductions
harmlessly by 1 ULP; strict bit-identity would be a false-positive.
6. §8 — Made explicit that parallel_sim_equivalence + independence
tests are BOTH required. Equivalence alone is necessary but not
sufficient (a shadow-backtest[0] bug still passes equivalence).
7. §3.3 — Specified output schema: `cell_W{n}/sim_<variant_name>/`
with summary.json carrying a resolved `sim_config` block (verdict
emitter reads that, not the directory name).
8. §7 P4 — Enumerated apply_fill_to_pos call sites + added grep-verify
step before commit, so no fill path silently loses cost.
9. §9 — Tightened rate-validation gates with hard targets (P2 ≤90s,
P5 ≤60s) instead of generous minute envelopes. Added §9.1 stride=8
fallback as explicit Plan-B if Graph capture under-delivers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brainstormed live with claude-opus-4-7. Targets ≤1h wall-clock per
4-quarter sweep on 1-3 L40S pods (vs ~750 GPU-hours naive). Three
multiplicative levers: per-backtest sim parameter matrix (~140×),
CUDA Graph capture (~2×), pod-level parallelism (~3×). Verified by
code-read during scoping:
- forward_only has no graph capture (X11 plan said so, X11 commit
didn't ship it)
- two host-roundtrip loops in sim.rs need GPU-ization for batching
to pay off (dispatch_latent_market_orders + close-detection)
- cost system is a literal-zero placeholder in pnl_track.cu:92
7-commit atomic ladder (P1-P7), each gated by tests + a measured rate
target. bf16 explicitly deferred to follow-up.
Awaiting review before transitioning to writing-plans for the
implementation plan.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After widening the smoke to max_events=0 (full quarter), the inherited
decision_stride=1 became 10M+ forward passes — hours of GPU per cell,
indistinguishable from a deadlock in pod logs.
Two operational fixes:
- sweep_smoke.yaml: decision_stride 1 -> 4 (matches the sweep
template's own default).
- harness.rs: emit a `progress: events=... decisions=... elapsed=...
rate=...ev/s` line every 1M events on stderr so multi-million-event
cells are observable from kubectl logs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three follow-ups to the cold-start floor fix:
1. Kernel: MIN_TRADES_FOR_VAR_CAP gate. After the first trade closed,
`isv_kelly_update_on_close` set `realised_return_var = ret²` — a
single-sample variance proxy that systematically collapses
`cap_units = target_vol / sqrt(var × ann_factor)` near zero for
any biased return. cap_lots → 0 → no further trades despite strong
alpha. Gate the variance-derived cap behind `n_trades_seen >= 10`;
below the threshold cap falls back to host-supplied `max_lots`,
same as the pre-first-trade path. Same gate applied to both
`decision_policy_default` and `decision_policy_program`.
Regression: `post_first_loss_state_does_not_lock_out_further_trades`
reproduces the exact pre-fix state from the smoke (n_trades_seen=1,
var=103.6) and asserts the kernel still fires a long with p=0.8.
2. Aggregate: add `nvidia.com/gpu: 1` resource request. Scaleway's
L40S device plugin mounts libcuda.so.1 into the container only on
GPU-requesting pods; the aggregate logic is CPU-only but the
binary's dynamic loader needs the driver libs. Cheapest correct
fix until a separate CPU-only aggregator binary exists.
3. Smoke YAML: `max_events: 0` (exhaust loader). 100k events is
minutes of ES.FUT, far shorter than the h6000 holding horizon the
model was trained on. Full quarter exercises sustained trading +
variance estimate ramp-up.
All three regression tests pass locally on RTX 3050 Ti.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Production smoke completed end-to-end but produced n_trades=0 across 99,969
decisions — `decision_policy_default` and `decision_policy_program` both
applied a sentinel-skip pattern: if `isv_kelly_d` had not been seeded
(pnl_ema_win == 0), each horizon's signed-size stayed zero, AND each
horizon's aggregation weight (= recent_sharpe) also stayed zero. The
cross-horizon w_sum was therefore 0, final_size was 0, every market_target
was noop. State only updates on trade close → no trade ever fires →
infinite cold-start.
Per pearl_blend_formulas_must_have_permanent_floor (`max(real, floor)`,
not blend) and pearl_kelly_cap_signal_driven_floors, replace the sentinel-
skip with a two-layer floor on each kernel:
1. Kelly fraction: `max(kelly_frac_floor, computed_kelly)` — when state
is sentinel, falls back to the floor directly. Cap_lots falls back
to `max_lots` when realised_return_var is sentinel.
2. Aggregation weight: `max(sharpe_weight_floor, recent_sharpe)` — lets
cross-horizon sum produce a non-zero size before recent_sharpe is
populated. Once a horizon shows positive sharpe it dominates.
Plumbed through `step_decision_with_latency` / `step_decision` as two
new f32 args (atomic contract change, every caller migrated). Defaults
0.20 / 0.10 chosen so a strong-conviction signal (sig_mag ≥ 0.5) fires
1 lot at cold-start under max_lots=5 while weaker signals stay flat
(see `default_kelly_frac_floor` comment for the arithmetic). Exposed
as CLI flags + sweep-grid base/cell overrides.
Regression test `decision_floor_coldstart` proves:
- default floors (0.20/0.10) fire a 1-lot buy with p_h=0.8 and zero state
- zero floors reproduce the original noop bug
Also moves `aggregate` step to the GPU pool because fxt-backtest is
dynamically linked against libcuda.so.1 (the ci-compile-cpu hosts
don't expose CUDA driver libs).
Verified locally on RTX 3050 Ti — workspace cargo check passes, both
regression tests pass, trunk save/load roundtrip still passes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
58b5ebbd3 may pre-date PerceptionTrainer's Mamba2-into-trunk move
(timestamp 2026-05-19 08:32, during the refactor window). dbd500ecf
(10:59) was trained after every X-migration landed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Production smoke panicked at trunk.rs:mamba2_l1() during
PerceptionTrainer::from_checkpoint. Root cause: load_checkpoint
called new_random which left mamba2_stack_1/2 = None, then tried
to upload weights into None blocks.
PerceptionTrainer::new was the only caller that populated the
Mamba2 stacks (X3/X5 migrations stopped at the accessor methods
but never moved construction). Inference paths that didn't go
through PerceptionTrainer::new — exactly what fxt-backtest does
via from_checkpoint → load_checkpoint — hit the gap.
Move Mamba2Block::new + state allocation into CfcTrunk::new_random.
PerceptionTrainer::new now sets up its optimizer + scratches against
the trunk-owned blocks via existing accessors (mamba2_l1/mamba2_l2).
Extends the trunk roundtrip test to:
- assert mamba2 weight slices are non-empty (catches empty-buffer
regressions that would let bit-equivalence pass trivially)
- bit-equivalence-check Mamba2 stack 1 + stack 2 weights through
save -> load, reproducing the exact failure path that hit prod.
Verified locally on RTX 3050 Ti — full workspace cargo check passes,
test passes with non-empty weight slices.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Template was missing feature-cache PVC mount on run-cell; trained
checkpoints + predecoded data live at /feature-cache/* on the
feature-cache-pvc claim. Mount it the same way alpha-perception does.
- sweep_smoke.yaml referenced /data/* (no such mount) and a 40-char
SHA path for the checkpoint dir. alpha-perception-runs subdirs are
9-char short SHAs; data path is /mnt/training-data/* (matches mount).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Working alpha-perception-template references
`foxhunt/foxhunt-training-runtime:latest`. Our lob-backtest-sweep
template stripped the `foxhunt-` prefix and got 404 from the registry
on run-cell + aggregate steps.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CARGO_TARGET_DIR=/cargo-target redirects all build output away from
the in-tree `<crate>/target/` path, so `cp $BUILD/target/release/...`
fails after a successful compile. Mirror alpha-perception-template's
correct pattern: `cp $CARGO_TARGET_DIR/release/...`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pods labeled `component: backtest` only match the `argo-base-egress`
NetworkPolicy (DNS, 443, MinIO, Mattermost). The ensure-binary step
needs SSH-to-gitlab-shell on port 2222, which is gated by
`argo-train-workflow` (selects `component=train`).
Relabel to `train` so the same NetworkPolicy that lets alpha-perception
clone the repo also covers lob-backtest-sweep. Egress shape (compile,
GPU run, PVC writes) is identical to a training run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The template wrote ~/.ssh/config but didn't chmod it. OpenSSH refused
the file with 'Bad owner or permissions on /root/.ssh/config' and the
git clone failed with 'exit status 128' before fxt-backtest could
compile.
Same one-line fix that alpha-perception-template.yaml has had since
forever. Verified: lob-backtest-sweep-<new> ensure-binary now passes
the SSH check and starts cargo build.
Matches the alpha-perception-template (which is the known-good
template in this namespace). Without the fix, ensure-binary +
run-cell + aggregate pods all stayed Init for ~20m with
'MountVolume.SetUp failed for volume "git-ssh-key" : secret
"git-ssh-key" not found' before the workflow timed out.
The actual secret in the foxhunt namespace is named
'argo-git-ssh-key' (see kubectl get secrets -n foxhunt). Same
template misconfig as the training-data PVC name.
The template referenced PVC 'training-data' but the actual claim in
the foxhunt namespace is 'training-data-pvc' (matches the convention
used by other workflows like alpha-perception-template). Without this
fix, ensure-binary + run-cell + aggregate pods all stayed Pending
forever with 'persistentvolumeclaim "training-data" not found',
blocking every smoke / threshold-tuning / deployability sweep.
Verified: smoke sweep lob-backtest-sweep-jp48v scheduled ensure-binary
onto the ci-compile-cpu pool (autoscaler triggered scale-up 0->1
within seconds of resubmission).
argo-lob-sweep.sh's awk substitution for # __SWEEP_CELLS__ matched BOTH
the actual marker (line 110, whitespace-indented) AND the docstring
reference at the top of the template that mentions `# __SWEEP_CELLS__`
inside a sentence (line 8). Result: cell task was injected twice,
once before `apiVersion:` (breaking kubectl apply with 'invalid object'
validation error) and once at the correct DAG location.
Fix: anchor the match to lines that START with whitespace + the marker
(^[[:space:]]+# __SWEEP_CELLS__), so the docstring sentence (which has
'# The' first) no longer matches.
sweep_smoke.yaml: rewrite from the spec's idealised (axes:/windows:)
schema to the actual base:/cells: schema that fxt-backtest sweep +
argo-lob-sweep.sh parse. Hardcodes SHA 58b5ebbd3 (the production
training run); bounds smoke runtime via max_events=100000.
Verified: smoke sweep workflow lob-backtest-sweep-hxwmq submits cleanly
and starts running.
Adds `fxt-backtest verdict <sweep_dir> --threshold X --windows W1,W2,W3,W4
--training-sha SHA --spec-sha SHA --out deployability_verdict.json`
that reads per-cell summary.json files at the realistic (1 tick, 200ms)
+ stress (1.5 tick, 400ms) anchors against the pre-registered threshold,
computes median Sharpe/max_dd/Sortino/profit_factor across windows,
classifies into Pass-robust / Pass-nominal / Fail-inconclusive / Fail /
Fail-degenerate per spec §3.5, and writes the audit JSON.
Adds serde_json workspace dep to fxt-backtest's Cargo.toml (was already
a transitive dep but not declared at this layer).
End-to-end CLI for Phase 2 runtime is now: argo-train.sh → argo-lob-sweep.sh
(smoke / threshold-tuning / deployability) → fxt-backtest aggregate →
fxt-backtest verdict → commit deployability_verdict.json.
Both tests exercised CfcTrunk::capture_graph_a + perception_forward_captured
+ snapshot_hidden, which feed V1-shaped CfC weights. After X8 the trunk's
CfC was reshaped to v2 layout (cfc_n_in=HIDDEN_DIM); the V1 forward path
now feeds FEATURE_DIM input into HIDDEN_DIM-shaped CfC — runtime garbage.
The methods themselves are still in trunk.rs (marked dead-code by rustc).
A deeper cleanup pass — deleting the V1 weight fields (heads_w_d, proj_*),
V1 per-step scratches, V1 cubin function handles, and the V1 forward
methods themselves — is a follow-up commit when fresh.
Verification: ml-alpha + ml-backtesting + fxt-backtest all build clean.
BacktestHarness now owns a PerceptionTrainer (in inference role) instead
of a raw CfcTrunk. The sliding K-window of recent snapshots accumulates
in the harness; at each decision-stride boundary (and only once the
window has reached cfg.seq_len), the harness calls
trainer.forward_only(&window) and broadcasts the last K position's
per-horizon probs to the LobSim.
fxt-backtest's main.rs constructs the trainer via
PerceptionTrainer::from_checkpoint when --checkpoint is supplied (else
random init for noise baseline).
Why this shape: PerceptionTrainer's evaluate_batched already runs the
full inference chain (snap → vsn → mamba2 → ln → mamba2 → ln →
attn_pool → cfc K-loop → grn heads) correctly. Duplicating that 400-line
forward chain on CfcTrunk would double the surface area for the same
result — the trunk's role is weight-source-of-truth (achieved in X1-X9),
not kernel-launch orchestration.
End-to-end status: alpha_train emits Checkpoint files via X14 wiring;
fxt-backtest now loads those Checkpoints via from_checkpoint and drives
forward via forward_only. Phase 2 (Argo runtime: training → smoke →
threshold pre-reg → 560-cell deployability sweep → verdict) is unblocked.
Adds PerceptionTrainer::config() accessor so the harness can read seq_len.
Verification: ml-alpha + ml-backtesting + fxt-backtest all build clean.
X11 inference interface for the deployability backtester. Two new
public methods on PerceptionTrainer:
forward_only(snapshots) -> Vec<f32>
Forward-only pass over a K-snapshot window. Wraps evaluate() with
dummy zero-valued labels and discards the loss. Returns the same
[K, B, N_HORIZONS]-shaped probability output as evaluate_batched.
from_checkpoint(dev, cfg, path) -> Result<Self>
Build a PerceptionTrainer ready for inference: constructs a fresh
trainer (with random init) to wire up kernel handles + grad
buffers + scratches, then overwrites the trunk's weights from the
Checkpoint file via CfcTrunk::load_checkpoint. The optimizer +
grad buffers stay allocated — unused at inference, but allocating
them keeps the struct invariant uniform. A leaner inference-only
struct can be added later if memory matters.
Architectural note: the trunk is the source of truth for weights (X1-X9
established that). PerceptionTrainer is the kernel-launch adapter that
drives forward + backward over those weights. Inference doesn't need a
separate forward path on the trunk — the trainer's evaluate_batched
already does the full chain correctly. fxt-backtest can now construct a
PerceptionTrainer in inference role via from_checkpoint + call
forward_only per decision.
Verification: ml-alpha + ml-backtesting + fxt-backtest build clean.
ml-alpha lib tests: 33 pass.
Per project_ml_alpha_starting_capital greenfield posture: there is no V1
to differentiate from (the V1 trunk forward was dead code, no V1
checkpoint files exist in the wild). The 'v2' prefix on every identifier
was historical baggage from the migration period.
Renames:
- CheckpointV2 -> Checkpoint (also drops the version: u32 field —
bincode either deserialises a current envelope or errors; no migration
path needed)
- CheckpointVersionProbe removed (was only for V1 rejection)
- LAYER_NORM_CUBIN_V2 / VARIABLE_SELECTION_CUBIN_V2 / ATTENTION_POOL_CUBIN_V2
-> LAYER_NORM_CUBIN / VARIABLE_SELECTION_CUBIN / ATTENTION_POOL_CUBIN
- _ln_module_v2 / _vsn_module_v2 / _attn_module_v2 -> drop _v2 suffix
- smoke_load_v2_checkpoint test -> smoke_load_checkpoint
- config/ml/sweep_v2_*.yaml -> config/ml/sweep_*.yaml
- migration-era 'V2 weight skeleton' / 'V2 fields' / etc. comments
cleaned to remove the v2 prefix
Pre-existing 'v2' references in ml-backtesting CUDA files
(decision_policy.cu, pnl_track.cu) are NOT touched — those refer to
future planned 'v2' refinements (Portfolio mode, multi-fill averaging)
from the C1-C19 commits and reflect aspirational features unrelated to
this session's trunk-grows work.
Verification: ml-alpha + ml-backtesting + fxt-backtest all build clean.
perception_forward_golden bit-exact (max_diff = 0.000000).
