dcffb11f5de271e2996b76fe3eb87bf7d03ee9bf
11 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
jgrusewski
|
fef5939556 |
feat(ml-backtesting): cold-start stopgap — max-confidence bytecode policy (Q1/Tier1)
The threshold-tuning smoke at
|
||
|
jgrusewski
|
9d4fda36ab |
feat(ml-backtesting): batched-cell sweep schema + 140-variant runner (P6)
Sweep YAML now supports the batched flow per spec §3.3 + Task 6:
- SweepBase.sim_variants: Vec<SimVariant> — list of (cost, latency,
threshold, ...) variants. When non-empty, each cell runs ONE harness
at n_parallel=variants.len() with BatchedSimConfig::from_grid instead
of the legacy one-harness-per-cell fan-out.
- SweepBase.data_template: Option<String> — when set with `{window}`
placeholder, each cell's `window` field interpolates the per-cell
data path. Replaces single scalar `data` for the windowed flow.
- SweepCell.window: Option<String> — window identifier (e.g., "2025-Q2").
- SimVariant: threshold + cost_per_lot_per_side required (the spec's
primary axes); other fields optional overrides on top of SweepBase
scalars.
New runner pieces:
- BatchedSimConfig::from_grid(&[ResolvedSimVariant]) in
crates/ml-backtesting/src/sim/batched_config.rs.
- ResolvedSimVariant — per-variant fully-resolved sim params.
- resolve_sim_variants(&SweepBase) in main.rs — layers per-variant
overrides over base scalars.
- run_batched_cell() in main.rs — builds the harness with
sim_config_override + variant_names plumbed through. Writes per-
backtest artifacts to sim_<variant_name>/ subdirs (spec §3.3).
Harness side:
- BacktestHarnessConfig gains variant_names + sim_config_override
Option fields. When sim_config_override is Some, harness uses that
directly instead of building from_uniform off scalar cfg. When
variant_names is Some, write_artifacts uses sim_<name>/ instead of
cell_NNNN/ subdirs. Both None preserve legacy single-cell behaviour
(smoke, fixtures unchanged).
YAML configs:
- config/ml/sweep_threshold_tuning.yaml: 1 cell (W0) × 8 sim_variants
(p60-p95 in 5pt steps) with cost=0.125 (1-tick anchor). Threshold
pre-registration pass.
- config/ml/sweep_deployability.yaml: 4 cells (W1-W4) × 140 variants
each (7 costs × 4 latencies × 5 thresholds). Generated by
scripts/generate_sweep_variants.py — placeholder threshold values
(p60-p95) until threshold-tuning publishes calibrated absolutes to
config/ml/v2_prod_thresholds.json.
Deferred to P7 (operational glue):
- argo-lob-sweep.sh adaptation for the batched flow (cells = windows,
not sim-variants; one Argo task per window invokes `fxt-backtest sweep`
end-to-end inside the pod rather than `fxt-backtest run`).
Regression: all 7 existing CUDA tests pass through the new harness
construction path (sim_config_override = None → from_uniform fallback).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
jgrusewski
|
cd82f9a4a0 |
feat(ml-backtesting): threshold gate + per-fill cost integration (P4)
Adds the two sweep axes that the spec's deployability grid needs but were missing from the kernels: Threshold gate (decision_policy.cu, both kernels): - New per-backtest `threshold_per_b` array kernel arg. - Pre-Kelly prelude: if max_h |alpha[h] - 0.5| * 2 < threshold[b], emit noop and return. Kept deterministic from alpha alone so the threshold pre-registration step (p60-p95 absolute calibration on a validation window, future P6) reflects exactly what gets gated in deployment. Per-fill cost integration (resting_orders.cu / apply_fill_to_pos): - apply_fill_to_pos signature grows three args: b, cost_per_lot_per_side_per_b, total_fees_per_b. Single insertion point at line 90. - After the close-leg realized_pnl math runs (so the gross unwind P&L is preserved), deduct fill_cost = filled_lots * cost_per_lot_per_side[b] from pos.realized_pnl AND accumulate into total_fees_per_b[b]. - Net-of-cost semantics: isv_kelly_update_on_close reads realized_pnl delta which is now net of cost — Kelly state learns from realistic return distribution. - All 3 apply_fill_to_pos call sites in step_resting_orders updated. order_match.cu's submit_market_immediate path is dead code in the post-P1 flow (everything routes through seed_inflight_limits_batched → step_resting_orders → apply_fill_to_pos) so not touched here. BatchedSimConfig + UniformSimParams + BacktestHarnessConfig gain threshold + cost_per_lot_per_side fields. All UniformSimParams constructors in tests and main.rs updated with defaults (0.0, 0.0 = gate disabled, frictionless). Regression: - threshold_gate_skips_low_conviction (p=0.51 + threshold=0.10 → noop) - threshold_gate_allows_high_conviction (p=0.8 + threshold=0.10 → buy 1+) - threshold_zero_is_passthrough (sanity) - All P1+P2+P3 tests continue to pass via the new ABI. cost_deducted_at_each_fill + kelly_state_sees_net_return end-to-end tests deferred — they require a full submit_market → fill → close sequence, which the production smoke exercises. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
jgrusewski
|
da21feb1b1 |
fix(ml-backtesting): cold-start Kelly + Sharpe-weight floors in decision kernel
Production smoke completed end-to-end but produced n_trades=0 across 99,969 decisions — `decision_policy_default` and `decision_policy_program` both applied a sentinel-skip pattern: if `isv_kelly_d` had not been seeded (pnl_ema_win == 0), each horizon's signed-size stayed zero, AND each horizon's aggregation weight (= recent_sharpe) also stayed zero. The cross-horizon w_sum was therefore 0, final_size was 0, every market_target was noop. State only updates on trade close → no trade ever fires → infinite cold-start. Per pearl_blend_formulas_must_have_permanent_floor (`max(real, floor)`, not blend) and pearl_kelly_cap_signal_driven_floors, replace the sentinel- skip with a two-layer floor on each kernel: 1. Kelly fraction: `max(kelly_frac_floor, computed_kelly)` — when state is sentinel, falls back to the floor directly. Cap_lots falls back to `max_lots` when realised_return_var is sentinel. 2. Aggregation weight: `max(sharpe_weight_floor, recent_sharpe)` — lets cross-horizon sum produce a non-zero size before recent_sharpe is populated. Once a horizon shows positive sharpe it dominates. Plumbed through `step_decision_with_latency` / `step_decision` as two new f32 args (atomic contract change, every caller migrated). Defaults 0.20 / 0.10 chosen so a strong-conviction signal (sig_mag ≥ 0.5) fires 1 lot at cold-start under max_lots=5 while weaker signals stay flat (see `default_kelly_frac_floor` comment for the arithmetic). Exposed as CLI flags + sweep-grid base/cell overrides. Regression test `decision_floor_coldstart` proves: - default floors (0.20/0.10) fire a 1-lot buy with p_h=0.8 and zero state - zero floors reproduce the original noop bug Also moves `aggregate` step to the GPU pool because fxt-backtest is dynamically linked against libcuda.so.1 (the ci-compile-cpu hosts don't expose CUDA driver libs). Verified locally on RTX 3050 Ti — workspace cargo check passes, both regression tests pass, trunk save/load roundtrip still passes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
jgrusewski
|
58b5ebbd38 |
feat(fxt-backtest): verdict subcommand wraps emit_deployability_verdict
Adds `fxt-backtest verdict <sweep_dir> --threshold X --windows W1,W2,W3,W4 --training-sha SHA --spec-sha SHA --out deployability_verdict.json` that reads per-cell summary.json files at the realistic (1 tick, 200ms) + stress (1.5 tick, 400ms) anchors against the pre-registered threshold, computes median Sharpe/max_dd/Sortino/profit_factor across windows, classifies into Pass-robust / Pass-nominal / Fail-inconclusive / Fail / Fail-degenerate per spec §3.5, and writes the audit JSON. Adds serde_json workspace dep to fxt-backtest's Cargo.toml (was already a transitive dep but not declared at this layer). End-to-end CLI for Phase 2 runtime is now: argo-train.sh → argo-lob-sweep.sh (smoke / threshold-tuning / deployability) → fxt-backtest aggregate → fxt-backtest verdict → commit deployability_verdict.json. |
||
|
jgrusewski
|
395e0d3000 |
refactor(ml-backtesting): drive forward via PerceptionTrainer.forward_only
BacktestHarness now owns a PerceptionTrainer (in inference role) instead of a raw CfcTrunk. The sliding K-window of recent snapshots accumulates in the harness; at each decision-stride boundary (and only once the window has reached cfg.seq_len), the harness calls trainer.forward_only(&window) and broadcasts the last K position's per-horizon probs to the LobSim. fxt-backtest's main.rs constructs the trainer via PerceptionTrainer::from_checkpoint when --checkpoint is supplied (else random init for noise baseline). Why this shape: PerceptionTrainer's evaluate_batched already runs the full inference chain (snap → vsn → mamba2 → ln → mamba2 → ln → attn_pool → cfc K-loop → grn heads) correctly. Duplicating that 400-line forward chain on CfcTrunk would double the surface area for the same result — the trunk's role is weight-source-of-truth (achieved in X1-X9), not kernel-launch orchestration. End-to-end status: alpha_train emits Checkpoint files via X14 wiring; fxt-backtest now loads those Checkpoints via from_checkpoint and drives forward via forward_only. Phase 2 (Argo runtime: training → smoke → threshold pre-reg → 560-cell deployability sweep → verdict) is unblocked. Adds PerceptionTrainer::config() accessor so the harness can read seq_len. Verification: ml-alpha + ml-backtesting + fxt-backtest all build clean. |
||
|
jgrusewski
|
f9b57f4c18 |
feat(fxt-backtest): sweep subcommand + example grid YAML (C15)
New `fxt-backtest sweep --grid <yaml> --out <dir>` subcommand: iterates
over a grid of Run configs, writes each cell's artifacts to
<out>/<cell_name>/, then automatically invokes the existing aggregate
path to produce aggregate.parquet + pareto_frontier.json at the root.
All cells run sequentially on the same GPU (single-machine). For
cluster fan-out the underlying mechanism is the same — Argo can wrap
this binary in a workflow that runs each cell as a separate pod
(left as infra-side work for a follow-up commit; the binary's
contract is the same).
Sweep grid YAML schema:
base: # defaults applied to every cell unless overridden
data: ...
n_parallel: ...
decision_stride: ...
latency_ns: ...
target_annual_vol_units: ...
annualisation_factor: ...
max_lots: ...
max_events: ...
seed: ...
checkpoint: ... # optional — load real trained weights
cells:
- name: cell_a
decision_stride: 1 # override base
- name: cell_b
latency_ns: 250000000
...
Each SweepCell may override any subset of fields; unset fields fall
back to base defaults. --max-cells gates the run for smoke-testing
large grids.
Adds an example grid at config/ml/sweep_decision_stride_example.yaml
that re-runs the decision_stride ∈ {1, 2, 4, 8} sweep deferred from
the original plan (task #202).
Closes #201 (sweep tool). #202 (first decision_stride sweep) is now
executable end-to-end via the example grid.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
3fa215ad2e |
feat(ml-alpha): CfcTrunk save/load checkpoint + --checkpoint CLI (C14)
CfcTrunk::save_checkpoint(path) reads each device weight tensor back
via memcpy_dtoh and bincode-serialises into a CheckpointV1 envelope:
{ version, n_in, n_hid,
w_in, w_rec, b, tau,
heads_w, heads_b,
proj_w, proj_b, proj_g, proj_n }
Total ~22k-25k f32 = ~90 KB per trunk. Tiny.
CfcTrunk::load_checkpoint(dev, cfg, path) deserialises + validates
(version == 1, n_in/n_hid match the supplied CfcConfig — a model
trained for one arch can't silently load against another). Constructs
a fresh trunk via new_random (for kernel bindings + scratch buffers)
then overwrites every weight tensor via memcpy_htod. The random init
values are thrown away — marginally wasteful, but keeps the
construction code paths unified.
Roundtrip test (--ignored, CUDA-required): save trunk_A → load → read
back every device tensor and assert bit-equality between trunk_A and
the loaded trunk_B. Passed locally. Dim-mismatch rejection test runs
without CUDA (verifies bincode envelope serialise/deserialise).
bin/fxt-backtest --checkpoint <path>: when set, overrides --seed and
loads from disk. When absent, warns loudly that the trunk is
random-initialised and backtest results are noise. This makes the
binary genuinely useful as a deployment tool — point it at a trained
checkpoint and run real backtests.
Adds bincode workspace dep to ml-alpha (was already in workspace
dependencies, just not in ml-alpha's [dependencies] block). serde
features bumped to ["derive"] (was using workspace default which
omits derive macros).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
ed19985c95 |
feat(ml-backtesting): bytecode VM dispatcher for Strategy compositions (C13)
The hardcoded WeightedByRealizedSharpe path from C7 is now joined by
a stack-based bytecode interpreter that consumes Strategy::flatten()
output, unlocking RegimeSwitch / Portfolio / non-default Ensemble /
single-Leaf compositions specified via policy-grid YAML.
cuda/decision_policy.cu — new kernel `decision_policy_program`:
Stack-based VM with parallel (value, attribution_mask) stacks.
Opcodes mirror src/policy/mod.rs::OpCode exactly:
NoOp / PushScalar / EvalRegime / BranchIfRegime
EmitPerHorizonSize (computes sized intent from alpha[h] +
IsvKellyState[h] using same Kelly + ISV-cap formula as the
hardcoded default)
AggMean / AggWeightedSharpe / AggMaxConfidence (pop n values,
push aggregated, OR attribution masks)
ApplyConflict (v1 no-op, reserved for Portfolio)
WriteOrder (terminal — converts top-of-stack to market_target +
open_horizon_masks attribution if currently flat)
AggWeightedSharpe recovers the source horizon from a single-bit
attribution mask to look up recent_sharpe; multi-bit masks (nested
aggregators that collapsed horizons) fall back to uniform weight.
decision_policy_default extended with a program_lens param: skips any
backtest whose plen > 0 (the program kernel handled it). The two
kernels run sequentially in step_decision_with_latency with mutual
exclusivity on each backtest slot.
LobSimCuda gains:
upload_program(b, &Program) — uploads a Strategy::flatten() output
to backtest b's slot in program_table_d, updates program_lens_d.
set_regime(b, regime_id) — writes regimes_d for OP_EVAL_REGIME /
OP_BRANCH_IF_REGIME consumption.
BacktestHarnessConfig.strategies (Vec<Strategy>) — empty means every
cell uses the hardcoded default; non-empty len must equal n_parallel
and each strategy is flattened + uploaded at construction.
bin/fxt-backtest --policy-grid <yaml> path now actually plumbs through
to the kernel (was parsed-but-ignored in C9). Empty grid keeps the
default behaviour.
New fixture decision_program_h4_only: uploads Strategy::Leaf(h4_only)
flattened to (EmitPerHorizonSize, WriteOrder) — 2 instructions, 16
bytes — and verifies the bytecode kernel produces equivalent end-state
to the hardcoded default (3 lot buy at vwap=5500). Proves the VM
dispatcher works end-to-end.
12/12 GPU fixtures green. 33 lib unit tests + 3 Ring 2 fuzz tests
still green.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
344d7a67d7 |
feat(ml-backtesting): wire --latency-ns end-to-end + fixture (C12)
The in-flight machinery from C11 is now reachable from the harness +
CLI. New step_decision_with_latency on LobSimCuda:
latency_ns == 0 → existing immediate-match path (submit_market_immediate
kernel fills against current book).
latency_ns > 0 → host reads market_targets, converts each non-noop
into seed_limit_order(active=2, price=very-aggressive,
arrival_ts_ns = current + latency_ns). The
resting_orders kernel promotes + fills at arrival
time, so the order sees whatever book exists then —
not the book at decision time.
Aggressive-price heuristic: buy at 1e9 / sell at 0 — the marketability
check (price ≥ best ask for buy, ≤ best bid for sell) unconditionally
crosses at arrival; the kernel then walks book levels for the actual
fill price. This models "market order with latency" correctly because
slippage emerges from the book-walking at arrival, not from a limit
price boundary.
BacktestHarness gains a latency_ns field; harness::run() always calls
step_decision_with_latency now. Also calls sim.step_resting_orders
per event (with trade_signed_vol=0.0 — the Mbp10RawInput trade-flow
hookup is deferred to a trades-feed integration commit) so in-flight
orders get a chance to promote on every snapshot.
bin/fxt-backtest no longer ignores --latency-ns; the flag value
propagates through BacktestHarnessConfig.latency_ns into the kernel
path. Default stays at 100_000_000 (IBKR + Scaleway baseline per
spec §4). Setting --latency-ns 0 selects the legacy immediate path.
Adds latency_in_flight_miss GPU fixture: limit at 5510 submitted
active=2 at T=1s with arrival_ts=T+100ms. step_resting at T+50ms
sees no promotion (still in-flight). Book moves +5 against buyer
during the 100ms window. step_resting at T+150ms promotes the
limit and fills it at the WORSE post-move book (vwap=5505 vs the
original 5500 it would have hit without latency). Slot ends active=0.
11/11 GPU fixtures green on RTX 3050. 33 lib tests still green.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
63ed6d0217 |
feat(ml-backtesting): artifacts + sweep aggregate + fxt-backtest CLI (C9)
artifacts.rs:
- Summary struct (total_pnl_usd, sharpe_ann, sortino_ann,
max_drawdown_usd, calmar, n_trades, win_rate, avg_win/avg_loss,
profit_factor, total_fees_usd, exposure_pct,
kelly_cap_history_sample).
- compute_summary(records, pnl_curve_usd) — non-overlapping
annualisation × √825 per pearl_phase1d4_backtest_cost_edge_frontier
(K=6000 holding × 250 trading days ≈ 825 trades/year).
Sharpe + Sortino + max drawdown + Calmar.
- write_summary (JSON pretty-printed), write_trades_csv (with USD
conversion from fp ×100), write_pnl_curve_bin (bytemuck-cast f32
slice). 5 unit tests with tempdir.
aggregate.rs:
- aggregate_sweep_dir walks <root>/<cell>/summary.json, builds an
arrow RecordBatch (cell name + 9 stats columns), writes
SNAPPY-compressed aggregate.parquet.
- pareto_frontier: cells are kept unless another cell weakly
dominates on all three of (sharpe_ann maxed, max_drawdown_usd
minimised, total_fees_usd minimised) AND strictly improves on one.
Written to pareto_frontier.json (Vec<cell-name>).
- 2 unit tests (3-cell mutual-non-dominance; B-dominates-A).
harness.rs:
- run() now samples Pos.realized_pnl × $50/index-pt per event into
self.pnl_curves[b], so the per-cell P&L curve is ready for
write_artifacts() without an extra sim pass.
- write_artifacts(out_dir) — per-cell <out>/cell_NNNN/{summary.json,
trades.csv, pnl_curve.bin}.
bin/fxt-backtest:
- clap-derive CLI with two subcommands:
run --data <dir> [--predecoded-dir <dir>] [--policy-grid <yaml>]
[--n-parallel N] [--decision-stride S] [--latency-ns N]
[--target-annual-vol-units F] [--annualisation-factor F]
[--max-lots N] [--max-events N] [--seed N] --out <dir>
aggregate <sweep_dir>
- Constructs MlDevice::cuda(0) + CfcTrunk::new_random for the trunk
(v1 — ml-alpha has no checkpoint format yet; --seed gates init).
- Parses --policy-grid YAML if given but doesn't yet plumb to the
LobSimCuda decision kernel (the v1 kernel hardcodes the
Strategy::default_for path; bytecode VM is C7's deferred follow-up).
Parse step kept end-to-end so the YAML format is validated now.
- --latency-ns parsed but not consumed — reserved for follow-up
resting-order in-flight promotion (deferred from C5).
Adds parquet + arrow + arrow-array + arrow-schema + serde_yaml to
ml-backtesting deps; bin/fxt-backtest added to workspace members.
All 33 lib tests + 6 GPU fixture tests green. CLI --help renders both
subcommands correctly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|