Serializes m, v moments + step_count + hyperparams to a binary writer.
Uses mapped-pinned DtoD (not raw memcpy_htod/dtoh) per
feedback_no_htod_htoh_only_mapped_pinned.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces the flawed Phase F+G LobEnv trait + LobSimEnvAdapter +
MockLobEnv fixture with a GPU-pure env-interaction path.
DELETED:
- LobEnv trait (host-method-per-batch surface)
- MockLobEnv fixture (the toy-bandit pattern that hid the production defects)
ADDED:
- RlLobBackend trait (src/rl/reward.rs): narrow device-oriented
surface (apply_snapshot, pos_and_market_targets_mut, pos_d,
pos_bytes, step_fill_from_market_targets, n_backtests). Single
purpose: break the ml-alpha ↔ ml-backtesting dep cycle. No
mocking layer.
- 3 new GPU-pure kernels (cuda/):
* extract_realized_pnl_delta.cu — reads pos.realized_pnl +
position_lots from device Pos array, writes per-batch
(reward delta, done flag), updates trainer's prev_* buffers.
* apply_reward_scale.cu — element-wise rewards *= ISV[406].
* actions_to_market_targets.cu — 9-action grid → market_targets
{side, size} on device, reading current position_lots for
conditional Flat-from-Long/Short.
- LobSimCuda impls RlLobBackend (ml-backtesting/src/sim/mod.rs)
plus a new step_fill_from_market_targets entry that runs
submit_market_immediate + step_pnl_track without the host-side
targets-vec build. pos_and_market_targets_mut returns disjoint
field borrows (&pos_d, &mut market_targets_d).
- IntegratedTrainer: 3 cubin includes, 3 module/function fields,
launch_apply_reward_scale launcher, prev_realized_pnl_d /
prev_position_lots_d / rewards_d / dones_d buffers,
step_with_lobsim signature switched to RlLobBackend, body's
Step 5 rewritten as kernel-driven (no per-batch host loop, no
individual submit_action calls).
R6 PARTIAL — work R7 lifts:
- Thompson + log_pi stay host (R7 uses R4 kernels)
- mean_abs_pnl EMA stays host (R7 uses R3 ema_update_on_done)
- Advantage/return stays host (R7 uses R3 compute_advantage_return)
- 4 final DtoH copies for step_synthetic's host-slice signature
(R7 lifts step_after_encoder_forward to device-buffer args)
The "DtoH the device rewards/dones at end" comment in
step_with_lobsim documents the boundary. After R7, hot path has
zero host loops other than the now-unused Thompson host loop
(which R7 retires in favour of rl_action_kernel from R4).
Tests:
- integrated_trainer_smoke.rs: rewritten — real LobSimCuda with
synthetic book, one step_with_lobsim call, asserts finite losses
+ λs sum to 1. Smoke gate, not a convergence test.
- dqn_toy.rs, ppo_toy.rs: retired (empty stubs documenting
rationale). Files preserved so rename history survives. The
MockLobEnv toy-bandit pattern intrinsically couldn't catch the
production defects #1, #2, #5 that motivated this rebuild.
ml-backtesting added as ml-alpha dev-dep (cycle-safe: production
direction is ml-backtesting → ml-alpha; dev edge only loads when
building ml-alpha's own tests/examples).
Build cache-bust v29. cargo check + cargo build for all R-phase
tests green (pre-existing heads_bit_equiv index-OOB persists).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Each file's load_or_predecode + label generation is pure CPU work over
disjoint inputs (snapshots, cfg.horizons, cfg.outcome_label_cost). The
sequential for loop was the parallel-friendly bottleneck — converting
to rayon::par_iter gives ~4-8x speedup on typical 4-8 core hosts.
Combined with SPEED-C's ~400x speedup on the inner Welford loop, total
preload throughput is ~1600-3200x faster than the prior single-threaded
O(W) recompute path.
File order is preserved by par_iter's collect contract. Too-few-snapshots
skips emit a warn during load and resolve to None at collection.
Per user direction: feature-gating is appropriate for per-step diagnostic
logging (real perf/memory cost) but the name should be specific to the
mechanism, not a generic "diag-log" catch-all. kernel-step-trace
describes what the feature provides — per-step records written to the
ring by kernels.
Pure rename, no behavioral changes. Touched: Cargo.toml feature decl,
lib.rs cfg attribute, perception.rs (~25 cfg attributes including
not(feature) pairs), gpu_log.rs + smoothness_lambda_controller.cu doc
comments, gpu_log_ring_invariants.rs file-level cfg + ignore-attribute
text.
CfcTrunk::save_checkpoint(path) reads each device weight tensor back
via memcpy_dtoh and bincode-serialises into a CheckpointV1 envelope:
{ version, n_in, n_hid,
w_in, w_rec, b, tau,
heads_w, heads_b,
proj_w, proj_b, proj_g, proj_n }
Total ~22k-25k f32 = ~90 KB per trunk. Tiny.
CfcTrunk::load_checkpoint(dev, cfg, path) deserialises + validates
(version == 1, n_in/n_hid match the supplied CfcConfig — a model
trained for one arch can't silently load against another). Constructs
a fresh trunk via new_random (for kernel bindings + scratch buffers)
then overwrites every weight tensor via memcpy_htod. The random init
values are thrown away — marginally wasteful, but keeps the
construction code paths unified.
Roundtrip test (--ignored, CUDA-required): save trunk_A → load → read
back every device tensor and assert bit-equality between trunk_A and
the loaded trunk_B. Passed locally. Dim-mismatch rejection test runs
without CUDA (verifies bincode envelope serialise/deserialise).
bin/fxt-backtest --checkpoint <path>: when set, overrides --seed and
loads from disk. When absent, warns loudly that the trunk is
random-initialised and backtest results are noise. This makes the
binary genuinely useful as a deployment tool — point it at a trained
checkpoint and run real backtests.
Adds bincode workspace dep to ml-alpha (was already in workspace
dependencies, just not in ml-alpha's [dependencies] block). serde
features bumped to ["derive"] (was using workspace default which
omits derive macros).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reuses ml-features::predecoded::load_or_predecode_mbp10 (no cycle —
ml-features doesn't depend on ml-alpha). Yields seq_len-sized windows
of Mbp10RawInput plus 5-horizon binary labels via
multi_horizon_labels::generate_labels.
Per-snapshot prev_mid / prev_ts_ns / trade_signed_vol come from the
prior snapshot in the source stream (not from the anchor), so the
CfC trunk sees a continuous-time signal across the entire seq.
Labels: NaN at edge positions (no forward window) or tied prices;
BCE kernel masks these (Task 10).
Tests:
- loader_errors_on_missing_root: passes (1/1 inline)
- loader_yields_seq_with_valid_labels: --ignored, runs at gate time
with FOXHUNT_TEST_DATA set
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Cargo.toml: drops gbdt; adds memmap2 + approx; keeps ml-core only
(cannot depend on ml: would cycle since ml depends on ml-alpha for
the Mamba2 gate baseline).
build.rs: compiles 7 cubins (mamba2_alpha + 6 new placeholders)
with -O3 --use_fast_math --ftz --fmad. Skips kernels whose source
isn't present yet so partial check-ins work. Every env::var paired
with rerun-if-env-changed per the canonical build pearl.
src/pinned_mem.rs: local copy of MappedF32Buffer (mirrors
ml::cuda_pipeline::mapped_pinned::MappedF32Buffer). Drives the only
permitted CPU<->GPU path per feedback_no_htod_htoh_only_mapped_pinned.
Eventually the move-to-ml-core refactor will deduplicate; out of
scope for the Phase A branch.
Addendum: updates the import path to ml_alpha::pinned_mem.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three things landing atomically because they're load-bearing for each other:
1. **Trend-scanning leakage fix** — trend_scanning.rs was emitting OLS slope+t-stat
over a *forward* window [t, t+L]. With the Phase 1a label = sign(price[t+60]
− price[t]), the forward feature window overlaps the label window, contaminating
it. Purged walk-forward only sterilizes forward-looking *labels* that cross
the train/val split, not forward-looking *features* that peek inside the same
horizon the label measures. The leak inflated MLP accuracy from 0.49
(legacy 74-dim baseline) to 0.75 — vanished to 0.50 after switching to a
trailing window. Bounded the perfect-fit t-stat sentinel from ±1e6 → ±20
(p<1e-30 is already meaningless); eliminated the 16k corruption-cap drops.
2. **Variable-dim alpha column** — fxcache schema now carries the alpha-feature
width via metadata (`alpha_feature_dim`), not a compile-time constant. Same
on-disk format hosts the 134-dim bar-level stack OR the 81-dim snapshot stack.
Reader + auto-detect honor the metadata-declared dim; downstream MLP auto-sizes
`in_dim`. Single schema, no forks.
3. **Snapshot pipeline (Phase 1c falsification)** — `snapshot_pipeline.rs`: 81-dim
per-MBP10-snapshot extractor reusing 10 snapshot-native alpha blocks + 6 new
snapshot-specific features (time-since-trade, time-since-snap, event-rate,
spread-bps, L1-imbalance, microprice-mid drift). `precompute_features` gets
`--row-unit snapshot` flag; emits one fxcache row per LOB update (1.97M rows
from MBP-10 data vs 206K for bar mode).
**Smoke verdict on real data** (ES.FUT, 1.97M snapshots, 384K val):
- Bar-level honest alpha: accuracy=0.5005, AUC=0.5043 (no signal)
- **Snapshot-level alpha**: accuracy=0.5241, AUC=0.6849 (real signal, 384K val)
- GBM corroboration: accuracy=0.5401 (non-linear partitioning sees more)
- Horizon decay: alpha peaks at K=20-50 snapshots (~5-25ms), gone by K=500
- Regime-conditional: spread-Q4 quintile hits 0.752 accuracy on 76k samples
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
