Three things landing atomically because they're load-bearing for each other:
1. **Trend-scanning leakage fix** — trend_scanning.rs was emitting OLS slope+t-stat
over a *forward* window [t, t+L]. With the Phase 1a label = sign(price[t+60]
− price[t]), the forward feature window overlaps the label window, contaminating
it. Purged walk-forward only sterilizes forward-looking *labels* that cross
the train/val split, not forward-looking *features* that peek inside the same
horizon the label measures. The leak inflated MLP accuracy from 0.49
(legacy 74-dim baseline) to 0.75 — vanished to 0.50 after switching to a
trailing window. Bounded the perfect-fit t-stat sentinel from ±1e6 → ±20
(p<1e-30 is already meaningless); eliminated the 16k corruption-cap drops.
2. **Variable-dim alpha column** — fxcache schema now carries the alpha-feature
width via metadata (`alpha_feature_dim`), not a compile-time constant. Same
on-disk format hosts the 134-dim bar-level stack OR the 81-dim snapshot stack.
Reader + auto-detect honor the metadata-declared dim; downstream MLP auto-sizes
`in_dim`. Single schema, no forks.
3. **Snapshot pipeline (Phase 1c falsification)** — `snapshot_pipeline.rs`: 81-dim
per-MBP10-snapshot extractor reusing 10 snapshot-native alpha blocks + 6 new
snapshot-specific features (time-since-trade, time-since-snap, event-rate,
spread-bps, L1-imbalance, microprice-mid drift). `precompute_features` gets
`--row-unit snapshot` flag; emits one fxcache row per LOB update (1.97M rows
from MBP-10 data vs 206K for bar mode).
**Smoke verdict on real data** (ES.FUT, 1.97M snapshots, 384K val):
- Bar-level honest alpha: accuracy=0.5005, AUC=0.5043 (no signal)
- **Snapshot-level alpha**: accuracy=0.5241, AUC=0.6849 (real signal, 384K val)
- GBM corroboration: accuracy=0.5401 (non-linear partitioning sees more)
- Horizon decay: alpha peaks at K=20-50 snapshots (~5-25ms), gone by K=500
- Regime-conditional: spread-Q4 quintile hits 0.752 accuracy on 76k samples
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
## Two architectural cleanups, both surfaced by the wgdc8 experiment
### Part 1: volume_bar_size in cache key
Mirrors the imbalance_bar_threshold/ewma_alpha fix from `f7718b376`. The
volume bar size constant (100 contracts/bar) was previously hardcoded and
not in the fxcache key. Tuning it would have hit the same fossilization
bug as imbalance_bar_threshold did pre-fix.
Changes:
- `Hyperparams.volume_bar_size: u64` field added (default 100, matches
`DEFAULT_VOLUME_BAR_SIZE` for backwards compat).
- TrainingProfile loader reads `volume_bar_size` TOML key.
- `calculate_dbn_cache_key_full` signature 7 → 8 args. Hashed via
`to_le_bytes()`. Test `test_cache_key_includes_volume_bar_size`
added; passes alongside the 5 existing tests.
- 4 callers updated atomically (per `feedback_no_partial_refactor`):
`discover_and_load`, `data_loading.rs:146`, `train_baseline_rl.rs:599`,
`precompute_features.rs:259,720`.
- `data_loading.rs:279` now passes `self.hyperparams.volume_bar_size`
to `build_volume_bars` instead of the hardcoded `DEFAULT_VOLUME_BAR_SIZE`.
- New `--volume-bar-size` CLI arg on both binaries (default 100).
- New Argo workflow params `volume-bar-size` (default "100") and
`data-source` (default "mbp10") on both `train-template.yaml` and
`train-multi-seed-template.yaml`. Threaded into precompute + trainer
invocations.
- `scripts/argo-train.sh` exposes `--volume-bar-size <n>` and
`--data-source <s>` for ad-hoc overrides.
### Part 2: OFI front-month filter (latent bug fix)
`crates/ml/examples/precompute_features.rs:539-557` (the OFI/VPIN/Kyle's
Lambda computation branch when MBP-10 + trades data is available) was
loading trades unfiltered for per-bar microstructure feature computation.
The volume bar formation path filters front-month per-file (line 354), but
the OFI path did not.
Effect pre-fix: during contract rollover windows (e.g., ESZ24 → ESH25),
OFI per-bar microstructure features included trades from BOTH contracts
simultaneously, distorting VPIN, Kyle's Lambda, and trade imbalance
signals. Severity in production: small (front-month dominates ES.FUT
volume by 10-100×) but real and present in every prior MBP-10+trades
production run.
Fix: mirror the per-file `filter_front_month` call from the volume bar
path. Volume bar formation and OFI computation now both see the same
in-month trade tape. Added log line shows raw vs filtered count per file
for transparency.
## Why bundled
Both fixes touch trade-data plumbing in `precompute_features.rs` and the
fxcache key contract. Per `feedback_no_partial_refactor`, related
architectural cleanups land atomically. Both surfaced from the same
wgdc8 audit; bundling avoids two cache-key-invalidating commits in
sequence (each would force full fxcache regen).
## Compatibility
- `volume_bar_size` defaults to 100 → existing wgdc7-equivalent runs
reproduce, but with a *new* fxcache key (the f7718b376-era cache file
is unreachable; harmless, can GC manually).
- OFI fix is strictly more correct; no opt-out needed. Existing models
trained on contaminated OFI features may show slight feature
distribution drift on first cache regen — expected, not a regression.
- `data_source = "ohlcv"` Argo param now possible; routes precompute
through volume bar branch directly. wgdc8 experiment uses this to test
bar resolution sensitivity at volume_bar_size=500 (5× DEFAULT).
Tests: 6/6 feature_cache tests pass. Workspace + examples compile clean.
Audit-doc: `docs/dqn-wire-up-audit.md` updated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## The bug (audit 2026-05-09)
`crates/ml/src/feature_cache.rs:calculate_dbn_cache_key_full` hashed only
`(symbol, data_source, dbn_filenames+sizes)` — NOT `imbalance_bar_threshold`
or `imbalance_bar_ewma_alpha`. Combined with `precompute_features.rs:346`
unconditionally calling `build_volume_bars` regardless of `data_source`,
this meant:
1. 14 audited production runs (Apr 11–May 8) all collided on the same
fxcache key (`a3f933aa...` / `c07c960a...`) regardless of TOML
`imbalance_bar_threshold` value
2. The imbalance-bar code path was reachable only via fxcache MISS, which
never happens in production because `ensure-fxcache` always populates
first
3. Every "tuning" of `imbalance_bar_threshold` across 16+ SP runs was a
silent no-op — the system was actually running volume bars at
DEFAULT_VOLUME_BAR_SIZE (100 contracts/bar)
## The fix (this commit)
**Part A — cache key includes bar formation params:**
- `calculate_dbn_cache_key_full` signature: 5 args → 7 args. Two new f64
params hashed via `to_le_bytes()`.
- 4 callers updated atomically (per `feedback_no_partial_refactor`).
- 2 new unit tests (`test_cache_key_includes_bar_threshold`,
`test_cache_key_includes_bar_alpha`) pin the contract.
**Part B — precompute_features actually USES data_source:**
- New CLI args `--imbalance-bar-threshold` (default 0.5) and
`--imbalance-bar-ewma-alpha` (default 0.1) on both train_baseline_rl
and precompute_features.
- `precompute_features.rs:346` now branches: when
`data_source == "mbp10"` AND `mbp10_data_dir.is_some()`, calls
`mbp10_to_imbalance_bars` instead of `build_volume_bars`.
**Argo plumbing:**
- `train-template.yaml` + `train-multi-seed-template.yaml`: new workflow
parameters threaded into BOTH precompute and trainer invocations so
both compute the same fxcache key.
- `scripts/argo-train.sh`: new CLI flags for ad-hoc overrides.
- ensure-fxcache regen path: removed `rm -f /feature-cache/*.fxcache`
(with bar-params now in key, parallel experiments coexist).
## Effects going forward
- Tuning `imbalance_bar_threshold` actually changes bar density
- Configuring `data_source = "mbp10"` actually produces imbalance bars
- Multiple parallel experiments at different thresholds coexist on PVC
- `dqn-production.toml: imbalance_bar_threshold = 0.5` no longer ignored
Default values match prior production behavior → existing wgdc7-equivalent
runs reproduce, just with a *new* fxcache key (the old volume-bar cache
file is still on disk but won't be hit; harmless, can GC manually).
Audit-doc: `docs/dqn-wire-up-audit.md` updated with full context.
Tests: 5/5 feature_cache tests pass, full workspace + examples compile clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes recs 2 + 3 in lookahead-bias-audit-2026-04-28.md.
Fix 2A (rec 2): add walk_forward::PURGE_BARS = 5 constant and insert
val_start = train_end + PURGE_BARS in all 4 fold-construction sites:
generate_walk_forward_indices, _from_timestamps, _windows (date-based,
drops first PURGE_BARS bars of val), and gpu_walk_forward::generate_folds
(both stratified variants preserve the gap on boundary shift). 5-bar
width matches max per-bar feature lookback (autocorr lags 1/5/10);
compile-time per feedback_isv_for_adaptive_bounds since the
feature-pipeline lookback is itself compile-time.
Fix 2B (rec 3): per-fold NormStats fit from train-only data.
- walk_forward.rs: add from_features_slice + denormalize/denormalize_batch
helpers (inverse of normalize_batch for clamp-bounded round-trip).
- train_baseline_rl.rs fold loop: load fxcache sidecar norm_stats.json,
denormalise back to RAW, refit per-fold via from_features_slice,
renormalise full dataset, re-upload via init_from_fxcache, save
per-fold norm_stats_fold{N}.json. DBN-fallback path keeps legacy
behaviour (no sidecar to denormalise from) with a warn! log.
Ensemble trainer block is documented follow-up (separate code path).
Behavioral tests:
- test_norm_stats_per_fold_fit_train_only: synthetic train-mean=1.0 /
val-mean=9.0 dataset; from_features_slice recovers train-only mean to
ε=1e-5 while from_features returns global 5.0
- test_norm_stats_denormalize_roundtrip: non-clamped features round-trip
bit-identically
- test_walk_forward_purge_gap_indices_from_timestamps: every fold has
val_start - train_end == PURGE_BARS
- test_fold_generation_basic (gpu_walk_forward): updated to expect the
purge gap
Validation: cargo check --workspace clean (12.30s); 20/20 walk_forward
+ gpu_walk_forward tests pass; audit_sp18_consumers.sh --check exit 0.
Pre-existing test failures on local RTX 3050 Ti are unrelated SIGSEGVs
(VRAM exhaustion in chunked Thompson tests, pre-existing on baseline).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per spec §6.6 / Q6 train/dev/test split (defaults Q1-Q7 train, Q8 dev,
Q9 sealed final test):
- DQNHyperparameters: holdout_quarters + dev_quarters (default 1+1)
- crates/ml/examples/train_baseline_rl.rs: --holdout-quarters /
--dev-quarters CLI flags forwarded to hyperparams (this is the actual
training binary; bin/fxt/src/commands/train.rs is a gRPC client and
services/ml_training_service/src/main.rs accepts training params via
proto not CLI — see audit doc note).
- DQNTrainer::train_walk_forward slices training_data BEFORE fold
generation; folds run on Q1..Q(9 - holdout - dev) only.
- DQNTrainer struct: dev_features/dev_targets/holdout_features/
holdout_targets fields stash trailing slices for end-of-training dev
eval and the Phase 4.3 separate eval-only workflow.
- debug_assert sealed-slice guard catches future refactors that
re-introduce holdout into the training path.
Per established Phase 1 precedent (1.1-1.5: kernel/state lands first,
consumer wiring deferred to follow-up commit per
feedback_no_partial_refactor): CLI plumbing + slicing + dev/holdout
storage land in this commit. The post-final-fold dev evaluation call
(consumer of dev_features) is deferred to a follow-up commit and will
mirror Task 1.7's evaluate_dqn_graphed integration pattern. Phase 4.3
argo-eval-final.sh is the sole legitimate consumer of holdout_features
(separate eval-only workflow that does NOT call train_walk_forward).
cargo check -p ml --features cuda --example train_baseline_rl: clean
cargo check -p fxt: clean (no fxt changes needed; gRPC client only)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stops chasing one-off corruption bugs. Three+ historical fixes patched
specific writers (#191 fxcache column-0, Bug 1 target schema,
label_scale=5443 leaks, today's state[0] heavy-tail). Each new
corruption shape found the next hole. This installs a structural
defense so corruption is REJECTED at the data-loading boundary
regardless of source.
Five independent layers, each mandatory:
1. Bar-level sanity (baseline_common::sanitize_bars):
drop bars with non-positive OHLC, high<low, non-finite values, or
close/prev_close outside [0.5, 2.0]. Catches DBN parse glitches,
broker tick errors, near-zero-open bars at source.
2. safe_log_return result clamp (extraction.rs:1246):
ratio.ln().clamp(±0.1). Real-market 1-bar log returns rarely
exceed ±0.05; ±0.1 traps every legitimate move while rejecting
the corruption shape (corrupt bar with bar.open ≈ 0 → ln = -30
→ previously normalized to -30000-magnitude state[0] outliers).
3. validate_features pre-norm bound (extraction.rs:538):
|val| ≤ 5.0 post-extraction. Pre-norm features come from
safe_normalize ([0,1]/[-1,1]), safe_clip (max ±3), or clamped
log-returns (±0.1); ±5 catches extractor invariant breaks.
4. NormStats::normalize post-norm clamp (walk_forward.rs:688):
((val - mean) / std).clamp(±20.0). Even if upstream produces
outliers, every value uploaded to GPU is bounded.
5. Shared validate_normalized_features gate (walk_forward.rs):
single source-of-truth invariant enforced at THREE sites:
- fxcache fast path (after discover_and_load)
- DBN fallback (after normalize_batch)
- precompute writer (before fxcache write — never persist
a poisoned cache)
Removed: DIAG_BUG2 + DIAG_BUG2_v2 one-shot diagnostics
(~125 lines of host-side download + outlier scan in
training_loop.rs). Replaced by structural defense — instrumentation
isn't needed when corruption can't reach state[0].
FEATURE_SCHEMA_HASH auto-bumps via build.rs FNV-1a hash over
SCHEMA_FILES (extraction.rs included). All pre-fix .fxcache files
on PVC are invalidated at load time; ensure-fxcache regen produces
clean cache with new clamps applied.
Why this finally closes the chapter: per-writer fixes are reactive
(land after corruption hits prod). Boundary validation is
proactive — every future regression to extraction or normalization
trips the gate at load, not at epoch 5 of a 50-epoch run. The 5
layers are independent: a bug in any one leaves the others as
backstop.
Validation: cargo check -p ml --all-targets --offline clean.
NormStats unit tests (walk_forward.rs:706+) still pass — clamp +
validate are additive; existing test inputs are well within bounds.
Refs: SP5 Bug 2 (state[0] std=570 outliers on smoke-test-xb78r),
historical #191#210#214#193#195 chains.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two-part fix for a class of bugs causing un-normalised features to
silently flow into training.
(1) data_source mismatch between precompute writer and trainer reader.
precompute_features.rs:214,633 hardcoded "ohlcv"
train_baseline_rl.rs:582 hardcoded "ohlcv"
config/training/dqn-production.toml: data_source = "mbp10"
Production runs the trainer with the production profile (data_source
= mbp10), but the actual cache lookup hardcoded "ohlcv". Smoke worked
by accident (smoke profile is also "ohlcv"). Any future profile with
a different data_source silently mismatches → cache MISS → DBN
fallback path.
Both call sites now hardcode "mbp10" (the canonical production data
source per CLAUDE.md). precompute_features adds a `--data-source`
CLI override for the rare case a smoke flow needs to regenerate
the local "ohlcv" fxcache; default is "mbp10".
(2) DBN-fallback path didn't normalise features.
precompute_features.rs:629 applies NormStats::normalize_batch on the
canonical fxcache write path. The fallback in train_baseline_rl.rs
(cache-miss → load DBN files → extract features → upload to GPU) did
NOT normalise. Any cache miss (data_source drift, schema-hash mismatch,
missing file) silently uploaded RAW features. Raw close prices
(~$5180 ES futures) flowed into next_states[:, 0]; the aux next-bar
head's label_scale EMA latched onto raw-price magnitude (~5443 vs
expected ~1.0 z-score); the shared trunk learned to predict next-bar
prices; the policy effectively traded with future-price knowledge →
train-h5gxb epoch-0 Sharpe = 141 with 0.32% max-drawdown over 214k
bars (impossibly good = oracle leak).
DBN fallback now applies the same z-score normalisation unconditionally
as defence-in-depth, so a future cache-miss cannot reintroduce raw
values into training.
Audit entry updated.
The first L40S deploy attempt (workflow `train-multi-seed-z2llf`, terminated)
failed at startup with `error: unexpected argument '--fold' found` on every
job: `train_baseline_rl` is a multi-fold walk-forward executor that accepts
`--max-folds K`, NOT `--fold N`. The original P5T1 harness assumed the
opposite and fanned out N seeds × K folds = N*K jobs, each invoking the
binary with `--seed N --fold K`.
User chose Path B: pivot to one job per seed (each runs all K folds via the
existing `--max-folds` mechanism). Per-job runtime is K× longer, but fanout
drops from N*K=30 → N=5 (matches L40S pool capacity better) and the binary
contract becomes the one the binary actually has.
4 surface changes:
1. crates/ml/examples/train_baseline_rl.rs — add `--seed N` CLI arg
(default 42 — historic implicit value). Sets `FOXHUNT_SEED` env var at
startup BEFORE any CUDA module spins up. Logs the seed value at the
training start banner.
2. crates/ml/src/cuda_pipeline/mod.rs — add `global_seed()` (reads
`FOXHUNT_SEED`, default 42) + `mix_seed(base)` (SplitMix64 avalanche
so adjacent global seeds produce uncorrelated module seeds). Six call
sites updated to mix the global seed into their previously-hardcoded
constants:
- trainer/action.rs: GpuActionSelector seed (0xDEAD_BEEF_CAFE) + the
epsilon-greedy fallback StdRng (0xAC7_DEF0).
- cuda_pipeline/gpu_iqn_head.rs: IQN Xavier-init RNG (0x1CA_1234).
- cuda_pipeline/gpu_iql_trainer.rs: V(s) Xavier-init RNG (0x1C1_9ABC).
- cuda_pipeline/gpu_her.rs: random-donor RNG (0x4E4_5678).
- cuda_pipeline/gpu_ppo_collector.rs: rng_seeds Vec for PPO
experience-collector init + reset (0xAA0_5EED).
- trainer/training_loop.rs: per-epoch regime_dropout_seed.
3. infra/k8s/argo/train-multi-seed-template.yaml — drop `fold` parameter
from `train-single` template; binary invoked as `--seed "$SEED"
--max-folds {{workflow.parameters.folds}}` so the walk-forward sweep
happens inside the single training process. Drop `FOLD` env var. Update
the nsys-rep upload filename to drop the fold suffix. Update banners /
doc comments to reflect "one-job-per-seed" semantics.
4. scripts/argo-train.sh — matrix generator drops the inner fold loop.
Each emitted task carries only `seed=${s}` and depends on the same
ensure-fxcache + gpu-warmup. The dry-run synthetic marker switches from
`seed=${s} fold=${f}` to `seed=${s} max_folds=${FOLDS}` so test harnesses
count the new shape correctly.
5. scripts/tests/test_multi_seed_harness.sh — assertions updated:
- `--multi-seed 3 --folds 2` produces 3 tasks (was 6).
- Rendered binary command must include `--max-folds
{{workflow.parameters.folds}}` placeholder.
- Rendered template must declare `folds` workflow parameter (so
`argo submit -p folds=K` overrides the default).
- Rendered binary command must NOT contain any per-fold flag — this
catches the failure mode that broke the first L40S deploy.
- Backward-compat: `--multi-seed 1 --folds 1` preserves the existing
single-template path (no DAG matrix tasks emitted).
6. docs/dqn-wire-up-audit.md — adds 1 Wired row documenting the pivot,
the new `--seed`/`mix_seed` plumbing, all 6 RNG call sites, and the
end-to-end seed-variation verification result.
Validation:
cargo check --workspace clean at 11 warnings (workspace baseline preserved).
cargo build --release --example train_baseline_rl succeeds; --help shows
the new --seed flag with documented default 42.
Seed-variation end-to-end test on RTX 3050 Ti (1 fold × 2 epochs each):
--seed 42 → F0 best Sharpe = -9.7831, best_val_metric = 1.957244,
epoch-2 train Sharpe = -16.12, val_Sharpe = +1.11.
--seed 999 → F0 best Sharpe = +92.9341, best_val_metric = 2.161012,
epoch-2 train Sharpe = +92.93, val_Sharpe = -0.25.
Different best Sharpe / best_val_metric / epoch-2 train + val Sharpe
across seeds proves the seed actually propagates through the RNG init
paths and is not just accepted-and-ignored. The seed=42 numbers match
the prompt's "deterministic baseline" expectation (F0 = -9.7831 was
bit-identical pre-pivot because no global-seed plumbing existed).
./scripts/argo-train.sh dqn --multi-seed 5 --folds 6 --dry-run produces
exactly 5 WorkflowTask markers (train-s0..train-s4), each with
`--max-folds {{workflow.parameters.folds}}` in the binary invocation.
All 3 harness tests PASS:
- test_multi_seed_harness.sh: 5 PASS lines, exit 0.
- test_nsys_harness.sh: 4 PASS lines + ALL PASS, exit 0.
- test_tier_checks.sh: PASS overall (good-fixture passes, bad-fixture
surfaces expected check rejections), exit 0.
Backward compat: existing single-job `argo-train.sh` callers (no
`--multi-seed`, no `--folds`) route to the original `train-template.yaml`
unchanged. `--seed 42` is a no-op offset for the SplitMix64 mix at the call
sites — the trajectory shifts only when the user passes `--seed` explicitly,
matching the prompt's "default 42 (historic implicit value)" requirement.
L40S pool: argo-train.sh defaults `--gpu-pool ci-training-h100`; user passes
`--gpu-pool ci-training-l40s` at deploy time. No script default change
(per constraint 5).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Local smoke-test run after Plan 1 C.6 completion surfaced three issues:
1. StateResetRegistry missing dispatch arms for the 8 ISV slots
pre-allocated in ac9bcab94 (isv_epoch_idx, isv_epsilon_eff, isv_tau_eff,
isv_gamma_eff, isv_kelly_cap_eff, + 3 SchemaContract which already no-op).
Fold boundary would error "unknown name 'isv_epoch_idx'". Added dispatch
arms in training_loop.rs::reset_named_state — reset each to 0.0; GPU
kernels repopulate on next epoch.
2. controller_activity smoke test's single 50% threshold was designed for
reactive CPU-compute controllers. Under GPU-drives-CPU-reads, tau is a
Polyak-EMA cosine schedule that fires every epoch by design (95%),
gamma is health-coupled monotonic (may fire every epoch as health
drifts). Split threshold per-controller: reactive (anti_lr, grad_clip,
cql_alpha, cost_anneal) = 0.50; schedule-based (tau, gamma) = 1.00.
3. examples/train_baseline_rl was broken since the f64→f32 ABI refactor
(d64adc14f) — hp_f64 returning f64 assigned to f32 fields. Added
hp_f32 helper that narrows JSON-born f64→f32 at ingest boundary. Use
hp_f32 for f32 fields, hp_f64 for f64 fields (learning_rate,
entropy_coefficient, weight_decay). No more "as f32" casts at call
sites. Also fixed replay_buffer_vram_fraction + bars_per_day f64→f32.
Local smoke tests now pass:
- controller_activity: ok (1 passed, 29.5s)
- multi_fold_convergence: ok (1 passed, 3 folds x 20 epochs, 534.9s)
- 24 new monitor + registry unit tests: all passing
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix kernel-read gap
Adds 12 features to the DQN input pipeline:
- 10 MicrostructureState::snapshot()[0..10] slots that were previously computed
every bar and then discarded before reaching fxcache: ofi_trajectory,
realized_variance, hawkes_intensity, book_pressure (weighted 10-level),
spread_dynamics, aggression_ratio, queue_depletion_asymmetry,
order_count_flux, intra_bar_momentum, regime_score.
- 2 TLOB-novel slots derived directly from Mbp10Snapshot:
order_count_imbalance = (Σbid_ct − Σask_ct) / Σ(bid_ct + ask_ct),
microprice_residual = (weighted_mid − mid) / mid.
Also fixes a production gap: ofi_acceleration (slot 18) and
toxicity_gradient (slot 19) were persisted to fxcache via OFI_DIM=20
but the OFI embed kernel (experience_kernels.cu:6146-6173) only read
[0..18), silently discarding them every bar. Kernel extended to
consume full SL_OFI_DIM=32.
Dimension bumps (all 8-aligned):
OFI_DIM 20 → 32
FXCACHE_VERSION 4 → 5 (invalidates existing caches; regen via
precompute_features)
STATE_DIM 96 → 104
PADDING_DIM 4 → 0 (OFI expansion consumed padding, still 8-aligned)
STATE_DIM_PADDED 128 (unchanged)
OFI_EMBED_IN 18 → 32 (MLP input width; W/grad/Adam/m/v buffers
resized in lockstep via named constants)
fxcache regen results (175874 bars ES.FUT 2024-Q1):
deltas_nonzero: 175781 / 175874 (99.9 percent)
book_aggression: 102137 / 175874 (58.1 percent)
microstructure[20-30): 175874 / 175874 (100 percent)
tlob_novel[30-32): 133615 / 175874 (76.0 percent)
Compile status: SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo check
--workspace --tests passes cleanly (0 errors, pre-existing warnings
only).
Test results:
fxcache roundtrip (unit + integration): PASS (4+6 tests)
magnitude_distribution smoke: ran through epoch 1 successfully
(OFI_DIAG fires, state_dim=104 confirmed, feature_dim=74 in
validation kernel); epoch 2 OOM on local RTX 3050 Ti (4 GB) —
expected hardware limit from state_dim growth. Full 20-epoch run
requires L40S/H100 CI verification.
multi_fold_convergence smoke: not verified locally (same VRAM
ceiling applies). L40S/H100 CI verification required.
The new slots follow the existing OFICalculator/MicrostructureState
pattern and consume signals already computed by ml-features — no new
crate, no ONNX, no stubs. All 12 sources were audited against their
implementation before persistence; every slot traces back to real
Mbp10Snapshot or MicrostructureState math.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds an optional --max-bars CLI cap to `examples/train_baseline_rl.rs`.
When >0, truncates the fxcache-loaded features/targets/OFI/timestamps in
lockstep per the data_loading.rs precedent (commit 2ac956298 OFI/bars
desync fix), then rebuilds FxCacheData with the capped bar_count. 0 =
unlimited, matching pre-change production behaviour.
Wires --max-bars=500000 into the multi_fold_convergence smoke test. Full
ES.FUT fxcache is ~697,732 bars (~290 MB on GPU with OFI per the
session_2026-04-20 memory note); the smoke's 3-fold × (6+2+2+2×step =
14-month total) walk-forward span needs ~350k bars at 1-min resolution,
so 500k gives ~43% headroom while freeing ~81 MB of VRAM — enough to
unblock RTX 3050 Ti 4 GB local runs that OOMed before.
Applied after fxcache load and before WalkForwardConfig so downstream
fold-range generation sees the capped range. Zero effect on production
training (default 0 = unlimited).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds --max-folds CLI flag to train_baseline_rl (0 = no cap). When set,
truncates the generated walk-forward fold list to the first N folds.
Used by the new multi_fold_convergence smoke test to run a 3-fold × 20-epoch
local variant of the Phase 3 L40S 6-fold × 50-epoch validation gate
(~5 min on RTX 3050 vs ~1 hour on L40S).
Test asserts:
* train_baseline_rl subprocess exits 0 (no NaN/Inf)
* ≥ 2/3 folds produce dqn_fold{N}_best.safetensors checkpoint
(checkpoint is only written when Best Sharpe improves during the
fold, so presence = policy learned something on that window)
Per plan Task 0.15 at docs/superpowers/plans/2026-04-21-policy-quality.md.
evaluate_baseline looks for <output_dir>/norm_stats_fold{N}.json per fold
(matching the convention written by train_baseline_supervised.rs:812).
train_baseline_rl never produced this file — the fxcache has a single
<hex_key>.norm_stats.json written by precompute_features, which RL
training uses implicitly for pre-normalized features but never copies to
the per-fold paths evaluate wants. Result (L40S train-7r9zf):
Error: NormStats not found at /workspace/output/norm_stats_fold0.json
- cannot evaluate without training-set statistics (computing from test
data would introduce lookahead bias). Run training first to generate
this file.
WARN: Evaluation failed, continuing
Evaluate fails gracefully but no report is produced.
Fix: new helper `fxcache::norm_stats_path_for_key(data_dir, override, cache_key)`
returns the canonical <hex_key>.norm_stats.json path that sits next to
the .fxcache file. train_baseline_rl resolves this once after the fxcache
load (falling back to None if the key is zero — i.e., DBN fallback path
with no precomputed stats) and copies it into the output dir as
norm_stats_fold{N}.json for every fold processed.
The RL-side fxcache norm stats are fold-independent (one z-score per
cache key, applied to all walk-forward windows), so the per-fold copies
are intentional duplicates of the same file — this matches evaluate's
per-fold lookup semantics without diverging from supervised convention.
Closes task #32.
target_dim expansion: adds raw_open (OHLCV) and mid_price_open
(MBP-10 midpoint at bar formation) to fxcache targets. FXCACHE_VERSION
2→3 for auto-rebuild. Legacy v2 files handled with close-price fallback.
Spec v5 adds 3 pearls:
- Bar duration encoding in Mamba2 (continuous-time SSM awareness)
- Order book center of mass from all 10 MBP-10 levels (aggression signal)
- Retrospective hold quality bonus (teaches exit timing)
Plus: Hold action (4th direction), DSR Sharpe EMA fix, counterfactual
magnitude/order sign fix, MFT mid-price mark-to-market.
OFI embed MLP now 18→10 (was 16→8). Mamba2 width SH2+10 (was SH2+8).
Attention width D+10 (was D+8).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Default was hardcoded as 64 in train_baseline_rl.rs and evaluate_baseline.rs,
overriding the config default of 32. Added num_quantiles=32 to production
config so it's explicit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removed: enforce_hold(), min_hold_bars from config/kernels/backtest.
Added: holding_cost_rate (inventory penalty), churn_threshold_bars +
churn_penalty_scale (graduated flip penalty) to reward in
experience_env_step and backtest kernels.
The model learns optimal hold timing from cost signals:
- Per-trade tx cost prevents churning (existing)
- Inventory penalty makes large positions expensive to hold
- Churn penalty graduates cost for rapid flips
- Temporal attention learns when holding cost > expected profit
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
bars_per_day was hardcoded 390 (1-minute candles) but we use MBP-10
imbalance bars (~7500/day). All Sharpe/Sortino/return annualization
was off by sqrt(7500/390) ≈ 4.4×.
Now computed from actual fxcache timestamps:
bars_per_day = total_bars / (trading_days from timestamp span)
Propagates to: val_Sharpe, train_Sharpe, financials, backtest evaluator.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
fxcache: OFI_DIM=20 (pub const), RECORD_F64_COUNT=66, 272 bytes/bar.
constructor: state_dim 74→86 (aligned 88) with OFI.
experience collector: ofi_dim detection 8→20.
All [f64; 8] → [f64; 20]. Existing 8 features preserved at 0-7.
New 12 features zero-padded until precompute is extended.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace 40-line inline fxcache discovery block in train_baseline_rl.rs
with a single call to ml::fxcache::discover_and_load(). precompute_features.rs
already uses correct symbol/data_source args — no changes needed there.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds `has_ofi: bool` to `FxCacheData` and the fxcache binary header
(stored in `reserved[0]`). Propagates through load_fxcache, write_fxcache,
all callers (precompute_features, train_baseline_rl, hyperopt dqn adapter),
existing roundtrip tests, and adds two new has_ofi-specific roundtrip tests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Different instruments (ES.FUT vs NQ.FUT) and data source modes (ohlcv vs mbp10)
now produce distinct .fxcache keys, preventing silent overwrites and wrong-bar-type
loads at cache lookup time.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The cache key in train_baseline_rl was computed from data_dir/symbol
(e.g. test_data/futures-baseline/ES.FUT) but precompute_features uses
data_dir (test_data/futures-baseline). Different paths → different keys
→ cache miss → OFI=false.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Normalize features once in precompute_features (single source of truth).
NormStats saved alongside .fxcache for inference denormalization.
Removed per-fold NormStats computation from train_baseline_rl, hyperopt
adapter, and smoketests — fxcache is pre-normalized, consumers use as-is.
NOTE: features still show raw price values (max=18000+) because
test_data/futures-baseline contains multiple symbols (ES, NQ, ZN, 6E)
mixed into one dataset. The feature extraction pipeline needs
investigation — log returns between different symbols produce garbage.
This is tracked as a separate task.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix 1 (HIGH): Ensemble trainers were created + data uploaded inside the
fold loop, causing redundant GPU uploads per fold per ensemble member.
Moved creation + init_from_fxcache before the fold loop; inside the loop
we now reuse trainers via set_training_range + reset_for_fold.
Fix 2 (MEDIUM): reset_for_fold cleared gpu_data = None, forcing re-upload
on every fold even though the full dataset was already GPU-resident via
init_from_fxcache. Removed the clearing — data stays on GPU across folds.
Fix 3 (LOW): train_fold_from_slices allocated a Vec<f64> per bar via
t.to_vec(). Added train_with_data_full_loop_slices that accepts
&[([f64; 42], [f64; 4])] — both are Copy stack types, zero heap alloc
per element. Added matching collect_gpu_experiences_slices and
run_training_steps_slices helpers.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add PpoTrainer::train_from_slices(&[[f64; 42]]) that converts to
Vec<Vec<f32>> internally (PPO train() API requires ownership). Replace
the PPO fold loop in train_baseline_rl.rs: eliminates the OHLCVBar
construction shim and calls train_from_slices directly on fxcache
feature slices. Delete the standalone train_ppo_fold function entirely.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds DQNTrainer::train_fold_from_slices(&[[f64;42]], &[[f64;4]]) so the
fold loop in train_baseline_rl passes raw fxcache slices directly, without
the caller constructing any Vec<(FeatureVector, Vec<f64>)>. Removes
features_to_trainer_format_fast helper (no longer needed) and updates
train_dqn_fold to call the new method.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Rewrites the train_baseline_rl fold loop to eliminate per-fold waste:
- Data loading: fxcache -> fold index ranges from timestamps (no bar
reconstruction). DBN fallback builds FxCacheData in-place.
- DQN trainer created ONCE before fold loop, fxcache uploaded to GPU
ONCE via init_from_fxcache. Each fold uses set_training_range +
set_val_data_from_slices + reset_for_fold instead of recreating.
- Tokio runtime created ONCE (not per fold).
- Hyperparams construction extracted to build_dqn_hyperparams().
- Deleted: prepare_fold_data, FoldData type, prefetch thread,
DoubleBufferedLoader GPU staging, features_to_trainer_format (old).
- Added: generate_walk_forward_indices_from_timestamps (i64 ns
timestamps, O(log n) partition_point, no OHLCVBar dependency).
- Added: features_to_trainer_format_fast (fxcache targets directly).
- PPO compatibility preserved: constructs minimal OHLCVBars from
fxcache targets for train_ppo_fold (Task 6 will refactor).
- Ensemble mode preserved with per-member trainers for k>0.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Print fxcache lookup key to diagnose cache miss on H100
- PREFETCH_K=16 (was usize::MAX causing 8M PER samples in one shot)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Epoch duration self-balances: bigger GPU → bigger auto-scaled batch →
fewer steps per epoch. The manual cap created 7 different values
(0, 8, 64, 100, 200, 300, 2000) across configs/tests/examples, making
behavior inconsistent between environments.
Removed from: DQNHyperparameters, training profiles (smoketest,
localdev, production), CLI args, Argo templates, hyperopt adapter,
all test overrides, supervised example.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All VRAM-derived parameters (batch_size, gpu_n_episodes, buffer_size)
are auto-scaled — CLI overrides bypass this and cause inconsistent
behavior between local testing and production.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
gpu_n_episodes was manually overridden in GPU profiles, training configs,
test files, and hyperopt — all set to 0 or small fixed values that
bypassed the auto-scaling logic, causing a div-by-zero crash in
train_baseline_rl.
Now: single auto-scaling path via optimal_n_episodes() from VRAM/SM
count. No manual override field. Cap at 16384 (consistent with
AutoBatchSizer's 8192 cap pattern). Floor at 32 for small GPUs.
Removed gpu_n_episodes from:
- DQNHyperparameters, PpoHyperparameters structs
- All 4 GPU profiles (rtx3050, h100, a100, default)
- Training profiles (smoketest, localdev)
- ExperienceProfile struct + serde
- Hyperopt adapter
- All test overrides
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sharpe calculation:
- Use per-trade returns (sum_returns/sum_sq_returns) instead of per-bar
step_returns. Per-bar Sharpe collapsed variance → bogus 19.75 with PF=0.03.
- Annualize by sqrt(trades_per_year) not sqrt(bars_per_year).
Batch sizing:
- Cap auto-computed batch_size at 8192 (VRAM ceiling of 2M is OOM limit, not
optimal RL batch).
- Add VRAM floor: batch_size < ceiling/4096 gets scaled UP (128 → 512 on H100).
- Only let hyperopt override batch_size when explicitly non-zero — preserve
profile's batch_size=0 auto-compute sentinel.
Replay buffer:
- Divide per_max_memory_bytes by 3.0 for regime heads (PER budget was 3x too
large, causing OOM cascade 74M → 37M → 18M → 9M → 4.6M).
- HER buffer uses original_buffer_size (pre-autosizer), not inflated 74M.
Q-value clipping:
- Wire hyperparams.q_clip_min/max to DQNConfig (was hardcoded ±500, production
TOML has ±50). Prevents Q-value overestimation ratio of 94.6x at epoch 2.
Training stability:
- Anti-LR warmup: skip first 5 epochs (early Sharpe unreliable from random
policy). Prevents bogus 3x LR boost at epoch 2.
- min_replay_size from profile (1000), not hyperopt batch_size (128).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- DAG: compile (ci-compile-cpu) + gpu-warmup → train (H100)
- Compiles from source targeting compute cap 90
- fxcache REQUIRED — fails if no .fxcache found
- Error chains printed with {:#} for full cause visibility
- Uses cargo-target-pvc for compile cache + sccache
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each record now starts with an i64 timestamp (nanoseconds since epoch)
before the feature/target/OFI data. v1 records grow from 432 to 440
bytes, v2 from 112 to 120 bytes. The timestamp is always i64 even in
bf16 mode. train_baseline_rl reconstructs bars with real timestamps
instead of placeholders so the walk-forward windower can split by month.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Auto-discovers fxcache via env var or sibling directory. Falls back
to DBN loading if no cache found. Reconstructs aligned bars from
cached targets for walk-forward windowing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add --min-hold-bars CLI arg to train_baseline_rl and hyperopt_baseline_rl.
Wire through Argo workflow as parameter. Default 5 (TOML), override via
CLI for quick A/B experiments without config changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
train_baseline_rl now accepts --initial-capital (default $35K) matching
hyperopt. Argo compile-and-train passes the workflow parameter to the
train-best step. Both hyperopt and training now use consistent capital.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Noisy nets are now always on (mandatory feature since config unification).
Epsilon start always uses the noisy-nets-aware default (0.05).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove use_double_dqn, use_dueling, use_per, use_branching,
use_distributional, use_noisy_nets, use_huber_loss, and use_cql
from DQNConfig, DQNHyperparameters, and DqnParams structs.
These features are always enabled (Rainbow DQN standard). The boolean
flags were dead code — every constructor set them to true, and the
only code paths that set them to false were in tests that disabled
features for simplicity. With the fields removed, the features are
unconditionally active, eliminating ~490 lines of dead configuration.
Key changes:
- Struct field declarations removed from 3 core config structs
- Conditional branches (if use_X { ... } else { ... }) simplified:
dueling/branching/PER network creation is now unconditional
- Checkpoint metadata hardcodes "true" for backward compatibility
- Hyperopt search space index 11 (use_branching) fixed at 1.0
- TOML/YAML config files cleaned of removed fields
- Tests that toggled these flags updated or rewritten
45 files changed, -487 net lines. Zero new test failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
