Stops chasing one-off corruption bugs. Three+ historical fixes patched
specific writers (#191 fxcache column-0, Bug 1 target schema,
label_scale=5443 leaks, today's state[0] heavy-tail). Each new
corruption shape found the next hole. This installs a structural
defense so corruption is REJECTED at the data-loading boundary
regardless of source.
Five independent layers, each mandatory:
1. Bar-level sanity (baseline_common::sanitize_bars):
drop bars with non-positive OHLC, high<low, non-finite values, or
close/prev_close outside [0.5, 2.0]. Catches DBN parse glitches,
broker tick errors, near-zero-open bars at source.
2. safe_log_return result clamp (extraction.rs:1246):
ratio.ln().clamp(±0.1). Real-market 1-bar log returns rarely
exceed ±0.05; ±0.1 traps every legitimate move while rejecting
the corruption shape (corrupt bar with bar.open ≈ 0 → ln = -30
→ previously normalized to -30000-magnitude state[0] outliers).
3. validate_features pre-norm bound (extraction.rs:538):
|val| ≤ 5.0 post-extraction. Pre-norm features come from
safe_normalize ([0,1]/[-1,1]), safe_clip (max ±3), or clamped
log-returns (±0.1); ±5 catches extractor invariant breaks.
4. NormStats::normalize post-norm clamp (walk_forward.rs:688):
((val - mean) / std).clamp(±20.0). Even if upstream produces
outliers, every value uploaded to GPU is bounded.
5. Shared validate_normalized_features gate (walk_forward.rs):
single source-of-truth invariant enforced at THREE sites:
- fxcache fast path (after discover_and_load)
- DBN fallback (after normalize_batch)
- precompute writer (before fxcache write — never persist
a poisoned cache)
Removed: DIAG_BUG2 + DIAG_BUG2_v2 one-shot diagnostics
(~125 lines of host-side download + outlier scan in
training_loop.rs). Replaced by structural defense — instrumentation
isn't needed when corruption can't reach state[0].
FEATURE_SCHEMA_HASH auto-bumps via build.rs FNV-1a hash over
SCHEMA_FILES (extraction.rs included). All pre-fix .fxcache files
on PVC are invalidated at load time; ensure-fxcache regen produces
clean cache with new clamps applied.
Why this finally closes the chapter: per-writer fixes are reactive
(land after corruption hits prod). Boundary validation is
proactive — every future regression to extraction or normalization
trips the gate at load, not at epoch 5 of a 50-epoch run. The 5
layers are independent: a bug in any one leaves the others as
backstop.
Validation: cargo check -p ml --all-targets --offline clean.
NormStats unit tests (walk_forward.rs:706+) still pass — clamp +
validate are additive; existing test inputs are well within bounds.
Refs: SP5 Bug 2 (state[0] std=570 outliers on smoke-test-xb78r),
historical #191#210#214#193#195 chains.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
test_all_42_features_bounded: verifies each feature within documented range
test_features_not_saturated: verifies non-zero variance (no constant features)
make_price_series: deterministic synthetic OHLCV bar generator for testing
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: shmem_max_in_dim only included trunk dims (state_dim,
shared_h1, shared_h2) but not head dims (value_h, adv_h). When
hidden_dim_base=32 made the trunk narrow while heads stayed at 128,
the BF16 weight tile for branch output (255×128=32640 BF16 elements)
overflowed the shared memory region (12288 BF16 elements). On H100
the overflow landed in unused-but-mapped hardware shmem (silent
corruption). On RTX 3050 (48KB physical shmem) it hit unmapped
memory → CUDA_ERROR_ILLEGAL_ADDRESS.
Changes:
- gpu_dqn_trainer.rs: shmem_max_in_dim includes value_h/adv_h
- Remove all #[ignore] from smoke tests (feature_coverage,
training_stability, gpu_residency)
- Smoke tests use real .dbn data from test_data/ (hard error if missing)
- Remove synthetic_data() fallback — no fake data in tests
- GPU-direct DtoD training path (train_step_gpu, FusedTrainScalars)
- GPU-native PER priority update kernel (zero CPU readback)
- IQN dual-head integration (gpu_iqn_head.rs)
- BF16 dtype fixes across 6 model adapters
- Hyperopt 30D→31D (iqn_lambda)
- portfolio_transformer: unconditional BF16 (remove dead CPU branches)
- liquid/adapter: all tests use Cuda(0) directly
- Fix pre-existing gpu_kernel_parity_test.rs (stale args)
- Fix pre-existing evaluate_baseline.rs (removed fields)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Cast input to weight dtype in DQN residual, rmsnorm, noisy_layers
- Set use_gpu=true in QNetworkConfig defaults and all config sites
- Resolve BF16 boundary mismatches in attention, curiosity, branching,
distributional_dueling across ml-dqn
- GPU-resident regime ops with BF16 boundary casts, eliminate .expect() in CUDA paths
- Eliminate all Device::Cpu fallbacks — GPU-only across 10 ML crates
- PPO: cast logits to F32 before softmax, cast batch tensors to training dtype
- Gradient collapse detection for RegimeConditionalDQN
- Wire halt_grad_collapse from CUDA guard kernel to halt training
- Dead neuron detection uses active network VarMap + squeeze factored readback
- Increment gradient_logging_step in GPU PER path
- Gradient collapse warmup guards use original buffer_size
- Cap training steps per epoch + tracing migration
- Replace Tensor::all() with sum_all() for pinned Candle compatibility
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Expand FeatureVector from 40 to 42 dimensions by including ADX(14) at
index 40 and CUSUM direction at index 41 from the existing CPU feature
extraction pipeline. This eliminates proxy-based regime classification
and enables GPU-native regime detection via tensor narrow/comparison ops.
Key changes:
- extraction.rs: wire RegimeADXFeatures + RegimeCUSUMFeatures into
extract_current_features_v2(), output 42 features per bar
- regime_conditional.rs: classify_regime_masks_gpu() creates per-regime
mask tensors entirely on GPU (ADX > 0.25 = trending, |CUSUM| > 0.7 =
volatile, else ranging). Zero CPU roundtrip in training hot path.
- trainer.rs/config.rs: state_dim 43→45 (no OFI), 51→53 (with OFI),
aligned dims unchanged (48/56). GPU batch insertion for all 3 heads.
- CUDA header: MARKET_DIM 40→42
- walk_forward.rs: FEATURE_DIM 40→42
- 42 files updated, all [f64;40]→[f64;42] propagated across workspace
Test results: ml=874/0, ml-dqn=354/0, ml-features=282/0, ml-core=274/0
Real data GPU smoke tests: 7/7 passed (OHLCV + OFI + trade enrichment)
Hyperopt baseline RL: 2 trials completed on local RTX 3050 Ti
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move all inline type definitions from ml/src/lib.rs to ml-core:
MLError, MLResult, Trade, MarketRegime, HealthStatus, Features,
MLModel trait, ModelRegistry, ParallelExecutor, LatencyOptimizer,
TrainingMetrics, ValidationMetrics, InferenceResult, ModelMetadata.
Dedup: consolidate ConfigError{reason}/ConfigurationError(msg) into
single ConfigError(String) tuple variant (was 2 variants, 174 refs).
Cleanup: convert create_hft_* free functions to associated methods
(HFTPerformanceProfile::ultra_low_latency(), ParallelExecutor::hft()).
ml facade re-exports via `pub use ml_core::*`.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The OFI calculator's compute_ofi_from_file() never called feed_trade(),
leaving 3/8 features (VPIN, Kyle's Lambda, trade_imbalance) at zero in
production. Added compute_ofi_with_trades() that interleaves trades by
timestamp into the MBP-10 streaming callback. Updated DQN and PPO
hyperopt adapters to derive trades_dir as sibling of dbn_data_dir.
Smoke test validates: without trades VPIN=0/5000, with trades VPIN=5000/5000.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes:
- Streaming MBP-10 parser (parse_mbp10_streaming): computes OFI inline
during decode — eliminates Vec<Mbp10Snapshot> allocation (41.9M clones)
- Parallel file processing: rayon par_iter across 9 MBP-10 files
(778s sequential → ~90s expected on H100 24-core)
- Fix hardcoded state_dim=54 in walk-forward backtest tensor creation
that caused panic "range end index 55296 out of range for slice of
length 52224" — now uses dynamic state_dim (43 or 51 with OFI)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add PositionFeatures struct that extracts 3 features for RL agent
position awareness (54→57 dim state vector):
- unrealized_pnl: normalized by EMA volatility
- bars_in_position: log-scaled to [0, 1]
- cost_basis: relative to price in bps, clamped [-500, 500]
8 unit tests, 0 clippy warnings. Not yet wired into trainers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The real_ prefix was misleading — there is no fake data loader.
Mechanical rename across 18 source files, no logic changes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split the build pipeline: one compile-services job builds all 8 service
binaries with PVC-backed sccache, saves as artifacts. Then 9 Kaniko jobs
just package pre-built binaries into slim runtime images (~30s each).
Before: 9 parallel Kaniko jobs each doing full cargo build --release
(~20min each, no sccache, 9x duplicated dep compilation)
After: 1 compile job with sccache (~5min cached) + 9 package jobs (~30s)
- Add compile stage between test and build
- Add Dockerfile.runtime (minimal debian + pre-built binary)
- Add Dockerfile.web-gateway-runtime (Node dashboard + pre-built binary)
- Keep Dockerfile.training via Kaniko (needs CUDA dev image for H100)
- Remove all SCCACHE_BUCKET build-args from service builds
- Use dir:// context for Kaniko (only sends build-out/ dir, not full repo)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move 17 library crates into crates/, CLI binary into bin/fxt,
consolidate 10 test crates into testing/, split config crate
from deployment config files.
Root directory reduced from 38+ to ~17 directories.
All Cargo.toml paths and build.rs proto refs updated.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
