9f2d0fffb5
v8 smoke (train-96wfk, commit 5694eb4df) eval pod crashed at the
same point as v7 with CUDA_ERROR_ILLEGAL_ADDRESS despite Phase 8.5's
set_branch_sizes fix. 15-minute runtime confirmed env_step decoder
no longer crashes (8.5 worked); a later kernel still OOBed.
Localization via local compute-sanitizer (RTX 3050 Ti):
Updated gpu_backtest_validation.rs to mirror production eval-baseline
(FEATURE_DIM=42, portfolio_dim=24, set_branch_sizes), reproducing the
v8 crash in 1.66s locally.
compute-sanitizer --tool=memcheck pinpointed:
Invalid __global__ read of size 4 bytes
at cost_net_sharpe_kernel+0x90
by thread (32,0,0) in block (0,0,0)
Access at 0x65c is out of bounds
0x65c = 1628 bytes = float index 407 = OFI_IMPACT_LAMBDA_INDEX.
Kernel reads isv[407]; isv pointer was 0 (null) because
isv_signals_ptr defaults to 0 in constructor and set_isv_signals_ptr
is only called by production training. Closure-path callers
(eval-baseline) dereferenced null + slot×4 bytes.
Fix:
Allocate zero-filled default_isv_buf of size ISV_TOTAL_DIM=536 f32
in constructor. Wire isv_signals_ptr to its dev_ptr by default.
Production training still overrides via set_isv_signals_ptr.
Zero-init semantics:
- isv[407]=0 → ofi_lambda=0 → c_ofi=0 in cost-net sharpe
(degraded but valid; matches LobBar.ofi=0.0 placeholder)
- Other slots default to 0 — Kelly health, controller anchors,
etc. all see degraded-but-valid defaults
Test updates (consumer migration):
- FEATURE_DIM 10 → 42 (production value; FEATURE_DIM < 32 makes
gather kernel's `market_dim = feat_dim - SL_OFI_DIM (32)` negative
→ OOB; previous tests were already broken even before our changes)
- portfolio_dim 3 → 24 (Phase 8.3+9 contract)
- set_branch_sizes call added (Phase 8.5 contract)
Verification:
cargo test -p ml --test gpu_backtest_validation \
gpu_tests::test_always_long_on_uptrend --features cuda --release
# PASSES
compute-sanitizer --tool=memcheck <test_bin>
# 0 CUDA errors across all 6 tests
# (2 PnL-assertion test failures are pre-existing data-expectation
# issues with new FEATURE_DIM=42, not OOB bugs)
Pearls honoured:
- feedback_no_hiding: null pointer surfaced via compute-sanitizer
instead of silently crashing in production
- feedback_no_partial_refactor: closure-path callers now have
self-contained ISV setup matching training path; consumer
migration (test FEATURE_DIM/portfolio_dim/set_branch_sizes)
atomic with the producer-side default ISV alloc
- pearl_no_deferrals_for_complementary_fixes: same SP cycle as
8.3+9, 8.5 (the layer-by-layer eval rot peeling)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Foxhunt
Production HFT trading system in Rust.
Architecture
The workspace contains 32 crates organized as follows:
Core Libraries (16)
| Crate | Purpose |
|---|---|
trading_engine |
Order processing, FIX 4.4, IB TWS, SIMD, RDTSC timing |
risk |
VaR, Kelly, circuit breakers, kill switches, compliance |
risk-data |
Risk data types and shared structures |
trading-data |
Trading data types |
ml |
DQN Rainbow, PPO, TFT, Mamba2, ensemble inference |
ml-data |
ML data types and feature definitions |
data |
Market data ingestion and storage |
backtesting |
Replay engine, strategy tester |
adaptive-strategy |
Ensemble execution, microstructure analysis |
common |
Shared types, resilience, error handling |
storage |
S3 and local model storage |
model_loader |
Model serialization and loading |
market-data |
Market data feed handlers |
database |
PostgreSQL access layer (SQLx) |
config |
Configuration management |
tli |
CLI commands and tooling |
Services (8)
| Service | Purpose |
|---|---|
backtesting_service |
gRPC backtesting service |
broker_gateway_service |
FIX routing, broker connectivity |
trading_service |
Core trading operations |
ml_training_service |
Model training orchestration |
data_acquisition_service |
Market data acquisition |
trading_agent_service |
Autonomous trading agents |
api_gateway |
gRPC API gateway with auth |
web-gateway |
Axum REST + WebSocket gateway |
Frontend
web-dashboard/ -- React 19 + TypeScript + Vite + TradingView charts.
Building
# Check compilation (no PostgreSQL required)
SQLX_OFFLINE=true cargo check --workspace
# Run tests for a specific crate
SQLX_OFFLINE=true cargo test -p <crate> --lib
# Clippy
SQLX_OFFLINE=true cargo clippy --workspace
ML Models
Four production model architectures on Candle v0.9.1 with CUDA:
- DQN Rainbow -- Deep Q-Network with prioritized replay, dueling heads, noisy nets
- PPO -- Proximal Policy Optimization with GAE and LSTM policies
- TFT -- Temporal Fusion Transformer for multi-horizon forecasting
- Mamba2 -- State space model for sequence prediction
Each model has a standalone trainer and a UnifiedTrainable adapter for the hyperopt pipeline.
Infrastructure
- Git: Gitea at
git.fxhnt.ai(Tailscale-only), Scaleway DEV1-S - Observability: OpenTelemetry OTLP (env
OTEL_EXPORTER_OTLP_ENDPOINT) - Database: PostgreSQL with SQLx offline mode for CI
License
Proprietary. All rights reserved.