396f0d09b5
Two related changes:
1) Determinism fix in barrier_gradient_direction + ib_gradient_direction
(c51_loss_kernel.cu)
Both kernels run ONE thread per sample i. Each thread writes to memory
regions [i, *, *] that are unique to that sample — no cross-thread races
are possible. The atomicAdd was overkill: within a single thread,
d_v_row[z] is written 2× (barrier) or up to b0_size× (IB) sequentially,
and d_adv_a[z] writes are to unique (a, z) slots per target/action.
Replaced with: accumulate d_v contributions in a register-sized local
array (MAX_ATOMS=128), then plain += writes once per z. d_adv_a uses
plain += directly (unique slot per write). No correctness change at all
— same gradient contributions in same order — but fully deterministic
(no atomic ordering effects on float reduction) and faster (atomicAdd
serializes on shared memory).
Of the 69 "atomicAdd" string occurrences in the codebase, 56 are in
COMMENTS (most saying "no atomicAdd" or describing what was removed).
Real call-site count was 13. After this commit: 9 remain. Of those:
- 5 in monitoring_kernel: diagnostic stats only, no training-path impact
- 1 in ensemble_kernels: diversity_loss per-block reduction (true cross-block accum)
- 3 in experience_kernels: atom_stats + penalty_out (diagnostic-ish)
The remaining 4 training-path atomics use the standard hierarchical
warp+block reduction pattern; making them deterministic requires the
two-pass per-block sum + deterministic reduction pattern (same as MSE
loss already uses). Doable but ~50-100 LOC each, deferred.
2) Multi-trial statistical test (td_propagation.rs)
Refactor: extract `run_one_trial() -> TrialMetrics` so the per-trial
logic is callable from both single- and multi-trial entry points.
New: `test_td_propagation_sparse_rewards_multi_trial` (#[ignore], ~3 min
runtime on RTX 3050 Ti) runs 5 independent trials and asserts on the
*distribution* of outcomes, not single-run values:
- ALL trials must produce finite sharpe_ema (NaN/Inf is a hard bug)
- Median q_gap > 0.05 (median is robust to single-run outliers)
- q_gap pass rate ≥ 80% on the 0.02 single-run threshold
- Mean sharpe_ema across trials > -10 (catches systematic divergence)
This decouples "did the algorithm work?" from "did this particular RNG
state produce a profitable model?" — the same pattern RL benchmark
suites use. Expected outcome: the determinism fix in (1) reduces the
variance enough that the median assertion is stable, and the multi-trial
median is a reliable indicator for future A/B comparisons of model
changes.
The original single-trial test is preserved for fast iteration ("did
I break compile / catastrophic regression").
Files touched:
crates/ml/src/cuda_pipeline/c51_loss_kernel.cu (+33 / -12)
crates/ml/src/trainers/dqn/smoke_tests/td_propagation.rs (+143 / -51)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Foxhunt
Production HFT trading system in Rust.
Architecture
The workspace contains 32 crates organized as follows:
Core Libraries (16)
| Crate | Purpose |
|---|---|
trading_engine |
Order processing, FIX 4.4, IB TWS, SIMD, RDTSC timing |
risk |
VaR, Kelly, circuit breakers, kill switches, compliance |
risk-data |
Risk data types and shared structures |
trading-data |
Trading data types |
ml |
DQN Rainbow, PPO, TFT, Mamba2, ensemble inference |
ml-data |
ML data types and feature definitions |
data |
Market data ingestion and storage |
backtesting |
Replay engine, strategy tester |
adaptive-strategy |
Ensemble execution, microstructure analysis |
common |
Shared types, resilience, error handling |
storage |
S3 and local model storage |
model_loader |
Model serialization and loading |
market-data |
Market data feed handlers |
database |
PostgreSQL access layer (SQLx) |
config |
Configuration management |
tli |
CLI commands and tooling |
Services (8)
| Service | Purpose |
|---|---|
backtesting_service |
gRPC backtesting service |
broker_gateway_service |
FIX routing, broker connectivity |
trading_service |
Core trading operations |
ml_training_service |
Model training orchestration |
data_acquisition_service |
Market data acquisition |
trading_agent_service |
Autonomous trading agents |
api_gateway |
gRPC API gateway with auth |
web-gateway |
Axum REST + WebSocket gateway |
Frontend
web-dashboard/ -- React 19 + TypeScript + Vite + TradingView charts.
Building
# Check compilation (no PostgreSQL required)
SQLX_OFFLINE=true cargo check --workspace
# Run tests for a specific crate
SQLX_OFFLINE=true cargo test -p <crate> --lib
# Clippy
SQLX_OFFLINE=true cargo clippy --workspace
ML Models
Four production model architectures on Candle v0.9.1 with CUDA:
- DQN Rainbow -- Deep Q-Network with prioritized replay, dueling heads, noisy nets
- PPO -- Proximal Policy Optimization with GAE and LSTM policies
- TFT -- Temporal Fusion Transformer for multi-horizon forecasting
- Mamba2 -- State space model for sequence prediction
Each model has a standalone trainer and a UnifiedTrainable adapter for the hyperopt pipeline.
Infrastructure
- Git: Gitea at
git.fxhnt.ai(Tailscale-only), Scaleway DEV1-S - Observability: OpenTelemetry OTLP (env
OTEL_EXPORTER_OTLP_ENDPOINT) - Database: PostgreSQL with SQLx offline mode for CI
License
Proprietary. All rights reserved.