jgrusewski bde510bf8e fix(cuda): harden PPO + hyperopt GPU fallbacks, fix gradient clip CPU roundtrip
Eliminate all CPU fallback paths in the PPO trainer, PPO hyperopt
adapter, and DQN hyperopt adapter. On H100, every GPU operation must
succeed or abort — silent CPU fallback runs at 1/10th throughput and
produces stale/inconsistent results.

PPO trainer (5 sites):
- GPU unavailable → hard error (was warn + CPU fallback)
- Collector init, episode reset, collection, weight sync → hard errors
- Test updated: GPU batch limit test expects error without CUDA

PPO hyperopt adapter (8 sites):
- ensure_gpu_data() now returns Result<(), MLError>
- Features/targets upload, collector init, weight sync → hard errors
- gpu_collect_trajectories() now returns Result<TrajectoryBatch>
- Experience collection caller uses ? instead of Option fallback
- GPU backtest failure → hard error

DQN hyperopt adapter (3 sites):
- CUDA synchronize between trials → hard error
- GPU backtest None on CUDA → hard error (upstream of extract_objective)
- extract_objective: panic! on CUDA build if backtest_metrics is None
- CPU backtest path guarded by #[cfg(not(feature = "cuda"))]

continuous_ppo.rs (1 site):
- clip_grads passed &Device::Cpu instead of actor's actual device
- Forces CPU roundtrip for gradient norm computation on every mini-batch
- Fixed: passes `device` (from self.actor.device()) — stays GPU-resident

gpu_experience_collector.rs (1 site):
- cuCtxSetLimit(STACK_SIZE) failure: warn → hard error
- 16KB stack is required — kernel segfaults without it

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 00:18:31 +01:00

Foxhunt

Production HFT trading system in Rust.

Architecture

The workspace contains 32 crates organized as follows:

Core Libraries (16)

Crate Purpose
trading_engine Order processing, FIX 4.4, IB TWS, SIMD, RDTSC timing
risk VaR, Kelly, circuit breakers, kill switches, compliance
risk-data Risk data types and shared structures
trading-data Trading data types
ml DQN Rainbow, PPO, TFT, Mamba2, ensemble inference
ml-data ML data types and feature definitions
data Market data ingestion and storage
backtesting Replay engine, strategy tester
adaptive-strategy Ensemble execution, microstructure analysis
common Shared types, resilience, error handling
storage S3 and local model storage
model_loader Model serialization and loading
market-data Market data feed handlers
database PostgreSQL access layer (SQLx)
config Configuration management
tli CLI commands and tooling

Services (8)

Service Purpose
backtesting_service gRPC backtesting service
broker_gateway_service FIX routing, broker connectivity
trading_service Core trading operations
ml_training_service Model training orchestration
data_acquisition_service Market data acquisition
trading_agent_service Autonomous trading agents
api_gateway gRPC API gateway with auth
web-gateway Axum REST + WebSocket gateway

Frontend

web-dashboard/ -- React 19 + TypeScript + Vite + TradingView charts.

Building

# Check compilation (no PostgreSQL required)
SQLX_OFFLINE=true cargo check --workspace

# Run tests for a specific crate
SQLX_OFFLINE=true cargo test -p <crate> --lib

# Clippy
SQLX_OFFLINE=true cargo clippy --workspace

ML Models

Four production model architectures on Candle v0.9.1 with CUDA:

  • DQN Rainbow -- Deep Q-Network with prioritized replay, dueling heads, noisy nets
  • PPO -- Proximal Policy Optimization with GAE and LSTM policies
  • TFT -- Temporal Fusion Transformer for multi-horizon forecasting
  • Mamba2 -- State space model for sequence prediction

Each model has a standalone trainer and a UnifiedTrainable adapter for the hyperopt pipeline.

Infrastructure

  • Git: Gitea at git.fxhnt.ai (Tailscale-only), Scaleway DEV1-S
  • Observability: OpenTelemetry OTLP (env OTEL_EXPORTER_OTLP_ENDPOINT)
  • Database: PostgreSQL with SQLx offline mode for CI

License

Proprietary. All rights reserved.

Description
No description provided
Readme 849 MiB
Languages
Rust 88.2%
Cuda 7.7%
Python 1.3%
Shell 1.1%
PLpgSQL 0.8%
Other 0.8%