1bcc703920
Real fix for the F1 step-3540 NaN. Commit 8956c2fb7 wired Mech 9
(post-Adam |p_val| clamp) into ONE Adam kernel (dqn_adam_update_kernel)
out of FIVE in the codebase. The slot-26 GEMM (apply_iqn_trunk_gradient
output) computes `grad_iqn @ W_iqn^T`; W_iqn lives in IQN's separate
param buffer, updated by iqn_adam_kernel — never reached by Mech 9.
Slots 44-45 only cover trunk + heads, leaving IQN weights without a
diagnostic. The chain: IQN weights drift via iqn_adam_kernel → finite
gradient × non-finite weight → slot-26 NaN. Same partial-refactor class
as Mech 4's missing GpuAttention reset (caught by B5 audit), corrected
the same way: every consumer of the contract migrates together.
Extended Mech 9 to all four remaining Adam kernels:
- iqn_adam_kernel (iqn_dual_head_kernel.cu)
- iql_adam_kernel (iql_value_kernel.cu)
- attn_adam_kernel (attention_backward_kernel.cu — used by GpuAttention + GpuTlob)
- curiosity_adam_step (curiosity_training_kernel.cu)
Same `if (weight_clamp_max_abs > 0.0f) p = fminf(fmaxf(p, -bound), bound)`
pattern. Same `100 × Q_ABS_REF.max(1.0)` ISV-driven bound. Each launch
site computes the bound host-side and passes it as the trailing kernel
arg, mirroring the existing dqn_adam_update_kernel pattern. DT launch
keeps the 0.0 disable (offline-RL, outside SP3 scope). Curiosity reads
ISV from FusedTrainingCtx in training_loop and threads through the
collector wrapper (collector doesn't own ISV).
Added IQN-weight diagnostic slot 48:
- nan_flags_buf 48 → 49
- Fused-kernel block count 24 → 25; new branch for absolute slot 48
(relative slot 24): threshold = 1e3 × q_abs_ref_eff (matches slot 44-45)
- Metadata buffers (nan_check_buf_ptrs/_lens) populate slot 48 with
IQN online_params pointer + length (new public accessors on GpuIqnHead)
- name table (halt_grad_collapse): 49 entries with "iqn_weight_max"
- Audit doc: slot 48 row in Mech 5 table; Mech 9 row updated to span
all 5 Adam kernels
No new ISV slots. No graph-topology change beyond the +1 block in the
already-fused NaN check. cargo check -p ml --lib clean (12 pre-existing
warnings, none new).
Validation: deferred to one L40S smoke at this HEAD. Expected: F1
trains past step 3720 (Mech-6-only ceiling) and slots 36-43 + new
slot 48 stay quiet.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Foxhunt
Production HFT trading system in Rust.
Architecture
The workspace contains 32 crates organized as follows:
Core Libraries (16)
| Crate | Purpose |
|---|---|
trading_engine |
Order processing, FIX 4.4, IB TWS, SIMD, RDTSC timing |
risk |
VaR, Kelly, circuit breakers, kill switches, compliance |
risk-data |
Risk data types and shared structures |
trading-data |
Trading data types |
ml |
DQN Rainbow, PPO, TFT, Mamba2, ensemble inference |
ml-data |
ML data types and feature definitions |
data |
Market data ingestion and storage |
backtesting |
Replay engine, strategy tester |
adaptive-strategy |
Ensemble execution, microstructure analysis |
common |
Shared types, resilience, error handling |
storage |
S3 and local model storage |
model_loader |
Model serialization and loading |
market-data |
Market data feed handlers |
database |
PostgreSQL access layer (SQLx) |
config |
Configuration management |
tli |
CLI commands and tooling |
Services (8)
| Service | Purpose |
|---|---|
backtesting_service |
gRPC backtesting service |
broker_gateway_service |
FIX routing, broker connectivity |
trading_service |
Core trading operations |
ml_training_service |
Model training orchestration |
data_acquisition_service |
Market data acquisition |
trading_agent_service |
Autonomous trading agents |
api_gateway |
gRPC API gateway with auth |
web-gateway |
Axum REST + WebSocket gateway |
Frontend
web-dashboard/ -- React 19 + TypeScript + Vite + TradingView charts.
Building
# Check compilation (no PostgreSQL required)
SQLX_OFFLINE=true cargo check --workspace
# Run tests for a specific crate
SQLX_OFFLINE=true cargo test -p <crate> --lib
# Clippy
SQLX_OFFLINE=true cargo clippy --workspace
ML Models
Four production model architectures on Candle v0.9.1 with CUDA:
- DQN Rainbow -- Deep Q-Network with prioritized replay, dueling heads, noisy nets
- PPO -- Proximal Policy Optimization with GAE and LSTM policies
- TFT -- Temporal Fusion Transformer for multi-horizon forecasting
- Mamba2 -- State space model for sequence prediction
Each model has a standalone trainer and a UnifiedTrainable adapter for the hyperopt pipeline.
Infrastructure
- Git: Gitea at
git.fxhnt.ai(Tailscale-only), Scaleway DEV1-S - Observability: OpenTelemetry OTLP (env
OTEL_EXPORTER_OTLP_ENDPOINT) - Database: PostgreSQL with SQLx offline mode for CI
License
Proprietary. All rights reserved.