d60e5375a9
Adds grad_mag_{iqn,cql,c51,ens} HEALTH_DIAG fields via in-graph
pinned-snapshot + in-graph reduction kernel (revised approach; first
Task 2.0 dispatch escalated BLOCKED on host-side-snapshots-inside-
captured-graph, plan revised at 5a4d56145 to use this pattern).
How it works:
* Three CudaSlice<f32>[branch01_elems] per-component scratch buffers
(iqn, cql, ens) + one permanent-zeros reference buffer for C51
(grad_buf is memset-zeroed at start of submit_forward_ops_main —
no pre-C51 snapshot needed). Snapshots only cover the branch 0+1
slice (direction+magnitude, tensors 8..16), NOT TOTAL_PARAMS —
~420 KB per snapshot, 4× = ~1.7 MB device memory total.
* BEFORE each component's backward (captured in graph):
graph_safe_copy_f32 kernel copies grad_buf[branch01] → scratch.
C51 uses the permanent-zeros reference buffer directly.
* AFTER each component's backward (captured in graph):
grad_component_delta_norm kernel (256 threads, one block) reads
(grad_buf[dir_slice] − snapshot[0..dir_len]) and
(grad_buf[mag_slice] − snapshot[dir_len..]), reduces via
shared-memory tree reduction (NO atomicAdd), writes
(mag_norm, dir_norm) into a pinned device-mapped 8-float result
slot at the component's 2-float offset.
* At epoch boundary: refresh_grad_component_norms() stream-syncs,
reads the pinned slot (zero-copy via cuMemHostGetDevicePointer),
populates host-side grad_component_norms_mag/_dir caches.
* HEALTH_DIAG emits `grad_split [iqn=... cql=... c51=... ens=...]`
with per-component magnitude/direction ratios.
Uses graph_safe_copy_f32 (already existing — see gpu_dqn_trainer.rs:3839
doc comment: "memcpy_dtod_async is NOT captured by CUDA Graph"). The
plan's claim that DtoD memcpy is captureable is contradicted by the
codebase's own history, so we follow the established pattern.
Matches Task 0.5 / 0.8 no-atomics pattern; matches Task 0.4's pinned
device-mapped pattern for kernel-written readback. New file:
crates/ml/src/cuda_pipeline/grad_decomp_kernel.cu (72 LOC, one kernel
+ header doc).
Zero per-step PCIe traffic: the 8-float pinned slot is device-mapped;
writes flow via the mapped device pointer during graph replay, reads
are direct host dereferences at epoch boundary.
20-epoch grad_split trajectory — final fold, epochs 15–19:
grad_split [iqn=0.0000 cql=33.5947 c51=418.0461 ens=0.0000]
grad_split [iqn=0.0000 cql=65.2418 c51=222.4561 ens=0.0000]
grad_split [iqn=0.0000 cql=23.8915 c51=145.3589 ens=0.0000]
grad_split [iqn=0.0000 cql=46.1190 c51=268.8549 ens=0.0000]
grad_split [iqn=0.0000 cql=147.4817 c51=213.1188 ens=0.0000]
Findings for Task 2.1 decision tree:
* IQN and Ensemble report 0.0000 ratios every epoch — architecturally
expected: apply_iqn_trunk_gradient writes only to trunk tensors 0..4
(w_s1/b_s1/w_s2/b_s2); run_ensemble_step flows through the value
head only (tensors 4..8). Neither touches branches 8..24 by design.
* C51 and CQL are the ONLY two loss components that write directly
to branch head weights — and both send MORE gradient to magnitude
than to direction (ratios routinely 50–400×).
* Task 0.4's grad_ratio_mag_dir=0.0000 (reported post-all-accumulations,
post-budget-scaling) vs per-component ratios >>1 shifts the
diagnosis: gradient IS reaching magnitude from C51/CQL — the
global mag/dir=0 at epoch boundary reflects cancellation with
other writers (distill SAXPY, recursive_confidence_backward,
predictive_coding_loss) that accumulate into grad_buf BETWEEN
CQL and Ens, AND the heavy c51_budget_scale (0.70) + cql_budget
(0.04) scaling that is applied AFTER our reduction.
This is H9 territory (data-driven magnitude Q-value collapse) more
than H4 (architectural gradient starvation) — the Task 2.1 decision
tree should pick branches A or C (init scale / direction-conditioning)
only after excluding data-driven factors, or skip straight to the
"revisit in Phase 3" row.
Per plan Task 2.0 (revised at 5a4d56145). Build clean; magnitude
distribution smoke green (21.10s local, RTX 3050 Ti).
Foxhunt
Production HFT trading system in Rust.
Architecture
The workspace contains 32 crates organized as follows:
Core Libraries (16)
| Crate | Purpose |
|---|---|
trading_engine |
Order processing, FIX 4.4, IB TWS, SIMD, RDTSC timing |
risk |
VaR, Kelly, circuit breakers, kill switches, compliance |
risk-data |
Risk data types and shared structures |
trading-data |
Trading data types |
ml |
DQN Rainbow, PPO, TFT, Mamba2, ensemble inference |
ml-data |
ML data types and feature definitions |
data |
Market data ingestion and storage |
backtesting |
Replay engine, strategy tester |
adaptive-strategy |
Ensemble execution, microstructure analysis |
common |
Shared types, resilience, error handling |
storage |
S3 and local model storage |
model_loader |
Model serialization and loading |
market-data |
Market data feed handlers |
database |
PostgreSQL access layer (SQLx) |
config |
Configuration management |
tli |
CLI commands and tooling |
Services (8)
| Service | Purpose |
|---|---|
backtesting_service |
gRPC backtesting service |
broker_gateway_service |
FIX routing, broker connectivity |
trading_service |
Core trading operations |
ml_training_service |
Model training orchestration |
data_acquisition_service |
Market data acquisition |
trading_agent_service |
Autonomous trading agents |
api_gateway |
gRPC API gateway with auth |
web-gateway |
Axum REST + WebSocket gateway |
Frontend
web-dashboard/ -- React 19 + TypeScript + Vite + TradingView charts.
Building
# Check compilation (no PostgreSQL required)
SQLX_OFFLINE=true cargo check --workspace
# Run tests for a specific crate
SQLX_OFFLINE=true cargo test -p <crate> --lib
# Clippy
SQLX_OFFLINE=true cargo clippy --workspace
ML Models
Four production model architectures on Candle v0.9.1 with CUDA:
- DQN Rainbow -- Deep Q-Network with prioritized replay, dueling heads, noisy nets
- PPO -- Proximal Policy Optimization with GAE and LSTM policies
- TFT -- Temporal Fusion Transformer for multi-horizon forecasting
- Mamba2 -- State space model for sequence prediction
Each model has a standalone trainer and a UnifiedTrainable adapter for the hyperopt pipeline.
Infrastructure
- Git: Gitea at
git.fxhnt.ai(Tailscale-only), Scaleway DEV1-S - Observability: OpenTelemetry OTLP (env
OTEL_EXPORTER_OTLP_ENDPOINT) - Database: PostgreSQL with SQLx offline mode for CI
License
Proprietary. All rights reserved.