8f93c11525
Curiosity stores params/grads/Adam state as 4 non-contiguous sub-buffers (w1, b1, w2, b2 — each its own CudaSlice<f32>). The single-pointer kernel signature couldn't describe them. The first fix-up wired 4 of 5 aux groups; Curiosity (group 7) was deferred. This commit extends the param_group_oracle kernel to accept a sub-buffer table (mapped-pinned u64 ptr-arrays + i32 counts), with `n_sub=1` for groups 0-6 (existing behavior) and `n_sub=4` for Curiosity. Pass A/B/C (p99 histograms) iterate sub-buffers via a new `sp4_histogram_p99_multi<BLOCK_SIZE>` template that mirrors the original three-pass structure but loops sub-buffers within the max-reduce + binning passes; Pass 3 (cumulative-from-top) divides by `total_count`. Pass D (4-way reduce) iterates sub-buffers in the accumulator loop; Pass E (L1 trunk only) reads `grads_ptrs[0]` (group 0 has n_sub=1 by construction). `Sp4ParamGroupBufs` redefined from a flat quartet to `Vec<Sp4SubBuffer>`. Convenience constructors `Sp4ParamGroupBufs::single(...)` and `Sp4ParamGroupBufs::empty()` keep call sites ergonomic; `total_count()` for kernel arg. Switched Copy → Clone since the descriptor now owns a Vec. `SP4AuxBuffers` extended with `curiosity` field (5-tuple from 4-tuple). Added 16 public accessors to GpuCuriosityTrainer (grad + Adam state) and 8 to CuriosityWeightSet (weights + lengths) for the 4 sub-buffers × 4 signal types. `oracle_subbuf_table_buf: MappedU64Buffer` (4×4 = 16 u64s) and `oracle_subbuf_counts_buf: MappedI32Buffer` (4 i32s) allocated at construction. Launcher overwrites entries [0..n_sub) per group launch via volatile writes; zeros the unused tail (defence-in-depth so a kernel bug reading past `n_sub` lands on count=0 no-op). Inter-launch `stream.synchronize()` added so the next iteration's host writes don't race with the in-flight kernel's coalesced loads from the persistent table. Cold-path producer; per-launch sync cost is negligible vs the kernel work. `build_sp4_aux_buffers` signature changed: takes `Option<&CuriosityWeightSet>` and `Option<&GpuCuriosityTrainer>` since Curiosity state lives outside FusedTrainingCtx (owned by GpuExperienceCollector). Layer B's training-loop caller threads them in from `collector.curiosity_weight_set()` + the collector's `curiosity_trainer` field; both must be Some together (caller responsibility — they live on the same collector). Passing None for either yields an empty curiosity descriptor and the launcher silently skips group 7. `param_group_buffers` return type changed from `Option<(u64, u64, u64, u64, usize, i32, i32)>` to `Option<(Sp4ParamGroupBufs, i32, i32)>`. All groups now return Some(...) (Curiosity included); None reserved for forward-compat. GPU test extended: group 7 exercises 4 sub-buffers of distinct shapes [1024, 32, 1024, 32] (w1/b1/w2/b2-like sizes scaled to keep test runtime small while still exercising the multi-sub-buffer iteration), each with its own seed offset so distinct sub-buffers have distinct distributions — catches buffer-mixup bugs in the kernel's sub-buffer iteration. Reference computation builds union vectors and computes p99/WD_RATE over the union. Test launcher refactored to `&[TestSubBuffer]` slice matching the production kernel's table-packing layout. All 8 SP4 param-groups now produce real outputs in Layer A. The launcher's `count == 0` short-circuit retained for the optional aux- trainer fallback (init failures on gpu_iqn / gpu_attention / curiosity). `MappedU64Buffer` gained a manual Debug impl (warn missing_debug_impls) for parity with MappedU32Buffer. cargo check -p ml --lib --tests clean. Workspace clean. Layer B can now safely consume all 8 ISV[WEIGHT_BOUND/ADAM_M_BOUND/ADAM_V_BOUND/WD_RATE[group]] slots. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Foxhunt
Production HFT trading system in Rust.
Architecture
The workspace contains 32 crates organized as follows:
Core Libraries (16)
| Crate | Purpose |
|---|---|
trading_engine |
Order processing, FIX 4.4, IB TWS, SIMD, RDTSC timing |
risk |
VaR, Kelly, circuit breakers, kill switches, compliance |
risk-data |
Risk data types and shared structures |
trading-data |
Trading data types |
ml |
DQN Rainbow, PPO, TFT, Mamba2, ensemble inference |
ml-data |
ML data types and feature definitions |
data |
Market data ingestion and storage |
backtesting |
Replay engine, strategy tester |
adaptive-strategy |
Ensemble execution, microstructure analysis |
common |
Shared types, resilience, error handling |
storage |
S3 and local model storage |
model_loader |
Model serialization and loading |
market-data |
Market data feed handlers |
database |
PostgreSQL access layer (SQLx) |
config |
Configuration management |
tli |
CLI commands and tooling |
Services (8)
| Service | Purpose |
|---|---|
backtesting_service |
gRPC backtesting service |
broker_gateway_service |
FIX routing, broker connectivity |
trading_service |
Core trading operations |
ml_training_service |
Model training orchestration |
data_acquisition_service |
Market data acquisition |
trading_agent_service |
Autonomous trading agents |
api_gateway |
gRPC API gateway with auth |
web-gateway |
Axum REST + WebSocket gateway |
Frontend
web-dashboard/ -- React 19 + TypeScript + Vite + TradingView charts.
Building
# Check compilation (no PostgreSQL required)
SQLX_OFFLINE=true cargo check --workspace
# Run tests for a specific crate
SQLX_OFFLINE=true cargo test -p <crate> --lib
# Clippy
SQLX_OFFLINE=true cargo clippy --workspace
ML Models
Four production model architectures on Candle v0.9.1 with CUDA:
- DQN Rainbow -- Deep Q-Network with prioritized replay, dueling heads, noisy nets
- PPO -- Proximal Policy Optimization with GAE and LSTM policies
- TFT -- Temporal Fusion Transformer for multi-horizon forecasting
- Mamba2 -- State space model for sequence prediction
Each model has a standalone trainer and a UnifiedTrainable adapter for the hyperopt pipeline.
Infrastructure
- Git: Gitea at
git.fxhnt.ai(Tailscale-only), Scaleway DEV1-S - Observability: OpenTelemetry OTLP (env
OTEL_EXPORTER_OTLP_ENDPOINT) - Database: PostgreSQL with SQLx offline mode for CI
License
Proprietary. All rights reserved.