jgrusewski fa1f299147 fix(sp7-cadence): migrate SP5 Pearl 2 budget + SP7 loss-balance controller from process_epoch_boundary to per-step submit_aux_ops
Bug audit finding #2 (post train-d2b2s diagnostic — same Pattern 1
class as SP14 B.11 commit 200f05fce and the 4-producer batch in
5608b866b). SP5 Pearl 2 budget producer and SP7 loss-balance
controller currently launch from process_epoch_boundary (fires once
per epoch), but the loss-balance budget output (ISV[BUDGET_CQL_BASE..]
/ ISV[BUDGET_C51_BASE..]) is consumed EVERY training step via
apply_c51_budget_scale (fused_training.rs:1941) and the dispatch
kernel that resolves the cached value into lb_budget_effective_buf
(fused_training.rs:3631).

The dispatch kernel is per-step, but the underlying flatness signal
ISV[FLATNESS_BASE..] is per-epoch. SP7's controller therefore reads
(steps_per_epoch − 1)-step-stale flatness — the same failure mode
that broke SP14's ALPHA_GRAD_SMOOTHED. The entire loss-balance
budget system has been operating on stale-flatness state for the
duration of training.

Migration (atomic, preserves Pearl 2 → SP7 dependency):
- Pearl 2 budget launch moved to submit_aux_ops (just-after the
  producer-cadence batch's MoE chain, just-before the IQL gather
  block).
- SP7 loss-balance controller follows immediately (reads Pearl 2
  output via ISV).
- Same captured-into-aux_child graph-replay semantics as SP14 B.11.
- training_loop.rs lines 4312, 4336 deleted; replaced with redirect
  comment.

Per feedback_no_partial_refactor: this is the 7th cadence-fix in
this branch since v8ztm. Other Pattern 1 candidates (SP5 Pearl 1
atom, Pearl 3 σ — Pearl 2 inputs, AND SP8 Fix 36 launch_max_budget_compute
— SP7 controller input) deferred per scope-tightening rule. They
sit one-epoch-stale at fold start; Pearl A bootstrap + the
controller's internal cold-start branch keep behavior functional.
Tracked in the audit doc top-of-file entry; independent migrations,
land separately.

Verification: cargo check -p ml --lib clean (dev profile);
sp14_oracle_tests 7/7 pass on RTX 3050 Ti.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 23:19:25 +02:00

Foxhunt

Production HFT trading system in Rust.

Architecture

The workspace contains 32 crates organized as follows:

Core Libraries (16)

Crate Purpose
trading_engine Order processing, FIX 4.4, IB TWS, SIMD, RDTSC timing
risk VaR, Kelly, circuit breakers, kill switches, compliance
risk-data Risk data types and shared structures
trading-data Trading data types
ml DQN Rainbow, PPO, TFT, Mamba2, ensemble inference
ml-data ML data types and feature definitions
data Market data ingestion and storage
backtesting Replay engine, strategy tester
adaptive-strategy Ensemble execution, microstructure analysis
common Shared types, resilience, error handling
storage S3 and local model storage
model_loader Model serialization and loading
market-data Market data feed handlers
database PostgreSQL access layer (SQLx)
config Configuration management
tli CLI commands and tooling

Services (8)

Service Purpose
backtesting_service gRPC backtesting service
broker_gateway_service FIX routing, broker connectivity
trading_service Core trading operations
ml_training_service Model training orchestration
data_acquisition_service Market data acquisition
trading_agent_service Autonomous trading agents
api_gateway gRPC API gateway with auth
web-gateway Axum REST + WebSocket gateway

Frontend

web-dashboard/ -- React 19 + TypeScript + Vite + TradingView charts.

Building

# Check compilation (no PostgreSQL required)
SQLX_OFFLINE=true cargo check --workspace

# Run tests for a specific crate
SQLX_OFFLINE=true cargo test -p <crate> --lib

# Clippy
SQLX_OFFLINE=true cargo clippy --workspace

ML Models

Four production model architectures on Candle v0.9.1 with CUDA:

  • DQN Rainbow -- Deep Q-Network with prioritized replay, dueling heads, noisy nets
  • PPO -- Proximal Policy Optimization with GAE and LSTM policies
  • TFT -- Temporal Fusion Transformer for multi-horizon forecasting
  • Mamba2 -- State space model for sequence prediction

Each model has a standalone trainer and a UnifiedTrainable adapter for the hyperopt pipeline.

Infrastructure

  • Git: Gitea at git.fxhnt.ai (Tailscale-only), Scaleway DEV1-S
  • Observability: OpenTelemetry OTLP (env OTEL_EXPORTER_OTLP_ENDPOINT)
  • Database: PostgreSQL with SQLx offline mode for CI

License

Proprietary. All rights reserved.

Description
No description provided
Readme 849 MiB
Languages
Rust 88.2%
Cuda 7.7%
Python 1.3%
Shell 1.1%
PLpgSQL 0.8%
Other 0.8%