6a58ac9465
mjzfk diag (commit 53aeef099) showed PPO clip ε pegged at MAX=0.5
for 100% of the 50k-step run, despite kl_pi_ema median = 5.3e-9 and
max = 3.4e-4 (well below KL_TARGET=0.01). The widened clip band is
why ratio_clamp settled at (1+0.5)*10=15 instead of 12, and why π
took no meaningful updates — KL was essentially zero meaning the
policy wasn't moving.
## Root cause
The controller's adaptation was `ratio = KL_TARGET / max(kl_ema, 1e-6)`
— a multiplicative formula with no per-step bound. With kl_ema=1e-11
the kernel sees:
ratio = 0.01 / max(1e-11, 1e-6) = 0.01 / 1e-6 = 10000
target = eps_prev * 10000 = clamped to EPS_MAX
First-observation replace-directly then locks ε at MAX immediately,
and the Wiener α-floor=0.4 blend keeps it there forever.
The cold-start `if (kl_ema == 0.0f) return;` gate only caught EXACT
zero — first observable but tiny KL (typical: cold-start LR not yet
producing measurable policy drift) blows past the gate and saturates
the multiplier.
Same dangerous pattern existed in `rl_target_tau_controller` (uses
`q_div / DIV_TARGET` ratio with no per-step bound) — hadn't bitten
because q_divergence_norm naturally lives in the [0.01, 0.1] range,
but a quiet initialisation could hit it the same way.
## Fix: Schulman-style bounded adaptive KL
Both controllers now use a discrete-step adjustment:
* input > target × TOLERANCE (1.5) → ratio = ADJUST_RATE (1.5)
* input < target / TOLERANCE → ratio = 1/ADJUST_RATE
* in-band → ratio = 1.0 (hold)
Per-step adjustment is bounded at 1.5× (50% expansion / 33%
shrinkage), so no single observation can swing the output across the
[MIN, MAX] range regardless of how outlier-tiny or outlier-huge it
is. After several consecutive out-of-band observations the output
drifts smoothly toward MIN/MAX, but the response is dampened.
## Noise-floor gate
In addition to the bounded step, both controllers now hold their
output when the input EMA is below a noise floor:
KL_NOISE_FLOOR = KL_TARGET × 0.01 = 1e-4 (ppo_clip)
DIV_NOISE_FLOOR = DIV_TARGET × 0.01 = 1e-4 (target_tau)
Two orders of magnitude below the design target = "policy isn't
actually updating" / "Q hasn't started learning" / numerical noise.
Reacting to this signal can only mis-tune the controller — we'd
rather hold a sane default than chase noise.
The TARGET-derived floors (rather than absolute constants) mean
adjusting KL_TARGET / DIV_TARGET shifts the floors proportionally —
consistent with the existing pattern.
## ISV discipline
Per `feedback_isv_for_adaptive_bounds`: KL_TARGET and DIV_TARGET
themselves are PPO/DQN design constants (like REWARD_CLAMP_WIN =
1.0 in apply_reward_scale, or LR_BOOTSTRAP = 1e-3 in
rl_lr_controller). The NOISE_FLOOR derives from them, and the
TOLERANCE / ADJUST_RATE constants are structural Schulman-recipe
parameters that don't adapt at runtime. Existing diag exposes both
controllers' outputs (isv_out.ppo_clip_eps, isv_out.target_tau) and
inputs (isv_ema_in.kl_pi, isv_ema_in.q_divergence) so the
controller behaviour is fully observable from the JSONL — no new
slots needed.
## Verified gates (local sm_86)
G1 isv_bootstrap ✅
G3 controllers ✅ (controllers still move outputs when fed real EMAs)
G4 target_update ✅
integrated_smoke ✅
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Foxhunt
Production HFT trading system in Rust.
Architecture
The workspace contains 32 crates organized as follows:
Core Libraries (16)
| Crate | Purpose |
|---|---|
trading_engine |
Order processing, FIX 4.4, IB TWS, SIMD, RDTSC timing |
risk |
VaR, Kelly, circuit breakers, kill switches, compliance |
risk-data |
Risk data types and shared structures |
trading-data |
Trading data types |
ml |
DQN Rainbow, PPO, TFT, Mamba2, ensemble inference |
ml-data |
ML data types and feature definitions |
data |
Market data ingestion and storage |
backtesting |
Replay engine, strategy tester |
adaptive-strategy |
Ensemble execution, microstructure analysis |
common |
Shared types, resilience, error handling |
storage |
S3 and local model storage |
model_loader |
Model serialization and loading |
market-data |
Market data feed handlers |
database |
PostgreSQL access layer (SQLx) |
config |
Configuration management |
tli |
CLI commands and tooling |
Services (8)
| Service | Purpose |
|---|---|
backtesting_service |
gRPC backtesting service |
broker_gateway_service |
FIX routing, broker connectivity |
trading_service |
Core trading operations |
ml_training_service |
Model training orchestration |
data_acquisition_service |
Market data acquisition |
trading_agent_service |
Autonomous trading agents |
api_gateway |
gRPC API gateway with auth |
web-gateway |
Axum REST + WebSocket gateway |
Frontend
web-dashboard/ -- React 19 + TypeScript + Vite + TradingView charts.
Building
# Check compilation (no PostgreSQL required)
SQLX_OFFLINE=true cargo check --workspace
# Run tests for a specific crate
SQLX_OFFLINE=true cargo test -p <crate> --lib
# Clippy
SQLX_OFFLINE=true cargo clippy --workspace
ML Models
Four production model architectures on Candle v0.9.1 with CUDA:
- DQN Rainbow -- Deep Q-Network with prioritized replay, dueling heads, noisy nets
- PPO -- Proximal Policy Optimization with GAE and LSTM policies
- TFT -- Temporal Fusion Transformer for multi-horizon forecasting
- Mamba2 -- State space model for sequence prediction
Each model has a standalone trainer and a UnifiedTrainable adapter for the hyperopt pipeline.
Infrastructure
- Git: Gitea at
git.fxhnt.ai(Tailscale-only), Scaleway DEV1-S - Observability: OpenTelemetry OTLP (env
OTEL_EXPORTER_OTLP_ENDPOINT) - Database: PostgreSQL with SQLx offline mode for CI
License
Proprietary. All rights reserved.