235e838422
Atomic Phase 1.4 commit per `feedback_no_partial_refactor`. Wires the
SP20 fused-producer chain (Stats → Aggregate → EMAs → Controllers) into
GpuExperienceCollector's per-rollout-step path with one new aggregation
kernel + Phase 1.2 EMA kernel-signature refactor + production wire-up
+ tests + audit/spec/plan amendments, all in one commit.
## Path C decision rationale
The Phase 1.2 EMA kernel originally took 9 scalar value-args. The Phase
1.4 wire-up site (per-rollout-step in GpuExperienceCollector) needs to
feed per-env GPU-resident trade signals (trade_close_per_sample,
step_ret_per_sample, hold_at_exit_per_sample, packed factored
actions_out) into the kernel — forbidden via host sync
(`feedback_cpu_is_read_only`) and via memcpy_dtoh/htod
(`feedback_no_htod_htoh_only_mapped_pinned`). Path C refactors the EMA
kernel to take a device struct pointer (`SP20EmaInputs* ema_inputs`) +
adds an aggregation kernel that writes the struct on the GPU. Phase 1.2
had zero production consumers yet, so the sig change + Phase 1.4 wire-up
land atomically.
## What this commit contains
1. **EMA kernel signature refactor** (Phase 1.2 → Path C):
`sp20_emas_compute_kernel.cu` swaps 9 scalar args for a single
`const SP20EmaInputs* __restrict__ ema_inputs` device pointer.
Math semantics bit-identical. Rust `EmaInputs` mirrored as
`#[repr(C)]` byte-for-byte; new `pack_inputs_into_f32_view` helper
for tests + collectors that fill the struct via a mapped-pinned
f32-aliased buffer.
2. **New aggregation kernel** (`sp20_aggregate_inputs_kernel.cu`):
Per-env arrays (sliced to current rollout step) + sp20_stats outputs
(p50/std) + aux_dir_acc_reduce_kernel output [0] → SP20EmaInputs
struct. Block tree-reduce (4 stripes × bdim × 4 bytes shmem); no
atomicAdd. Aggregation rules per spec §4.5:
- is_close = OR over envs
- is_win = (≥ 0.5 of closed envs were wins)
- trade_duration = round(mean(hold_at_exit) over closed envs)
- action_is_hold = strict majority (count*2 > n_envs) HOLD
- alpha = 0.0 (Phase 2 forward ref — reward kernel)
- per_bar_hold_reward = 0.0 (Phase 3.2 forward ref — Hold-cost dual)
- aux_logits_p50/std/dir_acc forwarded from upstream
3. **Production wire-up in GpuExperienceCollector**:
4 kernel handles + 4 mapped-pinned buffers (struct fields, alloc,
init); per-rollout-step launch sequence after env_step (step 5c):
`Stats → Aggregate → EMAs → Controllers`. All on the same stream;
stream-implicit producer→consumer ordering. Gated on
`isv_signals_dev_ptr != 0 && trainer_params_ptr != 0` (matches the
existing SP14-β EGF chain pattern).
4. **Fold-boundary reset**:
3 new RegistryEntry records (sp20_ema_inputs_buf,
sp20_emas_internal_buf, sp20_emas_obs_count_buf) + 13 new dispatch
arms in training_loop.rs (10 SP20 ISV slot resets + 3 buffer resets).
Closes the pre-existing `every_fold_and_soft_reset_entry_has_dispatch_arm`
regression (was failing on fresh check after the SP20 ISV slot
registrations landed in commit 4249ebc96).
5. **HEALTH_DIAG emit**:
Per-epoch `HEALTH_DIAG[N]: sp20_isv [loss_cap=… alpha_ema=…
wr_ema=… hold_cost_scale=… target_hold_pct=… hold_pct_ema=…
hold_reward_ema=… n_step=… aux_conf_threshold=… aux_gate_temp=…]`
right after the existing q_disagreement_diag emit.
6. **Tests** (5 test files, all green on RTX 3050 Ti sm_86):
- sp20_emas_compute_test.rs (4 GPU oracle): updated to use the
post-Path-C buffer-arg API (mapped-pinned f32-aliased struct).
- sp20_aggregate_inputs_test.rs (NEW, 6 GPU oracle): aggregation
rule coverage including the Phase 2 / Phase 3.2 placeholder
contract.
- sp20_phase1_4_wireup_test.rs (NEW, 2 GPU oracle): end-to-end
4-kernel chain integration test.
- sp20_stats_compute_test.rs (4 GPU oracle): unchanged, regression.
- sp20_controllers_compute_test.rs (7 GPU oracle): unchanged,
regression.
- 18 lib unit tests across the SP20 launchers.
7. **Phase 2 / Phase 3.2 forward references**:
The `alpha` (Phase 2) and `per_bar_hold_reward` (Phase 3.2) fields
are emitted as 0.0 placeholders and documented at:
- `sp20_aggregate_inputs_kernel.cu:46-52` (in-kernel docstring)
- `sp20_aggregate_inputs.rs:46-49` (launcher docstring)
- `dqn-wire-up-audit.md` "Phase 2 / Phase 3.2 forward references"
These are NOT stubs — Phase 2 / 3.2 will replace the kernel's 0.0
writes with real signals atomically with their respective producers.
The original Task 2.3 (host-side aggregation) is **subsumed** by
the GPU-side aggregation kernel.
## Hard rules
- `feedback_no_partial_refactor` — kernel sig change + aggregation
kernel + production wire-up + tests + audit/spec/plan amendments
in one atomic commit
- `feedback_no_atomicadd` — aggregation kernel uses block tree-reduce
- `feedback_no_cpu_compute_strict` — every aggregation lives on GPU
- `feedback_no_htod_htoh_only_mapped_pinned` — every Phase 1.4 buffer
is mapped-pinned with `cuMemHostAlloc(DEVICEMAP)` reachable via
`dev_ptr`; no memcpy_dtoh/htod
- `pearl_first_observation_bootstrap` — counter-based bootstrap
preserved; placeholder writes (0.0) keep the ALPHA / HOLD_REWARD
EMAs at 0.0 sentinel until Phase 2 / 3.2 wire real producers
- `pearl_no_host_branches_in_captured_graph` — every kernel is single-
block; no host branches; safe to capture in the per-step CUDA Graph
- `pearl_tests_must_prove_not_lock_observations` — integration test
asserts invariants (slots populate, bounds respected) rather than
locked observed values
## Verification
- `cargo check -p ml --features cuda` clean
- 18 lib unit tests for SP20 launchers pass
- 23 GPU oracle tests across 5 test files pass on RTX 3050 Ti (sm_86)
- `every_fold_and_soft_reset_entry_has_dispatch_arm` regression test
now passes (was failing pre-change)
- 14 baseline lib test failures unchanged (none introduced by this
commit)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Foxhunt
Production HFT trading system in Rust.
Architecture
The workspace contains 32 crates organized as follows:
Core Libraries (16)
| Crate | Purpose |
|---|---|
trading_engine |
Order processing, FIX 4.4, IB TWS, SIMD, RDTSC timing |
risk |
VaR, Kelly, circuit breakers, kill switches, compliance |
risk-data |
Risk data types and shared structures |
trading-data |
Trading data types |
ml |
DQN Rainbow, PPO, TFT, Mamba2, ensemble inference |
ml-data |
ML data types and feature definitions |
data |
Market data ingestion and storage |
backtesting |
Replay engine, strategy tester |
adaptive-strategy |
Ensemble execution, microstructure analysis |
common |
Shared types, resilience, error handling |
storage |
S3 and local model storage |
model_loader |
Model serialization and loading |
market-data |
Market data feed handlers |
database |
PostgreSQL access layer (SQLx) |
config |
Configuration management |
tli |
CLI commands and tooling |
Services (8)
| Service | Purpose |
|---|---|
backtesting_service |
gRPC backtesting service |
broker_gateway_service |
FIX routing, broker connectivity |
trading_service |
Core trading operations |
ml_training_service |
Model training orchestration |
data_acquisition_service |
Market data acquisition |
trading_agent_service |
Autonomous trading agents |
api_gateway |
gRPC API gateway with auth |
web-gateway |
Axum REST + WebSocket gateway |
Frontend
web-dashboard/ -- React 19 + TypeScript + Vite + TradingView charts.
Building
# Check compilation (no PostgreSQL required)
SQLX_OFFLINE=true cargo check --workspace
# Run tests for a specific crate
SQLX_OFFLINE=true cargo test -p <crate> --lib
# Clippy
SQLX_OFFLINE=true cargo clippy --workspace
ML Models
Four production model architectures on Candle v0.9.1 with CUDA:
- DQN Rainbow -- Deep Q-Network with prioritized replay, dueling heads, noisy nets
- PPO -- Proximal Policy Optimization with GAE and LSTM policies
- TFT -- Temporal Fusion Transformer for multi-horizon forecasting
- Mamba2 -- State space model for sequence prediction
Each model has a standalone trainer and a UnifiedTrainable adapter for the hyperopt pipeline.
Infrastructure
- Git: Gitea at
git.fxhnt.ai(Tailscale-only), Scaleway DEV1-S - Observability: OpenTelemetry OTLP (env
OTEL_EXPORTER_OTLP_ENDPOINT) - Database: PostgreSQL with SQLx offline mode for CI
License
Proprietary. All rights reserved.