629ebd667c
Two same-seed runs now produce bit-equal eval_summary.json, alpha_rl_train_summary.json,
and diag.jsonl (modulo wall-clock elapsed_s). The 5-phase falsification chain landed:
Phase 2 PER tree-rebuild: __threadfence is NOT a grid-wide barrier; multiple blocks
raced across sum-tree levels. Fix: Grid=(1) Block=(1024) + __syncthreads
in rl_per_tree_rebuild.cu.
Phase 2.3 cuBLAS GEMM_DFALT + TF32 default-math allowed split-K non-deterministic
accumulation at 3 sites. New crates/ml-alpha/src/cublas_determinism.rs
applies CUBLAS_PEDANTIC_MATH via FOXHUNT_DETERMINISTIC env toggle
(0=TF32 prod, 1=PEDANTIC dev default, 2=DEFAULT_MATH control).
Phase 2.6 Two bugs surfaced sequentially in the backward kernel chain:
(1) rl_iqn_tau_cos_features had a multi-block r/w race on prng_state[batch]
— all N_TAU=32 blocks read seed; only tau_idx==0 wrote back; no
inter-block barrier. Fix: split into READ-ONLY rl_iqn_tau_cos_features
+ new sibling rl_iqn_advance_prng_state launched on same stream
(kernel-launch ordering = grid-wide barrier).
(2) OutcomeHead::new called near_zero_xavier without scoped_init_seed,
falling back to time+thread-id RNG. Stayed dormant until first done
event activated non-sentinel labels and divergent weights flowed via
grad_h_t_outcome into encoder gradient. Fix: add seed param + install
scoped_init_seed(dqn_seed.wrapping_add(0x0CE0)) guard.
Validation (./scripts/determinism-check.sh --quick, RTX 3050, b=128, 200+50 steps):
- All 200 rows of checksums.* leaves match (rel-tol 1e-5, abs-tol 1e-7)
- eval_summary.json, alpha_rl_train_summary.json byte-equal between runs
- diag.jsonl byte-equal modulo elapsed_s
- Eval pnl identical run-A vs run-B at seed 42
Pre-fix baseline (Phase 2.5 measurement): same-seed eval pnl spread $450k
($187k vs -$261k). Post-fix: $0 spread.
Speed cost: ~1.5ms/step amortised; ~10-15% slower than TF32 production
(PEDANTIC tax — acceptable in dev, toggle to FOXHUNT_DETERMINISTIC=0 for prod).
Mapped-pinned discipline: all 11 NEW memcpy_dtoh sites in diagnostic dump methods
+ per-step checksum readback use a new pub(crate) helper
read_slice_d_into<T: Copy>(stream, src, dst) — MappedRecordBuffer + raw
memcpy_dtod_async + raw_stream_sync + volatile read. Generic over T (f32, f64,
i32, u32, u8). Satisfies feedback_no_htod_htoh_only_mapped_pinned + hook guard.
Bundled Tier 1.5 fast-dev-cycle infrastructure (spec
docs/superpowers/specs/2026-06-02-fast-dev-cycle.md):
- scripts/local-mid-smoke.sh b=128, 2000+500, ~10min on RTX 3050
- scripts/determinism-check.sh runs mid-smoke twice, diffs checksums
- scripts/tier1_5_verdict.py behavioral kill verdict
- AdamW checkpoint save/load (crates/ml-alpha/src/trainer/optim.rs)
- IntegratedTrainer checkpoint save/load (resume from checkpoint)
- 15 Phase 1 checksum leaves in build_diag_value
- Env-gated dump methods (FOXHUNT_DETERMINISM_DEBUG_PER/MAMBA2/RL/BACKWARD)
for future divergence-chasing — never run in production
Documentation:
- docs/superpowers/specs/2026-06-02-determinism-foundation.md
- docs/superpowers/specs/2026-06-02-fast-dev-cycle.md
- docs/superpowers/plans/2026-06-02-determinism-foundation-implementation.md
- docs/superpowers/notes/2026-06-02-determinism-phase{1,2,2.2,2.5,2.6}-*.md
- Adjacent specs/plans/notes from the analytical chain that surfaced determinism
as the load-bearing blocker (eval-summary, eval-boundary, regime-observer,
multi-head policy, regime-invariance, Phase 3 IQN-complement post-mortem)
Unlocks: every controller / architecture / reward-shaping A/B from this commit
onward attributes outcome differences to the change, not random-init kernel-race
drift cascading through training x eval LOB-sim trajectories. The eval-collapse
investigation (pearl_reward_signal_anti_aligned_with_pnl, multi-head spec,
regime-invariance spec) is now testable with trustworthy verdicts.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
5.7 KiB
5.7 KiB
Foxhunt — HFT Trading System
Behavioral Rules
- Do what has been asked; nothing more, nothing less
- NEVER create files unless absolutely necessary
- ALWAYS prefer editing an existing file to creating a new one
- NEVER proactively create documentation files (*.md) unless explicitly requested
- NEVER save working files, text/mds, or tests to the root folder
- ALWAYS read a file before editing it
- NEVER commit secrets, credentials, or .env files
Project Overview
Rust-based high-frequency trading system with ML-driven strategies (DQN, PPO, TFT, Mamba2), deployed on Kubernetes (Scaleway Kapsule) via Argo Workflows.
Tech Stack
| Component | Technology |
|---|---|
| Language | Rust 1.85+, Edition 2021 |
| Async | Tokio 1.40 (multi-threaded) |
| RPC | Tonic 0.14 (gRPC), tonic-web (browser gateway) |
| Database | PostgreSQL 16 (SQLx, offline mode), Redis 7, QuestDB 8.2.3 |
| ML | DQN, PPO, TFT, Mamba2 (Rust inference) |
| Frontend | React 19, TypeScript, Tailwind CSS, Vite (web-dashboard/) |
| Orchestration | Kubernetes + Argo Workflows |
| Build Cache | sccache (MinIO S3 backend) |
| GPU | CUDA 12.4, L40S (48GB), H100 (80GB) |
Project Structure
| Directory | Purpose |
|---|---|
crates/ (40 crates) |
Core libs: trading engine, ML models, data, backtesting, risk mgmt |
services/ (8 services) |
Microservices: API, Trading, ML Training, Data Acquisition, Backtesting, Broker Gateway, Trading Agent |
bin/fxt/ |
CLI binary |
web-dashboard/ |
React dashboard (gRPC-web) |
testing/ |
Integration, e2e, load, stress tests |
infra/k8s/ |
K8s manifests, Argo Workflows, network policies, monitoring |
infra/docker/ |
Dockerfiles (ci-builder, runtime, training-runtime) |
scripts/ |
Build, deployment, health-check, GPU scripts |
migrations/ |
PostgreSQL migrations |
proto/ |
Protocol Buffer definitions |
config/ |
Configuration files and templates |
Build & Test
# Build
cargo build --release
cargo build --profile=dev-release # Fast iteration (thin LTO)
cargo build -p <service-name> --release # Single service
# Test
cargo test --workspace
cargo test -p <crate-name>
cargo test --test integration
# Benchmarks
cargo bench --bench performance_regression
cargo bench --bench trading_latency
cargo bench --bench database_performance
SQLX_OFFLINE=true— compiled SQL queries (no live DB needed for build)SCCACHE_BUCKET=foxhunt-sccache— shared compile cache
Tiered local validation (fast dev cycle)
Three-tier funnel before cluster submit (per docs/superpowers/specs/2026-06-02-fast-dev-cycle.md):
| Tier | Setup | Time | What it validates |
|---|---|---|---|
| 1 (correctness) | RTX 3050, b=16, 200 steps, 1 file | ~6 sec | Kernel correctness, ISV slots, NaN |
| 1.5 (behavior) | RTX 3050, b=128, 2000+500 steps, 2 files | ~10 min | entropy, hold growth, controller stability, early Pearson |
| 2 (eval verdict) | L40S, b=1024, 20k+5k, 9 files | ~70 min | eval pnl, eval wr, regime stratification |
Tier 1.5 data path: test_data/futures-baseline-mid/ (symlinks to existing MBP-10 files for fold-1 train/eval split).
./scripts/local-mid-smoke.sh # Run Tier 1.5 mid-smoke (~10 min)
./scripts/determinism-check.sh # Reproducibility self-test (runs mid-smoke twice, diffs final 5 rows)
python3 scripts/tier1_5_verdict.py /tmp/foxhunt-mid-smoke # Behavioral kill verdict
Deployment (Argo Workflows)
./scripts/argo-compile-deploy.sh [--branch main] [--commit <sha>]
./scripts/argo-train.sh [--model dqn|ppo|supervised]
- Node pools:
ci-compile-cpu(compile/test),ci-training(L40S GPU),ci-training-h100(H100 GPU) - Training data: PVC at
/mnt/training-data/futures-baseline - Docker: Kaniko push to Scaleway Container Registry
- Legacy CI:
.gitlab-ci.ymlexists but disabled (when: never), Argo is primary
File Organization Rules
- NEVER save to root folder
/src,/crates— Rust source code/tests,/testing— test files/docs— documentation/config— configuration/scripts— utility scripts/infra— infrastructure (K8s, Docker)
Security Rules
- NEVER hardcode API keys, secrets, or credentials
- NEVER commit .env files or trading credentials
- Always validate input at system boundaries
- Sanitize file paths to prevent directory traversal
- Broker credentials (AMP Futures/CQG FIX 4.4) are K8s secrets — never log them
Concurrency: 1 MESSAGE = ALL RELATED OPERATIONS
- All operations MUST be concurrent/parallel in a single message
- Use Claude Code's Task tool for spawning agents
- ALWAYS batch ALL todos in ONE TodoWrite call
- ALWAYS spawn ALL agents in ONE message via Task tool
- ALWAYS batch ALL file reads/writes/edits in ONE message
- ALWAYS batch ALL Bash commands in ONE message
RuFlo V3 Integration
Hooks are configured in .claude/settings.json and run local scripts from .claude/helpers/. All hooks have timeouts — no network calls on the hot path.
CLI Commands
npx ruflo@latest status # Check status
npx ruflo@latest hooks metrics # View learning metrics
npx ruflo@latest hooks pretrain # Bootstrap from repository
npx ruflo@latest hooks build-agents # Generate agent configs
important-instruction-reminders
Do what has been asked; nothing more, nothing less. NEVER create files unless they're absolutely necessary for achieving your goal. ALWAYS prefer editing an existing file to creating a new one. NEVER proactively create documentation files (*.md) or README files. Only create documentation files if explicitly requested by the User. Never save working files, text/mds and tests to the root folder.