Files

jgrusewski 629ebd667c feat(ml-alpha): deterministic same-seed training + Tier 1.5 fast-dev-cycle

Two same-seed runs now produce bit-equal eval_summary.json, alpha_rl_train_summary.json,
and diag.jsonl (modulo wall-clock elapsed_s). The 5-phase falsification chain landed:

  Phase 2   PER tree-rebuild: __threadfence is NOT a grid-wide barrier; multiple blocks
            raced across sum-tree levels. Fix: Grid=(1) Block=(1024) + __syncthreads
            in rl_per_tree_rebuild.cu.

  Phase 2.3 cuBLAS GEMM_DFALT + TF32 default-math allowed split-K non-deterministic
            accumulation at 3 sites. New crates/ml-alpha/src/cublas_determinism.rs
            applies CUBLAS_PEDANTIC_MATH via FOXHUNT_DETERMINISTIC env toggle
            (0=TF32 prod, 1=PEDANTIC dev default, 2=DEFAULT_MATH control).

  Phase 2.6 Two bugs surfaced sequentially in the backward kernel chain:
            (1) rl_iqn_tau_cos_features had a multi-block r/w race on prng_state[batch]
                — all N_TAU=32 blocks read seed; only tau_idx==0 wrote back; no
                inter-block barrier. Fix: split into READ-ONLY rl_iqn_tau_cos_features
                + new sibling rl_iqn_advance_prng_state launched on same stream
                (kernel-launch ordering = grid-wide barrier).
            (2) OutcomeHead::new called near_zero_xavier without scoped_init_seed,
                falling back to time+thread-id RNG. Stayed dormant until first done
                event activated non-sentinel labels and divergent weights flowed via
                grad_h_t_outcome into encoder gradient. Fix: add seed param + install
                scoped_init_seed(dqn_seed.wrapping_add(0x0CE0)) guard.

Validation (./scripts/determinism-check.sh --quick, RTX 3050, b=128, 200+50 steps):
  - All 200 rows of checksums.* leaves match (rel-tol 1e-5, abs-tol 1e-7)
  - eval_summary.json, alpha_rl_train_summary.json byte-equal between runs
  - diag.jsonl byte-equal modulo elapsed_s
  - Eval pnl identical run-A vs run-B at seed 42

Pre-fix baseline (Phase 2.5 measurement): same-seed eval pnl spread $450k
($187k vs -$261k). Post-fix: $0 spread.

Speed cost: ~1.5ms/step amortised; ~10-15% slower than TF32 production
(PEDANTIC tax — acceptable in dev, toggle to FOXHUNT_DETERMINISTIC=0 for prod).

Mapped-pinned discipline: all 11 NEW memcpy_dtoh sites in diagnostic dump methods
+ per-step checksum readback use a new pub(crate) helper
read_slice_d_into<T: Copy>(stream, src, dst) — MappedRecordBuffer + raw
memcpy_dtod_async + raw_stream_sync + volatile read. Generic over T (f32, f64,
i32, u32, u8). Satisfies feedback_no_htod_htoh_only_mapped_pinned + hook guard.

Bundled Tier 1.5 fast-dev-cycle infrastructure (spec
docs/superpowers/specs/2026-06-02-fast-dev-cycle.md):
  - scripts/local-mid-smoke.sh        b=128, 2000+500, ~10min on RTX 3050
  - scripts/determinism-check.sh      runs mid-smoke twice, diffs checksums
  - scripts/tier1_5_verdict.py        behavioral kill verdict
  - AdamW checkpoint save/load (crates/ml-alpha/src/trainer/optim.rs)
  - IntegratedTrainer checkpoint save/load (resume from checkpoint)
  - 15 Phase 1 checksum leaves in build_diag_value
  - Env-gated dump methods (FOXHUNT_DETERMINISM_DEBUG_PER/MAMBA2/RL/BACKWARD)
    for future divergence-chasing — never run in production

Documentation:
  - docs/superpowers/specs/2026-06-02-determinism-foundation.md
  - docs/superpowers/specs/2026-06-02-fast-dev-cycle.md
  - docs/superpowers/plans/2026-06-02-determinism-foundation-implementation.md
  - docs/superpowers/notes/2026-06-02-determinism-phase{1,2,2.2,2.5,2.6}-*.md
  - Adjacent specs/plans/notes from the analytical chain that surfaced determinism
    as the load-bearing blocker (eval-summary, eval-boundary, regime-observer,
    multi-head policy, regime-invariance, Phase 3 IQN-complement post-mortem)

Unlocks: every controller / architecture / reward-shaping A/B from this commit
onward attributes outcome differences to the change, not random-init kernel-race
drift cascading through training x eval LOB-sim trajectories. The eval-collapse
investigation (pearl_reward_signal_anti_aligned_with_pnl, multi-head spec,
regime-invariance spec) is now testable with trustworthy verdicts.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

2026-06-02 17:56:00 +02:00

5.7 KiB

Raw Permalink Blame History

Foxhunt — HFT Trading System

Behavioral Rules

Do what has been asked; nothing more, nothing less
NEVER create files unless absolutely necessary
ALWAYS prefer editing an existing file to creating a new one
NEVER proactively create documentation files (*.md) unless explicitly requested
NEVER save working files, text/mds, or tests to the root folder
ALWAYS read a file before editing it
NEVER commit secrets, credentials, or .env files

Project Overview

Rust-based high-frequency trading system with ML-driven strategies (DQN, PPO, TFT, Mamba2), deployed on Kubernetes (Scaleway Kapsule) via Argo Workflows.

Tech Stack

Component	Technology
Language	Rust 1.85+, Edition 2021
Async	Tokio 1.40 (multi-threaded)
RPC	Tonic 0.14 (gRPC), tonic-web (browser gateway)
Database	PostgreSQL 16 (SQLx, offline mode), Redis 7, QuestDB 8.2.3
ML	DQN, PPO, TFT, Mamba2 (Rust inference)
Frontend	React 19, TypeScript, Tailwind CSS, Vite (`web-dashboard/`)
Orchestration	Kubernetes + Argo Workflows
Build Cache	sccache (MinIO S3 backend)
GPU	CUDA 12.4, L40S (48GB), H100 (80GB)

Project Structure

Directory	Purpose
`crates/` (40 crates)	Core libs: trading engine, ML models, data, backtesting, risk mgmt
`services/` (8 services)	Microservices: API, Trading, ML Training, Data Acquisition, Backtesting, Broker Gateway, Trading Agent
`bin/fxt/`	CLI binary
`web-dashboard/`	React dashboard (gRPC-web)
`testing/`	Integration, e2e, load, stress tests
`infra/k8s/`	K8s manifests, Argo Workflows, network policies, monitoring
`infra/docker/`	Dockerfiles (ci-builder, runtime, training-runtime)
`scripts/`	Build, deployment, health-check, GPU scripts
`migrations/`	PostgreSQL migrations
`proto/`	Protocol Buffer definitions
`config/`	Configuration files and templates

Build & Test

# Build
cargo build --release
cargo build --profile=dev-release          # Fast iteration (thin LTO)
cargo build -p <service-name> --release    # Single service

# Test
cargo test --workspace
cargo test -p <crate-name>
cargo test --test integration

# Benchmarks
cargo bench --bench performance_regression
cargo bench --bench trading_latency
cargo bench --bench database_performance

SQLX_OFFLINE=true — compiled SQL queries (no live DB needed for build)
SCCACHE_BUCKET=foxhunt-sccache — shared compile cache

Tiered local validation (fast dev cycle)

Three-tier funnel before cluster submit (per docs/superpowers/specs/2026-06-02-fast-dev-cycle.md):

Tier	Setup	Time	What it validates
1 (correctness)	RTX 3050, b=16, 200 steps, 1 file	~6 sec	Kernel correctness, ISV slots, NaN
1.5 (behavior)	RTX 3050, b=128, 2000+500 steps, 2 files	~10 min	entropy, hold growth, controller stability, early Pearson
2 (eval verdict)	L40S, b=1024, 20k+5k, 9 files	~70 min	eval pnl, eval wr, regime stratification

Tier 1.5 data path: test_data/futures-baseline-mid/ (symlinks to existing MBP-10 files for fold-1 train/eval split).

./scripts/local-mid-smoke.sh           # Run Tier 1.5 mid-smoke (~10 min)
./scripts/determinism-check.sh         # Reproducibility self-test (runs mid-smoke twice, diffs final 5 rows)
python3 scripts/tier1_5_verdict.py /tmp/foxhunt-mid-smoke   # Behavioral kill verdict

Deployment (Argo Workflows)

./scripts/argo-compile-deploy.sh [--branch main] [--commit <sha>]
./scripts/argo-train.sh [--model dqn|ppo|supervised]

Node pools: ci-compile-cpu (compile/test), ci-training (L40S GPU), ci-training-h100 (H100 GPU)
Training data: PVC at /mnt/training-data/futures-baseline
Docker: Kaniko push to Scaleway Container Registry
Legacy CI: .gitlab-ci.yml exists but disabled (when: never), Argo is primary

File Organization Rules

NEVER save to root folder
/src, /crates — Rust source code
/tests, /testing — test files
/docs — documentation
/config — configuration
/scripts — utility scripts
/infra — infrastructure (K8s, Docker)

Security Rules

NEVER hardcode API keys, secrets, or credentials
NEVER commit .env files or trading credentials
Always validate input at system boundaries
Sanitize file paths to prevent directory traversal
Broker credentials (AMP Futures/CQG FIX 4.4) are K8s secrets — never log them

All operations MUST be concurrent/parallel in a single message
Use Claude Code's Task tool for spawning agents
ALWAYS batch ALL todos in ONE TodoWrite call
ALWAYS spawn ALL agents in ONE message via Task tool
ALWAYS batch ALL file reads/writes/edits in ONE message
ALWAYS batch ALL Bash commands in ONE message

RuFlo V3 Integration

Hooks are configured in .claude/settings.json and run local scripts from .claude/helpers/. All hooks have timeouts — no network calls on the hot path.

CLI Commands

npx ruflo@latest status                    # Check status
npx ruflo@latest hooks metrics             # View learning metrics
npx ruflo@latest hooks pretrain            # Bootstrap from repository
npx ruflo@latest hooks build-agents        # Generate agent configs

important-instruction-reminders

Do what has been asked; nothing more, nothing less. NEVER create files unless they're absolutely necessary for achieving your goal. ALWAYS prefer editing an existing file to creating a new one. NEVER proactively create documentation files (*.md) or README files. Only create documentation files if explicitly requested by the User. Never save working files, text/mds and tests to the root folder.

5.7 KiB Raw Permalink Blame History