gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
foxhunt/CLAUDE.md
jgrusewski 629ebd667c feat(ml-alpha): deterministic same-seed training + Tier 1.5 fast-dev-cycle 
Two same-seed runs now produce bit-equal eval_summary.json, alpha_rl_train_summary.json,
and diag.jsonl (modulo wall-clock elapsed_s). The 5-phase falsification chain landed:

  Phase 2   PER tree-rebuild: __threadfence is NOT a grid-wide barrier; multiple blocks
            raced across sum-tree levels. Fix: Grid=(1) Block=(1024) + __syncthreads
            in rl_per_tree_rebuild.cu.

  Phase 2.3 cuBLAS GEMM_DFALT + TF32 default-math allowed split-K non-deterministic
            accumulation at 3 sites. New crates/ml-alpha/src/cublas_determinism.rs
            applies CUBLAS_PEDANTIC_MATH via FOXHUNT_DETERMINISTIC env toggle
            (0=TF32 prod, 1=PEDANTIC dev default, 2=DEFAULT_MATH control).

  Phase 2.6 Two bugs surfaced sequentially in the backward kernel chain:
            (1) rl_iqn_tau_cos_features had a multi-block r/w race on prng_state[batch]
                — all N_TAU=32 blocks read seed; only tau_idx==0 wrote back; no
                inter-block barrier. Fix: split into READ-ONLY rl_iqn_tau_cos_features
                + new sibling rl_iqn_advance_prng_state launched on same stream
                (kernel-launch ordering = grid-wide barrier).
            (2) OutcomeHead::new called near_zero_xavier without scoped_init_seed,
                falling back to time+thread-id RNG. Stayed dormant until first done
                event activated non-sentinel labels and divergent weights flowed via
                grad_h_t_outcome into encoder gradient. Fix: add seed param + install
                scoped_init_seed(dqn_seed.wrapping_add(0x0CE0)) guard.

Validation (./scripts/determinism-check.sh --quick, RTX 3050, b=128, 200+50 steps):
  - All 200 rows of checksums.* leaves match (rel-tol 1e-5, abs-tol 1e-7)
  - eval_summary.json, alpha_rl_train_summary.json byte-equal between runs
  - diag.jsonl byte-equal modulo elapsed_s
  - Eval pnl identical run-A vs run-B at seed 42

Pre-fix baseline (Phase 2.5 measurement): same-seed eval pnl spread $450k
($187k vs -$261k). Post-fix: $0 spread.

Speed cost: ~1.5ms/step amortised; ~10-15% slower than TF32 production
(PEDANTIC tax — acceptable in dev, toggle to FOXHUNT_DETERMINISTIC=0 for prod).

Mapped-pinned discipline: all 11 NEW memcpy_dtoh sites in diagnostic dump methods
+ per-step checksum readback use a new pub(crate) helper
read_slice_d_into<T: Copy>(stream, src, dst) — MappedRecordBuffer + raw
memcpy_dtod_async + raw_stream_sync + volatile read. Generic over T (f32, f64,
i32, u32, u8). Satisfies feedback_no_htod_htoh_only_mapped_pinned + hook guard.

Bundled Tier 1.5 fast-dev-cycle infrastructure (spec
docs/superpowers/specs/2026-06-02-fast-dev-cycle.md):
  - scripts/local-mid-smoke.sh        b=128, 2000+500, ~10min on RTX 3050
  - scripts/determinism-check.sh      runs mid-smoke twice, diffs checksums
  - scripts/tier1_5_verdict.py        behavioral kill verdict
  - AdamW checkpoint save/load (crates/ml-alpha/src/trainer/optim.rs)
  - IntegratedTrainer checkpoint save/load (resume from checkpoint)
  - 15 Phase 1 checksum leaves in build_diag_value
  - Env-gated dump methods (FOXHUNT_DETERMINISM_DEBUG_PER/MAMBA2/RL/BACKWARD)
    for future divergence-chasing — never run in production

Documentation:
  - docs/superpowers/specs/2026-06-02-determinism-foundation.md
  - docs/superpowers/specs/2026-06-02-fast-dev-cycle.md
  - docs/superpowers/plans/2026-06-02-determinism-foundation-implementation.md
  - docs/superpowers/notes/2026-06-02-determinism-phase{1,2,2.2,2.5,2.6}-*.md
  - Adjacent specs/plans/notes from the analytical chain that surfaced determinism
    as the load-bearing blocker (eval-summary, eval-boundary, regime-observer,
    multi-head policy, regime-invariance, Phase 3 IQN-complement post-mortem)

Unlocks: every controller / architecture / reward-shaping A/B from this commit
onward attributes outcome differences to the change, not random-init kernel-race
drift cascading through training x eval LOB-sim trajectories. The eval-collapse
investigation (pearl_reward_signal_anti_aligned_with_pnl, multi-head spec,
regime-invariance spec) is now testable with trustworthy verdicts.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-02 17:56:00 +02:00

146 lines
5.7 KiB
Markdown
Raw Permalink Blame History

# Foxhunt — HFT Trading System
## Behavioral Rules
- Do what has been asked; nothing more, nothing less
- NEVER create files unless absolutely necessary
- ALWAYS prefer editing an existing file to creating a new one
- NEVER proactively create documentation files (*.md) unless explicitly requested
- NEVER save working files, text/mds, or tests to the root folder
- ALWAYS read a file before editing it
- NEVER commit secrets, credentials, or .env files
## Project Overview
Rust-based high-frequency trading system with ML-driven strategies (DQN, PPO, TFT, Mamba2), deployed on Kubernetes (Scaleway Kapsule) via Argo Workflows.
### Tech Stack
| Component | Technology |
|-----------|------------|
| Language | Rust 1.85+, Edition 2021 |
| Async | Tokio 1.40 (multi-threaded) |
| RPC | Tonic 0.14 (gRPC), tonic-web (browser gateway) |
| Database | PostgreSQL 16 (SQLx, offline mode), Redis 7, QuestDB 8.2.3 |
| ML | DQN, PPO, TFT, Mamba2 (Rust inference) |
| Frontend | React 19, TypeScript, Tailwind CSS, Vite (`web-dashboard/`) |
| Orchestration | Kubernetes + Argo Workflows |
| Build Cache | sccache (MinIO S3 backend) |
| GPU | CUDA 12.4, L40S (48GB), H100 (80GB) |
## Project Structure
| Directory | Purpose |
|-----------|---------|
| `crates/` (40 crates) | Core libs: trading engine, ML models, data, backtesting, risk mgmt |
| `services/` (8 services) | Microservices: API, Trading, ML Training, Data Acquisition, Backtesting, Broker Gateway, Trading Agent |
| `bin/fxt/` | CLI binary |
| `web-dashboard/` | React dashboard (gRPC-web) |
| `testing/` | Integration, e2e, load, stress tests |
| `infra/k8s/` | K8s manifests, Argo Workflows, network policies, monitoring |
| `infra/docker/` | Dockerfiles (ci-builder, runtime, training-runtime) |
| `scripts/` | Build, deployment, health-check, GPU scripts |
| `migrations/` | PostgreSQL migrations |
| `proto/` | Protocol Buffer definitions |
| `config/` | Configuration files and templates |
## Build & Test
```bash
# Build
cargo build --release
cargo build --profile=dev-release # Fast iteration (thin LTO)
cargo build -p <service-name> --release # Single service
# Test
cargo test --workspace
cargo test -p <crate-name>
cargo test --test integration
# Benchmarks
cargo bench --bench performance_regression
cargo bench --bench trading_latency
cargo bench --bench database_performance
```
- `SQLX_OFFLINE=true` — compiled SQL queries (no live DB needed for build)
- `SCCACHE_BUCKET=foxhunt-sccache` — shared compile cache
### Tiered local validation (fast dev cycle)
Three-tier funnel before cluster submit (per `docs/superpowers/specs/2026-06-02-fast-dev-cycle.md`):
| Tier | Setup | Time | What it validates |
|---|---|---|---|
| 1 (correctness) | RTX 3050, b=16, 200 steps, 1 file | ~6 sec | Kernel correctness, ISV slots, NaN |
| 1.5 (behavior) | RTX 3050, b=128, 2000+500 steps, 2 files | ~10 min | entropy, hold growth, controller stability, early Pearson |
| 2 (eval verdict) | L40S, b=1024, 20k+5k, 9 files | ~70 min | eval pnl, eval wr, regime stratification |
**Tier 1.5 data path**: `test_data/futures-baseline-mid/` (symlinks to existing MBP-10 files for fold-1 train/eval split).
```bash
./scripts/local-mid-smoke.sh # Run Tier 1.5 mid-smoke (~10 min)
./scripts/determinism-check.sh # Reproducibility self-test (runs mid-smoke twice, diffs final 5 rows)
python3 scripts/tier1_5_verdict.py /tmp/foxhunt-mid-smoke # Behavioral kill verdict
```
## Deployment (Argo Workflows)
```bash
./scripts/argo-compile-deploy.sh [--branch main] [--commit <sha>]
./scripts/argo-train.sh [--model dqn|ppo|supervised]
```
- **Node pools**: `ci-compile-cpu` (compile/test), `ci-training` (L40S GPU), `ci-training-h100` (H100 GPU)
- **Training data**: PVC at `/mnt/training-data/futures-baseline`
- **Docker**: Kaniko push to Scaleway Container Registry
- **Legacy CI**: `.gitlab-ci.yml` exists but disabled (`when: never`), Argo is primary
## File Organization Rules
- NEVER save to root folder
- `/src`, `/crates` — Rust source code
- `/tests`, `/testing` — test files
- `/docs` — documentation
- `/config` — configuration
- `/scripts` — utility scripts
- `/infra` — infrastructure (K8s, Docker)
## Security Rules
- NEVER hardcode API keys, secrets, or credentials
- NEVER commit .env files or trading credentials
- Always validate input at system boundaries
- Sanitize file paths to prevent directory traversal
- Broker credentials (AMP Futures/CQG FIX 4.4) are K8s secrets — never log them
## Concurrency: 1 MESSAGE = ALL RELATED OPERATIONS
- All operations MUST be concurrent/parallel in a single message
- Use Claude Code's Task tool for spawning agents
- ALWAYS batch ALL todos in ONE TodoWrite call
- ALWAYS spawn ALL agents in ONE message via Task tool
- ALWAYS batch ALL file reads/writes/edits in ONE message
- ALWAYS batch ALL Bash commands in ONE message
## RuFlo V3 Integration
Hooks are configured in `.claude/settings.json` and run local scripts from `.claude/helpers/`. All hooks have timeouts — no network calls on the hot path.
### CLI Commands
```bash
npx ruflo@latest status # Check status
npx ruflo@latest hooks metrics # View learning metrics
npx ruflo@latest hooks pretrain # Bootstrap from repository
npx ruflo@latest hooks build-agents # Generate agent configs
```
## important-instruction-reminders
Do what has been asked; nothing more, nothing less.
NEVER create files unless they're absolutely necessary for achieving your goal.
ALWAYS prefer editing an existing file to creating a new one.
NEVER proactively create documentation files (*.md) or README files. Only create documentation files if explicitly requested by the User.
Never save working files, text/mds and tests to the root folder.
Reference in New Issue View Git Blame Copy Permalink