Commit Graph

1039 Commits

Author SHA1 Message Date
jgrusewski
9c3d741a08 refactor: restructure repo — crates/, bin/, testing/ layout
Move 17 library crates into crates/, CLI binary into bin/fxt,
consolidate 10 test crates into testing/, split config crate
from deployment config files.

Root directory reduced from 38+ to ~17 directories.
All Cargo.toml paths and build.rs proto refs updated.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 11:56:00 +01:00
Administrator
c821f2bec8 fix: allow dead_code in data/ml crates (unintegrated provider/model infrastructure) 2026-02-25 10:32:34 +00:00
Administrator
2af537d37c fix: clippy warnings in trading_engine brokers (bool::then, needless_return) 2026-02-25 10:12:12 +00:00
Administrator
91b15ec293 fix: implement cache_timeout in config loaders (was dead code) 2026-02-25 10:00:06 +00:00
Administrator
bdcd016f33 ci: install protoc 28.3 from GitHub (Ubuntu protoc too old for proto3 optional) 2026-02-25 09:37:16 +00:00
jgrusewski
055751b3c3 chore: delete legacy artifacts (RunPod, GitHub Actions, disabled tests, systemd, diagnostic data)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 10:32:41 +01:00
Administrator
39cfb037ea ci: add libfontconfig1-dev to CI builder (plotters dependency) 2026-02-25 09:24:59 +00:00
Administrator
b3e8f69631 ci: set CUDA_COMPUTE_CAP=86 for GPU-less CI nodes (candle-kernels needs it) 2026-02-25 09:19:16 +00:00
Administrator
f541a3fb9d ci: remove build-ci-builder dependency from check/test (image exists in SCR) 2026-02-25 09:13:50 +00:00
Administrator
541ddcb7c1 ci: remove build-ci-builder dependency from check/test (image exists in SCR) 2026-02-25 09:09:24 +00:00
Administrator
84f935a70e ci: CUDA CI builder image via Scaleway CR
- Push ci-builder to rg.fr-par.scw.cloud/foxhunt-ci (Kapsule nodes can pull natively)
- Add SCR + GitLab imagePullSecrets to runner config
- Dockerfile.ci-builder unchanged (CUDA 12.4 + Rust 1.89 + protoc + sccache)
2026-02-25 08:54:55 +00:00
Administrator
cdd13783c4 fix(ci): use username/password auth for Kaniko instead of base64 auth field 2026-02-25 08:32:40 +00:00
Administrator
8a73f47624 ci: pre-baked CI builder image, pipeline prepare stage
- Add Dockerfile.ci-builder (CUDA 12.4 + Rust 1.89 + protoc + sccache + git + lld)
- Add prepare stage to build CI builder image via Kaniko (auto on Dockerfile change, manual otherwise)
- Replace before_script apt-get installs with pre-baked image in .rust-base
- Add git to Dockerfile.service for candle git dependency
- Add sccache stats output to check stage
2026-02-25 08:22:30 +00:00
Administrator
08e4d3720b fix(ci): add git to build deps, runner clone_url to internal service 2026-02-25 08:11:36 +00:00
Administrator
fa0e2a3809 infra: HTTPS wildcard cert, subdomain routing, CI/CD pipeline, Grafana
- Wildcard LE cert for *.fxhnt.ai
- Subdomain routing: git/grafana/prometheus.fxhnt.ai
- CI/CD: Kaniko image builds, sccache, deploy stage
- Grafana with Prometheus datasource
- GitLab node upgraded to GP1-XS (16GB)
2026-02-25 01:18:21 +00:00
jgrusewski
5886ae57a6 chore: remove Gitea CI config and old ci-runner module
Gitea replaced by GitLab CE on Kapsule. CI runner replaced by
GitLab Runner with Kubernetes executor on ci-build node pool.

Removed:
- .gitea/workflows/ci.yaml (Gitea Actions config)
- infra/modules/ci-runner/ (Scaleway VM-based runner)
- infra/live/production/ci-runner/ (Terragrunt live config)

Note: The ci-runner VM instance still needs manual `terragrunt destroy`
before the Scaleway resource can be released.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 01:14:59 +01:00
jgrusewski
a21c534ed9 chore: untrack 928 large binary files (safetensors/onnx/dbn)
filter-repo stripped the blobs but left tree entries. Remove from
index so .gitignore rules take effect. Adds checkpoints/ to gitignore.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 01:13:35 +01:00
Administrator
8f98230dc3 fix: .gitignore large files, runner internal URL, proxy buffering
- Add .gitignore entries for .dbn, .safetensors, .onnx and other large binaries
- Point GitLab Runner at internal cluster URL (not external Tailscale)
- Disable nginx proxy_buffering for git clone/push through Tailscale proxy
- Increase socat SSH proxy buffer sizes to 1MB

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 00:09:23 +00:00
jgrusewski
840fe15471 Merge branch 'worktree-gitlab-migration'
GitLab CE migration infrastructure: Helm values, Tailscale proxy,
CI/CD pipeline, GitLab Runner, DNS, K8s manifests, S3 storage.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 00:56:29 +01:00
jgrusewski
184c07bb82 Merge branch 'worktree-audit-fixes'
# Conflicts:
#	infra/modules/secrets/main.tf
#	infra/modules/secrets/variables.tf
2026-02-25 00:33:10 +01:00
jgrusewski
86f7f1fa76 fix: comprehensive audit — real brokers, deployment fixes, production safety
Codebase audit identified 23 findings across 4 dimensions (production safety,
code health, deployment readiness, test quality). This commit fixes all of them.

Broker execution layer (was entirely stubbed):
- Real IBKR TWS client via ibapi crate (950+ lines, feature-gated)
- ICMarkets ctrader-openapi now always-on (removed feature flag)
- Real broker routing with health monitoring and exponential backoff reconnect
- Validated against live IB Gateway Docker (6/6 connectivity tests pass)

Deployment blockers:
- Fixed 6 broken Dockerfiles (removed COPY foxhunt-deploy)
- Created foxhunt K8s namespace, secret templates, migration job
- Added liveness probes to all 7 K8s services
- IB Gateway manifest (ghcr.io/gnzsnz/ib-gateway:stable)
- IBKR credentials in Scaleway Secret Manager via Terragrunt
- Fixed port collisions and mismatches across services

Production safety (9 critical + 6 high/medium fixes):
- Asset-class-specific VaR volatility (not flat 2%)
- Real parametric VaR with z-score 95th percentile
- Kyle's lambda regression (100-bar rolling window)
- Per-feature running statistics from historical data
- VWAP-based slippage reference, regime duration tracking
- Real Databento JSON parsing for OHLCV/Trade/Quote

Code health:
- Removed #![allow(dead_code)] from ml, data, config
- Fixed log:: → tracing:: in 4 production files
- Removed dead workspace deps (ratatui, crossterm)

Verified: cargo check --workspace (0 errors), trading_engine 330 tests pass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 00:32:10 +01:00
jgrusewski
9c37d0edbb infra(gitlab): enable Prometheus, SSH proxy, explicit always-on type
- Enable Prometheus server (GitLab chart bundled) on gitlab node pool
- Add socat SSH proxy sidecar (port 2222 → gitlab-shell) to Tailscale proxy
- Remove nginx stream module (not available in alpine) in favor of socat
- Set unlimited nginx client_max_body_size for large git pushes
- Add workhorse extraArgs for API limits
- Explicit always_on_type = DEV1-M in kapsule terragrunt

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 00:29:13 +01:00
jgrusewski
26e0caf67c fix(gitlab): Helm values schema, Tailscale proxy, DNS record
- Fix certmanager: use installCertmanager top-level key (not certmanager.install)
- Fix redis: global.redis.auth.enabled (not password.enabled)
- Remove grafana (not in GitLab chart), remove top-level minio (use global.minio.enabled)
- Add toolbox backup objectStorage config (required when minio disabled)
- Set HPA minReplicas/maxReplicas=1 for single-node DEV1-L
- Add Tailscale proxy (nginx + tailscale sidecar) for private access
- DNS: git.fxhnt.ai → 100.90.76.85 (foxhunt-gitlab Tailscale IP)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 00:06:10 +01:00
jgrusewski
ad4619ae64 fix(gitlab): rewrite postgres init job to avoid psql \gexec 2026-02-24 23:38:29 +01:00
jgrusewski
b507b1bdea infra: rename root terragrunt.hcl → root.hcl (fix anti-pattern warning)
Terragrunt v0.77+ warns about using terragrunt.hcl as root config.
Rename to root.hcl and update all child includes to use
find_in_parent_folders("root.hcl").

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:29:11 +01:00
jgrusewski
6cd46e4796 ci: deploy stage applies manifests + sets images
Deploy now runs kubectl apply on manifests first (picking up
config changes), then updates images to the commit SHA.
Adds GitLab environment tracking for production.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:27:56 +01:00
jgrusewski
020e85af2b infra: add DNS module + rename gitlab.fxhnt.ai → git.fxhnt.ai
Keep existing git.fxhnt.ai hostname for GitLab CE (repoint DNS
to new Tailscale IP after deploy). Add Terraform DNS module to
manage the A record via Terragrunt instead of manual scw CLI.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:26:51 +01:00
jgrusewski
4fb398a531 infra(k8s): switch image refs from Scaleway registry to GitLab registry
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:25:05 +01:00
jgrusewski
a082d52313 ci: add GitLab CI/CD pipeline (check, test, build, deploy)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:23:26 +01:00
jgrusewski
678f526225 infra(gitlab): Runner Helm values with K8s executor for ci-build pool
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:23:16 +01:00
jgrusewski
24c1b94fb0 infra(gitlab): Helm values for GitLab CE deployment
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:23:05 +01:00
jgrusewski
1045b190dc infra(gitlab): postgres init job for GitLab database
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:22:54 +01:00
jgrusewski
1c36941ded infra(kapsule): add gitlab and ci-build node pools
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:21:14 +01:00
jgrusewski
ce8eeba6fb infra(storage): add GitLab registry and artifacts buckets 2026-02-24 23:21:06 +01:00
jgrusewski
1bc4e7e10a docs: GitLab CE migration implementation plan (12 tasks, 5 phases)
Covers: Terragrunt node pools, object storage buckets, Postgres init,
GitLab CE Helm deploy, Runner with K8s executor, .gitlab-ci.yml
translation (Kaniko builds), manifest cutover, decommission steps.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:16:50 +01:00
jgrusewski
3c4af61cb0 docs: GitLab CE migration design (Gitea → GitLab on Kapsule)
Covers: in-cluster GitLab CE with external Postgres/Redis, dedicated
ci-build node pool (scale-to-zero), Kaniko image builds, GitLab
built-in registry, and 4-phase migration plan with rollback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:13:00 +01:00
jgrusewski
019bc140d7 fix(ci): increase idle timeout to 60min, check act_runner children
The 15min timeout was too short for Rust builds. Also checks
for act_runner child processes (image pull, git clone phases)
not just Docker containers.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 22:52:28 +01:00
jgrusewski
903bf4fbf6 infra(ci): scale-to-zero with idle shutdown and auto-start watcher
CI runner auto-shuts down after 15min idle (no GITEA-ACTIONS containers).
Gitea server polls every 60s for queued runs and starts instance via SCW API.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 22:33:42 +01:00
jgrusewski
a47e0cb947 infra(ci): add Terragrunt-managed CI runner with Tailscale + act_runner
GP1-S instance (8 vCPU, 32GB, 100GB root) provisioned via Terraform
with cloud-init: Docker CE, Tailscale (pre-auth key), Gitea Actions
runner (act_runner v0.3.0). Security group drops all inbound — access
via Tailscale only. Solves: Kapsule pods cannot reach Tailscale IPs.

Also stores Tailscale API key in SCW Secret Manager for future auth
key generation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 22:15:37 +01:00
jgrusewski
c8f2dacc5f feat(infra): H100 smoketest pipeline — S3 output sync, sccache, registry fix
- Fix container registry region nl-ams → fr-par across all CI jobs
- Add sccache build-args to training image build (no-op fallback for local)
- Add rclone to training Docker runtime for S3 output sync
- Update train.sh: S3 sync on Job completion via rclone env-var config,
  --run-id tracking, evaluate preset for walk-forward evaluation
- Add s3-credentials Secret template (.example, apply via kubectl)
- Add design doc and implementation plan

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 21:31:22 +01:00
jgrusewski
d4b707bfa8 fix(ml): repair download_l2_data example (time→chrono, GetRangeToFileParams)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 20:57:00 +01:00
jgrusewski
5fb84a02f0 feat: training pipeline for all 10 ML ensemble models
- Add standalone training binaries: TGGN, KAN, xLSTM, Diffusion (DBN data)
- Update Dockerfile.training: 6 → 16 binaries (all 10 models + hyperopt + baseline)
- Expand train.sh: 4 → 10 models, fix registry URL and GPU pool nodeSelector
- Add GPU overlay manifests for trading-service and ml-training-service
- Create training data PVC and upload pod manifests
- Expand web-gateway model validation: 4 → 10 types (training + tune routes)
- Extend dashboard: 10 model cards grouped by category (RL/Temporal/Graph/Generative)
- Add training image build job to Gitea CI workflow
- Update GPU taint controller to exclude inference pool from tainting
- Fix job-template nodeSelector: gpu → gpu-training

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 20:47:50 +01:00
jgrusewski
e45bda9412 Merge branch 'fix/clippy-warnings' 2026-02-24 19:16:59 +01:00
jgrusewski
001624c5b2 fix: eliminate all 8,384 clippy warnings across workspace
Systematic clippy warning cleanup achieving zero warnings:

- Add domain-appropriate crate-level #![allow(...)] to 20+ crate roots
  for pedantic lints that are noise in HFT/ML code (float_arithmetic,
  indexing_slicing, missing_const_for_fn, cognitive_complexity, etc.)
- Fix attribute ordering in risk/src/lib.rs: move #![warn(clippy::pedantic)]
  before #![allow(...)] so individual allows correctly override pedantic
- Remove module-level #![warn(clippy::pedantic)] from 8 trading_engine
  submodules that were overriding crate-level allows
- Add 45+ workspace-level lint allows in Cargo.toml for common pedantic
  noise (mixed_attributes_style, cargo_common_metadata, etc.)
- Auto-fix 67 machine-applicable warnings (redundant_closure, clone_on_copy,
  unnecessary_cast, etc.) via cargo clippy --fix
- Fix 3 unsafe JSON indexing in risk/circuit_breaker.rs with safe .get()
- Fix unused variables, unused mut, unnecessary parens in 4 files
- Proto-generated code: suppress missing_const_for_fn, indexing_slicing,
  cognitive_complexity in ctrader-openapi and service crates

75 files changed across 20+ crates. All tests pass (3,122+ verified).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 19:16:35 +01:00
foxhunt-admin
04f3db65ab Merge pull request 'infra: provision Kapsule cluster with services and GPU pool split' (#1) from worktree-infra-provisioning into main 2026-02-24 16:43:38 +00:00
jgrusewski
5380bd9186 infra: split GPU pool into training (H100) and inference (L4), add auto-taint
- Split single H100 GPU pool into two purpose-specific pools:
  - gpu-training: H100-1-80G (€2.73/hr) for 10-model ensemble training
  - gpu-inference: L4-1-24G (€0.75/hr) for cost-effective trading inference
- Add GPU taint controller DaemonSet that auto-taints new GPU nodes
  with nvidia.com/gpu=true:NoSchedule to prevent non-GPU workloads
- Fix service log directory permissions with emptyDir volumes
  (observability init fails creating /app/logs as non-root user)
- Increase postgres max_connections from 25 to 100
  (7 services each requesting connection pools exhausted the limit)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 17:39:56 +01:00
jgrusewski
4cc37057e1 Merge branch 'fix/ml-clippy-cleanup'
Resolve all 179 clippy deny violations in the ml crate: replace
.unwrap()/.expect() with safe alternatives across 51 files.
Zero clippy errors remain across the entire workspace.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 17:19:57 +01:00
jgrusewski
7fe064c6e0 fix: resolve all 179 clippy deny violations in ml crate
Replace .unwrap()/.expect() with safe alternatives across 51 files:
- 41 `let _ = writeln!()` → `_ = writeln!()` (wildcard assignment)
- 53 unwrap() in features/ → unwrap_or/match/early-return
- 20 expect() in inference/metrics → module-level #[allow] for static init
- 16 unwrap/expect in hyperopt/ → ?, map_err, unwrap_or
- 20 unwrap in dqn/trainers/ → ?, map_err, unwrap_or
- 28 unwrap in misc files → context-appropriate safe patterns

Zero clippy errors remain across the entire workspace.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 17:19:35 +01:00
jgrusewski
3db5c950c7 infra: deploy all services to Kapsule cluster
- Fix DB pool connect/acquire timeouts (50ms→5s) for cluster networking
  (pool-level timeouts, not query timeouts — HFT query timeout stays at 800μs)
- Fix secret key references (DATABASE_PASSWORD→db-password) in all manifests
- Fix api-gateway port (50050→50051) to match actual gRPC listen port
- Fix web-gateway health probe path (/api/health→/health)
- Fix S3 endpoint region (nl-ams→fr-par) in ml-training-service
- Add TLS cert volume mount for ml-training-service
- Add BENZINGA_API_KEY placeholder for backtesting-service startup
- Remove always-on nodeSelector from services (let autoscaler handle)
- Add serve subcommand to ml-training-service container

All 10 pods (3 databases + 7 services) now 1/1 Running.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 17:18:47 +01:00
jgrusewski
f6fb67a58f infra: add build-and-push script and fix Dockerfiles for Scaleway registry
Remove deleted foxhunt-deploy COPY from all Dockerfiles, update sccache
default region to fr-par, add build-and-push.sh for building all service
images and pushing to rg.fr-par.scw.cloud/foxhunt/.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 16:29:43 +01:00