jgrusewski
022036cb96
refactor(ml): consolidate 21 training binaries into 2 unified baselines
...
Replace 20 per-model training examples with:
- train_baseline_rl: DQN + PPO (renamed from train_baseline)
- train_baseline_supervised: TFT, Mamba2, Liquid, TGGN, TLOB, KAN, xLSTM, Diffusion
via model factory + UnifiedTrainable generic training loop
Update Dockerfile.training (16→7 binaries), train.sh MODEL_BINARY map,
and job-template.yaml default. -12,759 lines.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-02-25 23:16:25 +01:00
jgrusewski
86f7f1fa76
fix: comprehensive audit — real brokers, deployment fixes, production safety
...
Codebase audit identified 23 findings across 4 dimensions (production safety,
code health, deployment readiness, test quality). This commit fixes all of them.
Broker execution layer (was entirely stubbed):
- Real IBKR TWS client via ibapi crate (950+ lines, feature-gated)
- ICMarkets ctrader-openapi now always-on (removed feature flag)
- Real broker routing with health monitoring and exponential backoff reconnect
- Validated against live IB Gateway Docker (6/6 connectivity tests pass)
Deployment blockers:
- Fixed 6 broken Dockerfiles (removed COPY foxhunt-deploy)
- Created foxhunt K8s namespace, secret templates, migration job
- Added liveness probes to all 7 K8s services
- IB Gateway manifest (ghcr.io/gnzsnz/ib-gateway:stable)
- IBKR credentials in Scaleway Secret Manager via Terragrunt
- Fixed port collisions and mismatches across services
Production safety (9 critical + 6 high/medium fixes):
- Asset-class-specific VaR volatility (not flat 2%)
- Real parametric VaR with z-score 95th percentile
- Kyle's lambda regression (100-bar rolling window)
- Per-feature running statistics from historical data
- VWAP-based slippage reference, regime duration tracking
- Real Databento JSON parsing for OHLCV/Trade/Quote
Code health:
- Removed #![allow(dead_code)] from ml, data, config
- Fixed log:: → tracing:: in 4 production files
- Removed dead workspace deps (ratatui, crossterm)
Verified: cargo check --workspace (0 errors), trading_engine 330 tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-02-25 00:32:10 +01:00
jgrusewski
c8f2dacc5f
feat(infra): H100 smoketest pipeline — S3 output sync, sccache, registry fix
...
- Fix container registry region nl-ams → fr-par across all CI jobs
- Add sccache build-args to training image build (no-op fallback for local)
- Add rclone to training Docker runtime for S3 output sync
- Update train.sh: S3 sync on Job completion via rclone env-var config,
--run-id tracking, evaluate preset for walk-forward evaluation
- Add s3-credentials Secret template (.example, apply via kubectl)
- Add design doc and implementation plan
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-02-24 21:31:22 +01:00
jgrusewski
5fb84a02f0
feat: training pipeline for all 10 ML ensemble models
...
- Add standalone training binaries: TGGN, KAN, xLSTM, Diffusion (DBN data)
- Update Dockerfile.training: 6 → 16 binaries (all 10 models + hyperopt + baseline)
- Expand train.sh: 4 → 10 models, fix registry URL and GPU pool nodeSelector
- Add GPU overlay manifests for trading-service and ml-training-service
- Create training data PVC and upload pod manifests
- Expand web-gateway model validation: 4 → 10 types (training + tune routes)
- Extend dashboard: 10 model cards grouped by category (RL/Temporal/Graph/Generative)
- Add training image build job to Gitea CI workflow
- Update GPU taint controller to exclude inference pool from tainting
- Fix job-template nodeSelector: gpu → gpu-training
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-02-24 20:47:50 +01:00
jgrusewski
f434309f59
infra: provision Kapsule cluster in fr-par with GPU support
...
Move all infrastructure from nl-ams to fr-par-2 for H100 GPU availability.
Kapsule cluster provisioned with 3 node pools (always-on DEV1-M, CI GP1-XS,
GPU H100-1-80G), databases deployed (Postgres/TimescaleDB, Redis, QuestDB),
and Tailscale subnet router configured.
Key changes:
- Region: nl-ams -> fr-par-2 (GPU types only available in Paris)
- K8s version: 1.30 -> 1.34 with auto-upgrade
- Block storage: migrate b_ssd -> SBS (scaleway_block_volume)
- Kapsule: add VPC private network, delete_additional_resources
- GPU pool: conditional via enable_gpu_pool variable
- K8s manifests: fix registry URLs, secret key names, PG subPath
- Add terragrunt cache/lockfile to .gitignore
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-02-24 15:35:40 +01:00
jgrusewski
2a3c107b31
infra: add K8s manifests for full paper trading stack
...
- Tailscale subnet router (zero public access)
- PostgreSQL, Redis, QuestDB (PVCs on always-on node)
- 7 service deployments (trading, api-gw, broker-gw, ml, backtest, agent, web-gw)
- GPU training Job template + idle reaper CronJob
- Block Storage PV/PVC for shared training data
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-02-24 14:27:32 +01:00
