Commit Graph

15 Commits

Author SHA1 Message Date
jgrusewski
5886ae57a6 chore: remove Gitea CI config and old ci-runner module
Gitea replaced by GitLab CE on Kapsule. CI runner replaced by
GitLab Runner with Kubernetes executor on ci-build node pool.

Removed:
- .gitea/workflows/ci.yaml (Gitea Actions config)
- infra/modules/ci-runner/ (Scaleway VM-based runner)
- infra/live/production/ci-runner/ (Terragrunt live config)

Note: The ci-runner VM instance still needs manual `terragrunt destroy`
before the Scaleway resource can be released.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 01:14:59 +01:00
jgrusewski
840fe15471 Merge branch 'worktree-gitlab-migration'
GitLab CE migration infrastructure: Helm values, Tailscale proxy,
CI/CD pipeline, GitLab Runner, DNS, K8s manifests, S3 storage.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 00:56:29 +01:00
jgrusewski
184c07bb82 Merge branch 'worktree-audit-fixes'
# Conflicts:
#	infra/modules/secrets/main.tf
#	infra/modules/secrets/variables.tf
2026-02-25 00:33:10 +01:00
jgrusewski
86f7f1fa76 fix: comprehensive audit — real brokers, deployment fixes, production safety
Codebase audit identified 23 findings across 4 dimensions (production safety,
code health, deployment readiness, test quality). This commit fixes all of them.

Broker execution layer (was entirely stubbed):
- Real IBKR TWS client via ibapi crate (950+ lines, feature-gated)
- ICMarkets ctrader-openapi now always-on (removed feature flag)
- Real broker routing with health monitoring and exponential backoff reconnect
- Validated against live IB Gateway Docker (6/6 connectivity tests pass)

Deployment blockers:
- Fixed 6 broken Dockerfiles (removed COPY foxhunt-deploy)
- Created foxhunt K8s namespace, secret templates, migration job
- Added liveness probes to all 7 K8s services
- IB Gateway manifest (ghcr.io/gnzsnz/ib-gateway:stable)
- IBKR credentials in Scaleway Secret Manager via Terragrunt
- Fixed port collisions and mismatches across services

Production safety (9 critical + 6 high/medium fixes):
- Asset-class-specific VaR volatility (not flat 2%)
- Real parametric VaR with z-score 95th percentile
- Kyle's lambda regression (100-bar rolling window)
- Per-feature running statistics from historical data
- VWAP-based slippage reference, regime duration tracking
- Real Databento JSON parsing for OHLCV/Trade/Quote

Code health:
- Removed #![allow(dead_code)] from ml, data, config
- Fixed log:: → tracing:: in 4 production files
- Removed dead workspace deps (ratatui, crossterm)

Verified: cargo check --workspace (0 errors), trading_engine 330 tests pass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 00:32:10 +01:00
jgrusewski
020e85af2b infra: add DNS module + rename gitlab.fxhnt.ai → git.fxhnt.ai
Keep existing git.fxhnt.ai hostname for GitLab CE (repoint DNS
to new Tailscale IP after deploy). Add Terraform DNS module to
manage the A record via Terragrunt instead of manual scw CLI.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:26:51 +01:00
jgrusewski
1c36941ded infra(kapsule): add gitlab and ci-build node pools
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:21:14 +01:00
jgrusewski
ce8eeba6fb infra(storage): add GitLab registry and artifacts buckets 2026-02-24 23:21:06 +01:00
jgrusewski
019bc140d7 fix(ci): increase idle timeout to 60min, check act_runner children
The 15min timeout was too short for Rust builds. Also checks
for act_runner child processes (image pull, git clone phases)
not just Docker containers.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 22:52:28 +01:00
jgrusewski
903bf4fbf6 infra(ci): scale-to-zero with idle shutdown and auto-start watcher
CI runner auto-shuts down after 15min idle (no GITEA-ACTIONS containers).
Gitea server polls every 60s for queued runs and starts instance via SCW API.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 22:33:42 +01:00
jgrusewski
a47e0cb947 infra(ci): add Terragrunt-managed CI runner with Tailscale + act_runner
GP1-S instance (8 vCPU, 32GB, 100GB root) provisioned via Terraform
with cloud-init: Docker CE, Tailscale (pre-auth key), Gitea Actions
runner (act_runner v0.3.0). Security group drops all inbound — access
via Tailscale only. Solves: Kapsule pods cannot reach Tailscale IPs.

Also stores Tailscale API key in SCW Secret Manager for future auth
key generation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 22:15:37 +01:00
jgrusewski
5fb84a02f0 feat: training pipeline for all 10 ML ensemble models
- Add standalone training binaries: TGGN, KAN, xLSTM, Diffusion (DBN data)
- Update Dockerfile.training: 6 → 16 binaries (all 10 models + hyperopt + baseline)
- Expand train.sh: 4 → 10 models, fix registry URL and GPU pool nodeSelector
- Add GPU overlay manifests for trading-service and ml-training-service
- Create training data PVC and upload pod manifests
- Expand web-gateway model validation: 4 → 10 types (training + tune routes)
- Extend dashboard: 10 model cards grouped by category (RL/Temporal/Graph/Generative)
- Add training image build job to Gitea CI workflow
- Update GPU taint controller to exclude inference pool from tainting
- Fix job-template nodeSelector: gpu → gpu-training

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 20:47:50 +01:00
jgrusewski
5380bd9186 infra: split GPU pool into training (H100) and inference (L4), add auto-taint
- Split single H100 GPU pool into two purpose-specific pools:
  - gpu-training: H100-1-80G (€2.73/hr) for 10-model ensemble training
  - gpu-inference: L4-1-24G (€0.75/hr) for cost-effective trading inference
- Add GPU taint controller DaemonSet that auto-taints new GPU nodes
  with nvidia.com/gpu=true:NoSchedule to prevent non-GPU workloads
- Fix service log directory permissions with emptyDir volumes
  (observability init fails creating /app/logs as non-root user)
- Increase postgres max_connections from 25 to 100
  (7 services each requesting connection pools exhausted the limit)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 17:39:56 +01:00
jgrusewski
1441f5fbbb infra: security hardening and autoscale-to-zero for CI/GPU pools
- K8s API ACL restricted to Tailscale CGNAT (100.64.0.0/10) + admin IPs
- CI and GPU pools: min_size=0 with lifecycle ignore_changes on size
  so autoscaler manages actual node count without Terraform drift
- Add harden.sh script for post-provisioning security lockdown
  (ACL setup, exposed service check, bucket ACL verify, registry visibility)
- Update smoke-test.sh default region to fr-par

Security audit results:
- All services ClusterIP only (no LoadBalancer/NodePort)
- No Ingress resources
- Container registry: private
- S3 buckets: HTTP 403 on anonymous access
- Gitea: all ports closed on public IP (Tailscale-only via security group)
- Kapsule nodes: no open ports on external IPs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 15:46:24 +01:00
jgrusewski
f434309f59 infra: provision Kapsule cluster in fr-par with GPU support
Move all infrastructure from nl-ams to fr-par-2 for H100 GPU availability.
Kapsule cluster provisioned with 3 node pools (always-on DEV1-M, CI GP1-XS,
GPU H100-1-80G), databases deployed (Postgres/TimescaleDB, Redis, QuestDB),
and Tailscale subnet router configured.

Key changes:
- Region: nl-ams -> fr-par-2 (GPU types only available in Paris)
- K8s version: 1.30 -> 1.34 with auto-upgrade
- Block storage: migrate b_ssd -> SBS (scaleway_block_volume)
- Kapsule: add VPC private network, delete_additional_resources
- GPU pool: conditional via enable_gpu_pool variable
- K8s manifests: fix registry URLs, secret key names, PG subPath
- Add terragrunt cache/lockfile to .gitignore

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 15:35:40 +01:00
jgrusewski
3e75afa160 infra: add OpenTofu modules for all Scaleway resources
- kapsule: cluster + 3 node pools (always-on, ci, gpu)
- object-storage: S3 bucket with 30-day sccache expiry
- registry: private Container Registry namespace
- secrets: JWT + DB password via Secret Manager
- block-storage: 100GB b_ssd for training data

All managed via Terragrunt with shared provider/backend.
Gitignore exception added for infra/*/secrets/ (TF code, not actual secrets).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 14:23:03 +01:00