Commit Graph

150 Commits

Author SHA1 Message Date
jgrusewski
c20ce9fdb8 docs: GPU resource optimization implementation plan
4 tasks: add .train-rl-base template, switch RL jobs, reduce epochs, validate.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 16:16:55 +01:00
jgrusewski
2286b8976a docs: GPU resource optimization design — RL→L4, supervised→L40S
Route DQN/PPO to cheaper L4 (CPU-bound), supervised models to L40S
(GPU-bound). Separate CI templates with tuned resource requests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 16:15:07 +01:00
jgrusewski
94f98c65f8 docs: add autonomous agents implementation plan (23 tasks, 6 phases)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 08:20:56 +01:00
jgrusewski
9937c67b4a docs: autonomous CI agents design — OpenHands + GitLab issue bus
5-phase agent loop (diagnose → plan → implement → verify → deliver)
with GitLab issues as durable communication/handoff layer. OpenHands
on Kapsule K8s runtime, LiteLLM proxy routing to Claude Opus/Sonnet
and Scaleway devstral. Rust dispatcher service. Auto-merge for CI
failure fixes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 08:14:17 +01:00
jgrusewski
4860395dbf docs: add implementation plan for Cockpit metrics + NAT gateway
10 tasks across 2 workstreams (NAT + Metrics), mostly parallelizable.
Includes Terraform modules, K8s manifests, Helm values, and runbook.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 00:30:06 +01:00
jgrusewski
084f11a8e7 docs: add Cockpit metrics + NAT gateway design
Covers two workstreams:
1. Scaleway Public Gateway for NAT (all pools private-only)
2. Scaleway Cockpit + Alloy + DCGM Exporter for full metrics

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 00:27:23 +01:00
jgrusewski
1a3e89f0c7 docs: add on-demand training dispatch implementation plan
10 tasks: completion markers, training-uploader crate, proto RPCs,
K8s dispatcher, promotion manager, fxt CLI commands, infra updates.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 11:42:54 +01:00
jgrusewski
6c39413680 docs: add on-demand training dispatch design
K8s Job dispatch from ml_training_service with Rust sidecar uploader,
S3 artifact storage, and model promotion with fxt CLI approval gate.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 11:39:05 +01:00
jgrusewski
6e339316cf feat(ml): add manually-triggered GitLab CI training pipeline
Adds a parent/child GitLab CI pipeline for ML model training:

- Generator script produces per-model hyperopt/train/evaluate jobs
- Parent pipeline (.gitlab-ci-training.yml) with manual trigger
- NFS-backed ReadWriteMany PVC for shared training outputs
- Hyperopt params wired into training binaries (DQN, PPO, TFT, Mamba2)
- Shared DBN loader eliminates duplicate code across hyperopt adapters
- Supervised hyperopt unified to DBN data (was parquet-only)

Pipeline: hyperopt (4 models) → train (10 models) → evaluate ensemble

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 09:04:58 +01:00
jgrusewski
754baeb434 perf(ci): skip check+clippy stage — test already compiles workspace
The check stage ran cargo check + clippy (~6min) before tests, but
the test stage compiles the same workspace anyway. Skip check with
an echo pass-through to save ~6min per pipeline. Easy to uncomment
when clippy enforcement is needed (e.g., before releases).

Also adds sccache stats to the test job for cache monitoring.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 00:24:57 +01:00
jgrusewski
387bc0d400 docs: add unified training binaries implementation plan
8-task plan covering rename, new supervised binary, deletions, and infra updates.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 23:05:10 +01:00
jgrusewski
45483d203a docs: unified training binaries design
Consolidate 21 training examples into 2 binaries:
- train_baseline_rl (DQN, PPO with RL walk-forward)
- train_baseline_supervised (8 models via UnifiedTrainable)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 22:59:51 +01:00
jgrusewski
ac688cac24 fix(ci): skip Rust builds on infra-only changes, remove stale files
Add `rules:` with `changes:` filters to check, test, and .kaniko-base
jobs so the Rust build pipeline only triggers when source code changes
(crates/, bin/, services/, Cargo.*). Prevents H100 spin-up on
infra-only pushes.

Remove 31 stale/orphan files (-11,885 lines):
- monitoring/ directory (duplicated by config/grafana + config/prometheus)
- 10 broken/placeholder migration files (.broken, .skip)
- 10 unreferenced config files (haproxy, nginx-lb, mutants, etc.)

Add 3 historical design docs from previous restructure work.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 22:45:44 +01:00
jgrusewski
254b72d111 docs: IaC pipeline implementation plan (10 tasks)
Tasks 1-7: code changes (cleanup, module updates, Dockerfile, CI jobs,
root.hcl migration). Tasks 8-10: manual ops (state migration, resource
imports, image build, schedule setup).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 21:44:20 +01:00
jgrusewski
7d80a0db48 docs: IaC pipeline & tfstate migration design
Covers tfstate migration nl-ams→fr-par, resource import for
drift gaps (sccache bucket, foxhunt-ci registry, DNS records),
and GitLab CI/CD pipeline (plan on MR, apply on merge, weekly
drift detection).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 21:41:45 +01:00
jgrusewski
74615823ce docs: add DevPod implementation plan (10 tasks)
Terraform pool, PVC, Dockerfile, devcontainer.json, pod template,
CI job, setup script, infra apply, GitLab badge. TDD-style steps
with exact file paths and commands.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 21:09:31 +01:00
jgrusewski
f3a3b61049 docs: add DevPod remote development environment design
DevPod on Kapsule with GP1-L (16 vCPU, 64GB) autoscaling to zero,
persistent PVCs for Claude Code + cargo cache, "Open in DevPod"
GitLab badge, CI-built devcontainer image.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 21:06:51 +01:00
jgrusewski
1bc4e7e10a docs: GitLab CE migration implementation plan (12 tasks, 5 phases)
Covers: Terragrunt node pools, object storage buckets, Postgres init,
GitLab CE Helm deploy, Runner with K8s executor, .gitlab-ci.yml
translation (Kaniko builds), manifest cutover, decommission steps.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:16:50 +01:00
jgrusewski
3c4af61cb0 docs: GitLab CE migration design (Gitea → GitLab on Kapsule)
Covers: in-cluster GitLab CE with external Postgres/Redis, dedicated
ci-build node pool (scale-to-zero), Kaniko image builds, GitLab
built-in registry, and 4-phase migration plan with rollback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:13:00 +01:00
jgrusewski
c8f2dacc5f feat(infra): H100 smoketest pipeline — S3 output sync, sccache, registry fix
- Fix container registry region nl-ams → fr-par across all CI jobs
- Add sccache build-args to training image build (no-op fallback for local)
- Add rclone to training Docker runtime for S3 output sync
- Update train.sh: S3 sync on Job completion via rclone env-var config,
  --run-id tracking, evaluate preset for walk-forward evaluation
- Add s3-credentials Secret template (.example, apply via kubectl)
- Add design doc and implementation plan

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 21:31:22 +01:00
jgrusewski
5fb84a02f0 feat: training pipeline for all 10 ML ensemble models
- Add standalone training binaries: TGGN, KAN, xLSTM, Diffusion (DBN data)
- Update Dockerfile.training: 6 → 16 binaries (all 10 models + hyperopt + baseline)
- Expand train.sh: 4 → 10 models, fix registry URL and GPU pool nodeSelector
- Add GPU overlay manifests for trading-service and ml-training-service
- Create training data PVC and upload pod manifests
- Expand web-gateway model validation: 4 → 10 types (training + tune routes)
- Extend dashboard: 10 model cards grouped by category (RL/Temporal/Graph/Generative)
- Add training image build job to Gitea CI workflow
- Update GPU taint controller to exclude inference pool from tainting
- Fix job-template nodeSelector: gpu → gpu-training

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 20:47:50 +01:00
jgrusewski
6210a5d4a3 docs: update implementation plan for OpenTofu + Terragrunt
Replaced imperative shell scripts with declarative OpenTofu modules
and Terragrunt DRY configuration. 16 tasks across 4 batches.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 14:28:16 +01:00
jgrusewski
67f01ac4a1 docs: add Kapsule infrastructure implementation plan
14 tasks across 5 batches: cluster foundation, CI/CD,
paper trading deployment, GPU training, and verification.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 14:12:45 +01:00
jgrusewski
73cb7e500f docs: add Kapsule infrastructure design
Scaleway Kapsule with 3 node pools (always-on, CI, GPU),
Tailscale-only networking, H100-preferred GPU training,
and Gitea Actions CI/CD pipeline.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 14:07:56 +01:00
jgrusewski
ff15341c3e Merge branch 'worktree-ml-production-hardening' 2026-02-24 09:56:03 +01:00
jgrusewski
57bae2cb68 fix(ml): OOM hardening + battle-test KAN/xLSTM/Diffusion models
Replace 8 unbounded Vec accumulation patterns with bounded VecDeque
across ensemble, PPO, DQN, Mamba2, and data pipeline code to prevent
OOM on RTX 3050 Ti (4GB VRAM) during live trading and extended training.

Key OOM fixes:
- Ensemble price/volatility history: Vec → VecDeque with O(1) eviction
- Data pipeline: MAX_FEATURES=500K cap (~512MB) prevents unbounded loading
- DQN replay buffer: full-array shuffle → HashSet random sampling (8MB → 256B)
- PPO loss histories: bounded VecDeque (cap 1K), eliminated batch.clone()
- Mamba2 scan: pre-allocated Vecs, explicit drop() after Tensor::cat
- Mamba2 training history: capped at 100, Tensor::randn replaces Vec→Tensor
- Mamba2 SSM reset: 2 unwrap() violations replaced with proper error handling

Battle-testing (19 new integration tests):
- KAN: 5 tests (forward, 50-epoch training 89.9% loss reduction, checkpoint)
- xLSTM: 7 tests (2D+3D forward, 30-epoch training 82% reduction, checkpoint)
- Diffusion: 7 tests (2D+3D forward, 20-epoch pipeline, checkpoint, validation)

Bonus: fix pre-existing cache test failure (match .dbn.zst files, graceful skip)

All 2390 lib tests pass, 0 new clippy errors.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 09:55:39 +01:00
jgrusewski
f2d9fb9328 docs: update stub audit results with actual fix status
All 18 findings now have resolution status:
- 8 fixed with real implementations
- 3 fixed with honest errors (instead of fake data)
- 4 deferred (need infrastructure)
- 3 acceptable as-is

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 09:46:17 +01:00
jgrusewski
9dd967a016 Merge cleanup: warnings, legacy vendor, dependency trimming
- fix(ml): resolve all 12 warnings (unused imports, Debug impls, lifetimes)
- chore: remove broken vendor/candle-optimisers gitlink
- chore: replace tokio features=["full"] with workspace defaults in 5 crates
- chore: remove unused plotters dep from services/load_tests
- feat(api_gateway): feature-gate MFA deps behind optional 'mfa' feature

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 02:08:34 +01:00
jgrusewski
5be297501b docs: add stub audit results — 18 findings across 37 crates
8 critical stubs found in risk-data, adaptive-strategy, trading_service,
and ml/integration. 3 crate groups clean (infra, frontend, CLI).
Priority: VPIN calculator, portfolio positions, enhanced_ml endpoints.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:48:12 +01:00
jgrusewski
5f7a913a7b docs: add stub detection implementation plan
9-task plan: 6 parallel audit agents, aggregate report,
pre-commit lint rules, and TLI→CLI rename.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:42:48 +01:00
jgrusewski
decf9ef957 docs: add stub detection & prevention design
Approved design for full-codebase audit using 6 parallel agents
scanning 14 stub patterns, plus lint prevention in pre-commit hook.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:39:42 +01:00
jgrusewski
72addbbbc4 docs: add cleanup implementation plan
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:29:24 +01:00
jgrusewski
b1a12ce7d9 docs: add cleanup design plan for warnings, deps, and legacy removal
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:26:23 +01:00
jgrusewski
c3b5e124f0 chore: update .gitignore and add design plan docs
Ignore ML checkpoints, trained model safetensors, stray ml/ml/ dir,
and .claude/worktrees/. Clean up duplicate hive-mind-prompt entries.
Add 17 design/implementation plan docs from 2026-02-20 to 2026-02-22.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 00:30:15 +01:00
jgrusewski
d45692934a docs: production safety audit implementation plan (38 tasks, 5 layers)
Layer 0: Data Integrity (7 fixes) — position race, price validation, counters
Layer 1: Risk Enforcement (10 fixes) — order atomicity, overfill, risk bypass
Layer 2: ML Pipeline (10 fixes) — random weights, epsilon-greedy, NaN validation
Layer 3: Broker Safety (5 fixes) — connection drops, volume overflow, DB sync
Layer 4: Auth & Ops (6 fixes) — auth stub gating, live trading confirmation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 19:00:29 +01:00
jgrusewski
7e9a3841f0 docs: production safety audit design — 38 fixes across 5 layers
4-domain code audit found 13 CRITICAL, 14 HIGH, 10 MEDIUM issues.
Organized as layer-by-layer remediation:
- Layer 0: Data integrity (positions, prices, market data)
- Layer 1: Risk enforcement (make checks actually block)
- Layer 2: ML pipeline (real weights, bounded predictions)
- Layer 3: Broker safety (connection handling, volumes)
- Layer 4: Auth & ops (credentials, rate limiting, monitoring)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 18:51:54 +01:00
jgrusewski
17c49f0788 docs: add real data training pipeline design and implementation plan
Design: Download 730 days Databento OHLCV-1m for 4 CME futures,
train DQN + PPO with hyperopt, walk-forward evaluation.

Implementation: 9 tasks — quarterly download binary, walk-forward
splitter, hyperopt runner, training binary, evaluation binary,
DBN wiring, integration tests, manual execution.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 18:07:03 +01:00
jgrusewski
7b870726ab docs: add production pipeline wiring implementation plan (10 tasks)
Detailed TDD implementation plan for wiring all 10 ML models into
the ensemble coordinator. Covers 5 new inference adapters (KAN, TGGN,
xLSTM, TLOB, Diffusion), main.rs registration with equal weights,
market data wiring, E2E integration test, Docker/CI validation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 14:59:18 +01:00
jgrusewski
d268a9dca8 docs: add production pipeline wiring design (6 tasks)
Wire existing components into end-to-end pipeline: 5 missing inference
adapters, 10-model registration, market data feeding, integration test,
Docker build validation, and CI pipeline triage.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 14:43:22 +01:00
jgrusewski
ae225b0e31 docs: add trading universe & data organization implementation plan
10-task plan: unify AssetClass, add trading_symbol field, futures_baseline()
preset, TOML config loading, DatasetSpec::from_universe() wiring, data
reorganization into cache structure, stale data cleanup, manifest generation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 14:21:48 +01:00
jgrusewski
66966806c7 docs: add trading universe & data organization design
Define futures baseline universe (ES, NQ, ZN, 6E) with micro
contract mapping for limited capital. Reorganize scattered test
data into cache structure. Download spec for 730 days of OHLCV-1m.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 14:17:22 +01:00
jgrusewski
9f499282ac docs: add data pipeline & asset selection implementation plan
8-task plan across 3 phases: data pipeline (types, cache, manager,
prepared dataset), asset selection (universe, scorer, selector),
and integration wiring. 7 new files, ~51 tests estimated.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 12:36:45 +01:00
jgrusewski
df251a9d7e docs: add data pipeline and asset selection design
Covers automated data downloading/caching for ML training and
a tiered asset selection funnel (universe → predictability → regime → signal).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 12:28:02 +01:00
jgrusewski
f0bafdf30b docs: add operational maturity implementation plan (12 tasks, 6 phases)
Three pillars: 7-gate conviction system, autonomous feedback loop
with kill switch, Rust-native model registry. QuestDB analytics layer.
~55 tests, 7 new files planned across ml/ and trading_service/.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 12:11:28 +01:00
jgrusewski
abf4baf30e docs: add operational maturity system design (3 pillars)
7-gate conviction system, autonomous feedback loop with kill switch,
and Rust-native model registry — backed by PostgreSQL + QuestDB.
Resolve merge conflicts in hyperopt/adapters/mod.rs and enhanced_ml.rs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 12:02:52 +01:00
jgrusewski
10f9cfadb7 feat(infra): GPU training launcher with local/cloud routing
Add train_launcher.sh that detects local GPU VRAM and routes training
to local or Scaleway cloud. Auto-selects batch size per model based on
available VRAM tier. Maps model names to actual ml/examples/train_*.rs
cargo targets. Document Scaleway GPU instance types, setup procedure,
batch size tables, and cost estimates.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 11:19:58 +01:00
jgrusewski
17727d5db9 docs: add dynamic GPU detection implementation plan
8 bite-sized tasks with TDD approach, exact file paths, and complete code.
Covers: DeviceConfig relocation, GpuCapabilities detection,
ModelMemoryEstimate per-model profiles, and wiring into PPO trainer
and hyperopt campaign.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 10:56:45 +01:00
jgrusewski
ea81c751c0 docs: add dynamic GPU detection design plan
Centralizes GPU device selection, capability detection, and per-model
batch size optimization to replace hardcoded RTX 3050 Ti limits.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 10:53:04 +01:00
jgrusewski
15c0fb5397 merge: pull liquid CfC v2 and codebase deduplication from main into production-hardening 2026-02-23 10:09:25 +01:00
jgrusewski
7fd5ad4286 docs: production hardening phase 2 implementation plan — 19 tasks
4 layers: Safety Net (6 tasks), Correctness (6 tasks incl. CfC ensemble),
Verification (4 tasks), Training Infrastructure (3 tasks incl. Databento).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 09:39:13 +01:00