Commit Graph

130 Commits

Author SHA1 Message Date
jgrusewski
5fb84a02f0 feat: training pipeline for all 10 ML ensemble models
- Add standalone training binaries: TGGN, KAN, xLSTM, Diffusion (DBN data)
- Update Dockerfile.training: 6 → 16 binaries (all 10 models + hyperopt + baseline)
- Expand train.sh: 4 → 10 models, fix registry URL and GPU pool nodeSelector
- Add GPU overlay manifests for trading-service and ml-training-service
- Create training data PVC and upload pod manifests
- Expand web-gateway model validation: 4 → 10 types (training + tune routes)
- Extend dashboard: 10 model cards grouped by category (RL/Temporal/Graph/Generative)
- Add training image build job to Gitea CI workflow
- Update GPU taint controller to exclude inference pool from tainting
- Fix job-template nodeSelector: gpu → gpu-training

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 20:47:50 +01:00
jgrusewski
6210a5d4a3 docs: update implementation plan for OpenTofu + Terragrunt
Replaced imperative shell scripts with declarative OpenTofu modules
and Terragrunt DRY configuration. 16 tasks across 4 batches.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 14:28:16 +01:00
jgrusewski
67f01ac4a1 docs: add Kapsule infrastructure implementation plan
14 tasks across 5 batches: cluster foundation, CI/CD,
paper trading deployment, GPU training, and verification.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 14:12:45 +01:00
jgrusewski
73cb7e500f docs: add Kapsule infrastructure design
Scaleway Kapsule with 3 node pools (always-on, CI, GPU),
Tailscale-only networking, H100-preferred GPU training,
and Gitea Actions CI/CD pipeline.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 14:07:56 +01:00
jgrusewski
ff15341c3e Merge branch 'worktree-ml-production-hardening' 2026-02-24 09:56:03 +01:00
jgrusewski
57bae2cb68 fix(ml): OOM hardening + battle-test KAN/xLSTM/Diffusion models
Replace 8 unbounded Vec accumulation patterns with bounded VecDeque
across ensemble, PPO, DQN, Mamba2, and data pipeline code to prevent
OOM on RTX 3050 Ti (4GB VRAM) during live trading and extended training.

Key OOM fixes:
- Ensemble price/volatility history: Vec → VecDeque with O(1) eviction
- Data pipeline: MAX_FEATURES=500K cap (~512MB) prevents unbounded loading
- DQN replay buffer: full-array shuffle → HashSet random sampling (8MB → 256B)
- PPO loss histories: bounded VecDeque (cap 1K), eliminated batch.clone()
- Mamba2 scan: pre-allocated Vecs, explicit drop() after Tensor::cat
- Mamba2 training history: capped at 100, Tensor::randn replaces Vec→Tensor
- Mamba2 SSM reset: 2 unwrap() violations replaced with proper error handling

Battle-testing (19 new integration tests):
- KAN: 5 tests (forward, 50-epoch training 89.9% loss reduction, checkpoint)
- xLSTM: 7 tests (2D+3D forward, 30-epoch training 82% reduction, checkpoint)
- Diffusion: 7 tests (2D+3D forward, 20-epoch pipeline, checkpoint, validation)

Bonus: fix pre-existing cache test failure (match .dbn.zst files, graceful skip)

All 2390 lib tests pass, 0 new clippy errors.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 09:55:39 +01:00
jgrusewski
f2d9fb9328 docs: update stub audit results with actual fix status
All 18 findings now have resolution status:
- 8 fixed with real implementations
- 3 fixed with honest errors (instead of fake data)
- 4 deferred (need infrastructure)
- 3 acceptable as-is

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 09:46:17 +01:00
jgrusewski
9dd967a016 Merge cleanup: warnings, legacy vendor, dependency trimming
- fix(ml): resolve all 12 warnings (unused imports, Debug impls, lifetimes)
- chore: remove broken vendor/candle-optimisers gitlink
- chore: replace tokio features=["full"] with workspace defaults in 5 crates
- chore: remove unused plotters dep from services/load_tests
- feat(api_gateway): feature-gate MFA deps behind optional 'mfa' feature

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 02:08:34 +01:00
jgrusewski
5be297501b docs: add stub audit results — 18 findings across 37 crates
8 critical stubs found in risk-data, adaptive-strategy, trading_service,
and ml/integration. 3 crate groups clean (infra, frontend, CLI).
Priority: VPIN calculator, portfolio positions, enhanced_ml endpoints.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:48:12 +01:00
jgrusewski
5f7a913a7b docs: add stub detection implementation plan
9-task plan: 6 parallel audit agents, aggregate report,
pre-commit lint rules, and TLI→CLI rename.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:42:48 +01:00
jgrusewski
decf9ef957 docs: add stub detection & prevention design
Approved design for full-codebase audit using 6 parallel agents
scanning 14 stub patterns, plus lint prevention in pre-commit hook.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:39:42 +01:00
jgrusewski
72addbbbc4 docs: add cleanup implementation plan
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:29:24 +01:00
jgrusewski
b1a12ce7d9 docs: add cleanup design plan for warnings, deps, and legacy removal
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 01:26:23 +01:00
jgrusewski
c3b5e124f0 chore: update .gitignore and add design plan docs
Ignore ML checkpoints, trained model safetensors, stray ml/ml/ dir,
and .claude/worktrees/. Clean up duplicate hive-mind-prompt entries.
Add 17 design/implementation plan docs from 2026-02-20 to 2026-02-22.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 00:30:15 +01:00
jgrusewski
d45692934a docs: production safety audit implementation plan (38 tasks, 5 layers)
Layer 0: Data Integrity (7 fixes) — position race, price validation, counters
Layer 1: Risk Enforcement (10 fixes) — order atomicity, overfill, risk bypass
Layer 2: ML Pipeline (10 fixes) — random weights, epsilon-greedy, NaN validation
Layer 3: Broker Safety (5 fixes) — connection drops, volume overflow, DB sync
Layer 4: Auth & Ops (6 fixes) — auth stub gating, live trading confirmation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 19:00:29 +01:00
jgrusewski
7e9a3841f0 docs: production safety audit design — 38 fixes across 5 layers
4-domain code audit found 13 CRITICAL, 14 HIGH, 10 MEDIUM issues.
Organized as layer-by-layer remediation:
- Layer 0: Data integrity (positions, prices, market data)
- Layer 1: Risk enforcement (make checks actually block)
- Layer 2: ML pipeline (real weights, bounded predictions)
- Layer 3: Broker safety (connection handling, volumes)
- Layer 4: Auth & ops (credentials, rate limiting, monitoring)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 18:51:54 +01:00
jgrusewski
17c49f0788 docs: add real data training pipeline design and implementation plan
Design: Download 730 days Databento OHLCV-1m for 4 CME futures,
train DQN + PPO with hyperopt, walk-forward evaluation.

Implementation: 9 tasks — quarterly download binary, walk-forward
splitter, hyperopt runner, training binary, evaluation binary,
DBN wiring, integration tests, manual execution.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 18:07:03 +01:00
jgrusewski
7b870726ab docs: add production pipeline wiring implementation plan (10 tasks)
Detailed TDD implementation plan for wiring all 10 ML models into
the ensemble coordinator. Covers 5 new inference adapters (KAN, TGGN,
xLSTM, TLOB, Diffusion), main.rs registration with equal weights,
market data wiring, E2E integration test, Docker/CI validation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 14:59:18 +01:00
jgrusewski
d268a9dca8 docs: add production pipeline wiring design (6 tasks)
Wire existing components into end-to-end pipeline: 5 missing inference
adapters, 10-model registration, market data feeding, integration test,
Docker build validation, and CI pipeline triage.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 14:43:22 +01:00
jgrusewski
ae225b0e31 docs: add trading universe & data organization implementation plan
10-task plan: unify AssetClass, add trading_symbol field, futures_baseline()
preset, TOML config loading, DatasetSpec::from_universe() wiring, data
reorganization into cache structure, stale data cleanup, manifest generation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 14:21:48 +01:00
jgrusewski
66966806c7 docs: add trading universe & data organization design
Define futures baseline universe (ES, NQ, ZN, 6E) with micro
contract mapping for limited capital. Reorganize scattered test
data into cache structure. Download spec for 730 days of OHLCV-1m.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 14:17:22 +01:00
jgrusewski
9f499282ac docs: add data pipeline & asset selection implementation plan
8-task plan across 3 phases: data pipeline (types, cache, manager,
prepared dataset), asset selection (universe, scorer, selector),
and integration wiring. 7 new files, ~51 tests estimated.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 12:36:45 +01:00
jgrusewski
df251a9d7e docs: add data pipeline and asset selection design
Covers automated data downloading/caching for ML training and
a tiered asset selection funnel (universe → predictability → regime → signal).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 12:28:02 +01:00
jgrusewski
f0bafdf30b docs: add operational maturity implementation plan (12 tasks, 6 phases)
Three pillars: 7-gate conviction system, autonomous feedback loop
with kill switch, Rust-native model registry. QuestDB analytics layer.
~55 tests, 7 new files planned across ml/ and trading_service/.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 12:11:28 +01:00
jgrusewski
abf4baf30e docs: add operational maturity system design (3 pillars)
7-gate conviction system, autonomous feedback loop with kill switch,
and Rust-native model registry — backed by PostgreSQL + QuestDB.
Resolve merge conflicts in hyperopt/adapters/mod.rs and enhanced_ml.rs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 12:02:52 +01:00
jgrusewski
10f9cfadb7 feat(infra): GPU training launcher with local/cloud routing
Add train_launcher.sh that detects local GPU VRAM and routes training
to local or Scaleway cloud. Auto-selects batch size per model based on
available VRAM tier. Maps model names to actual ml/examples/train_*.rs
cargo targets. Document Scaleway GPU instance types, setup procedure,
batch size tables, and cost estimates.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 11:19:58 +01:00
jgrusewski
17727d5db9 docs: add dynamic GPU detection implementation plan
8 bite-sized tasks with TDD approach, exact file paths, and complete code.
Covers: DeviceConfig relocation, GpuCapabilities detection,
ModelMemoryEstimate per-model profiles, and wiring into PPO trainer
and hyperopt campaign.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 10:56:45 +01:00
jgrusewski
ea81c751c0 docs: add dynamic GPU detection design plan
Centralizes GPU device selection, capability detection, and per-model
batch size optimization to replace hardcoded RTX 3050 Ti limits.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 10:53:04 +01:00
jgrusewski
15c0fb5397 merge: pull liquid CfC v2 and codebase deduplication from main into production-hardening 2026-02-23 10:09:25 +01:00
jgrusewski
7fd5ad4286 docs: production hardening phase 2 implementation plan — 19 tasks
4 layers: Safety Net (6 tasks), Correctness (6 tasks incl. CfC ensemble),
Verification (4 tasks), Training Infrastructure (3 tasks incl. Databento).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 09:39:13 +01:00
jgrusewski
fad71033ff docs: production hardening phase 2 design — $100K live trading readiness
4-layer risk-prioritized approach: safety net (crash prevention),
correctness (accurate calculations + liquid CfC ensemble integration),
verification (test coverage), training infrastructure (Databento + GPU).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 09:33:55 +01:00
jgrusewski
d9fa536511 docs: add ML ensemble expansion implementation plan (36 tasks, 6 phases)
Sequential build plan for 5 model integrations (TGGN, TLOB, KAN,
xLSTM, Diffusion) with TDD, worktree isolation, and full trait
implementations (UnifiedTrainable + ParameterSpace + hyperopt).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 00:05:28 +01:00
jgrusewski
1408d33f4f docs: add ML ensemble expansion design (7→10 models, 5 integrations)
Approved design for expanding the ensemble with KAN, xLSTM, and
Diffusion architectures plus bringing TGGN/TLOB to full integration.
Sequential build order, no god classes, modular decomposition.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 23:56:38 +01:00
jgrusewski
8b74a9a42e docs: add production hardening implementation plan (53 tasks, 5 phases)
Phase 1: ML pipeline verification (checkpoint roundtrip, feature extraction)
Phase 2: Service production logic (hardcoded values, stub responses)
Phases 3-5: Backtesting, ML crate TODOs, infrastructure/security/cleanup

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 22:51:56 +01:00
jgrusewski
340ec0d68f docs: add production hardening design (53 tasks, 5 phases)
Covers all TODO/FIXME items across workspace, checkpoint roundtrip
tests, feature extraction pipeline tests, and infrastructure cleanup.
No overlap with dedup-cleanup or liquid-cfc plans.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 22:33:08 +01:00
jgrusewski
3e1c415b8d docs: add codebase de-duplication implementation plan
18 tasks across 4 phases:
- Phase 1: OrderType, ConfigError, TLS types (zero risk)
- Phase 2: ErrorCategory, ModelType, re-export cleanup (low risk)
- Phase 3: Dead code audit, config warnings, Adam move (low risk)
- Phase 4: CircuitBreaker trait + inline CB replacement (medium risk)

Scoped out: ErrorSeverity (semantically different variants),
RetryStrategy (complex merge), trading_engine/risk CBs (domain-specific)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 20:01:12 +01:00
jgrusewski
f31c852a45 docs: add codebase de-duplication design
4-phase bottom-up consolidation plan:
- Phase 1: TLS, ErrorSeverity, ConfigError, OrderType (zero risk)
- Phase 2: RetryStrategy, ErrorCategory, ModelType, re-exports (low risk)
- Phase 3: Dead code audit, hardcoded config warnings, Adam move (low risk)
- Phase 4: CircuitBreaker trait hierarchy (medium risk)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 19:54:53 +01:00
jgrusewski
30a96db895 infra: update cloud-init with lessons from Gitea deployment
Fixes discovered during live provisioning: app.ini permissions
(root:gitea + 660→640 for token generation), HTTPS via acme.sh
DNS-01, setcap for port 443, admin user creation, INSTALL_LOCK
lifecycle. Secrets replaced with placeholders.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 19:23:30 +01:00
jgrusewski
2fc92c80d5 docs: Phase 5 production readiness design and implementation plan
24 tasks across 5 pillars: runtime crash elimination, service wiring,
credential cleanup, clippy hardening, and remaining HIGH items.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 23:06:50 +01:00
jgrusewski
3e410f29e3 docs: Phase 4 production readiness design and implementation plan
16 tasks across 4 pillars: Safety & Crash Elimination, Trading
Correctness, Compliance & Code Quality, ML Pipeline Quality.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 20:21:01 +01:00
jgrusewski
86712a1216 docs: production readiness phase 3 implementation plan — 18 tasks
Bite-sized steps for each task with exact file paths, code snippets,
verification commands, and parallel execution strategy.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 19:00:43 +01:00
jgrusewski
aa6669ede0 docs: production readiness phase 3 design — 18 tasks across 4 pillars
Safety, live trading, security, and observability gaps identified
from comprehensive codebase audit. Organized for parallel swarm execution.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 18:53:38 +01:00
jgrusewski
4069eb473c docs: backtesting vertical slice implementation plan
8-task TDD plan to bridge 8 disconnected layers in the backtesting
pipeline: DBN converter, replay engine, 51-dim feature wiring,
model loader, registry startup, position tracking, PnL tracking,
and end-to-end integration test.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 15:30:00 +01:00
jgrusewski
4354098d11 docs: Phase 1-2 data pipeline and trading service implementation plan
13-task plan covering data pipeline fixes (Databento stream, data acquisition,
DBN uploader) and trading service core wiring (RiskEngine, MLEngine, VaR,
feature extraction, order matching, API gateway, Prometheus panics).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 15:25:17 +01:00
jgrusewski
30241e858d docs: backtesting vertical slice production readiness design
8-task plan to bridge disconnected backtesting pipeline layers:
DBN parser → feature extraction → ML inference → position tracking → PnL

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 15:21:22 +01:00
jgrusewski
e2a712593a docs: full system production readiness design (6-phase plan)
Covers data pipeline, trading service core, broker connectivity
(FIX 4.4 + TWS), execution algorithms, security/compliance, and
observability — based on comprehensive non-ML audit.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 15:18:19 +01:00
jgrusewski
faa7b85cea docs: real-data ensemble backtest and hyperopt deploy design
Two-phase design:
A) Real 6E.FUT data → 4-model ensemble → PaperBroker → P&L metrics
B) Best hyperopt params → train → deploy to ensemble → ValidationHarness

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 15:04:58 +01:00
jgrusewski
2dd3326aec docs: ensemble real inference design and implementation plan
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 14:58:36 +01:00
jgrusewski
919773a741 docs: full-stack ML integration design (cleanup → ensemble → hyperopt → paper trading)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 13:19:17 +01:00
jgrusewski
fbf9c8a5aa docs: capital-ready validation roadmap implementation plan
14 TDD tasks across 4 sections: data integrity (TemporalGuard,
SlippageModel), walk-forward robustness (DegradationTracker, NoiseInjector,
SensitivityAnalyzer), risk enforcement (RiskAction, RiskEnforcer,
GraduatedRecovery, CorrelationMonitor), and E2E integration (pipeline
traits, RiskGate, operating modes, DriftResponder, observability).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 10:39:54 +01:00