Commit Graph

213 Commits

Author SHA1 Message Date
jgrusewski
7c8fa991cc feat(infra): standalone Prometheus deployment for foxhunt namespace
- ServiceAccount + ClusterRole (nodes, pods, endpoints, metrics)
- ConfigMap with 7 scrape jobs: self, foxhunt-services, annotated-pods,
  gitlab-annotated-pods, node-exporter, kube-state-metrics, dcgm-exporter,
  pushgateway
- Deployment (Prometheus v3.8.1, 14d retention, 1500MB size limit)
- PVC (2Gi scw-bssd), Service (port 80 → 9090)

Also includes pods-panel-versioning design doc.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 13:24:52 +01:00
jgrusewski
6d37b95072 chore: add SubscribeClusterPods stub to trading_service + plan doc
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 13:17:41 +01:00
jgrusewski
d14b6a38f1 docs: TUI live data wiring implementation plan
17-task plan for replacing hardcoded mock data in fxt watch
with real gRPC streaming data from the API Gateway.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 02:04:08 +01:00
jgrusewski
1d0227b79e docs: TUI live data wiring design
Streaming-first architecture: 11 new poll-to-stream gateway adapters,
channel-based DataFetcher in the TUI, auto-reconnect with backoff.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 01:54:33 +01:00
jgrusewski
fb4df12376 Merge branch 'feature/fxt-overhaul'
fxt 2.0 overhaul: consolidated proto/, absorbed monitoring into API Gateway,
new 15-command CLI with gRPC, MCP server mode, TUI cockpit framework.

159 files changed, net -11,835 lines.
2026-03-03 23:16:45 +01:00
jgrusewski
7f5dfb70c6 docs: add DQN production hardening design
4-phase plan from deep audit: critical bug fixes (weight_decay, IQN PER,
q_value_std, 54/51 dim mismatch), train/eval parity (TradeExecutor,
checkpoint validation), SOTA improvements (differential Sharpe, PQN
LayerNorm, temporal PER, dormant neuron resets), and search space cleanup.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 20:40:46 +01:00
jgrusewski
ba8fb8eedd feat(ppo): wire symlog, adaptive entropy, percentile scaling into training loop
Integrate all PPO improvement modules into the core training paths:
- Symlog value predictions in compute_value_loss (MLP + LSTM)
- Adaptive entropy auto-tuning replaces fixed entropy_coeff
- Percentile P5/P95 advantage scaling for heavy-tailed returns
- DAPO clip_epsilon_high wired in all 7 PPOConfig construction sites
- Shape mismatch fix in adaptive_entropy (unsqueeze scalar)

2726 tests pass, 0 clippy errors.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 20:12:00 +01:00
jgrusewski
df93b4c534 docs: fxt 2.0 design and implementation plan
Clean rewrite into full-scale operations platform.
Single proto/ root, monitoring_service removed,
CLI + MCP + cockpit TUI with purple/cyan theme.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 20:11:21 +01:00
jgrusewski
a0a2a1d736 docs: add hyperopt improvements design and implementation plan
A+B approach: smooth penalties + position limits + CUDA cleanup +
search space reduction (45D→25D) + TPE optimizer replacing PSO.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 18:17:23 +01:00
jgrusewski
e9e6c431de docs: add BF16 training implementation plan (21 tasks, ~150 sites)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 16:02:10 +01:00
jgrusewski
2d16926d71 docs: add BF16 training design — pure BF16 for all 10 models
Dynamic dtype detection (Ampere+ → BF16, else F32), zero casts in
training hot path, cast only at data ingestion and loss scalar.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 15:56:14 +01:00
jgrusewski
866da8bbbb docs: RL agent gaps implementation plan — 7 tasks, vertical slices
Position-aware state, regime detection, slippage, curriculum learning,
multi-timeframe fusion, online learning with EWC, A/B testing extension.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 02:30:00 +01:00
jgrusewski
44b38e8014 docs: RL agent gaps design — 7 features for training quality and production readiness
Position-aware state encoding, regime detection (feature-based + HMM),
slippage modeling, curriculum learning, multi-timeframe fusion (default on),
online learning with EWC, and A/B testing with Thompson Sampling.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 02:23:38 +01:00
jgrusewski
0840f8a971 Merge branch 'worktree-dqn-action-collapse-fix' 2026-03-03 02:04:57 +01:00
jgrusewski
e9c0edd298 docs: fxt dashboard implementation plan (15 tasks)
Bite-sized TDD tasks: ratatui deps, proto RPCs, state types, stream
manager, renderer, event loop, CLI wiring, 6 stub fixes, clippy.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 01:20:27 +01:00
jgrusewski
5b328750a4 docs: fxt dashboard & CLI completion design
Ratatui streaming dashboard (fxt watch) with 4 tabs, stub command
wiring plan, and new ApproveModel/RejectModel proto RPCs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 01:14:02 +01:00
jgrusewski
97f69111c5 docs: DQN action collapse fix design — 5 fixes for 45-action diversity
Root causes: entropy regularizer dead code, noisy nets disable epsilon,
diversity penalty calibrated for 3 actions not 45.

Fixes: wire entropy into loss, fix normalization, epsilon floor for
noisy nets, sigma scheduling, UCB count-based bonus.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 00:59:20 +01:00
jgrusewski
2e3ac28e1b docs: implementation plan for 17 additional training metrics
9 tasks covering: training_metrics.rs (definitions + verbose gate),
monitoring.proto (12 new fields), service.rs (match arms + tests),
fxt monitor CLI (grad norm, RL diagnostics, hyperopt trial detail),
DQN/PPO trainers (per-epoch recording), supervised binary (LR + duration),
hyperopt binaries (elapsed_seconds), and full workspace verification.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 23:48:18 +01:00
jgrusewski
477b9cf1cf docs: GPU PER sum-tree implementation plan (13 tasks)
TDD plan covering GpuReplayBuffer, proportional + rank-based GPU
sampling, priority scatter updates, async loss readback, trainer
integration, OOM fallback, and distribution correctness tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 23:45:27 +01:00
jgrusewski
9105921f8d docs: additional training metrics design — 17 new Prometheus metrics
Two-tier system: 12 always-on gauges (RL diagnostics, gradient health,
hyperopt intra-trial) + 5 opt-in verbose metrics gated behind
FOXHUNT_VERBOSE_METRICS=1. Extends proto, monitoring service, and CLI.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 23:44:45 +01:00
jgrusewski
995c473ad9 docs: GPU-resident PER sum-tree design
Approved design for moving Prioritized Experience Replay entirely to
GPU — flat priority array with parallel prefix-sum sampling, GPU-resident
ring buffer for experiences, async loss readback. Eliminates the last
major CPU bottleneck in the DQN training loop.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 23:40:57 +01:00
jgrusewski
0c227bb7d0 docs: add monitoring service design — live training metrics over gRPC
New standalone monitoring_service that bridges Prometheus → gRPC,
enabling `fxt train monitor` and web-gateway WS streaming for
all training jobs (CI and service-managed).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 21:05:48 +01:00
jgrusewski
eefb191a62 docs: add implementation plan for fxt broker check command
7 tasks: deps → config → proto → broker.rs → main.rs wiring → lint → verify.
TDD approach with exact file paths and commands.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 13:28:51 +01:00
jgrusewski
50d1aff97c docs: add design for fxt broker check CLI command
Two-layer IBKR connectivity validation: direct ibapi handshake +
broker-gateway gRPC health check. Pure-client architecture using
ibapi as optional dep on fxt (not trading_engine).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 13:26:17 +01:00
jgrusewski
a489e6455e chore(infra): delete 8 per-service binary-cache PVCs (replaced by foxhunt-binaries)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 11:20:03 +01:00
jgrusewski
d35f4e0565 docs: binary cache PVC consolidation design (8 PVCs → 1 shared)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 10:58:33 +01:00
jgrusewski
58f8478320 infra(tf): add platform pool, rename services → foxhunt in Kapsule module
Terraform changes for 3-pool node split:
- New `platform` pool resource (DEV1-L, databases + monitoring)
- Renamed `services` → `foxhunt` pool (DEV1-L, max_size 1)
- Updated variables, outputs, and live terragrunt inputs
- Updated implementation plan with Terraform-based workflow

Apply in 2 phases:
  Phase 1: terragrunt apply -target=scaleway_k8s_pool.platform
  Phase 2: terragrunt apply (after databases migrated to platform)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 09:59:21 +01:00
jgrusewski
9942ba8c29 docs: 3-pool node split implementation plan
11-task step-by-step plan: create platform pool, move monitoring
(stateless), migrate databases (PVC migration with backups), scale
services pool to 1 node. Includes verification and rollback procedures.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 09:54:28 +01:00
jgrusewski
4da3a16bfe docs: 3-pool node split design (gitlab/platform/foxhunt)
Isolate platform infra (postgres, redis, minio, monitoring) from foxhunt
app services on separate DEV1-L nodes. Same EUR 61/mo total cost, better
blast radius containment and burst headroom for trading services.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 09:49:46 +01:00
jgrusewski
3799981321 docs: add infrastructure security hardening implementation plan
7-task plan: NetworkPolicy (default-deny + per-service), SecurityContext
(pod + container hardening), secret scoping (split foxhunt-secrets),
deploy-secrets.sh (Scaleway integration), Trivy CI scanning.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 23:16:44 +01:00
jgrusewski
a5e0558e08 docs: add infrastructure security hardening design
Minimal hardening plan for production deployment: NetworkPolicy
(default-deny + explicit allow), SecurityContext on all pods,
secret scoping (split monolithic foxhunt-secrets), and Trivy
image scanning in CI pipeline.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 23:13:13 +01:00
jgrusewski
5296b44fe7 docs: workspace cleanup design and implementation plan
Delete 92 scattered .md files, 137 plan docs, create/update all READMEs,
add lib.rs doc comments, clean stale worktrees.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 22:32:54 +01:00
jgrusewski
7524406e3b docs: training observability implementation plan (reduced scope)
DCGM exporter + dashboards already exist. Actual work: 3 tasks —
per-fold metrics in train_baseline_rl, OTLP tracing in all 6 binaries,
OTEL env var in K8s job template.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 22:31:54 +01:00
jgrusewski
638d7c733f docs: training observability + Grafana dashboards design
GPU metrics polling, per-epoch RL metrics, OTLP tracing in training
binaries, and three Grafana dashboards (training cockpit, traces,
infra cockpit trace row).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 22:25:11 +01:00
jgrusewski
8910591f85 docs: I/O pipeline wiring implementation plan — 7 tasks
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 21:42:38 +01:00
jgrusewski
93ad873e40 docs: I/O pipeline wiring design — prefetcher, buffer pool, double-buffer
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 21:39:50 +01:00
jgrusewski
69a9808107 docs: GPU full sweep implementation plan — 17 tasks covering P0-P3
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 21:25:36 +01:00
jgrusewski
7cb8c305df docs: GPU optimization full sweep design — training binary, PPO parity, NCCL multi-GPU
Audit found training binary bypasses all DQNTrainer/PpoTrainer infrastructure
and explicitly disables Rainbow DQN. Design covers 6 sections:
- P0: Wire train_baseline_rl.rs to Trainers
- P1: PPO mixed precision + gradient accumulation + Liquid/TLOB detach fixes
- P2: Mamba2 HardwareBudget + CUDA pipeline wiring + NCCL multi-GPU
- P3: Ensemble parallel inference + BF16 benchmarks

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 21:25:36 +01:00
jgrusewski
6157ea0f77 docs: add deferred stubs implementation plan
6-task plan: proto change, client compilation, retrain wiring,
ensemble confidence RPC, portfolio positions DB query, verification.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 21:21:53 +01:00
jgrusewski
e006518ab0 docs: add deferred stubs implementation design
Design for 3 remaining deferred stubs from 2026-02-24 audit:
- Retrain Model: fine-tune via gRPC with TrainingMode proto field
- Portfolio Positions: PositionProvider trait with broker + DB fallback
- ML Confidence: ensemble RPC with liquidity heuristic fallback

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 21:17:30 +01:00
jgrusewski
eaa2cf4e5a docs: add codebase consolidation implementation plan
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 19:25:01 +01:00
jgrusewski
cbd36fb7eb docs: add codebase consolidation & DRY cleanup design
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 19:22:22 +01:00
jgrusewski
169821b3da docs: GPU max performance Phase 2 design (L4 → H100)
8 optimizations: dynamic batch size, tensor core alignment, no-grad
inference, CUDA stream double-buffering, epoch prefetch, INT8
quantized inference, memory pinning, CUDA kernel optimization.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 18:37:03 +01:00
jgrusewski
b91fa43b32 Merge branch 'worktree-gpu-max-performance'
# Conflicts:
#	crates/ml/examples/hyperopt_baseline_rl.rs
2026-03-01 18:00:10 +01:00
jgrusewski
2b2ff4ffa5 feat(ml): maximize GPU utilization — BF16 mixed precision, dynamic sizing, sync reduction
Wire BF16/FP16 mixed precision end-to-end for DQN and PPO with auto-detection
from GPU name (Ampere+ → BF16, Volta/Turing → FP16). Add hidden_dim_base to
hyperopt and wire through training/eval binaries. Reduce GPU sync points: make
DQN NaN checks periodic (every 100 steps), replace PPO GAE GPU round-trip with
pure CPU implementation. Cache training data across hyperopt trials for all 10
models via Arc. Batch DQN experience storage (128x fewer lock acquisitions).
Correct VRAM constants and batch bounds for all 9 supervised model adapters.

28 files changed, +1207/-208 lines. 2418 tests pass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 17:54:25 +01:00
jgrusewski
259082279d docs: ML inference production cleanup implementation plan
8-task plan covering:
- Delete ~4,600 lines of dead code (push_metrics, legacy tests/benches)
- Remove MLFeatureExtractor from trading_agent_service
- Delete MLFeatureExtractor + SimpleDQNAdapter from ml_strategy.rs (~2,100 lines)
- Refactor SharedMLStrategy to new_with_models() constructor
- Update all callers (backtesting, integration tests)
- Create EnsembleModelAdapter + build_production_strategy() factory
- Harden metrics HTTP server (timeout, size limit, charset)
- Full verification

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 17:23:07 +01:00
jgrusewski
c9de023b32 docs: ML inference production cleanup design
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 17:16:08 +01:00
jgrusewski
415480929d docs: training metrics design — common::metrics ownership
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 15:23:41 +01:00
jgrusewski
d417807132 docs: add real metrics overhaul implementation plan
13-task plan across 4 layers: fix dashboard PromQL, add metrics
servers to data-acquisition and web-gateway, create gRPC metrics
Tower layer, deploy Pushgateway for training jobs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 02:00:08 +01:00
jgrusewski
43235686a8 docs: add real metrics overhaul design
Four-layer plan: fix dashboard queries to use real metric names,
add metrics servers to data-acquisition and web-gateway, add shared
gRPC metrics tower layer, deploy Pushgateway for training job metrics.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 01:51:24 +01:00