Clean rewrite into full-scale operations platform.
Single proto/ root, monitoring_service removed,
CLI + MCP + cockpit TUI with purple/cyan theme.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dynamic dtype detection (Ampere+ → BF16, else F32), zero casts in
training hot path, cast only at data ingestion and loss scalar.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ratatui streaming dashboard (fxt watch) with 4 tabs, stub command
wiring plan, and new ApproveModel/RejectModel proto RPCs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TDD plan covering GpuReplayBuffer, proportional + rank-based GPU
sampling, priority scatter updates, async loss readback, trainer
integration, OOM fallback, and distribution correctness tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Approved design for moving Prioritized Experience Replay entirely to
GPU — flat priority array with parallel prefix-sum sampling, GPU-resident
ring buffer for experiences, async loss readback. Eliminates the last
major CPU bottleneck in the DQN training loop.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New standalone monitoring_service that bridges Prometheus → gRPC,
enabling `fxt train monitor` and web-gateway WS streaming for
all training jobs (CI and service-managed).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two-layer IBKR connectivity validation: direct ibapi handshake +
broker-gateway gRPC health check. Pure-client architecture using
ibapi as optional dep on fxt (not trading_engine).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
11-task step-by-step plan: create platform pool, move monitoring
(stateless), migrate databases (PVC migration with backups), scale
services pool to 1 node. Includes verification and rollback procedures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Isolate platform infra (postgres, redis, minio, monitoring) from foxhunt
app services on separate DEV1-L nodes. Same EUR 61/mo total cost, better
blast radius containment and burst headroom for trading services.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Minimal hardening plan for production deployment: NetworkPolicy
(default-deny + explicit allow), SecurityContext on all pods,
secret scoping (split monolithic foxhunt-secrets), and Trivy
image scanning in CI pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Delete 92 scattered .md files, 137 plan docs, create/update all READMEs,
add lib.rs doc comments, clean stale worktrees.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DCGM exporter + dashboards already exist. Actual work: 3 tasks —
per-fold metrics in train_baseline_rl, OTLP tracing in all 6 binaries,
OTEL env var in K8s job template.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GPU metrics polling, per-epoch RL metrics, OTLP tracing in training
binaries, and three Grafana dashboards (training cockpit, traces,
infra cockpit trace row).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Design for 3 remaining deferred stubs from 2026-02-24 audit:
- Retrain Model: fine-tune via gRPC with TrainingMode proto field
- Portfolio Positions: PositionProvider trait with broker + DB fallback
- ML Confidence: ensemble RPC with liquidity heuristic fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire BF16/FP16 mixed precision end-to-end for DQN and PPO with auto-detection
from GPU name (Ampere+ → BF16, Volta/Turing → FP16). Add hidden_dim_base to
hyperopt and wire through training/eval binaries. Reduce GPU sync points: make
DQN NaN checks periodic (every 100 steps), replace PPO GAE GPU round-trip with
pure CPU implementation. Cache training data across hyperopt trials for all 10
models via Arc. Batch DQN experience storage (128x fewer lock acquisitions).
Correct VRAM constants and batch bounds for all 9 supervised model adapters.
28 files changed, +1207/-208 lines. 2418 tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
13-task plan across 4 layers: fix dashboard PromQL, add metrics
servers to data-acquisition and web-gateway, create gRPC metrics
Tower layer, deploy Pushgateway for training jobs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Four-layer plan: fix dashboard queries to use real metric names,
add metrics servers to data-acquisition and web-gateway, add shared
gRPC metrics tower layer, deploy Pushgateway for training job metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Parallel hyperopt trials, VRAM-aware batch bounds, 5-layer OOM protection.
Scales from L4 through H100.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Detailed step-by-step plan covering: S3 bucket creation, CI builder
updates, 3 base image Dockerfiles, CI pipeline rewiring, 8 cache PVCs,
all service deployment YAML updates, training job template, and cleanup.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace per-service Docker images with generic base images + S3-based binary
distribution. CI compiles and uploads stripped binaries to S3. Pods fetch
binaries via initContainer with PVC cache fallback for trading resilience.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Single monolithic kernel with in-kernel GAE, full portfolio simulation,
shared device functions header, 5-layer critic matching trainer exactly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Full design for monolithic CUDA kernel that runs the entire DQN
experience collection pipeline on GPU: Dueling Q-network inference
(online + target), epsilon-greedy action selection, portfolio
simulation, barrier tracking, diversity entropy, curiosity inference,
and reward combination. 128 parallel episodes × 500 timesteps =
64,000 experiences per kernel launch with zero CPU roundtrips.
Key decisions:
- Approach A (monolithic kernel) for speed + accuracy
- Parallel episode architecture (N threads × L timesteps)
- Q-network + curiosity inference in raw CUDA (matrix-vector multiply)
- Curiosity training stays on Candle (periodic batch updates)
- TD error pre-computed for priority replay
- Weight sync after gradient updates (~1.2MB upload)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
