TDD plan covering GpuReplayBuffer, proportional + rank-based GPU
sampling, priority scatter updates, async loss readback, trainer
integration, OOM fallback, and distribution correctness tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Approved design for moving Prioritized Experience Replay entirely to
GPU — flat priority array with parallel prefix-sum sampling, GPU-resident
ring buffer for experiences, async loss readback. Eliminates the last
major CPU bottleneck in the DQN training loop.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New standalone monitoring_service that bridges Prometheus → gRPC,
enabling `fxt train monitor` and web-gateway WS streaming for
all training jobs (CI and service-managed).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two-layer IBKR connectivity validation: direct ibapi handshake +
broker-gateway gRPC health check. Pure-client architecture using
ibapi as optional dep on fxt (not trading_engine).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
11-task step-by-step plan: create platform pool, move monitoring
(stateless), migrate databases (PVC migration with backups), scale
services pool to 1 node. Includes verification and rollback procedures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Isolate platform infra (postgres, redis, minio, monitoring) from foxhunt
app services on separate DEV1-L nodes. Same EUR 61/mo total cost, better
blast radius containment and burst headroom for trading services.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Minimal hardening plan for production deployment: NetworkPolicy
(default-deny + explicit allow), SecurityContext on all pods,
secret scoping (split monolithic foxhunt-secrets), and Trivy
image scanning in CI pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Delete 92 scattered .md files, 137 plan docs, create/update all READMEs,
add lib.rs doc comments, clean stale worktrees.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DCGM exporter + dashboards already exist. Actual work: 3 tasks —
per-fold metrics in train_baseline_rl, OTLP tracing in all 6 binaries,
OTEL env var in K8s job template.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GPU metrics polling, per-epoch RL metrics, OTLP tracing in training
binaries, and three Grafana dashboards (training cockpit, traces,
infra cockpit trace row).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Design for 3 remaining deferred stubs from 2026-02-24 audit:
- Retrain Model: fine-tune via gRPC with TrainingMode proto field
- Portfolio Positions: PositionProvider trait with broker + DB fallback
- ML Confidence: ensemble RPC with liquidity heuristic fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire BF16/FP16 mixed precision end-to-end for DQN and PPO with auto-detection
from GPU name (Ampere+ → BF16, Volta/Turing → FP16). Add hidden_dim_base to
hyperopt and wire through training/eval binaries. Reduce GPU sync points: make
DQN NaN checks periodic (every 100 steps), replace PPO GAE GPU round-trip with
pure CPU implementation. Cache training data across hyperopt trials for all 10
models via Arc. Batch DQN experience storage (128x fewer lock acquisitions).
Correct VRAM constants and batch bounds for all 9 supervised model adapters.
28 files changed, +1207/-208 lines. 2418 tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
13-task plan across 4 layers: fix dashboard PromQL, add metrics
servers to data-acquisition and web-gateway, create gRPC metrics
Tower layer, deploy Pushgateway for training jobs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Four-layer plan: fix dashboard queries to use real metric names,
add metrics servers to data-acquisition and web-gateway, add shared
gRPC metrics tower layer, deploy Pushgateway for training job metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Parallel hyperopt trials, VRAM-aware batch bounds, 5-layer OOM protection.
Scales from L4 through H100.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Detailed step-by-step plan covering: S3 bucket creation, CI builder
updates, 3 base image Dockerfiles, CI pipeline rewiring, 8 cache PVCs,
all service deployment YAML updates, training job template, and cleanup.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace per-service Docker images with generic base images + S3-based binary
distribution. CI compiles and uploads stripped binaries to S3. Pods fetch
binaries via initContainer with PVC cache fallback for trading resilience.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Single monolithic kernel with in-kernel GAE, full portfolio simulation,
shared device functions header, 5-layer critic matching trainer exactly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Full design for monolithic CUDA kernel that runs the entire DQN
experience collection pipeline on GPU: Dueling Q-network inference
(online + target), epsilon-greedy action selection, portfolio
simulation, barrier tracking, diversity entropy, curiosity inference,
and reward combination. 128 parallel episodes × 500 timesteps =
64,000 experiences per kernel launch with zero CPU roundtrips.
Key decisions:
- Approach A (monolithic kernel) for speed + accuracy
- Parallel episode architecture (N threads × L timesteps)
- Q-network + curiosity inference in raw CUDA (matrix-vector multiply)
- Curiosity training stays on Candle (periodic batch updates)
- TD error pre-computed for priority replay
- Weight sync after gradient updates (~1.2MB upload)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
12-worktree phased migration: foundation types → core crates →
ML boundary → services → peripheral → cleanup. Full code for
Phase 1 (Price/Quantity/Money/Ratio in common/financial_types),
migration patterns for Phases 2-6.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Consolidate competing Price/Quantity implementations into canonical
i64 fixed-point types in common/. Adds Money and Ratio newtypes.
Phased migration plan across 5 phases using parallel worktrees.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace all `.to_string().parse::<f32/f64>()` patterns with
`num_traits::ToPrimitive` methods (`.to_f32()`, `.to_f64()`).
Each string roundtrip heap-allocated per conversion — fatal in
DQN hot loop (300K+ bars × epochs). Decimal stays as canonical
financial type; conversions happen at GPU/float boundaries only.
Also fixes blocking_read() in async context (risk_integration.rs).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Route DQN/PPO to cheaper L4 (CPU-bound), supervised models to L40S
(GPU-bound). Separate CI templates with tuned resource requests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
5-phase agent loop (diagnose → plan → implement → verify → deliver)
with GitLab issues as durable communication/handoff layer. OpenHands
on Kapsule K8s runtime, LiteLLM proxy routing to Claude Opus/Sonnet
and Scaleway devstral. Rust dispatcher service. Auto-merge for CI
failure fixes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
