Delete 92 scattered .md files, 137 plan docs, create/update all READMEs,
add lib.rs doc comments, clean stale worktrees.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DCGM exporter + dashboards already exist. Actual work: 3 tasks —
per-fold metrics in train_baseline_rl, OTLP tracing in all 6 binaries,
OTEL env var in K8s job template.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GPU metrics polling, per-epoch RL metrics, OTLP tracing in training
binaries, and three Grafana dashboards (training cockpit, traces,
infra cockpit trace row).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Design for 3 remaining deferred stubs from 2026-02-24 audit:
- Retrain Model: fine-tune via gRPC with TrainingMode proto field
- Portfolio Positions: PositionProvider trait with broker + DB fallback
- ML Confidence: ensemble RPC with liquidity heuristic fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire BF16/FP16 mixed precision end-to-end for DQN and PPO with auto-detection
from GPU name (Ampere+ → BF16, Volta/Turing → FP16). Add hidden_dim_base to
hyperopt and wire through training/eval binaries. Reduce GPU sync points: make
DQN NaN checks periodic (every 100 steps), replace PPO GAE GPU round-trip with
pure CPU implementation. Cache training data across hyperopt trials for all 10
models via Arc. Batch DQN experience storage (128x fewer lock acquisitions).
Correct VRAM constants and batch bounds for all 9 supervised model adapters.
28 files changed, +1207/-208 lines. 2418 tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
13-task plan across 4 layers: fix dashboard PromQL, add metrics
servers to data-acquisition and web-gateway, create gRPC metrics
Tower layer, deploy Pushgateway for training jobs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Four-layer plan: fix dashboard queries to use real metric names,
add metrics servers to data-acquisition and web-gateway, add shared
gRPC metrics tower layer, deploy Pushgateway for training job metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Parallel hyperopt trials, VRAM-aware batch bounds, 5-layer OOM protection.
Scales from L4 through H100.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Detailed step-by-step plan covering: S3 bucket creation, CI builder
updates, 3 base image Dockerfiles, CI pipeline rewiring, 8 cache PVCs,
all service deployment YAML updates, training job template, and cleanup.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace per-service Docker images with generic base images + S3-based binary
distribution. CI compiles and uploads stripped binaries to S3. Pods fetch
binaries via initContainer with PVC cache fallback for trading resilience.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Single monolithic kernel with in-kernel GAE, full portfolio simulation,
shared device functions header, 5-layer critic matching trainer exactly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Full design for monolithic CUDA kernel that runs the entire DQN
experience collection pipeline on GPU: Dueling Q-network inference
(online + target), epsilon-greedy action selection, portfolio
simulation, barrier tracking, diversity entropy, curiosity inference,
and reward combination. 128 parallel episodes × 500 timesteps =
64,000 experiences per kernel launch with zero CPU roundtrips.
Key decisions:
- Approach A (monolithic kernel) for speed + accuracy
- Parallel episode architecture (N threads × L timesteps)
- Q-network + curiosity inference in raw CUDA (matrix-vector multiply)
- Curiosity training stays on Candle (periodic batch updates)
- TD error pre-computed for priority replay
- Weight sync after gradient updates (~1.2MB upload)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
12-worktree phased migration: foundation types → core crates →
ML boundary → services → peripheral → cleanup. Full code for
Phase 1 (Price/Quantity/Money/Ratio in common/financial_types),
migration patterns for Phases 2-6.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Consolidate competing Price/Quantity implementations into canonical
i64 fixed-point types in common/. Adds Money and Ratio newtypes.
Phased migration plan across 5 phases using parallel worktrees.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace all `.to_string().parse::<f32/f64>()` patterns with
`num_traits::ToPrimitive` methods (`.to_f32()`, `.to_f64()`).
Each string roundtrip heap-allocated per conversion — fatal in
DQN hot loop (300K+ bars × epochs). Decimal stays as canonical
financial type; conversions happen at GPU/float boundaries only.
Also fixes blocking_read() in async context (risk_integration.rs).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Route DQN/PPO to cheaper L4 (CPU-bound), supervised models to L40S
(GPU-bound). Separate CI templates with tuned resource requests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
5-phase agent loop (diagnose → plan → implement → verify → deliver)
with GitLab issues as durable communication/handoff layer. OpenHands
on Kapsule K8s runtime, LiteLLM proxy routing to Claude Opus/Sonnet
and Scaleway devstral. Rust dispatcher service. Auto-merge for CI
failure fixes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
10 tasks across 2 workstreams (NAT + Metrics), mostly parallelizable.
Includes Terraform modules, K8s manifests, Helm values, and runbook.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Covers two workstreams:
1. Scaleway Public Gateway for NAT (all pools private-only)
2. Scaleway Cockpit + Alloy + DCGM Exporter for full metrics
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
K8s Job dispatch from ml_training_service with Rust sidecar uploader,
S3 artifact storage, and model promotion with fxt CLI approval gate.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a parent/child GitLab CI pipeline for ML model training:
- Generator script produces per-model hyperopt/train/evaluate jobs
- Parent pipeline (.gitlab-ci-training.yml) with manual trigger
- NFS-backed ReadWriteMany PVC for shared training outputs
- Hyperopt params wired into training binaries (DQN, PPO, TFT, Mamba2)
- Shared DBN loader eliminates duplicate code across hyperopt adapters
- Supervised hyperopt unified to DBN data (was parquet-only)
Pipeline: hyperopt (4 models) → train (10 models) → evaluate ensemble
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The check stage ran cargo check + clippy (~6min) before tests, but
the test stage compiles the same workspace anyway. Skip check with
an echo pass-through to save ~6min per pipeline. Easy to uncomment
when clippy enforcement is needed (e.g., before releases).
Also adds sccache stats to the test job for cache monitoring.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Consolidate 21 training examples into 2 binaries:
- train_baseline_rl (DQN, PPO with RL walk-forward)
- train_baseline_supervised (8 models via UnifiedTrainable)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Covers tfstate migration nl-ams→fr-par, resource import for
drift gaps (sccache bucket, foxhunt-ci registry, DNS records),
and GitLab CI/CD pipeline (plan on MR, apply on merge, weekly
drift detection).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Terraform pool, PVC, Dockerfile, devcontainer.json, pod template,
CI job, setup script, infra apply, GitLab badge. TDD-style steps
with exact file paths and commands.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DevPod on Kapsule with GP1-L (16 vCPU, 64GB) autoscaling to zero,
persistent PVCs for Claude Code + cargo cache, "Open in DevPod"
GitLab badge, CI-built devcontainer image.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
