Plan to reduce ml crate from 91K to ~12K LOC by extracting trainers
into model sub-crates, hyperopt adapters into ml-hyperopt, and
infrastructure into existing sub-crates. ml becomes an orchestration
layer owning inference, model factory, and training pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split ml-rl into ml-dqn and ml-ppo for 3-way parallel compilation
and better sccache hit rate. Extract TradingAction to ml-core to
enable full DQN/PPO independence.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split the monolithic ml crate (260K lines, 55s compile) into 5 crates:
ml-core, ml-rl, ml-supervised, ml-infra, ml (facade).
15-task plan with full module inventory and import migration guide.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace MinIO binary distribution with GitLab Generic Package Registry.
Every code push to main auto-creates a CalVer tag (vYYYY.MM.N),
compiles, uploads binaries to GitLab packages, creates a Release
with auto-generated notes, and deploys via deployment patching.
- New CI templates: create-tag, upload-release
- Modified: compile-services/training upload to GitLab packages
- Modified: deploy-services patches FOXHUNT_RELEASE on deployments
- All 7 service initContainers fetch from GitLab (curl, deploy token)
- Training job-template binary fetch from GitLab (data stays MinIO)
- MinIO retains: sccache, training data, model checkpoints
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. Walk-forward windows: replaced 3 non-overlapping with sliding (50% overlap, ~5 windows).
Aggregation changed from mean-0.5*std to median-0.5*IQR for outlier robustness.
2. Composite score: tanh normalization prevents Calmar ratio scale dominance
(0.02% drawdowns → values in thousands drowning out Sharpe/Sortino).
3. Q-value overestimation: new Prometheus gauge foxhunt_training_q_overestimation_ratio,
warning log when ratio>10 or q_mean>5, adaptive tau doubles when Q-mean growth>0.5/epoch
(capped at 0.01), decays back when stable.
2742 tests pass, 0 failures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Addresses eval/training mismatch (B1-B3), reward architecture (C1-C3),
and early stopping (C4). See design doc for full analysis.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Unify both gateways into a single gRPC-only `services/api/` with tonic-web
for browser access. Drop REST+WebSocket, keep full 6-layer auth.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replaces monolithic compile-services with 8 per-service compile jobs,
each with dependency-aware changes: filters. Deploy job only restarts
services whose binary actually changed. Also renames service crates
from snake_case to kebab-case to match k8s deployment names.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add live training metrics monitor CLI command (streaming & one-shot) using
the monitoring gRPC service. Update DQN tests to match post-fix defaults:
IQN disabled, CQL alpha=0.1, v_min/v_max widened, 26D search space.
- train.rs: `fxt train monitor [--once] [--model X] [--interval N]`
- Rewrite gradient collapse test for BF16 mixed precision awareness
- Update inference test config to match trainer defaults (IQN off, CQL on)
- Update production smoke test for 26D parameter space
- Add dqn_action_collapse_fix_test.rs verifying all 6 root cause fixes
- Add planning docs for monitoring service and epoch financial metrics
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Helm values (controller + server on platform node, MinIO artifact repo)
- WorkflowTemplate: parameterized 5-step DAG (fetch→hyperopt→train→eval→upload)
- Nginx proxy for argo.fxhnt.ai → Argo Server :2746
- DNS A record for argo.fxhnt.ai
- MinIO bucket foxhunt-training-results for Argo artifacts
- Kustomization for kubectl apply -k
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Full GitLab CI → Argo migration: Argo Events webhook trigger,
per-service compile WorkflowTemplates, selective deploy, test
workspace, training compile, and IaC templates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminate foxhunt-binaries and training-binaries PVCs — services and
training jobs now fetch binaries from MinIO via rclone initContainers.
This removes L40S GPU autoscale-for-PVC-writes, removes RWO node
affinity constraints, and requires zero image rebuilds.
Changes:
- .gitlab-ci.yml: replace ~115 lines of binary-writer pod logic with
aws s3 cp to MinIO (~25 lines)
- 8 service YAMLs + 2 GPU overlays: add rclone initContainer + emptyDir
- Training job template: MinIO rclone fetch replaces PVC copy
- Delete foxhunt-binaries-pvc.yaml and training-binaries-pvc.yaml
- Add s3.fxhnt.ai DNS record (Terraform) and nginx proxy block
- Replace pod-writer-deploy skill with MinIO-based deploy skill
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
17-task plan for replacing hardcoded mock data in fxt watch
with real gRPC streaming data from the API Gateway.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Streaming-first architecture: 11 new poll-to-stream gateway adapters,
channel-based DataFetcher in the TUI, auto-reconnect with backoff.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fxt 2.0 overhaul: consolidated proto/, absorbed monitoring into API Gateway,
new 15-command CLI with gRPC, MCP server mode, TUI cockpit framework.
159 files changed, net -11,835 lines.
4-phase plan from deep audit: critical bug fixes (weight_decay, IQN PER,
q_value_std, 54/51 dim mismatch), train/eval parity (TradeExecutor,
checkpoint validation), SOTA improvements (differential Sharpe, PQN
LayerNorm, temporal PER, dormant neuron resets), and search space cleanup.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Integrate all PPO improvement modules into the core training paths:
- Symlog value predictions in compute_value_loss (MLP + LSTM)
- Adaptive entropy auto-tuning replaces fixed entropy_coeff
- Percentile P5/P95 advantage scaling for heavy-tailed returns
- DAPO clip_epsilon_high wired in all 7 PPOConfig construction sites
- Shape mismatch fix in adaptive_entropy (unsqueeze scalar)
2726 tests pass, 0 clippy errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Clean rewrite into full-scale operations platform.
Single proto/ root, monitoring_service removed,
CLI + MCP + cockpit TUI with purple/cyan theme.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
A+B approach: smooth penalties + position limits + CUDA cleanup +
search space reduction (45D→25D) + TPE optimizer replacing PSO.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dynamic dtype detection (Ampere+ → BF16, else F32), zero casts in
training hot path, cast only at data ingestion and loss scalar.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ratatui streaming dashboard (fxt watch) with 4 tabs, stub command
wiring plan, and new ApproveModel/RejectModel proto RPCs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TDD plan covering GpuReplayBuffer, proportional + rank-based GPU
sampling, priority scatter updates, async loss readback, trainer
integration, OOM fallback, and distribution correctness tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Approved design for moving Prioritized Experience Replay entirely to
GPU — flat priority array with parallel prefix-sum sampling, GPU-resident
ring buffer for experiences, async loss readback. Eliminates the last
major CPU bottleneck in the DQN training loop.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New standalone monitoring_service that bridges Prometheus → gRPC,
enabling `fxt train monitor` and web-gateway WS streaming for
all training jobs (CI and service-managed).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two-layer IBKR connectivity validation: direct ibapi handshake +
broker-gateway gRPC health check. Pure-client architecture using
ibapi as optional dep on fxt (not trading_engine).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
11-task step-by-step plan: create platform pool, move monitoring
(stateless), migrate databases (PVC migration with backups), scale
services pool to 1 node. Includes verification and rollback procedures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Isolate platform infra (postgres, redis, minio, monitoring) from foxhunt
app services on separate DEV1-L nodes. Same EUR 61/mo total cost, better
blast radius containment and burst headroom for trading services.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Minimal hardening plan for production deployment: NetworkPolicy
(default-deny + explicit allow), SecurityContext on all pods,
secret scoping (split monolithic foxhunt-secrets), and Trivy
image scanning in CI pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
