0f9d756caa
Extend ml_training_service to dispatch GPU training jobs as K8s batch/v1 Jobs, collect results via a Rust sidecar uploader, and support model promotion with operator approval via fxt CLI. - K8s dispatcher creates Jobs on gpu-training pool with native sidecar - training_uploader crate: watches DONE/FAILED marker, uploads to S3, reports completion via ReportJobCompletion gRPC - PromotionManager compares metrics, queues better models for approval - 4 new proto RPCs: ReportJobCompletion, ListPendingPromotions, ApprovePromotion, RejectPromotion - fxt commands: train start, model list/approve/reject - Training binaries write DONE/FAILED markers + metrics.json - Dockerfile, K8s job template, and CI pipeline updated - StartTraining gracefully falls back to in-process when outside K8s - 27 new tests (16 service + 11 promotion), 141 total service tests pass Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ml_training_service
Model training orchestration and lifecycle management for the Foxhunt HFT trading system. Manages training jobs for DQN, PPO, TFT, Mamba2, TLOB, and Liquid models with progress tracking, resource allocation, and model artifact storage.
Building
# Default (minimal features)
cargo build --release -p ml_training_service
# With GPU acceleration (requires CUDA)
cargo build --release -p ml_training_service --features gpu
# With mock training data (testing only, bypasses database)
cargo build --release -p ml_training_service --features mock-data
Features
| Feature | Default | Description |
|---|---|---|
minimal |
Yes | Minimal ML feature set for financial models |
gpu |
No | SIMD GPU acceleration (requires CUDA) |
debug |
No | Additional debug logging |
mock-data |
No | Use mock training data instead of PostgreSQL |
Configuration
The gRPC listen port is set via the GRPC_PORT environment variable. Prometheus metrics are exposed on port 9094.
PostgreSQL (via sqlx) is used for job metadata, training history, and state management. Set the connection string with DATABASE_URL.
Running
GRPC_PORT=50053 DATABASE_URL="postgresql://user:pass@localhost:5432/foxhunt_training" \
./target/release/ml_training_service serve
Testing
# Unit tests (offline, no database required)
SQLX_OFFLINE=true cargo test -p ml_training_service --lib
# Integration tests (requires running PostgreSQL)
cargo test -p ml_training_service