foxhunt/services/ml_training_service

ML Training Service

Production-ready ML training service for the Foxhunt HFT trading system. This service orchestrates model training jobs, manages GPU/CPU resources, and provides comprehensive progress tracking for financial ML models.

Features

🚀 Core Capabilities

• Model Training Orchestration: Manages training jobs for TLOB, MAMBA-2, DQN, PPO, Liquid, and TFT models
• Resource Management: Intelligent GPU/CPU allocation with concurrent job limiting
• Real-time Progress Tracking: Live streaming of training metrics and status updates
• Model Lifecycle Management: From training initiation to artifact storage and retrieval
• Financial Safety Guarantees: Built-in validation for financial data and model outputs

🏗️ Architecture

• gRPC API: High-performance streaming API with type-safe protobuf definitions
• PostgreSQL Persistence: Reliable job metadata and training history storage
• Flexible Storage: Local filesystem or S3-compatible object storage for model artifacts
• Production Safety: Comprehensive error handling, gradient safety, and NaN detection
• Monitoring Integration: Prometheus metrics and structured logging

📊 Supported Models

Model	Description	Estimated Training Time	GPU Required
TLOB	Time-Limit Order Book Transformer	45 min	✅
MAMBA-2	State Space Model for long sequences	90 min	✅
DQN	Deep Q-Network for RL trading	120 min	✅
PPO	Proximal Policy Optimization	75 min	✅
Liquid	Liquid Neural Network for regime detection	60 min	❌
TFT	Temporal Fusion Transformer	100 min	✅

Quick Start

Prerequisites

• Rust 1.75+
• PostgreSQL 12+
• CUDA 12.0+ (for GPU acceleration)
• Optional: S3-compatible storage

Installation

# Clone the repository
git clone https://github.com/user/foxhunt
cd foxhunt

# Build the service (production - uses real data)
cargo build --release -p ml_training_service

# Or build for testing with mock data
cargo build --release -p ml_training_service --features mock-data

# Set up configuration
cp config/ml_training_service.example.toml config/ml_training_service.toml
# Edit configuration as needed

# Run database migrations
./target/release/ml_training_service database migrate

# Start the service
./target/release/ml_training_service serve

Feature Flags

The service supports the following Cargo feature flags:

Feature	Default	Description
minimal	✅ Yes	Minimal ML feature set for financial models
gpu	❌ No	Enable SIMD GPU acceleration (requires CUDA)
debug	❌ No	Enable debug mode with additional logging
mock-data	❌ No	TESTING ONLY - Use mock training data instead of database

Important: The mock-data feature is for testing and development only. Production builds should use the default features which load real historical data from PostgreSQL.

# Production build (default)
cargo build --release -p ml_training_service

# Testing with mock data (bypasses database)
cargo build --release -p ml_training_service --features mock-data

# GPU-accelerated build
cargo build --release -p ml_training_service --features gpu

Configuration

[server]
host = "0.0.0.0"
port = 50053
max_concurrent_jobs = 4

[database]
url = "postgresql://user:pass@localhost:5432/foxhunt_training"
max_connections = 10

[training]
default_device = "cuda"
max_gpu_memory_gb = 8.0
worker_threads = 4

[storage]
storage_type = "local"  # or "s3"
local_base_path = "./models"

[monitoring]
enable_prometheus = true
prometheus_port = 9090

API Usage

Starting a Training Job

import grpc
from ml_training_pb2 import *
from ml_training_pb2_grpc import MLTrainingServiceStub

# Connect to service
channel = grpc.insecure_channel('localhost:50053')
client = MLTrainingServiceStub(channel)

# Configure TLOB training
request = StartTrainingRequest(
    model_type="TLOB",
    hyperparameters=Hyperparameters(
        tlob_params=TlobParams(
            epochs=100,
            learning_rate=0.001,
            batch_size=64,
            hidden_dim=256,
            num_heads=8
        )
    ),
    use_gpu=True,
    description="TLOB training for EURUSD orderbook prediction"
)

# Submit job
response = client.StartTraining(request)
job_id = response.job_id
print(f"Training job started: {job_id}")

Monitoring Training Progress

# Subscribe to real-time updates
status_request = SubscribeToTrainingStatusRequest(job_id=job_id)
status_stream = client.SubscribeToTrainingStatus(status_request)

for update in status_stream:
    print(f"Epoch {update.current_epoch}/{update.total_epochs}")
    print(f"Progress: {update.progress_percentage:.1f}%")
    print(f"Loss: {update.metrics.get('loss', 0.0):.6f}")
    print(f"Sharpe Ratio: {update.financial_metrics.sharpe_ratio:.3f}")
    
    if update.status == TrainingStatus.COMPLETED:
        print("Training completed successfully!")
        break

Listing Training Jobs

# List recent jobs
jobs_request = ListTrainingJobsRequest(
    page=1,
    page_size=10,
    status_filter=TrainingStatus.COMPLETED
)

jobs_response = client.ListTrainingJobs(jobs_request)
for job in jobs_response.jobs:
    print(f"{job.job_id}: {job.model_type} - {job.status}")
    print(f"  Final Loss: {job.final_loss:.6f}")
    print(f"  Duration: {job.completed_at - job.started_at}")

CLI Usage

Server Management

# Start the service
ml_training_service serve --config config.toml --port 50053

# Enable development mode with debug logging
ml_training_service serve --dev

# Health check
ml_training_service health --endpoint http://localhost:50053

Database Operations

# Run migrations
ml_training_service database migrate

# Check database health
ml_training_service database health

# Clean up old jobs (retain 30 days)
ml_training_service database cleanup --retain-days 30

Configuration Management

# Validate configuration
ml_training_service config --file config.toml

Integration with Existing ML Infrastructure

The service integrates seamlessly with the existing Foxhunt ML infrastructure:

Training Pipeline Integration

use ml::training_pipeline::{ProductionMLTrainingSystem, ProductionTrainingConfig};
use ml::safety::{MLSafetyManager, GradientSafetyManager};

// The service orchestrates the existing training system
let training_system = ProductionMLTrainingSystem::new(config).await?;
let result = training_system.train_model(training_data, validation_data).await?;

Financial Feature Processing

use ml::training_pipeline::{FinancialFeatures, MicrostructureFeatures, RiskFeatures};

// Financial features are validated and processed automatically
let features = FinancialFeatures {
    prices: vec![IntegerPrice::from_f64(100.50)],
    volumes: vec![1000],
    technical_indicators: indicators,
    microstructure: MicrostructureFeatures { /* ... */ },
    risk_metrics: RiskFeatures { /* ... */ },
    timestamp: Utc::now(),
};

Safety and Validation

// Built-in safety guarantees
- Gradient clipping and NaN detection
- Financial data validation (positive prices, finite indicators)
- Resource allocation limits
- Training timeout protection
- Model artifact integrity checks

Monitoring and Observability

Prometheus Metrics

The service exposes comprehensive metrics on :9090/metrics:

# Training job metrics
ml_training_jobs_total{status="completed"} 45
ml_training_jobs_total{status="running"} 2
ml_training_jobs_total{status="failed"} 1

# Resource utilization
ml_training_gpu_utilization_percent 78.5
ml_training_memory_usage_bytes 4294967296

# Performance metrics
ml_training_job_duration_seconds{model_type="TLOB"} 2700
ml_training_final_loss{model_type="MAMBA_2"} 0.001234

Structured Logging

{
  "timestamp": "2025-01-21T10:30:45Z",
  "level": "INFO",
  "target": "ml_training_service::orchestrator",
  "message": "Training job completed successfully",
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "model_type": "TLOB",
  "final_loss": 0.001234,
  "training_duration_secs": 2700,
  "epochs_completed": 100
}

Performance Characteristics

Throughput

• Concurrent Jobs: Up to 4 simultaneous training jobs (configurable)
• Job Submission: <10ms latency for job creation
• Status Updates: Real-time streaming with <100ms latency
• Database Operations: <5ms for job metadata queries

Resource Usage

• Memory: ~1-2GB base + 4-8GB per training job
• GPU Memory: 4-8GB per GPU-accelerated job
• CPU: 1-2 cores for orchestration + 4-8 cores per training job
• Storage: Variable (10MB-1GB+ per model artifact)

Scalability

• Horizontal: Can run multiple service instances with shared database
• Vertical: Scales with available GPU/CPU resources
• Storage: Unlimited with S3-compatible backends
• Concurrent Clients: 100+ simultaneous gRPC connections

Security and Compliance

Data Protection

• Encryption: TLS 1.3 for gRPC communication
• Authentication: Integration with Foxhunt auth system
• Audit Logging: Complete training job audit trail
• Access Control: Role-based access to training operations

Financial Compliance

• Model Validation: Automatic financial data sanity checks
• Reproducibility: Complete training configuration persistence
• Model Governance: Artifact integrity and versioning
• Risk Controls: Automated position sizing validation

Development

Building from Source

# Development build
cargo build -p ml_training_service

# Release build
cargo build --release -p ml_training_service

# Run tests
cargo test -p ml_training_service

# Run with debug logging
RUST_LOG=debug cargo run -p ml_training_service -- serve --dev

Testing

# Unit tests
cargo test -p ml_training_service

# Integration tests (requires database)
cargo test -p ml_training_service --features integration-tests

# End-to-end tests
cargo test -p ml_training_service --test e2e

gRPC Development

# Generate protobuf code
cargo build -p ml_training_service

# Test with grpcurl
grpcurl -plaintext localhost:50053 list
grpcurl -plaintext localhost:50053 ml_training.MLTrainingService/HealthCheck

Troubleshooting

Common Issues

Service Won't Start

# Check configuration
ml_training_service config --file config.toml

# Check database connectivity
ml_training_service database health

# Check port availability
lsof -i :50053

Training Jobs Fail

# Check GPU availability
nvidia-smi

# Check logs for detailed error messages
tail -f /var/log/ml_training_service.log

# Verify model artifacts storage
ls -la ./models/

Performance Issues

# Check resource utilization
htop

# Monitor GPU usage
watch -n 1 nvidia-smi

# Check database performance
EXPLAIN ANALYZE SELECT * FROM training_jobs WHERE status = 'running';

Debugging

# Enable debug logging
export RUST_LOG=ml_training_service=debug

# Enable trace logging for specific modules
export RUST_LOG=ml_training_service::orchestrator=trace

# Profile memory usage
valgrind --tool=massif target/release/ml_training_service serve

Contributing

Code Style

• Follow Rust standard formatting (cargo fmt)
• Add documentation for public APIs
• Include comprehensive error handling
• Write tests for new functionality

Pull Request Process

1. Create feature branch from main
2. Implement changes with tests
3. Update documentation
4. Submit PR with clear description

Performance Testing

# Benchmark training job throughput
cargo run --release --bin bench_training_service

# Load test gRPC API
ghz --insecure --proto proto/ml_training.proto --call ml_training.MLTrainingService/HealthCheck localhost:50053

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.

Support

• Documentation: docs.rs/foxhunt
• Issues: GitHub Issues
• Discussions: GitHub Discussions