foxhunt/services/ml_training_service/README.md

# ML Training Service

Production-ready ML training service for the Foxhunt HFT trading system. This service orchestrates model training jobs, manages GPU/CPU resources, and provides comprehensive progress tracking for financial ML models.

## Features

### 🚀 Core Capabilities
- **Model Training Orchestration**: Manages training jobs for TLOB, MAMBA-2, DQN, PPO, Liquid, and TFT models
- **Resource Management**: Intelligent GPU/CPU allocation with concurrent job limiting
- **Real-time Progress Tracking**: Live streaming of training metrics and status updates
- **Model Lifecycle Management**: From training initiation to artifact storage and retrieval
- **Financial Safety Guarantees**: Built-in validation for financial data and model outputs

### 🏗️ Architecture
- **gRPC API**: High-performance streaming API with type-safe protobuf definitions
- **PostgreSQL Persistence**: Reliable job metadata and training history storage
- **Flexible Storage**: Local filesystem or S3-compatible object storage for model artifacts
- **Production Safety**: Comprehensive error handling, gradient safety, and NaN detection
- **Monitoring Integration**: Prometheus metrics and structured logging

### 📊 Supported Models

| Model | Description | Estimated Training Time | GPU Required |
|-------|-------------|------------------------|--------------|
| **TLOB** | Time-Limit Order Book Transformer | 45 min | ✅ |
| **MAMBA-2** | State Space Model for long sequences | 90 min | ✅ |
| **DQN** | Deep Q-Network for RL trading | 120 min | ✅ |
| **PPO** | Proximal Policy Optimization | 75 min | ✅ |
| **Liquid** | Liquid Neural Network for regime detection | 60 min | ❌ |
| **TFT** | Temporal Fusion Transformer | 100 min | ✅ |

## Quick Start

### Prerequisites
- Rust 1.75+
- PostgreSQL 12+
- CUDA 12.0+ (for GPU acceleration)
- Optional: S3-compatible storage

### Installation

```bash
# Clone the repository
git clone https://github.com/user/foxhunt
cd foxhunt

# Build the service (production - uses real data)
cargo build --release -p ml_training_service

# Or build for testing with mock data
cargo build --release -p ml_training_service --features mock-data

# Set up configuration
cp config/ml_training_service.example.toml config/ml_training_service.toml
# Edit configuration as needed

# Run database migrations
./target/release/ml_training_service database migrate

# Start the service
./target/release/ml_training_service serve
```

### Feature Flags

The service supports the following Cargo feature flags:

| Feature | Default | Description |
|---------|---------|-------------|
| `minimal` | ✅ Yes | Minimal ML feature set for financial models |
| `gpu` | ❌ No | Enable SIMD GPU acceleration (requires CUDA) |
| `debug` | ❌ No | Enable debug mode with additional logging |
| `mock-data` | ❌ No | **TESTING ONLY** - Use mock training data instead of database |

**Important**: The `mock-data` feature is for testing and development only. Production builds should use the default features which load real historical data from PostgreSQL.

```bash
# Production build (default)
cargo build --release -p ml_training_service

# Testing with mock data (bypasses database)
cargo build --release -p ml_training_service --features mock-data

# GPU-accelerated build
cargo build --release -p ml_training_service --features gpu
```

### Configuration

```toml
[server]
host = "0.0.0.0"
port = 50053
max_concurrent_jobs = 4

[database]
url = "postgresql://user:pass@localhost:5432/foxhunt_training"
max_connections = 10

[training]
default_device = "cuda"
max_gpu_memory_gb = 8.0
worker_threads = 4

[storage]
storage_type = "local"  # or "s3"
local_base_path = "./models"

[monitoring]
enable_prometheus = true
prometheus_port = 9090
```

## API Usage

### Starting a Training Job

```python
import grpc
from ml_training_pb2 import *
from ml_training_pb2_grpc import MLTrainingServiceStub

# Connect to service
channel = grpc.insecure_channel('localhost:50053')
client = MLTrainingServiceStub(channel)

# Configure TLOB training
request = StartTrainingRequest(
    model_type="TLOB",
    hyperparameters=Hyperparameters(
        tlob_params=TlobParams(
            epochs=100,
            learning_rate=0.001,
            batch_size=64,
            hidden_dim=256,
            num_heads=8
        )
    ),
    use_gpu=True,
    description="TLOB training for EURUSD orderbook prediction"
)

# Submit job
response = client.StartTraining(request)
job_id = response.job_id
print(f"Training job started: {job_id}")
```

### Monitoring Training Progress

```python
# Subscribe to real-time updates
status_request = SubscribeToTrainingStatusRequest(job_id=job_id)
status_stream = client.SubscribeToTrainingStatus(status_request)

for update in status_stream:
    print(f"Epoch {update.current_epoch}/{update.total_epochs}")
    print(f"Progress: {update.progress_percentage:.1f}%")
    print(f"Loss: {update.metrics.get('loss', 0.0):.6f}")
    print(f"Sharpe Ratio: {update.financial_metrics.sharpe_ratio:.3f}")
    
    if update.status == TrainingStatus.COMPLETED:
        print("Training completed successfully!")
        break
```

### Listing Training Jobs

```python
# List recent jobs
jobs_request = ListTrainingJobsRequest(
    page=1,
    page_size=10,
    status_filter=TrainingStatus.COMPLETED
)

jobs_response = client.ListTrainingJobs(jobs_request)
for job in jobs_response.jobs:
    print(f"{job.job_id}: {job.model_type} - {job.status}")
    print(f"  Final Loss: {job.final_loss:.6f}")
    print(f"  Duration: {job.completed_at - job.started_at}")
```

## CLI Usage

### Server Management

```bash
# Start the service
ml_training_service serve --config config.toml --port 50053

# Enable development mode with debug logging
ml_training_service serve --dev

# Health check
ml_training_service health --endpoint http://localhost:50053
```

### Database Operations

```bash
# Run migrations
ml_training_service database migrate

# Check database health
ml_training_service database health

# Clean up old jobs (retain 30 days)
ml_training_service database cleanup --retain-days 30
```

### Configuration Management

```bash
# Validate configuration
ml_training_service config --file config.toml
```

## Integration with Existing ML Infrastructure

The service integrates seamlessly with the existing Foxhunt ML infrastructure:

### Training Pipeline Integration

```rust
use ml::training_pipeline::{ProductionMLTrainingSystem, ProductionTrainingConfig};
use ml::safety::{MLSafetyManager, GradientSafetyManager};

// The service orchestrates the existing training system
let training_system = ProductionMLTrainingSystem::new(config).await?;
let result = training_system.train_model(training_data, validation_data).await?;
```

### Financial Feature Processing

```rust
use ml::training_pipeline::{FinancialFeatures, MicrostructureFeatures, RiskFeatures};

// Financial features are validated and processed automatically
let features = FinancialFeatures {
    prices: vec![IntegerPrice::from_f64(100.50)],
    volumes: vec![1000],
    technical_indicators: indicators,
    microstructure: MicrostructureFeatures { /* ... */ },
    risk_metrics: RiskFeatures { /* ... */ },
    timestamp: Utc::now(),
};
```

### Safety and Validation

```rust
// Built-in safety guarantees
- Gradient clipping and NaN detection
- Financial data validation (positive prices, finite indicators)
- Resource allocation limits
- Training timeout protection
- Model artifact integrity checks
```

## Monitoring and Observability

### Prometheus Metrics

The service exposes comprehensive metrics on `:9090/metrics`:

```
# Training job metrics
ml_training_jobs_total{status="completed"} 45
ml_training_jobs_total{status="running"} 2
ml_training_jobs_total{status="failed"} 1

# Resource utilization
ml_training_gpu_utilization_percent 78.5
ml_training_memory_usage_bytes 4294967296

# Performance metrics
ml_training_job_duration_seconds{model_type="TLOB"} 2700
ml_training_final_loss{model_type="MAMBA_2"} 0.001234
```

### Structured Logging

```json
{
  "timestamp": "2025-01-21T10:30:45Z",
  "level": "INFO",
  "target": "ml_training_service::orchestrator",
  "message": "Training job completed successfully",
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "model_type": "TLOB",
  "final_loss": 0.001234,
  "training_duration_secs": 2700,
  "epochs_completed": 100
}
```

## Performance Characteristics

### Throughput
- **Concurrent Jobs**: Up to 4 simultaneous training jobs (configurable)
- **Job Submission**: <10ms latency for job creation
- **Status Updates**: Real-time streaming with <100ms latency
- **Database Operations**: <5ms for job metadata queries

### Resource Usage
- **Memory**: ~1-2GB base + 4-8GB per training job
- **GPU Memory**: 4-8GB per GPU-accelerated job
- **CPU**: 1-2 cores for orchestration + 4-8 cores per training job
- **Storage**: Variable (10MB-1GB+ per model artifact)

### Scalability
- **Horizontal**: Can run multiple service instances with shared database
- **Vertical**: Scales with available GPU/CPU resources
- **Storage**: Unlimited with S3-compatible backends
- **Concurrent Clients**: 100+ simultaneous gRPC connections

## Security and Compliance

### Data Protection
- **Encryption**: TLS 1.3 for gRPC communication
- **Authentication**: Integration with Foxhunt auth system
- **Audit Logging**: Complete training job audit trail
- **Access Control**: Role-based access to training operations

### Financial Compliance
- **Model Validation**: Automatic financial data sanity checks
- **Reproducibility**: Complete training configuration persistence
- **Model Governance**: Artifact integrity and versioning
- **Risk Controls**: Automated position sizing validation

## Development

### Building from Source

```bash
# Development build
cargo build -p ml_training_service

# Release build
cargo build --release -p ml_training_service

# Run tests
cargo test -p ml_training_service

# Run with debug logging
RUST_LOG=debug cargo run -p ml_training_service -- serve --dev
```

### Testing

```bash
# Unit tests
cargo test -p ml_training_service

# Integration tests (requires database)
cargo test -p ml_training_service --features integration-tests

# End-to-end tests
cargo test -p ml_training_service --test e2e
```

### gRPC Development

```bash
# Generate protobuf code
cargo build -p ml_training_service

# Test with grpcurl
grpcurl -plaintext localhost:50053 list
grpcurl -plaintext localhost:50053 ml_training.MLTrainingService/HealthCheck
```

## Troubleshooting

### Common Issues

#### Service Won't Start
```bash
# Check configuration
ml_training_service config --file config.toml

# Check database connectivity
ml_training_service database health

# Check port availability
lsof -i :50053
```

#### Training Jobs Fail
```bash
# Check GPU availability
nvidia-smi

# Check logs for detailed error messages
tail -f /var/log/ml_training_service.log

# Verify model artifacts storage
ls -la ./models/
```

#### Performance Issues
```bash
# Check resource utilization
htop

# Monitor GPU usage
watch -n 1 nvidia-smi

# Check database performance
EXPLAIN ANALYZE SELECT * FROM training_jobs WHERE status = 'running';
```

### Debugging

```bash
# Enable debug logging
export RUST_LOG=ml_training_service=debug

# Enable trace logging for specific modules
export RUST_LOG=ml_training_service::orchestrator=trace

# Profile memory usage
valgrind --tool=massif target/release/ml_training_service serve
```

## Contributing

### Code Style
- Follow Rust standard formatting (`cargo fmt`)
- Add documentation for public APIs
- Include comprehensive error handling
- Write tests for new functionality

### Pull Request Process
1. Create feature branch from `main`
2. Implement changes with tests
3. Update documentation
4. Submit PR with clear description

### Performance Testing
```bash
# Benchmark training job throughput
cargo run --release --bin bench_training_service

# Load test gRPC API
ghz --insecure --proto proto/ml_training.proto --call ml_training.MLTrainingService/HealthCheck localhost:50053
```

## License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.

## Support

- **Documentation**: [docs.rs/foxhunt](https://docs.rs/foxhunt)
- **Issues**: [GitHub Issues](https://github.com/user/foxhunt/issues)
- **Discussions**: [GitHub Discussions](https://github.com/user/foxhunt/discussions)