foxhunt/docs/ML_TRAINING_SERVICE_API.md

MLTrainingService API Documentation

Overview

The MLTrainingService is a core component of the Foxhunt HFT system that provides comprehensive machine learning model training capabilities with enterprise-grade safety controls, real-time monitoring, and production-ready workflow management.

Table of Contents

1. Service Architecture
2. gRPC Service Definition
3. Training Workflow
4. API Reference
5. Data Pipeline
6. Lifecycle Management
7. Monitoring & Observability
8. Integration Examples
9. Error Handling
10. Production Deployment

Service Architecture

The MLTrainingService follows a microservice architecture pattern with the following components:

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   TLI Client    │    │  MLTraining     │    │   Training      │
│   (gRPC)        │◄──►│   Service       │◄──►│   Pipeline      │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                              │                        │
                              ▼                        ▼
                       ┌─────────────────┐    ┌─────────────────┐
                       │  Model Registry │    │  Safety Manager │
                       │   (DashMap)     │    │  (Gradient/NaN) │
                       └─────────────────┘    └─────────────────┘
                              │                        │
                              ▼                        ▼
                       ┌─────────────────┐    ┌─────────────────┐
                       │  Persistence    │    │  Resource Mgmt  │
                       │  (PostgreSQL)   │    │  (GPU/CPU)      │
                       └─────────────────┘    └─────────────────┘

Key Features

• Real-time Training Monitoring: Streaming progress updates with sub-second latency
• Enterprise Safety Controls: Mathematical safety guarantees, gradient clipping, NaN detection
• Multi-Model Support: DQN, PPO, MAMBA, TFT, Liquid Neural Networks, Transformers
• Resource Management: Dynamic GPU/CPU allocation with utilization monitoring
• Financial Type Safety: Unified decimal types preventing precision loss
• Automatic Deployment: Optional auto-deployment upon successful training completion

gRPC Service Definition

The MLTrainingService is defined in /tli/proto/ml.proto and provides the following service interface:

service MLTrainingService {
  // Training job management
  rpc StartTraining(StartTrainingRequest) returns (TrainingJob);
  rpc StopTraining(StopTrainingRequest) returns (TrainingJob);
  rpc ListTrainingJobs(ListTrainingJobsRequest) returns (ListTrainingJobsResponse);

  // Real-time training monitoring (streaming)
  rpc WatchTrainingProgress(WatchTrainingRequest) returns (stream TrainingProgressUpdate);

  // Training configuration and validation
  rpc ValidateTrainingConfig(TrainingConfigRequest) returns (TrainingConfigResponse);
  rpc GetTrainingTemplates(TrainingTemplatesRequest) returns (TrainingTemplatesResponse);

  // Resource management
  rpc GetResourceUtilization(ResourceRequest) returns (ResourceResponse);
  rpc StreamResourceMetrics(ResourceRequest) returns (stream ResourceMetricsUpdate);
}

Training Workflow

1. Training Job Lifecycle

graph TD
    A[Submit Training Request] --> B[Validate Configuration]
    B --> C{Configuration Valid?}
    C -->|No| D[Return Validation Errors]
    C -->|Yes| E[Allocate Resources]
    E --> F[Load Dataset]
    F --> G[Initialize Model]
    G --> H[Start Training Loop]
    H --> I[Monitor Progress]
    I --> J{Training Complete?}
    J -->|No| K[Update Metrics]
    K --> H
    J -->|Yes| L[Validate Model]
    L --> M{Auto Deploy?}
    M -->|Yes| N[Deploy Model]
    M -->|No| O[Store Model]
    N --> P[Cleanup Resources]
    O --> P
    P --> Q[Training Complete]

2. Training States

State	Description	Next Possible States
QUEUED	Training job submitted and waiting for resources	PREPARING, CANCELLED
PREPARING	Allocating resources and loading data	RUNNING, FAILED
RUNNING	Active training in progress	COMPLETED, FAILED, STOPPING
STOPPING	Graceful shutdown in progress	CANCELLED, FAILED
COMPLETED	Training finished successfully	Terminal state
FAILED	Training failed with error	Terminal state
CANCELLED	Training was cancelled by user	Terminal state

API Reference

StartTraining

Initiates a new training job with comprehensive validation and resource allocation.

Request:

message StartTrainingRequest {
  string model_name = 1;                          // "DQN_EURUSD_v2"
  string dataset_id = 2;                          // "market_data_2024_q1"
  TrainingHyperparameters hyperparameters = 3;    // Training configuration
  ResourceRequirements resource_requirements = 4;  // GPU/CPU requirements
  repeated string tags = 5;                       // ["production", "eurusd"]
  string description = 6;                         // Human description
  bool auto_deploy = 7;                           // Auto-deploy on success
}

Response:

message TrainingJob {
  string job_id = 1;                    // "train_550e8400-e29b-41d4-a716-446655440000"
  string model_name = 2;                // Echo from request
  TrainingStatus status = 3;            // Current status
  int64 start_time = 4;                 // Unix timestamp nanoseconds
  // ... additional fields
}

Example Usage:

use tli::ml_training_service_client::MlTrainingServiceClient;

let mut client = MlTrainingServiceClient::connect("http://localhost:50051").await?;

let request = tonic::Request::new(StartTrainingRequest {
    model_name: "DQN_EURUSD_Production".to_string(),
    dataset_id: "market_data_2024_q3".to_string(),
    hyperparameters: Some(TrainingHyperparameters {
        learning_rate: 0.001,
        batch_size: 128,
        epochs: 1000,
        dropout_rate: Some(0.1),
        ..Default::default()
    }),
    resource_requirements: Some(ResourceRequirements {
        gpu_count: 1,
        cpu_cores: 8,
        memory_gb: 16,
        gpu_type: Some("A100".to_string()),
        disk_gb: 100,
    }),
    tags: vec!["production".to_string(), "eurusd".to_string()],
    description: "Production DQN training for EURUSD pair".to_string(),
    auto_deploy: true,
});

let response = client.start_training(request).await?;
let job = response.into_inner();
println!("Training job started: {}", job.job_id);

WatchTrainingProgress

Streams real-time training progress updates with metrics, logs, and resource utilization.

Request:

message WatchTrainingRequest {
  string job_id = 1;           // Training job to monitor
  bool include_logs = 2;       // Include log messages
  bool include_metrics = 3;    // Include training metrics
}

Response Stream:

message TrainingProgressUpdate {
  string job_id = 1;
  TrainingStatus status = 2;
  int32 current_epoch = 3;
  int32 total_epochs = 4;
  double progress_percentage = 5;         // 0.0 to 100.0
  TrainingMetrics metrics = 6;            // Loss, accuracy, etc.
  optional string log_message = 7;        // Log output
  int64 timestamp = 8;
  optional ResourceUtilization resource_usage = 9;
}

Example Usage:

let request = tonic::Request::new(WatchTrainingRequest {
    job_id: job.job_id.clone(),
    include_logs: true,
    include_metrics: true,
});

let mut stream = client.watch_training_progress(request).await?.into_inner();

while let Some(update) = stream.next().await {
    let update = update?;
    println!("Epoch {}/{}: {:.2}% complete",
        update.current_epoch,
        update.total_epochs,
        update.progress_percentage
    );
    
    if let Some(metrics) = update.metrics {
        println!("Loss: {:.4}, Accuracy: {:.2}%",
            metrics.loss,
            metrics.accuracy * 100.0
        );
    }
    
    if let Some(log) = update.log_message {
        println!("Log: {}", log);
    }
}

ListTrainingJobs

Retrieves training jobs with filtering and pagination support.

Request:

message ListTrainingJobsRequest {
  optional string model_name = 1;        // Filter by model
  optional TrainingStatus status = 2;     // Filter by status
  optional int64 start_time_after = 3;   // Filter by start time
  optional int64 start_time_before = 4;
  repeated string tags = 5;              // Filter by tags
  int32 limit = 6;                       // Max results (default: 50)
  string cursor = 7;                     // Pagination cursor
}

Response:

message ListTrainingJobsResponse {
  repeated TrainingJob jobs = 1;
  string next_cursor = 2;               // For pagination
  int32 total_count = 3;                // Total matching jobs
}

ValidateTrainingConfig

Validates training configuration before job submission with suggestions for optimization.

Request:

message TrainingConfigRequest {
  string model_name = 1;
  TrainingHyperparameters hyperparameters = 2;
  ResourceRequirements resource_requirements = 3;
}

Response:

message TrainingConfigResponse {
  bool valid = 1;
  repeated string validation_errors = 2;
  repeated string validation_warnings = 3;
  optional TrainingHyperparameters suggested_params = 4;
  optional ResourceRequirements suggested_resources = 5;
  double estimated_duration_hours = 6;
}

Data Pipeline

Feature Engineering Pipeline

The MLTrainingService integrates with a sophisticated feature engineering pipeline:

pub struct FinancialFeatures {
    /// Price features (normalized, safe decimal representation)
    pub prices: Vec<IntegerPrice>,
    /// Volume features (safe integers to prevent overflow)
    pub volumes: Vec<i64>,
    /// Technical indicators (bounded and validated)
    pub technical_indicators: HashMap<String, f64>,
    /// Market microstructure features
    pub microstructure: MicrostructureFeatures,
    /// Risk metrics (VaR, Expected Shortfall, etc.)
    pub risk_metrics: RiskFeatures,
    /// Timestamp for temporal alignment
    pub timestamp: chrono::DateTime<chrono::Utc>,
}

Data Validation

All training data undergoes comprehensive validation:

1. Financial Type Safety: All prices use unified IntegerPrice type preventing floating-point precision loss
2. Range Validation: Technical indicators bounded to expected ranges
3. Temporal Consistency: Timestamps validated for proper chronological order
4. Missing Data Handling: Configurable strategies for missing value imputation
5. Outlier Detection: Statistical outlier detection with configurable thresholds

Supported Data Sources

• Real-time Market Data: Direct integration with Polygon.io and broker feeds
• Historical Data: PostgreSQL and InfluxDB time-series data
• Alternative Data: Economic indicators, sentiment data, news feeds
• Custom Datasets: User-provided datasets with validation

Lifecycle Management

Model Versioning

The service implements comprehensive model versioning:

pub struct ModelVersion {
    pub version_id: String,           // "v1.2.3"
    pub model_id: String,             // "DQN_EURUSD"
    pub training_job_id: String,      // Reference to training job
    pub created_at: DateTime<Utc>,
    pub performance_metrics: PerformanceMetrics,
    pub hyperparameters: TrainingHyperparameters,
    pub deployment_status: DeploymentStatus,
}

Deployment Pipeline

graph LR
    A[Training Complete] --> B[Model Validation]
    B --> C{Auto Deploy?}
    C -->|Yes| D[Staging Deployment]
    C -->|No| E[Model Stored]
    D --> F[Integration Tests]
    F --> G{Tests Pass?}
    G -->|Yes| H[Production Deployment]
    G -->|No| I[Rollback to Previous]
    H --> J[Health Monitoring]

Model Registry Integration

Models are automatically registered in the global registry upon successful training:

let registry = get_global_registry();
let trained_model = Arc::new(TLOBModelWrapper::new(tlob_model));
registry.register(trained_model).await?;

Monitoring & Observability

Training Metrics

Real-time metrics tracked during training:

• Loss Functions: Training and validation loss with convergence analysis
• Accuracy Metrics: Precision, recall, F1-score, AUC-ROC
• Financial Metrics: Sharpe ratio, Calmar ratio, maximum drawdown
• Performance Metrics: Training speed, GPU utilization, memory usage
• Safety Metrics: Gradient norms, NaN detection, numerical stability

Resource Monitoring

pub struct ResourceUtilization {
    pub gpu_utilization: f64,     // 0.0 to 1.0
    pub gpu_memory_used: f64,     // 0.0 to 1.0
    pub cpu_utilization: f64,
    pub memory_used: f64,
    pub disk_used: f64,
    pub timestamp: i64,
}

Alerting

Automated alerts for:

• Training failures or divergence
• Resource exhaustion
• Safety violations (NaN, gradient explosion)
• Performance degradation
• Hardware failures

Integration Examples

Basic Training Job

use foxhunt_tli::ml_training::{MLTrainingServiceClient, StartTrainingRequest};

async fn train_dqn_model() -> Result<(), Box<dyn std::error::Error>> {
    let mut client = MLTrainingServiceClient::connect("http://localhost:50051").await?;
    
    let training_request = StartTrainingRequest {
        model_name: "DQN_EURUSD_v3".to_string(),
        dataset_id: "market_data_q3_2024".to_string(),
        hyperparameters: Some(TrainingHyperparameters {
            learning_rate: 0.0001,
            batch_size: 64,
            epochs: 2000,
            dropout_rate: Some(0.15),
            hidden_layers: Some(3),
            hidden_units: Some(256),
            custom_params: hashmap! {
                "epsilon_decay".to_string() => "0.995".to_string(),
                "target_update_frequency".to_string() => "100".to_string(),
            },
        }),
        resource_requirements: Some(ResourceRequirements {
            gpu_count: 1,
            cpu_cores: 8,
            memory_gb: 32,
            gpu_type: Some("A100".to_string()),
            disk_gb: 200,
        }),
        tags: vec!["production".to_string(), "dqn".to_string(), "eurusd".to_string()],
        description: "Production DQN training for EURUSD with enhanced safety controls".to_string(),
        auto_deploy: true,
    };
    
    let response = client.start_training(tonic::Request::new(training_request)).await?;
    let job = response.into_inner();
    
    println!("Training job started: {} (ID: {})", job.model_name, job.job_id);
    
    // Monitor training progress
    let watch_request = WatchTrainingRequest {
        job_id: job.job_id.clone(),
        include_logs: true,
        include_metrics: true,
    };
    
    let mut stream = client.watch_training_progress(
        tonic::Request::new(watch_request)
    ).await?.into_inner();
    
    while let Some(update) = stream.next().await {
        let update = update?;
        
        match update.status() {
            TrainingStatus::Running => {
                if let Some(metrics) = &update.metrics {
                    println!("Epoch {}/{}: Loss={:.4}, Acc={:.2}%, GPU={:.1}%",
                        update.current_epoch,
                        update.total_epochs,
                        metrics.loss,
                        metrics.accuracy * 100.0,
                        update.resource_usage.as_ref().map(|r| r.gpu_utilization * 100.0).unwrap_or(0.0)
                    );
                }
            }
            TrainingStatus::Completed => {
                println!("Training completed successfully!");
                if let Some(model_id) = &job.resulting_model_id {
                    println!("Model deployed with ID: {}", model_id);
                }
                break;
            }
            TrainingStatus::Failed => {
                println!("Training failed: {}", update.log_message.unwrap_or_default());
                break;
            }
            _ => {}
        }
    }
    
    Ok(())
}

Ensemble Training

async fn train_ensemble_model() -> Result<(), Box<dyn std::error::Error>> {
    let mut client = MLTrainingServiceClient::connect("http://localhost:50051").await?;
    
    // Train individual models for ensemble
    let base_models = vec!["DQN", "PPO", "MAMBA", "TFT"];
    let mut training_jobs = Vec::new();
    
    for model_type in base_models {
        let request = StartTrainingRequest {
            model_name: format!("{}_EURUSD_ensemble_base", model_type),
            dataset_id: "market_data_ensemble_2024".to_string(),
            hyperparameters: Some(get_model_hyperparameters(model_type)),
            resource_requirements: Some(get_resource_requirements(model_type)),
            tags: vec!["ensemble".to_string(), "base_model".to_string()],
            description: format!("Base {} model for ensemble training", model_type),
            auto_deploy: false, // Don't auto-deploy base models
        };
        
        let response = client.start_training(tonic::Request::new(request)).await?;
        training_jobs.push(response.into_inner());
    }
    
    // Wait for all base models to complete
    let completed_models = wait_for_training_completion(&mut client, training_jobs).await?;
    
    // Train ensemble meta-model
    let ensemble_request = StartTrainingRequest {
        model_name: "ENSEMBLE_EURUSD_v1".to_string(),
        dataset_id: "market_data_ensemble_2024".to_string(),
        hyperparameters: Some(TrainingHyperparameters {
            learning_rate: 0.01,
            batch_size: 32,
            epochs: 500,
            custom_params: hashmap! {
                "base_models".to_string() => completed_models.join(","),
                "ensemble_method".to_string() => "stacking".to_string(),
            },
        }),
        resource_requirements: Some(ResourceRequirements {
            gpu_count: 1,
            cpu_cores: 16,
            memory_gb: 64,
            disk_gb: 500,
        }),
        tags: vec!["ensemble".to_string(), "production".to_string()],
        description: "Ensemble meta-model combining DQN, PPO, MAMBA, and TFT".to_string(),
        auto_deploy: true,
    };
    
    let ensemble_job = client.start_training(tonic::Request::new(ensemble_request)).await?;
    println!("Ensemble training started: {}", ensemble_job.into_inner().job_id);
    
    Ok(())
}

Error Handling

Error Categories

The MLTrainingService defines comprehensive error categories:

pub enum ProductionTrainingError {
    ConfigError { reason: String },          // Configuration validation errors
    ArchitectureError { reason: String },    // Model architecture issues
    DataError { reason: String },            // Data loading/validation errors
    OptimizationError { reason: String },    // Training optimization failures
    FinancialError { reason: String },       // Financial type validation errors
    SafetyViolation { reason: String },      // Safety control violations
    ConvergenceError { reason: String },     // Model convergence failures
    ResourceError { reason: String },        // Hardware resource errors
    GpuRequired { reason: String },          // GPU acceleration required
}

Error Recovery

The service implements sophisticated error recovery mechanisms:

1. Automatic Retries: Transient failures trigger automatic retries with exponential backoff
2. Checkpoint Recovery: Training resumes from last valid checkpoint on recoverable errors
3. Resource Reallocation: Automatic reallocation of resources on hardware failures
4. Graceful Degradation: CPU fallback when GPU resources unavailable
5. Data Validation: Comprehensive data validation with automatic cleaning

Safety Controls

Mathematical safety is enforced through multiple layers:

pub struct GradientSafetyConfig {
    pub max_gradient_norm: f64,              // Gradient clipping threshold
    pub nan_detection_enabled: bool,         // NaN detection
    pub inf_detection_enabled: bool,         // Infinity detection
    pub numerical_stability_threshold: f64,  // Numerical stability threshold
    pub gradient_explosion_threshold: f64,   // Gradient explosion detection
}

Production Deployment

Performance Requirements

• Training Latency: < 10ms per forward pass for real-time training
• Memory Efficiency: < 2GB memory usage per model during training
• GPU Utilization: > 80% GPU utilization during active training
• Fault Tolerance: Automatic recovery within 30 seconds of failures
• Scalability: Support for 100+ concurrent training jobs

Security & Compliance

• Authentication: mTLS certificate-based authentication
• Authorization: Role-based access control (RBAC)
• Audit Logging: Comprehensive audit trail for all training activities
• Data Privacy: PII anonymization and secure data handling
• Regulatory Compliance: SOX, MiFID II compliance for financial models

Infrastructure Requirements

Minimum Requirements:

• 2x NVIDIA A100 GPUs (40GB VRAM each)
• 64 cores CPU (Intel Xeon or AMD EPYC)
• 256GB RAM
• 2TB NVMe SSD storage
• 10GbE network connectivity

Recommended Production Setup:

• 8x NVIDIA H100 GPUs (80GB VRAM each)
• 128 cores CPU
• 1TB RAM
• 10TB NVMe SSD storage
• InfiniBand network (200Gb/s)
• Redundant power supplies

Monitoring & Alerting

Production deployment includes comprehensive monitoring:

# Prometheus metrics configuration
training_metrics:
  - name: ml_training_jobs_total
    help: Total number of training jobs
    labels: [model_type, status]
  
  - name: ml_training_duration_seconds
    help: Training duration histogram
    buckets: [60, 300, 900, 3600, 14400]
  
  - name: ml_gpu_utilization_percent
    help: GPU utilization percentage
    labels: [gpu_id, job_id]
  
  - name: ml_memory_usage_bytes
    help: Memory usage during training
    labels: [job_id, memory_type]

High Availability

The service supports high availability deployment:

• Active-Passive Failover: Automatic failover to standby instances
• Load Balancing: Intelligent load balancing across GPU resources
• Data Replication: Real-time data replication for disaster recovery
• Health Checks: Comprehensive health monitoring with auto-restart
• Rolling Updates: Zero-downtime updates and deployments

---

Conclusion

The MLTrainingService provides a production-ready, enterprise-grade machine learning training platform specifically designed for high-frequency trading applications. With comprehensive safety controls, real-time monitoring, and seamless integration with the Foxhunt ecosystem, it enables reliable and scalable ML model development and deployment.

For additional information, see:

• TLI Operations Manual
• System Architecture
• Performance Tuning
• Security Documentation