foxhunt/tests/chaos/README.md

Foxhunt Chaos Engineering Framework

A comprehensive chaos engineering framework specifically designed for high-frequency trading (HFT) systems with sub-100ms recovery requirements.

🎯 Overview

This framework provides systematic failure injection and recovery validation for the Foxhunt HFT trading system, focusing on:

• MLTrainingService Resilience: Kill/restart scenarios with checkpoint recovery
• Performance Validation: Sub-100ms recovery time requirements
• Failure Injection: Network, memory, GPU, disk, and database failures
• Automated Testing: Nightly chaos job scheduling with reporting
• HFT-Specific Requirements: Ultra-low latency validation and monitoring

🏗️ Architecture

tests/chaos/
├── chaos_framework.rs          # Core chaos orchestration engine
├── ml_training_chaos.rs        # ML-specific chaos tests
├── nightly_chaos_runner.rs     # Automated scheduling and execution
├── chaos_cli.rs                # Command-line interface
├── examples/
│   ├── usage_examples.rs       # Comprehensive usage examples
│   └── chaos_config.toml       # Configuration template
└── README.md                   # This file

🚀 Quick Start

1. Basic ML Service Kill Test

use foxhunt_tests::chaos::*;

#[tokio::main]
async fn main() -> Result<()> {
    // Initialize chaos framework
    let runner = initialize_foxhunt_chaos().await?;
    
    // Run quick ML service resilience test
    let results = run_quick_chaos_test().await?;
    
    println!("Chaos test results: {:?}", results);
    Ok(())
}

2. Command Line Usage

# Run a single process kill experiment
cargo run --bin foxhunt-chaos run --experiment-type process-kill --service ml_training_service

# Run comprehensive ML test suite
cargo run --bin foxhunt-chaos ml-suite --endpoint http://localhost:8080 --generate-report

# Start nightly scheduler
cargo run --bin foxhunt-chaos schedule --time 02:00 --exclude-weekends --webhook https://hooks.slack.com/...

# Validate system readiness
cargo run --bin foxhunt-chaos validate --check-ml-service --check-database

3. Configuration File

# chaos_config.toml
[general]
enabled = true
schedule_time = "02:00"
max_duration_hours = 3

[ml_chaos_config]
ml_service_endpoint = "http://localhost:8080"
max_recovery_time_ms = 100  # HFT requirement
model_types = ["tlob", "dqn", "mamba2"]

🧪 Supported Failure Types

Process Failures

• SIGTERM/SIGKILL: Graceful and forceful process termination
• Service Restart: Automatic restart with configurable delays

Resource Exhaustion

• Memory Pressure: Configurable memory consumption (2GB-8GB)
• GPU Exhaustion: GPU memory filling (80%-95% capacity)
• CPU Throttling: CPU limit enforcement (25%-75%)

Infrastructure Failures

• Network Partitions: Port-specific network isolation
• Disk I/O Failures: File system failure injection
• Database Disconnections: Connection pool exhaustion

📊 ML Model Support

The framework supports chaos testing across all Foxhunt ML models:

Model	Type	Recovery Target	Checkpoint Interval
TLOB	Transformer	25ms	30s
MAMBA-2	State Space	40ms	60s
DQN	Deep Q-Learning	80ms	120s
PPO	Policy Optimization	60ms	90s
Liquid	Neural Network	35ms	45s
TFT	Temporal Fusion	95ms	180s

🕒 Nightly Automation

Scheduling Features

• Configurable Time: Any time zone and schedule
• Weekend Exclusion: Skip weekends for production safety
• Retry Logic: Automatic retry on failure with exponential backoff
• Notification Integration: Slack/Teams webhooks for alerts

Alert Thresholds

• Critical: SLA violations (>100ms recovery)
• Warning: Checkpoint failures or performance regressions
• Info: Successful completion notifications

📈 Performance Requirements

HFT Latency Targets

• Order Processing: <50μs end-to-end
• Market Data: <30μs ingestion latency
• Risk Calculation: <25μs computation
• Recovery Time: <100ms system restoration

Validation Metrics

• P50/P95/P99 Latency: Histogram tracking
• Recovery Time Distribution: Statistical analysis
• Checkpoint Integrity: Binary validation
• Performance Regression: Pre/post comparison

🔧 Integration

Test Infrastructure Integration

// In your tests/lib.rs
pub mod chaos;

#[tokio::test]
async fn test_ml_service_resilience() {
    use crate::chaos::*;
    
    let results = run_quick_chaos_test().await.unwrap();
    assert!(!results.is_empty());
}

CI/CD Integration

# .github/workflows/chaos.yml
name: Chaos Engineering
on:
  schedule:
    - cron: '0 2 * * *'  # Run at 2 AM daily

jobs:
  chaos-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run Chaos Tests
        run: cargo test --test chaos_integration

📋 Example Scenarios

1. ML Training Kill/Restart

let experiment = ChaosExperiment {
    name: "TLOB Training Kill Test".to_string(),
    target_service: "ml_training_service".to_string(),
    failure_type: FailureType::ProcessKill {
        signal: Signal::SIGTERM,
        delay_before_restart_ms: 2000,
    },
    max_recovery_time_ms: 25, // TLOB target
    // ...
};

2. GPU Memory Exhaustion

let experiment = ChaosExperiment {
    name: "GPU Memory Pressure Test".to_string(),
    failure_type: FailureType::GpuResourceExhaustion {
        memory_fill_percent: 95,
        duration_ms: 20000,
    },
    max_recovery_time_ms: 150, // Relaxed for GPU
    // ...
};

3. Network Partition

let experiment = ChaosExperiment {
    name: "Database Partition Test".to_string(),
    failure_type: FailureType::NetworkPartition {
        target_ports: vec![5432, 6379], // PostgreSQL, Redis
        duration_ms: 15000,
    },
    // ...
};

🛡️ Safety Features

Production Safeguards

• Weekend Exclusion: Automatic weekend skipping
• Duration Limits: Maximum 3-hour chaos windows
• Recovery Timeouts: Automatic experiment termination
• Checkpoint Validation: Pre/post integrity checks

Monitoring Integration

• Real-time Alerting: Immediate notification of failures
• Performance Tracking: Latency histogram recording
• Report Generation: Automated markdown reports
• Event Streaming: Live experiment status updates

📊 Reporting

Automated Reports

# ML Training Chaos Engineering Report

**Generated:** 2025-01-21 02:30:00 UTC
**Total Tests:** 18

## Summary
- ✅ **Successful:** 16 (88.9%)
- ❌ **Failed:** 2 (11.1%)
- 📊 **Success Rate:** 88.9%

## Results by Model Type
- **tlob:** 6/6 (100.0%)
- **dqn:** 5/6 (83.3%)
- **mamba2:** 5/6 (83.3%)

## Performance Analysis
- ✅ **No Performance Regressions Detected**
- **Average Recovery Time:** 45.2ms
- **Max Recovery Time:** 78ms

🔍 Troubleshooting

Common Issues

1. Service Not Found

   # Check service status
   systemctl status ml_training_service
   

2. GPU Not Available

   # Verify GPU access
   nvidia-smi
   

3. Permission Errors

   # Check chaos framework permissions
   sudo usermod -a -G docker $USER
   

Debug Mode

# Enable verbose logging
cargo run --bin foxhunt-chaos --verbose run --experiment-type process-kill

🤝 Contributing

1. Add New Failure Types: Extend FailureType enum
2. ML Model Support: Add new models to ModelType
3. Monitoring Integration: Extend metrics collection
4. Custom Experiments: Create domain-specific tests

📝 License

This chaos engineering framework is part of the Foxhunt HFT trading system and follows the same licensing terms.

---

⚠️ Important: This framework is designed specifically for HFT systems with sub-100ms recovery requirements. Always test in non-production environments first and ensure proper safeguards are in place.