foxhunt/tests/chaos/README.md

# Foxhunt Chaos Engineering Framework

A comprehensive chaos engineering framework specifically designed for high-frequency trading (HFT) systems with sub-100ms recovery requirements.

## 🎯 Overview

This framework provides systematic failure injection and recovery validation for the Foxhunt HFT trading system, focusing on:

- **MLTrainingService Resilience**: Kill/restart scenarios with checkpoint recovery
- **Performance Validation**: Sub-100ms recovery time requirements
- **Failure Injection**: Network, memory, GPU, disk, and database failures
- **Automated Testing**: Nightly chaos job scheduling with reporting
- **HFT-Specific Requirements**: Ultra-low latency validation and monitoring

## 🏗️ Architecture

```
tests/chaos/
├── chaos_framework.rs          # Core chaos orchestration engine
├── ml_training_chaos.rs        # ML-specific chaos tests
├── nightly_chaos_runner.rs     # Automated scheduling and execution
├── chaos_cli.rs                # Command-line interface
├── examples/
│   ├── usage_examples.rs       # Comprehensive usage examples
│   └── chaos_config.toml       # Configuration template
└── README.md                   # This file
```

## 🚀 Quick Start

### 1. Basic ML Service Kill Test

```rust
use foxhunt_tests::chaos::*;

#[tokio::main]
async fn main() -> Result<()> {
    // Initialize chaos framework
    let runner = initialize_foxhunt_chaos().await?;
    
    // Run quick ML service resilience test
    let results = run_quick_chaos_test().await?;
    
    println!("Chaos test results: {:?}", results);
    Ok(())
}
```

### 2. Command Line Usage

```bash
# Run a single process kill experiment
cargo run --bin foxhunt-chaos run --experiment-type process-kill --service ml_training_service

# Run comprehensive ML test suite
cargo run --bin foxhunt-chaos ml-suite --endpoint http://localhost:8080 --generate-report

# Start nightly scheduler
cargo run --bin foxhunt-chaos schedule --time 02:00 --exclude-weekends --webhook https://hooks.slack.com/...

# Validate system readiness
cargo run --bin foxhunt-chaos validate --check-ml-service --check-database
```

### 3. Configuration File

```toml
# chaos_config.toml
[general]
enabled = true
schedule_time = "02:00"
max_duration_hours = 3

[ml_chaos_config]
ml_service_endpoint = "http://localhost:8080"
max_recovery_time_ms = 100  # HFT requirement
model_types = ["tlob", "dqn", "mamba2"]
```

## 🧪 Supported Failure Types

### Process Failures
- **SIGTERM/SIGKILL**: Graceful and forceful process termination
- **Service Restart**: Automatic restart with configurable delays

### Resource Exhaustion
- **Memory Pressure**: Configurable memory consumption (2GB-8GB)
- **GPU Exhaustion**: GPU memory filling (80%-95% capacity)
- **CPU Throttling**: CPU limit enforcement (25%-75%)

### Infrastructure Failures
- **Network Partitions**: Port-specific network isolation
- **Disk I/O Failures**: File system failure injection
- **Database Disconnections**: Connection pool exhaustion

## 📊 ML Model Support

The framework supports chaos testing across all Foxhunt ML models:

| Model | Type | Recovery Target | Checkpoint Interval |
|-------|------|----------------|-------------------|
| **TLOB** | Transformer | 25ms | 30s |
| **MAMBA-2** | State Space | 40ms | 60s |
| **DQN** | Deep Q-Learning | 80ms | 120s |
| **PPO** | Policy Optimization | 60ms | 90s |
| **Liquid** | Neural Network | 35ms | 45s |
| **TFT** | Temporal Fusion | 95ms | 180s |

## 🕒 Nightly Automation

### Scheduling Features
- **Configurable Time**: Any time zone and schedule
- **Weekend Exclusion**: Skip weekends for production safety
- **Retry Logic**: Automatic retry on failure with exponential backoff
- **Notification Integration**: Slack/Teams webhooks for alerts

### Alert Thresholds
- **Critical**: SLA violations (>100ms recovery)
- **Warning**: Checkpoint failures or performance regressions
- **Info**: Successful completion notifications

## 📈 Performance Requirements

### HFT Latency Targets
- **Order Processing**: <50μs end-to-end
- **Market Data**: <30μs ingestion latency
- **Risk Calculation**: <25μs computation
- **Recovery Time**: <100ms system restoration

### Validation Metrics
- **P50/P95/P99 Latency**: Histogram tracking
- **Recovery Time Distribution**: Statistical analysis
- **Checkpoint Integrity**: Binary validation
- **Performance Regression**: Pre/post comparison

## 🔧 Integration

### Test Infrastructure Integration

```rust
// In your tests/lib.rs
pub mod chaos;

#[tokio::test]
async fn test_ml_service_resilience() {
    use crate::chaos::*;
    
    let results = run_quick_chaos_test().await.unwrap();
    assert!(!results.is_empty());
}
```

### CI/CD Integration

```yaml
# .github/workflows/chaos.yml
name: Chaos Engineering
on:
  schedule:
    - cron: '0 2 * * *'  # Run at 2 AM daily

jobs:
  chaos-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run Chaos Tests
        run: cargo test --test chaos_integration
```

## 📋 Example Scenarios

### 1. ML Training Kill/Restart
```rust
let experiment = ChaosExperiment {
    name: "TLOB Training Kill Test".to_string(),
    target_service: "ml_training_service".to_string(),
    failure_type: FailureType::ProcessKill {
        signal: Signal::SIGTERM,
        delay_before_restart_ms: 2000,
    },
    max_recovery_time_ms: 25, // TLOB target
    // ...
};
```

### 2. GPU Memory Exhaustion
```rust
let experiment = ChaosExperiment {
    name: "GPU Memory Pressure Test".to_string(),
    failure_type: FailureType::GpuResourceExhaustion {
        memory_fill_percent: 95,
        duration_ms: 20000,
    },
    max_recovery_time_ms: 150, // Relaxed for GPU
    // ...
};
```

### 3. Network Partition
```rust
let experiment = ChaosExperiment {
    name: "Database Partition Test".to_string(),
    failure_type: FailureType::NetworkPartition {
        target_ports: vec![5432, 6379], // PostgreSQL, Redis
        duration_ms: 15000,
    },
    // ...
};
```

## 🛡️ Safety Features

### Production Safeguards
- **Weekend Exclusion**: Automatic weekend skipping
- **Duration Limits**: Maximum 3-hour chaos windows
- **Recovery Timeouts**: Automatic experiment termination
- **Checkpoint Validation**: Pre/post integrity checks

### Monitoring Integration
- **Real-time Alerting**: Immediate notification of failures
- **Performance Tracking**: Latency histogram recording
- **Report Generation**: Automated markdown reports
- **Event Streaming**: Live experiment status updates

## 📊 Reporting

### Automated Reports
```markdown
# ML Training Chaos Engineering Report

**Generated:** 2025-01-21 02:30:00 UTC
**Total Tests:** 18

## Summary
- ✅ **Successful:** 16 (88.9%)
- ❌ **Failed:** 2 (11.1%)
- 📊 **Success Rate:** 88.9%

## Results by Model Type
- **tlob:** 6/6 (100.0%)
- **dqn:** 5/6 (83.3%)
- **mamba2:** 5/6 (83.3%)

## Performance Analysis
- ✅ **No Performance Regressions Detected**
- **Average Recovery Time:** 45.2ms
- **Max Recovery Time:** 78ms
```

## 🔍 Troubleshooting

### Common Issues

1. **Service Not Found**
   ```bash
   # Check service status
   systemctl status ml_training_service
   ```

2. **GPU Not Available**
   ```bash
   # Verify GPU access
   nvidia-smi
   ```

3. **Permission Errors**
   ```bash
   # Check chaos framework permissions
   sudo usermod -a -G docker $USER
   ```

### Debug Mode
```bash
# Enable verbose logging
cargo run --bin foxhunt-chaos --verbose run --experiment-type process-kill
```

## 🤝 Contributing

1. **Add New Failure Types**: Extend `FailureType` enum
2. **ML Model Support**: Add new models to `ModelType`
3. **Monitoring Integration**: Extend metrics collection
4. **Custom Experiments**: Create domain-specific tests

## 📝 License

This chaos engineering framework is part of the Foxhunt HFT trading system and follows the same licensing terms.

---

**⚠️ Important**: This framework is designed specifically for HFT systems with sub-100ms recovery requirements. Always test in non-production environments first and ensure proper safeguards are in place.