foxhunt/deployment/README.md

# FOXHUNT HFT TRADING SYSTEM - DEPLOYMENT ARTIFACTS

This directory contains all the deployment artifacts needed to deploy the Foxhunt HFT Trading System to production environments. The system supports multiple deployment modes with comprehensive monitoring and health checking.

## 🎯 Quick Start

For a standard production deployment:

```bash
# 1. Build optimized release binaries
./deployment/build_release.sh --cpu-target native

# 2. Deploy to production (bare-metal, recommended for HFT)
sudo ./deployment/deploy_production.sh --mode bare-metal

# 3. Set up monitoring
./deployment/monitoring_setup.sh --mode docker

# 4. Verify deployment health
./deployment/health_check.sh --mode comprehensive
```

## 📁 Deployment Artifacts

### Core Deployment Scripts

| File | Description | Usage |
|------|-------------|-------|
| `build_release.sh` | Builds optimized release binaries | `./build_release.sh --cpu-target native` |
| `deploy_production.sh` | Complete production deployment | `sudo ./deploy_production.sh --mode bare-metal` |
| `health_check.sh` | Comprehensive health monitoring | `./health_check.sh --mode comprehensive` |

### Docker & Container Support

| File | Description | Usage |
|------|-------------|-------|
| `Dockerfile.production` | Multi-stage production Docker builds | Used by `build_docker_images.sh` |
| `build_docker_images.sh` | Docker image build automation | `./build_docker_images.sh --service all` |

### System Integration

| File | Description | Usage |
|------|-------------|-------|
| `create_systemd_services.sh` | SystemD service file generator | `./create_systemd_services.sh` |
| `monitoring_setup.sh` | Monitoring stack deployment | `./monitoring_setup.sh --mode docker` |

### Documentation & Checklists

| File | Description |
|------|-------------|
| `DEPLOYMENT_CHECKLIST.md` | Comprehensive deployment checklist |
| `README.md` | This deployment guide |

## 🚀 Deployment Modes

### 1. Bare-Metal Deployment (Recommended for HFT)

**Best Performance**: Sub-50μs latency, direct hardware access

```bash
# Build optimized binaries
./deployment/build_release.sh --cpu-target native

# Deploy as SystemD services
sudo ./deployment/deploy_production.sh --mode bare-metal --env production
```

**Features:**
- CPU affinity optimization
- Hardware timing (RDTSC)
- Memory lock capabilities
- Priority scheduling
- Direct network access

### 2. Docker Deployment

**Development/Testing**: Easy deployment and scaling

```bash
# Build Docker images
./deployment/build_docker_images.sh --service all --enable-gpu

# Deploy with Docker Compose
./deployment/deploy_production.sh --mode docker --env production
```

**Features:**
- Container isolation
- Resource limits
- Easy scaling
- Consistent environments

### 3. Kubernetes Deployment

**Cloud/Enterprise**: Orchestrated container deployment

```bash
# Build and push images
./deployment/build_docker_images.sh --service all --push --registry your-registry.com

# Deploy to Kubernetes
./deployment/deploy_production.sh --mode k8s --env production
```

**Features:**
- Auto-scaling
- Service discovery
- Rolling updates
- High availability

## 🏗️ Architecture Overview

The Foxhunt HFT system consists of four main services:

```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Trading       │    │  ML Training    │    │  Backtesting    │
│   Service       │◄──►│   Service       │◄──►│   Service       │
│   (Port 8080)   │    │  (Port 8082)    │    │  (Port 8083)    │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         ▲                       ▲                       ▲
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│      TLI        │    │   PostgreSQL    │    │     Redis       │
│  (Port 8081)    │    │   (Port 5432)   │    │  (Port 6379)    │
└─────────────────┘    └─────────────────┘    └─────────────────┘
```

### Service Details

| Service | Binary | HTTP Port | gRPC Port | Metrics Port | Purpose |
|---------|--------|-----------|-----------|--------------|---------|
| Trading | `trading_service` | 8080 | 50051 | 9090 | Core trading engine |
| ML Training | `ml_training_service` | 8082 | 50052 | 9091 | ML model training |
| Backtesting | `backtesting_service` | 8083 | 50053 | 9092 | Strategy testing |
| TLI | `tli` | 8081 | 50054 | 9093 | Management interface |

## 🔧 Configuration Management

### Environment Variables

Core system configuration through environment variables:

```bash
# Application Configuration
FOXHUNT_ENV=production
FOXHUNT_CONFIG_PATH=/opt/foxhunt/config
FOXHUNT_DATA_PATH=/opt/foxhunt/data

# Performance Configuration  
FOXHUNT_LATENCY_TARGET_MS=50
FOXHUNT_THREAD_POOL_SIZE=0  # Auto-detect
FOXHUNT_MAX_CONNECTIONS=1000

# Database Configuration
DATABASE_URL=postgresql://foxhunt:password@localhost:5432/foxhunt
REDIS_URL=redis://localhost:6379/0
INFLUXDB_URL=http://localhost:8086

# Security Configuration
FOXHUNT_SECURITY_ENABLED=true
FOXHUNT_TLS_ENABLED=true
VAULT_ADDR=http://localhost:8200
```

### Configuration Files

Configuration files are deployed to `/opt/foxhunt/config/`:

- `trading.toml` - Trading service configuration
- `ml.toml` - ML training parameters
- `backtesting.toml` - Backtesting settings
- `tli.toml` - TLI interface configuration

## 📊 Monitoring & Observability

### Health Checking

```bash
# Quick health check
./deployment/health_check.sh

# Comprehensive health check
./deployment/health_check.sh --mode comprehensive --verbose

# Continuous monitoring
./deployment/health_check.sh --mode continuous

# JSON output for automation
./deployment/health_check.sh --format json > health.json
```

### Monitoring Stack

Deploy complete monitoring with Prometheus, Grafana, and AlertManager:

```bash
# Deploy monitoring stack
./deployment/monitoring_setup.sh --mode docker --grafana-password mypassword

# Access monitoring
# Grafana: http://localhost:3000 (admin/mypassword)
# Prometheus: http://localhost:9090
# AlertManager: http://localhost:9093
```

### Key Metrics Monitored

- **Trading Latency**: < 50μs target, < 100μs critical
- **Order Processing Rate**: Orders per second
- **System Resources**: CPU, Memory, Disk usage
- **Service Availability**: HTTP/gRPC health checks
- **Database Performance**: Query response times
- **ML Model Accuracy**: Model performance metrics

## 🔒 Security Considerations

### Production Security Checklist

- [ ] Services run as non-root user (`foxhunt`)
- [ ] File permissions properly restricted
- [ ] Network access limited to required ports
- [ ] SSL/TLS encryption enabled
- [ ] Secrets stored in HashiCorp Vault
- [ ] Audit logging enabled
- [ ] Regular security updates applied

### Security Hardening

SystemD services include security hardening:
- `NoNewPrivileges=yes`
- `ProtectSystem=strict`
- `PrivateTmp=yes`
- `RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX`
- Memory execution protection
- System call filtering

## 🔄 Deployment Workflows

### Initial Production Deployment

1. **Pre-deployment** (Use `DEPLOYMENT_CHECKLIST.md`)
   - [ ] Hardware validation
   - [ ] Network configuration
   - [ ] Security setup
   - [ ] Database preparation

2. **Build & Deploy**
   ```bash
   # Build optimized binaries
   ./deployment/build_release.sh --cpu-target native
   
   # Deploy to production
   sudo ./deployment/deploy_production.sh --mode bare-metal
   ```

3. **Post-deployment**
   ```bash
   # Setup monitoring
   ./deployment/monitoring_setup.sh
   
   # Comprehensive health check
   ./deployment/health_check.sh --mode comprehensive
   ```

### Rolling Updates

For zero-downtime updates:

```bash
# Build new binaries
./deployment/build_release.sh --cpu-target native

# Rolling deployment
sudo ./deployment/deploy_production.sh --mode bare-metal --rolling-update
```

### Rollback Procedures

```bash
# Rollback to previous deployment
sudo ./deployment/deploy_production.sh --rollback
```

## 🚨 Troubleshooting

### Common Issues

| Issue | Solution |
|-------|----------|
| Service won't start | Check `journalctl -f -u foxhunt-<service>` |
| High latency | Verify CPU affinity and frequency scaling |
| Memory issues | Check SystemD memory limits |
| Database connectivity | Verify PostgreSQL/Redis status |
| Permission errors | Check file ownership (`chown foxhunt:foxhunt`) |

### Log Locations

- **SystemD Logs**: `journalctl -f -u foxhunt-*`
- **Application Logs**: `/opt/foxhunt/logs/`
- **System Logs**: `/var/log/foxhunt/`

### Performance Debugging

```bash
# Check service performance
./deployment/health_check.sh --mode comprehensive --verbose

# Monitor real-time metrics
curl http://localhost:9090/metrics

# Check system resources
htop
iostat -x 1
```

## 📋 Production Readiness Checklist

Before going live, ensure all items in `DEPLOYMENT_CHECKLIST.md` are completed:

- [ ] Infrastructure validated
- [ ] Services deployed and healthy
- [ ] Security hardening applied
- [ ] Monitoring configured
- [ ] Backup procedures tested
- [ ] Disaster recovery plan ready
- [ ] Team training completed
- [ ] Documentation updated

## 🔗 Related Documentation

- [DEPLOYMENT_CHECKLIST.md](DEPLOYMENT_CHECKLIST.md) - Complete deployment checklist
- [../README.md](../README.md) - Project overview
- [../docs/](../docs/) - Technical documentation
- [../CLAUDE.md](../CLAUDE.md) - Project instructions

## 🆘 Support

For deployment issues:

1. Check the troubleshooting section above
2. Review service logs: `journalctl -f -u foxhunt-*`
3. Run comprehensive health check: `./deployment/health_check.sh --mode comprehensive`
4. Consult the deployment checklist for missed steps

---

**Deployment Version**: 1.0.0  
**Last Updated**: $(date)  
**Compatible with**: Foxhunt HFT v1.0.0

*This deployment package enables production-ready deployment of the Foxhunt HFT Trading System with comprehensive monitoring, security, and operational excellence.*