# Running Cross-Service Integration Tests

This guide explains how to run the comprehensive integration tests for the Foxhunt HFT Trading System.

## Prerequisites

1. **Docker services running**:
   ```bash
   docker-compose up -d
   docker-compose ps  # Verify all services healthy
   ```

2. **Services operational**:
   - API Gateway (ports 50051, 9091)
   - Trading Service (ports 50052, 9092)
   - Backtesting Service (ports 50053, 9093)
   - ML Training Service (ports 50054, 9094)
   - PostgreSQL (port 5432)
   - Redis (port 6379)
   - Prometheus (port 9090)

## Test Suites

### 1. Infrastructure Tests

Tests HTTP health endpoints, database connectivity, Redis, and Prometheus monitoring.

```bash
./cross_service_integration_test.sh
```

**Expected Output**:
```
================================
Cross-Service Integration Tests
================================

Total Tests:  21
Passed:       20
Failed:       1

Pass Rate:    95.2%
```

---

### 2. gRPC Integration Tests

Tests gRPC port connectivity, database performance, cache performance, and latency.

```bash
./grpc_integration_test.sh
```

**Expected Output**:
```
======================================
gRPC Cross-Service Integration Tests
======================================

Total Tests:  10
Passed:       10
Failed:       0

Pass Rate:    100.0%
```

---

### 3. E2E Tests (Rust)

Comprehensive end-to-end tests covering order flow, ML integration, and backtesting.

```bash
# All E2E tests
cargo test -p foxhunt_e2e --test '*' -- --nocapture

# Specific test file
cargo test -p foxhunt_e2e --test multi_service_integration -- --nocapture
cargo test -p foxhunt_e2e --test full_trading_flow_e2e -- --nocapture
```

**Expected Output**: 15/15 tests passing (100%)

---

## Test Reports

After running tests, review the generated reports:

1. **Comprehensive Report**: `CROSS_SERVICE_INTEGRATION_REPORT.md` (655 lines)
2. **Quick Summary**: `INTEGRATION_TEST_SUMMARY.md`
3. **Test Logs**: `/tmp/cross_service_results.txt`, `/tmp/grpc_integration_results.txt`

---

## Troubleshooting

### Services Not Healthy

```bash
# Check service status
docker-compose ps

# Restart unhealthy services
docker-compose restart api-gateway trading-service

# Check logs
docker-compose logs -f api-gateway
```

---

### Port Conflicts

```bash
# Check which ports are in use
lsof -i :50051,50052,50053,50054

# Kill conflicting processes
kill -9 $(lsof -ti:50051)
```

---

### Database Connection Issues

```bash
# Test PostgreSQL connection
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT 1"

# Run migrations if needed
cargo sqlx migrate run
```

---

### Redis Connection Issues

```bash
# Test Redis via Docker
docker exec 496d979ef7da redis-cli PING

# Check Redis container status
docker ps | grep redis
```

---

## Quick Health Check

Run this one-liner to verify all services:

```bash
for port in 9091 9092 8083 8095; do 
  echo -n "Port $port: "; 
  curl -s http://localhost:$port/health >/dev/null && echo "✓ OK" || curl -s http://localhost:$port/metrics >/dev/null && echo "✓ OK (metrics)" || echo "✗ FAIL"; 
done
```

Expected output:
```
Port 9091: ✓ OK (metrics)
Port 9092: ✓ OK
Port 8083: ✓ OK
Port 8095: ✓ OK
```

---

## Production Validation

Before deploying to production, ensure:

1. ✅ All integration tests passing (>95% pass rate)
2. ✅ Service health endpoints responding
3. ✅ Database write performance >2,000 inserts/sec
4. ✅ Inter-service latency <100ms
5. ✅ Prometheus targets healthy (5/5)
6. ✅ E2E tests passing (15/15)

---

**Last Updated**: 2025-10-11 (Wave 135 Complete)