foxhunt/tests/smoke_tests/README.md

# Smoke Tests - End-to-End System Validation

## Overview

Automated smoke test suite for validating complete Foxhunt HFT system functionality after deployment. These tests provide quick validation that all critical components are operational.

## Test Categories

### 1. Infrastructure Health (`infrastructure_health.rs`)
Tests core infrastructure services:
- **PostgreSQL**: Connection, schema validation, TimescaleDB extension
- **Redis**: Connection, SET/GET operations, key expiration
- **Vault**: Connectivity and health status
- **InfluxDB**: Ping endpoint and availability
- **Prometheus**: Health endpoint and metrics availability
- **Grafana**: API health check

### 2. Service Health (`service_health.rs`)
Tests gRPC microservices:
- **Trading Service**: HTTP health check, gRPC connection (port 50052)
- **API Gateway**: gRPC connection (port 50051)
- **Backtesting Service**: gRPC connection (port 50053) - BLOCKED
- **ML Training Service**: gRPC connection (port 50054) - BLOCKED
- **Service Ports**: Validates all services are listening
- **Response Times**: Measures connection latency
- **Metrics Endpoints**: Validates Prometheus exporters (ports 9091-9094)

### 3. Authentication Flow (`authentication_flow.rs`)
Tests authentication and security:
- **JWT Generation**: Token creation with claims
- **JWT Validation**: Signature verification and claim extraction
- **JWT Expiration**: Expired token rejection
- **JWT Signature**: Invalid signature detection
- **Session Storage**: Redis-based session management
- **Token Revocation**: Revocation list checking
- **Rate Limiting**: Request throttling with Redis

### 4. Basic Order Flow (`basic_order_flow.rs`)
Tests core trading operations:
- **Order Submission**: Insert orders into PostgreSQL
- **Order Query**: Retrieve order by ID
- **Order Cancellation**: Update order status to cancelled
- **Position Management**: Create and query positions
- **Order History**: Query user's order history
- **Complete Lifecycle**: Full order flow from submission to cleanup

## Usage

### Run All Smoke Tests

```bash
# Using the automated script (recommended)
./run_smoke_tests.sh

# Using Cargo directly
cargo test --test smoke_tests --features smoke-tests
```

### Run Specific Categories

```bash
# Infrastructure only
./run_smoke_tests.sh --category infrastructure

# Service health only
./run_smoke_tests.sh --category service

# Authentication only
./run_smoke_tests.sh --category authentication

# Order flow only
./run_smoke_tests.sh --category order_flow
```

### Fast Mode (Critical Tests Only)

```bash
# Run only infrastructure and service health checks
./run_smoke_tests.sh --fast
```

### Verbose Mode

```bash
# Run with debug logging
./run_smoke_tests.sh --verbose
```

## Environment Variables

All tests use environment variables with sensible defaults:

```bash
# Infrastructure
DATABASE_URL=postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt
REDIS_URL=redis://localhost:6379
VAULT_ADDR=http://localhost:8200
INFLUXDB_URL=http://localhost:8086

# Services
API_GATEWAY_URL=http://localhost:50051
TRADING_SERVICE_URL=http://localhost:50052
BACKTESTING_SERVICE_URL=http://localhost:50053
ML_TRAINING_SERVICE_URL=http://localhost:50054

# Monitoring
PROMETHEUS_URL=http://localhost:9090
GRAFANA_URL=http://localhost:3000

# Authentication
JWT_SECRET=dev_secret_key_change_in_production

# Logging
RUST_LOG=info  # Set to 'debug' for verbose output
```

## Known Issues and Blockers

### Blocked Tests (Agent 96 Findings)

1. **Backtesting Service** - Configuration issues prevent deployment
   - Test marked with `#[ignore]` attribute
   - Skips gracefully when service unavailable

2. **ML Training Service** - Configuration issues prevent deployment
   - Test marked with `#[ignore]` attribute
   - Skips gracefully when service unavailable

### Working Tests

- ✅ Infrastructure Health (PostgreSQL, Redis, Vault, InfluxDB, Prometheus, Grafana)
- ✅ Trading Service Health (confirmed working by Agent 96)
- ✅ API Gateway Health
- ✅ Authentication Flow (JWT, Sessions, Rate Limiting)
- ✅ Basic Order Flow (Database operations)

## Test Architecture

### Graceful Failure Handling

Tests use the `skip_if_unavailable!` macro to handle service unavailability:

```rust
skip_if_unavailable!("Service Name", {
    // Test code here
    result
});
```

This allows tests to:
- Pass when services are available
- Skip gracefully when services are down (connection refused, timeout)
- Fail hard for actual test failures

### Timeout Management

All tests have configurable timeouts:
- **Standard smoke tests**: 10 seconds
- **Infrastructure tests**: 5 seconds

Timeouts prevent hanging tests and provide quick feedback.

### Parallel Execution

Tests can run in parallel by default, but use `--test-threads=1` for sequential execution when debugging:

```bash
cargo test --test smoke_tests -- --test-threads=1 --nocapture
```

## Integration with CI/CD

### Docker Deployment Validation

After deploying with Docker Compose:

```bash
# 1. Start all services
docker-compose up -d

# 2. Wait for health checks
docker-compose ps

# 3. Run smoke tests
./run_smoke_tests.sh

# 4. Check exit code
if [ $? -eq 0 ]; then
    echo "Deployment validated - ready for production"
else
    echo "Deployment issues detected - review logs"
fi
```

### Kubernetes Deployment Validation

```bash
# 1. Deploy to cluster
kubectl apply -f k8s/

# 2. Wait for pods
kubectl wait --for=condition=ready pod -l app=foxhunt --timeout=300s

# 3. Port forward services
kubectl port-forward svc/api-gateway 50051:50051 &
kubectl port-forward svc/trading-service 50052:50051 &

# 4. Run smoke tests
./run_smoke_tests.sh

# 5. Cleanup port forwards
pkill -f "kubectl port-forward"
```

## Metrics and Reporting

### Test Pass Rate

The smoke test runner reports:
- Total categories tested
- Passed/failed counts
- Pass percentage
- Exit code (0 = success, 1 = failure)

### Sample Output

```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Foxhunt HFT System - Smoke Test Suite
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment Configuration:
  Database: postgresql://foxhunt:***@localhost:5432/foxhunt
  Redis: redis://localhost:6379
  Vault: http://localhost:8200
  API Gateway: http://localhost:50051
  Trading Service: http://localhost:50052
  Log Level: info

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Testing: Infrastructure Health
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

✅ PostgreSQL connection successful (TimescaleDB: true)
✅ Redis connection and operations successful
✅ Vault connectivity successful (status: 200)

✅ Infrastructure Health - PASSED

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Smoke Test Summary
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  Total Categories: 4
  Passed: 4
  Failed: 0

  Pass Rate: 100%

✅ All smoke tests passed!
   System is ready for deployment.
```

## Future Enhancements

### Planned Test Categories (Not Implemented)

5. **Market Data Tests** (blocked by service availability)
   - Subscribe to market data stream
   - Receive quote updates
   - Historical data queries

6. **ML Inference Tests** (blocked by service availability)
   - Model prediction requests
   - Feature engineering pipeline
   - Inference latency validation (<100ms)

7. **Compliance Tests** (requires ML service)
   - Audit log creation
   - Best execution analysis
   - Risk check validation

8. **Advanced Monitoring** (requires full deployment)
   - Alert manager connectivity
   - Custom dashboard validation
   - Log aggregation checks

## Troubleshooting

### Common Issues

1. **Connection Refused**
   ```bash
   # Check if services are running
   docker-compose ps

   # Restart failed services
   docker-compose restart trading_service
   ```

2. **Database Schema Missing**
   ```bash
   # Run migrations
   cargo sqlx migrate run
   ```

3. **Redis Connection Failed**
   ```bash
   # Check Redis is running
   redis-cli ping

   # Restart Redis
   docker-compose restart redis
   ```

4. **JWT Secret Not Set**
   ```bash
   # Set JWT secret
   export JWT_SECRET="your-secret-key-here"
   ```

### Debug Mode

Run tests with full debug output:

```bash
RUST_LOG=debug ./run_smoke_tests.sh --verbose
```

This shows:
- Full test execution flow
- Connection attempts
- Query execution
- Error details

## Contributing

When adding new smoke tests:

1. Create test file in `tests/smoke_tests/`
2. Add module to `tests/smoke_tests/mod.rs`
3. Use `skip_if_unavailable!` macro for graceful failures
4. Add timeout with `with_timeout()` wrapper
5. Update this README with test documentation
6. Update `run_smoke_tests.sh` with new category

### Test Template

```rust
#[tokio::test]
async fn test_new_feature() {
    let result = with_timeout(async {
        // Your test code here
        Ok(())
    }).await;

    skip_if_unavailable!("Service Name", { result });
}
```