foxhunt/testing/integration/smoke_tests

Smoke Tests - End-to-End System Validation

Overview

Automated smoke test suite for validating complete Foxhunt HFT system functionality after deployment. These tests provide quick validation that all critical components are operational.

Test Categories

1. Infrastructure Health (infrastructure_health.rs)

Tests core infrastructure services:

• PostgreSQL: Connection, schema validation, TimescaleDB extension
• Redis: Connection, SET/GET operations, key expiration
• Vault: Connectivity and health status
• InfluxDB: Ping endpoint and availability
• Prometheus: Health endpoint and metrics availability
• Grafana: API health check

2. Service Health (service_health.rs)

Tests gRPC microservices:

• Trading Service: HTTP health check, gRPC connection (port 50052)
• API Gateway: gRPC connection (port 50051)
• Backtesting Service: gRPC connection (port 50053) - BLOCKED
• ML Training Service: gRPC connection (port 50054) - BLOCKED
• Service Ports: Validates all services are listening
• Response Times: Measures connection latency
• Metrics Endpoints: Validates Prometheus exporters (ports 9091-9094)

3. Authentication Flow (authentication_flow.rs)

Tests authentication and security:

• JWT Generation: Token creation with claims
• JWT Validation: Signature verification and claim extraction
• JWT Expiration: Expired token rejection
• JWT Signature: Invalid signature detection
• Session Storage: Redis-based session management
• Token Revocation: Revocation list checking
• Rate Limiting: Request throttling with Redis

4. Basic Order Flow (basic_order_flow.rs)

Tests core trading operations:

• Order Submission: Insert orders into PostgreSQL
• Order Query: Retrieve order by ID
• Order Cancellation: Update order status to cancelled
• Position Management: Create and query positions
• Order History: Query user's order history
• Complete Lifecycle: Full order flow from submission to cleanup

Usage

Run All Smoke Tests

# Using the automated script (recommended)
./run_smoke_tests.sh

# Using Cargo directly
cargo test --test smoke_tests --features smoke-tests

Run Specific Categories

# Infrastructure only
./run_smoke_tests.sh --category infrastructure

# Service health only
./run_smoke_tests.sh --category service

# Authentication only
./run_smoke_tests.sh --category authentication

# Order flow only
./run_smoke_tests.sh --category order_flow

Fast Mode (Critical Tests Only)

# Run only infrastructure and service health checks
./run_smoke_tests.sh --fast

Verbose Mode

# Run with debug logging
./run_smoke_tests.sh --verbose

Environment Variables

All tests use environment variables with sensible defaults:

# Infrastructure
DATABASE_URL=postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt
REDIS_URL=redis://localhost:6379
VAULT_ADDR=http://localhost:8200
INFLUXDB_URL=http://localhost:8086

# Services
API_GATEWAY_URL=http://localhost:50051
TRADING_SERVICE_URL=http://localhost:50052
BACKTESTING_SERVICE_URL=http://localhost:50053
ML_TRAINING_SERVICE_URL=http://localhost:50054

# Monitoring
PROMETHEUS_URL=http://localhost:9090
GRAFANA_URL=http://localhost:3000

# Authentication
JWT_SECRET=dev_secret_key_change_in_production

# Logging
RUST_LOG=info  # Set to 'debug' for verbose output

Known Issues and Blockers

Blocked Tests (Agent 96 Findings)

1. Backtesting Service - Configuration issues prevent deployment

   • Test marked with #[ignore] attribute
   • Skips gracefully when service unavailable

2. ML Training Service - Configuration issues prevent deployment

   • Test marked with #[ignore] attribute
   • Skips gracefully when service unavailable

Working Tests

• ✅ Infrastructure Health (PostgreSQL, Redis, Vault, InfluxDB, Prometheus, Grafana)
• ✅ Trading Service Health (confirmed working by Agent 96)
• ✅ API Gateway Health
• ✅ Authentication Flow (JWT, Sessions, Rate Limiting)
• ✅ Basic Order Flow (Database operations)

Test Architecture

Graceful Failure Handling

Tests use the skip_if_unavailable! macro to handle service unavailability:

skip_if_unavailable!("Service Name", {
    // Test code here
    result
});

This allows tests to:

• Pass when services are available
• Skip gracefully when services are down (connection refused, timeout)
• Fail hard for actual test failures

Timeout Management

All tests have configurable timeouts:

• Standard smoke tests: 10 seconds
• Infrastructure tests: 5 seconds

Timeouts prevent hanging tests and provide quick feedback.

Parallel Execution

Tests can run in parallel by default, but use --test-threads=1 for sequential execution when debugging:

cargo test --test smoke_tests -- --test-threads=1 --nocapture

Integration with CI/CD

Docker Deployment Validation

After deploying with Docker Compose:

# 1. Start all services
docker-compose up -d

# 2. Wait for health checks
docker-compose ps

# 3. Run smoke tests
./run_smoke_tests.sh

# 4. Check exit code
if [ $? -eq 0 ]; then
    echo "Deployment validated - ready for production"
else
    echo "Deployment issues detected - review logs"
fi

Kubernetes Deployment Validation

# 1. Deploy to cluster
kubectl apply -f k8s/

# 2. Wait for pods
kubectl wait --for=condition=ready pod -l app=foxhunt --timeout=300s

# 3. Port forward services
kubectl port-forward svc/api-gateway 50051:50051 &
kubectl port-forward svc/trading-service 50052:50051 &

# 4. Run smoke tests
./run_smoke_tests.sh

# 5. Cleanup port forwards
pkill -f "kubectl port-forward"

Metrics and Reporting

Test Pass Rate

The smoke test runner reports:

• Total categories tested
• Passed/failed counts
• Pass percentage
• Exit code (0 = success, 1 = failure)

Sample Output

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Foxhunt HFT System - Smoke Test Suite
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment Configuration:
  Database: postgresql://foxhunt:***@localhost:5432/foxhunt
  Redis: redis://localhost:6379
  Vault: http://localhost:8200
  API Gateway: http://localhost:50051
  Trading Service: http://localhost:50052
  Log Level: info

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Testing: Infrastructure Health
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

✅ PostgreSQL connection successful (TimescaleDB: true)
✅ Redis connection and operations successful
✅ Vault connectivity successful (status: 200)

✅ Infrastructure Health - PASSED

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Smoke Test Summary
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  Total Categories: 4
  Passed: 4
  Failed: 0

  Pass Rate: 100%

✅ All smoke tests passed!
   System is ready for deployment.

Future Enhancements

Planned Test Categories (Not Implemented)

5. Market Data Tests (blocked by service availability)

   • Subscribe to market data stream
   • Receive quote updates
   • Historical data queries

6. ML Inference Tests (blocked by service availability)

   • Model prediction requests
   • Feature engineering pipeline
   • Inference latency validation (<100ms)

7. Compliance Tests (requires ML service)

   • Audit log creation
   • Best execution analysis
   • Risk check validation

8. Advanced Monitoring (requires full deployment)

   • Alert manager connectivity
   • Custom dashboard validation
   • Log aggregation checks

Troubleshooting

Common Issues

1. Connection Refused

   # Check if services are running
   docker-compose ps
   
   # Restart failed services
   docker-compose restart trading_service
   

2. Database Schema Missing

   # Run migrations
   cargo sqlx migrate run
   

3. Redis Connection Failed

   # Check Redis is running
   redis-cli ping
   
   # Restart Redis
   docker-compose restart redis
   

4. JWT Secret Not Set

   # Set JWT secret
   export JWT_SECRET="your-secret-key-here"
   

Debug Mode

Run tests with full debug output:

RUST_LOG=debug ./run_smoke_tests.sh --verbose

This shows:

• Full test execution flow
• Connection attempts
• Query execution
• Error details

Contributing

When adding new smoke tests:

1. Create test file in tests/smoke_tests/
2. Add module to tests/smoke_tests/mod.rs
3. Use skip_if_unavailable! macro for graceful failures
4. Add timeout with with_timeout() wrapper
5. Update this README with test documentation
6. Update run_smoke_tests.sh with new category

Test Template

#[tokio::test]
async fn test_new_feature() {
    let result = with_timeout(async {
        // Your test code here
        Ok(())
    }).await;

    skip_if_unavailable!("Service Name", { result });
}