foxhunt/CI_CD_SETUP.md

CI/CD Quality Gates Setup

This document describes the automated quality enforcement system for the Foxhunt HFT Trading System.

Overview

The CI/CD pipeline enforces strict quality gates to ensure code reliability, security, and performance for high-frequency trading operations. All checks must pass before code can be merged to main.

Quality Gates

1. Compilation Check ✅

• Zero compilation errors tolerance
• All workspace crates must compile successfully
• Enforced via cargo check --workspace --all-targets

2. Clippy Linting 🔍

• Zero warnings tolerance with -D warnings flag
• All clippy lints must pass
• Enforced via cargo clippy --workspace --all-targets -- -D warnings

3. Test Suite 🧪

• Unit tests, integration tests, and doc tests
• Tests excluding external dependencies (redis, kill_switch)
• Concurrency testing with Loom
• Cross-platform testing (Linux, macOS, Windows)

4. Code Coverage 📊

• Minimum coverage threshold enforced
• Generated with cargo-tarpaulin
• Reports uploaded to Codecov
• HTML reports available as artifacts

5. Security Audit 🔒

• Daily automated security scans
• Dependency vulnerability checks with cargo-audit
• License compliance with cargo-deny
• Supply chain security validation
• Cryptographic security validation
• Memory safety analysis with cargo-geiger

6. Warning Count Check ⚠️

• Maximum 50 warnings allowed
• Tracks warning trends over time
• Encourages clean code practices

GitHub Actions Workflows

Main CI Pipeline (.github/workflows/ci.yml)

Triggers: Push to main/develop, pull requests to main

Jobs:

1. check: Fast compilation and lint checks
2. test: Comprehensive test matrix (stable, beta, nightly)
3. quality: Enterprise-grade linting and security
4. coverage: Code coverage analysis
5. concurrency: Loom-based concurrency testing
6. benchmarks: Performance regression detection
7. integration: Tests with real PostgreSQL/Redis
8. cross-platform: Multi-platform builds
9. documentation: API docs generation

Security Workflow (.github/workflows/security.yml)

Triggers: Daily at midnight UTC, push to main, manual trigger

Jobs:

1. audit: Security vulnerability scanning
2. dependency-check: License and security policy enforcement
3. outdated-check: Outdated dependency detection
4. security-summary: Aggregated security report

Financial Security Audit (.github/workflows/financial-security-audit.yml)

Triggers: Weekly, push to main/develop, pull requests

Enhanced checks:

• Financial system vulnerability scanning
• Supply chain security analysis
• Cryptographic security validation
• Numeric precision security check
• Memory safety deep analysis
• Network security validation

Local Development Commands

Using Make (Available Now)

# Quick checks before committing
make pre-commit

# Full quality gate checks
make check-all

# Run all tests
make test

# Run tests excluding external dependencies
make test-fast

# Generate code coverage
make coverage

# Run security audit
make audit

# Check for outdated dependencies
make outdated

# Format code
make fmt

# Run clippy lints
make clippy

# Count warnings
make warnings

# Build all services (release mode)
make build-release

# Clean build artifacts
make clean

# Simulate CI pipeline locally
make ci-local

# Pre-merge validation (same as CI)
make pre-merge

# Run trading service
make run-trading

# Run TLI terminal interface
make run-tli

# Show all available commands
make help

Using just (Install: cargo install just)

# Quick checks before committing
just pre-commit

# Full quality gate checks
just check-all

# Run all tests
just test

# Run unit tests only
just test-unit

# Generate code coverage
just coverage

# Run security audit
just audit

# Format code
just fmt

# Run clippy lints
just clippy

# Count warnings
just warnings

# Build in release mode
just build-release

# Watch for changes and run checks
just watch

# Fix common issues automatically
just fix

# Show project statistics
just stats

# Show environment info
just env-info

# Show all available commands
just

CI Configuration Details

Environment Variables

RUST_BACKTRACE: 1              # Enable backtraces
CARGO_TERM_COLOR: always       # Colored output
CARGO_INCREMENTAL: 0           # Disable incremental for CI
RUSTFLAGS: "-Dwarnings"        # Treat warnings as errors

Caching Strategy

• Uses Swatinem/rust-cache@v2 for dependency caching
• Separate cache keys for different Rust versions and platforms
• Significant speedup for subsequent CI runs

Service Dependencies

Integration tests use real service containers:

• PostgreSQL 16: Database testing
• Redis 7: Cache testing
• Health checks ensure services are ready

Artifact Retention

• Coverage reports: 30 days
• Security audit results: 90 days
• Benchmark results: 90 days
• Quality reports: Available per job

Quality Standards

Zero Tolerance Policies

1. ❌ Compilation errors: BLOCKED
2. ❌ Clippy warnings: BLOCKED
3. ❌ Security vulnerabilities: BLOCKED
4. ❌ Placeholder code: BLOCKED (TODO, FIXME, unimplemented!)
5. ❌ Formatting issues: BLOCKED

Warning Thresholds

• Maximum 50 warnings allowed workspace-wide
• Encourages progressive warning reduction
• Tracks warning count trends

Code Coverage

• Minimum coverage threshold enforced
• Coverage reports generated for all jobs
• Trends tracked over time

Pre-Commit Checklist

Before committing, run:

make pre-commit        # or: just pre-commit

Before creating a PR, run:

make pre-merge         # or: just pre-merge

This ensures your code passes CI checks locally before pushing.

Continuous Improvement

Adding New Checks

1. Add check to appropriate workflow YAML
2. Update this documentation
3. Add corresponding command to Makefile/justfile
4. Test locally before pushing

Modifying Thresholds

• Warning threshold: Update .github/workflows/ci.yml line with warning count check
• Coverage threshold: Update tarpaulin configuration
• Test timeouts: Update workflow timeout settings

Performance Benchmarking

Benchmarks run on every push to main:

make bench             # or: just bench

Results stored as artifacts for comparison.

Troubleshooting

CI Failures

Compilation errors:

make check

Clippy warnings:

make clippy
make fix    # Auto-fix where possible

Test failures:

make test-verbose

Coverage too low:

make coverage
# Review: target/tarpaulin/index.html

Security issues:

make audit
make outdated

Local vs CI Differences

If CI fails but local passes:

1. Ensure Rust version matches CI (stable)
2. Check environment variables
3. Run with CI flags: RUSTFLAGS="-Dwarnings" cargo check
4. Clear cache: make clean && make check-all

References

• Cargo Documentation
• GitHub Actions Documentation
• cargo-audit
• cargo-tarpaulin
• cargo-deny

Status Summary

✅ Main CI Pipeline: Comprehensive quality gates ✅ Security Audit: Daily automated scanning ✅ Financial Security: Weekly deep analysis ✅ Local Development: Make and just support ✅ Documentation: Complete setup guide ✅ Quality Standards: Zero tolerance for critical issues

---

Last Updated: 2025-10-01 Status: Production Ready - All quality gates operational