foxhunt/DATABASE_SETUP.md

Database Setup Guide for Foxhunt HFT System

This guide explains how to set up the database for the Foxhunt HFT trading system and resolve SQLx compilation issues.

Quick Setup (Recommended)

Run the automated setup script:

./setup-database.sh

This script will:

1. Start PostgreSQL using Docker (if available) or use local PostgreSQL
2. Create the database and run all migrations
3. Generate SQLx metadata for offline compilation
4. Test compilation to ensure everything works

Manual Setup

Option 1: Docker (Recommended)

1. Start PostgreSQL:

docker-compose -f docker-compose.dev.yml up -d postgres

2. Wait for PostgreSQL to be ready and migrations to complete:

docker-compose -f docker-compose.dev.yml logs postgres

3. Generate SQLx metadata:

export DATABASE_URL="postgresql://foxhunt:foxhunt123@localhost:5432/foxhunt"
cargo sqlx prepare

4. Test compilation:

SQLX_OFFLINE=true cargo check --workspace

Option 2: Local PostgreSQL

1. Install PostgreSQL and create the database:

createdb foxhunt

2. Run the initialization script:

psql -d foxhunt -f init-db.sql

3. Run migrations in order:

psql -d foxhunt -f migrations/001_up_create_core_tables.sql
psql -d foxhunt -f migrations/002_up_create_risk_performance_tables.sql
# ... continue with all migrations in numerical order
psql -d foxhunt -f migrations/011_create_market_data_tables.sql
psql -d foxhunt -f migrations/012_create_event_and_config_tables.sql

4. Generate SQLx metadata:

export DATABASE_URL="postgresql://localhost/foxhunt"
cargo sqlx prepare

Database Schema

The database includes the following main components:

Core Trading Tables (Migration 001)

• orders - Trading orders with optimized indexing
• fills - Trade executions with foreign keys to orders
• positions - Current holdings by symbol and account

Market Data Tables (Migration 011)

• prices - Bid/ask/last prices and OHLCV data
• order_book_levels - Order book depth data
• technical_indicators - Computed technical indicators
• market_ticks - Raw market tick data
• candles - OHLCV candlestick data

Event Processing Tables (Migration 012)

• market_events - Market-related events
• trading_events - Trading-related events
• event_processing_stats - Processing statistics
• configuration - Application configuration
• secrets - Encrypted sensitive data

Risk Management Tables (Various Migrations)

• risk_alerts - Risk management alerts
• position_risks - Position-level risk calculations
• var_calculations - Value at Risk calculations

SQLx Configuration

The system uses SQLx for compile-time verification of SQL queries. The configuration includes:

• DATABASE_URL: Set in .env file
• Migration Path: /home/jgrusewski/Work/foxhunt/migrations
• Offline Mode: Enabled via SQLX_OFFLINE=true environment variable
• Query Cache: Stored in sqlx-data.json (generated by cargo sqlx prepare)

Environment Variables

Key environment variables for database operation:

# Database connection
DATABASE_URL=postgresql://foxhunt:foxhunt123@localhost:5432/foxhunt

# Migration configuration
MIGRATIONS_PATH=/home/jgrusewski/Work/foxhunt/migrations

# SQLx offline mode (for compilation without live database)
SQLX_OFFLINE=true

Troubleshooting

SQLx Compilation Errors

If you see errors like "password authentication failed" or "no cached data for this query":

1. For database connection issues: Ensure PostgreSQL is running and credentials are correct
2. For missing query cache: Run cargo sqlx prepare to generate metadata
3. For offline compilation: Set SQLX_OFFLINE=true and ensure sqlx-data.json exists

Migration Issues

1. Check migration order: Migrations should be applied in numerical order
2. Verify database exists: Ensure the target database exists before running migrations
3. Check permissions: Ensure the database user has necessary privileges

Docker Issues

1. Port conflicts: Ensure port 5432 is available
2. Volume permissions: Check that Docker can access the migration files
3. Container startup: Wait for the healthcheck to pass before connecting

Development Workflow

1. Starting development:

   ./setup-database.sh
   

2. Adding new queries:

   # After adding sqlx::query! macros
   cargo sqlx prepare
   git add sqlx-data.json
   

3. Creating new migrations:

   # Create migration file: migrations/013_new_feature.sql
   # Test locally, then update setup scripts
   

4. Testing changes:

   SQLX_OFFLINE=true cargo check --workspace
   

Production Considerations

• Use proper PostgreSQL authentication (not trust mode)
• Set up connection pooling with appropriate limits
• Enable query logging and monitoring
• Regular backups and point-in-time recovery
• Partition large tables (prices, events) by time
• Monitor and optimize query performance

Files Created/Modified

This setup creates or modifies:

• docker-compose.dev.yml - Docker setup for development
• docker-init-migrations.sh - Migration runner script
• setup-database.sh - Automated setup script
• migrations/011_create_market_data_tables.sql - Market data schema
• migrations/012_create_event_and_config_tables.sql - Event and config schema
• sqlx-data.json - SQLx query metadata cache
• .env - Updated with correct DATABASE_URL

The existing migration system in core/src/persistence/migrations.rs remains unchanged and continues to work with the new schema files.