foxhunt/DATABASE_SETUP.md

# Database Setup Guide for Foxhunt HFT System

This guide explains how to set up the database for the Foxhunt HFT trading system and resolve SQLx compilation issues.

## Quick Setup (Recommended)

Run the automated setup script:

```bash
./setup-database.sh
```

This script will:
1. Start PostgreSQL using Docker (if available) or use local PostgreSQL
2. Create the database and run all migrations
3. Generate SQLx metadata for offline compilation
4. Test compilation to ensure everything works

## Manual Setup

### Option 1: Docker (Recommended)

1. Start PostgreSQL:
```bash
docker-compose -f docker-compose.dev.yml up -d postgres
```

2. Wait for PostgreSQL to be ready and migrations to complete:
```bash
docker-compose -f docker-compose.dev.yml logs postgres
```

3. Generate SQLx metadata:
```bash
export DATABASE_URL="postgresql://foxhunt:foxhunt123@localhost:5432/foxhunt"
cargo sqlx prepare
```

4. Test compilation:
```bash
SQLX_OFFLINE=true cargo check --workspace
```

### Option 2: Local PostgreSQL

1. Install PostgreSQL and create the database:
```bash
createdb foxhunt
```

2. Run the initialization script:
```bash
psql -d foxhunt -f init-db.sql
```

3. Run migrations in order:
```bash
psql -d foxhunt -f migrations/001_up_create_core_tables.sql
psql -d foxhunt -f migrations/002_up_create_risk_performance_tables.sql
# ... continue with all migrations in numerical order
psql -d foxhunt -f migrations/011_create_market_data_tables.sql
psql -d foxhunt -f migrations/012_create_event_and_config_tables.sql
```

4. Generate SQLx metadata:
```bash
export DATABASE_URL="postgresql://localhost/foxhunt"
cargo sqlx prepare
```

## Database Schema

The database includes the following main components:

### Core Trading Tables (Migration 001)
- `orders` - Trading orders with optimized indexing
- `fills` - Trade executions with foreign keys to orders
- `positions` - Current holdings by symbol and account

### Market Data Tables (Migration 011)
- `prices` - Bid/ask/last prices and OHLCV data
- `order_book_levels` - Order book depth data
- `technical_indicators` - Computed technical indicators
- `market_ticks` - Raw market tick data
- `candles` - OHLCV candlestick data

### Event Processing Tables (Migration 012)
- `market_events` - Market-related events
- `trading_events` - Trading-related events
- `event_processing_stats` - Processing statistics
- `configuration` - Application configuration
- `secrets` - Encrypted sensitive data

### Risk Management Tables (Various Migrations)
- `risk_alerts` - Risk management alerts
- `position_risks` - Position-level risk calculations
- `var_calculations` - Value at Risk calculations

## SQLx Configuration

The system uses SQLx for compile-time verification of SQL queries. The configuration includes:

- **DATABASE_URL**: Set in `.env` file
- **Migration Path**: `/home/jgrusewski/Work/foxhunt/migrations`
- **Offline Mode**: Enabled via `SQLX_OFFLINE=true` environment variable
- **Query Cache**: Stored in `sqlx-data.json` (generated by `cargo sqlx prepare`)

## Environment Variables

Key environment variables for database operation:

```bash
# Database connection
DATABASE_URL=postgresql://foxhunt:foxhunt123@localhost:5432/foxhunt

# Migration configuration
MIGRATIONS_PATH=/home/jgrusewski/Work/foxhunt/migrations

# SQLx offline mode (for compilation without live database)
SQLX_OFFLINE=true
```

## Troubleshooting

### SQLx Compilation Errors

If you see errors like "password authentication failed" or "no cached data for this query":

1. **For database connection issues**: Ensure PostgreSQL is running and credentials are correct
2. **For missing query cache**: Run `cargo sqlx prepare` to generate metadata
3. **For offline compilation**: Set `SQLX_OFFLINE=true` and ensure `sqlx-data.json` exists

### Migration Issues

1. **Check migration order**: Migrations should be applied in numerical order
2. **Verify database exists**: Ensure the target database exists before running migrations
3. **Check permissions**: Ensure the database user has necessary privileges

### Docker Issues

1. **Port conflicts**: Ensure port 5432 is available
2. **Volume permissions**: Check that Docker can access the migration files
3. **Container startup**: Wait for the healthcheck to pass before connecting

## Development Workflow

1. **Starting development**:
   ```bash
   ./setup-database.sh
   ```

2. **Adding new queries**:
   ```bash
   # After adding sqlx::query! macros
   cargo sqlx prepare
   git add sqlx-data.json
   ```

3. **Creating new migrations**:
   ```bash
   # Create migration file: migrations/013_new_feature.sql
   # Test locally, then update setup scripts
   ```

4. **Testing changes**:
   ```bash
   SQLX_OFFLINE=true cargo check --workspace
   ```

## Production Considerations

- Use proper PostgreSQL authentication (not trust mode)
- Set up connection pooling with appropriate limits
- Enable query logging and monitoring
- Regular backups and point-in-time recovery
- Partition large tables (prices, events) by time
- Monitor and optimize query performance

## Files Created/Modified

This setup creates or modifies:

- `docker-compose.dev.yml` - Docker setup for development
- `docker-init-migrations.sh` - Migration runner script
- `setup-database.sh` - Automated setup script
- `migrations/011_create_market_data_tables.sql` - Market data schema
- `migrations/012_create_event_and_config_tables.sql` - Event and config schema
- `sqlx-data.json` - SQLx query metadata cache
- `.env` - Updated with correct DATABASE_URL

The existing migration system in `core/src/persistence/migrations.rs` remains unchanged and continues to work with the new schema files.