foxhunt/adaptive-strategy/docs/hot_reload_testing.md

# Hot-Reload Integration Testing Documentation

## Overview

This document describes the comprehensive integration test suite for the adaptive strategy configuration hot-reload functionality, implemented via PostgreSQL NOTIFY/LISTEN.

## Test Suite Structure

### File: `tests/hot_reload_integration.rs`

The test suite contains **25 comprehensive tests** organized into 5 categories:

1. **Hot-Reload Notification Tests** (6 tests) - Core NOTIFY/LISTEN functionality
2. **ACID Transaction Tests** (5 tests) - Transaction integrity and isolation
3. **Concurrent Update Tests** (1 test) - Race condition handling
4. **Performance Tests** (2 tests, `#[ignore]`) - Latency and throughput benchmarks
5. **Edge Cases & Failure Modes** (4 tests) - Error handling and resilience

## Prerequisites

### Database Setup

```bash
# Set database connection
export DATABASE_URL="postgresql://postgres:postgres@localhost:5432/foxhunt_test"

# Run migrations
sqlx migrate run
```

### Running Tests

```bash
# Run all hot-reload integration tests
cargo test --test hot_reload_integration --features postgres

# Run with output
cargo test --test hot_reload_integration --features postgres -- --nocapture

# Run performance tests (normally ignored)
cargo test --test hot_reload_integration --features postgres -- --ignored --nocapture

# Run specific test
cargo test --test hot_reload_integration test_hot_reload_enable_and_listen --features postgres
```

## Test Categories

### Category 1: Hot-Reload Notification Tests

#### Test 1: `test_hot_reload_enable_and_listen`
**Purpose:** Verify basic NOTIFY/LISTEN lifecycle

**What it tests:**
- `enable_hot_reload()` successfully creates listener
- Configuration updates trigger notifications
- `check_for_updates()` receives correct strategy_id
- Timeout handling for notification waits

**Expected behavior:**
- Notification received within 5 seconds
- Payload contains `default-production` strategy_id

---

#### Test 2: `test_hot_reload_notification_on_model_update`
**Purpose:** Verify model table changes trigger notifications

**What it tests:**
- Trigger on `adaptive_strategy_models` table
- Notification propagation for model updates
- Foreign key relationship integrity

**Expected behavior:**
- Model weight update triggers notification
- Strategy_id correctly extracted from notification

---

#### Test 3: `test_hot_reload_notification_on_feature_update`
**Purpose:** Verify feature table changes trigger notifications

**What it tests:**
- Trigger on `adaptive_strategy_features` table
- Notification for feature enable/disable
- Related configuration updates

**Expected behavior:**
- Feature toggle triggers notification
- Notification received for correct strategy

---

#### Test 4: `test_hot_reload_multiple_listeners`
**Purpose:** Verify multiple services receive same notification

**What it tests:**
- Multi-service coordination
- Broadcast nature of PostgreSQL NOTIFY
- Payload consistency across listeners

**Expected behavior:**
- All 3 loaders receive identical notification
- No message loss or duplication
- Payloads are identical

---

#### Test 5: `test_hot_reload_notification_format`
**Purpose:** Validate notification payload structure

**What it tests:**
- JSON payload format
- Required fields: table, action, strategy_id, timestamp
- Payload parsing logic

**Expected behavior:**
- strategy_id correctly extracted
- Timestamp within reasonable range
- No parsing errors

---

#### Test 6: `test_hot_reload_listener_reconnection`
**Purpose:** Verify recovery from connection loss

**What it tests:**
- Listener reconnection after disconnect
- State recovery after re-enable
- Notification delivery post-reconnection

**Expected behavior:**
- Re-enable succeeds without error
- Notifications work after reconnection
- No message loss

---

### Category 2: ACID Transaction Tests

#### Test 7: `test_atomic_config_update_all_or_nothing`
**Purpose:** Verify atomic commit across 3 tables

**What it tests:**
- **Atomicity:** All 3 tables update or none
- Transaction BEGIN/COMMIT sequence
- Multi-table consistency

**Expected behavior:**
- Config, models, and features all inserted
- Single transaction commits successfully
- All changes visible after commit

---

#### Test 8: `test_transaction_rollback_on_constraint_violation`
**Purpose:** Verify rollback on constraint violation

**What it tests:**
- **Consistency:** Invalid data rejected
- UNIQUE constraint enforcement
- Transaction rollback on error

**Expected behavior:**
- Duplicate strategy_id fails
- Original config unchanged
- No partial updates

---

#### Test 9: `test_transaction_isolation_read_committed`
**Purpose:** Verify READ COMMITTED isolation level

**What it tests:**
- **Isolation:** Uncommitted changes not visible
- Transaction T1 and T2 interaction
- Commit visibility timing

**Expected behavior:**
- T2 sees original value before T1 commits
- T2 sees updated value after T1 commits
- No dirty reads

---

#### Test 10: `test_transaction_consistency_validation`
**Purpose:** Verify validation prevents invalid configs

**What it tests:**
- Business rule validation
- **Consistency:** Invalid state detection
- Validation after database load

**Expected behavior:**
- Invalid config (max_position_size > 1.0) inserted to DB
- `load_config()` fails validation
- Error returned to caller

---

#### Test 11: `test_transaction_durability_after_commit`
**Purpose:** Verify committed changes persist

**What it tests:**
- **Durability:** Changes survive connection close
- Persistence after pool drop
- Crash recovery simulation

**Expected behavior:**
- Config persists after connection close
- New pool sees committed data
- No data loss

---

### Category 3: Concurrent Update Tests

#### Test 15: `test_concurrent_updates_with_version`
**Purpose:** Verify optimistic locking via version field

**What it tests:**
- Version-based concurrency control
- Lost update prevention
- Concurrent update detection

**Expected behavior:**
- Only one update succeeds (WHERE version = 1)
- One task gets 0 rows affected
- Final version is 2 (incremented once)

---

### Category 4: Performance Tests (Ignored by Default)

#### Test 18: `test_config_load_latency_benchmark`
**Purpose:** Measure config loading performance

**What it tests:**
- Database query latency
- Connection pool performance
- P50, P95, P99 percentiles

**Expected behavior:**
- p99 latency < 100ms
- Consistent performance across 100 iterations

---

#### Test 20: `test_notification_propagation_delay`
**Purpose:** Measure NOTIFY latency

**What it tests:**
- Time from UPDATE to notification received
- Network + database latency
- LISTEN mechanism performance

**Expected behavior:**
- Average delay < 100ms
- Consistent propagation timing

---

### Category 5: Edge Cases & Failure Modes

#### Test 22: `test_malformed_notification_payload`
**Purpose:** Handle invalid JSON in notification

**What it tests:**
- Error handling for malformed payloads
- Graceful degradation
- No panic on invalid data

**Expected behavior:**
- No panic or crash
- Returns None or skips invalid notification
- Continues processing valid notifications

---

#### Test 24: `test_config_update_during_service_restart`
**Purpose:** Verify resilience during service lifecycle

**What it tests:**
- Config persistence across restarts
- Loader initialization with latest data
- No stale data after restart

**Expected behavior:**
- New loader sees latest config
- No caching issues
- Immediate consistency

---

#### Test 25: `test_partial_transaction_failure`
**Purpose:** Verify rollback on partial failure

**What it tests:**
- Transaction rollback on mid-transaction error
- Foreign key constraint enforcement
- All-or-nothing guarantee

**Expected behavior:**
- Config update rolls back despite success
- Foreign key violation causes full rollback
- Original config unchanged

---

## Configuration Lifecycle

### 1. Initial Load
```rust
let loader = DatabaseConfigLoader::new(&database_url).await?;
let config = loader.load_config("default-production").await?;
```

### 2. Enable Hot-Reload
```rust
loader.enable_hot_reload().await?;
```

### 3. Monitor for Changes
```rust
loop {
    if let Some(strategy_id) = loader.check_for_updates().await? {
        println!("Configuration changed for: {}", strategy_id);
        let new_config = loader.load_config(&strategy_id).await?;
        // Apply new configuration...
    }
    tokio::time::sleep(Duration::from_secs(1)).await;
}
```

### 4. Database Trigger Mechanism

```sql
-- Trigger function (from migration 015)
CREATE OR REPLACE FUNCTION notify_adaptive_strategy_config_change()
RETURNS TRIGGER AS $$
DECLARE
    payload JSON;
BEGIN
    payload = json_build_object(
        'table', TG_TABLE_NAME,
        'action', TG_OP,
        'strategy_id', COALESCE(NEW.strategy_id, OLD.strategy_id),
        'timestamp', EXTRACT(EPOCH FROM NOW())
    );
    PERFORM pg_notify('adaptive_strategy_config_change', payload::text);
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

-- Triggers on all 3 tables
CREATE TRIGGER adaptive_strategy_config_notify
AFTER INSERT OR UPDATE OR DELETE ON adaptive_strategy_config
FOR EACH ROW EXECUTE FUNCTION notify_adaptive_strategy_config_change();
```

## Test Helpers

### `create_loader()`
Creates a DatabaseConfigLoader connected to test database.

### `create_test_pool()`
Creates a separate connection pool for test operations (updates, verifications).

### `wait_for_notification(loader, timeout_secs)`
Waits for notification with timeout to prevent test hangs.

### `cleanup_test_strategy(pool, strategy_id)`
Removes test data after test completion (cascade delete).

## Common Test Patterns

### Pattern 1: Dual Connection Test
```rust
// Loader for listening
let mut loader = create_loader().await;
loader.enable_hot_reload().await?;

// Separate pool for updates
let pool = create_test_pool().await;
sqlx::query("UPDATE ...").execute(&pool).await?;

// Wait for notification
let notif = wait_for_notification(&mut loader, 5).await?;
```

### Pattern 2: Transaction Verification
```rust
let pool = create_test_pool().await;

let mut tx = pool.begin().await?;
// ... perform operations ...
tx.commit().await?;

// Verify via separate query
let result = sqlx::query(...).fetch_one(&pool).await?;
assert_eq!(result, expected);
```

### Pattern 3: Concurrent Tasks
```rust
let task1 = tokio::spawn(async move { /* update 1 */ });
let task2 = tokio::spawn(async move { /* update 2 */ });

let result1 = task1.await??;
let result2 = task2.await??;

// Verify only one succeeded
assert_eq!(result1.rows_affected() + result2.rows_affected(), 1);
```

## Troubleshooting

### Test Hangs
- **Cause:** Notification not received, `wait_for_notification()` times out
- **Fix:** Check PostgreSQL logs, verify triggers are installed, ensure migrations ran

### Connection Errors
- **Cause:** DATABASE_URL incorrect or PostgreSQL not running
- **Fix:** Verify `psql $DATABASE_URL` works, check PostgreSQL service status

### Constraint Violations
- **Cause:** Test cleanup didn't run from previous test
- **Fix:** Manually clean test database:
  ```sql
  DELETE FROM adaptive_strategy_config WHERE strategy_id LIKE 'test-%';
  ```

### Version Conflicts
- **Cause:** Multiple tests updating same config without cleanup
- **Fix:** Use unique strategy_ids per test, ensure cleanup runs

## CI/CD Integration

### GitHub Actions Example
```yaml
jobs:
  test-hot-reload:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: foxhunt_test
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    steps:
      - uses: actions/checkout@v3
      - name: Run migrations
        run: sqlx migrate run
        env:
          DATABASE_URL: postgresql://postgres:postgres@localhost:5432/foxhunt_test

      - name: Run hot-reload tests
        run: cargo test --test hot_reload_integration --features postgres
        env:
          DATABASE_URL: postgresql://postgres:postgres@localhost:5432/foxhunt_test

      - name: Run performance tests
        run: cargo test --test hot_reload_integration --features postgres -- --ignored
        env:
          DATABASE_URL: postgresql://postgres:postgres@localhost:5432/foxhunt_test
```

## Performance Benchmarks

### Expected Latencies (p99)
- **Config Load:** < 100ms
- **Notification Propagation:** < 100ms
- **Transaction Commit:** < 50ms

### Tested Load
- **Concurrent Listeners:** 3+ simultaneous services
- **Update Frequency:** 10 updates/second
- **Config Size:** Production config with 3-5 models, 4-6 features

## Future Enhancements

1. **Cache Invalidation Tests:**
   - Test ConfigManager cache behavior with hot-reload
   - Verify stale data not served after notification

2. **Multi-Database Tests:**
   - Test across PostgreSQL read replicas
   - Verify notification propagation in distributed setup

3. **Failure Injection:**
   - Network partition simulation
   - Database crash recovery
   - Listener process kill/restart

4. **Load Testing:**
   - 1000+ concurrent updates
   - Notification buffer overflow handling
   - Connection pool exhaustion

## References

- **Migration:** `/home/jgrusewski/Work/foxhunt/database/migrations/015_adaptive_strategy_config.sql`
- **Loader:** `/home/jgrusewski/Work/foxhunt/adaptive-strategy/src/database_loader.rs`
- **Config Types:** `/home/jgrusewski/Work/foxhunt/adaptive-strategy/src/config_types.rs`
- **Existing Tests:** `/home/jgrusewski/Work/foxhunt/adaptive-strategy/tests/database_config_integration.rs`

---

**Last Updated:** 2025-10-03
**Test Suite Version:** 1.0
**Total Tests:** 25 (17 active, 2 ignored performance tests, 6 hot-reload core tests)