foxhunt/adaptive-strategy/docs/hot_reload_testing.md

Hot-Reload Integration Testing Documentation

Overview

This document describes the comprehensive integration test suite for the adaptive strategy configuration hot-reload functionality, implemented via PostgreSQL NOTIFY/LISTEN.

Test Suite Structure

File: tests/hot_reload_integration.rs

The test suite contains 25 comprehensive tests organized into 5 categories:

1. Hot-Reload Notification Tests (6 tests) - Core NOTIFY/LISTEN functionality
2. ACID Transaction Tests (5 tests) - Transaction integrity and isolation
3. Concurrent Update Tests (1 test) - Race condition handling
4. Performance Tests (2 tests, #[ignore]) - Latency and throughput benchmarks
5. Edge Cases & Failure Modes (4 tests) - Error handling and resilience

Prerequisites

Database Setup

# Set database connection
export DATABASE_URL="postgresql://postgres:postgres@localhost:5432/foxhunt_test"

# Run migrations
sqlx migrate run

Running Tests

# Run all hot-reload integration tests
cargo test --test hot_reload_integration --features postgres

# Run with output
cargo test --test hot_reload_integration --features postgres -- --nocapture

# Run performance tests (normally ignored)
cargo test --test hot_reload_integration --features postgres -- --ignored --nocapture

# Run specific test
cargo test --test hot_reload_integration test_hot_reload_enable_and_listen --features postgres

Test Categories

Category 1: Hot-Reload Notification Tests

Test 1: test_hot_reload_enable_and_listen

Purpose: Verify basic NOTIFY/LISTEN lifecycle

What it tests:

• enable_hot_reload() successfully creates listener
• Configuration updates trigger notifications
• check_for_updates() receives correct strategy_id
• Timeout handling for notification waits

Expected behavior:

• Notification received within 5 seconds
• Payload contains default-production strategy_id

---

Test 2: test_hot_reload_notification_on_model_update

Purpose: Verify model table changes trigger notifications

What it tests:

• Trigger on adaptive_strategy_models table
• Notification propagation for model updates
• Foreign key relationship integrity

Expected behavior:

• Model weight update triggers notification
• Strategy_id correctly extracted from notification

---

Test 3: test_hot_reload_notification_on_feature_update

Purpose: Verify feature table changes trigger notifications

What it tests:

• Trigger on adaptive_strategy_features table
• Notification for feature enable/disable
• Related configuration updates

Expected behavior:

• Feature toggle triggers notification
• Notification received for correct strategy

---

Test 4: test_hot_reload_multiple_listeners

Purpose: Verify multiple services receive same notification

What it tests:

• Multi-service coordination
• Broadcast nature of PostgreSQL NOTIFY
• Payload consistency across listeners

Expected behavior:

• All 3 loaders receive identical notification
• No message loss or duplication
• Payloads are identical

---

Test 5: test_hot_reload_notification_format

Purpose: Validate notification payload structure

What it tests:

• JSON payload format
• Required fields: table, action, strategy_id, timestamp
• Payload parsing logic

Expected behavior:

• strategy_id correctly extracted
• Timestamp within reasonable range
• No parsing errors

---

Test 6: test_hot_reload_listener_reconnection

Purpose: Verify recovery from connection loss

What it tests:

• Listener reconnection after disconnect
• State recovery after re-enable
• Notification delivery post-reconnection

Expected behavior:

• Re-enable succeeds without error
• Notifications work after reconnection
• No message loss

---

Category 2: ACID Transaction Tests

Test 7: test_atomic_config_update_all_or_nothing

Purpose: Verify atomic commit across 3 tables

What it tests:

• Atomicity: All 3 tables update or none
• Transaction BEGIN/COMMIT sequence
• Multi-table consistency

Expected behavior:

• Config, models, and features all inserted
• Single transaction commits successfully
• All changes visible after commit

---

Test 8: test_transaction_rollback_on_constraint_violation

Purpose: Verify rollback on constraint violation

What it tests:

• Consistency: Invalid data rejected
• UNIQUE constraint enforcement
• Transaction rollback on error

Expected behavior:

• Duplicate strategy_id fails
• Original config unchanged
• No partial updates

---

Test 9: test_transaction_isolation_read_committed

Purpose: Verify READ COMMITTED isolation level

What it tests:

• Isolation: Uncommitted changes not visible
• Transaction T1 and T2 interaction
• Commit visibility timing

Expected behavior:

• T2 sees original value before T1 commits
• T2 sees updated value after T1 commits
• No dirty reads

---

Test 10: test_transaction_consistency_validation

Purpose: Verify validation prevents invalid configs

What it tests:

• Business rule validation
• Consistency: Invalid state detection
• Validation after database load

Expected behavior:

• Invalid config (max_position_size > 1.0) inserted to DB
• load_config() fails validation
• Error returned to caller

---

Test 11: test_transaction_durability_after_commit

Purpose: Verify committed changes persist

What it tests:

• Durability: Changes survive connection close
• Persistence after pool drop
• Crash recovery simulation

Expected behavior:

• Config persists after connection close
• New pool sees committed data
• No data loss

---

Category 3: Concurrent Update Tests

Test 15: test_concurrent_updates_with_version

Purpose: Verify optimistic locking via version field

What it tests:

• Version-based concurrency control
• Lost update prevention
• Concurrent update detection

Expected behavior:

• Only one update succeeds (WHERE version = 1)
• One task gets 0 rows affected
• Final version is 2 (incremented once)

---

Category 4: Performance Tests (Ignored by Default)

Test 18: test_config_load_latency_benchmark

Purpose: Measure config loading performance

What it tests:

• Database query latency
• Connection pool performance
• P50, P95, P99 percentiles

Expected behavior:

• p99 latency < 100ms
• Consistent performance across 100 iterations

---

Test 20: test_notification_propagation_delay

Purpose: Measure NOTIFY latency

What it tests:

• Time from UPDATE to notification received
• Network + database latency
• LISTEN mechanism performance

Expected behavior:

• Average delay < 100ms
• Consistent propagation timing

---

Category 5: Edge Cases & Failure Modes

Test 22: test_malformed_notification_payload

Purpose: Handle invalid JSON in notification

What it tests:

• Error handling for malformed payloads
• Graceful degradation
• No panic on invalid data

Expected behavior:

• No panic or crash
• Returns None or skips invalid notification
• Continues processing valid notifications

---

Test 24: test_config_update_during_service_restart

Purpose: Verify resilience during service lifecycle

What it tests:

• Config persistence across restarts
• Loader initialization with latest data
• No stale data after restart

Expected behavior:

• New loader sees latest config
• No caching issues
• Immediate consistency

---

Test 25: test_partial_transaction_failure

Purpose: Verify rollback on partial failure

What it tests:

• Transaction rollback on mid-transaction error
• Foreign key constraint enforcement
• All-or-nothing guarantee

Expected behavior:

• Config update rolls back despite success
• Foreign key violation causes full rollback
• Original config unchanged

---

Configuration Lifecycle

1. Initial Load

let loader = DatabaseConfigLoader::new(&database_url).await?;
let config = loader.load_config("default-production").await?;

2. Enable Hot-Reload

loader.enable_hot_reload().await?;

3. Monitor for Changes

loop {
    if let Some(strategy_id) = loader.check_for_updates().await? {
        println!("Configuration changed for: {}", strategy_id);
        let new_config = loader.load_config(&strategy_id).await?;
        // Apply new configuration...
    }
    tokio::time::sleep(Duration::from_secs(1)).await;
}

4. Database Trigger Mechanism

-- Trigger function (from migration 015)
CREATE OR REPLACE FUNCTION notify_adaptive_strategy_config_change()
RETURNS TRIGGER AS $$
DECLARE
    payload JSON;
BEGIN
    payload = json_build_object(
        'table', TG_TABLE_NAME,
        'action', TG_OP,
        'strategy_id', COALESCE(NEW.strategy_id, OLD.strategy_id),
        'timestamp', EXTRACT(EPOCH FROM NOW())
    );
    PERFORM pg_notify('adaptive_strategy_config_change', payload::text);
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

-- Triggers on all 3 tables
CREATE TRIGGER adaptive_strategy_config_notify
AFTER INSERT OR UPDATE OR DELETE ON adaptive_strategy_config
FOR EACH ROW EXECUTE FUNCTION notify_adaptive_strategy_config_change();

Test Helpers

create_loader()

Creates a DatabaseConfigLoader connected to test database.

create_test_pool()

Creates a separate connection pool for test operations (updates, verifications).

wait_for_notification(loader, timeout_secs)

Waits for notification with timeout to prevent test hangs.

cleanup_test_strategy(pool, strategy_id)

Removes test data after test completion (cascade delete).

Common Test Patterns

Pattern 1: Dual Connection Test

// Loader for listening
let mut loader = create_loader().await;
loader.enable_hot_reload().await?;

// Separate pool for updates
let pool = create_test_pool().await;
sqlx::query("UPDATE ...").execute(&pool).await?;

// Wait for notification
let notif = wait_for_notification(&mut loader, 5).await?;

Pattern 2: Transaction Verification

let pool = create_test_pool().await;

let mut tx = pool.begin().await?;
// ... perform operations ...
tx.commit().await?;

// Verify via separate query
let result = sqlx::query(...).fetch_one(&pool).await?;
assert_eq!(result, expected);

Pattern 3: Concurrent Tasks

let task1 = tokio::spawn(async move { /* update 1 */ });
let task2 = tokio::spawn(async move { /* update 2 */ });

let result1 = task1.await??;
let result2 = task2.await??;

// Verify only one succeeded
assert_eq!(result1.rows_affected() + result2.rows_affected(), 1);

Troubleshooting

Test Hangs

• Cause: Notification not received, wait_for_notification() times out
• Fix: Check PostgreSQL logs, verify triggers are installed, ensure migrations ran

Connection Errors

• Cause: DATABASE_URL incorrect or PostgreSQL not running
• Fix: Verify psql $DATABASE_URL works, check PostgreSQL service status

Constraint Violations

• Cause: Test cleanup didn't run from previous test
• Fix: Manually clean test database:

  DELETE FROM adaptive_strategy_config WHERE strategy_id LIKE 'test-%';
  

Version Conflicts

• Cause: Multiple tests updating same config without cleanup
• Fix: Use unique strategy_ids per test, ensure cleanup runs

CI/CD Integration

GitHub Actions Example

jobs:
  test-hot-reload:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: foxhunt_test
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    steps:
      - uses: actions/checkout@v3
      - name: Run migrations
        run: sqlx migrate run
        env:
          DATABASE_URL: postgresql://postgres:postgres@localhost:5432/foxhunt_test

      - name: Run hot-reload tests
        run: cargo test --test hot_reload_integration --features postgres
        env:
          DATABASE_URL: postgresql://postgres:postgres@localhost:5432/foxhunt_test

      - name: Run performance tests
        run: cargo test --test hot_reload_integration --features postgres -- --ignored
        env:
          DATABASE_URL: postgresql://postgres:postgres@localhost:5432/foxhunt_test

Performance Benchmarks

Expected Latencies (p99)

• Config Load: < 100ms
• Notification Propagation: < 100ms
• Transaction Commit: < 50ms

Tested Load

• Concurrent Listeners: 3+ simultaneous services
• Update Frequency: 10 updates/second
• Config Size: Production config with 3-5 models, 4-6 features

Future Enhancements

1. Cache Invalidation Tests:

   • Test ConfigManager cache behavior with hot-reload
   • Verify stale data not served after notification

2. Multi-Database Tests:

   • Test across PostgreSQL read replicas
   • Verify notification propagation in distributed setup

3. Failure Injection:

   • Network partition simulation
   • Database crash recovery
   • Listener process kill/restart

4. Load Testing:

   • 1000+ concurrent updates
   • Notification buffer overflow handling
   • Connection pool exhaustion

References

• Migration: /home/jgrusewski/Work/foxhunt/database/migrations/015_adaptive_strategy_config.sql
• Loader: /home/jgrusewski/Work/foxhunt/adaptive-strategy/src/database_loader.rs
• Config Types: /home/jgrusewski/Work/foxhunt/adaptive-strategy/src/config_types.rs
• Existing Tests: /home/jgrusewski/Work/foxhunt/adaptive-strategy/tests/database_config_integration.rs

---

Last Updated: 2025-10-03 Test Suite Version: 1.0 Total Tests: 25 (17 active, 2 ignored performance tests, 6 hot-reload core tests)