foxhunt/AGENT_DB01_ROLLBACK_PROCEDURES.md

# Database Rollback Procedures - Wave D Migration 045

**Agent**: DB-01 (Database Migration Validator)
**Date**: 2025-10-18
**Migration**: 045_wave_d_regime_tracking
**Status**: VALIDATED - PRODUCTION READY

---

## Overview

This document provides comprehensive rollback procedures for Migration 045 (Wave D Regime Tracking), including emergency rollback, partial rollback, and data preservation strategies.

---

## Rollback Levels

### Level 1: Emergency Rollback (Production Issue)
**Time Required**: ~30 seconds
**Data Loss**: All Wave D regime tracking data
**Use When**: Critical production issue requiring immediate rollback

```sql
-- Execute the down migration
\i /home/jgrusewski/Work/foxhunt/migrations/045_wave_d_regime_tracking.down.sql
```

**Verification Steps**:
```sql
-- Verify tables are dropped
SELECT COUNT(*) FROM information_schema.tables 
WHERE table_name IN ('regime_states', 'regime_transitions', 'adaptive_strategy_metrics');
-- Expected: 0

-- Verify functions are dropped
SELECT COUNT(*) FROM information_schema.routines 
WHERE routine_name IN ('get_latest_regime', 'get_regime_transition_matrix', 'get_regime_performance');
-- Expected: 0
```

---

### Level 2: Data Preservation Rollback
**Time Required**: ~5-10 minutes
**Data Loss**: None (data exported before rollback)
**Use When**: Need to preserve data for analysis or future re-migration

```bash
# Step 1: Export data (replace <DATE> with current date in YYYYMMDD format)
pg_dump postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  --table=regime_states \
  --table=regime_transitions \
  --table=adaptive_strategy_metrics \
  --data-only \
  --inserts \
  --file=/tmp/wave_d_backup_<DATE>.sql

# Step 2: Execute rollback
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -f /home/jgrusewski/Work/foxhunt/migrations/045_wave_d_regime_tracking.down.sql

# Step 3: Verify and archive backup
gzip /tmp/wave_d_backup_<DATE>.sql
mv /tmp/wave_d_backup_<DATE>.sql.gz /home/jgrusewski/Work/foxhunt/backups/
```

---

### Level 3: Partial Rollback (Feature-Specific)
**Time Required**: ~1-2 minutes
**Data Loss**: Specific feature data only
**Use When**: Need to disable specific Wave D features while keeping others

#### Disable Regime Transitions Only
```sql
-- Drop transition-related function
DROP FUNCTION IF EXISTS get_regime_transition_matrix(TEXT, INTEGER);

-- Archive and clear transition data
CREATE TABLE regime_transitions_archive AS SELECT * FROM regime_transitions;
TRUNCATE TABLE regime_transitions;
```

#### Disable Adaptive Strategy Metrics Only
```sql
-- Drop performance function
DROP FUNCTION IF EXISTS get_regime_performance(TEXT, INTEGER);

-- Archive and clear metrics data
CREATE TABLE adaptive_strategy_metrics_archive AS SELECT * FROM adaptive_strategy_metrics;
TRUNCATE TABLE adaptive_strategy_metrics;
```

---

## Rollback Decision Matrix

| Situation | Rollback Level | Data Preservation | Downtime |
|-----------|---------------|-------------------|----------|
| Critical production bug | Level 1 | No | <1 min |
| Performance degradation | Level 3 | Yes | <2 min |
| Data integrity issue | Level 2 | Yes | 5-10 min |
| Feature disable request | Level 3 | Yes | <2 min |
| Schema conflict | Level 1 | No | <1 min |

---

## Re-Migration Process

If rollback was executed and you need to re-apply Wave D:

```bash
# Step 1: Ensure rollback is complete
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "
SELECT table_name FROM information_schema.tables 
WHERE table_name IN ('regime_states', 'regime_transitions', 'adaptive_strategy_metrics');
"
# Expected: 0 rows

# Step 2: Re-apply migration (using sqlx)
cargo sqlx migrate run

# Step 3: Verify migration
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "
SELECT version, description, success FROM _sqlx_migrations WHERE version = 45;
"
# Expected: 1 row with success = true

# Step 4: Restore data (if Level 2 rollback was used)
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -f /home/jgrusewski/Work/foxhunt/backups/wave_d_backup_<DATE>.sql
```

---

## Rollback Testing (Pre-Production)

Before production deployment, test rollback procedures:

```bash
# Test Environment Setup
export TEST_DB="postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt_test"

# Test 1: Full Rollback
psql $TEST_DB -f migrations/045_wave_d_regime_tracking.sql
psql $TEST_DB -f migrations/045_wave_d_regime_tracking.down.sql

# Test 2: Rollback with Data
psql $TEST_DB -f migrations/045_wave_d_regime_tracking.sql
psql $TEST_DB -c "INSERT INTO regime_states (symbol, event_timestamp, regime, confidence) VALUES ('TEST', NOW(), 'Normal', 0.8);"
pg_dump $TEST_DB --table=regime_states --data-only --inserts --file=/tmp/test_backup.sql
psql $TEST_DB -f migrations/045_wave_d_regime_tracking.down.sql
psql $TEST_DB -f migrations/045_wave_d_regime_tracking.sql
psql $TEST_DB -f /tmp/test_backup.sql

# Verify data restored
psql $TEST_DB -c "SELECT * FROM regime_states WHERE symbol = 'TEST';"
```

---

## Emergency Contacts

- **DBA On-Call**: [Contact Info]
- **DevOps Lead**: [Contact Info]
- **Wave D Lead**: Agent DB-01
- **Escalation Path**: DBA → DevOps Lead → CTO

---

## Rollback Checklist

### Pre-Rollback
- [ ] Identify rollback level (1, 2, or 3)
- [ ] Notify stakeholders (if Level 1 or 2)
- [ ] Create backup (if Level 2)
- [ ] Document reason for rollback
- [ ] Verify application services can handle missing tables/functions

### During Rollback
- [ ] Stop dependent services (trading_service, backtesting_service, ml_training_service)
- [ ] Execute rollback SQL
- [ ] Verify tables/functions dropped
- [ ] Check for orphaned sequences or constraints
- [ ] Review PostgreSQL logs for errors

### Post-Rollback
- [ ] Restart application services
- [ ] Verify services operational without Wave D features
- [ ] Monitor error logs for 24 hours
- [ ] Document rollback completion
- [ ] Schedule post-mortem (if production rollback)
- [ ] Update migration tracking

---

## Known Issues & Gotchas

1. **Checksum Mismatch**: Migration 045 shows a checksum difference in `sqlx migrate info`. This is expected due to local edits but does not affect functionality.
2. **Dependent Services**: Ensure Trading Service, Backtesting Service, and ML Training Service are stopped before rollback to avoid connection errors.
3. **Partitioned Tables**: Wave D tables are not partitioned. If data volume grows, consider partitioning before re-migration.
4. **Foreign Keys**: No foreign keys reference Wave D tables, so rollback is safe without cascade concerns.
5. **Materialized Views**: No materialized views depend on Wave D tables.

---

## Performance Impact

### Rollback Performance
- Level 1 (Emergency): <1 second
- Level 2 (Data Preservation): 5-10 minutes (depends on data volume)
- Level 3 (Partial): <30 seconds per feature

### Application Impact
- **Zero Downtime**: If application gracefully handles missing tables (recommended)
- **2-5 Minute Downtime**: If services must be restarted
- **Feature Degradation**: Wave D features unavailable, fallback to Wave C (201 features)

---

## Compliance & Audit

All rollbacks must be:
1. Logged in audit_log table (use transaction_audit_events)
2. Documented in change management system
3. Reported to compliance team (if production)
4. Included in monthly incident report

```sql
-- Example audit log entry
INSERT INTO transaction_audit_events (
  event_type, 
  event_data, 
  user_id, 
  event_timestamp
) VALUES (
  'MIGRATION_ROLLBACK',
  '{"migration": "045_wave_d_regime_tracking", "level": 1, "reason": "Critical production issue"}',
  'system',
  NOW()
);
```

---

## Appendix A: Rollback Validation Queries

```sql
-- Verify all Wave D components removed
SELECT 
  'Tables' AS component_type,
  COUNT(*) AS remaining_count
FROM information_schema.tables 
WHERE table_name IN ('regime_states', 'regime_transitions', 'adaptive_strategy_metrics')
UNION ALL
SELECT 
  'Functions' AS component_type,
  COUNT(*) AS remaining_count
FROM information_schema.routines 
WHERE routine_name IN ('get_latest_regime', 'get_regime_transition_matrix', 'get_regime_performance')
UNION ALL
SELECT 
  'Indexes' AS component_type,
  COUNT(*) AS remaining_count
FROM pg_indexes 
WHERE tablename IN ('regime_states', 'regime_transitions', 'adaptive_strategy_metrics');

-- Expected: 0 for all component types
```

---

## Appendix B: Common Rollback Errors

### Error: "relation does not exist"
**Cause**: Application attempting to access Wave D tables after rollback
**Solution**: Restart application services or deploy application version without Wave D features

### Error: "function does not exist"
**Cause**: Application calling Wave D functions after rollback
**Solution**: Ensure application code handles missing functions gracefully

### Error: "permission denied"
**Cause**: Database user lacks DROP privileges
**Solution**: Execute rollback as `foxhunt` user (owner of tables)

### Error: "cannot drop table because other objects depend on it"
**Cause**: Unexpected foreign key or view dependency
**Solution**: Use CASCADE option: `DROP TABLE regime_states CASCADE;`

---

**End of Rollback Procedures**