7458f1be01
✅ Validation Results: - PPO training: 24.2s (1 epoch, 950 samples, dim=225) - Feature extraction: 105μs/bar (9.5x faster than target) - Model checkpoint: 293KB (147KB actor + 146KB critic) - GPU memory: 145MB used (96.4% headroom) - Zero dimension mismatches 📊 Success Criteria (5/5): ✅ Feature dimension = 225 (Wave C 201 + Wave D 24) ✅ Model state_dim = 225 ✅ Training completed without errors ✅ Checkpoint saved successfully ✅ No dimension mismatch errors 📁 Training Data Ready: - ES.FUT: 2.9MB, 180 days - NQ.FUT: 4.4MB, 180 days - 6E.FUT: 2.8MB, 180 days - ZN.FUT: 65KB, 90 days (clean) 🚀 Next: Full production model retraining (4 models, ~10min GPU time) 🤖 Generated with Claude Code (https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
134 lines
3.7 KiB
Markdown
134 lines
3.7 KiB
Markdown
# Database Migration Runbook
|
|
|
|
**Last Updated**: 2025-10-22
|
|
**Purpose**: Safe database schema changes
|
|
**RPO**: 1 hour (hourly backups)
|
|
|
|
---
|
|
|
|
## Pre-Migration Checklist
|
|
|
|
- [ ] Migration tested in development environment
|
|
- [ ] Migration is backward-compatible (or multi-step plan documented)
|
|
- [ ] Database backup completed within last hour
|
|
- [ ] Estimated migration time calculated
|
|
- [ ] Rollback migration tested
|
|
- [ ] Application code supports both old and new schema
|
|
- [ ] Low-traffic window scheduled (if long-running migration)
|
|
|
|
---
|
|
|
|
## Migration Procedure
|
|
|
|
### Step 1: Backup Database
|
|
|
|
```bash
|
|
# Kubernetes
|
|
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
|
|
pg_dump -U foxhunt -d foxhunt -Fc > /backup/foxhunt_pre_migration_$(date +%Y%m%d_%H%M%S).dump
|
|
|
|
# Verify backup
|
|
ls -lh /backup/foxhunt_pre_migration_*.dump
|
|
```
|
|
|
|
### Step 2: Test Migration (Dry Run)
|
|
|
|
```bash
|
|
# Create test database
|
|
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
|
|
psql -U foxhunt -d postgres -c "CREATE DATABASE foxhunt_test TEMPLATE foxhunt;"
|
|
|
|
# Run migration on test database
|
|
kubectl exec -i -n foxhunt foxhunt-postgresql-0 -- \
|
|
psql -U foxhunt -d foxhunt_test < migrations/045_regime_detection.up.sql
|
|
|
|
# Verify schema
|
|
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
|
|
psql -U foxhunt -d foxhunt_test -c "\dt"
|
|
|
|
# Measure migration time
|
|
time kubectl exec -i -n foxhunt foxhunt-postgresql-0 -- \
|
|
psql -U foxhunt -d foxhunt_test < migrations/045_regime_detection.up.sql
|
|
|
|
# Drop test database
|
|
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
|
|
psql -U foxhunt -d postgres -c "DROP DATABASE foxhunt_test;"
|
|
```
|
|
|
|
### Step 3: Apply Migration
|
|
|
|
```bash
|
|
# Option A: Using sqlx (recommended)
|
|
kubectl exec -n foxhunt foxhunt-api-gateway-0 -- sqlx migrate run
|
|
|
|
# Option B: Manual SQL
|
|
kubectl exec -i -n foxhunt foxhunt-postgresql-0 -- \
|
|
psql -U foxhunt -d foxhunt < migrations/045_regime_detection.up.sql
|
|
|
|
# Verify migration applied
|
|
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
|
|
psql -U foxhunt -d foxhunt -c "SELECT version, description, success FROM _sqlx_migrations ORDER BY version DESC LIMIT 5;"
|
|
```
|
|
|
|
### Step 4: Validate Migration
|
|
|
|
```bash
|
|
# Check table structure
|
|
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
|
|
psql -U foxhunt -d foxhunt -c "\d regime_states"
|
|
|
|
# Test queries
|
|
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
|
|
psql -U foxhunt -d foxhunt -c "SELECT count(*) FROM regime_states;"
|
|
|
|
# Verify indexes created
|
|
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
|
|
psql -U foxhunt -d foxhunt -c "SELECT indexname, indexdef FROM pg_indexes WHERE tablename='regime_states';"
|
|
```
|
|
|
|
### Step 5: Rollback (if needed)
|
|
|
|
```bash
|
|
# Option A: Restore from backup
|
|
kubectl exec -i -n foxhunt foxhunt-postgresql-0 -- \
|
|
pg_restore -U foxhunt -d foxhunt -c /backup/foxhunt_pre_migration_20251022_100000.dump
|
|
|
|
# Option B: Run down migration
|
|
kubectl exec -i -n foxhunt foxhunt-postgresql-0 -- \
|
|
psql -U foxhunt -d foxhunt < migrations/045_regime_detection.down.sql
|
|
```
|
|
|
|
---
|
|
|
|
## Long-Running Migrations
|
|
|
|
For migrations expected to take >5 minutes:
|
|
|
|
### Use `pt-online-schema-change` (for large tables)
|
|
|
|
```bash
|
|
# Install percona-toolkit
|
|
kubectl exec -n foxhunt foxhunt-postgresql-0 -- apt-get update && apt-get install -y percona-toolkit
|
|
|
|
# Run online schema change (creates shadow table, migrates data, swaps tables)
|
|
pt-online-schema-change \
|
|
--alter "ADD COLUMN new_field TEXT" \
|
|
--execute \
|
|
h=localhost,D=foxhunt,t=orders,u=foxhunt,p=${DB_PASSWORD}
|
|
```
|
|
|
|
### Background Index Creation
|
|
|
|
```sql
|
|
-- Instead of:
|
|
CREATE INDEX idx_orders_symbol ON orders(symbol);
|
|
|
|
-- Use:
|
|
CREATE INDEX CONCURRENTLY idx_orders_symbol ON orders(symbol);
|
|
-- Allows concurrent writes, takes longer but no downtime
|
|
```
|
|
|
|
---
|
|
|
|
**End of Database Migration Runbook**
|