Files

jgrusewski 7458f1be01 feat(wave12): E2E validation complete - 225-feature pipeline ready

✅ Validation Results:
- PPO training: 24.2s (1 epoch, 950 samples, dim=225)
- Feature extraction: 105μs/bar (9.5x faster than target)
- Model checkpoint: 293KB (147KB actor + 146KB critic)
- GPU memory: 145MB used (96.4% headroom)
- Zero dimension mismatches

📊 Success Criteria (5/5):
✅ Feature dimension = 225 (Wave C 201 + Wave D 24)
✅ Model state_dim = 225
✅ Training completed without errors
✅ Checkpoint saved successfully
✅ No dimension mismatch errors

📁 Training Data Ready:
- ES.FUT: 2.9MB, 180 days
- NQ.FUT: 4.4MB, 180 days
- 6E.FUT: 2.8MB, 180 days
- ZN.FUT: 65KB, 90 days (clean)

🚀 Next: Full production model retraining (4 models, ~10min GPU time)

🤖 Generated with Claude Code (https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

2025-10-22 22:48:04 +02:00

3.7 KiB

Raw Blame History

Database Migration Runbook

Last Updated: 2025-10-22 Purpose: Safe database schema changes RPO: 1 hour (hourly backups)

Pre-Migration Checklist

Migration tested in development environment
Migration is backward-compatible (or multi-step plan documented)
Database backup completed within last hour
Estimated migration time calculated
Rollback migration tested
Application code supports both old and new schema
Low-traffic window scheduled (if long-running migration)

Migration Procedure

Step 1: Backup Database

# Kubernetes
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    pg_dump -U foxhunt -d foxhunt -Fc > /backup/foxhunt_pre_migration_$(date +%Y%m%d_%H%M%S).dump

# Verify backup
ls -lh /backup/foxhunt_pre_migration_*.dump

Step 2: Test Migration (Dry Run)

# Create test database
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d postgres -c "CREATE DATABASE foxhunt_test TEMPLATE foxhunt;"

# Run migration on test database
kubectl exec -i -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt_test < migrations/045_regime_detection.up.sql

# Verify schema
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt_test -c "\dt"

# Measure migration time
time kubectl exec -i -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt_test < migrations/045_regime_detection.up.sql

# Drop test database
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d postgres -c "DROP DATABASE foxhunt_test;"

Step 3: Apply Migration

# Option A: Using sqlx (recommended)
kubectl exec -n foxhunt foxhunt-api-gateway-0 -- sqlx migrate run

# Option B: Manual SQL
kubectl exec -i -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt < migrations/045_regime_detection.up.sql

# Verify migration applied
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt -c "SELECT version, description, success FROM _sqlx_migrations ORDER BY version DESC LIMIT 5;"

Step 4: Validate Migration

# Check table structure
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt -c "\d regime_states"

# Test queries
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt -c "SELECT count(*) FROM regime_states;"

# Verify indexes created
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt -c "SELECT indexname, indexdef FROM pg_indexes WHERE tablename='regime_states';"

Step 5: Rollback (if needed)

# Option A: Restore from backup
kubectl exec -i -n foxhunt foxhunt-postgresql-0 -- \
    pg_restore -U foxhunt -d foxhunt -c /backup/foxhunt_pre_migration_20251022_100000.dump

# Option B: Run down migration
kubectl exec -i -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt < migrations/045_regime_detection.down.sql

Long-Running Migrations

For migrations expected to take >5 minutes:

Use `pt-online-schema-change` (for large tables)

# Install percona-toolkit
kubectl exec -n foxhunt foxhunt-postgresql-0 -- apt-get update && apt-get install -y percona-toolkit

# Run online schema change (creates shadow table, migrates data, swaps tables)
pt-online-schema-change \
    --alter "ADD COLUMN new_field TEXT" \
    --execute \
    h=localhost,D=foxhunt,t=orders,u=foxhunt,p=${DB_PASSWORD}

Background Index Creation

-- Instead of:
CREATE INDEX idx_orders_symbol ON orders(symbol);

-- Use:
CREATE INDEX CONCURRENTLY idx_orders_symbol ON orders(symbol);
-- Allows concurrent writes, takes longer but no downtime

End of Database Migration Runbook

3.7 KiB Raw Blame History