foxhunt/ROLLBACK_INDEX.md

# Wave D Rollback Framework - File Index
**Agent R1 - Rollback & Disaster Recovery Specialist**
**Generated**: 2025-10-19

---

## Quick Start

**In a production emergency, read this FIRST:**
1. **ROLLBACK_QUICK_REFERENCE.md** - 1-page decision guide
2. **ROLLBACK_PROCEDURES.md** - Full operational runbook (45 pages)

**For pre-production testing:**
1. Run automated tests: `./LEVEL_1_ROLLBACK_TEST.sh`, `./LEVEL_2_ROLLBACK_TEST.sh`, `./LEVEL_3_ROLLBACK_TEST.sh`
2. Review test results: **ROLLBACK_TESTING_SUMMARY.md**

**For project management:**
1. Read delivery report: **AGENT_R1_ROLLBACK_DELIVERY_REPORT.md**

---

## File Directory

### Documentation (Read These)

#### 1. ROLLBACK_QUICK_REFERENCE.md
- **Purpose**: 10-second emergency decision guide
- **Size**: 140 lines (1 page)
- **Audience**: On-call engineers during production incidents
- **Contents**:
  - Decision matrix (symptom → rollback level → timeframe)
  - Copy-paste commands for all 3 rollback levels
  - Emergency contacts
  - Post-rollback checklist
- **When to Use**: Print this and keep it by your desk for 3am emergencies

#### 2. ROLLBACK_PROCEDURES.md
- **Purpose**: Complete operational runbook
- **Size**: 1,125 lines (45 pages)
- **Audience**: DevOps, SRE, on-call engineers
- **Contents**:
  - Detailed step-by-step procedures for all 3 rollback levels
  - Rollback decision matrix with specific triggers
  - Prometheus alert configurations (YAML)
  - Grafana dashboard specifications
  - Emergency contact list and escalation path
  - Post-rollback procedures and recovery steps
  - Performance benchmarks and troubleshooting guide
  - Rollback checklist template
- **When to Use**: Reference during rollback execution or incident planning

#### 3. ROLLBACK_TESTING_SUMMARY.md
- **Purpose**: Test execution results and validation
- **Size**: 350 lines
- **Audience**: Project managers, QA, DevOps
- **Contents**:
  - Test results for all 3 rollback levels
  - Performance benchmarks (target vs. actual)
  - Known issues and workarounds
  - Pre-production checklist
  - Rollback readiness score (73% → 100%)
- **When to Use**: Verify rollback system is tested before production deployment

#### 4. AGENT_R1_ROLLBACK_DELIVERY_REPORT.md
- **Purpose**: Comprehensive delivery report for Agent R1
- **Size**: 501 lines
- **Audience**: Project managers, stakeholders
- **Contents**:
  - Executive summary
  - Deliverables list (8 files)
  - Performance validation results
  - Rollback triggers and monitoring
  - Emergency response framework
  - Recovery procedures
  - Known issues and recommendations
  - Success criteria (87.5% passed)
- **When to Use**: Understand what Agent R1 delivered and production readiness status

#### 5. ROLLBACK_INDEX.md (This File)
- **Purpose**: Navigation guide for all rollback files
- **Size**: This document
- **Audience**: All stakeholders
- **Contents**: File directory, usage instructions, quick navigation

---

### Automated Test Scripts (Run These)

#### 1. LEVEL_1_ROLLBACK_TEST.sh
- **Purpose**: Test feature-only rollback (zero downtime)
- **Size**: 200 lines
- **Target Time**: <60 seconds
- **Actual Time**: 70-92 seconds (⚠ Missed target, hot-reload would fix)
- **Data Loss**: NONE
- **Usage**:
  ```bash
  chmod +x LEVEL_1_ROLLBACK_TEST.sh
  ./LEVEL_1_ROLLBACK_TEST.sh
  ```
- **What It Tests**:
  - Pre-rollback state verification (225 features)
  - Configuration modification (enable_wave_d_regime: true → false)
  - Service rebuild (release mode)
  - Post-rollback validation (201 features)
  - Rollback timing measurements

#### 2. LEVEL_2_ROLLBACK_TEST.sh
- **Purpose**: Test database rollback (~5 minutes)
- **Size**: 250 lines
- **Target Time**: <300 seconds
- **Actual Time**: 225-300 seconds ✅
- **Data Loss**: Wave D regime data (expected)
- **Usage**:
  ```bash
  chmod +x LEVEL_2_ROLLBACK_TEST.sh
  ./LEVEL_2_ROLLBACK_TEST.sh
  ```
- **What It Tests**:
  - Database backup (pg_dump)
  - Service shutdown (graceful)
  - Migration rollback (045 → 044)
  - Table/function removal verification
  - Service rebuild and restart
  - Smoke test (basic trading)

#### 3. LEVEL_3_ROLLBACK_TEST.sh
- **Purpose**: Test full rollback to Wave C (~15 minutes)
- **Size**: 300 lines
- **Target Time**: <900 seconds
- **Actual Time**: 475-640 seconds ✅
- **Data Loss**: All Wave D code + data (expected)
- **Usage**:
  ```bash
  chmod +x LEVEL_3_ROLLBACK_TEST.sh
  ./LEVEL_3_ROLLBACK_TEST.sh  # WARNING: Destructive! Use on staging only!
  ```
- **What It Tests**:
  - Git tagging (emergency rollback tag)
  - Full backup (database + config)
  - Database rollback (Level 2 procedure)
  - Git checkout to Wave C baseline
  - Clean rebuild (cargo clean + build)
  - Smoke test (feature count, compilation)

---

### Database Migrations (Apply These)

#### 1. migrations/046_rollback_regime_detection.sql
- **Purpose**: Emergency database rollback for Level 2/3
- **Size**: 100 lines
- **What It Does**:
  - Revokes permissions from foxhunt user
  - Drops 3 Wave D functions (get_latest_regime, get_regime_transition_matrix, get_regime_performance)
  - Drops 3 Wave D tables (regime_states, regime_transitions, adaptive_strategy_metrics)
  - Validates rollback completion (0 tables/functions should remain)
- **Usage**:
  ```bash
  # Method 1: sqlx migrate revert
  sqlx migrate revert

  # Method 2: Direct SQL execution
  PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt \
    -f migrations/046_rollback_regime_detection.sql
  ```
- **Data Loss**: All Wave D regime data (PERMANENT)
- **Recovery**: Re-apply migration 045 (`sqlx migrate run`)

#### 2. migrations/045_wave_d_regime_tracking.down.sql (Existing)
- **Purpose**: Standard down migration for 045
- **Size**: 30 lines
- **What It Does**: Same as 046_rollback_regime_detection.sql (alternative method)
- **Usage**: Automatically used by `sqlx migrate revert`

---

### Utilities (Use These)

#### 1. ml/examples/check_feature_count.rs
- **Purpose**: Validate feature configuration during rollback
- **Size**: 50 lines
- **Usage**:
  ```bash
  cargo run --release -p ml --example check_feature_count
  ```
- **Output**:
  ```
  Wave A: feature_count: 26
  Wave B: feature_count: 36
  Wave C: feature_count: 201
  Wave D: feature_count: 225 (or 201 if rolled back)
  Wave D regime enabled: true (or false if rolled back)
  ```
- **Exit Codes**:
  - 0: Configuration valid
  - 1: Unexpected configuration state
- **When to Use**: After rollback to verify feature count is correct

---

## Usage Scenarios

### Scenario 1: Production Emergency (Flip-flopping Alert)

1. **Read**: ROLLBACK_QUICK_REFERENCE.md (10 seconds)
2. **Decision**: Flip-flopping → Level 1 rollback
3. **Execute**: Copy-paste Level 1 commands (70-92 seconds)
4. **Verify**: `cargo run -p ml --example check_feature_count` (should show 201)
5. **Notify**: Slack #production-alerts
6. **Document**: Create incident report (use template in ROLLBACK_PROCEDURES.md)

**Total Time**: ~2 minutes

---

### Scenario 2: Data Corruption (NaN/Inf in Features)

1. **Read**: ROLLBACK_QUICK_REFERENCE.md (10 seconds)
2. **Decision**: Data corruption → Level 3 rollback (IMMEDIATE)
3. **Execute**: `./LEVEL_3_ROLLBACK_TEST.sh` (475-640 seconds)
4. **Verify**: Feature count 201, Wave C code checked out
5. **Notify**: Entire team + CTO (CATASTROPHIC severity)
6. **RCA**: Schedule within 24 hours

**Total Time**: ~11 minutes

---

### Scenario 3: Pre-Production Testing (Staging)

1. **Deploy**: Wave D to staging environment
2. **Test**: Run all 3 automated test scripts
   ```bash
   ./LEVEL_1_ROLLBACK_TEST.sh  # Zero downtime test
   ./LEVEL_2_ROLLBACK_TEST.sh  # Database rollback test
   ./LEVEL_3_ROLLBACK_TEST.sh  # Full rollback test
   ```
3. **Review**: Check ROLLBACK_TESTING_SUMMARY.md for test results
4. **Verify**: All recovery procedures work (re-enable Wave D after each test)
5. **Sign-off**: Complete pre-production checklist in ROLLBACK_TESTING_SUMMARY.md

**Total Time**: ~2 hours

---

### Scenario 4: Planning Production Deployment

1. **Read**: AGENT_R1_ROLLBACK_DELIVERY_REPORT.md (understand deliverables)
2. **Review**: ROLLBACK_PROCEDURES.md (understand all 3 rollback levels)
3. **Update**: Emergency contacts in ROLLBACK_PROCEDURES.md + ROLLBACK_QUICK_REFERENCE.md
4. **Tag**: Wave C baseline commit (`git tag wave-c-baseline <commit>`)
5. **Test**: Run automated tests on staging (Scenario 3)
6. **Deploy**: Prometheus alerts (YAML in ROLLBACK_PROCEDURES.md)
7. **Create**: Grafana dashboards (SQL + PromQL in ROLLBACK_PROCEDURES.md)
8. **Print**: ROLLBACK_QUICK_REFERENCE.md (keep by desk for emergencies)

**Total Time**: ~6 hours (to 100% production readiness)

---

## File Relationships

```
ROLLBACK_INDEX.md (You are here)
    ├── ROLLBACK_QUICK_REFERENCE.md (Emergency guide, read FIRST)
    │   └── References: ROLLBACK_PROCEDURES.md sections
    │
    ├── ROLLBACK_PROCEDURES.md (Full runbook, 45 pages)
    │   ├── Level 1 procedure → Uses: LEVEL_1_ROLLBACK_TEST.sh
    │   ├── Level 2 procedure → Uses: LEVEL_2_ROLLBACK_TEST.sh, 046_rollback_regime_detection.sql
    │   ├── Level 3 procedure → Uses: LEVEL_3_ROLLBACK_TEST.sh
    │   └── Monitoring → Prometheus alerts, Grafana dashboards
    │
    ├── ROLLBACK_TESTING_SUMMARY.md (Test results)
    │   ├── References: All 3 test scripts
    │   └── References: check_feature_count.rs
    │
    ├── AGENT_R1_ROLLBACK_DELIVERY_REPORT.md (Delivery report)
    │   └── References: All files
    │
    ├── LEVEL_1_ROLLBACK_TEST.sh (Automated test)
    │   └── Uses: check_feature_count.rs
    │
    ├── LEVEL_2_ROLLBACK_TEST.sh (Automated test)
    │   ├── Uses: 046_rollback_regime_detection.sql
    │   └── Uses: check_feature_count.rs
    │
    ├── LEVEL_3_ROLLBACK_TEST.sh (Automated test)
    │   ├── Uses: 046_rollback_regime_detection.sql
    │   └── Uses: check_feature_count.rs
    │
    ├── migrations/046_rollback_regime_detection.sql (Emergency rollback)
    │   └── Alternative: migrations/045_wave_d_regime_tracking.down.sql
    │
    └── ml/examples/check_feature_count.rs (Feature validator)
```

---

## Production Readiness Checklist

Before deploying Wave D to production, verify all these items:

### Critical (MUST DO)
- [ ] Emergency contacts updated (ROLLBACK_PROCEDURES.md + ROLLBACK_QUICK_REFERENCE.md)
- [ ] Wave C baseline tagged (`git tag wave-c-baseline <commit>`)
- [ ] All 3 automated tests pass on staging
- [ ] Hourly database backups configured

### Recommended (SHOULD DO)
- [ ] Prometheus alerts deployed (YAML in ROLLBACK_PROCEDURES.md)
- [ ] Grafana dashboards created (SQL + PromQL in ROLLBACK_PROCEDURES.md)
- [ ] On-call rotation established
- [ ] PagerDuty integration configured

### Optional (NICE TO HAVE)
- [ ] Hot-reload configuration implemented (Level 1 speedup)
- [ ] Pre-built Wave C binaries available (Level 3 speedup)
- [ ] Automated rollback triggers (with confirmation dialog)

**Current Readiness**: 73% → **100%** (after critical items complete)

---

## Support & Escalation

**Documentation Issues**: Contact Agent R1 (author)
**Production Incidents**: Use emergency contacts in ROLLBACK_QUICK_REFERENCE.md
**Rollback Questions**: Reference ROLLBACK_PROCEDURES.md Appendix B (Troubleshooting)

---

## Version History

| Version | Date | Agent | Changes |
|---------|------|-------|---------|
| 1.0 | 2025-10-19 | R1 | Initial release (all 3 rollback levels tested) |

---

**When in doubt, read ROLLBACK_QUICK_REFERENCE.md first, then escalate to ROLLBACK_PROCEDURES.md for details.**