foxhunt/docs/archive/wave_d/reports/THRASHING_PREVENTION_STRATEGY.md

Thrashing Prevention Strategy

Version: 1.0 Date: 2025-10-23 Status: APPROVED - Ready for Implementation Estimated Rollout: 4 weeks (incremental)

---

Executive Summary

This document defines a comprehensive strategy to eliminate recurring technical thrashing in the Foxhunt trading system. Analysis of 3 major thrashing incidents (QAT 3x fixes, database migrations 4x attempts, clippy 40-min estimate → 1-2 weeks actual) reveals a systemic pattern: symptom-driven fixes without root cause analysis, combined with testing pyramid inversion (99.4% unit test coverage masking integration failures).

Solution: Implement 6 mandatory quality gates + 1 advisory gate focused on integration validation rather than isolated unit testing. Expected outcomes: <5% PR thrashing rate (down from current ~15%), >80% integration test coverage, >95% documentation-code sync.

---

Part 1: Root Cause Analysis

1.1 Thrashing Pattern Evidence

QAT Implementation (3 Fix Cycles)

Cycle 1: QAT-01 to QAT-12 (12 agents) - Initial implementation
Cycle 2: Test fixes (4 agents) - Fixed 97 test errors
Cycle 3: Benchmark fixes (4 agents) - Fixed 18 benchmark errors
Cycle 4: GPU validation - Discovered device mismatch, memory bugs (STILL PENDING)

Pattern: Implementation → Unit tests pass → Integration failures discovered → Fix cycle repeats

Database Migrations (4 Attempts)

Attempt 1: Migration 045 created
Attempt 2: Migration 046 conflict discovered
Attempt 3: Hard migration to remove 046
Attempt 4: Wave 10 SQLX offline mode conflicts

Pattern: Schema changes → CI breaks → Manual fixes → Repeat

Clippy Fixes (Optimistic Estimates)

Claimed: "40-minute fix path" (Phase 0: 10 min, Phase 1: 30 min)
Reality: Phase 0 + Phase 1 → 380 warnings remaining → Phase 2: 1-2 weeks (NOT STARTED)

Pattern: Optimistic estimates → Partial fixes → Technical debt accumulation

1.2 Core Root Causes

Root Cause	Evidence	Impact
1. Symptom-Driven Development	QAT device mismatch fixed 3 times without understanding GPU memory model	3x wasted effort, production blockers remain
2. Testing Pyramid Inversion	2,086 unit tests (99.4% pass rate) but integration gaps exist (Adaptive Position Sizer claimed in CLAUDE.md, not wired in code)	False confidence in system stability
3. Documentation-Code Divergence	CLAUDE.md promises "Migration 045 operational" but Wave 10 discovered SQLX conflicts	Misleading status reporting
4. No Pre-Merge Integration Validation	Database migrations not tested in SQLX offline mode before merge	CI breaks in production
5. Time Pressure Culture	"Quick fix" mentality leading to incomplete solutions	Technical debt compounds

1.3 Key Insight

The system has 99.4% unit test pass rate but still has production blockers.

This indicates:

• Testing the wrong things: Isolated component behavior (unit tests)
• Not testing the right things: Component interaction (integration tests)

Solution: Shift from "tests passing" metric to "integration validated" metric.

---

Part 2: Quality Gate Framework

2.1 Mandatory Gates (6)

Gate 1: Root Cause Documentation

Trigger (Risk-Based):

• Any PR modifying files in >2 architectural boundaries (e.g., trading_engine/ + database/)
• Any database migration file change
• Any public gRPC service definition change
• Fallback: >100 lines of code changed (if above rules don't apply)

Requirement: ROOT_CAUSE.md file in PR with:

## What Broke?
[Describe the symptom - what was the visible failure?]

## Why Did It Break?
[Explain the mechanism - what caused the failure at a technical level?]

## Why Didn't Existing Tests Catch It?
[Identify the testing gap that allowed this to reach production/CI]

## What Prevents Recurrence?
[Describe the systemic fix - not just the code change, but process improvements]

Enforcement: GitHub PR template validation (script: scripts/check_root_cause.sh)

---

Gate 2: Integration Test Coverage

Trigger: Any cross-component change

Requirements by Component:

Database Changes (migrations/):

# Required tests
test_migration_up_down_cycle()       # Apply + rollback
test_sqlx_offline_mode_build()       # CI compatibility
test_query_performance()             # <10ms for trading queries
test_foreign_key_constraints()       # Data integrity

ML Model Changes (ml/):

#[test]
fn test_end_to_end_inference_pipeline() {
    // Load DBN data → Preprocess → Model inference → Validate output shape
}

#[test]
fn test_225_feature_extraction_integration() {
    // Market data → 225-feature pipeline → Model input
}

#[test]
fn test_gpu_memory_budget() {
    // Load all 4 models → Verify <4GB total
}

gRPC Service Changes (services/):

#[tokio::test]
async fn test_gateway_to_trading_service_routing() {
    // Start both services → Submit order via gateway → Verify reaches trading service
}

#[tokio::test]
async fn test_auth_flow_end_to_end() {
    // Login → Get JWT → Use JWT for authenticated request → Verify
}

Trading Flow Changes (trading_agent/, trading_engine/):

#[tokio::test]
async fn test_order_lifecycle_integration() {
    // Universe selection → Asset selection → Position sizing → Order generation →
    // Execution → PnL tracking → Database persistence
}

#[test]
fn test_regime_adaptive_position_sizing() {
    // Detect regime → kelly_criterion_regime_adaptive → Verify 0.2x-1.5x range
}

Enforcement: CI step cargo test --test integration_* must pass

Expert Recommendation: Use transaction-based rollback for 99% of database tests (fast, simple), reserve per-test schemas for DDL-specific tests only.

---

Gate 3: Documentation Sync Validation

Trigger: Any change to CLAUDE.md or major feature completion

Validation Method: Declarative claim verification via docs_validation.yml

Example Configuration:

# docs_validation.yml
claims:
  - feature: "Adaptive Position Sizer"
    file: "CLAUDE.md"
    line: 125
    validation:
      type: integration_test
      name: "test_adaptive_position_sizer_e2e"

  - feature: "Migration 045 operational"
    file: "CLAUDE.md"
    line: 89
    validation:
      type: command
      command: "sqlx migrate list | grep 045"
      expected_exit: 0

  - feature: "QAT infrastructure complete"
    file: "CLAUDE.md"
    line: 234
    validation:
      type: test
      command: "cargo test -p ml test_qat"
      min_passing: 24

Enforcement: scripts/validate_docs.sh runs on every PR, blocks merge if claims unverified

Expert Recommendation: Link claims directly to integration test names, not grep searches. This creates an unbreakable link between documentation and working code.

---

Gate 4: Pre-Merge Smoke Test

Trigger: Every PR before merge

Required Tests (5 steps):

#!/bin/bash
# scripts/smoke_test.sh

# 1. Service health
docker-compose up -d
sleep 10
curl -f http://localhost:8080/health || exit 1

# 2. Database migration
cargo sqlx migrate run || exit 1

# 3. Clean compilation (with SQLX offline mode)
cargo build --workspace --release --offline || exit 1

# 4. Clippy ratcheting (baseline: 380 warnings)
WARNINGS=$(cargo clippy --workspace 2>&1 | grep 'warning:' | wc -l)
if [ $WARNINGS -gt 380 ]; then
  echo "ERROR: Clippy warnings increased from 380 to $WARNINGS"
  exit 1
fi

# 5. Load test
cargo run --bin load_test -- --orders 100 --min-success-rate 95 || exit 1

Enforcement: GitHub Actions workflow .github/workflows/smoke-test.yml

---

Gate 5: Rollback Plan (Production Deployments Only)

Trigger: Any production deployment

Requirement: ROLLBACK.md file with:

## Rollback Command
git revert <commit-hash>
# OR
kubectl rollout undo deployment/trading-service

## Data Migration Rollback
psql -f migrations/rollback_045.sql
# OR
No data migration (feature flag only)

## Feature Flag Disable
redis-cli SET feature:regime_detection false

## Expected Downtime
<5 minutes (blue-green deployment)
# OR
~30 seconds (feature flag toggle)

## Validation Steps
1. Check service health: curl http://localhost:8080/health
2. Verify no new errors: kubectl logs -f trading-service
3. Confirm orders flowing: psql -c "SELECT COUNT(*) FROM orders WHERE created_at > NOW() - INTERVAL '1 minute'"

Enforcement: Production deployment checklist (not automated)

---

Gate 6: Monthly Retrospective (Learning Mechanism)

Trigger: First week of each month

Template: RETROSPECTIVE_TEMPLATE.md

# Monthly Thrashing Retrospective - [Month YYYY]

## Recurring Issues This Month
| Issue | Occurrences | Root Cause | Systemic Fix Needed? |
|-------|-------------|------------|---------------------|
| Example: SQLX offline mode breaks | 2 | Migration script not validated | Yes - Add to CI |

## Time Estimate Accuracy
| Task | Estimated | Actual | Variance | Learning |
|------|-----------|--------|----------|----------|
| Clippy fixes | 40 min | 1-2 weeks | +2000% | Need phase-based estimates |

## Quality Gate Effectiveness
| Gate | Blocked PRs | False Positives | Adjustments Needed? |
|------|-------------|-----------------|---------------------|
| Integration tests | 3 | 0 | No |
| Doc sync validation | 5 | 2 | Yes - Refine claim rules |

## Action Items for Next Month
- [ ] Adjust estimation models based on variance
- [ ] Update quality gate thresholds
- [ ] Add new claim validation rules
- [ ] Schedule training on root cause analysis

Enforcement: None (advisory for continuous improvement)

---

2.2 Advisory Gates (1)

Gate 7: Time Estimate Calibration

Trigger: Every PR (optional)

Process:

1. Developer logs initial time estimate in PR description
2. Developer logs actual time in completion comment
3. Monthly retrospective analyzes variance trends
4. Estimation models adjusted (e.g., "clippy fixes are 20x longer than estimated")

Enforcement: None (learning tool only)

Purpose: Build realistic planning models, avoid optimistic estimates like "40-minute clippy fix"

---

Part 3: Implementation Tooling

3.1 Script: scripts/validate_migration.sh

#!/bin/bash
# Validates database migrations before merge

set -e

echo "Step 1: Apply migration in test database..."
docker-compose exec -T postgres psql -U foxhunt -d foxhunt_test -f /migrations/$1

echo "Step 2: Regenerate SQLX offline metadata..."
cargo sqlx prepare --database-url postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt_test

echo "Step 3: Test offline build..."
cargo build --workspace --offline

echo "Step 4: Test migration rollback..."
if [ -f "migrations/rollback_$1" ]; then
  docker-compose exec -T postgres psql -U foxhunt -d foxhunt_test -f /migrations/rollback_$1
fi

echo "✅ Migration validated successfully"

3.2 Script: scripts/validate_docs.sh

#!/bin/bash
# Validates CLAUDE.md claims against code reality

set -e

# Parse docs_validation.yml
while IFS= read -r claim; do
  FEATURE=$(echo "$claim" | yq '.feature')
  TYPE=$(echo "$claim" | yq '.validation.type')

  case $TYPE in
    integration_test)
      TEST_NAME=$(echo "$claim" | yq '.validation.name')
      cargo test --test integration_tests "$TEST_NAME" || {
        echo "❌ Claim '$FEATURE' failed: Test $TEST_NAME not found or failing"
        exit 1
      }
      ;;
    command)
      CMD=$(echo "$claim" | yq '.validation.command')
      eval "$CMD" || {
        echo "❌ Claim '$FEATURE' failed: Command '$CMD' failed"
        exit 1
      }
      ;;
    test)
      CMD=$(echo "$claim" | yq '.validation.command')
      MIN_PASSING=$(echo "$claim" | yq '.validation.min_passing')
      PASSED=$(eval "$CMD" | grep -c 'test result: ok' || echo 0)
      if [ "$PASSED" -lt "$MIN_PASSING" ]; then
        echo "❌ Claim '$FEATURE' failed: Only $PASSED tests passed (expected $MIN_PASSING)"
        exit 1
      fi
      ;;
  esac

  echo "✅ Claim '$FEATURE' validated"
done < <(yq '.claims[]' docs_validation.yml)

echo "✅ All documentation claims validated"

3.3 Script: scripts/check_root_cause.sh

#!/bin/bash
# Enforces ROOT_CAUSE.md for high-risk PRs

set -e

# Get changed files
CHANGED_FILES=$(git diff --name-only origin/main)

# Count architectural boundaries crossed
BOUNDARIES=0
echo "$CHANGED_FILES" | grep -q 'trading_engine/' && BOUNDARIES=$((BOUNDARIES + 1))
echo "$CHANGED_FILES" | grep -q 'database/' && BOUNDARIES=$((BOUNDARIES + 1))
echo "$CHANGED_FILES" | grep -q 'ml/' && BOUNDARIES=$((BOUNDARIES + 1))
echo "$CHANGED_FILES" | grep -q 'services/' && BOUNDARIES=$((BOUNDARIES + 1))

# Check for high-risk changes
HIGH_RISK=false
echo "$CHANGED_FILES" | grep -q 'migrations/' && HIGH_RISK=true
echo "$CHANGED_FILES" | grep -q '\.proto$' && HIGH_RISK=true

# Require ROOT_CAUSE.md if high risk
if [ "$BOUNDARIES" -gt 1 ] || [ "$HIGH_RISK" = true ]; then
  if [ ! -f "ROOT_CAUSE.md" ]; then
    echo "❌ HIGH RISK CHANGE DETECTED"
    echo "This PR crosses $BOUNDARIES architectural boundaries or modifies high-risk files."
    echo "Please create ROOT_CAUSE.md with root cause analysis."
    exit 1
  fi

  # Validate ROOT_CAUSE.md completeness
  grep -q '## What Broke?' ROOT_CAUSE.md || {
    echo "❌ ROOT_CAUSE.md missing 'What Broke?' section"
    exit 1
  }
  grep -q '## Why Did It Break?' ROOT_CAUSE.md || {
    echo "❌ ROOT_CAUSE.md missing 'Why Did It Break?' section"
    exit 1
  }
  grep -q '## Why Didn'\''t Existing Tests Catch It?' ROOT_CAUSE.md || {
    echo "❌ ROOT_CAUSE.md missing testing gap analysis"
    exit 1
  }
  grep -q '## What Prevents Recurrence?' ROOT_CAUSE.md || {
    echo "❌ ROOT_CAUSE.md missing systemic fix documentation"
    exit 1
  }
fi

echo "✅ Root cause documentation check passed"

3.4 GitHub Actions: .github/workflows/pr-validation.yml

name: PR Validation
on: [pull_request]

jobs:
  quality-gates:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3
        with:
          fetch-depth: 0  # Need full history for ROOT_CAUSE check

      - name: Setup Rust
        uses: actions-rs/toolchain@v1
        with:
          toolchain: stable

      - name: Setup Docker
        run: docker-compose up -d

      - name: Check ROOT_CAUSE.md (Gate 1)
        run: scripts/check_root_cause.sh

      - name: Run integration tests (Gate 2)
        run: |
          cargo test --test integration_tests
          cargo test --test ml_integration --features cuda
          cargo test --test service_integration

      - name: Validate documentation sync (Gate 3)
        run: scripts/validate_docs.sh

      - name: Pre-merge smoke test (Gate 4)
        run: scripts/smoke_test.sh

      - name: Post results
        if: always()
        uses: actions/github-script@v6
        with:
          script: |
            const fs = require('fs');
            const results = fs.readFileSync('test_results.txt', 'utf8');
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: '## Quality Gate Results\n\n' + results
            });

---

Part 4: Implementation Roadmap

Phase 1: Immediate Wins (Week 1) - START HERE

Goal: Demonstrate value with lowest-friction gate

Tasks:

1. Create .github/workflows/smoke-test.yml with 5-step validation
2. Create scripts/smoke_test.sh script
3. Run pilot on 3 PRs to demonstrate bug detection
4. Team demo: Show how smoke test caught SQLX offline mode issue

Success Criteria:

• Smoke test catches 1+ real bug in week 1
• Team sees immediate value
• <5 min execution time

Expected Outcome: Buy-in for more comprehensive gates

---

Phase 2: Integration Test Framework (Weeks 2-3)

Goal: Build robust integration test coverage

Week 2 Tasks:

1. Database integration tests (Day 1):
   • Implement transaction-based rollback pattern (expert recommendation)
   • Add test_migration_up_down_cycle() for all migrations
   • Add test_sqlx_offline_mode_build() to CI
2. ML integration tests (Day 2):
   • Add test_end_to_end_inference_pipeline() (DBN → prediction)
   • Add test_225_feature_extraction_integration() (market data → features)
   • Add test_gpu_memory_budget() (validate <4GB total)
3. gRPC integration tests (Day 3):
   • Add test_gateway_to_trading_service_routing()
   • Add test_auth_flow_end_to_end() (login → JWT → authenticated request)
4. Trading flow integration tests (Day 4):
   • Add test_order_lifecycle_integration() (universe → asset → sizing → execution → PnL → persistence)
   • Add test_regime_adaptive_position_sizing() (regime detection → Kelly adjustment)
5. Validate all integration tests (Day 5):
   • Run on clean main branch
   • Fix any failures
   • Document patterns for future tests

Week 3 Tasks:

1. Pilot integration test requirement on 1 component (database module)
2. Collect feedback from developers
3. Refine test templates based on learnings
4. Extend to all components

Success Criteria:

• All 4 component types have integration test examples
• At least 1 real integration bug caught during pilot
• Developer feedback: "This is worth the effort"

---

Phase 3: Documentation & Process (Week 4)

Goal: Operationalize quality gates with tooling and training

Tasks:

1. Create templates (Days 1-2):
   • PULL_REQUEST_TEMPLATE.md with gate checklist
   • RETROSPECTIVE_TEMPLATE.md for monthly learning
   • docs_validation.yml with 5+ initial claim rules
2. Create validation scripts (Day 3):
   • scripts/validate_docs.sh
   • scripts/check_root_cause.sh
   • scripts/validate_migration.sh
3. Documentation updates (Day 4):
   • Update CLAUDE.md with quality gate documentation
   • Create docs/quality_gates/INTEGRATION_TESTING_GUIDE.md
   • Create docs/quality_gates/ROOT_CAUSE_ANALYSIS_GUIDE.md
4. Team training (Day 5):
   • Workshop: "Root Cause Analysis 101"
   • Demo: Using docs_validation.yml for claim verification
   • Q&A: Address concerns about added process overhead

Success Criteria:

• All templates and scripts operational
• Team trained on new process
• First monthly retrospective scheduled

---

Phase 4: Full Rollout & Refinement (Ongoing)

Goal: Make quality gates mandatory, iterate based on data

Month 1 Tasks:

1. Enable all mandatory gates in .github/workflows/pr-validation.yml
2. Run first monthly retrospective (Gate 6)
3. Adjust thresholds based on data (e.g., ROOT_CAUSE.md LOC limit, clippy warning baseline)

Month 2+ Tasks:

1. Track success metrics (see Section 5)
2. Add new claim validation rules to docs_validation.yml as features ship
3. Refine integration test templates based on common patterns
4. Celebrate wins: PRs that demonstrate improved quality

Expert Recommendation: Start with smoke test only in Week 1 to build momentum. Don't roll out all gates at once.

---

Part 5: Success Metrics

Track monthly in retrospectives:

Metric	Target	Measurement Method
Thrashing Rate	<5% of PRs	Count PRs requiring 3+ fix attempts after merge
Integration Test Coverage	>80% of PRs	Count PRs with new integration tests / total PRs
Documentation-Code Sync	>95%	scripts/validate_docs.sh success rate
Estimate Accuracy	0.8-1.2 range	Actual time / Estimated time (from Gate 7 data)
Production Incidents	<2 per month	Count post-deployment bugs requiring hotfixes

Leading Indicator: Integration test coverage (predicts thrashing rate) Lagging Indicator: Production incidents (validates effectiveness)

---

Part 6: Enforcement Philosophy

What to Automate (Mandatory)

• ✅ Smoke tests (Gate 4) - CI blocks merge if failed
• ✅ Integration tests (Gate 2) - CI blocks merge if missing
• ✅ Documentation sync (Gate 3) - CI blocks merge if claims unverified
• ✅ ROOT_CAUSE.md check (Gate 1) - CI blocks merge for high-risk PRs

What to Guide (Advisory)

• 📋 Time estimate calibration (Gate 7) - No blocking, just data collection
• 📋 Monthly retrospective (Gate 6) - Scheduled, not enforced

What to Keep Flexible

• 🔧 ROOT_CAUSE.md LOC threshold (currently 100, adjust based on data)
• 🔧 Clippy warning baseline (currently 380, should decrease over time via ratcheting)
• 🔧 Integration test templates (evolve as patterns emerge)

Principle: Automate correctness checks, guide learning processes, keep thresholds flexible.

---

Part 7: Rollback & Emergency Override

When to Override Gates

1. P0 Production Incident: Security vulnerability or trading system down
2. Critical Hotfix: Must deploy immediately to prevent financial loss
3. False Positive: Gate incorrectly blocks a valid change

Override Process

# Emergency merge (requires 2 approvals)
git commit -m "OVERRIDE: [REASON] - Original PR #123"
git push --force-with-lease

# Post-incident review
# 1. Document in ROOT_CAUSE.md why override was necessary
# 2. Fix the false positive in next sprint
# 3. Add test case to prevent future false positives

Important: Every override MUST have a post-incident review within 48 hours.

---

Part 8: Expert Recommendations (Incorporated)

Based on expert validation, the following refinements were made:

8.1 Database Testing Strategy

Original: Per-test schemas for all tests Refined: Transaction-based rollback for 99% of tests (faster, simpler), per-test schemas only for DDL-specific tests

Rationale: Transaction rollback is the standard approach (Rails, Django) with significantly faster execution. Provides adequate isolation for non-DDL tests.

8.2 ROOT_CAUSE.md Trigger

Original: >100 LOC changed Refined: Risk-based triggers (crosses >2 architectural boundaries, modifies migrations, changes gRPC definitions)

Rationale: A 200-line refactor in a single function is less risky than a 50-line change touching database + trading engine + gRPC. Risk-based triggers align with actual failure modes.

8.3 Documentation Validation

Original: grep for code patterns Refined: Link claims directly to integration test names in docs_validation.yml

Rationale: Creates an unbreakable link between documentation claims and working code. If test passes, claim is verified. If test fails, claim is invalidated.

Example:

claims:
  - feature: "Adaptive Position Sizer"
    validation:
      type: integration_test
      name: "test_adaptive_position_sizer_e2e"  # Must exist and pass

8.4 Rollout Strategy

Original: 4-week big-bang rollout Refined: Incremental rollout starting with highest-value, lowest-friction gate (smoke test in Week 1)

Rationale: Demonstrates immediate value, builds team buy-in, reduces risk of process rejection.

8.5 Clippy Ratcheting Implementation

Original: Simple grep 'warning:' | wc -l Refined: Store baseline as CI artifact, compare on every run

Implementation:

# In CI
CURRENT_WARNINGS=$(cargo clippy --workspace 2>&1 | grep 'warning:' | wc -l)
BASELINE=$(cat clippy_baseline.txt || echo 380)

if [ "$CURRENT_WARNINGS" -gt "$BASELINE" ]; then
  echo "❌ Clippy warnings increased: $BASELINE → $CURRENT_WARNINGS"
  exit 1
elif [ "$CURRENT_WARNINGS" -lt "$BASELINE" ]; then
  echo "✅ Clippy warnings reduced: $BASELINE → $CURRENT_WARNINGS"
  echo "$CURRENT_WARNINGS" > clippy_baseline.txt
fi

Bonus: Add social incentive - "PRs that reduce warnings by 5+ get a ⭐"

---

Part 9: Adoption Risks & Mitigations

Risk	Probability	Impact	Mitigation
Developer resistance ("too much process")	High	High	Start with smoke test only (Week 1), demonstrate bug detection, iterate based on feedback
False positives (gates block valid changes)	Medium	Medium	Provide emergency override process, fix false positives within 48 hours, adjust thresholds
Slow CI times (integration tests take >10 min)	Medium	Medium	Parallelize tests, use transaction rollback (faster than per-test schemas), optimize database fixtures
Documentation validation drift (claims go stale)	Medium	Low	Monthly retrospective reviews claim accuracy, automated validation catches 95%+
Time estimate gaming (developers pad estimates)	Low	Low	Keep Gate 7 advisory (no punishment for inaccuracy), focus on learning not blame

Key Success Factor: Start small, demonstrate value, iterate based on data.

---

Part 10: Measuring Success

Week 1 (Smoke Test Only)

• Metric: Bugs caught by smoke test
• Target: 1+ real bug detected
• Action: If target met, proceed to Week 2. If not, refine smoke test.

Month 1 (All Gates Enabled)

• Metric: Thrashing rate (PRs requiring 3+ fixes)
• Baseline: ~15% (based on QAT, database, clippy examples)
• Target: <10%
• Action: If target met, declare success. If not, retrospective to identify gaps.

Month 3 (Steady State)

• Metric: All 5 success metrics (thrashing rate, integration coverage, doc sync, estimate accuracy, production incidents)
• Target: All targets met
• Action: Continuous improvement via monthly retrospectives

Month 6 (Long-Term)

• Metric: Developer sentiment ("Is this process worth it?")
• Target: >80% positive feedback
• Action: If negative, simplify process. If positive, evangelize to other teams.

---

Appendix A: PR Review Checklist

(To be added to PULL_REQUEST_TEMPLATE.md)

## Change Description
- [ ] What changed? (1-2 sentences)
- [ ] Why this change? (business/technical justification)

## Root Cause Analysis (Required for high-risk changes)
- [ ] `ROOT_CAUSE.md` file included (if crosses >2 boundaries, modifies migrations, or changes gRPC)
- [ ] Root cause documented (not just symptoms)
- [ ] Existing test gap identified
- [ ] Systemic fix implemented (not quick fix)

## Integration Testing
- [ ] Database changes: `scripts/validate_migration.sh` passed
- [ ] ML changes: E2E inference test added (`test_end_to_end_inference_pipeline`)
- [ ] gRPC changes: Client-server integration test added
- [ ] Trading flow changes: Order lifecycle test added

## Documentation Sync
- [ ] CLAUDE.md updated (if applicable)
- [ ] `scripts/validate_docs.sh` passed
- [ ] API documentation updated (if applicable)

## Pre-Merge Validation
- [ ] All unit tests pass (`cargo test --workspace`)
- [ ] All integration tests pass (`cargo test --test integration_*`)
- [ ] Smoke test passed (`scripts/smoke_test.sh`)
- [ ] No new clippy warnings introduced (ratcheting enforced)

## Production Readiness (For production deployments only)
- [ ] `ROLLBACK.md` file included
- [ ] Rollback tested in staging
- [ ] Feature flags configured (if applicable)
- [ ] Monitoring dashboards updated
- [ ] On-call team notified

## Time Estimate Calibration (Advisory)
- Estimated time: ___ hours
- Actual time: ___ hours (to be filled at completion)

---

Appendix B: Monthly Retrospective Template

(To be added to RETROSPECTIVE_TEMPLATE.md)

# Monthly Thrashing Retrospective - [Month YYYY]

## Recurring Issues This Month
| Issue | Occurrences | Root Cause | Systemic Fix Needed? |
|-------|-------------|------------|---------------------|
| Example: SQLX offline mode breaks | 2 | Migration script not validated | Yes - Add to CI |

## Time Estimate Accuracy
| Task | Estimated | Actual | Variance | Learning |
|------|-----------|--------|----------|----------|
| Clippy fixes | 40 min | 1-2 weeks | +2000% | Need phase-based estimates |

## Quality Gate Effectiveness
| Gate | Blocked PRs | False Positives | Adjustments Needed? |
|------|-------------|-----------------|---------------------|
| Integration tests | 3 | 0 | No |
| Doc sync validation | 5 | 2 | Yes - Refine claim rules |

## Success Metrics
| Metric | Target | Actual | Status |
|--------|--------|--------|--------|
| Thrashing rate | <5% | 8% | ⚠️ Needs improvement |
| Integration coverage | >80% | 85% | ✅ On track |
| Doc sync | >95% | 92% | ⚠️ Close |
| Estimate accuracy | 0.8-1.2 | 1.5 | ❌ Over-estimating |
| Production incidents | <2 | 1 | ✅ Excellent |

## Action Items for Next Month
- [ ] Adjust estimation models based on variance (focus on clippy/testing tasks)
- [ ] Update quality gate thresholds (consider raising integration coverage to 85%)
- [ ] Add new claim validation rules (3 new features shipped this month)
- [ ] Schedule training on transaction-based test rollback pattern

---

Conclusion

This strategy addresses the root causes of systemic thrashing by shifting focus from "tests passing" to "integration validated". Key innovations:

1. Risk-based triggers (not arbitrary LOC limits)
2. Integration test focus (transaction rollback for speed)
3. Documentation-code linking (claims tied to test names)
4. Incremental rollout (smoke test first, full gates later)
5. Continuous improvement (monthly retrospectives, flexible thresholds)

Expected Outcomes:

• <5% thrashing rate (down from ~15%)

• 80% integration test coverage

• 95% documentation-code sync

• 0.8-1.2 estimate accuracy
• <2 production incidents per month

Next Steps: Begin Phase 1 (Week 1) - Implement smoke test and demonstrate value.

---

Document Status: APPROVED Implementation Owner: Development Team Review Cadence: Monthly (via Gate 6 retrospectives) Last Updated: 2025-10-23