foxhunt/docs/MODEL_RETRAINING_SOP.md

# Model Retraining Standard Operating Procedure (SOP)

**Last Updated**: 2025-10-14
**Version**: 1.0
**Owner**: ML Training Team

---

## Overview

This document defines the standard operating procedure for quarterly ML model retraining in the Foxhunt HFT trading system. Retraining ensures models adapt to evolving market conditions and maintain competitive edge.

---

## Table of Contents

1. [When to Retrain](#when-to-retrain)
2. [Prerequisites](#prerequisites)
3. [Retraining Schedule](#retraining-schedule)
4. [Execution Procedure](#execution-procedure)
5. [Validation Criteria](#validation-criteria)
6. [Rollout Procedure](#rollout-procedure)
7. [Rollback Procedure](#rollback-procedure)
8. [Checkpoint Management](#checkpoint-management)
9. [Quality Gates](#quality-gates)
10. [Troubleshooting](#troubleshooting)

---

## When to Retrain

### Scheduled Retraining (Quarterly)

Models should be retrained **every quarter** to adapt to market regime changes:

- **Q1**: January 1-15 (retrain with Oct-Dec data)
- **Q2**: April 1-15 (retrain with Jan-Mar data)
- **Q3**: July 1-15 (retrain with Apr-Jun data)
- **Q4**: October 1-15 (retrain with Jul-Sep data)

**Rationale**: 90 days provides sufficient data for statistical significance while staying current with market conditions.

### Ad-Hoc Retraining Triggers

Retrain **immediately** if any of the following occur:

1. **Performance Degradation**
   - Production Sharpe ratio drops below 1.0 for 2+ consecutive weeks
   - Win rate falls below 50% for 1+ week
   - Max drawdown exceeds 20%
   - Daily PnL negative for 5+ consecutive days

2. **Market Regime Shift**
   - VIX spikes >50% in single day
   - Major policy changes (Fed rate changes >0.5%)
   - Black swan events (COVID-19, flash crashes)
   - Sector rotation >20% in single week

3. **Data Distribution Drift**
   - Feature distribution drift >15% (KL divergence)
   - Volume patterns change >25%
   - Volatility regime shift (Hurst exponent change >0.2)

4. **Model Staleness**
   - Model age >6 months without retraining
   - Training data >12 months old

---

## Prerequisites

### Infrastructure Requirements

- [ ] PostgreSQL database accessible (port 5432)
- [ ] Redis cache accessible (port 6379)
- [ ] MinIO/S3 storage accessible
- [ ] GPU available (RTX 3050 Ti or better, 4GB+ VRAM)
- [ ] Disk space: 50GB+ free for training data + checkpoints
- [ ] Memory: 16GB+ RAM recommended

### Data Requirements

- [ ] 90 days of OHLCV-1m data (ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT)
- [ ] Data quality validated (no gaps >5 bars)
- [ ] Price anomalies corrected (spike detection applied)
- [ ] Volume data normalized
- [ ] Estimated bars: ~180,000 (90 days × 4 symbols × ~500 bars/day)

### Software Requirements

- [ ] Rust 1.70+ installed
- [ ] CUDA 12.0+ installed (for GPU training)
- [ ] Docker Compose running (for infrastructure)
- [ ] TLI CLI configured and authenticated

### Access Requirements

- [ ] Production database read access
- [ ] MinIO write access (for checkpoints)
- [ ] Model registry write access
- [ ] Staging environment deployment permissions

---

## Retraining Schedule

### Timeline (Quarterly)

| Week | Activity | Duration | Responsible |
|------|----------|----------|-------------|
| **Week 1** | Data acquisition & validation | 2-3 days | Data Engineering |
| **Week 2** | Training execution (all models) | 4-6 days | ML Training Service |
| **Week 3** | Validation & quality gates | 3-5 days | ML Engineering |
| **Week 4** | Staging deployment & monitoring | 7-10 days | DevOps + Trading |
| **Week 5-6** | Production rollout (gradual) | 10-14 days | Trading Operations |

**Total**: ~6 weeks from data acquisition to full production rollout

### Resource Allocation

- **GPU hours**: 150-200 hours (DQN: 30h, PPO: 40h, MAMBA-2: 80h, TFT: 40h)
- **Engineer time**: 40-60 hours (1-1.5 FTE weeks)
- **Compute cost**: $0 (local RTX 3050 Ti) or $50-75 (cloud A100)

---

## Execution Procedure

### Step 1: Prepare Environment

```bash
# 1. Navigate to project directory
cd /home/jgrusewski/Work/foxhunt

# 2. Start infrastructure
docker-compose up -d

# 3. Verify services healthy
docker-compose ps
grpc_health_probe -addr=localhost:50051  # API Gateway
grpc_health_probe -addr=localhost:50054  # ML Training Service

# 4. Check GPU availability
nvidia-smi
nvcc --version
```

### Step 2: Acquire Training Data

```bash
# Option A: Download latest 90 days from Databento
cargo run -p ml --example download_training_data --release -- \
  --symbols ES.FUT,NQ.FUT,ZN.FUT,6E.FUT \
  --start-date 2024-07-01 \
  --end-date 2024-10-01 \
  --output-dir test_data/real/databento/ml_training

# Option B: Use existing data (verify coverage)
./verify_dataset_coverage.sh test_data/real/databento/ml_training
```

**Expected Output**:
- 90 files (4 symbols × ~22 trading days/month × 3 months)
- Total size: ~2-5GB
- Bars per file: 400-500 (1-minute OHLCV for 6.5-hour trading day)

### Step 3: Validate Hyperparameters

```bash
# Review best hyperparameters from previous tuning
cat ml/config/best_hyperparameters.yaml

# If tuning needed (optional, adds 8-12 hours):
tli tune start --model DQN --trials 50 --watch
tli tune best --job-id <uuid> > ml/config/best_hyperparameters.yaml
```

### Step 4: Execute Retraining

```bash
# Full retraining (all models, sequential)
cargo run -p ml --example retrain_all_models --release --features cuda -- \
  --models DQN,PPO,MAMBA2,TFT \
  --data-dir test_data/real/databento/ml_training \
  --output-dir ml/trained_models/quarterly/2024Q4 \
  --latest-days 90 \
  --min-sharpe 1.5 \
  --min-win-rate 0.55 \
  --version-tag 2024Q4_v1

# Monitor progress (separate terminal)
tail -f ml/trained_models/quarterly/2024Q4/training.log

# Expected duration (sequential):
# - DQN: 6-8 hours
# - PPO: 8-12 hours
# - MAMBA-2: 20-30 hours
# - TFT: 10-15 hours
# Total: 44-65 hours (2-3 days on RTX 3050 Ti)
```

**Parallel Training** (if GPU memory permits):

```bash
# ⚠️ WARNING: May cause OOM on RTX 3050 Ti (4GB VRAM)
cargo run -p ml --example retrain_all_models --release --features cuda -- \
  --models DQN,PPO,MAMBA2,TFT \
  --parallel
```

### Step 5: Review Results

```bash
# View summary report
cat ml/trained_models/quarterly/2024Q4/retraining_summary_2024Q4_v1.json | jq

# Check quality gates
grep "quality_gate_passed" ml/trained_models/quarterly/2024Q4/retraining_summary_2024Q4_v1.json
```

**Expected Output**:
```json
{
  "run_id": "abc-123-def",
  "models_attempted": 4,
  "models_succeeded": 4,
  "models_passed_quality_gate": 3,
  "results": [
    {
      "model_type": "DQN",
      "quality_gate_passed": true,
      "validation_metrics": {
        "sharpe_ratio": 1.8,
        "win_rate": 0.58,
        "max_drawdown": 0.15
      }
    },
    ...
  ]
}
```

---

## Validation Criteria

### Quality Gates

All models must pass **ALL** of the following criteria before staging deployment:

| Metric | Threshold | Severity |
|--------|-----------|----------|
| **Sharpe Ratio** | ≥ 1.5 | CRITICAL |
| **Win Rate** | ≥ 55% | CRITICAL |
| **Max Drawdown** | ≤ 25% | CRITICAL |
| **Total Trades** | ≥ 100 | WARNING |
| **Profit Factor** | ≥ 1.3 | WARNING |
| **Training Convergence** | True | CRITICAL |

**CRITICAL**: Model deployment blocked if failed
**WARNING**: Review required, deployment decision by trading team

### Manual Review Checklist

Before approving staging deployment:

- [ ] Review training loss curves (converged, not overfitting)
- [ ] Inspect sample predictions (reasonable, not degenerate)
- [ ] Verify checkpoint file integrity (SHA-256 checksum)
- [ ] Compare metrics with previous production model
- [ ] Check for data leakage (train/test split properly isolated)
- [ ] Validate hyperparameters match configuration
- [ ] Confirm training data covers expected date range
- [ ] Review any quality gate failures with trading team

---

## Rollout Procedure

### Phase 1: Staging Deployment (Week 4)

```bash
# 1. Register new model versions in model registry
tli model register \
  --model-type DQN \
  --version 2024Q4_v1 \
  --checkpoint ml/trained_models/quarterly/2024Q4/dqn_2024Q4_v1_final.safetensors \
  --status experimental

# 2. Deploy to staging environment
kubectl apply -f k8s/staging/ml-deployment-2024Q4.yaml

# 3. Run integration tests
cargo test -p integration_tests --test staging_ml_models

# 4. Monitor staging for 7-10 days
# - Check Grafana dashboard: http://localhost:3000/d/staging-ml
# - Review prediction distribution
# - Verify latency <100ms P99
# - Monitor memory usage
```

### Phase 2: Production Canary (Week 5)

```bash
# 1. Route 10% traffic to new models
tli deploy canary \
  --models DQN:2024Q4_v1,PPO:2024Q4_v1 \
  --traffic-percentage 10 \
  --duration-hours 48

# 2. Monitor canary metrics (48 hours)
# Key metrics to watch:
# - Sharpe ratio: Should match validation (±0.2)
# - Win rate: Should be ≥55%
# - Latency: Should be <100ms P99
# - Error rate: Should be <0.1%

# 3. If metrics good, increase to 25%
tli deploy canary --traffic-percentage 25 --duration-hours 48

# 4. If metrics still good, increase to 50%
tli deploy canary --traffic-percentage 50 --duration-hours 72
```

### Phase 3: Full Production Rollout (Week 6)

```bash
# 1. Promote to production (100% traffic)
tli model promote \
  --model-type DQN \
  --version 2024Q4_v1 \
  --status production

# 2. Mark old models as archived
tli model archive \
  --model-type DQN \
  --version 2024Q3_v1

# 3. Verify production metrics (24 hours)
# - Sharpe ratio: ≥1.5
# - Win rate: ≥55%
# - PnL: Positive cumulative
# - Max drawdown: ≤25%

# 4. Document deployment
# - Update PRODUCTION_MODELS.md
# - Log deployment in audit trail
# - Notify trading team
```

---

## Rollback Procedure

### When to Rollback

Initiate **immediate rollback** if any occur within first 48 hours:

- Production Sharpe ratio <0.5
- Win rate <45%
- Max drawdown >30%
- Daily PnL negative for 2+ consecutive days
- Model errors/crashes >1%
- Prediction latency >500ms P99

### Rollback Steps

```bash
# 1. Revert traffic to previous production model
tli model rollback \
  --model-type DQN \
  --to-version 2024Q3_v1

# 2. Verify old model restored
grpc_health_probe -addr=localhost:50054

# 3. Mark failed model as archived
tli model archive \
  --model-type DQN \
  --version 2024Q4_v1 \
  --reason "Failed production validation - Sharpe 0.4 < 1.5 threshold"

# 4. Investigate root cause
# - Review training logs
# - Check data quality
# - Validate hyperparameters
# - Analyze prediction distribution

# 5. Document incident
# - Create postmortem report
# - Update SOP if needed
# - Plan corrective action
```

---

## Checkpoint Management

### Naming Convention

```
{model}_{version}_epoch{N}_{timestamp}.safetensors

Examples:
- dqn_2024Q4_v1_epoch150_20250115_143022.safetensors
- ppo_2024Q4_v1_final_20250116_093045.safetensors
```

### Metadata Format

Each checkpoint includes JSON metadata:

```json
{
  "model_type": "DQN",
  "version": "2024Q4_v1",
  "parent_checkpoint": "dqn_2024Q3_v1_final.safetensors",
  "training_date": "2025-01-15T14:30:22Z",
  "data_range": {
    "start_date": "2024-10-01",
    "end_date": "2024-12-31",
    "total_bars": 180000,
    "symbols": ["ES.FUT", "NQ.FUT", "ZN.FUT", "6E.FUT"]
  },
  "hyperparameters": {
    "learning_rate": 0.0001,
    "batch_size": 128,
    "epochs": 200,
    "gamma": 0.99
  },
  "training_metrics": {
    "final_loss": 0.001234,
    "best_epoch": 187,
    "training_time_seconds": 28800
  },
  "validation_metrics": {
    "sharpe_ratio": 1.8,
    "win_rate": 0.58,
    "max_drawdown": 0.15,
    "total_trades": 250
  },
  "quality_gate_passed": true,
  "checksum": "sha256:abc123def456..."
}
```

### Retention Policy

- **Production checkpoints**: Keep last 3 versions (9 months)
- **Staging checkpoints**: Keep last 2 versions (6 months)
- **Experimental checkpoints**: Keep last 1 version (3 months)
- **Failed checkpoints**: Delete after 30 days (archive metadata)

### Storage Locations

```
ml/trained_models/
├── production/        # Active production models
│   ├── dqn/
│   ├── ppo/
│   ├── mamba2/
│   └── tft/
├── staging/           # Models in staging validation
│   └── 2024Q4/
├── quarterly/         # Quarterly retraining outputs
│   ├── 2024Q1/
│   ├── 2024Q2/
│   ├── 2024Q3/
│   └── 2024Q4/
└── archived/          # Old models (metadata only)
    └── 2023Q4/
```

---

## Quality Gates

### Implementation

Quality gates are enforced automatically by the retraining pipeline:

```rust
// ml/examples/retrain_all_models.rs
fn apply_quality_gates(
    metrics: &ValidationMetricsSnapshot,
    gates: &QualityGates,
) -> (bool, Vec<String>) {
    let mut failures = Vec::new();

    if metrics.sharpe_ratio < gates.min_sharpe {
        failures.push(format!(
            "Sharpe ratio {:.2} < {:.2}",
            metrics.sharpe_ratio, gates.min_sharpe
        ));
    }

    // ... other checks

    (failures.is_empty(), failures)
}
```

### Override Procedure

If quality gates fail but trading team approves deployment:

```bash
# 1. Document override reason
tli model override-quality-gate \
  --model-type DQN \
  --version 2024Q4_v1 \
  --reason "Market regime shift - lower Sharpe acceptable" \
  --approved-by "John Doe (Trading Lead)" \
  --approval-ticket JIRA-12345

# 2. Deploy with override flag
tli deploy --force --override-quality-gate

# 3. Monitor closely (daily reviews for 2 weeks)
```

⚠️ **WARNING**: Quality gate overrides require:
- Director-level approval
- Documented business justification
- Enhanced monitoring (daily reviews)
- Automatic rollback if performance degrades further

---

## Troubleshooting

### Common Issues

#### 1. GPU Out of Memory (OOM)

**Symptoms**: Training crashes with "CUDA out of memory" error

**Solutions**:
```bash
# Option A: Reduce batch size
--batch-size 64  # Instead of 128

# Option B: Use CPU (slower, but works)
export CUDA_VISIBLE_DEVICES=""

# Option C: Train models sequentially (default)
# Do NOT use --parallel flag

# Option D: Use gradient accumulation
--gradient-accumulation-steps 2
```

#### 2. Data Loading Fails

**Symptoms**: "No DBN files found" or "Failed to decode record"

**Solutions**:
```bash
# Verify data directory
ls -lh test_data/real/databento/ml_training/*.dbn

# Check file permissions
chmod 644 test_data/real/databento/ml_training/*.dbn

# Validate DBN files
cargo run -p ml --example test_dbn_loading -- \
  --file test_data/real/databento/ml_training/ES.FUT_ohlcv-1m_2024-01-02.dbn
```

#### 3. Training Not Converging

**Symptoms**: Loss plateaus at high value (>10.0), accuracy <50%

**Solutions**:
```bash
# Option A: Adjust learning rate
--learning-rate 0.0005  # Increase from 0.0001

# Option B: Increase epochs
--epochs 300  # Instead of 200

# Option C: Check data quality
./verify_dataset_coverage.sh test_data/real/databento/ml_training

# Option D: Review hyperparameters
cat ml/config/best_hyperparameters.yaml
```

#### 4. Quality Gate Failures

**Symptoms**: Model trained successfully but Sharpe <1.5 or Win Rate <55%

**Analysis**:
```bash
# Run comprehensive backtest analysis
cargo run -p ml --example comprehensive_model_backtest --release -- \
  --checkpoint ml/trained_models/quarterly/2024Q4/dqn_2024Q4_v1_final.safetensors \
  --output-report backtest_analysis.json

# Review failure reasons
cat ml/trained_models/quarterly/2024Q4/retraining_summary_2024Q4_v1.json | \
  jq '.results[] | select(.quality_gate_passed == false) | .quality_gate_failures'
```

**Common Causes**:
- **Insufficient data**: Need 90+ days (not 30-60 days)
- **Data quality issues**: Price spikes, gaps, anomalies
- **Overfitting**: Model memorized training data, poor generalization
- **Hyperparameter mismatch**: Using suboptimal hyperparameters
- **Market regime change**: Training data not representative of current market

**Solutions**:
1. Acquire more/better training data
2. Adjust hyperparameters (run Optuna tuning)
3. Add regularization (dropout, L2 penalty)
4. Use ensemble methods (combine multiple models)
5. Consult with trading team (may need strategy adjustment)

---

## Appendix

### A. Automation Scripts

#### Quarterly Retraining Cron Job

```bash
# /etc/cron.d/foxhunt-quarterly-retrain

# Run on first Sunday of Jan/Apr/Jul/Oct at 2 AM
0 2 1-7 1,4,7,10 0 /home/jgrusewski/scripts/quarterly_retrain.sh

# Content of quarterly_retrain.sh:
#!/bin/bash
set -e

cd /home/jgrusewski/Work/foxhunt

# Generate version tag
QUARTER=$(date +"%Y")Q$(($(date +"%m")/3))
VERSION_TAG="${QUARTER}_v1"

# Execute retraining
cargo run -p ml --example retrain_all_models --release --features cuda -- \
  --models DQN,PPO,MAMBA2,TFT \
  --latest-days 90 \
  --version-tag "$VERSION_TAG" \
  2>&1 | tee "logs/retraining_${VERSION_TAG}.log"

# Send notification
curl -X POST https://hooks.slack.com/services/YOUR/WEBHOOK/URL \
  -H 'Content-Type: application/json' \
  -d "{\"text\":\"Quarterly retraining completed: $VERSION_TAG\"}"
```

#### Systemd Timer (Alternative)

```ini
# /etc/systemd/system/foxhunt-retrain.timer

[Unit]
Description=Foxhunt Quarterly Retraining Timer

[Timer]
OnCalendar=Sun *-01,04,07,10-01..07 02:00:00
Persistent=true

[Install]
WantedBy=timers.target
```

### B. Metrics Dashboard

Grafana dashboard for monitoring retraining:

```yaml
# grafana/dashboards/ml_retraining.json

{
  "dashboard": {
    "title": "ML Retraining Metrics",
    "panels": [
      {
        "title": "Training Progress",
        "targets": [
          "ml_training_loss",
          "ml_training_accuracy",
          "ml_training_epoch"
        ]
      },
      {
        "title": "Validation Metrics",
        "targets": [
          "ml_validation_sharpe_ratio",
          "ml_validation_win_rate",
          "ml_validation_max_drawdown"
        ]
      },
      {
        "title": "Quality Gate Status",
        "targets": [
          "ml_quality_gate_passed_total",
          "ml_quality_gate_failed_total"
        ]
      }
    ]
  }
}
```

### C. Contacts

| Role | Name | Email | Slack |
|------|------|-------|-------|
| ML Engineering Lead | TBD | ml-lead@foxhunt.ai | @ml-lead |
| Trading Operations | TBD | trading-ops@foxhunt.ai | @trading-ops |
| DevOps Lead | TBD | devops@foxhunt.ai | @devops |
| On-call Engineer | TBD | oncall@foxhunt.ai | @oncall |

### D. Change Log

| Date | Version | Changes | Author |
|------|---------|---------|--------|
| 2025-10-14 | 1.0 | Initial SOP creation | Agent 79 |

---

**Document Status**: DRAFT
**Next Review**: 2025-01-15 (after first quarterly retraining)
**Approval Required**: ML Engineering Lead, Trading Operations Lead