foxhunt/docs/MODEL_RETRAINING_SOP.md

Model Retraining Standard Operating Procedure (SOP)

Last Updated: 2025-10-14 Version: 1.0 Owner: ML Training Team

---

Overview

This document defines the standard operating procedure for quarterly ML model retraining in the Foxhunt HFT trading system. Retraining ensures models adapt to evolving market conditions and maintain competitive edge.

---

Table of Contents

1. When to Retrain
2. Prerequisites
3. Retraining Schedule
4. Execution Procedure
5. Validation Criteria
6. Rollout Procedure
7. Rollback Procedure
8. Checkpoint Management
9. Quality Gates
10. Troubleshooting

---

When to Retrain

Scheduled Retraining (Quarterly)

Models should be retrained every quarter to adapt to market regime changes:

• Q1: January 1-15 (retrain with Oct-Dec data)
• Q2: April 1-15 (retrain with Jan-Mar data)
• Q3: July 1-15 (retrain with Apr-Jun data)
• Q4: October 1-15 (retrain with Jul-Sep data)

Rationale: 90 days provides sufficient data for statistical significance while staying current with market conditions.

Ad-Hoc Retraining Triggers

Retrain immediately if any of the following occur:

1. Performance Degradation

   • Production Sharpe ratio drops below 1.0 for 2+ consecutive weeks
   • Win rate falls below 50% for 1+ week
   • Max drawdown exceeds 20%
   • Daily PnL negative for 5+ consecutive days

2. Market Regime Shift

   • VIX spikes >50% in single day
   • Major policy changes (Fed rate changes >0.5%)
   • Black swan events (COVID-19, flash crashes)
   • Sector rotation >20% in single week

3. Data Distribution Drift

   • Feature distribution drift >15% (KL divergence)
   • Volume patterns change >25%
   • Volatility regime shift (Hurst exponent change >0.2)

4. Model Staleness

   • Model age >6 months without retraining
   • Training data >12 months old

---

Prerequisites

Infrastructure Requirements

• PostgreSQL database accessible (port 5432)
• Redis cache accessible (port 6379)
• MinIO/S3 storage accessible
• GPU available (RTX 3050 Ti or better, 4GB+ VRAM)
• Disk space: 50GB+ free for training data + checkpoints
• Memory: 16GB+ RAM recommended

Data Requirements

• 90 days of OHLCV-1m data (ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT)
• Data quality validated (no gaps >5 bars)
• Price anomalies corrected (spike detection applied)
• Volume data normalized
• Estimated bars: ~180,000 (90 days × 4 symbols × ~500 bars/day)

Software Requirements

• Rust 1.70+ installed
• CUDA 12.0+ installed (for GPU training)
• Docker Compose running (for infrastructure)
• TLI CLI configured and authenticated

Access Requirements

• Production database read access
• MinIO write access (for checkpoints)
• Model registry write access
• Staging environment deployment permissions

---

Retraining Schedule

Timeline (Quarterly)

Week	Activity	Duration	Responsible
Week 1	Data acquisition & validation	2-3 days	Data Engineering
Week 2	Training execution (all models)	4-6 days	ML Training Service
Week 3	Validation & quality gates	3-5 days	ML Engineering
Week 4	Staging deployment & monitoring	7-10 days	DevOps + Trading
Week 5-6	Production rollout (gradual)	10-14 days	Trading Operations

Total: ~6 weeks from data acquisition to full production rollout

Resource Allocation

• GPU hours: 150-200 hours (DQN: 30h, PPO: 40h, MAMBA-2: 80h, TFT: 40h)
• Engineer time: 40-60 hours (1-1.5 FTE weeks)
• Compute cost: $0 (local RTX 3050 Ti) or $50-75 (cloud A100)

---

Execution Procedure

Step 1: Prepare Environment

# 1. Navigate to project directory
cd /home/jgrusewski/Work/foxhunt

# 2. Start infrastructure
docker-compose up -d

# 3. Verify services healthy
docker-compose ps
grpc_health_probe -addr=localhost:50051  # API Gateway
grpc_health_probe -addr=localhost:50054  # ML Training Service

# 4. Check GPU availability
nvidia-smi
nvcc --version

Step 2: Acquire Training Data

# Option A: Download latest 90 days from Databento
cargo run -p ml --example download_training_data --release -- \
  --symbols ES.FUT,NQ.FUT,ZN.FUT,6E.FUT \
  --start-date 2024-07-01 \
  --end-date 2024-10-01 \
  --output-dir test_data/real/databento/ml_training

# Option B: Use existing data (verify coverage)
./verify_dataset_coverage.sh test_data/real/databento/ml_training

Expected Output:

• 90 files (4 symbols × ~22 trading days/month × 3 months)
• Total size: ~2-5GB
• Bars per file: 400-500 (1-minute OHLCV for 6.5-hour trading day)

Step 3: Validate Hyperparameters

# Review best hyperparameters from previous tuning
cat ml/config/best_hyperparameters.yaml

# If tuning needed (optional, adds 8-12 hours):
tli tune start --model DQN --trials 50 --watch
tli tune best --job-id <uuid> > ml/config/best_hyperparameters.yaml

Step 4: Execute Retraining

# Full retraining (all models, sequential)
cargo run -p ml --example retrain_all_models --release --features cuda -- \
  --models DQN,PPO,MAMBA2,TFT \
  --data-dir test_data/real/databento/ml_training \
  --output-dir ml/trained_models/quarterly/2024Q4 \
  --latest-days 90 \
  --min-sharpe 1.5 \
  --min-win-rate 0.55 \
  --version-tag 2024Q4_v1

# Monitor progress (separate terminal)
tail -f ml/trained_models/quarterly/2024Q4/training.log

# Expected duration (sequential):
# - DQN: 6-8 hours
# - PPO: 8-12 hours
# - MAMBA-2: 20-30 hours
# - TFT: 10-15 hours
# Total: 44-65 hours (2-3 days on RTX 3050 Ti)

Parallel Training (if GPU memory permits):

# ⚠️ WARNING: May cause OOM on RTX 3050 Ti (4GB VRAM)
cargo run -p ml --example retrain_all_models --release --features cuda -- \
  --models DQN,PPO,MAMBA2,TFT \
  --parallel

Step 5: Review Results

# View summary report
cat ml/trained_models/quarterly/2024Q4/retraining_summary_2024Q4_v1.json | jq

# Check quality gates
grep "quality_gate_passed" ml/trained_models/quarterly/2024Q4/retraining_summary_2024Q4_v1.json

Expected Output:

{
  "run_id": "abc-123-def",
  "models_attempted": 4,
  "models_succeeded": 4,
  "models_passed_quality_gate": 3,
  "results": [
    {
      "model_type": "DQN",
      "quality_gate_passed": true,
      "validation_metrics": {
        "sharpe_ratio": 1.8,
        "win_rate": 0.58,
        "max_drawdown": 0.15
      }
    },
    ...
  ]
}

---

Validation Criteria

Quality Gates

All models must pass ALL of the following criteria before staging deployment:

Metric	Threshold	Severity
Sharpe Ratio	≥ 1.5	CRITICAL
Win Rate	≥ 55%	CRITICAL
Max Drawdown	≤ 25%	CRITICAL
Total Trades	≥ 100	WARNING
Profit Factor	≥ 1.3	WARNING
Training Convergence	True	CRITICAL

CRITICAL: Model deployment blocked if failed WARNING: Review required, deployment decision by trading team

Manual Review Checklist

Before approving staging deployment:

• Review training loss curves (converged, not overfitting)
• Inspect sample predictions (reasonable, not degenerate)
• Verify checkpoint file integrity (SHA-256 checksum)
• Compare metrics with previous production model
• Check for data leakage (train/test split properly isolated)
• Validate hyperparameters match configuration
• Confirm training data covers expected date range
• Review any quality gate failures with trading team

---

Rollout Procedure

Phase 1: Staging Deployment (Week 4)

# 1. Register new model versions in model registry
tli model register \
  --model-type DQN \
  --version 2024Q4_v1 \
  --checkpoint ml/trained_models/quarterly/2024Q4/dqn_2024Q4_v1_final.safetensors \
  --status experimental

# 2. Deploy to staging environment
kubectl apply -f k8s/staging/ml-deployment-2024Q4.yaml

# 3. Run integration tests
cargo test -p integration_tests --test staging_ml_models

# 4. Monitor staging for 7-10 days
# - Check Grafana dashboard: http://localhost:3000/d/staging-ml
# - Review prediction distribution
# - Verify latency <100ms P99
# - Monitor memory usage

Phase 2: Production Canary (Week 5)

# 1. Route 10% traffic to new models
tli deploy canary \
  --models DQN:2024Q4_v1,PPO:2024Q4_v1 \
  --traffic-percentage 10 \
  --duration-hours 48

# 2. Monitor canary metrics (48 hours)
# Key metrics to watch:
# - Sharpe ratio: Should match validation (±0.2)
# - Win rate: Should be ≥55%
# - Latency: Should be <100ms P99
# - Error rate: Should be <0.1%

# 3. If metrics good, increase to 25%
tli deploy canary --traffic-percentage 25 --duration-hours 48

# 4. If metrics still good, increase to 50%
tli deploy canary --traffic-percentage 50 --duration-hours 72

Phase 3: Full Production Rollout (Week 6)

# 1. Promote to production (100% traffic)
tli model promote \
  --model-type DQN \
  --version 2024Q4_v1 \
  --status production

# 2. Mark old models as archived
tli model archive \
  --model-type DQN \
  --version 2024Q3_v1

# 3. Verify production metrics (24 hours)
# - Sharpe ratio: ≥1.5
# - Win rate: ≥55%
# - PnL: Positive cumulative
# - Max drawdown: ≤25%

# 4. Document deployment
# - Update PRODUCTION_MODELS.md
# - Log deployment in audit trail
# - Notify trading team

---

Rollback Procedure

When to Rollback

Initiate immediate rollback if any occur within first 48 hours:

• Production Sharpe ratio <0.5
• Win rate <45%
• Max drawdown >30%
• Daily PnL negative for 2+ consecutive days
• Model errors/crashes >1%
• Prediction latency >500ms P99

Rollback Steps

# 1. Revert traffic to previous production model
tli model rollback \
  --model-type DQN \
  --to-version 2024Q3_v1

# 2. Verify old model restored
grpc_health_probe -addr=localhost:50054

# 3. Mark failed model as archived
tli model archive \
  --model-type DQN \
  --version 2024Q4_v1 \
  --reason "Failed production validation - Sharpe 0.4 < 1.5 threshold"

# 4. Investigate root cause
# - Review training logs
# - Check data quality
# - Validate hyperparameters
# - Analyze prediction distribution

# 5. Document incident
# - Create postmortem report
# - Update SOP if needed
# - Plan corrective action

---

Checkpoint Management

Naming Convention

{model}_{version}_epoch{N}_{timestamp}.safetensors

Examples:
- dqn_2024Q4_v1_epoch150_20250115_143022.safetensors
- ppo_2024Q4_v1_final_20250116_093045.safetensors

Metadata Format

Each checkpoint includes JSON metadata:

{
  "model_type": "DQN",
  "version": "2024Q4_v1",
  "parent_checkpoint": "dqn_2024Q3_v1_final.safetensors",
  "training_date": "2025-01-15T14:30:22Z",
  "data_range": {
    "start_date": "2024-10-01",
    "end_date": "2024-12-31",
    "total_bars": 180000,
    "symbols": ["ES.FUT", "NQ.FUT", "ZN.FUT", "6E.FUT"]
  },
  "hyperparameters": {
    "learning_rate": 0.0001,
    "batch_size": 128,
    "epochs": 200,
    "gamma": 0.99
  },
  "training_metrics": {
    "final_loss": 0.001234,
    "best_epoch": 187,
    "training_time_seconds": 28800
  },
  "validation_metrics": {
    "sharpe_ratio": 1.8,
    "win_rate": 0.58,
    "max_drawdown": 0.15,
    "total_trades": 250
  },
  "quality_gate_passed": true,
  "checksum": "sha256:abc123def456..."
}

Retention Policy

• Production checkpoints: Keep last 3 versions (9 months)
• Staging checkpoints: Keep last 2 versions (6 months)
• Experimental checkpoints: Keep last 1 version (3 months)
• Failed checkpoints: Delete after 30 days (archive metadata)

Storage Locations

ml/trained_models/
├── production/        # Active production models
│   ├── dqn/
│   ├── ppo/
│   ├── mamba2/
│   └── tft/
├── staging/           # Models in staging validation
│   └── 2024Q4/
├── quarterly/         # Quarterly retraining outputs
│   ├── 2024Q1/
│   ├── 2024Q2/
│   ├── 2024Q3/
│   └── 2024Q4/
└── archived/          # Old models (metadata only)
    └── 2023Q4/

---

Quality Gates

Implementation

Quality gates are enforced automatically by the retraining pipeline:

// ml/examples/retrain_all_models.rs
fn apply_quality_gates(
    metrics: &ValidationMetricsSnapshot,
    gates: &QualityGates,
) -> (bool, Vec<String>) {
    let mut failures = Vec::new();

    if metrics.sharpe_ratio < gates.min_sharpe {
        failures.push(format!(
            "Sharpe ratio {:.2} < {:.2}",
            metrics.sharpe_ratio, gates.min_sharpe
        ));
    }

    // ... other checks

    (failures.is_empty(), failures)
}

Override Procedure

If quality gates fail but trading team approves deployment:

# 1. Document override reason
tli model override-quality-gate \
  --model-type DQN \
  --version 2024Q4_v1 \
  --reason "Market regime shift - lower Sharpe acceptable" \
  --approved-by "John Doe (Trading Lead)" \
  --approval-ticket JIRA-12345

# 2. Deploy with override flag
tli deploy --force --override-quality-gate

# 3. Monitor closely (daily reviews for 2 weeks)

⚠️ WARNING: Quality gate overrides require:

• Director-level approval
• Documented business justification
• Enhanced monitoring (daily reviews)
• Automatic rollback if performance degrades further

---

Troubleshooting

Common Issues

1. GPU Out of Memory (OOM)

Symptoms: Training crashes with "CUDA out of memory" error

Solutions:

# Option A: Reduce batch size
--batch-size 64  # Instead of 128

# Option B: Use CPU (slower, but works)
export CUDA_VISIBLE_DEVICES=""

# Option C: Train models sequentially (default)
# Do NOT use --parallel flag

# Option D: Use gradient accumulation
--gradient-accumulation-steps 2

2. Data Loading Fails

Symptoms: "No DBN files found" or "Failed to decode record"

Solutions:

# Verify data directory
ls -lh test_data/real/databento/ml_training/*.dbn

# Check file permissions
chmod 644 test_data/real/databento/ml_training/*.dbn

# Validate DBN files
cargo run -p ml --example test_dbn_loading -- \
  --file test_data/real/databento/ml_training/ES.FUT_ohlcv-1m_2024-01-02.dbn

3. Training Not Converging

Symptoms: Loss plateaus at high value (>10.0), accuracy <50%

Solutions:

# Option A: Adjust learning rate
--learning-rate 0.0005  # Increase from 0.0001

# Option B: Increase epochs
--epochs 300  # Instead of 200

# Option C: Check data quality
./verify_dataset_coverage.sh test_data/real/databento/ml_training

# Option D: Review hyperparameters
cat ml/config/best_hyperparameters.yaml

4. Quality Gate Failures

Symptoms: Model trained successfully but Sharpe <1.5 or Win Rate <55%

Analysis:

# Run comprehensive backtest analysis
cargo run -p ml --example comprehensive_model_backtest --release -- \
  --checkpoint ml/trained_models/quarterly/2024Q4/dqn_2024Q4_v1_final.safetensors \
  --output-report backtest_analysis.json

# Review failure reasons
cat ml/trained_models/quarterly/2024Q4/retraining_summary_2024Q4_v1.json | \
  jq '.results[] | select(.quality_gate_passed == false) | .quality_gate_failures'

Common Causes:

• Insufficient data: Need 90+ days (not 30-60 days)
• Data quality issues: Price spikes, gaps, anomalies
• Overfitting: Model memorized training data, poor generalization
• Hyperparameter mismatch: Using suboptimal hyperparameters
• Market regime change: Training data not representative of current market

Solutions:

1. Acquire more/better training data
2. Adjust hyperparameters (run Optuna tuning)
3. Add regularization (dropout, L2 penalty)
4. Use ensemble methods (combine multiple models)
5. Consult with trading team (may need strategy adjustment)

---

Appendix

A. Automation Scripts

Quarterly Retraining Cron Job

# /etc/cron.d/foxhunt-quarterly-retrain

# Run on first Sunday of Jan/Apr/Jul/Oct at 2 AM
0 2 1-7 1,4,7,10 0 /home/jgrusewski/scripts/quarterly_retrain.sh

# Content of quarterly_retrain.sh:
#!/bin/bash
set -e

cd /home/jgrusewski/Work/foxhunt

# Generate version tag
QUARTER=$(date +"%Y")Q$(($(date +"%m")/3))
VERSION_TAG="${QUARTER}_v1"

# Execute retraining
cargo run -p ml --example retrain_all_models --release --features cuda -- \
  --models DQN,PPO,MAMBA2,TFT \
  --latest-days 90 \
  --version-tag "$VERSION_TAG" \
  2>&1 | tee "logs/retraining_${VERSION_TAG}.log"

# Send notification
curl -X POST https://hooks.slack.com/services/YOUR/WEBHOOK/URL \
  -H 'Content-Type: application/json' \
  -d "{\"text\":\"Quarterly retraining completed: $VERSION_TAG\"}"

Systemd Timer (Alternative)

# /etc/systemd/system/foxhunt-retrain.timer

[Unit]
Description=Foxhunt Quarterly Retraining Timer

[Timer]
OnCalendar=Sun *-01,04,07,10-01..07 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

B. Metrics Dashboard

Grafana dashboard for monitoring retraining:

# grafana/dashboards/ml_retraining.json

{
  "dashboard": {
    "title": "ML Retraining Metrics",
    "panels": [
      {
        "title": "Training Progress",
        "targets": [
          "ml_training_loss",
          "ml_training_accuracy",
          "ml_training_epoch"
        ]
      },
      {
        "title": "Validation Metrics",
        "targets": [
          "ml_validation_sharpe_ratio",
          "ml_validation_win_rate",
          "ml_validation_max_drawdown"
        ]
      },
      {
        "title": "Quality Gate Status",
        "targets": [
          "ml_quality_gate_passed_total",
          "ml_quality_gate_failed_total"
        ]
      }
    ]
  }
}

C. Contacts

Role	Name	Email	Slack
ML Engineering Lead	TBD	ml-lead@foxhunt.ai	@ml-lead
Trading Operations	TBD	trading-ops@foxhunt.ai	@trading-ops
DevOps Lead	TBD	devops@foxhunt.ai	@devops
On-call Engineer	TBD	oncall@foxhunt.ai	@oncall

D. Change Log

Date	Version	Changes	Author
2025-10-14	1.0	Initial SOP creation	Agent 79

---

Document Status: DRAFT Next Review: 2025-01-15 (after first quarterly retraining) Approval Required: ML Engineering Lead, Trading Operations Lead