foxhunt/CHECKPOINT_QUICK_REFERENCE.md

# Checkpoint Management Quick Reference

**Status**: ✅ PRODUCTION READY (100% operational, 7/7 tests passing)

---

## Quick Commands

### List All Checkpoints
```bash
# PostgreSQL query
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -c "SELECT model_type, version, metrics->>'sharpe_ratio' as sharpe, training_date 
      FROM ml_model_versions WHERE is_archived = false ORDER BY training_date DESC LIMIT 10;"
```

### Run Checkpoint Tests
```bash
# Full checkpoint manager test suite (7/7 tests)
cargo test --package ml_training_service --test checkpoint_manager_tests

# MinIO E2E tests (requires MinIO running)
docker-compose up -d minio
cargo test --test minio_e2e_tests -- --ignored
```

### Apply Retention Policy
```rust
// Keep best 5 checkpoints by Sharpe ratio
let retention_policy = RetentionPolicy {
    max_checkpoints_per_model: 5,
    ranking_metric: "sharpe_ratio".to_string(),
    ascending: false,  // Higher is better
};

let manager = CheckpointManager::new(pool, retention_policy).await?;
let archived_count = manager.apply_retention_policy(ModelType::DQN, "my_model").await?;
```

### Cleanup Old Checkpoints
```rust
// Remove checkpoints older than 30 days
let cleanup_count = manager.cleanup_old_checkpoints(
    ModelType::PPO,
    "my_model",
    30  // days threshold
).await?;
```

---

## Storage Backends

### FileSystem (Development)
```bash
# Default location
./checkpoints/
├── model_name_v1.0.0_e100_s10000_timestamp.dqn
└── metadata/
    └── model_name_v1.0.0_e100_s10000_timestamp.dqn.metadata.json

# Configure custom directory
export CHECKPOINT_STORAGE_DIR="/data/ml_checkpoints"
```

### S3/MinIO (Production)
```bash
# Environment variables
export S3_CHECKPOINT_BUCKET=foxhunt-checkpoints
export S3_CHECKPOINT_PREFIX=ml-checkpoints
export AWS_REGION=us-east-1
export AWS_ACCESS_KEY_ID=<key>
export AWS_SECRET_ACCESS_KEY=<secret>
export S3_ENABLE_ENCRYPTION=true

# Start MinIO (development)
docker-compose up -d minio
# Console: http://localhost:9001 (foxhunt_test / foxhunt_test_password)
```

### PostgreSQL Metadata
```bash
# Connect
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt

# Useful queries
\d ml_model_versions                    # Table schema
SELECT * FROM v_active_ml_models;       # Active models view
SELECT * FROM v_production_ml_models;   # Production models view
```

---

## Version Management

### Valid Semantic Versions
```
✅ 1.0.0            - Standard release
✅ 1.0.1            - Patch release
✅ 2.1.3            - Minor release
✅ 1.0.0-alpha      - Pre-release
✅ 1.0.0-beta+build1 - Pre-release with build metadata

❌ 1.0              - Missing patch
❌ v1.0.0           - Prefix not allowed
❌ 1.0.0.0          - Too many components
```

### Version Compatibility
```rust
// Check compatibility
let manager = VersionManager::new();
let compat_info = manager.check_compatibility(
    "1.0.0",     // Current version
    "1.1.0",     // Checkpoint version
    ModelType::DQN
)?;

println!("Compatible: {}", compat_info.compatible);
println!("Risk: {:?}", compat_info.risk);  // None/Low/Medium/High
```

### Version Progression
```rust
// Suggest next version
manager.suggest_next_version("1.0.0", VersionChangeType::Patch)?;  // → "1.0.1"
manager.suggest_next_version("1.0.0", VersionChangeType::Minor)?;  // → "1.1.0"
manager.suggest_next_version("1.0.0", VersionChangeType::Major)?;  // → "2.0.0"
```

---

## Integrity Validation

### SHA256 Checksum
```rust
// Automatically calculated on save
let checkpoint_id = manager.save_checkpoint(&model, tags).await?;

// Validate on load (automatic if config.validate_checksums = true)
let metadata = manager.load_checkpoint(&mut model, &checkpoint_id).await?;

// Manual validation
manager.validate_checksum(&checkpoint_id, &data).await?;
```

### Test Integrity
```bash
# Run integrity validation test
cargo test --package ml_training_service --test checkpoint_manager_tests \
  test_sha256_integrity_validation
```

---

## Rollback Scenarios

### Automated Rollback Triggers

| Scenario | Trigger | Recovery Time | Action |
|----------|---------|---------------|--------|
| **DailyLossExceeded** | Loss > $2,000 | <5 minutes | Emergency halt + reduce positions |
| **HighDisagreement** | Disagreement >70% for 1 hour | <5 minutes | Disable models + baseline mode |
| **ModelFailure** | >3 consecutive errors | <5 minutes | Disable failed model |
| **CascadeFailure** | 2+ models fail | <5 minutes | Emergency halt + revert to baseline |

### Rollback Configuration
```rust
let rollback_config = RollbackConfig {
    daily_loss_threshold_usd: 2000.0,
    high_disagreement_threshold: 0.70,
    disagreement_duration_secs: 3600,       // 1 hour
    max_consecutive_errors: 3,
    cascade_failure_threshold: 2,           // 2 models
    position_reduction_factor: 0.50,        // 50% reduction
    monitoring_interval_secs: 10,
    recovery_timeout_secs: 300,             // 5 minutes
    enable_automatic_rollback: true,
};
```

### Manual Rollback
```rust
// Find best recent checkpoint
let checkpoint = manager.revert_to_stable_checkpoint(
    ModelType::DQN,
    "trading_model"
).await?;

// Load checkpoint
manager.load_checkpoint(&mut model, &checkpoint.checkpoint_id).await?;
```

---

## Common Workflows

### Save Checkpoint
```rust
let manager = CheckpointManager::new(config)?;

// Save with tags
let checkpoint_id = manager.save_checkpoint(
    &model,
    Some(vec!["production-candidate".to_string()])
).await?;
```

### Load Latest Checkpoint
```rust
// Load most recent checkpoint
let metadata = manager.load_latest_checkpoint(&mut model).await?;
```

### List Checkpoints
```rust
let checkpoints = manager.list_checkpoints(
    ModelType::MAMBA,
    "my_model"  // Empty string matches all names
).await;

for checkpoint in checkpoints {
    println!("{}: Sharpe {:.2}", 
        checkpoint.version,
        checkpoint.metrics.get("sharpe_ratio").unwrap_or(&0.0)
    );
}
```

### Delete Checkpoint
```rust
manager.delete_checkpoint(&checkpoint_id).await?;
```

---

## Database Queries

### Find Best Checkpoint by Metric
```sql
SELECT model_id, version, 
       (metrics->>'sharpe_ratio')::float as sharpe
FROM ml_model_versions
WHERE model_type = 'DQN'
  AND is_archived = false
  AND training_date > NOW() - INTERVAL '30 days'
ORDER BY (metrics->>'sharpe_ratio')::float DESC
LIMIT 1;
```

### Count Checkpoints by Model
```sql
SELECT model_type, 
       COUNT(*) as total,
       COUNT(*) FILTER (WHERE is_archived = false) as active,
       COUNT(*) FILTER (WHERE is_production = true) as production
FROM ml_model_versions
GROUP BY model_type;
```

### Cleanup Archived Checkpoints
```sql
-- Mark checkpoints older than 90 days as archived
UPDATE ml_model_versions
SET is_archived = true, updated_at = NOW()
WHERE training_date < NOW() - INTERVAL '90 days'
  AND is_archived = false
  AND is_production = false;
```

---

## Performance Tuning

### Checkpoint Configuration
```rust
let config = CheckpointConfig {
    base_dir: PathBuf::from("./checkpoints"),
    compression: CompressionType::LZ4,      // Fast compression
    format: CheckpointFormat::Binary,       // Fast format
    max_checkpoints_per_model: 10,
    auto_cleanup: true,
    validate_checksums: true,
    incremental_checkpoints: true,
    compression_level: 3,                   // Balance speed/compression
    async_io: true,
    buffer_size: 64 * 1024,                 // 64KB buffer
};
```

### Performance Metrics
```rust
let stats = manager.get_stats();
println!("Total saved: {}", stats["total_saved"]);
println!("Avg save time: {}μs", stats["avg_save_time_us"]);
println!("Compression savings: {} bytes", stats["compression_savings"]);
```

---

## Troubleshooting

### Checksum Mismatch
```
ERROR: Checksum mismatch for checkpoint XYZ: expected abc123, got def456
```
**Cause**: Data corruption during storage/transfer  
**Fix**: Re-save checkpoint or restore from S3 backup

### Version Format Invalid
```
ERROR: Invalid semantic version: '1.0'. Expected format: major.minor.patch
```
**Cause**: Version string doesn't follow SemVer 2.0  
**Fix**: Use valid version like "1.0.0"

### PostgreSQL Connection Error
```
ERROR: Failed to connect to database: connection refused
```
**Cause**: PostgreSQL not running or wrong credentials  
**Fix**: Check `docker-compose ps` and `DATABASE_URL` env var

### MinIO Not Running
```
ERROR: Failed to access S3 bucket: connection refused
```
**Cause**: MinIO service not started  
**Fix**: `docker-compose up -d minio`

---

## File Locations

| Component | Path |
|-----------|------|
| CheckpointManager | `/home/jgrusewski/Work/foxhunt/services/ml_training_service/src/checkpoint_manager.rs` |
| Core Checkpoint System | `/home/jgrusewski/Work/foxhunt/ml/src/checkpoint/mod.rs` |
| Storage Backends | `/home/jgrusewski/Work/foxhunt/ml/src/checkpoint/storage.rs` |
| Versioning | `/home/jgrusewski/Work/foxhunt/ml/src/checkpoint/versioning.rs` |
| Tests | `/home/jgrusewski/Work/foxhunt/services/ml_training_service/tests/checkpoint_manager_tests.rs` |
| Database Schema | `/home/jgrusewski/Work/foxhunt/migrations/021_ml_model_versioning.sql` |
| Rollback Automation | `/home/jgrusewski/Work/foxhunt/services/trading_service/src/rollback_automation.rs` |

---

## Next Steps

✅ **System is production-ready** (100% operational)

**Recommended Actions**:
1. Execute GPU training benchmark (30-60 min)
2. Download 90 days ES/NQ/ZN/6E data (~$2)
3. Begin 4-6 week ML training based on benchmark results

**Documentation**: See `WAVE_1_AGENT_5_CHECKPOINT_ANALYSIS.md` for full analysis