foxhunt/CHECKPOINT_QUICK_REFERENCE.md

Checkpoint Management Quick Reference

Status: ✅ PRODUCTION READY (100% operational, 7/7 tests passing)

---

Quick Commands

List All Checkpoints

# PostgreSQL query
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -c "SELECT model_type, version, metrics->>'sharpe_ratio' as sharpe, training_date 
      FROM ml_model_versions WHERE is_archived = false ORDER BY training_date DESC LIMIT 10;"

Run Checkpoint Tests

# Full checkpoint manager test suite (7/7 tests)
cargo test --package ml_training_service --test checkpoint_manager_tests

# MinIO E2E tests (requires MinIO running)
docker-compose up -d minio
cargo test --test minio_e2e_tests -- --ignored

Apply Retention Policy

// Keep best 5 checkpoints by Sharpe ratio
let retention_policy = RetentionPolicy {
    max_checkpoints_per_model: 5,
    ranking_metric: "sharpe_ratio".to_string(),
    ascending: false,  // Higher is better
};

let manager = CheckpointManager::new(pool, retention_policy).await?;
let archived_count = manager.apply_retention_policy(ModelType::DQN, "my_model").await?;

Cleanup Old Checkpoints

// Remove checkpoints older than 30 days
let cleanup_count = manager.cleanup_old_checkpoints(
    ModelType::PPO,
    "my_model",
    30  // days threshold
).await?;

---

Storage Backends

FileSystem (Development)

# Default location
./checkpoints/
├── model_name_v1.0.0_e100_s10000_timestamp.dqn
└── metadata/
    └── model_name_v1.0.0_e100_s10000_timestamp.dqn.metadata.json

# Configure custom directory
export CHECKPOINT_STORAGE_DIR="/data/ml_checkpoints"

S3/MinIO (Production)

# Environment variables
export S3_CHECKPOINT_BUCKET=foxhunt-checkpoints
export S3_CHECKPOINT_PREFIX=ml-checkpoints
export AWS_REGION=us-east-1
export AWS_ACCESS_KEY_ID=<key>
export AWS_SECRET_ACCESS_KEY=<secret>
export S3_ENABLE_ENCRYPTION=true

# Start MinIO (development)
docker-compose up -d minio
# Console: http://localhost:9001 (foxhunt_test / foxhunt_test_password)

PostgreSQL Metadata

# Connect
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt

# Useful queries
\d ml_model_versions                    # Table schema
SELECT * FROM v_active_ml_models;       # Active models view
SELECT * FROM v_production_ml_models;   # Production models view

---

Version Management

Valid Semantic Versions

✅ 1.0.0            - Standard release
✅ 1.0.1            - Patch release
✅ 2.1.3            - Minor release
✅ 1.0.0-alpha      - Pre-release
✅ 1.0.0-beta+build1 - Pre-release with build metadata

❌ 1.0              - Missing patch
❌ v1.0.0           - Prefix not allowed
❌ 1.0.0.0          - Too many components

Version Compatibility

// Check compatibility
let manager = VersionManager::new();
let compat_info = manager.check_compatibility(
    "1.0.0",     // Current version
    "1.1.0",     // Checkpoint version
    ModelType::DQN
)?;

println!("Compatible: {}", compat_info.compatible);
println!("Risk: {:?}", compat_info.risk);  // None/Low/Medium/High

Version Progression

// Suggest next version
manager.suggest_next_version("1.0.0", VersionChangeType::Patch)?;  // → "1.0.1"
manager.suggest_next_version("1.0.0", VersionChangeType::Minor)?;  // → "1.1.0"
manager.suggest_next_version("1.0.0", VersionChangeType::Major)?;  // → "2.0.0"

---

Integrity Validation

SHA256 Checksum

// Automatically calculated on save
let checkpoint_id = manager.save_checkpoint(&model, tags).await?;

// Validate on load (automatic if config.validate_checksums = true)
let metadata = manager.load_checkpoint(&mut model, &checkpoint_id).await?;

// Manual validation
manager.validate_checksum(&checkpoint_id, &data).await?;

Test Integrity

# Run integrity validation test
cargo test --package ml_training_service --test checkpoint_manager_tests \
  test_sha256_integrity_validation

---

Rollback Scenarios

Automated Rollback Triggers

Scenario	Trigger	Recovery Time	Action
DailyLossExceeded	Loss > $2,000	<5 minutes	Emergency halt + reduce positions
HighDisagreement	Disagreement >70% for 1 hour	<5 minutes	Disable models + baseline mode
ModelFailure	>3 consecutive errors	<5 minutes	Disable failed model
CascadeFailure	2+ models fail	<5 minutes	Emergency halt + revert to baseline

Rollback Configuration

let rollback_config = RollbackConfig {
    daily_loss_threshold_usd: 2000.0,
    high_disagreement_threshold: 0.70,
    disagreement_duration_secs: 3600,       // 1 hour
    max_consecutive_errors: 3,
    cascade_failure_threshold: 2,           // 2 models
    position_reduction_factor: 0.50,        // 50% reduction
    monitoring_interval_secs: 10,
    recovery_timeout_secs: 300,             // 5 minutes
    enable_automatic_rollback: true,
};

Manual Rollback

// Find best recent checkpoint
let checkpoint = manager.revert_to_stable_checkpoint(
    ModelType::DQN,
    "trading_model"
).await?;

// Load checkpoint
manager.load_checkpoint(&mut model, &checkpoint.checkpoint_id).await?;

---

Common Workflows

Save Checkpoint

let manager = CheckpointManager::new(config)?;

// Save with tags
let checkpoint_id = manager.save_checkpoint(
    &model,
    Some(vec!["production-candidate".to_string()])
).await?;

Load Latest Checkpoint

// Load most recent checkpoint
let metadata = manager.load_latest_checkpoint(&mut model).await?;

List Checkpoints

let checkpoints = manager.list_checkpoints(
    ModelType::MAMBA,
    "my_model"  // Empty string matches all names
).await;

for checkpoint in checkpoints {
    println!("{}: Sharpe {:.2}", 
        checkpoint.version,
        checkpoint.metrics.get("sharpe_ratio").unwrap_or(&0.0)
    );
}

Delete Checkpoint

manager.delete_checkpoint(&checkpoint_id).await?;

---

Database Queries

Find Best Checkpoint by Metric

SELECT model_id, version, 
       (metrics->>'sharpe_ratio')::float as sharpe
FROM ml_model_versions
WHERE model_type = 'DQN'
  AND is_archived = false
  AND training_date > NOW() - INTERVAL '30 days'
ORDER BY (metrics->>'sharpe_ratio')::float DESC
LIMIT 1;

Count Checkpoints by Model

SELECT model_type, 
       COUNT(*) as total,
       COUNT(*) FILTER (WHERE is_archived = false) as active,
       COUNT(*) FILTER (WHERE is_production = true) as production
FROM ml_model_versions
GROUP BY model_type;

Cleanup Archived Checkpoints

-- Mark checkpoints older than 90 days as archived
UPDATE ml_model_versions
SET is_archived = true, updated_at = NOW()
WHERE training_date < NOW() - INTERVAL '90 days'
  AND is_archived = false
  AND is_production = false;

---

Performance Tuning

Checkpoint Configuration

let config = CheckpointConfig {
    base_dir: PathBuf::from("./checkpoints"),
    compression: CompressionType::LZ4,      // Fast compression
    format: CheckpointFormat::Binary,       // Fast format
    max_checkpoints_per_model: 10,
    auto_cleanup: true,
    validate_checksums: true,
    incremental_checkpoints: true,
    compression_level: 3,                   // Balance speed/compression
    async_io: true,
    buffer_size: 64 * 1024,                 // 64KB buffer
};

Performance Metrics

let stats = manager.get_stats();
println!("Total saved: {}", stats["total_saved"]);
println!("Avg save time: {}μs", stats["avg_save_time_us"]);
println!("Compression savings: {} bytes", stats["compression_savings"]);

---

Troubleshooting

Checksum Mismatch

ERROR: Checksum mismatch for checkpoint XYZ: expected abc123, got def456

Cause: Data corruption during storage/transfer
Fix: Re-save checkpoint or restore from S3 backup

Version Format Invalid

ERROR: Invalid semantic version: '1.0'. Expected format: major.minor.patch

Cause: Version string doesn't follow SemVer 2.0
Fix: Use valid version like "1.0.0"

PostgreSQL Connection Error

ERROR: Failed to connect to database: connection refused

Cause: PostgreSQL not running or wrong credentials
Fix: Check docker-compose ps and DATABASE_URL env var

MinIO Not Running

ERROR: Failed to access S3 bucket: connection refused

Cause: MinIO service not started
Fix: docker-compose up -d minio

---

File Locations

Component	Path
CheckpointManager	/home/jgrusewski/Work/foxhunt/services/ml_training_service/src/checkpoint_manager.rs
Core Checkpoint System	/home/jgrusewski/Work/foxhunt/ml/src/checkpoint/mod.rs
Storage Backends	/home/jgrusewski/Work/foxhunt/ml/src/checkpoint/storage.rs
Versioning	/home/jgrusewski/Work/foxhunt/ml/src/checkpoint/versioning.rs
Tests	/home/jgrusewski/Work/foxhunt/services/ml_training_service/tests/checkpoint_manager_tests.rs
Database Schema	/home/jgrusewski/Work/foxhunt/migrations/021_ml_model_versioning.sql
Rollback Automation	/home/jgrusewski/Work/foxhunt/services/trading_service/src/rollback_automation.rs

---

Next Steps

✅ System is production-ready (100% operational)

Recommended Actions:

1. Execute GPU training benchmark (30-60 min)
2. Download 90 days ES/NQ/ZN/6E data (~$2)
3. Begin 4-6 week ML training based on benchmark results

Documentation: See WAVE_1_AGENT_5_CHECKPOINT_ANALYSIS.md for full analysis