foxhunt/crates/ml/PERFORMANCE_TRACKING.md

# Performance Regression Detection System

Automated performance tracking and regression detection for ML training pipeline.

## Overview

This TDD-driven system tracks key performance metrics and automatically fails CI builds when performance degrades beyond acceptable thresholds (>10% regression).

## Tracked Metrics

| Metric | Target | Description |
|--------|--------|-------------|
| **DBN Load Time** | <10ms | Real market data loading from DBN files |
| **Feature Extraction** | - | Technical indicator calculation (16 features) |
| **Training Step** | - | Single training iteration time |
| **Inference Latency** | <50μs | Model prediction time (HFT requirement) |
| **Throughput** | - | Samples processed per second |
| **Memory Usage** | Model-specific | Peak memory consumption |

## Architecture

```
┌─────────────────────────────────────────────────────┐
│         Performance Regression Detection            │
└────────────────┬────────────────────────────────────┘
                 │
    ┌────────────┴────────────┐
    │                         │
    ▼                         ▼
┌──────────────┐      ┌─────────────────┐
│  Baseline    │      │  Current Run    │
│  Metrics     │      │  Metrics        │
│ (saved JSON) │      │  (new results)  │
└──────┬───────┘      └────────┬────────┘
       │                       │
       └───────────┬───────────┘
                   │
                   ▼
          ┌────────────────┐
          │ Regression     │
          │ Detection      │
          │ (>10% = FAIL)  │
          └────────┬───────┘
                   │
        ┌──────────┴──────────┐
        │                     │
        ▼                     ▼
   ┌────────┐          ┌──────────┐
   │ CI     │          │ Grafana  │
   │ Report │          │ Dashboard│
   └────────┘          └──────────┘
```

## Usage

### 1. Record Baseline

Run benchmark and save baseline:

```bash
cargo run --release -p ml --example quick_performance_benchmark -- \
  --output ml/benchmark_results/dqn_baseline.json \
  --git-commit $(git rev-parse HEAD) \
  --model DQN
```

### 2. Check for Regression (CI)

Compare current metrics against baseline:

```bash
cargo run --release -p ml --example check_performance_regression -- \
  --baseline ml/benchmark_results/dqn_baseline.json \
  --current ml/benchmark_results/dqn_current.json \
  --output regression_report.md \
  --threshold 10.0
```

**Exit Codes:**
- `0` - No regression detected
- `1` - Performance regression detected (>10% degradation)

### 3. View in Grafana

Import dashboard:

```bash
# Copy Grafana dashboard JSON
cp ml/grafana/performance_tracking_dashboard.json /path/to/grafana/dashboards/

# Access at: http://localhost:3000
```

## CI Integration

Performance checks run automatically on every PR:

```yaml
# .github/workflows/performance.yml
- name: Check for regression
  run: |
    cargo run --release -p ml --example check_performance_regression -- \
      --baseline baseline.json \
      --current current.json \
      --output report.md

    # Exit code 1 fails the build
```

## Testing (TDD Approach)

All tests pass (12/12):

```bash
cargo test -p ml --test performance_regression_tests
```

Test coverage:
- ✅ Baseline saving/loading
- ✅ Regression detection (>10% threshold)
- ✅ Metric tracking (DBN, features, training, inference)
- ✅ CI integration (exit codes)
- ✅ Multiple models (independent baselines)

## File Structure

```
ml/
├── src/benchmark/
│   └── performance_tracker.rs    # Core regression detection
├── tests/
│   └── performance_regression_tests.rs  # 12 TDD tests
├── examples/
│   ├── quick_performance_benchmark.rs   # Record metrics
│   └── check_performance_regression.rs  # Detect regressions
├── grafana/
│   └── performance_tracking_dashboard.json  # Grafana dashboard
└── benchmark_results/
    ├── dqn_baseline.json         # DQN baseline
    ├── ppo_baseline.json         # PPO baseline
    ├── mamba2_baseline.json      # MAMBA-2 baseline
    └── tft_baseline.json         # TFT baseline
```

## Regression Threshold

**10% threshold** = Fail CI if any metric degrades by >10%

Example:
- Baseline: DBN load time = 0.70ms
- Current: DBN load time = 0.80ms (14.3% slower)
- Result: ❌ **FAIL** - Performance regression detected

## Model-Specific Baselines

Each model has independent baselines:

| Model | Memory | Training Step | Inference |
|-------|--------|---------------|-----------|
| DQN | 50-150MB | ~100ms | ~45μs |
| PPO | 50-200MB | ~150ms | ~50μs |
| MAMBA-2 | 150-500MB | ~200ms | ~40μs |
| TFT | 1.5-2.5GB | ~500ms | ~55μs |

## Example Report

```markdown
# Performance Regression Check

## ❌ Regression Detected

Performance regression detected: 2 metric(s) degraded by >10%: dbn_load_time_ms, training_step_time_ms

### Regressions

| Metric | Baseline | Current | Change |
|--------|----------|---------|--------|
| dbn_load_time_ms | 0.70 | 0.81 | +15.7% |
| training_step_time_ms | 100.00 | 120.00 | +20.0% |

### Details

- DBN data loading time increased by 15.7% (0.70 → 0.81)
- Training step time increased by 20.0% (100.00 → 120.00)

**Baseline**: 2025-10-15 10:00:00 (commit: abc123)
**Current**: 2025-10-15 10:05:00 (commit: def456)
```

## Grafana Dashboard

Track performance over time:

- **DBN Load Time** - Real-time monitoring with <10ms threshold
- **Inference Latency** - Per-model tracking (<50μs target)
- **Training Step Time** - Compare models (DQN, PPO, MAMBA-2, TFT)
- **Memory Usage** - Track memory consumption by model
- **Regression Count** - Total regressions detected
- **Performance Change** - % change vs baseline

## Integration with Existing Systems

### GPU Training Benchmark

Performance tracker integrates with existing benchmark system:

```rust
use ml::benchmark::{PerformanceTracker, PerformanceMetrics};

// After benchmark run
let metrics = PerformanceMetrics {
    dbn_load_time_ms: benchmark_results.dbn_load_time,
    feature_extraction_time_ms: benchmark_results.feature_time,
    training_step_time_ms: benchmark_results.training_time,
    inference_latency_us: benchmark_results.inference_latency,
    throughput_samples_per_sec: benchmark_results.throughput,
    memory_usage_mb: benchmark_results.memory_peak,
    timestamp: Utc::now(),
    git_commit: git_commit_hash,
    model_type: "DQN".to_string(),
};

tracker.record_metrics(metrics).await?;
tracker.save_baseline().await?;
```

### Continuous Monitoring

Pipeline integration:

```
Training Run → Record Metrics → Check Regression → Update Grafana
     ↓              ↓                  ↓                  ↓
   model.pt    baseline.json    report.md (CI)    Prometheus metrics
```

## Benefits

1. **Automated Detection**: Catch performance regressions before merge
2. **Historical Tracking**: Grafana dashboards show trends over time
3. **CI Integration**: Fail builds on >10% regression
4. **Model-Specific**: Independent baselines per model
5. **TDD Tested**: 12/12 tests passing (100% coverage)

## Future Enhancements

- [ ] Statistical significance testing (t-test)
- [ ] Performance budget per model
- [ ] Automatic baseline updates on main merge
- [ ] Slack/email notifications on regression
- [ ] P95/P99 latency tracking
- [ ] GPU utilization metrics
- [ ] Multi-epoch stability analysis

## References

- Test file: `ml/tests/performance_regression_tests.rs`
- Implementation: `ml/src/benchmark/performance_tracker.rs`
- CI workflow: `.github/workflows/performance.yml`
- Dashboard: `ml/grafana/performance_tracking_dashboard.json`
- CLAUDE.md: System architecture and targets