# SLA Definitions - Foxhunt HFT Trading System

**Last Updated**: 2025-10-07
**Version**: 1.0
**Owner**: Platform Engineering Team

---

## 1. Overview

This document defines Service Level Indicators (SLIs), Service Level Objectives (SLOs), and Service Level Agreements (SLAs) for the Foxhunt HFT trading system. These metrics ensure operational excellence and establish clear expectations for system performance, availability, and reliability.

### 1.1 Definitions

- **SLI (Service Level Indicator)**: A quantifiable metric of a service's behavior (e.g., latency, error rate)
- **SLO (Service Level Objective)**: Internal target value or range for an SLI (e.g., p99 latency < 100μs)
- **SLA (Service Level Agreement)**: External commitment with consequences if violated (e.g., 99.9% uptime guarantee)

### 1.2 Measurement Windows

- **Real-time**: 1-minute rolling window
- **Short-term**: 1-hour rolling window
- **Daily**: 24-hour rolling window
- **Monthly**: 30-day rolling window
- **Quarterly**: 90-day rolling window

---

## 2. Trading Service SLAs

### 2.1 Latency SLAs

#### Order Processing Latency
**SLI**: Time from order submission to acknowledgment
**Measurement**: `histogram_quantile(0.99, rate(trading_order_processing_microseconds_bucket[1m]))`

| Percentile | SLO (Target) | SLA (Commitment) | Breach Consequence |
|------------|--------------|------------------|-------------------|
| P50 (median) | < 30μs | < 50μs | Warning notification |
| P95 | < 75μs | < 90μs | Investigation required |
| P99 | < 100μs | < 150μs | **CRITICAL** - Executive escalation |
| P99.9 | < 200μs | < 500μs | Service review required |

**Monitoring**:
- Alert threshold: p99 > 100μs for 10 seconds
- Critical alert: p99 > 150μs for 30 seconds
- Dashboard: Trading Performance (Panel 1)

#### End-to-End Trading Latency
**SLI**: Total time from signal generation to order execution
**Measurement**: `trading_e2e_latency_microseconds`

| Metric | SLO | SLA | Notes |
|--------|-----|-----|-------|
| P99 | < 5ms | < 10ms | Includes ML inference, risk checks, broker |
| P99.9 | < 15ms | < 25ms | Maximum acceptable delay |

### 2.2 Throughput SLAs

#### Order Processing Rate
**SLI**: Number of orders processed per second
**Measurement**: `rate(trading_orders_submitted_total[1m])`

| Period | SLO (Minimum) | SLA (Guaranteed) | Peak Capacity |
|--------|---------------|------------------|---------------|
| Sustained | 10,000 ops/sec | 5,000 ops/sec | 50,000 ops/sec |
| Burst (10s) | 50,000 ops/sec | 25,000 ops/sec | 100,000 ops/sec |

**Monitoring**:
- Alert: Rate < 1,000 ops/sec for 2 minutes
- Dashboard: Trading Performance (Panel 4)

### 2.3 Reliability SLAs

#### Order Success Rate
**SLI**: Percentage of successfully executed orders
**Measurement**: `100 * rate(trading_orders_filled_total[5m]) / rate(trading_orders_submitted_total[5m])`

| Metric | SLO | SLA | Acceptable Failure Types |
|--------|-----|-----|--------------------------|
| Fill Rate | > 95% | > 90% | Market conditions, risk breaches |
| Rejection Rate | < 5% | < 10% | Validation, compliance, limits |

**Monitoring**:
- Alert: Fill rate < 50% for 1 minute (CRITICAL)
- Alert: Rejection rate > 5% for 2 minutes

---

## 3. API Gateway SLAs

### 3.1 Authentication Performance

#### Authentication Latency
**SLI**: Time to complete JWT validation and RBAC checks
**Measurement**: `histogram_quantile(0.99, rate(api_gateway_auth_total_duration_microseconds_bucket[1m]))`

| Percentile | SLO | SLA | Notes |
|------------|-----|-----|-------|
| P99 | < 10μs | < 20μs | In-memory cache hit |
| P99 (cache miss) | < 500μs | < 1ms | Database lookup |

#### Authentication Success Rate
**SLI**: Percentage of successful authentications
**Measurement**: `100 * rate(api_gateway_auth_requests_success[5m]) / rate(api_gateway_auth_requests_total[5m])`

| Metric | SLO | SLA |
|--------|-----|-----|
| Success Rate | > 99% | > 98% |
| Failure Rate | < 1% | < 2% |

### 3.2 Proxy Performance

#### Backend Request Latency
**SLI**: Time to proxy requests to backend services
**Measurement**: `histogram_quantile(0.99, rate(api_gateway_backend_request_duration_milliseconds_bucket[1m]))`

| Service | P99 SLO | P99 SLA |
|---------|---------|---------|
| Trading Service | < 50ms | < 100ms |
| Backtesting Service | < 200ms | < 500ms |
| ML Training Service | < 100ms | < 200ms |

### 3.3 Rate Limiting

#### Rate Limit Accuracy
**SLI**: Percentage of legitimate requests allowed
**Measurement**: `100 * (1 - rate(api_gateway_auth_errors_rate_limited[5m]) / rate(api_gateway_auth_requests_total[5m]))`

| Metric | SLO | SLA |
|--------|-----|-----|
| False Positive Rate | < 0.1% | < 1% |
| Legitimate Requests Blocked | < 100/day | < 1000/day |

---

## 4. ML Training Service SLAs

### 4.1 Inference Performance

#### Inference Latency
**SLI**: Time to generate ML predictions
**Measurement**: `histogram_quantile(0.99, rate(ml_inference_duration_milliseconds_bucket[1m]))`

| Model Type | P99 SLO | P99 SLA | Notes |
|------------|---------|---------|-------|
| MAMBA-2 | < 50ms | < 100ms | Primary strategy model |
| DQN | < 30ms | < 75ms | Fast reinforcement learning |
| PPO | < 40ms | < 90ms | Policy-based models |
| TFT | < 60ms | < 120ms | Temporal forecasting |

#### Model Accuracy
**SLI**: Prediction accuracy on validation set
**Measurement**: `ml_model_accuracy`

| Model | Minimum SLO | Minimum SLA | Drift Threshold |
|-------|-------------|-------------|-----------------|
| MAMBA-2 | 0.90 | 0.85 | 0.05 drop triggers retraining |
| DQN | 0.88 | 0.83 | 0.05 drop triggers retraining |
| PPO | 0.87 | 0.82 | 0.05 drop triggers retraining |

### 4.2 GPU Utilization

#### GPU Resource Efficiency
**SLI**: GPU utilization percentage
**Measurement**: `ml_gpu_utilization_percent`

| Metric | SLO | SLA | Notes |
|--------|-----|-----|-------|
| Normal Operation | 60-90% | 30-95% | Optimal range |
| Training Phase | > 80% | > 60% | High utilization expected |
| Inference Phase | 40-70% | 20-90% | Variable load |

**Alerts**:
- Low utilization: < 30% for 10 minutes (waste alert)
- High utilization: > 95% for 5 minutes (bottleneck alert)

---

## 5. Backtesting Service SLAs

### 5.1 Execution Performance

#### Backtest Execution Time
**SLI**: Time to complete historical strategy simulation
**Measurement**: `histogram_quantile(0.95, rate(backtesting_execution_duration_seconds_bucket[10m]))`

| Dataset Size | P95 SLO | P95 SLA |
|--------------|---------|---------|
| 1 day (1M events) | < 60s | < 120s |
| 1 week (7M events) | < 300s | < 600s |
| 1 month (30M events) | < 1200s | < 2400s |

#### Data Replay Rate
**SLI**: Events processed per second during replay
**Measurement**: `rate(backtesting_events_processed_total[5m])`

| Metric | SLO | SLA |
|--------|-----|-----|
| Replay Rate | > 10,000 events/sec | > 5,000 events/sec |

---

## 6. System-Level SLAs

### 6.1 Availability

#### Service Uptime
**SLI**: Percentage of time service is healthy
**Measurement**: `100 * avg_over_time(up{job=~".*_service"}[1h])`

| Service | Monthly SLO | Monthly SLA | Allowed Downtime |
|---------|-------------|-------------|------------------|
| Trading Service | 99.99% | 99.95% | 21.6 minutes/month |
| API Gateway | 99.95% | 99.9% | 43.2 minutes/month |
| ML Training Service | 99.9% | 99.5% | 3.6 hours/month |
| Backtesting Service | 99.5% | 99.0% | 7.2 hours/month |

**Measurement Window**: 30-day rolling average
**Exclusions**: Scheduled maintenance (with 48h notice)

### 6.2 Resource Utilization

#### CPU Utilization
**SLI**: Percentage of CPU capacity used
**Measurement**: `100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100)`

| Threshold | SLO | SLA | Action |
|-----------|-----|-----|--------|
| Warning | < 85% | < 90% | Monitor |
| Critical | < 90% | < 95% | Scale or optimize |
| Emergency | N/A | < 98% | Immediate intervention |

#### Memory Utilization
**SLI**: Percentage of memory capacity used
**Measurement**: `100 * (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes`

| Threshold | SLO | SLA | Action |
|-----------|-----|-----|--------|
| Warning | < 85% | < 90% | Monitor |
| Critical | < 90% | < 95% | Scale or free memory |
| Emergency | N/A | < 98% | Kill processes |

#### Disk Space
**SLI**: Percentage of disk space available
**Measurement**: `100 * node_filesystem_avail_bytes / node_filesystem_size_bytes`

| Threshold | SLO | SLA | Action |
|-----------|-----|-----|--------|
| Warning | > 15% | > 10% | Archive or cleanup |
| Critical | > 10% | > 5% | Emergency cleanup |
| Emergency | N/A | > 2% | Stop non-critical services |

### 6.3 Database Performance

#### PostgreSQL Query Latency
**SLI**: Time to execute database queries
**Measurement**: `histogram_quantile(0.95, rate(database_query_duration_seconds_bucket[5m]))`

| Query Type | P95 SLO | P95 SLA |
|------------|---------|---------|
| Simple SELECT | < 10ms | < 20ms |
| Complex JOIN | < 50ms | < 100ms |
| Aggregation | < 100ms | < 200ms |
| Write Operations | < 30ms | < 60ms |

#### Connection Pool Health
**SLI**: Percentage of connection pool capacity available
**Measurement**: `100 * (1 - pg_stat_database_numbackends / pg_settings_max_connections)`

| Metric | SLO | SLA |
|--------|-----|-----|
| Available Capacity | > 20% | > 10% |
| Pool Exhaustion | Never | < 1 minute/day |

### 6.4 Cache Performance

#### Redis Hit Rate
**SLI**: Percentage of cache requests that hit
**Measurement**: `100 * rate(redis_keyspace_hits_total[5m]) / (rate(redis_keyspace_hits_total[5m]) + rate(redis_keyspace_misses_total[5m]))`

| Cache Type | SLO | SLA | Typical Workload |
|------------|-----|-----|------------------|
| RBAC Permissions | > 95% | > 90% | Frequent auth checks |
| JWT Revocation | > 99% | > 95% | All requests |
| Model Metadata | > 85% | > 80% | ML inference |

---

## 7. Error Budgets

### 7.1 Monthly Error Budgets

Error budgets define the acceptable amount of unreliability. Budget is consumed by downtime, high latency, or errors.

| Service | SLA | Error Budget (30 days) | Budget Consumption |
|---------|-----|------------------------|-------------------|
| Trading Service | 99.95% | 21.6 minutes | 1 minute = 4.6% |
| API Gateway | 99.9% | 43.2 minutes | 1 minute = 2.3% |
| ML Training | 99.5% | 3.6 hours | 1 minute = 0.46% |
| Backtesting | 99.0% | 7.2 hours | 1 minute = 0.23% |

### 7.2 Error Budget Policies

**50% Budget Consumed** (Warning):
- Incident review required
- Root cause analysis initiated
- Preventative measures planned

**75% Budget Consumed** (Critical):
- Feature freeze (non-critical deployments halted)
- Focus on reliability improvements
- Daily status reviews

**100% Budget Consumed** (Emergency):
- All non-emergency deployments blocked
- All engineering resources on reliability
- Executive escalation
- SLA violation consequences triggered

---

## 8. Compliance & Audit SLAs

### 8.1 Audit Trail Completeness

#### Audit Event Capture Rate
**SLI**: Percentage of critical operations logged
**Measurement**: `100 * rate(audit_events_logged_total[1h]) / rate(critical_operations_total[1h])`

| Metric | SLO | SLA | Regulatory Requirement |
|--------|-----|-----|------------------------|
| Capture Rate | 100% | 99.99% | SOX, MiFID II |
| Missing Events | 0 | < 10/day | Critical threshold |

#### Audit Log Retention
**SLI**: Percentage of required logs available
**Measurement**: Manual quarterly audit

| Period | SLO | SLA | Notes |
|--------|-----|-----|-------|
| 7 years | 100% | 100% | Legal requirement |
| Integrity | 100% | 100% | Tamper-proof |

---

## 9. Monitoring & Alerting SLAs

### 9.1 Metrics Collection

#### Metrics Ingestion Rate
**SLI**: Percentage of metrics successfully ingested
**Measurement**: `100 * prometheus_tsdb_head_samples_appended_total / prometheus_tsdb_head_samples_total`

| Metric | SLO | SLA |
|--------|-----|-----|
| Ingestion Success Rate | > 99.9% | > 99.5% |
| Data Loss | < 0.1% | < 0.5% |

#### Alert Latency
**SLI**: Time from condition occurrence to alert firing
**Measurement**: Manual testing + `prometheus_notifications_latency_seconds`

| Alert Severity | SLO | SLA |
|----------------|-----|-----|
| Critical | < 10s | < 30s |
| High | < 30s | < 60s |
| Warning | < 60s | < 120s |

### 9.2 Alert Accuracy

#### False Positive Rate
**SLI**: Percentage of alerts that are not actionable
**Measurement**: Manual weekly review

| Metric | SLO | SLA |
|--------|-----|-----|
| False Positive Rate | < 5% | < 10% |
| Alert Fatigue Prevention | < 10 alerts/hour | < 20 alerts/hour |

---

## 10. SLA Reporting & Review

### 10.1 Reporting Schedule

| Report Type | Frequency | Audience | Contents |
|-------------|-----------|----------|----------|
| Real-time Dashboard | Continuous | Engineering | Current SLI values, active alerts |
| Daily Report | Daily 9AM | Engineering, Management | 24h SLO compliance, incidents |
| Weekly Summary | Monday 9AM | Management | 7-day trends, error budget |
| Monthly Review | 1st of month | Executive | 30-day compliance, SLA violations |
| Quarterly Business Review | End of quarter | Board, Investors | 90-day performance, budget status |

### 10.2 SLA Review Cadence

- **Monthly**: Review SLI/SLO thresholds for accuracy
- **Quarterly**: Adjust SLAs based on business needs
- **Annually**: Complete SLA framework review

### 10.3 Breach Notification

**Immediate (< 15 minutes)**:
- Critical SLA violations (Trading Service down, 99.95% breach)
- Security incidents
- Data loss events

**Same Day (< 4 hours)**:
- High-severity SLA violations
- Error budget > 75% consumed

**Weekly Summary**:
- Warning-level SLO misses
- Trend analysis

---

## 11. Contact & Escalation

### 11.1 Ownership

| Component | Primary Owner | Secondary Owner |
|-----------|---------------|-----------------|
| Trading Service | Trading Platform Team | SRE Team |
| API Gateway | Platform Team | Security Team |
| ML Training | ML Engineering | Data Science |
| Backtesting | Strategy Team | ML Engineering |
| Infrastructure | SRE Team | DevOps |

### 11.2 Escalation Path

**Level 1**: On-call Engineer (immediate response)
**Level 2**: Service Owner / Tech Lead (< 15 min)
**Level 3**: Engineering Manager (< 30 min)
**Level 4**: VP Engineering (< 1 hour)
**Level 5**: CTO / CEO (critical business impact)

### 11.3 Incident Communication

- **Status Page**: https://status.foxhunt.io
- **PagerDuty**: Critical alerts
- **Slack**: #incidents channel
- **Email**: incidents@foxhunt.io

---

## Appendix A: Metric Reference

### Prometheus Metric Names

| Service | Latency Metric | Throughput Metric | Error Metric |
|---------|----------------|-------------------|--------------|
| Trading | `trading_order_processing_microseconds` | `trading_orders_submitted_total` | `trading_errors_total` |
| API Gateway | `api_gateway_auth_total_duration_microseconds` | `api_gateway_requests_total` | `api_gateway_auth_errors_total` |
| ML Training | `ml_inference_duration_milliseconds` | `ml_predictions_total` | `ml_prediction_errors_total` |
| Backtesting | `backtesting_execution_duration_seconds` | `backtesting_events_processed_total` | `backtesting_simulation_errors_total` |

### Dashboard Links

- Trading Performance: http://grafana.foxhunt.io/d/trading-performance
- System Health: http://grafana.foxhunt.io/d/system-health
- ML Monitoring: http://grafana.foxhunt.io/d/ml-monitoring
- Compliance: http://grafana.foxhunt.io/d/compliance-audit

---

**Document Version History**:
- v1.0 (2025-10-07): Initial SLA definitions
- Next Review: 2025-11-07