# Foxhunt Monitoring Stack - Quick Reference

## Quick Access URLs

| Service | URL | Credentials |
|---------|-----|-------------|
| Grafana | http://localhost:3000 | admin/foxhunt123 |
| Prometheus | http://localhost:9090 | None |
| InfluxDB | http://localhost:8086 | foxhunt/foxhunt_dev_password |

## Service Metrics Endpoints

| Service | Metrics URL | Scrape Latency |
|---------|-------------|----------------|
| API Gateway | http://localhost:9091/metrics | 1.0ms |
| Trading Service | http://localhost:9092/metrics | 0.4ms |
| Backtesting Service | http://localhost:9093/metrics | 0.4ms |
| ML Training Service | http://localhost:9094/metrics | 0.5ms |

## Health Check Commands

```bash
# Check all Docker services
docker-compose ps

# Check Prometheus targets
curl http://localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | "\(.labels.job) - \(.health)"'

# Check Grafana health
curl http://localhost:3000/api/health

# Count active metrics
curl http://localhost:9090/api/v1/label/__name__/values | jq -r '.data | length'
```

## Common Prometheus Queries

### Service Health
```promql
# All services up
up

# Count healthy services
count(up == 1)

# Service uptime
trading_service_uptime_seconds
backtesting_service_uptime_seconds
ml_training_service_uptime_seconds
```

### Authentication Metrics
```promql
# JWT token count
api_gateway_active_jwt_tokens

# Auth errors
rate(api_gateway_auth_errors_total[5m])

# Auth latency P95
histogram_quantile(0.95, api_gateway_auth_total_duration_microseconds_bucket)
```

### Trading Metrics
```promql
# Order processing latency
trading_order_processing_seconds

# Risk check latency
trading_risk_check_seconds

# Total trading latency
trading_total_latency_seconds
```

### Backtesting Metrics
```promql
# Backtests per minute
rate(backtesting_backtests_started_total[1m])

# Backtest completion rate
rate(backtesting_backtests_completed_total[1m])

# Error rate
rate(backtesting_errors_total[5m])
```

### ML Training Metrics
```promql
# Training jobs per hour
rate(ml_training_jobs_started_total[1h])

# Job completion rate
rate(ml_training_jobs_completed_total[1h])

# Error rate
rate(ml_training_errors_total[5m])
```

## Grafana Dashboard Access

### Active Dashboards
1. **Foxhunt System Overview** - http://localhost:3000/d/foxhunt-system-overview
   - CPU, Memory, Disk, Service Status
2. **Foxhunt Trading Overview** - http://localhost:3000/d/foxhunt-trading-overview
   - Trading metrics, performance

### Available Dashboards (12+ JSON files in config/grafana/dashboards/)
- hft-trading-performance.json (default home)
- hft-risk-management.json
- hft-latency-monitor.json
- hft-system-health.json
- api-gateway-overview.json
- trading-service.json
- ml-training-monitoring.json
- And 5 more...

## Alert Rules

### Configured Alert Groups (7)
1. **foxhunt-database-alerts** - PostgreSQL health, connections
2. **foxhunt-trading-alerts** - Trading errors, latency SLA
3. **foxhunt-trading-metrics** - Order metrics, PnL thresholds
4. **foxhunt-database-health** - Database query performance
5. **foxhunt-grpc-health** - gRPC service health
6. **foxhunt-service-health** - Service uptime, restarts
7. **foxhunt-system-resources** - CPU, memory, disk

## Troubleshooting

### No Metrics Showing Up
```bash
# Check if Prometheus is scraping
curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.health!="up")'

# Check service metrics endpoint directly
curl http://localhost:9091/metrics | head -20

# Restart Prometheus
docker-compose restart prometheus
```

### Grafana Dashboard Not Loading
```bash
# Check Grafana logs
docker logs foxhunt-grafana | tail -50

# Verify datasource connectivity
curl -u admin:foxhunt123 http://localhost:3000/api/datasources

# Test Prometheus datasource
curl -u admin:foxhunt123 http://localhost:3000/api/datasources/1/health
```

### High Scrape Latency
```bash
# Check scrape duration per target
curl http://localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | "\(.labels.job): \(.lastScrapeDuration)"'

# Check service health
docker-compose ps

# Check service logs for errors
docker logs foxhunt-trading-service | tail -50
```

## Configuration Files

### Prometheus
- **Active Config**: `/home/jgrusewski/Work/foxhunt/config/prometheus/prometheus.yml`
- **Docker Mount**: `/etc/prometheus/prometheus.yml` (inside container)
- **Reload Config**: `curl -X POST http://localhost:9090/-/reload`

### Grafana
- **Datasources**: `/home/jgrusewski/Work/foxhunt/config/grafana/provisioning/datasources/datasources.yml`
- **Dashboards**: `/home/jgrusewski/Work/foxhunt/config/grafana/provisioning/dashboards/dashboards.yml`
- **Dashboard Files**: `/home/jgrusewski/Work/foxhunt/config/grafana/dashboards/*.json`

## Performance Targets (All Met ✅)

| Metric | Target | Actual | Status |
|--------|--------|--------|--------|
| API Gateway Scrape | <10ms | 1.0ms | ✅ |
| Trading Service Scrape | <10ms | 0.4ms | ✅ |
| Backtesting Service Scrape | <10ms | 0.4ms | ✅ |
| ML Training Service Scrape | <10ms | 0.5ms | ✅ |
| Postgres Exporter Scrape | <500ms | 282ms | ✅ |
| Prometheus Scrape | <50ms | 13ms | ✅ |

## Metrics Retention

- **Prometheus**: 15 days
- **InfluxDB**: 30 days
- **PostgreSQL**: Unlimited (manual cleanup)

## Backup and Restore

### Grafana Dashboards
```bash
# Export dashboard
curl -u admin:foxhunt123 http://localhost:3000/api/dashboards/uid/foxhunt-system-overview > dashboard_backup.json

# Import dashboard
curl -u admin:foxhunt123 -X POST -H "Content-Type: application/json" -d @dashboard_backup.json http://localhost:3000/api/dashboards/db
```

### Prometheus Data
```bash
# Prometheus data is stored in Docker volume
docker volume inspect foxhunt_prometheus_data

# Backup
docker run --rm -v foxhunt_prometheus_data:/data -v $(pwd):/backup alpine tar czf /backup/prometheus_backup.tar.gz /data
```

## Production Checklist

- ✅ All 6 Prometheus targets up
- ✅ All scrape latencies under target
- ✅ 794 unique metrics collected
- ✅ Grafana healthy with 4 datasources
- ✅ 7 alert rule groups configured
- ✅ 15-day retention configured
- ✅ All Docker services healthy

## Support

For issues or questions:
1. Check service logs: `docker logs foxhunt-<service>`
2. Verify Prometheus targets: http://localhost:9090/targets
3. Check Grafana datasources: http://localhost:3000/datasources
4. Review this document: `/home/jgrusewski/Work/foxhunt/WAVE_16_AGENT_16.16_MONITORING_STACK_VALIDATION.md`