foxhunt/MONITORING_QUICKSTART.md

Foxhunt Monitoring Stack - Quick Reference

Quick Access URLs

Service	URL	Credentials
Grafana	http://localhost:3000	admin/foxhunt123
Prometheus	http://localhost:9090	None
InfluxDB	http://localhost:8086	foxhunt/foxhunt_dev_password

Service Metrics Endpoints

Service	Metrics URL	Scrape Latency
API Gateway	http://localhost:9091/metrics	1.0ms
Trading Service	http://localhost:9092/metrics	0.4ms
Backtesting Service	http://localhost:9093/metrics	0.4ms
ML Training Service	http://localhost:9094/metrics	0.5ms

Health Check Commands

# Check all Docker services
docker-compose ps

# Check Prometheus targets
curl http://localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | "\(.labels.job) - \(.health)"'

# Check Grafana health
curl http://localhost:3000/api/health

# Count active metrics
curl http://localhost:9090/api/v1/label/__name__/values | jq -r '.data | length'

Common Prometheus Queries

Service Health

# All services up
up

# Count healthy services
count(up == 1)

# Service uptime
trading_service_uptime_seconds
backtesting_service_uptime_seconds
ml_training_service_uptime_seconds

Authentication Metrics

# JWT token count
api_gateway_active_jwt_tokens

# Auth errors
rate(api_gateway_auth_errors_total[5m])

# Auth latency P95
histogram_quantile(0.95, api_gateway_auth_total_duration_microseconds_bucket)

Trading Metrics

# Order processing latency
trading_order_processing_seconds

# Risk check latency
trading_risk_check_seconds

# Total trading latency
trading_total_latency_seconds

Backtesting Metrics

# Backtests per minute
rate(backtesting_backtests_started_total[1m])

# Backtest completion rate
rate(backtesting_backtests_completed_total[1m])

# Error rate
rate(backtesting_errors_total[5m])

ML Training Metrics

# Training jobs per hour
rate(ml_training_jobs_started_total[1h])

# Job completion rate
rate(ml_training_jobs_completed_total[1h])

# Error rate
rate(ml_training_errors_total[5m])

Grafana Dashboard Access

Active Dashboards

1. Foxhunt System Overview - http://localhost:3000/d/foxhunt-system-overview
   • CPU, Memory, Disk, Service Status
2. Foxhunt Trading Overview - http://localhost:3000/d/foxhunt-trading-overview
   • Trading metrics, performance

Available Dashboards (12+ JSON files in config/grafana/dashboards/)

• hft-trading-performance.json (default home)
• hft-risk-management.json
• hft-latency-monitor.json
• hft-system-health.json
• api-gateway-overview.json
• trading-service.json
• ml-training-monitoring.json
• And 5 more...

Alert Rules

Configured Alert Groups (7)

1. foxhunt-database-alerts - PostgreSQL health, connections
2. foxhunt-trading-alerts - Trading errors, latency SLA
3. foxhunt-trading-metrics - Order metrics, PnL thresholds
4. foxhunt-database-health - Database query performance
5. foxhunt-grpc-health - gRPC service health
6. foxhunt-service-health - Service uptime, restarts
7. foxhunt-system-resources - CPU, memory, disk

Troubleshooting

No Metrics Showing Up

# Check if Prometheus is scraping
curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.health!="up")'

# Check service metrics endpoint directly
curl http://localhost:9091/metrics | head -20

# Restart Prometheus
docker-compose restart prometheus

Grafana Dashboard Not Loading

# Check Grafana logs
docker logs foxhunt-grafana | tail -50

# Verify datasource connectivity
curl -u admin:foxhunt123 http://localhost:3000/api/datasources

# Test Prometheus datasource
curl -u admin:foxhunt123 http://localhost:3000/api/datasources/1/health

High Scrape Latency

# Check scrape duration per target
curl http://localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | "\(.labels.job): \(.lastScrapeDuration)"'

# Check service health
docker-compose ps

# Check service logs for errors
docker logs foxhunt-trading-service | tail -50

Configuration Files

Prometheus

• Active Config: /home/jgrusewski/Work/foxhunt/config/prometheus/prometheus.yml
• Docker Mount: /etc/prometheus/prometheus.yml (inside container)
• Reload Config: curl -X POST http://localhost:9090/-/reload

Grafana

• Datasources: /home/jgrusewski/Work/foxhunt/config/grafana/provisioning/datasources/datasources.yml
• Dashboards: /home/jgrusewski/Work/foxhunt/config/grafana/provisioning/dashboards/dashboards.yml
• Dashboard Files: /home/jgrusewski/Work/foxhunt/config/grafana/dashboards/*.json

Performance Targets (All Met ✅)

Metric	Target	Actual	Status
API Gateway Scrape	<10ms	1.0ms	✅
Trading Service Scrape	<10ms	0.4ms	✅
Backtesting Service Scrape	<10ms	0.4ms	✅
ML Training Service Scrape	<10ms	0.5ms	✅
Postgres Exporter Scrape	<500ms	282ms	✅
Prometheus Scrape	<50ms	13ms	✅

Metrics Retention

• Prometheus: 15 days
• InfluxDB: 30 days
• PostgreSQL: Unlimited (manual cleanup)

Backup and Restore

Grafana Dashboards

# Export dashboard
curl -u admin:foxhunt123 http://localhost:3000/api/dashboards/uid/foxhunt-system-overview > dashboard_backup.json

# Import dashboard
curl -u admin:foxhunt123 -X POST -H "Content-Type: application/json" -d @dashboard_backup.json http://localhost:3000/api/dashboards/db

Prometheus Data

# Prometheus data is stored in Docker volume
docker volume inspect foxhunt_prometheus_data

# Backup
docker run --rm -v foxhunt_prometheus_data:/data -v $(pwd):/backup alpine tar czf /backup/prometheus_backup.tar.gz /data

Production Checklist

• ✅ All 6 Prometheus targets up
• ✅ All scrape latencies under target
• ✅ 794 unique metrics collected
• ✅ Grafana healthy with 4 datasources
• ✅ 7 alert rule groups configured
• ✅ 15-day retention configured
• ✅ All Docker services healthy

Support

For issues or questions:

1. Check service logs: docker logs foxhunt-<service>
2. Verify Prometheus targets: http://localhost:9090/targets
3. Check Grafana datasources: http://localhost:3000/datasources
4. Review this document: /home/jgrusewski/Work/foxhunt/WAVE_16_AGENT_16.16_MONITORING_STACK_VALIDATION.md