foxhunt/docs/monitoring/LOG_AGGREGATION.md

# Log Aggregation Infrastructure - Foxhunt HFT

**Last Updated**: 2025-10-07
**Version**: 1.0
**Log Stack**: Loki + Promtail + Grafana

---

## 1. Overview

Foxhunt uses **Grafana Loki** for centralized log aggregation with **Promtail** for log shipping. This provides:
- Centralized log storage and search
- Integration with Grafana dashboards
- Long-term retention for compliance (7 years)
- High-performance log ingestion (100+ MB/s)

### 1.1 Architecture

```
┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Service   │────▶│  Promtail   │────▶│    Loki     │
│    Logs     │     │  (Shipper)  │     │  (Storage)  │
└─────────────┘     └─────────────┘     └─────────────┘
                                              │
                                              ▼
                                        ┌─────────────┐
                                        │   Grafana   │
                                        │   (Query)   │
                                        └─────────────┘
```

---

## 2. Log Retention Policies

### 2.1 Retention Periods

| Log Type | Retention | Location | Purpose |
|----------|-----------|----------|---------|
| **Audit Logs** | 7 years | S3 + Loki | SOX/MiFID II compliance |
| **Trading Logs** | 90 days | Loki | Operational analysis |
| **Debug Logs** | 30 days | Loki | Troubleshooting |
| **Access Logs** | 90 days | Loki | Security analysis |
| **System Logs** | 30 days | Loki | Infrastructure monitoring |

### 2.2 Retention Configuration

**Loki retention** (deployment/monitoring/loki-config.yml):
```yaml
limits_config:
  retention_period: 2160h  # 90 days for most logs

# Separate retention for audit logs
table_manager:
  retention_deletes_enabled: true
  retention_period: 61320h  # 7 years (2557 days)

  # Label-based retention
  retention_stream:
    - selector: '{log_type="audit"}'
      period: 61320h  # 7 years
    - selector: '{log_type="trading"}'
      period: 2160h   # 90 days
    - selector: '{log_type="debug"}'
      period: 720h    # 30 days
```

---

## 3. Log Format Standards

### 3.1 Structured JSON Logging

All services MUST log in structured JSON format:

```json
{
  "timestamp": "2025-10-07T14:23:45.123Z",
  "level": "INFO|WARN|ERROR|DEBUG",
  "service": "trading_service",
  "component": "order_manager",
  "message": "Order submitted successfully",
  "context": {
    "order_id": "ORD-12345",
    "symbol": "AAPL",
    "quantity": 100,
    "price": 150.25
  },
  "trace_id": "abc123def456",
  "span_id": "span789",
  "user_id": "user@example.com",
  "request_id": "req-xyz789"
}
```

### 3.2 Log Levels

| Level | Usage | Examples |
|-------|-------|----------|
| **ERROR** | Errors requiring immediate attention | Order submission failed, database connection lost |
| **WARN** | Potential issues, degraded performance | High latency detected, approaching limits |
| **INFO** | Important operational events | Order placed, position opened, config reloaded |
| **DEBUG** | Detailed diagnostic information | Request parameters, internal state changes |
| **TRACE** | Very verbose debugging | Function calls, variable values |

### 3.3 Required Fields

Every log entry MUST include:
- `timestamp` (ISO 8601 format with milliseconds)
- `level` (ERROR/WARN/INFO/DEBUG/TRACE)
- `service` (service name)
- `message` (human-readable description)

Recommended fields:
- `trace_id` (distributed tracing)
- `span_id` (tracing span)
- `user_id` (for user actions)
- `request_id` (for request correlation)
- `component` (internal component name)

---

## 4. Log Search & Query

### 4.1 Loki Query Language (LogQL)

#### Basic Queries

**Get logs from specific service**:
```logql
{service="trading_service"}
```

**Filter by log level**:
```logql
{service="trading_service", level="ERROR"}
```

**Time range**:
```logql
{service="trading_service"} [5m]
```

**Multiple services**:
```logql
{service=~"trading_service|ml_training_service"}
```

#### Advanced Queries

**Search for specific text**:
```logql
{service="trading_service"} |= "order_id"
```

**Regex search**:
```logql
{service="trading_service"} |~ "order_id: ORD-[0-9]+"
```

**JSON field extraction**:
```logql
{service="trading_service"} | json | order_id="ORD-12345"
```

**Rate of errors**:
```logql
rate({service="trading_service", level="ERROR"}[5m])
```

**Count by component**:
```logql
sum by (component) (count_over_time({service="trading_service"}[5m]))
```

### 4.2 Common Search Examples

#### Find all errors in last hour
```logql
{level="ERROR"} [1h]
```

#### Find high latency orders
```logql
{service="trading_service"} | json | latency_ms > 100
```

#### Track specific user activity
```logql
{user_id="user@example.com"} [24h]
```

#### Find failed authentications
```logql
{service="api_gateway", component="auth"} |= "authentication failed"
```

#### Database connection errors
```logql
{service=~".*_service"} |= "database" |= "connection" |= "error"
```

---

## 5. Log Access & Tools

### 5.1 Grafana Explore

**Access**: http://grafana.foxhunt.io/explore

1. Select "Loki" as data source
2. Enter LogQL query
3. Select time range
4. Click "Run Query"
5. Use filters to narrow results

**Tips**:
- Use `Ctrl+Space` for autocomplete
- Save frequent queries as favorites
- Export results to CSV for analysis

### 5.2 Command Line (logcli)

Install logcli:
```bash
brew install logcli  # macOS
# OR
go install github.com/grafana/loki/cmd/logcli@latest
```

Configure:
```bash
export LOKI_ADDR=http://loki.foxhunt.io:3100
export LOKI_USERNAME=<username>
export LOKI_PASSWORD=<password>
```

Query logs:
```bash
# Last 5 minutes of errors
logcli query '{level="ERROR"}' --since=5m

# Follow logs in real-time
logcli query '{service="trading_service"}' --tail

# Export to JSON
logcli query '{service="trading_service"}' --since=1h --output=jsonl > logs.json

# Count log entries
logcli series '{service="trading_service"}'
```

### 5.3 API Access

Query Loki API directly:
```bash
# Range query
curl -G -s "http://loki.foxhunt.io:3100/loki/api/v1/query_range" \
  --data-urlencode 'query={service="trading_service"}' \
  --data-urlencode 'start=1696680000000000000' \
  --data-urlencode 'end=1696683600000000000' | jq

# Instant query
curl -G -s "http://loki.foxhunt.io:3100/loki/api/v1/query" \
  --data-urlencode 'query={service="trading_service"}' \
  --data-urlencode 'time=1696680000000000000' | jq

# Label values
curl -s "http://loki.foxhunt.io:3100/loki/api/v1/label/service/values" | jq
```

---

## 6. Log-Based Alerting

### 6.1 Loki Alerting Rules

**Configure in Loki** (deployment/monitoring/loki-config.yml):

```yaml
ruler:
  storage:
    type: local
    local:
      directory: /loki/rules
  rule_path: /tmp/loki-rules
  alertmanager_url: http://alertmanager:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true
```

**Create alert rule** (loki/rules/trading_alerts.yml):

```yaml
groups:
  - name: trading_errors
    interval: 1m
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate({service="trading_service", level="ERROR"}[5m])) > 10
        for: 2m
        labels:
          severity: critical
          service: trading
        annotations:
          summary: "High error rate in trading service"
          description: "{{ $value }} errors/sec detected"

      - alert: AuthenticationFailures
        expr: |
          sum(rate({service="api_gateway"} |= "authentication failed"[5m])) > 5
        for: 1m
        labels:
          severity: warning
          service: api_gateway
        annotations:
          summary: "High authentication failure rate"
          description: "{{ $value }} failures/sec detected"
```

### 6.2 Common Alert Patterns

**High error rate**:
```logql
sum(rate({service="trading_service", level="ERROR"}[5m])) > 10
```

**Missing logs** (service might be down):
```logql
absent_over_time({service="trading_service"}[5m])
```

**Specific error pattern**:
```logql
count_over_time({service="trading_service"} |= "database connection failed"[5m]) > 0
```

**Slow requests**:
```logql
sum(rate({service="api_gateway"} | json | latency_ms > 100[5m])) > 5
```

---

## 7. Log Archival to S3

### 7.1 Audit Log Archival

**Purpose**: Long-term storage for compliance (7 years)

**Configuration** (deployment/monitoring/loki-config.yml):
```yaml
storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    shared_store: s3

  aws:
    s3: s3://foxhunt-audit-logs
    region: us-east-1
    access_key_id: ${AWS_ACCESS_KEY_ID}
    secret_access_key: ${AWS_SECRET_ACCESS_KEY}
```

### 7.2 Archival Schedule

**Daily archival** (cronjob):
```bash
#!/bin/bash
# Archive audit logs to S3 daily

DATE=$(date -d "yesterday" +%Y-%m-%d)
BUCKET="s3://foxhunt-audit-logs"

# Export audit logs from Loki
logcli query '{log_type="audit"}' \
  --since="${DATE}T00:00:00Z" \
  --until="${DATE}T23:59:59Z" \
  --output=jsonl \
  --limit=1000000 \
  > /tmp/audit_logs_${DATE}.jsonl

# Compress
gzip /tmp/audit_logs_${DATE}.jsonl

# Upload to S3
aws s3 cp /tmp/audit_logs_${DATE}.jsonl.gz \
  ${BUCKET}/${DATE}/audit_logs_${DATE}.jsonl.gz \
  --storage-class GLACIER_IR

# Cleanup
rm /tmp/audit_logs_${DATE}.jsonl.gz

# Verify upload
aws s3 ls ${BUCKET}/${DATE}/
```

**Cron schedule** (daily at 2 AM):
```cron
0 2 * * * /opt/foxhunt/scripts/archive_logs.sh
```

### 7.3 S3 Lifecycle Policies

```json
{
  "Rules": [
    {
      "Id": "AuditLogRetention",
      "Status": "Enabled",
      "Prefix": "",
      "Transitions": [
        {
          "Days": 90,
          "StorageClass": "GLACIER_IR"
        },
        {
          "Days": 365,
          "StorageClass": "DEEP_ARCHIVE"
        }
      ],
      "Expiration": {
        "Days": 2557
      }
    }
  ]
}
```

**Cost optimization**:
- 0-90 days: S3 Standard (frequent access for investigations)
- 90-365 days: Glacier Instant Retrieval (occasional access)
- 1-7 years: Glacier Deep Archive (compliance only, rarely accessed)
- After 7 years: Auto-delete (compliance requirement met)

---

## 8. Performance Tuning

### 8.1 Loki Performance

**Ingestion rate** (per tenant):
```yaml
limits_config:
  ingestion_rate_mb: 100        # 100 MB/s sustained
  ingestion_burst_size_mb: 200  # 200 MB/s burst
  max_line_size: 256KB          # Maximum log line size
```

**Query optimization**:
```yaml
query_range:
  max_retries: 5
  parallelise_shardable_queries: true
  cache_results: true
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 1000  # 1 GB cache
```

**Indexing**:
```yaml
schema_config:
  configs:
    - from: 2025-01-01
      store: tsdb           # Time-series database (faster)
      object_store: s3
      schema: v12
      index:
        prefix: foxhunt_
        period: 24h         # Daily index rotation
```

### 8.2 Promtail Configuration

**High-performance log shipping** (promtail-config.yml):

```yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push
    batchwait: 1s          # Batch logs for 1 second
    batchsize: 1048576     # 1 MB batches
    timeout: 10s

scrape_configs:
  - job_name: trading_service
    static_configs:
      - targets:
          - localhost
        labels:
          job: trading_service
          __path__: /var/log/trading_service/*.log

    pipeline_stages:
      # Parse JSON logs
      - json:
          expressions:
            timestamp: timestamp
            level: level
            message: message
            service: service

      # Add labels
      - labels:
          level:
          service:

      # Parse timestamps
      - timestamp:
          source: timestamp
          format: RFC3339

      # Drop debug logs in production
      - match:
          selector: '{level="DEBUG"}'
          action: drop
```

### 8.3 Monitoring Loki

**Key metrics**:
```promql
# Ingestion rate
rate(loki_ingester_chunks_created_total[1m])

# Query latency
histogram_quantile(0.99, rate(loki_request_duration_seconds_bucket[5m]))

# Storage usage
loki_ingester_memory_chunks

# Active streams
loki_ingester_streams
```

---

## 9. Troubleshooting

### 9.1 Common Issues

#### Logs not appearing in Grafana

**Check Promtail**:
```bash
# Check Promtail status
curl http://promtail:9080/metrics | grep promtail_targets_active_total

# Check Promtail logs
docker logs promtail --tail=50
```

**Check Loki ingestion**:
```bash
# Check ingestion rate
curl http://loki:3100/metrics | grep loki_distributor_bytes_received_total
```

**Check labels**:
```bash
# List all label names
curl http://loki:3100/loki/api/v1/labels | jq

# List values for specific label
curl http://loki:3100/loki/api/v1/label/service/values | jq
```

#### Slow queries

**Reduce time range**:
- Query smaller time windows
- Use `--limit` to cap results

**Add more specific labels**:
```logql
# Bad (scans everything)
{} |= "error"

# Good (scans only trading service)
{service="trading_service"} |= "error"
```

**Use metrics instead of logs**:
- For aggregations, use Prometheus metrics
- Logs are for debugging, not dashboards

#### Out of disk space

**Check Loki storage**:
```bash
du -sh /loki/chunks
du -sh /loki/index
```

**Manually clean old data**:
```bash
# Delete chunks older than 90 days
find /loki/chunks -mtime +90 -delete

# Compact indexes
curl -X POST http://loki:3100/loki/api/v1/push/compact
```

---

## 10. Best Practices

### 10.1 Logging Guidelines

**DO**:
- ✅ Use structured JSON logging
- ✅ Include correlation IDs (trace_id, request_id)
- ✅ Log important business events (orders, trades)
- ✅ Log errors with full context
- ✅ Use appropriate log levels
- ✅ Add timestamps in ISO 8601 format
- ✅ Include service and component labels

**DON'T**:
- ❌ Log sensitive data (passwords, API keys, PII)
- ❌ Log excessively (every function call)
- ❌ Use unstructured text logs
- ❌ Log binary data
- ❌ Log without context
- ❌ Use printf-style formatting (use structured fields)

### 10.2 Performance Tips

1. **Use label cardinality wisely**:
   - Low cardinality labels: service, level, component (< 100 values)
   - High cardinality as fields: user_id, order_id, request_id

2. **Batch log writes**:
   - Don't send individual log lines
   - Buffer and send in batches (1 second or 1 MB)

3. **Compress logs**:
   - Promtail automatically compresses with snappy
   - Use gzip for archival

4. **Index appropriately**:
   - Too many indexes → slow writes
   - Too few indexes → slow queries
   - Daily rotation is good balance

### 10.3 Security

**Access Control**:
- Grafana authentication required for log access
- Separate credentials for production/staging
- Audit who accessed sensitive logs

**Data Protection**:
- Encrypt logs in transit (TLS)
- Encrypt logs at rest (S3 encryption)
- Redact sensitive fields before logging

**Compliance**:
- Audit logs are immutable (write-once)
- Tamper detection via checksums
- Regular compliance audits

---

## 11. Quick Reference

### 11.1 Service Endpoints

| Service | URL | Port |
|---------|-----|------|
| Loki | http://loki.foxhunt.io | 3100 |
| Promtail | http://promtail.foxhunt.io | 9080 |
| Grafana Explore | http://grafana.foxhunt.io/explore | 3000 |

### 11.2 Key Commands

```bash
# Query last 5 minutes
logcli query '{service="trading_service"}' --since=5m

# Follow logs live
logcli query '{service="trading_service"}' --tail

# Count errors
logcli query 'sum(count_over_time({level="ERROR"}[1h]))'

# Export logs
logcli query '{service="trading_service"}' --since=1h -o jsonl > logs.json
```

### 11.3 Support

- **Documentation**: https://grafana.com/docs/loki/latest/
- **Slack**: #observability
- **Email**: oncall@foxhunt.io
- **Escalation**: See RUNBOOKS.md

---

**Document Version History**:
- v1.0 (2025-10-07): Initial log aggregation documentation
- Next Review: 2025-11-07