foxhunt/docs/monitoring/LOG_AGGREGATION.md

Log Aggregation Infrastructure - Foxhunt HFT

Last Updated: 2025-10-07 Version: 1.0 Log Stack: Loki + Promtail + Grafana

---

1. Overview

Foxhunt uses Grafana Loki for centralized log aggregation with Promtail for log shipping. This provides:

• Centralized log storage and search
• Integration with Grafana dashboards
• Long-term retention for compliance (7 years)
• High-performance log ingestion (100+ MB/s)

1.1 Architecture

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Service   │────▶│  Promtail   │────▶│    Loki     │
│    Logs     │     │  (Shipper)  │     │  (Storage)  │
└─────────────┘     └─────────────┘     └─────────────┘
                                              │
                                              ▼
                                        ┌─────────────┐
                                        │   Grafana   │
                                        │   (Query)   │
                                        └─────────────┘

---

2. Log Retention Policies

2.1 Retention Periods

Log Type	Retention	Location	Purpose
Audit Logs	7 years	S3 + Loki	SOX/MiFID II compliance
Trading Logs	90 days	Loki	Operational analysis
Debug Logs	30 days	Loki	Troubleshooting
Access Logs	90 days	Loki	Security analysis
System Logs	30 days	Loki	Infrastructure monitoring

2.2 Retention Configuration

Loki retention (deployment/monitoring/loki-config.yml):

limits_config:
  retention_period: 2160h  # 90 days for most logs

# Separate retention for audit logs
table_manager:
  retention_deletes_enabled: true
  retention_period: 61320h  # 7 years (2557 days)

  # Label-based retention
  retention_stream:
    - selector: '{log_type="audit"}'
      period: 61320h  # 7 years
    - selector: '{log_type="trading"}'
      period: 2160h   # 90 days
    - selector: '{log_type="debug"}'
      period: 720h    # 30 days

---

3. Log Format Standards

3.1 Structured JSON Logging

All services MUST log in structured JSON format:

{
  "timestamp": "2025-10-07T14:23:45.123Z",
  "level": "INFO|WARN|ERROR|DEBUG",
  "service": "trading_service",
  "component": "order_manager",
  "message": "Order submitted successfully",
  "context": {
    "order_id": "ORD-12345",
    "symbol": "AAPL",
    "quantity": 100,
    "price": 150.25
  },
  "trace_id": "abc123def456",
  "span_id": "span789",
  "user_id": "user@example.com",
  "request_id": "req-xyz789"
}

3.2 Log Levels

Level	Usage	Examples
ERROR	Errors requiring immediate attention	Order submission failed, database connection lost
WARN	Potential issues, degraded performance	High latency detected, approaching limits
INFO	Important operational events	Order placed, position opened, config reloaded
DEBUG	Detailed diagnostic information	Request parameters, internal state changes
TRACE	Very verbose debugging	Function calls, variable values

3.3 Required Fields

Every log entry MUST include:

• timestamp (ISO 8601 format with milliseconds)
• level (ERROR/WARN/INFO/DEBUG/TRACE)
• service (service name)
• message (human-readable description)

Recommended fields:

• trace_id (distributed tracing)
• span_id (tracing span)
• user_id (for user actions)
• request_id (for request correlation)
• component (internal component name)

---

4. Log Search & Query

4.1 Loki Query Language (LogQL)

Basic Queries

Get logs from specific service:

{service="trading_service"}

Filter by log level:

{service="trading_service", level="ERROR"}

Time range:

{service="trading_service"} [5m]

Multiple services:

{service=~"trading_service|ml_training_service"}

Advanced Queries

Search for specific text:

{service="trading_service"} |= "order_id"

Regex search:

{service="trading_service"} |~ "order_id: ORD-[0-9]+"

JSON field extraction:

{service="trading_service"} | json | order_id="ORD-12345"

Rate of errors:

rate({service="trading_service", level="ERROR"}[5m])

Count by component:

sum by (component) (count_over_time({service="trading_service"}[5m]))

4.2 Common Search Examples

Find all errors in last hour

{level="ERROR"} [1h]

Find high latency orders

{service="trading_service"} | json | latency_ms > 100

Track specific user activity

{user_id="user@example.com"} [24h]

Find failed authentications

{service="api_gateway", component="auth"} |= "authentication failed"

Database connection errors

{service=~".*_service"} |= "database" |= "connection" |= "error"

---

5. Log Access & Tools

5.1 Grafana Explore

Access: http://grafana.foxhunt.io/explore

1. Select "Loki" as data source
2. Enter LogQL query
3. Select time range
4. Click "Run Query"
5. Use filters to narrow results

Tips:

• Use Ctrl+Space for autocomplete
• Save frequent queries as favorites
• Export results to CSV for analysis

5.2 Command Line (logcli)

Install logcli:

brew install logcli  # macOS
# OR
go install github.com/grafana/loki/cmd/logcli@latest

Configure:

export LOKI_ADDR=http://loki.foxhunt.io:3100
export LOKI_USERNAME=<username>
export LOKI_PASSWORD=<password>

Query logs:

# Last 5 minutes of errors
logcli query '{level="ERROR"}' --since=5m

# Follow logs in real-time
logcli query '{service="trading_service"}' --tail

# Export to JSON
logcli query '{service="trading_service"}' --since=1h --output=jsonl > logs.json

# Count log entries
logcli series '{service="trading_service"}'

5.3 API Access

Query Loki API directly:

# Range query
curl -G -s "http://loki.foxhunt.io:3100/loki/api/v1/query_range" \
  --data-urlencode 'query={service="trading_service"}' \
  --data-urlencode 'start=1696680000000000000' \
  --data-urlencode 'end=1696683600000000000' | jq

# Instant query
curl -G -s "http://loki.foxhunt.io:3100/loki/api/v1/query" \
  --data-urlencode 'query={service="trading_service"}' \
  --data-urlencode 'time=1696680000000000000' | jq

# Label values
curl -s "http://loki.foxhunt.io:3100/loki/api/v1/label/service/values" | jq

---

6. Log-Based Alerting

6.1 Loki Alerting Rules

Configure in Loki (deployment/monitoring/loki-config.yml):

ruler:
  storage:
    type: local
    local:
      directory: /loki/rules
  rule_path: /tmp/loki-rules
  alertmanager_url: http://alertmanager:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true

Create alert rule (loki/rules/trading_alerts.yml):

groups:
  - name: trading_errors
    interval: 1m
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate({service="trading_service", level="ERROR"}[5m])) > 10
        for: 2m
        labels:
          severity: critical
          service: trading
        annotations:
          summary: "High error rate in trading service"
          description: "{{ $value }} errors/sec detected"

      - alert: AuthenticationFailures
        expr: |
          sum(rate({service="api_gateway"} |= "authentication failed"[5m])) > 5
        for: 1m
        labels:
          severity: warning
          service: api_gateway
        annotations:
          summary: "High authentication failure rate"
          description: "{{ $value }} failures/sec detected"

6.2 Common Alert Patterns

High error rate:

sum(rate({service="trading_service", level="ERROR"}[5m])) > 10

Missing logs (service might be down):

absent_over_time({service="trading_service"}[5m])

Specific error pattern:

count_over_time({service="trading_service"} |= "database connection failed"[5m]) > 0

Slow requests:

sum(rate({service="api_gateway"} | json | latency_ms > 100[5m])) > 5

---

7. Log Archival to S3

7.1 Audit Log Archival

Purpose: Long-term storage for compliance (7 years)

Configuration (deployment/monitoring/loki-config.yml):

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    shared_store: s3

  aws:
    s3: s3://foxhunt-audit-logs
    region: us-east-1
    access_key_id: ${AWS_ACCESS_KEY_ID}
    secret_access_key: ${AWS_SECRET_ACCESS_KEY}

7.2 Archival Schedule

Daily archival (cronjob):

#!/bin/bash
# Archive audit logs to S3 daily

DATE=$(date -d "yesterday" +%Y-%m-%d)
BUCKET="s3://foxhunt-audit-logs"

# Export audit logs from Loki
logcli query '{log_type="audit"}' \
  --since="${DATE}T00:00:00Z" \
  --until="${DATE}T23:59:59Z" \
  --output=jsonl \
  --limit=1000000 \
  > /tmp/audit_logs_${DATE}.jsonl

# Compress
gzip /tmp/audit_logs_${DATE}.jsonl

# Upload to S3
aws s3 cp /tmp/audit_logs_${DATE}.jsonl.gz \
  ${BUCKET}/${DATE}/audit_logs_${DATE}.jsonl.gz \
  --storage-class GLACIER_IR

# Cleanup
rm /tmp/audit_logs_${DATE}.jsonl.gz

# Verify upload
aws s3 ls ${BUCKET}/${DATE}/

Cron schedule (daily at 2 AM):

0 2 * * * /opt/foxhunt/scripts/archive_logs.sh

7.3 S3 Lifecycle Policies

{
  "Rules": [
    {
      "Id": "AuditLogRetention",
      "Status": "Enabled",
      "Prefix": "",
      "Transitions": [
        {
          "Days": 90,
          "StorageClass": "GLACIER_IR"
        },
        {
          "Days": 365,
          "StorageClass": "DEEP_ARCHIVE"
        }
      ],
      "Expiration": {
        "Days": 2557
      }
    }
  ]
}

Cost optimization:

• 0-90 days: S3 Standard (frequent access for investigations)
• 90-365 days: Glacier Instant Retrieval (occasional access)
• 1-7 years: Glacier Deep Archive (compliance only, rarely accessed)
• After 7 years: Auto-delete (compliance requirement met)

---

8. Performance Tuning

8.1 Loki Performance

Ingestion rate (per tenant):

limits_config:
  ingestion_rate_mb: 100        # 100 MB/s sustained
  ingestion_burst_size_mb: 200  # 200 MB/s burst
  max_line_size: 256KB          # Maximum log line size

Query optimization:

query_range:
  max_retries: 5
  parallelise_shardable_queries: true
  cache_results: true
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 1000  # 1 GB cache

Indexing:

schema_config:
  configs:
    - from: 2025-01-01
      store: tsdb           # Time-series database (faster)
      object_store: s3
      schema: v12
      index:
        prefix: foxhunt_
        period: 24h         # Daily index rotation

8.2 Promtail Configuration

High-performance log shipping (promtail-config.yml):

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push
    batchwait: 1s          # Batch logs for 1 second
    batchsize: 1048576     # 1 MB batches
    timeout: 10s

scrape_configs:
  - job_name: trading_service
    static_configs:
      - targets:
          - localhost
        labels:
          job: trading_service
          __path__: /var/log/trading_service/*.log

    pipeline_stages:
      # Parse JSON logs
      - json:
          expressions:
            timestamp: timestamp
            level: level
            message: message
            service: service

      # Add labels
      - labels:
          level:
          service:

      # Parse timestamps
      - timestamp:
          source: timestamp
          format: RFC3339

      # Drop debug logs in production
      - match:
          selector: '{level="DEBUG"}'
          action: drop

8.3 Monitoring Loki

Key metrics:

# Ingestion rate
rate(loki_ingester_chunks_created_total[1m])

# Query latency
histogram_quantile(0.99, rate(loki_request_duration_seconds_bucket[5m]))

# Storage usage
loki_ingester_memory_chunks

# Active streams
loki_ingester_streams

---

9. Troubleshooting

9.1 Common Issues

Logs not appearing in Grafana

Check Promtail:

# Check Promtail status
curl http://promtail:9080/metrics | grep promtail_targets_active_total

# Check Promtail logs
docker logs promtail --tail=50

Check Loki ingestion:

# Check ingestion rate
curl http://loki:3100/metrics | grep loki_distributor_bytes_received_total

Check labels:

# List all label names
curl http://loki:3100/loki/api/v1/labels | jq

# List values for specific label
curl http://loki:3100/loki/api/v1/label/service/values | jq

Slow queries

Reduce time range:

• Query smaller time windows
• Use --limit to cap results

Add more specific labels:

# Bad (scans everything)
{} |= "error"

# Good (scans only trading service)
{service="trading_service"} |= "error"

Use metrics instead of logs:

• For aggregations, use Prometheus metrics
• Logs are for debugging, not dashboards

Out of disk space

Check Loki storage:

du -sh /loki/chunks
du -sh /loki/index

Manually clean old data:

# Delete chunks older than 90 days
find /loki/chunks -mtime +90 -delete

# Compact indexes
curl -X POST http://loki:3100/loki/api/v1/push/compact

---

10. Best Practices

10.1 Logging Guidelines

DO:

• ✅ Use structured JSON logging
• ✅ Include correlation IDs (trace_id, request_id)
• ✅ Log important business events (orders, trades)
• ✅ Log errors with full context
• ✅ Use appropriate log levels
• ✅ Add timestamps in ISO 8601 format
• ✅ Include service and component labels

DON'T:

• ❌ Log sensitive data (passwords, API keys, PII)
• ❌ Log excessively (every function call)
• ❌ Use unstructured text logs
• ❌ Log binary data
• ❌ Log without context
• ❌ Use printf-style formatting (use structured fields)

10.2 Performance Tips

1. Use label cardinality wisely:

   • Low cardinality labels: service, level, component (< 100 values)
   • High cardinality as fields: user_id, order_id, request_id

2. Batch log writes:

   • Don't send individual log lines
   • Buffer and send in batches (1 second or 1 MB)

3. Compress logs:

   • Promtail automatically compresses with snappy
   • Use gzip for archival

4. Index appropriately:

   • Too many indexes → slow writes
   • Too few indexes → slow queries
   • Daily rotation is good balance

10.3 Security

Access Control:

• Grafana authentication required for log access
• Separate credentials for production/staging
• Audit who accessed sensitive logs

Data Protection:

• Encrypt logs in transit (TLS)
• Encrypt logs at rest (S3 encryption)
• Redact sensitive fields before logging

Compliance:

• Audit logs are immutable (write-once)
• Tamper detection via checksums
• Regular compliance audits

---

11. Quick Reference

11.1 Service Endpoints

Service	URL	Port
Loki	http://loki.foxhunt.io	3100
Promtail	http://promtail.foxhunt.io	9080
Grafana Explore	http://grafana.foxhunt.io/explore	3000

11.2 Key Commands

# Query last 5 minutes
logcli query '{service="trading_service"}' --since=5m

# Follow logs live
logcli query '{service="trading_service"}' --tail

# Count errors
logcli query 'sum(count_over_time({level="ERROR"}[1h]))'

# Export logs
logcli query '{service="trading_service"}' --since=1h -o jsonl > logs.json

11.3 Support

• Documentation: https://grafana.com/docs/loki/latest/
• Slack: #observability
• Email: oncall@foxhunt.io
• Escalation: See RUNBOOKS.md

---

Document Version History:

• v1.0 (2025-10-07): Initial log aggregation documentation
• Next Review: 2025-11-07