foxhunt/tli/README_EVENT_STREAMING.md

TLI Event Streaming System

Overview

The TLI Event Streaming System provides comprehensive real-time event handling for the Foxhunt HFT Trading System with advanced capabilities including:

• gRPC streaming client management with automatic reconnection and exponential backoff
• Event aggregation and buffering with back-pressure handling and memory management
• Event replay capabilities for historical analysis and debugging
• WebSocket support for browser clients with real-time updates
• Event deduplication and ordering with configurable rules
• Performance metrics and monitoring with comprehensive health checks

Architecture

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   gRPC Services │───▶│   StreamManager  │───▶│   EventBuffer   │
│  (Trading, etc) │    │  - Reconnection  │    │  - Buffering    │
│                 │    │  - Circuit Break │    │  - Back-pressure│
└─────────────────┘    └──────────────────┘    └─────────────────┘
                                 │                        │
                                 ▼                        ▼
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│  WebSocket      │◀───│   Aggregator     │◀───│   ReplaySystem  │
│  - Browser UI   │    │  - Deduplication │    │  - Historical   │
│  - Real-time    │    │  - Correlation   │    │  - Time-travel  │
└─────────────────┘    └──────────────────┘    └─────────────────┘

Core Components

1. EventStreamingSystem

The main orchestrator that coordinates all components:

use tli::events::{
    EventStreamingSystem, StreamConfig, EventBufferConfig,
    AggregationConfig, ReplayConfig, WebSocketConfig
};

let streaming_system = EventStreamingSystem::new(
    stream_config,
    buffer_config,
    aggregation_config,
    replay_config,
    Some(websocket_config),
).await?;

streaming_system.start().await?;

2. StreamManager

Manages multiple concurrent gRPC streams with resilient connections:

let stream_config = StreamConfig {
    endpoints: ServiceEndpoints::default(),
    max_concurrent_streams: 10,
    initial_reconnect_delay_ms: 1000,
    max_reconnect_delay_ms: 30000,
    backoff_multiplier: 2.0,
    enable_circuit_breaker: true,
    circuit_breaker_threshold: 5,
    ..Default::default()
};

Features:

• Automatic reconnection with exponential backoff
• Circuit breaker pattern for failed services
• Connection pooling and health monitoring
• Stream sequence numbering
• Performance metrics collection

3. EventBuffer

Memory-efficient event storage with intelligent management:

let buffer_config = EventBufferConfig {
    max_events: 100_000,
    max_memory_bytes: 100 * 1024 * 1024, // 100MB
    enable_backpressure: true,
    backpressure_threshold_percent: 0.8,
    enable_compression: true,
    enable_priority_queue: true,
    ..Default::default()
};

Features:

• Circular buffer with size and memory limits
• Back-pressure handling and flow control
• Event TTL and automatic cleanup
• Priority queue for critical events
• Memory usage monitoring and alerts

4. EventAggregator

Intelligent event processing with deduplication and correlation:

let aggregation_config = AggregationConfig {
    enable_deduplication: true,
    dedup_window_seconds: 60,
    enable_time_aggregation: true,
    aggregation_window_seconds: 300,
    enable_pattern_matching: true,
    ..Default::default()
};

Features:

• Event deduplication based on configurable keys
• Time-based aggregation windows
• Statistical operations (count, sum, avg, min, max)
• Event pattern matching and correlation
• Real-time enrichment and transformation

5. ReplaySystem

Historical event replay with database persistence:

let replay_config = ReplayConfig {
    database_path: "events.db".to_string(),
    retention_days: 30,
    max_concurrent_sessions: 10,
    default_replay_speed: 1.0,
    enable_indexing: true,
    ..Default::default()
};

Features:

• SQLite-based event storage with indexing
• Multiple concurrent replay sessions
• Configurable replay speed (0.1x to 100x)
• Time-based filtering and selection
• Session management and state tracking

6. WebSocketServer

Real-time browser connectivity with advanced features:

let websocket_config = WebSocketConfig {
    bind_address: "127.0.0.1".to_string(),
    port: 8080,
    max_connections: 1000,
    enable_auth: false,
    rate_limit_per_second: 100,
    enable_rooms: true,
    ..Default::default()
};

Features:

• WebSocket connection management
• Authentication and authorization
• Room-based event distribution
• Message compression and rate limiting
• Connection health monitoring

Event Types and Structure

Event Structure

pub struct Event {
    pub id: Uuid,                          // Unique identifier
    pub event_type: EventType,             // Event classification
    pub severity: EventSeverity,           // Priority level
    pub source: String,                    // Source service
    pub timestamp_nanos: i64,              // Precise timestamp
    pub sequence: u64,                     // Ordering sequence
    pub payload: serde_json::Value,        // Event data
    pub correlation_id: Option<Uuid>,      // Related events
    pub metadata: HashMap<String, String>, // Additional labels
    pub ttl_seconds: u64,                  // Time-to-live
}

Event Types

• Trading: Orders, executions, positions
• MarketData: Quotes, trades, order book updates
• Risk: Limits, breaches, VaR calculations
• MlSignal: Predictions, recommendations
• System: Health, metrics, alerts
• Config: Configuration changes
• Custom: User-defined events

Event Severity Levels

• Info: Informational events
• Warning: Events requiring attention
• Error: Events requiring immediate action
• Critical: Events requiring urgent response

Usage Examples

Basic Event Subscription

use tli::events::{EventFilter, EventType, EventSeverity};

// Subscribe to all trading events
let filter = EventFilter::for_types(vec![EventType::Trading]);
let mut subscription = streaming_system.subscribe(filter).await?;

// Process events
while let Some(event) = subscription.receiver.recv().await {
    println!("Received: {} from {}", event.event_type.as_str(), event.source);
}

Advanced Filtering

// Complex filter with multiple criteria
let filter = EventFilter {
    event_types: vec![EventType::Trading, EventType::Risk],
    min_severity: EventSeverity::Warning,
    sources: vec!["trading_engine".to_string()],
    metadata_filters: {
        let mut map = HashMap::new();
        map.insert("symbol".to_string(), "AAPL".to_string());
        map
    },
    start_time_nanos: Some(start_time.timestamp_nanos()),
    end_time_nanos: Some(end_time.timestamp_nanos()),
    correlation_id: Some(correlation_uuid),
};

Event Replay

// Create replay session for last 24 hours
let filter = ReplayFilter::last_hours(24);
let session_id = streaming_system.replay_system
    .create_session("analysis_session".to_string(), filter).await?;

// Load and start replay
streaming_system.replay_system.load_session_events(session_id).await?;

let (sender, mut receiver) = tokio::sync::mpsc::unbounded_channel();
streaming_system.replay_system.start_replay(session_id, sender).await?;

// Set 10x speed replay
streaming_system.replay_system.set_replay_speed(session_id, 10.0).await?;

// Process replayed events
while let Some(event) = receiver.recv().await {
    // Analyze historical event
}

Aggregation Rules

use tli::events::{AggregationRule, AggregationType};

// Count trading events per minute by symbol
let rule = AggregationRule {
    id: "trading_events_per_minute".to_string(),
    name: "Trading Events Count".to_string(),
    filter: EventFilter::for_types(vec![EventType::Trading]),
    aggregation_type: AggregationType::Count,
    window_seconds: 60,
    group_by: vec!["symbol".to_string()],
    output_event_type: EventType::System,
    enabled: true,
    ..Default::default()
};

streaming_system.aggregator.add_rule(rule).await?;

WebSocket Client (JavaScript)

const ws = new WebSocket('ws://127.0.0.1:8080');

ws.onopen = function() {
    // Subscribe to critical events
    ws.send(JSON.stringify({
        type: 'Subscribe',
        data: {
            filter: {
                event_types: [],
                min_severity: 'Critical',
                sources: [],
                metadata_filters: {},
                correlation_id: null,
                start_time_nanos: null,
                end_time_nanos: null
            }
        }
    }));
    
    // Join trading room
    ws.send(JSON.stringify({
        type: 'JoinRoom',
        data: { room: 'trading' }
    }));
};

ws.onmessage = function(event) {
    const message = JSON.parse(event.data);
    if (message.type === 'Event') {
        console.log('Event:', message.data.event);
    }
};

Performance Characteristics

Latency

• Event ingestion: Sub-millisecond buffering
• Stream processing: ~100μs per event
• WebSocket delivery: <5ms end-to-end
• Database storage: Batched for efficiency

Throughput

• Maximum events/sec: 100,000+ (depending on configuration)
• Concurrent streams: 100+ gRPC connections
• WebSocket clients: 1,000+ simultaneous connections
• Replay sessions: 10+ concurrent sessions

Memory Usage

• Event buffer: Configurable limits with back-pressure
• Deduplication cache: LRU with TTL-based cleanup
• Connection state: Minimal per-connection overhead
• Aggregation windows: Sliding window management

Monitoring and Metrics

System Metrics

let metrics = streaming_system.get_metrics().await;
println!("Events processed: {}", metrics.events_processed);
println!("Events per second: {:.2}", metrics.events_per_second);
println!("Active subscriptions: {}", metrics.active_subscriptions);
println!("Memory usage: {} bytes", metrics.memory_usage_bytes);

Health Checks

let stream_health = streaming_system.stream_manager.get_stream_health().await;
for (service, health) in stream_health {
    println!("Service {}: {:?}", service, health);
}

Buffer Metrics

let buffer_metrics = streaming_system.event_buffer.get_metrics().await;
println!("Buffer utilization: {:.1}%", buffer_metrics.utilization_percent);
println!("Back-pressure active: {}", buffer_metrics.backpressure_active);

Configuration Best Practices

Production Settings

// High-throughput production configuration
let config = StreamConfig {
    max_concurrent_streams: 50,
    initial_reconnect_delay_ms: 500,
    max_reconnect_delay_ms: 10000,
    enable_circuit_breaker: true,
    circuit_breaker_threshold: 3,
    ..Default::default()
};

let buffer_config = EventBufferConfig {
    max_events: 1_000_000,
    max_memory_bytes: 1024 * 1024 * 1024, // 1GB
    enable_backpressure: true,
    backpressure_threshold_percent: 0.9,
    enable_compression: true,
    ..Default::default()
};

Development Settings

// Development configuration with verbose logging
let config = StreamConfig {
    max_concurrent_streams: 5,
    initial_reconnect_delay_ms: 1000,
    enable_circuit_breaker: false, // Disable for testing
    ..Default::default()
};

let buffer_config = EventBufferConfig {
    max_events: 10_000,
    max_memory_bytes: 50 * 1024 * 1024, // 50MB
    cleanup_interval_seconds: 30,
    ..Default::default()
};

Error Handling

The system provides comprehensive error handling with specific error types:

use tli::error::TliError;

match result {
    Err(TliError::BufferFull(msg)) => {
        // Handle back-pressure
        warn!("Buffer full: {}", msg);
    }
    Err(TliError::ConnectionClosed(msg)) => {
        // Handle disconnection
        info!("Connection closed: {}", msg);
    }
    Err(TliError::WebSocket(msg)) => {
        // Handle WebSocket errors
        error!("WebSocket error: {}", msg);
    }
    Ok(result) => {
        // Success case
    }
}

Testing

Run the comprehensive demo:

cargo run --example event_streaming_demo

Run unit tests:

cargo test events::

Run integration tests:

cargo test --test event_streaming_integration

Troubleshooting

Common Issues

1. High Memory Usage

   • Reduce max_events or max_memory_bytes in buffer config
   • Enable compression and adjust TTL settings
   • Monitor aggregation window sizes

2. Connection Issues

   • Check service endpoints and network connectivity
   • Verify circuit breaker settings
   • Review reconnection delay configuration

3. Performance Issues

   • Adjust batch sizes and processing intervals
   • Enable back-pressure handling
   • Monitor event processing rates

4. WebSocket Problems

   • Check rate limiting settings
   • Verify CORS configuration for browser clients
   • Review authentication requirements

Debug Logging

Enable detailed logging:

RUST_LOG=tli::events=debug cargo run --example event_streaming_demo

Future Enhancements

• Event compression improvements with more algorithms
• Distributed replay across multiple nodes
• Advanced pattern matching with complex rules
• Machine learning integration for anomaly detection
• Cloud storage backends for historical data
• GraphQL subscription support for flexible queries