foxhunt/tli/README_EVENT_STREAMING.md

# TLI Event Streaming System

## Overview

The TLI Event Streaming System provides comprehensive real-time event handling for the Foxhunt HFT Trading System with advanced capabilities including:

- **gRPC streaming client management** with automatic reconnection and exponential backoff
- **Event aggregation and buffering** with back-pressure handling and memory management
- **Event replay capabilities** for historical analysis and debugging
- **WebSocket support** for browser clients with real-time updates
- **Event deduplication and ordering** with configurable rules
- **Performance metrics and monitoring** with comprehensive health checks

## Architecture

```text
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   gRPC Services │───▶│   StreamManager  │───▶│   EventBuffer   │
│  (Trading, etc) │    │  - Reconnection  │    │  - Buffering    │
│                 │    │  - Circuit Break │    │  - Back-pressure│
└─────────────────┘    └──────────────────┘    └─────────────────┘
                                 │                        │
                                 ▼                        ▼
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│  WebSocket      │◀───│   Aggregator     │◀───│   ReplaySystem  │
│  - Browser UI   │    │  - Deduplication │    │  - Historical   │
│  - Real-time    │    │  - Correlation   │    │  - Time-travel  │
└─────────────────┘    └──────────────────┘    └─────────────────┘
```

## Core Components

### 1. EventStreamingSystem

The main orchestrator that coordinates all components:

```rust
use tli::events::{
    EventStreamingSystem, StreamConfig, EventBufferConfig,
    AggregationConfig, ReplayConfig, WebSocketConfig
};

let streaming_system = EventStreamingSystem::new(
    stream_config,
    buffer_config,
    aggregation_config,
    replay_config,
    Some(websocket_config),
).await?;

streaming_system.start().await?;
```

### 2. StreamManager

Manages multiple concurrent gRPC streams with resilient connections:

```rust
let stream_config = StreamConfig {
    endpoints: ServiceEndpoints::default(),
    max_concurrent_streams: 10,
    initial_reconnect_delay_ms: 1000,
    max_reconnect_delay_ms: 30000,
    backoff_multiplier: 2.0,
    enable_circuit_breaker: true,
    circuit_breaker_threshold: 5,
    ..Default::default()
};
```

**Features:**
- Automatic reconnection with exponential backoff
- Circuit breaker pattern for failed services
- Connection pooling and health monitoring
- Stream sequence numbering
- Performance metrics collection

### 3. EventBuffer

Memory-efficient event storage with intelligent management:

```rust
let buffer_config = EventBufferConfig {
    max_events: 100_000,
    max_memory_bytes: 100 * 1024 * 1024, // 100MB
    enable_backpressure: true,
    backpressure_threshold_percent: 0.8,
    enable_compression: true,
    enable_priority_queue: true,
    ..Default::default()
};
```

**Features:**
- Circular buffer with size and memory limits
- Back-pressure handling and flow control
- Event TTL and automatic cleanup
- Priority queue for critical events
- Memory usage monitoring and alerts

### 4. EventAggregator

Intelligent event processing with deduplication and correlation:

```rust
let aggregation_config = AggregationConfig {
    enable_deduplication: true,
    dedup_window_seconds: 60,
    enable_time_aggregation: true,
    aggregation_window_seconds: 300,
    enable_pattern_matching: true,
    ..Default::default()
};
```

**Features:**
- Event deduplication based on configurable keys
- Time-based aggregation windows
- Statistical operations (count, sum, avg, min, max)
- Event pattern matching and correlation
- Real-time enrichment and transformation

### 5. ReplaySystem

Historical event replay with database persistence:

```rust
let replay_config = ReplayConfig {
    database_path: "events.db".to_string(),
    retention_days: 30,
    max_concurrent_sessions: 10,
    default_replay_speed: 1.0,
    enable_indexing: true,
    ..Default::default()
};
```

**Features:**
- SQLite-based event storage with indexing
- Multiple concurrent replay sessions
- Configurable replay speed (0.1x to 100x)
- Time-based filtering and selection
- Session management and state tracking

### 6. WebSocketServer

Real-time browser connectivity with advanced features:

```rust
let websocket_config = WebSocketConfig {
    bind_address: "127.0.0.1".to_string(),
    port: 8080,
    max_connections: 1000,
    enable_auth: false,
    rate_limit_per_second: 100,
    enable_rooms: true,
    ..Default::default()
};
```

**Features:**
- WebSocket connection management
- Authentication and authorization
- Room-based event distribution
- Message compression and rate limiting
- Connection health monitoring

## Event Types and Structure

### Event Structure

```rust
pub struct Event {
    pub id: Uuid,                          // Unique identifier
    pub event_type: EventType,             // Event classification
    pub severity: EventSeverity,           // Priority level
    pub source: String,                    // Source service
    pub timestamp_nanos: i64,              // Precise timestamp
    pub sequence: u64,                     // Ordering sequence
    pub payload: serde_json::Value,        // Event data
    pub correlation_id: Option<Uuid>,      // Related events
    pub metadata: HashMap<String, String>, // Additional labels
    pub ttl_seconds: u64,                  // Time-to-live
}
```

### Event Types

- **Trading**: Orders, executions, positions
- **MarketData**: Quotes, trades, order book updates
- **Risk**: Limits, breaches, VaR calculations
- **MlSignal**: Predictions, recommendations
- **System**: Health, metrics, alerts
- **Config**: Configuration changes
- **Custom**: User-defined events

### Event Severity Levels

- **Info**: Informational events
- **Warning**: Events requiring attention
- **Error**: Events requiring immediate action
- **Critical**: Events requiring urgent response

## Usage Examples

### Basic Event Subscription

```rust
use tli::events::{EventFilter, EventType, EventSeverity};

// Subscribe to all trading events
let filter = EventFilter::for_types(vec![EventType::Trading]);
let mut subscription = streaming_system.subscribe(filter).await?;

// Process events
while let Some(event) = subscription.receiver.recv().await {
    println!("Received: {} from {}", event.event_type.as_str(), event.source);
}
```

### Advanced Filtering

```rust
// Complex filter with multiple criteria
let filter = EventFilter {
    event_types: vec![EventType::Trading, EventType::Risk],
    min_severity: EventSeverity::Warning,
    sources: vec!["trading_engine".to_string()],
    metadata_filters: {
        let mut map = HashMap::new();
        map.insert("symbol".to_string(), "AAPL".to_string());
        map
    },
    start_time_nanos: Some(start_time.timestamp_nanos()),
    end_time_nanos: Some(end_time.timestamp_nanos()),
    correlation_id: Some(correlation_uuid),
};
```

### Event Replay

```rust
// Create replay session for last 24 hours
let filter = ReplayFilter::last_hours(24);
let session_id = streaming_system.replay_system
    .create_session("analysis_session".to_string(), filter).await?;

// Load and start replay
streaming_system.replay_system.load_session_events(session_id).await?;

let (sender, mut receiver) = tokio::sync::mpsc::unbounded_channel();
streaming_system.replay_system.start_replay(session_id, sender).await?;

// Set 10x speed replay
streaming_system.replay_system.set_replay_speed(session_id, 10.0).await?;

// Process replayed events
while let Some(event) = receiver.recv().await {
    // Analyze historical event
}
```

### Aggregation Rules

```rust
use tli::events::{AggregationRule, AggregationType};

// Count trading events per minute by symbol
let rule = AggregationRule {
    id: "trading_events_per_minute".to_string(),
    name: "Trading Events Count".to_string(),
    filter: EventFilter::for_types(vec![EventType::Trading]),
    aggregation_type: AggregationType::Count,
    window_seconds: 60,
    group_by: vec!["symbol".to_string()],
    output_event_type: EventType::System,
    enabled: true,
    ..Default::default()
};

streaming_system.aggregator.add_rule(rule).await?;
```

### WebSocket Client (JavaScript)

```javascript
const ws = new WebSocket('ws://127.0.0.1:8080');

ws.onopen = function() {
    // Subscribe to critical events
    ws.send(JSON.stringify({
        type: 'Subscribe',
        data: {
            filter: {
                event_types: [],
                min_severity: 'Critical',
                sources: [],
                metadata_filters: {},
                correlation_id: null,
                start_time_nanos: null,
                end_time_nanos: null
            }
        }
    }));
    
    // Join trading room
    ws.send(JSON.stringify({
        type: 'JoinRoom',
        data: { room: 'trading' }
    }));
};

ws.onmessage = function(event) {
    const message = JSON.parse(event.data);
    if (message.type === 'Event') {
        console.log('Event:', message.data.event);
    }
};
```

## Performance Characteristics

### Latency

- **Event ingestion**: Sub-millisecond buffering
- **Stream processing**: ~100μs per event
- **WebSocket delivery**: <5ms end-to-end
- **Database storage**: Batched for efficiency

### Throughput

- **Maximum events/sec**: 100,000+ (depending on configuration)
- **Concurrent streams**: 100+ gRPC connections
- **WebSocket clients**: 1,000+ simultaneous connections
- **Replay sessions**: 10+ concurrent sessions

### Memory Usage

- **Event buffer**: Configurable limits with back-pressure
- **Deduplication cache**: LRU with TTL-based cleanup
- **Connection state**: Minimal per-connection overhead
- **Aggregation windows**: Sliding window management

## Monitoring and Metrics

### System Metrics

```rust
let metrics = streaming_system.get_metrics().await;
println!("Events processed: {}", metrics.events_processed);
println!("Events per second: {:.2}", metrics.events_per_second);
println!("Active subscriptions: {}", metrics.active_subscriptions);
println!("Memory usage: {} bytes", metrics.memory_usage_bytes);
```

### Health Checks

```rust
let stream_health = streaming_system.stream_manager.get_stream_health().await;
for (service, health) in stream_health {
    println!("Service {}: {:?}", service, health);
}
```

### Buffer Metrics

```rust
let buffer_metrics = streaming_system.event_buffer.get_metrics().await;
println!("Buffer utilization: {:.1}%", buffer_metrics.utilization_percent);
println!("Back-pressure active: {}", buffer_metrics.backpressure_active);
```

## Configuration Best Practices

### Production Settings

```rust
// High-throughput production configuration
let config = StreamConfig {
    max_concurrent_streams: 50,
    initial_reconnect_delay_ms: 500,
    max_reconnect_delay_ms: 10000,
    enable_circuit_breaker: true,
    circuit_breaker_threshold: 3,
    ..Default::default()
};

let buffer_config = EventBufferConfig {
    max_events: 1_000_000,
    max_memory_bytes: 1024 * 1024 * 1024, // 1GB
    enable_backpressure: true,
    backpressure_threshold_percent: 0.9,
    enable_compression: true,
    ..Default::default()
};
```

### Development Settings

```rust
// Development configuration with verbose logging
let config = StreamConfig {
    max_concurrent_streams: 5,
    initial_reconnect_delay_ms: 1000,
    enable_circuit_breaker: false, // Disable for testing
    ..Default::default()
};

let buffer_config = EventBufferConfig {
    max_events: 10_000,
    max_memory_bytes: 50 * 1024 * 1024, // 50MB
    cleanup_interval_seconds: 30,
    ..Default::default()
};
```

## Error Handling

The system provides comprehensive error handling with specific error types:

```rust
use tli::error::TliError;

match result {
    Err(TliError::BufferFull(msg)) => {
        // Handle back-pressure
        warn!("Buffer full: {}", msg);
    }
    Err(TliError::ConnectionClosed(msg)) => {
        // Handle disconnection
        info!("Connection closed: {}", msg);
    }
    Err(TliError::WebSocket(msg)) => {
        // Handle WebSocket errors
        error!("WebSocket error: {}", msg);
    }
    Ok(result) => {
        // Success case
    }
}
```

## Testing

Run the comprehensive demo:

```bash
cargo run --example event_streaming_demo
```

Run unit tests:

```bash
cargo test events::
```

Run integration tests:

```bash
cargo test --test event_streaming_integration
```

## Troubleshooting

### Common Issues

1. **High Memory Usage**
   - Reduce `max_events` or `max_memory_bytes` in buffer config
   - Enable compression and adjust TTL settings
   - Monitor aggregation window sizes

2. **Connection Issues** 
   - Check service endpoints and network connectivity
   - Verify circuit breaker settings
   - Review reconnection delay configuration

3. **Performance Issues**
   - Adjust batch sizes and processing intervals
   - Enable back-pressure handling
   - Monitor event processing rates

4. **WebSocket Problems**
   - Check rate limiting settings
   - Verify CORS configuration for browser clients
   - Review authentication requirements

### Debug Logging

Enable detailed logging:

```bash
RUST_LOG=tli::events=debug cargo run --example event_streaming_demo
```

## Future Enhancements

- **Event compression** improvements with more algorithms
- **Distributed replay** across multiple nodes
- **Advanced pattern matching** with complex rules
- **Machine learning** integration for anomaly detection
- **Cloud storage** backends for historical data
- **GraphQL subscription** support for flexible queries