foxhunt/TLI_PLAN.md

TLI_PLAN.md - Comprehensive Real-Time Trading Terminal Implementation Plan

EXECUTIVE SUMMARY

This document outlines the complete implementation plan for the Terminal Line Interface (TLI) - a comprehensive real-time trading terminal for the Foxhunt HFT trading system. The TLI provides multi-dashboard monitoring, configuration management, and complete system oversight through a Ratatui-based terminal interface with gRPC connectivity to the monolithic trading service.

SYSTEM ARCHITECTURE OVERVIEW

TLI Client (Terminal)           Trading Service (Monolithic)        Backtesting Service

┌─────────────────────────┐    ┌───────────────────────────────┐    ┌─────────────────────┐
│    DASHBOARD MANAGER    │    │       UNIFIED gRPC SERVICE    │    │  BACKTESTING ENGINE │
│                         │    │                               │    │                     │
│ ┌─ Trading Dashboard ─┐ │gRPC│ ┌─ Trading Operations      ─┐ │    │ ┌─ Strategy Engine ─┐ │
│ ├─ Risk Dashboard   ─┤ │<-->│ ├─ Risk Management (built-in)─┤ │gRPC│ ├─ Performance     ─┤ │
│ ├─ ML Dashboard     ─┤ │    │ ├─ Market Data              ─┤ │<-->│ ├─ Results Storage ─┤ │
│ ├─ Performance Dash.─┤ │    │ ├─ ML Signal Processing     ─┤ │    │ └─ Report Generation─┘ │
│ ├─ Backtesting Dash.─┤ │    │ ├─ System Monitoring        ─┤ │    │                     │
│ └─ Configuration D. ─┘ │    │ └─ Configuration Management ─┘ │    │ PostgreSQL/InfluxDB │
│                         │    │                               │    └─────────────────────┘
│ Real-Time Data Streams  │    │ SQLite Configuration DB       │
│ Connection Manager      │    │ Event Publisher System        │
└─────────────────────────┘    └───────────────────────────────┘

Core Components

ONLY 3 SERVICES:

• TLI Client: Ratatui-based terminal with 6 dashboards connecting to services
• Trading Service: Monolithic service with ALL functionality (trading, risk, monitoring, config, ML)
• Backtesting Service: Isolated service for strategy testing and analysis

Supporting Infrastructure:

• SQLite Configuration: Centralized configuration database with live updates
• PostgreSQL: ACID-compliant backtesting metadata and trade storage
• InfluxDB: High-frequency time-series backtesting performance data
• gRPC Streaming: Real-time data feeds for live monitoring

PHASE 1: gRPC SERVICE ARCHITECTURE

Comprehensive gRPC API Suite

TradingService - Real-Time Trading Operations (with Integrated Risk Management)

service TradingService {
  // Real-time data streams
  rpc StreamMarketData(MarketDataRequest) returns (stream MarketDataResponse);
  rpc StreamPositions(PositionRequest) returns (stream PositionResponse);
  rpc StreamOrders(OrderRequest) returns (stream OrderResponse);
  rpc StreamExecutions(ExecutionRequest) returns (stream ExecutionResponse);

  // Trading operations
  rpc GetTradingStatus(Empty) returns (TradingStatusResponse);
  rpc PlaceOrder(PlaceOrderRequest) returns (OrderResponse);
  rpc CancelOrder(CancelOrderRequest) returns (CancelResponse);
  rpc GetOrderBook(OrderBookRequest) returns (OrderBookResponse);

  // Portfolio management
  rpc GetPortfolioSummary(PortfolioRequest) returns (PortfolioResponse);
  rpc GetPnLSummary(PnLRequest) returns (PnLResponse);

  // Integrated Risk Management
  rpc GetVaR(GetVaRRequest) returns (GetVaRResponse);
  rpc GetPositionRisk(GetPositionRiskRequest) returns (GetPositionRiskResponse);
  rpc ValidateOrder(ValidateOrderRequest) returns (ValidateOrderResponse);
  rpc GetRiskMetrics(GetRiskMetricsRequest) returns (GetRiskMetricsResponse);
  rpc SubscribeRiskAlerts(SubscribeRiskAlertsRequest) returns (stream RiskAlertEvent);
  rpc EmergencyStop(EmergencyStopRequest) returns (EmergencyStopResponse);
}

MLService - Model Insights & Predictions

service MLService {
  // Real-time ML streams
  rpc StreamModelPredictions(ModelRequest) returns (stream PredictionResponse);
  rpc StreamSignalStrength(SignalRequest) returns (stream SignalResponse);
  rpc StreamModelMetrics(MetricsRequest) returns (stream ModelMetricsResponse);

  // Model management
  rpc GetModelPerformance(ModelPerformanceRequest) returns (ModelPerformanceResponse);
  rpc GetEnsembleVote(EnsembleRequest) returns (EnsembleResponse);
  rpc GetFeatureImportance(FeatureRequest) returns (FeatureResponse);
  rpc RetrainModel(RetrainRequest) returns (RetrainResponse);

  // Model status
  rpc GetModelStatus(ModelStatusRequest) returns (ModelStatusResponse);
  rpc GetAvailableModels(Empty) returns (AvailableModelsResponse);
}

ConfigurationService - SQLite-Based Configuration Management

service ConfigurationService {
  // Configuration CRUD operations
  rpc GetConfiguration(ConfigRequest) returns (ConfigResponse);
  rpc UpdateConfiguration(UpdateConfigRequest) returns (UpdateResponse);
  rpc DeleteConfiguration(DeleteConfigRequest) returns (DeleteResponse);
  rpc ListCategories(Empty) returns (CategoriesResponse);

  // Real-time configuration updates
  rpc StreamConfigChanges(Empty) returns (stream ConfigChangeResponse);

  // Configuration management
  rpc ValidateConfiguration(ValidateRequest) returns (ValidationResponse);
  rpc GetConfigurationHistory(HistoryRequest) returns (HistoryResponse);
  rpc RollbackConfiguration(RollbackRequest) returns (RollbackResponse);
  rpc ExportConfiguration(ExportRequest) returns (ExportResponse);
  rpc ImportConfiguration(ImportRequest) returns (ImportResponse);

  // Schema management
  rpc GetConfigSchema(SchemaRequest) returns (SchemaResponse);
  rpc UpdateConfigSchema(UpdateSchemaRequest) returns (UpdateSchemaResponse);
}

Data Streaming Strategy

• Server-side streaming for real-time data feeds
• Client-side connection pooling for multiple simultaneous streams
• Automatic reconnection with exponential backoff
• Data compression and batching for network efficiency
• Backpressure handling for slow clients
• Connection health monitoring with automatic failover

PHASE 2: SQLITE CONFIGURATION DATABASE

Comprehensive Configuration Schema

-- Configuration categories for hierarchical organization
CREATE TABLE config_categories (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT UNIQUE NOT NULL,
    description TEXT,
    parent_id INTEGER,
    display_order INTEGER DEFAULT 0,
    icon TEXT, -- Unicode icon for UI display
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY(parent_id) REFERENCES config_categories(id)
);

-- Core configuration settings with full metadata
CREATE TABLE config_settings (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    category_id INTEGER NOT NULL,
    key TEXT NOT NULL,
    value TEXT NOT NULL,
    data_type TEXT NOT NULL CHECK (data_type IN ('string', 'number', 'boolean', 'json', 'encrypted')),
    hot_reload BOOLEAN DEFAULT TRUE,
    validation_rule TEXT, -- JSON schema for validation
    description TEXT,
    default_value TEXT,
    required BOOLEAN DEFAULT FALSE,
    sensitive BOOLEAN DEFAULT FALSE, -- For API keys, passwords, etc.
    environment_override TEXT, -- Environment variable name for override
    min_value REAL, -- For numeric types
    max_value REAL, -- For numeric types
    enum_values TEXT, -- JSON array for enum validation
    depends_on TEXT, -- JSON array of setting IDs this depends on
    tags TEXT, -- JSON array of tags for grouping/searching
    display_order INTEGER DEFAULT 0,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    modified_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    UNIQUE(category_id, key),
    FOREIGN KEY(category_id) REFERENCES config_categories(id)
);

-- Configuration change history with full audit trail
CREATE TABLE config_history (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    setting_id INTEGER NOT NULL,
    old_value TEXT,
    new_value TEXT,
    change_reason TEXT,
    changed_by TEXT NOT NULL, -- User/system that made the change
    changed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    change_source TEXT, -- 'tli', 'api', 'migration', 'system'
    validation_result TEXT, -- JSON validation result
    rollback_id INTEGER, -- Reference to rollback transaction
    FOREIGN KEY(setting_id) REFERENCES config_settings(id)
);

-- Environment-specific configuration overrides
CREATE TABLE config_environments (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT UNIQUE NOT NULL, -- 'development', 'staging', 'production'
    description TEXT,
    is_active BOOLEAN DEFAULT FALSE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE config_environment_overrides (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    environment_id INTEGER NOT NULL,
    setting_id INTEGER NOT NULL,
    override_value TEXT NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    UNIQUE(environment_id, setting_id),
    FOREIGN KEY(environment_id) REFERENCES config_environments(id),
    FOREIGN KEY(setting_id) REFERENCES config_settings(id)
);

-- Configuration validation rules and schemas
CREATE TABLE config_validation_schemas (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT UNIQUE NOT NULL,
    schema_definition TEXT NOT NULL, -- JSON schema
    description TEXT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Configuration change notifications/subscriptions
CREATE TABLE config_subscribers (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    setting_id INTEGER,
    category_id INTEGER,
    client_id TEXT NOT NULL,
    last_notified TIMESTAMP,
    notification_type TEXT DEFAULT 'change', -- 'change', 'validation_error', 'rollback'
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY(setting_id) REFERENCES config_settings(id),
    FOREIGN KEY(category_id) REFERENCES config_categories(id)
);

-- Encrypted storage for sensitive configuration data
CREATE TABLE config_encrypted_values (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    setting_id INTEGER UNIQUE NOT NULL,
    encrypted_value BLOB NOT NULL, -- AES-256 encrypted value
    encryption_key_id TEXT NOT NULL, -- Key management identifier
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY(setting_id) REFERENCES config_settings(id)
);

-- Configuration migration tracking
CREATE TABLE config_migrations (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    version TEXT UNIQUE NOT NULL,
    description TEXT,
    migration_sql TEXT,
    applied_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    rollback_sql TEXT
);

Configuration Categories Structure

-- Insert base configuration categories
INSERT INTO config_categories (name, description, display_order, icon) VALUES
('system', 'Core system configuration', 1, '⚙️'),
('trading', 'Trading engine settings', 2, '📈'),
('risk', 'Risk management parameters', 3, '🛡️'),
('ml', 'Machine learning model configuration', 4, '🧠'),
('data', 'Market data provider settings', 5, '📊'),
('brokers', 'Broker connectivity settings', 6, '🔗'),
('security', 'Security and authentication settings', 7, '🔐'),
('monitoring', 'Monitoring and alerting configuration', 8, '📡'),
('performance', 'Performance optimization settings', 9, '⚡');

-- Insert subcategories
INSERT INTO config_categories (name, description, parent_id, display_order, icon) VALUES
('logging', 'Logging configuration', 1, 1, '📝'),
('database', 'Database connection settings', 1, 2, '🗄️'),
('grpc', 'gRPC server configuration', 1, 3, '🔄'),

('execution', 'Order execution settings', 2, 1, '⚡'),
('strategies', 'Trading strategy parameters', 2, 2, '🎯'),
('position_sizing', 'Position sizing algorithms', 2, 3, '📏'),

('var', 'Value at Risk calculations', 3, 1, '📉'),
('limits', 'Position and exposure limits', 3, 2, '🚫'),
('alerts', 'Risk alert thresholds', 3, 3, '🚨'),

('models', 'ML model configurations', 4, 1, '🤖'),
('training', 'Model training parameters', 4, 2, '🎓'),
('inference', 'Model inference settings', 4, 3, '🔮'),

('databento', 'Databento market data settings', 5, 1, '📊'),
('benzinga', 'Benzinga news and sentiment settings', 5, 2, '📰'),
('alpha_vantage', 'Alpha Vantage API settings', 5, 2, '📈'),
('real_time', 'Real-time data feed settings', 5, 3, '⚡'),

('interactive_brokers', 'Interactive Brokers TWS settings', 6, 1, '🏦'),
('icmarkets', 'ICMarkets FIX settings', 6, 2, '💱'),
('paper_trading', 'Paper trading broker settings', 6, 3, '📄');

Comprehensive Configuration Settings

-- System Configuration
INSERT INTO config_settings (category_id, key, value, data_type, description, hot_reload, required) VALUES
-- Logging
((SELECT id FROM config_categories WHERE name = 'logging'), 'log_level', 'info', 'string', 'Global log level', TRUE, TRUE),
((SELECT id FROM config_categories WHERE name = 'logging'), 'log_file_path', '/var/log/foxhunt/trading.log', 'string', 'Log file location', FALSE, TRUE),
((SELECT id FROM config_categories WHERE name = 'logging'), 'max_log_file_size', '100MB', 'string', 'Maximum log file size before rotation', TRUE, TRUE),
((SELECT id FROM config_categories WHERE name = 'logging'), 'log_retention_days', '30', 'number', 'Number of days to retain log files', TRUE, TRUE),

-- Database
((SELECT id FROM config_categories WHERE name = 'database'), 'postgres_url', 'postgresql://localhost:5432/foxhunt', 'string', 'PostgreSQL connection URL', FALSE, TRUE),
((SELECT id FROM config_categories WHERE name = 'database'), 'redis_url', 'redis://localhost:6379', 'string', 'Redis connection URL', FALSE, TRUE),
((SELECT id FROM config_categories WHERE name = 'database'), 'sqlite_config_path', '/etc/foxhunt/config.db', 'string', 'SQLite configuration database path', FALSE, TRUE),
((SELECT id FROM config_categories WHERE name = 'database'), 'connection_pool_size', '10', 'number', 'Database connection pool size', TRUE, TRUE),

-- gRPC
((SELECT id FROM config_categories WHERE name = 'grpc'), 'server_address', '0.0.0.0:50051', 'string', 'gRPC server bind address', FALSE, TRUE),
((SELECT id FROM config_categories WHERE name = 'grpc'), 'max_message_size', '4MB', 'string', 'Maximum gRPC message size', TRUE, TRUE),
((SELECT id FROM config_categories WHERE name = 'grpc'), 'compression_enabled', 'true', 'boolean', 'Enable gRPC compression', TRUE, TRUE),

-- Trading Configuration
INSERT INTO config_settings (category_id, key, value, data_type, description, hot_reload, required, sensitive) VALUES
-- Execution
((SELECT id FROM config_categories WHERE name = 'execution'), 'max_order_size', '1000000.0', 'number', 'Maximum order size in USD', TRUE, TRUE, FALSE),
((SELECT id FROM config_categories WHERE name = 'execution'), 'order_timeout_seconds', '30', 'number', 'Order execution timeout', TRUE, TRUE, FALSE),
((SELECT id FROM config_categories WHERE name = 'execution'), 'price_improvement_threshold', '0.001', 'number', 'Minimum price improvement for execution', TRUE, TRUE, FALSE),
((SELECT id FROM config_categories WHERE name = 'execution'), 'slippage_tolerance', '0.005', 'number', 'Maximum acceptable slippage', TRUE, TRUE, FALSE),

-- Strategies
((SELECT id FROM config_categories WHERE name = 'strategies'), 'default_strategy', 'adaptive_ensemble', 'string', 'Default trading strategy', TRUE, TRUE, FALSE),
((SELECT id FROM config_categories WHERE name = 'strategies'), 'strategy_rotation_enabled', 'true', 'boolean', 'Enable automatic strategy rotation', TRUE, TRUE, FALSE),
((SELECT id FROM config_categories WHERE name = 'strategies'), 'max_concurrent_strategies', '5', 'number', 'Maximum concurrent active strategies', TRUE, TRUE, FALSE),

-- Position Sizing
((SELECT id FROM config_categories WHERE name = 'position_sizing'), 'kelly_criterion_enabled', 'true', 'boolean', 'Enable Kelly Criterion position sizing', TRUE, TRUE, FALSE),
((SELECT id FROM config_categories WHERE name = 'position_sizing'), 'max_position_pct', '0.10', 'number', 'Maximum position as percentage of portfolio', TRUE, TRUE, FALSE),
((SELECT id FROM config_categories WHERE name = 'position_sizing'), 'risk_per_trade', '0.02', 'number', 'Risk per trade as percentage of portfolio', TRUE, TRUE, FALSE),

-- Risk Management Configuration
INSERT INTO config_settings (category_id, key, value, data_type, description, hot_reload, required) VALUES
-- VaR
((SELECT id FROM config_categories WHERE name = 'var'), 'confidence_level', '0.95', 'number', 'VaR confidence level', TRUE, TRUE),
((SELECT id FROM config_categories WHERE name = 'var'), 'lookback_days', '252', 'number', 'VaR calculation lookback period', TRUE, TRUE),
((SELECT id FROM config_categories WHERE name = 'var'), 'monte_carlo_simulations', '10000', 'number', 'Number of Monte Carlo simulations', TRUE, TRUE),
((SELECT id FROM config_categories WHERE name = 'var'), 'calculation_frequency_minutes', '5', 'number', 'VaR calculation frequency', TRUE, TRUE),

-- Limits
((SELECT id FROM config_categories WHERE name = 'limits'), 'max_daily_loss', '50000.0', 'number', 'Maximum daily loss in USD', TRUE, TRUE),
((SELECT id FROM config_categories WHERE name = 'limits'), 'max_position_per_symbol', '100000.0', 'number', 'Maximum position per symbol in USD', TRUE, TRUE),
((SELECT id FROM config_categories WHERE name = 'limits'), 'max_portfolio_exposure', '2000000.0', 'number', 'Maximum total portfolio exposure in USD', TRUE, TRUE),
((SELECT id FROM config_categories WHERE name = 'limits'), 'concentration_limit_pct', '0.25', 'number', 'Maximum concentration per symbol', TRUE, TRUE),

-- Alerts
((SELECT id FROM config_categories WHERE name = 'alerts'), 'drawdown_alert_threshold', '0.05', 'number', 'Drawdown alert threshold (5%)', TRUE, TRUE),
((SELECT id FROM config_categories WHERE name = 'alerts'), 'var_breach_threshold', '1.5', 'number', 'VaR breach threshold multiplier', TRUE, TRUE),
((SELECT id FROM config_categories WHERE name = 'alerts'), 'emergency_stop_threshold', '0.10', 'number', 'Emergency stop threshold (10% loss)', TRUE, TRUE),

-- ML Configuration
INSERT INTO config_settings (category_id, key, value, data_type, description, hot_reload, required) VALUES
-- Models
((SELECT id FROM config_categories WHERE name = 'models'), 'ensemble_enabled', 'true', 'boolean', 'Enable ensemble model predictions', TRUE, TRUE),
((SELECT id FROM config_categories WHERE name = 'models'), 'model_confidence_threshold', '0.7', 'number', 'Minimum confidence for model predictions', TRUE, TRUE),
((SELECT id FROM config_categories WHERE name = 'models'), 'model_update_frequency_minutes', '15', 'number', 'Model update frequency', TRUE, TRUE),

-- Training
((SELECT id FROM config_categories WHERE name = 'training'), 'auto_retrain_enabled', 'true', 'boolean', 'Enable automatic model retraining', TRUE, TRUE),
((SELECT id FROM config_categories WHERE name = 'training'), 'retrain_performance_threshold', '0.6', 'number', 'Performance threshold for retraining', TRUE, TRUE),
((SELECT id FROM config_categories WHERE name = 'training'), 'training_data_lookback_days', '90', 'number', 'Training data lookback period', TRUE, TRUE),

-- Data Provider Configuration (Including API Keys)
INSERT INTO config_settings (category_id, key, value, data_type, description, hot_reload, required, sensitive) VALUES
-- Databento Market Data
((SELECT id FROM config_categories WHERE name = 'databento'), 'api_key', '', 'encrypted', 'Databento API key', FALSE, TRUE, TRUE),
((SELECT id FROM config_categories WHERE name = 'databento'), 'base_url', 'https://hist.databento.com', 'string', 'Databento API base URL', TRUE, TRUE, FALSE),
((SELECT id FROM config_categories WHERE name = 'databento'), 'live_gateway', 'gateway.databento.com', 'string', 'Databento live data gateway', TRUE, TRUE, FALSE),
((SELECT id FROM config_categories WHERE name = 'databento'), 'dataset', 'XNAS.ITCH', 'string', 'Databento dataset (e.g., XNAS.ITCH)', TRUE, TRUE, FALSE),
((SELECT id FROM config_categories WHERE name = 'databento'), 'timeout_seconds', '30', 'number', 'API request timeout', TRUE, TRUE, FALSE),

-- Benzinga News & Sentiment
((SELECT id FROM config_categories WHERE name = 'benzinga'), 'api_key', '', 'encrypted', 'Benzinga Pro API key', FALSE, TRUE, TRUE),
((SELECT id FROM config_categories WHERE name = 'benzinga'), 'base_url', 'https://api.benzinga.com', 'string', 'Benzinga API base URL', TRUE, TRUE, FALSE),
((SELECT id FROM config_categories WHERE name = 'benzinga'), 'rate_limit_per_minute', '300', 'number', 'API rate limit per minute', TRUE, TRUE, FALSE),
((SELECT id FROM config_categories WHERE name = 'benzinga'), 'timeout_seconds', '15', 'number', 'API request timeout', TRUE, TRUE, FALSE),

-- Alpha Vantage
((SELECT id FROM config_categories WHERE name = 'alpha_vantage'), 'api_key', '', 'encrypted', 'Alpha Vantage API key', FALSE, FALSE, TRUE),
((SELECT id FROM config_categories WHERE name = 'alpha_vantage'), 'base_url', 'https://www.alphavantage.co', 'string', 'Alpha Vantage API base URL', TRUE, FALSE, FALSE),
((SELECT id FROM config_categories WHERE name = 'alpha_vantage'), 'rate_limit_per_minute', '5', 'number', 'API rate limit per minute', TRUE, FALSE, FALSE),

-- Broker Configuration
INSERT INTO config_settings (category_id, key, value, data_type, description, hot_reload, required, sensitive) VALUES
-- Interactive Brokers
((SELECT id FROM config_categories WHERE name = 'interactive_brokers'), 'enabled', 'false', 'boolean', 'Enable Interactive Brokers connectivity', TRUE, TRUE, FALSE),
((SELECT id FROM config_categories WHERE name = 'interactive_brokers'), 'tws_host', 'localhost', 'string', 'TWS host address', FALSE, TRUE, FALSE),
((SELECT id FROM config_categories WHERE name = 'interactive_brokers'), 'tws_port', '7497', 'number', 'TWS port number', FALSE, TRUE, FALSE),
((SELECT id FROM config_categories WHERE name = 'interactive_brokers'), 'client_id', '1', 'number', 'TWS client ID', FALSE, TRUE, FALSE),
((SELECT id FROM config_categories WHERE name = 'interactive_brokers'), 'account_id', '', 'encrypted', 'IB account ID', FALSE, FALSE, TRUE),

-- ICMarkets
((SELECT id FROM config_categories WHERE name = 'icmarkets'), 'enabled', 'false', 'boolean', 'Enable ICMarkets connectivity', TRUE, TRUE, FALSE),
((SELECT id FROM config_categories WHERE name = 'icmarkets'), 'fix_host', '', 'string', 'FIX server host', FALSE, FALSE, FALSE),
((SELECT id FROM config_categories WHERE name = 'icmarkets'), 'fix_port', '5201', 'number', 'FIX server port', FALSE, FALSE, FALSE),
((SELECT id FROM config_categories WHERE name = 'icmarkets'), 'sender_comp_id', '', 'encrypted', 'FIX sender comp ID', FALSE, FALSE, TRUE),
((SELECT id FROM config_categories WHERE name = 'icmarkets'), 'target_comp_id', '', 'encrypted', 'FIX target comp ID', FALSE, FALSE, TRUE),
((SELECT id FROM config_categories WHERE name = 'icmarkets'), 'username', '', 'encrypted', 'ICMarkets username', FALSE, FALSE, TRUE),
((SELECT id FROM config_categories WHERE name = 'icmarkets'), 'password', '', 'encrypted', 'ICMarkets password', FALSE, FALSE, TRUE),

-- Security Configuration
INSERT INTO config_settings (category_id, key, value, data_type, description, hot_reload, required, sensitive) VALUES
((SELECT id FROM config_categories WHERE name = 'security'), 'encryption_key_rotation_days', '90', 'number', 'Encryption key rotation period', FALSE, TRUE, FALSE),
((SELECT id FROM config_categories WHERE name = 'security'), 'session_timeout_minutes', '60', 'number', 'TLI session timeout', TRUE, TRUE, FALSE),
((SELECT id FROM config_categories WHERE name = 'security'), 'max_failed_auth_attempts', '5', 'number', 'Maximum failed authentication attempts', TRUE, TRUE, FALSE),
((SELECT id FROM config_categories WHERE name = 'security'), 'audit_log_retention_days', '365', 'number', 'Audit log retention period', TRUE, TRUE, FALSE);

Configuration Validation Rules

-- Insert validation schemas for different data types
INSERT INTO config_validation_schemas (name, schema_definition, description) VALUES
('percentage', '{"type": "number", "minimum": 0, "maximum": 1}', 'Percentage value between 0 and 1'),
('positive_number', '{"type": "number", "minimum": 0}', 'Positive numeric value'),
('log_level', '{"type": "string", "enum": ["trace", "debug", "info", "warn", "error"]}', 'Valid log levels'),
('url', '{"type": "string", "format": "uri"}', 'Valid URL format'),
('api_key', '{"type": "string", "minLength": 8}', 'API key with minimum length'),
('email', '{"type": "string", "format": "email"}', 'Valid email address');

PHASE 3: TLI DASHBOARD FRAMEWORK

Multi-Dashboard Architecture

use ratatui::prelude::*;
use tokio::sync::mpsc;
use std::collections::HashMap;

pub struct DashboardManager {
    pub active_dashboard: DashboardType,
    pub dashboards: HashMap<DashboardType, Box<dyn Dashboard>>,
    pub grpc_client_pool: GrpcClientPool,
    pub data_streams: DataStreamManager,
    pub config_manager: ConfigManager,
    pub event_receiver: mpsc::Receiver<DashboardEvent>,
}

#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum DashboardType {
    Trading,      // Live positions, orders, executions, market data
    Risk,         // VaR, drawdown, position limits, safety controls
    ML,           // Model predictions, signal strength, confidence
    Performance,  // PnL, Sharpe ratios, strategy performance
    Config,       // System configuration management
}

pub trait Dashboard {
    fn render(&mut self, frame: &mut Frame, area: Rect) -> Result<(), Box<dyn std::error::Error>>;
    fn handle_input(&mut self, key: KeyEvent) -> Result<Option<DashboardEvent>, Box<dyn std::error::Error>>;
    fn update(&mut self, event: DashboardEvent) -> Result<(), Box<dyn std::error::Error>>;
    fn title(&self) -> &str;
    fn shortcut_key(&self) -> char;
}

#[derive(Debug, Clone)]
pub enum DashboardEvent {
    // Navigation
    SwitchDashboard(DashboardType),
    Exit,

    // Data updates
    MarketDataUpdate(MarketDataEvent),
    PositionUpdate(PositionEvent),
    OrderUpdate(OrderEvent),
    RiskMetricsUpdate(RiskMetricsEvent),
    MLPredictionUpdate(MLPredictionEvent),
    ConfigurationUpdate(ConfigurationEvent),

    // User actions
    PlaceOrder(OrderRequest),
    CancelOrder(OrderId),
    UpdateConfiguration(ConfigUpdate),
    TriggerEmergencyStop,

    // System events
    ConnectionStatus(ConnectionEvent),
    Error(String),
}

Ratatui Layout System

use ratatui::{
    layout::{Constraint, Direction, Layout, Rect},
    style::{Color, Modifier, Style},
    widgets::{Block, Borders, Paragraph, Tabs},
    Frame,
};

pub struct LayoutManager {
    header_height: u16,
    footer_height: u16,
    sidebar_width: u16,
}

impl LayoutManager {
    pub fn new() -> Self {
        Self {
            header_height: 3,
            footer_height: 3,
            sidebar_width: 20,
        }
    }

    pub fn create_layout(&self, area: Rect) -> (Rect, Rect, Rect, Rect) {
        let main_layout = Layout::default()
            .direction(Direction::Vertical)
            .constraints([
                Constraint::Length(self.header_height),
                Constraint::Min(0),
                Constraint::Length(self.footer_height),
            ])
            .split(area);

        let content_layout = Layout::default()
            .direction(Direction::Horizontal)
            .constraints([
                Constraint::Min(0),
                Constraint::Length(self.sidebar_width),
            ])
            .split(main_layout[1]);

        (
            main_layout[0], // header
            content_layout[0], // main content
            content_layout[1], // sidebar
            main_layout[2], // footer
        )
    }
}

// Global UI layout structure
/*
┌─ HEADER BAR ─────────────────────────────────────────────────────┐
│ [T]rading [R]isk [M]L [P]erf [C]onfig | Connected: ●●● | 14:35:21 │
├──────────────────────────────────────────────────┬───────────────┤
│                                                  │   SIDEBAR     │
│              MAIN DASHBOARD CONTENT              │               │
│               (Dashboard-Specific)               │  Quick Stats  │
│                                                  │   Alerts      │
│                                                  │ Health Status │
├──────────────────────────────────────────────────┴───────────────┤
│ [F1] Help [F2] Alerts [F3] Export [ESC] Menu | Status: ACTIVE   │
└──────────────────────────────────────────────────────────────────┘
*/

Real-Time Data Stream Management

use tokio::sync::{broadcast, mpsc};
use tonic::Streaming;

pub struct DataStreamManager {
    // Individual stream receivers
    market_data_rx: broadcast::Receiver<MarketDataEvent>,
    position_rx: broadcast::Receiver<PositionEvent>,
    order_rx: broadcast::Receiver<OrderEvent>,
    risk_rx: broadcast::Receiver<RiskMetricsEvent>,
    ml_rx: broadcast::Receiver<MLPredictionEvent>,
    config_rx: broadcast::Receiver<ConfigurationEvent>,

    // Dashboard event sender
    dashboard_tx: mpsc::Sender<DashboardEvent>,

    // Stream health monitoring
    connection_status: HashMap<String, ConnectionStatus>,
}

impl DataStreamManager {
    pub async fn start_all_streams(&mut self) -> Result<(), Box<dyn std::error::Error>> {
        // Start all gRPC streaming connections concurrently
        let market_data_task = self.start_market_data_stream();
        let position_task = self.start_position_stream();
        let order_task = self.start_order_stream();
        let risk_task = self.start_risk_stream();
        let ml_task = self.start_ml_stream();
        let config_task = self.start_config_stream();

        tokio::try_join!(
            market_data_task,
            position_task,
            order_task,
            risk_task,
            ml_task,
            config_task
        )?;

        Ok(())
    }

    async fn start_market_data_stream(&mut self) -> Result<(), Box<dyn std::error::Error>> {
        // Implementation for market data streaming
        Ok(())
    }

    // Additional stream implementations...
}

PHASE 4: INDIVIDUAL DASHBOARD IMPLEMENTATIONS

Trading Dashboard

┌─ TRADING DASHBOARD ──────────────────────────────────────────────┐
│ Market Data      │ Active Positions  │ Order Book │ Executions  │
│ AAPL: $150.25 ↑  │ AAPL: +1000 @150 │ Bid: 150.20│ AAPL +500   │
│ TSLA: $800.50 ↓  │ TSLA: -500 @800  │     150.15 │ @150.25     │
│ SPY:  $420.10 ↑  │ SPY:  +2000 @420 │     150.10 │ 14:35:21    │
│ QQQ:  $350.75 ↑  │ QQQ:  -1500 @350 │ Ask: 150.30│             │
│                  │                  │     150.35 │ TSLA -200   │
│                  │                  │     150.40 │ @800.75     │
├──────────────────┼──────────────────┼────────────┼─────────────┤
│ Order Entry      │ PnL Summary      │ Strategy Status          │
│ Symbol: [AAPL  ] │ Daily: +$2,500   │ Strategy: ACTIVE         │
│ Side: [BUY ▼]    │ Total: +$15,000  │ Models: 6/6 ONLINE       │
│ Qty: [500    ]   │ Unrealized: -$500│ Risk: GREEN              │
│ Price: [MKT ▼]   │ Win Rate: 67%    │ Last Signal: BUY 85%     │
│ [SUBMIT ORDER]   │ Sharpe: 1.85     │ Next Review: 14:40       │
└──────────────────┴──────────────────┴──────────────────────────┘

Risk Dashboard

┌─ RISK DASHBOARD ─────────────────────────────────────────────────┐
│ VaR Metrics      │ Position Limits  │ Drawdown Monitor         │
│ 1-Day: $5,000    │ Max Per Symbol:  │ Current: -2.5%           │
│ 5-Day: $8,000    │ $100K (50% used) │ Max Daily: -5.0%         │
│ 30-Day: $12,000  │ Total Exposure:  │ Max Lifetime: -15.0%     │
│ Confidence: 95%  │ $2.5M (80% used) │ Time in DD: 2h 15m       │
│ Method: MC       │ Concentration:   │ Recovery Time: 1h 45m    │
│ Last Calc: 14:30 │ 25% (limit 30%)  │ ██████████░░░░░░         │
├──────────────────┼──────────────────┼──────────────────────────┤
│ Safety Controls  │ Circuit Breakers │ Emergency Actions        │
│ Kill Switch:     │ Portfolio: OFF   │ [EMERGENCY STOP]         │
│ ●●●● ACTIVE      │ Symbol: OFF      │ [FLATTEN ALL]            │
│ Auto Recovery:   │ Strategy: OFF    │ [RISK OVERRIDE]          │
│ ●●●● ENABLED     │ Volatility: OFF  │ [CONTACT SUPPORT]        │
│ Last Test: 14:00 │                  │ [EXPORT POSITIONS]       │
└──────────────────┴──────────────────┴──────────────────────────┘

ML Dashboard

┌─ ML DASHBOARD ───────────────────────────────────────────────────┐
│ Model Status     │ Signal Strength  │ Prediction Confidence    │
│ DQN: ●●● ACTIVE  │ AAPL: ██████ 85% │ Next 1m: BUY (92%)       │
│ MAMBA: ●●● ACTIVE│ TSLA: ███▒▒▒ 60% │ Next 5m: HOLD (78%)      │
│ TFT: ●●● ACTIVE  │ SPY:  ████▒▒ 70% │ Next 15m: SELL (65%)     │
│ LIQUID: ●●● ACTV │ QQQ:  ██▒▒▒▒ 40% │ Ensemble: BUY (82%)      │
│ TLOB: ●●● ACTIVE │ BTC:  ████▒▒ 68% │ Volatility: HIGH         │
│ PPO: ●●● ACTIVE  │ ETH:  ███▒▒▒ 55% │ Market Regime: TRENDING  │
├──────────────────┼──────────────────┼──────────────────────────┤
│ Ensemble Vote    │ Model Performance│ Feature Importance       │
│ BUY: 4/6 models  │ DQN: 67% Win     │ Price: ████████ 45%      │
│ SELL: 2/6 models │ MAMBA: 72% Win   │ Volume: ██████▒ 32%      │
│ Confidence: 82%  │ TFT: 69% Win     │ Time: ████▒▒▒ 23%        │
│ Strength: HIGH   │ Avg: 69% Win     │ Volatility: ██▒▒ 15%     │
│ Last Update: Now │ Best: MAMBA      │ Momentum: ███▒▒ 18%      │
└──────────────────┴──────────────────┴──────────────────────────┘

Performance Dashboard

┌─ PERFORMANCE DASHBOARD ──────────────────────────────────────────┐
│ Portfolio Metrics│ Strategy Returns │ Risk-Adjusted Metrics    │
│ Total Return:    │ Daily: +1.25%    │ Sharpe Ratio: 1.85       │
│ +15.67% YTD      │ Weekly: +5.67%   │ Sortino Ratio: 2.34      │
│ +8.23% MTD       │ Monthly: +12.34% │ Calmar Ratio: 3.12       │
│ +1.25% Daily     │ YTD: +15.67%     │ Information Ratio: 1.67   │
│                  │                  │                          │
│ Alpha: +2.34%    │ Beta: 0.87       │ Max Drawdown: -5.67%     │
├──────────────────┼──────────────────┼──────────────────────────┤
│ Trade Statistics │ Win/Loss Analysis│ Model Performance        │
│ Total Trades: 247│ Winners: 165     │ Best Model: MAMBA        │
│ Avg Trade: +$125 │ Losers: 82       │ Worst Model: PPO         │
│ Win Rate: 66.8%  │ Win Rate: 66.8%  │ Ensemble Accuracy: 72%   │
│ Profit Factor:   │ Avg Win: +$245   │ Signal Quality: HIGH     │
│ 2.15             │ Avg Loss: -$95   │ Model Drift: NONE        │
│ Best Trade: +$750│ Largest Loss:-$89│ Last Retrain: 2 days    │
└──────────────────┴──────────────────┴──────────────────────────┘

Configuration Dashboard

┌─ CONFIGURATION DASHBOARD ────────────────────────────────────────┐
│ Category Tree    │ Settings Editor  │ Validation & History     │
│ ▼ System         │ Key: log_level   │ Status: ✓ VALID          │
│   ├─ Logging     │ Value: [info ▼]  │ Type: string             │
│   ├─ Database    │ Description:     │ Required: Yes            │
│   └─ gRPC        │ Global log level │ Hot Reload: Yes          │
│ ▼ Trading        │ for all services │                          │
│   ├─ Execution   │                  │ Recent Changes:          │
│   ├─ Strategies  │ [SAVE CHANGES]   │ 14:30 - risk.var_conf   │
│   └─ Position    │ [RESET]          │ 14:25 - ml.model_thresh │
│ ▼ Risk           │ [VALIDATE]       │ 14:20 - trade.max_order │
│   ├─ VaR         │                  │                          │
│   ├─ Limits      │ Validation:      │ [VIEW HISTORY]           │
│   └─ Alerts      │ ✓ Format OK      │ [EXPORT CONFIG]          │
│ ▼ ML             │ ✓ Range OK       │ [IMPORT CONFIG]          │
│   ├─ Models      │ ✓ Dependencies   │ [ROLLBACK]               │
│   └─ Training    │ ✓ Ready to apply │                          │
└──────────────────┴──────────────────┴──────────────────────────┘

PHASE 5: REAL-TIME EVENT STREAMING SYSTEM

High-Performance Event Publisher

use tokio::sync::broadcast;
use serde::{Deserialize, Serialize};

pub struct EventPublisher {
    // Separate channels for different event types
    market_data_tx: broadcast::Sender<MarketDataEvent>,
    trading_tx: broadcast::Sender<TradingEvent>,
    risk_tx: broadcast::Sender<RiskEvent>,
    ml_tx: broadcast::Sender<MLEvent>,
    config_tx: broadcast::Sender<ConfigEvent>,

    // Channel capacity and overflow handling
    channel_capacity: usize,
    overflow_strategy: OverflowStrategy,
}

#[derive(Debug, Clone)]
pub enum OverflowStrategy {
    DropOldest,
    DropNewest,
    Block,
}

impl EventPublisher {
    pub fn new(capacity: usize, strategy: OverflowStrategy) -> Self {
        let (market_data_tx, _) = broadcast::channel(capacity);
        let (trading_tx, _) = broadcast::channel(capacity);
        let (risk_tx, _) = broadcast::channel(capacity);
        let (ml_tx, _) = broadcast::channel(capacity);
        let (config_tx, _) = broadcast::channel(capacity);

        Self {
            market_data_tx,
            trading_tx,
            risk_tx,
            ml_tx,
            config_tx,
            channel_capacity: capacity,
            overflow_strategy: strategy,
        }
    }

    // Non-blocking event publishing with overflow handling
    pub fn publish_market_data(&self, event: MarketDataEvent) -> Result<(), PublishError> {
        match self.market_data_tx.try_send(event) {
            Ok(_) => Ok(()),
            Err(broadcast::error::TrySendError::Full(_)) => {
                match self.overflow_strategy {
                    OverflowStrategy::DropNewest => Err(PublishError::Dropped),
                    OverflowStrategy::DropOldest => {
                        // Force send to drop oldest
                        let _ = self.market_data_tx.send(event);
                        Ok(())
                    },
                    OverflowStrategy::Block => Err(PublishError::WouldBlock),
                }
            },
            Err(broadcast::error::TrySendError::Closed(_)) => Err(PublishError::ChannelClosed),
        }
    }

    pub fn publish_execution(&self, execution: ExecutionEvent) -> Result<(), PublishError> {
        let event = TradingEvent::Execution(execution);
        self.try_publish(&self.trading_tx, event)
    }

    pub fn publish_risk_alert(&self, alert: RiskAlert) -> Result<(), PublishError> {
        let event = RiskEvent::Alert(alert);
        self.try_publish(&self.risk_tx, event)
    }

    pub fn publish_ml_prediction(&self, prediction: MLPrediction) -> Result<(), PublishError> {
        let event = MLEvent::Prediction(prediction);
        self.try_publish(&self.ml_tx, event)
    }

    pub fn publish_config_change(&self, change: ConfigChange) -> Result<(), PublishError> {
        let event = ConfigEvent::Change(change);
        self.try_publish(&self.config_tx, event)
    }

    fn try_publish<T>(&self, tx: &broadcast::Sender<T>, event: T) -> Result<(), PublishError>
    where
        T: Clone,
    {
        match tx.try_send(event) {
            Ok(_) => Ok(()),
            Err(broadcast::error::TrySendError::Full(event)) => {
                match self.overflow_strategy {
                    OverflowStrategy::DropNewest => Err(PublishError::Dropped),
                    OverflowStrategy::DropOldest => {
                        let _ = tx.send(event);
                        Ok(())
                    },
                    OverflowStrategy::Block => Err(PublishError::WouldBlock),
                }
            },
            Err(broadcast::error::TrySendError::Closed(_)) => Err(PublishError::ChannelClosed),
        }
    }
}

#[derive(Debug, Clone)]
pub enum PublishError {
    Dropped,
    WouldBlock,
    ChannelClosed,
}

// Event type definitions
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct MarketDataEvent {
    pub symbol: String,
    pub price: f64,
    pub volume: u64,
    pub timestamp: i64,
    pub event_type: MarketDataType,
}

#[derive(Debug, Clone, Serialize, Deserialize)]
pub enum MarketDataType {
    Trade,
    Quote,
    OrderBook,
    News,
}

#[derive(Debug, Clone, Serialize, Deserialize)]
pub enum TradingEvent {
    Execution(ExecutionEvent),
    OrderPlaced(OrderEvent),
    OrderCancelled(OrderEvent),
    PositionUpdate(PositionEvent),
}

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ExecutionEvent {
    pub order_id: String,
    pub symbol: String,
    pub side: Side,
    pub quantity: f64,
    pub price: f64,
    pub timestamp: i64,
    pub execution_id: String,
}

#[derive(Debug, Clone, Serialize, Deserialize)]
pub enum RiskEvent {
    Alert(RiskAlert),
    VaRUpdate(VaRUpdate),
    DrawdownUpdate(DrawdownUpdate),
    LimitBreach(LimitBreach),
}

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct RiskAlert {
    pub alert_type: RiskAlertType,
    pub severity: AlertSeverity,
    pub message: String,
    pub timestamp: i64,
    pub affected_symbols: Vec<String>,
}

#[derive(Debug, Clone, Serialize, Deserialize)]
pub enum MLEvent {
    Prediction(MLPrediction),
    ModelUpdate(ModelUpdate),
    SignalStrength(SignalStrength),
    EnsembleVote(EnsembleVote),
}

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct MLPrediction {
    pub model_name: String,
    pub symbol: String,
    pub prediction: PredictionType,
    pub confidence: f64,
    pub timestamp: i64,
    pub features: Vec<f64>,
}

#[derive(Debug, Clone, Serialize, Deserialize)]
pub enum ConfigEvent {
    Change(ConfigChange),
    Validation(ConfigValidation),
    Rollback(ConfigRollback),
}

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ConfigChange {
    pub setting_id: i64,
    pub category: String,
    pub key: String,
    pub old_value: String,
    pub new_value: String,
    pub changed_by: String,
    pub timestamp: i64,
    pub hot_reload: bool,
}

Configuration Hot-Reload System

use sqlx::SqlitePool;
use tokio::sync::watch;
use std::collections::HashMap;

pub struct ConfigManager {
    db_pool: SqlitePool,
    config_cache: Arc<RwLock<HashMap<String, ConfigValue>>>,
    change_notifiers: HashMap<String, watch::Sender<ConfigValue>>,
    event_publisher: Arc<EventPublisher>,
}

impl ConfigManager {
    pub async fn new(db_pool: SqlitePool, event_publisher: Arc<EventPublisher>) -> Result<Self, ConfigError> {
        let mut manager = Self {
            db_pool,
            config_cache: Arc::new(RwLock::new(HashMap::new())),
            change_notifiers: HashMap::new(),
            event_publisher,
        };

        // Load all configuration on startup
        manager.load_all_configuration().await?;

        // Start configuration change monitoring
        manager.start_change_monitor().await?;

        Ok(manager)
    }

    pub async fn get_config<T>(&self, key: &str) -> Result<T, ConfigError>
    where
        T: for<'de> Deserialize<'de>,
    {
        let cache = self.config_cache.read().await;
        if let Some(value) = cache.get(key) {
            serde_json::from_str(&value.value)
                .map_err(|e| ConfigError::DeserializationError(e.to_string()))
        } else {
            Err(ConfigError::KeyNotFound(key.to_string()))
        }
    }

    pub async fn update_config(&self, key: &str, value: serde_json::Value, changed_by: &str) -> Result<(), ConfigError> {
        // Start transaction for atomic update
        let mut tx = self.db_pool.begin().await?;

        // Get current value for history
        let current_value = sqlx::query_as::<_, (String, bool)>(
            "SELECT value, hot_reload FROM config_settings WHERE key = ?"
        )
        .bind(key)
        .fetch_optional(&mut *tx)
        .await?;

        let (old_value, hot_reload) = current_value
            .ok_or_else(|| ConfigError::KeyNotFound(key.to_string()))?;

        let new_value_str = value.to_string();

        // Validate the new value
        self.validate_config_value(key, &new_value_str).await?;

        // Update the configuration
        sqlx::query(
            "UPDATE config_settings SET value = ?, modified_at = CURRENT_TIMESTAMP WHERE key = ?"
        )
        .bind(&new_value_str)
        .bind(key)
        .execute(&mut *tx)
        .await?;

        // Add to history
        sqlx::query(
            "INSERT INTO config_history (setting_id, old_value, new_value, changed_by)
             VALUES ((SELECT id FROM config_settings WHERE key = ?), ?, ?, ?)"
        )
        .bind(key)
        .bind(&old_value)
        .bind(&new_value_str)
        .bind(changed_by)
        .execute(&mut *tx)
        .await?;

        // Commit transaction
        tx.commit().await?;

        // Update cache
        {
            let mut cache = self.config_cache.write().await;
            cache.insert(key.to_string(), ConfigValue {
                value: new_value_str.clone(),
                hot_reload,
            });
        }

        // Notify subscribers if hot reload is enabled
        if hot_reload {
            if let Some(notifier) = self.change_notifiers.get(key) {
                let _ = notifier.send(ConfigValue {
                    value: new_value_str.clone(),
                    hot_reload,
                });
            }

            // Publish configuration change event
            let change_event = ConfigChange {
                setting_id: 0, // TODO: Get actual setting ID
                category: self.get_category_for_key(key).await?,
                key: key.to_string(),
                old_value,
                new_value: new_value_str,
                changed_by: changed_by.to_string(),
                timestamp: chrono::Utc::now().timestamp(),
                hot_reload,
            };

            let _ = self.event_publisher.publish_config_change(change_event);
        }

        Ok(())
    }

    pub async fn subscribe_to_changes(&mut self, key: &str) -> watch::Receiver<ConfigValue> {
        if let Some(notifier) = self.change_notifiers.get(key) {
            notifier.subscribe()
        } else {
            let current_value = self.config_cache.read().await
                .get(key)
                .cloned()
                .unwrap_or_else(|| ConfigValue {
                    value: "".to_string(),
                    hot_reload: false,
                });

            let (tx, rx) = watch::channel(current_value);
            self.change_notifiers.insert(key.to_string(), tx);
            rx
        }
    }

    async fn validate_config_value(&self, key: &str, value: &str) -> Result<(), ConfigError> {
        // Get validation rule for the key
        let validation_rule = sqlx::query_as::<_, (Option<String>,)>(
            "SELECT validation_rule FROM config_settings WHERE key = ?"
        )
        .bind(key)
        .fetch_optional(&self.db_pool)
        .await?;

        if let Some((Some(rule))) = validation_rule {
            // Validate using JSON schema
            let schema: serde_json::Value = serde_json::from_str(&rule)?;
            // TODO: Implement JSON schema validation
            // For now, just basic type checking
        }

        Ok(())
    }

    async fn load_all_configuration(&mut self) -> Result<(), ConfigError> {
        let configs = sqlx::query_as::<_, (String, String, bool)>(
            "SELECT key, value, hot_reload FROM config_settings"
        )
        .fetch_all(&self.db_pool)
        .await?;

        let mut cache = self.config_cache.write().await;
        for (key, value, hot_reload) in configs {
            cache.insert(key, ConfigValue { value, hot_reload });
        }

        Ok(())
    }

    async fn start_change_monitor(&self) -> Result<(), ConfigError> {
        // TODO: Implement file system watcher or database trigger
        // to monitor configuration changes from external sources
        Ok(())
    }

    async fn get_category_for_key(&self, key: &str) -> Result<String, ConfigError> {
        let category = sqlx::query_as::<_, (String,)>(
            "SELECT c.name FROM config_categories c
             JOIN config_settings s ON c.id = s.category_id
             WHERE s.key = ?"
        )
        .bind(key)
        .fetch_one(&self.db_pool)
        .await?;

        Ok(category.0)
    }
}

#[derive(Debug, Clone)]
pub struct ConfigValue {
    pub value: String,
    pub hot_reload: bool,
}

#[derive(Debug, thiserror::Error)]
pub enum ConfigError {
    #[error("Configuration key not found: {0}")]
    KeyNotFound(String),
    #[error("Database error: {0}")]
    DatabaseError(#[from] sqlx::Error),
    #[error("JSON error: {0}")]
    JsonError(#[from] serde_json::Error),
    #[error("Validation error: {0}")]
    ValidationError(String),
    #[error("Deserialization error: {0}")]
    DeserializationError(String),
}

PHASE 6: IMPLEMENTATION ROADMAP

Development Timeline (8-10 Weeks)

Week 1-2: Backend Foundation

# Trading Service gRPC Infrastructure
Tasks:
- Implement EventPublisher with broadcast channels for all event types
- Create gRPC service implementations (Trading, Risk, ML, Config)
- Add SQLite configuration database with comprehensive schema
- Implement ConfigManager with hot-reload mechanism
- Add configuration validation engine with JSON schema support
- Implement encrypted storage for sensitive configuration (API keys)

Deliverables:
- Working gRPC server with all 4 services
- SQLite database with full configuration schema
- Configuration hot-reload system
- Encrypted storage for API keys and credentials

Testing:
- Unit tests for all gRPC service methods
- Configuration validation testing
- Hot-reload mechanism testing
- Encryption/decryption testing for sensitive data

Week 3-4: TLI Framework Development

# Core TLI Infrastructure
Tasks:
- Build DashboardManager with Ratatui integration
- Implement gRPC client pool with connection management
- Create dashboard navigation system with keyboard shortcuts
- Add real-time data stream handling with Tokio
- Implement layout manager for consistent UI structure
- Add connection health monitoring and auto-reconnection

Deliverables:
- Working TLI framework with navigation
- gRPC client connectivity to trading service
- Real-time data stream infrastructure
- Layout system for all dashboards

Testing:
- TLI framework integration tests
- gRPC client connection testing
- UI navigation testing
- Stream handling performance tests

Week 5-6: Dashboard Implementation Phase 1

# Core Dashboards (Trading, Risk, ML)
Tasks:
- Trading Dashboard: positions, orders, market data, executions
- Risk Dashboard: VaR metrics, limits, drawdown, safety controls
- ML Dashboard: predictions, signals, model performance
- Implement real-time data visualization widgets
- Add interactive controls for order entry and risk management
- Create emergency stop and safety control interfaces

Deliverables:
- Fully functional Trading Dashboard
- Complete Risk Dashboard with safety controls
- ML Dashboard with model insights
- Real-time data updates across all dashboards

Testing:
- Dashboard rendering performance tests
- Real-time update testing
- User interaction testing
- Safety control testing

Week 7-8: Dashboard Implementation Phase 2 & Integration

# Remaining Dashboards and System Integration
Tasks:
- Performance Dashboard: analytics, Sharpe ratios, returns
- Configuration Dashboard: settings management, live updates
- End-to-end testing with live data streams
- Performance optimization for HFT requirements
- Remote connectivity testing and security
- Documentation and deployment preparation

Deliverables:
- Complete Performance Dashboard with analytics
- Fully functional Configuration Dashboard
- Production-ready TLI system
- Complete documentation and deployment guides

Testing:
- End-to-end system testing
- Performance benchmarking
- Security and remote access testing
- Load testing with high-frequency data

Success Criteria & Performance Targets

Performance Requirements

• Real-time data latency: < 10ms from trading service to TLI display
• UI responsiveness: Smooth updates at 30+ FPS without blocking
• Configuration updates: Applied within 1 second of change
• Memory usage: < 100MB for TLI client under normal operation
• Trading service impact: Zero measurable performance degradation
• Remote operation: Stable operation over WAN connections with < 100ms RTT

Functional Requirements

• Dashboard switching: Sub-100ms response time for navigation
• Data accuracy: 100% accuracy in real-time data display
• Configuration validation: All invalid configurations rejected with clear error messages
• Error recovery: Automatic recovery from network disconnections
• Security: All sensitive configuration data encrypted at rest

Deliverables Checklist

• Production-Ready TLI Terminal - Complete 5-dashboard system
• High-Performance gRPC Streaming - Real-time data feeds with < 10ms latency
• SQLite Configuration Management - Live configuration updates with validation
• Remote Operation Capability - Local and remote connectivity with security
• Comprehensive Trading Oversight - Complete system monitoring and control
• Encrypted Configuration Storage - Secure storage for API keys and credentials
• Documentation - Complete user and deployment documentation
• Testing Suite - Comprehensive test coverage for all components

Deployment Architecture

Local Deployment

┌─────────────────┐    ┌─────────────────┐
│   TLI Client    │    │ Trading Service │
│  (Terminal UI)  │<-->│   (Monolith)    │
│                 │    │                 │
│ - Dashboards    │    │ - gRPC Server   │
│ - Config Mgmt   │    │ - Event Publish │
│ - Real-time UI  │    │ - SQLite Config │
└─────────────────┘    └─────────────────┘
        │                        │
        └────────── localhost ───┘

Remote Deployment

┌─────────────────┐    Network     ┌─────────────────┐
│   TLI Client    │   (Internet/   │ Trading Service │
│  (Local/Remote) │    VPN/LAN)    │    (Remote)     │
│                 │<-------------->│                 │
│ - Dashboards    │    gRPC/TLS    │ - gRPC Server   │
│ - Config Mgmt   │    Auth        │ - Event Publish │
│ - Real-time UI  │    Security    │ - SQLite Config │
└─────────────────┘                └─────────────────┘

Security Considerations

• gRPC TLS encryption for all remote communications
• Authentication tokens for client verification
• API key encryption in SQLite database
• Network security with VPN or firewall rules
• Audit logging for all configuration changes
• Session management with configurable timeouts

This comprehensive plan provides a complete roadmap for implementing a production-ready TLI system with real-time trading insights, comprehensive configuration management, and secure remote operation capabilities.