# FXT Dashboard & CLI Completion Implementation Plan > **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. **Goal:** Finalize `fxt` as a fully functional CLI with a live ratatui streaming dashboard (`fxt watch`) and all stub commands wired to real gRPC backends. **Architecture:** New `watch` subcommand with ratatui TUI, 4 tabs (Training/Trading/Risk/System) fed by gRPC server-side streams via `tokio::select!` event loop. Stub commands wired to existing RPCs. Two new proto RPCs (ApproveModel/RejectModel) added to ml_training.proto. **Tech Stack:** Rust, ratatui 0.29, crossterm 0.28, tonic gRPC, tokio --- ### Task 1: Add Dependencies **Files:** - Modify: `bin/fxt/Cargo.toml` **Step 1: Add ratatui and crossterm to dependencies** In `bin/fxt/Cargo.toml`, add after the existing `[dependencies]` entries: ```toml ratatui = "0.29" crossterm = { version = "0.28", features = ["event-stream"] } ``` **Step 2: Verify compilation** Run: `SQLX_OFFLINE=true cargo check -p fxt` Expected: compiles with 0 errors **Step 3: Commit** ```bash git add bin/fxt/Cargo.toml git commit -m "feat(fxt): add ratatui and crossterm dependencies" ``` --- ### Task 2: Add ApproveModel / RejectModel Proto RPCs **Files:** - Modify: `bin/fxt/proto/ml_training.proto` **Step 1: Add messages and RPCs** In `bin/fxt/proto/ml_training.proto`, add two RPCs to the `MLTrainingService` service block (after the existing `StreamTuningProgress` RPC at line ~47): ```protobuf // Model promotion management rpc ApproveModel(ApproveModelRequest) returns (ApproveModelResponse); rpc RejectModel(RejectModelRequest) returns (RejectModelResponse); ``` Add messages at the end of the file: ```protobuf // Model promotion messages message ApproveModelRequest { string model_id = 1; string promoted_to = 2; // e.g. "production", "staging", "canary" } message ApproveModelResponse { bool success = 1; string message = 2; } message RejectModelRequest { string model_id = 1; string reason = 2; } message RejectModelResponse { bool success = 1; string message = 2; } ``` **Step 2: Verify proto compiles** Run: `SQLX_OFFLINE=true cargo check -p fxt` Expected: compiles (tonic-prost-build picks up new messages automatically) **Step 3: Commit** ```bash git add bin/fxt/proto/ml_training.proto git commit -m "feat(fxt): add ApproveModel and RejectModel proto RPCs" ``` --- ### Task 3: Dashboard State Types **Files:** - Create: `bin/fxt/src/commands/watch/state.rs` - Create: `bin/fxt/src/commands/watch/mod.rs` **Step 1: Create watch module directory** ```bash mkdir -p bin/fxt/src/commands/watch ``` **Step 2: Write state types** Create `bin/fxt/src/commands/watch/state.rs`: ```rust use std::collections::VecDeque; const CHART_HISTORY: usize = 200; #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum Tab { Training, Trading, Risk, System, } impl Tab { pub const ALL: [Tab; 4] = [Tab::Training, Tab::Trading, Tab::Risk, Tab::System]; pub fn title(&self) -> &'static str { match self { Tab::Training => "Training", Tab::Trading => "Trading", Tab::Risk => "Risk", Tab::System => "System", } } pub fn index(&self) -> usize { match self { Tab::Training => 0, Tab::Trading => 1, Tab::Risk => 2, Tab::System => 3, } } } #[derive(Debug, Default)] pub struct DashboardState { pub active_tab: usize, pub training: TrainingTabState, pub trading: TradingTabState, pub risk: RiskTabState, pub system: SystemTabState, pub dirty: bool, } impl DashboardState { pub fn current_tab(&self) -> Tab { Tab::ALL.get(self.active_tab).copied().unwrap_or(Tab::Training) } pub fn select_tab(&mut self, index: usize) { if index < Tab::ALL.len() { self.active_tab = index; self.dirty = true; } } pub fn scroll_up(&mut self) { match self.current_tab() { Tab::Training => self.training.scroll_up(), Tab::Trading => self.trading.scroll_up(), Tab::Risk => self.risk.scroll_up(), Tab::System => self.system.scroll_up(), } self.dirty = true; } pub fn scroll_down(&mut self) { match self.current_tab() { Tab::Training => self.training.scroll_down(), Tab::Trading => self.trading.scroll_down(), Tab::Risk => self.risk.scroll_down(), Tab::System => self.system.scroll_down(), } self.dirty = true; } } // ── Training Tab ── #[derive(Debug, Clone)] pub struct TrainingSession { pub model: String, pub fold: String, pub is_hyperopt: bool, pub epoch: f32, pub epoch_loss: f32, pub validation_loss: f32, pub learning_rate: f32, pub gpu_percent: f32, pub batches_per_second: f32, pub gradient_norm: f32, pub epoch_duration_seconds: f32, } #[derive(Debug, Default)] pub struct GpuInfo { pub utilization_percent: f32, pub memory_used_mb: f32, pub memory_total_mb: f32, pub temperature_celsius: f32, } #[derive(Debug, Default)] pub struct TrainingTabState { pub sessions: Vec, pub gpu: GpuInfo, pub loss_history: VecDeque, pub scroll_offset: usize, pub connected: bool, } impl TrainingTabState { pub fn push_loss(&mut self, loss: f64) { self.loss_history.push_back(loss); if self.loss_history.len() > CHART_HISTORY { self.loss_history.pop_front(); } } pub fn scroll_up(&mut self) { self.scroll_offset = self.scroll_offset.saturating_sub(1); } pub fn scroll_down(&mut self) { self.scroll_offset = self.scroll_offset.saturating_add(1); } } // ── Trading Tab ── #[derive(Debug, Clone)] pub struct PositionRow { pub symbol: String, pub side: String, pub quantity: f64, pub entry_price: f64, pub unrealized_pnl: f64, pub status: String, } #[derive(Debug, Clone)] pub struct FillRow { pub time: String, pub side: String, pub symbol: String, pub quantity: f64, pub price: f64, pub status: String, } #[derive(Debug, Default)] pub struct TradingTabState { pub positions: Vec, pub fills: VecDeque, pub day_pnl: f64, pub realized_pnl: f64, pub unrealized_pnl: f64, pub scroll_offset: usize, pub connected: bool, } impl TradingTabState { pub fn push_fill(&mut self, fill: FillRow) { self.fills.push_front(fill); if self.fills.len() > 50 { self.fills.pop_back(); } } pub fn scroll_up(&mut self) { self.scroll_offset = self.scroll_offset.saturating_sub(1); } pub fn scroll_down(&mut self) { self.scroll_offset = self.scroll_offset.saturating_add(1); } } // ── Risk Tab ── #[derive(Debug, Clone)] pub struct CircuitBreakerRow { pub name: String, pub status: String, pub current: String, pub limit: String, } #[derive(Debug, Default)] pub struct RiskTabState { pub portfolio_var: f64, pub max_drawdown: f64, pub current_drawdown: f64, pub sharpe_ratio: f64, pub circuit_breakers: Vec, pub scroll_offset: usize, pub connected: bool, } impl RiskTabState { pub fn scroll_up(&mut self) { self.scroll_offset = self.scroll_offset.saturating_sub(1); } pub fn scroll_down(&mut self) { self.scroll_offset = self.scroll_offset.saturating_add(1); } } // ── System Tab ── #[derive(Debug, Clone)] pub struct ServiceRow { pub name: String, pub status: String, pub latency_ms: f64, pub message: String, } #[derive(Debug, Default)] pub struct SystemTabState { pub services: Vec, pub gpu: GpuInfo, pub cpu_percent: f64, pub ram_used_gb: f64, pub ram_total_gb: f64, pub scroll_offset: usize, pub connected: bool, } impl SystemTabState { pub fn scroll_up(&mut self) { self.scroll_offset = self.scroll_offset.saturating_sub(1); } pub fn scroll_down(&mut self) { self.scroll_offset = self.scroll_offset.saturating_add(1); } } #[cfg(test)] #[allow(clippy::unwrap_used, clippy::expect_used)] mod tests { use super::*; #[test] fn test_tab_selection() { let mut state = DashboardState::default(); assert_eq!(state.current_tab(), Tab::Training); state.select_tab(2); assert_eq!(state.current_tab(), Tab::Risk); assert!(state.dirty); } #[test] fn test_tab_selection_out_of_bounds() { let mut state = DashboardState::default(); state.select_tab(99); assert_eq!(state.current_tab(), Tab::Training); } #[test] fn test_loss_history_ring_buffer() { let mut tab = TrainingTabState::default(); for i in 0..250 { tab.push_loss(i as f64); } assert_eq!(tab.loss_history.len(), CHART_HISTORY); assert!((tab.loss_history[0] - 50.0).abs() < f64::EPSILON); } #[test] fn test_fill_ring_buffer() { let mut tab = TradingTabState::default(); for i in 0..60 { tab.push_fill(FillRow { time: format!("{}", i), side: "BUY".to_string(), symbol: "ES".to_string(), quantity: 1.0, price: 5000.0, status: "FILLED".to_string(), }); } assert_eq!(tab.fills.len(), 50); } #[test] fn test_scroll() { let mut state = DashboardState::default(); state.scroll_down(); assert_eq!(state.training.scroll_offset, 1); state.scroll_up(); assert_eq!(state.training.scroll_offset, 0); state.scroll_up(); // should not underflow assert_eq!(state.training.scroll_offset, 0); } } ``` **Step 3: Create watch/mod.rs** Create `bin/fxt/src/commands/watch/mod.rs`: ```rust pub mod state; mod streams; mod render; mod event_loop; use anyhow::Result; pub async fn run(api_gateway_url: &str, jwt_token: &str) -> Result<()> { event_loop::run_dashboard(api_gateway_url, jwt_token).await } ``` **Step 4: Run tests** Run: `SQLX_OFFLINE=true cargo test -p fxt --lib watch::state` Expected: 5 tests pass **Step 5: Commit** ```bash git add bin/fxt/src/commands/watch/ git commit -m "feat(fxt): add dashboard state types with tests" ``` --- ### Task 4: gRPC Stream Manager **Files:** - Create: `bin/fxt/src/commands/watch/streams.rs` **Step 1: Write stream manager** Create `bin/fxt/src/commands/watch/streams.rs`. This module spawns one tokio task per gRPC stream, all feeding into a single `mpsc::Sender`. ```rust use anyhow::{Context, Result}; use tokio::sync::mpsc; use tokio::time::{sleep, Duration}; use tonic::metadata::MetadataValue; use tonic::transport::Channel; use tonic::Request; use tracing::{debug, warn}; use crate::proto::monitoring::{ monitoring_service_client::MonitoringServiceClient, StreamTrainingMetricsRequest, }; use crate::proto::foxhunt_tli::{ trading_service_client::TradingServiceClient, GetPositionsRequest, GetRiskMetricsRequest, SubscribeOrderUpdatesRequest, SubscribeRiskAlertsRequest, SubscribeSystemStatusRequest, }; use super::state::*; /// Events pushed from gRPC streams into the dashboard state. #[derive(Debug)] pub enum StreamEvent { TrainingUpdate { sessions: Vec, gpu: GpuInfo, }, OrderUpdate { order_id: String, symbol: String, status: String, filled_qty: f64, last_fill_price: f64, timestamp_nanos: i64, }, PositionsSnapshot { positions: Vec, }, RiskMetrics { var: f64, max_drawdown: f64, current_drawdown: f64, sharpe: f64, }, RiskAlert { severity: String, symbol: String, message: String, threshold: f64, current: f64, }, SystemStatus { service: String, status: String, message: String, }, StreamDisconnected { stream_name: String, }, } const MAX_BACKOFF_SECS: u64 = 30; async fn connect_channel(url: &str) -> Result { Channel::from_shared(url.to_owned()) .context("Invalid API Gateway URL")? .connect() .await .context("Failed to connect to API Gateway") } fn auth_metadata(jwt: &str) -> Result> { MetadataValue::try_from(format!("Bearer {}", jwt)) .context("Invalid JWT token format") } /// Spawn all stream tasks. Returns the receiving end. pub fn spawn_all_streams( api_gateway_url: String, jwt_token: String, ) -> mpsc::Receiver { let (tx, rx) = mpsc::channel(256); // Training metrics stream (monitoring.proto) { let tx = tx.clone(); let url = api_gateway_url.clone(); let jwt = jwt_token.clone(); tokio::spawn(async move { stream_training_loop(&url, &jwt, tx).await; }); } // Order updates stream (trading.proto) { let tx = tx.clone(); let url = api_gateway_url.clone(); let jwt = jwt_token.clone(); tokio::spawn(async move { stream_order_updates_loop(&url, &jwt, tx).await; }); } // Risk alerts stream (trading.proto) { let tx = tx.clone(); let url = api_gateway_url.clone(); let jwt = jwt_token.clone(); tokio::spawn(async move { stream_risk_alerts_loop(&url, &jwt, tx).await; }); } // System status stream (trading.proto) { let tx = tx.clone(); let url = api_gateway_url.clone(); let jwt = jwt_token.clone(); tokio::spawn(async move { stream_system_status_loop(&url, &jwt, tx).await; }); } // Initial positions + risk snapshot (unary RPCs, then done) { let tx = tx.clone(); let url = api_gateway_url.clone(); let jwt = jwt_token.clone(); tokio::spawn(async move { if let Err(e) = fetch_initial_state(&url, &jwt, &tx).await { warn!("Failed to fetch initial state: {}", e); } }); } rx } async fn stream_training_loop(url: &str, jwt: &str, tx: mpsc::Sender) { let mut backoff = 1u64; loop { match try_stream_training(url, jwt, &tx).await { Ok(()) => backoff = 1, Err(e) => { warn!("Training stream error: {}", e); let _ = tx.send(StreamEvent::StreamDisconnected { stream_name: "Training".to_string(), }).await; } } sleep(Duration::from_secs(backoff)).await; backoff = (backoff * 2).min(MAX_BACKOFF_SECS); } } async fn try_stream_training( url: &str, jwt: &str, tx: &mpsc::Sender, ) -> Result<()> { let channel = connect_channel(url).await?; let mut client = MonitoringServiceClient::new(channel); let mut request = Request::new(StreamTrainingMetricsRequest { model_filter: String::new(), interval_seconds: 3, }); request.metadata_mut().insert("authorization", auth_metadata(jwt)?); let mut stream = client .stream_training_metrics(request) .await .context("StreamTrainingMetrics failed")? .into_inner(); use tokio_stream::StreamExt; while let Some(result) = stream.next().await { let resp = result.context("Training stream message error")?; let sessions: Vec = resp.sessions.iter().map(|s| TrainingSession { model: s.model.clone(), fold: s.fold.clone(), is_hyperopt: s.is_hyperopt, epoch: s.current_epoch, epoch_loss: s.epoch_loss, validation_loss: s.validation_loss, learning_rate: s.learning_rate, gpu_percent: 0.0, // filled from gpu snapshot batches_per_second: s.batches_per_second, gradient_norm: s.gradient_norm, epoch_duration_seconds: s.epoch_duration_seconds, }).collect(); let gpu = resp.gpu.as_ref().map(|g| GpuInfo { utilization_percent: g.utilization_percent, memory_used_mb: g.memory_used_mb, memory_total_mb: g.memory_total_mb, temperature_celsius: g.temperature_celsius, }).unwrap_or_default(); tx.send(StreamEvent::TrainingUpdate { sessions, gpu }).await .context("Dashboard receiver dropped")?; } Ok(()) } async fn stream_order_updates_loop(url: &str, jwt: &str, tx: mpsc::Sender) { let mut backoff = 1u64; loop { match try_stream_order_updates(url, jwt, &tx).await { Ok(()) => backoff = 1, Err(e) => { warn!("Order updates stream error: {}", e); let _ = tx.send(StreamEvent::StreamDisconnected { stream_name: "Trading".to_string(), }).await; } } sleep(Duration::from_secs(backoff)).await; backoff = (backoff * 2).min(MAX_BACKOFF_SECS); } } async fn try_stream_order_updates( url: &str, jwt: &str, tx: &mpsc::Sender, ) -> Result<()> { let channel = connect_channel(url).await?; let mut client = TradingServiceClient::new(channel); let mut request = Request::new(SubscribeOrderUpdatesRequest { account_id: None }); request.metadata_mut().insert("authorization", auth_metadata(jwt)?); let mut stream = client .subscribe_order_updates(request) .await .context("SubscribeOrderUpdates failed")? .into_inner(); use tokio_stream::StreamExt; while let Some(result) = stream.next().await { let ev = result.context("Order update stream message error")?; tx.send(StreamEvent::OrderUpdate { order_id: ev.order_id, symbol: ev.symbol, status: format!("{:?}", ev.status), filled_qty: ev.filled_quantity, last_fill_price: ev.last_fill_price, timestamp_nanos: ev.timestamp_unix_nanos, }).await.context("Dashboard receiver dropped")?; } Ok(()) } async fn stream_risk_alerts_loop(url: &str, jwt: &str, tx: mpsc::Sender) { let mut backoff = 1u64; loop { match try_stream_risk_alerts(url, jwt, &tx).await { Ok(()) => backoff = 1, Err(e) => { warn!("Risk alerts stream error: {}", e); let _ = tx.send(StreamEvent::StreamDisconnected { stream_name: "Risk".to_string(), }).await; } } sleep(Duration::from_secs(backoff)).await; backoff = (backoff * 2).min(MAX_BACKOFF_SECS); } } async fn try_stream_risk_alerts( url: &str, jwt: &str, tx: &mpsc::Sender, ) -> Result<()> { let channel = connect_channel(url).await?; let mut client = TradingServiceClient::new(channel); let mut request = Request::new(SubscribeRiskAlertsRequest { min_severity: vec![], symbols: vec![], }); request.metadata_mut().insert("authorization", auth_metadata(jwt)?); let mut stream = client .subscribe_risk_alerts(request) .await .context("SubscribeRiskAlerts failed")? .into_inner(); use tokio_stream::StreamExt; while let Some(result) = stream.next().await { let ev = result.context("Risk alert stream message error")?; tx.send(StreamEvent::RiskAlert { severity: format!("{:?}", ev.severity), symbol: ev.symbol, message: ev.message, threshold: ev.threshold_value, current: ev.current_value, }).await.context("Dashboard receiver dropped")?; } Ok(()) } async fn stream_system_status_loop(url: &str, jwt: &str, tx: mpsc::Sender) { let mut backoff = 1u64; loop { match try_stream_system_status(url, jwt, &tx).await { Ok(()) => backoff = 1, Err(e) => { warn!("System status stream error: {}", e); let _ = tx.send(StreamEvent::StreamDisconnected { stream_name: "System".to_string(), }).await; } } sleep(Duration::from_secs(backoff)).await; backoff = (backoff * 2).min(MAX_BACKOFF_SECS); } } async fn try_stream_system_status( url: &str, jwt: &str, tx: &mpsc::Sender, ) -> Result<()> { let channel = connect_channel(url).await?; let mut client = TradingServiceClient::new(channel); let mut request = Request::new(SubscribeSystemStatusRequest { service_names: vec![], }); request.metadata_mut().insert("authorization", auth_metadata(jwt)?); let mut stream = client .subscribe_system_status(request) .await .context("SubscribeSystemStatus failed")? .into_inner(); use tokio_stream::StreamExt; while let Some(result) = stream.next().await { let ev = result.context("System status stream message error")?; tx.send(StreamEvent::SystemStatus { service: ev.service_name, status: format!("{:?}", ev.status), message: ev.message, }).await.context("Dashboard receiver dropped")?; } Ok(()) } async fn fetch_initial_state( url: &str, jwt: &str, tx: &mpsc::Sender, ) -> Result<()> { let channel = connect_channel(url).await?; let mut client = TradingServiceClient::new(channel); let auth = auth_metadata(jwt)?; // Fetch positions let mut pos_req = Request::new(GetPositionsRequest { symbol: None }); pos_req.metadata_mut().insert("authorization", auth.clone()); if let Ok(resp) = client.get_positions(pos_req).await { let positions = resp.into_inner().positions.iter().map(|p| PositionRow { symbol: p.symbol.clone(), side: if p.quantity >= 0.0 { "LONG" } else { "SHORT" }.to_string(), quantity: p.quantity.abs(), entry_price: p.average_cost, unrealized_pnl: p.unrealized_pnl, status: "OPEN".to_string(), }).collect(); let _ = tx.send(StreamEvent::PositionsSnapshot { positions }).await; } // Fetch risk metrics let mut risk_req = Request::new(GetRiskMetricsRequest { portfolio_id: None, start_time_unix_nanos: None, end_time_unix_nanos: None, }); risk_req.metadata_mut().insert("authorization", auth); if let Ok(resp) = client.get_risk_metrics(risk_req).await { let r = resp.into_inner(); let _ = tx.send(StreamEvent::RiskMetrics { var: r.value_at_risk, max_drawdown: r.max_drawdown, current_drawdown: r.current_drawdown, sharpe: r.sharpe_ratio, }).await; } Ok(()) } ``` **Step 2: Verify compilation** Run: `SQLX_OFFLINE=true cargo check -p fxt` Expected: compiles (streams.rs may need import path adjustments — the proto module paths depend on the `include_proto!` setup in lib.rs) **Step 3: Commit** ```bash git add bin/fxt/src/commands/watch/streams.rs git commit -m "feat(fxt): add gRPC stream manager with reconnect" ``` **NOTE to implementer:** The exact proto module paths (`crate::proto::monitoring`, `crate::proto::foxhunt_tli`) must match the `include_proto!` setup in `bin/fxt/src/lib.rs`. Check `lib.rs` for the actual module paths and adjust imports accordingly. The package names are `monitoring` and `foxhunt.tli` (which becomes `foxhunt_tli` in Rust). --- ### Task 5: Renderer **Files:** - Create: `bin/fxt/src/commands/watch/render.rs` **Step 1: Write renderer** Create `bin/fxt/src/commands/watch/render.rs`: ```rust use ratatui::{ Frame, layout::{Constraint, Direction, Layout, Rect}, style::{Color, Modifier, Style}, text::{Line, Span}, widgets::{Block, Borders, Cell, Paragraph, Row, Sparkline, Table, Tabs}, }; use super::state::*; pub fn render(frame: &mut Frame, state: &DashboardState) { let chunks = Layout::default() .direction(Direction::Vertical) .constraints([ Constraint::Length(3), // tab bar Constraint::Min(0), // tab content Constraint::Length(1), // status bar ]) .split(frame.area()); render_tab_bar(frame, chunks[0], state); match state.current_tab() { Tab::Training => render_training_tab(frame, chunks[1], &state.training), Tab::Trading => render_trading_tab(frame, chunks[1], &state.trading), Tab::Risk => render_risk_tab(frame, chunks[1], &state.risk), Tab::System => render_system_tab(frame, chunks[1], &state.system), } render_status_bar(frame, chunks[2]); } fn render_tab_bar(frame: &mut Frame, area: Rect, state: &DashboardState) { let titles: Vec = Tab::ALL.iter().map(|t| { Line::from(Span::styled(t.title(), Style::default().fg(Color::White))) }).collect(); let tabs = Tabs::new(titles) .block(Block::default().borders(Borders::ALL).title(" fxt watch ")) .select(state.active_tab) .style(Style::default().fg(Color::DarkGray)) .highlight_style(Style::default().fg(Color::Cyan).add_modifier(Modifier::BOLD)); frame.render_widget(tabs, area); } fn render_status_bar(frame: &mut Frame, area: Rect) { let bar = Paragraph::new(Line::from(vec![ Span::styled(" 1-4", Style::default().fg(Color::Cyan)), Span::raw(" tabs "), Span::styled("j/k", Style::default().fg(Color::Cyan)), Span::raw(" scroll "), Span::styled("q", Style::default().fg(Color::Cyan)), Span::raw(" quit"), ])); frame.render_widget(bar, area); } fn render_training_tab(frame: &mut Frame, area: Rect, tab: &TrainingTabState) { let chunks = Layout::default() .direction(Direction::Vertical) .constraints([Constraint::Min(8), Constraint::Length(6)]) .split(area); // Sessions table let header = Row::new(vec!["Model", "Fold", "Epoch", "Loss", "Val Loss", "LR", "Batch/s", "Grad"]) .style(Style::default().fg(Color::Yellow).add_modifier(Modifier::BOLD)); let rows: Vec = tab.sessions.iter().skip(tab.scroll_offset).map(|s| { Row::new(vec![ Cell::from(s.model.clone()), Cell::from(s.fold.clone()), Cell::from(format!("{:.0}", s.epoch)), Cell::from(format!("{:.4}", s.epoch_loss)), Cell::from(format!("{:.4}", s.validation_loss)), Cell::from(format!("{:.1e}", s.learning_rate)), Cell::from(format!("{:.1}", s.batches_per_second)), Cell::from(format!("{:.2}", s.gradient_norm)), ]) }).collect(); let connected_indicator = if tab.connected { " [connected]" } else { " [disconnected]" }; let table = Table::new(rows, [ Constraint::Length(10), Constraint::Length(8), Constraint::Length(8), Constraint::Length(10), Constraint::Length(10), Constraint::Length(10), Constraint::Length(8), Constraint::Length(8), ]) .header(header) .block(Block::default().borders(Borders::ALL).title(format!(" Training{} ", connected_indicator))); frame.render_widget(table, chunks[0]); // GPU + loss sparkline let bottom = Layout::default() .direction(Direction::Horizontal) .constraints([Constraint::Percentage(50), Constraint::Percentage(50)]) .split(chunks[1]); let gpu_text = format!( "GPU: {:.0}% VRAM: {:.0}/{:.0} MB Temp: {:.0}C", tab.gpu.utilization_percent, tab.gpu.memory_used_mb, tab.gpu.memory_total_mb, tab.gpu.temperature_celsius, ); let gpu_para = Paragraph::new(gpu_text) .block(Block::default().borders(Borders::ALL).title(" GPU ")); frame.render_widget(gpu_para, bottom[0]); let loss_data: Vec = tab.loss_history.iter().map(|v| { (v.clamp(0.0, 10.0) * 100.0) as u64 }).collect(); let sparkline = Sparkline::default() .block(Block::default().borders(Borders::ALL).title(" Loss History ")) .data(&loss_data) .style(Style::default().fg(Color::Green)); frame.render_widget(sparkline, bottom[1]); } fn render_trading_tab(frame: &mut Frame, area: Rect, tab: &TradingTabState) { let chunks = Layout::default() .direction(Direction::Vertical) .constraints([Constraint::Percentage(40), Constraint::Percentage(40), Constraint::Length(3)]) .split(area); // Positions table let header = Row::new(vec!["Symbol", "Side", "Qty", "Entry", "PnL", "Status"]) .style(Style::default().fg(Color::Yellow).add_modifier(Modifier::BOLD)); let rows: Vec = tab.positions.iter().map(|p| { let pnl_color = if p.unrealized_pnl >= 0.0 { Color::Green } else { Color::Red }; Row::new(vec![ Cell::from(p.symbol.clone()), Cell::from(p.side.clone()), Cell::from(format!("{:.0}", p.quantity)), Cell::from(format!("{:.2}", p.entry_price)), Cell::from(Span::styled(format!("{:+.2}", p.unrealized_pnl), Style::default().fg(pnl_color))), Cell::from(p.status.clone()), ]) }).collect(); let connected_indicator = if tab.connected { " [connected]" } else { " [disconnected]" }; let table = Table::new(rows, [ Constraint::Length(10), Constraint::Length(6), Constraint::Length(6), Constraint::Length(10), Constraint::Length(12), Constraint::Length(8), ]) .header(header) .block(Block::default().borders(Borders::ALL).title(format!(" Positions{} ", connected_indicator))); frame.render_widget(table, chunks[0]); // Recent fills let fill_header = Row::new(vec!["Time", "Side", "Symbol", "Qty", "Price", "Status"]) .style(Style::default().fg(Color::Yellow).add_modifier(Modifier::BOLD)); let fill_rows: Vec = tab.fills.iter().skip(tab.scroll_offset).take(10).map(|f| { Row::new(vec![ Cell::from(f.time.clone()), Cell::from(f.side.clone()), Cell::from(f.symbol.clone()), Cell::from(format!("{:.0}", f.quantity)), Cell::from(format!("{:.2}", f.price)), Cell::from(f.status.clone()), ]) }).collect(); let fills_table = Table::new(fill_rows, [ Constraint::Length(10), Constraint::Length(6), Constraint::Length(10), Constraint::Length(6), Constraint::Length(10), Constraint::Length(8), ]) .header(fill_header) .block(Block::default().borders(Borders::ALL).title(" Recent Fills ")); frame.render_widget(fills_table, chunks[1]); // PnL summary let pnl_color = if tab.day_pnl >= 0.0 { Color::Green } else { Color::Red }; let pnl_text = format!( "Day PnL: {:+.2} | Realized: {:+.2} Unrealized: {:+.2}", tab.day_pnl, tab.realized_pnl, tab.unrealized_pnl, ); let pnl_bar = Paragraph::new(Span::styled(pnl_text, Style::default().fg(pnl_color))) .block(Block::default().borders(Borders::ALL)); frame.render_widget(pnl_bar, chunks[2]); } fn render_risk_tab(frame: &mut Frame, area: Rect, tab: &RiskTabState) { let chunks = Layout::default() .direction(Direction::Vertical) .constraints([Constraint::Length(5), Constraint::Min(0)]) .split(area); // Risk metrics summary let connected_indicator = if tab.connected { " [connected]" } else { " [disconnected]" }; let summary = Paragraph::new(vec![ Line::from(format!("Portfolio VaR (95%): ${:.0} Max DD: {:.1}%", tab.portfolio_var, tab.max_drawdown * 100.0)), Line::from(format!("Current DD: {:.1}% Sharpe: {:.2}", tab.current_drawdown * 100.0, tab.sharpe_ratio)), ]) .block(Block::default().borders(Borders::ALL).title(format!(" Risk Metrics{} ", connected_indicator))); frame.render_widget(summary, chunks[0]); // Circuit breakers let header = Row::new(vec!["Breaker", "Status", "Current", "Limit"]) .style(Style::default().fg(Color::Yellow).add_modifier(Modifier::BOLD)); let rows: Vec = tab.circuit_breakers.iter().map(|cb| { let status_color = if cb.status == "OK" { Color::Green } else { Color::Red }; Row::new(vec![ Cell::from(cb.name.clone()), Cell::from(Span::styled(&cb.status, Style::default().fg(status_color))), Cell::from(cb.current.clone()), Cell::from(cb.limit.clone()), ]) }).collect(); let table = Table::new(rows, [ Constraint::Length(18), Constraint::Length(10), Constraint::Length(15), Constraint::Length(15), ]) .header(header) .block(Block::default().borders(Borders::ALL).title(" Circuit Breakers ")); frame.render_widget(table, chunks[1]); } fn render_system_tab(frame: &mut Frame, area: Rect, tab: &SystemTabState) { let chunks = Layout::default() .direction(Direction::Vertical) .constraints([Constraint::Min(8), Constraint::Length(5)]) .split(area); // Services table let header = Row::new(vec!["Service", "Status", "Latency", "Message"]) .style(Style::default().fg(Color::Yellow).add_modifier(Modifier::BOLD)); let rows: Vec = tab.services.iter().map(|s| { let status_color = match s.status.as_str() { "HEALTHY" | "UP" => Color::Green, "DEGRADED" => Color::Yellow, _ => Color::Red, }; Row::new(vec![ Cell::from(s.name.clone()), Cell::from(Span::styled(&s.status, Style::default().fg(status_color))), Cell::from(format!("{:.0}ms", s.latency_ms)), Cell::from(s.message.clone()), ]) }).collect(); let connected_indicator = if tab.connected { " [connected]" } else { " [disconnected]" }; let table = Table::new(rows, [ Constraint::Length(22), Constraint::Length(12), Constraint::Length(10), Constraint::Min(20), ]) .header(header) .block(Block::default().borders(Borders::ALL).title(format!(" Services{} ", connected_indicator))); frame.render_widget(table, chunks[0]); // GPU/CPU summary let sys_text = format!( "GPU: {:.0}% VRAM: {:.0}/{:.0} MB | CPU: {:.0}% RAM: {:.1}/{:.1} GB", tab.gpu.utilization_percent, tab.gpu.memory_used_mb, tab.gpu.memory_total_mb, tab.cpu_percent, tab.ram_used_gb, tab.ram_total_gb, ); let sys_para = Paragraph::new(sys_text) .block(Block::default().borders(Borders::ALL).title(" Resources ")); frame.render_widget(sys_para, chunks[1]); } ``` **Step 2: Verify compilation** Run: `SQLX_OFFLINE=true cargo check -p fxt` Expected: compiles **Step 3: Commit** ```bash git add bin/fxt/src/commands/watch/render.rs git commit -m "feat(fxt): add ratatui dashboard renderer for 4 tabs" ``` --- ### Task 6: Event Loop **Files:** - Create: `bin/fxt/src/commands/watch/event_loop.rs` **Step 1: Write event loop** Create `bin/fxt/src/commands/watch/event_loop.rs`: ```rust use anyhow::Result; use crossterm::{ event::{self, Event, KeyCode, KeyEvent, KeyModifiers, EventStream}, execute, terminal::{disable_raw_mode, enable_raw_mode, EnterAlternateScreen, LeaveAlternateScreen}, }; use ratatui::prelude::*; use std::io; use tokio::time::{interval, Duration}; use super::render; use super::state::DashboardState; use super::streams::{self, StreamEvent}; pub async fn run_dashboard(api_gateway_url: &str, jwt_token: &str) -> Result<()> { // Setup terminal enable_raw_mode()?; let mut stdout = io::stdout(); execute!(stdout, EnterAlternateScreen)?; let backend = CrosstermBackend::new(stdout); let mut terminal = Terminal::new(backend)?; let result = run_event_loop(&mut terminal, api_gateway_url, jwt_token).await; // Restore terminal disable_raw_mode()?; execute!(terminal.backend_mut(), LeaveAlternateScreen)?; terminal.show_cursor()?; result } async fn run_event_loop( terminal: &mut Terminal>, api_gateway_url: &str, jwt_token: &str, ) -> Result<()> { let mut state = DashboardState::default(); state.dirty = true; // initial render let mut stream_rx = streams::spawn_all_streams( api_gateway_url.to_string(), jwt_token.to_string(), ); let mut tick = interval(Duration::from_millis(200)); let mut event_stream = EventStream::new(); use tokio_stream::StreamExt; loop { tokio::select! { // gRPC stream events Some(event) = stream_rx.recv() => { apply_stream_event(&mut state, event); } // Terminal key events Some(Ok(event)) = event_stream.next() => { if let Event::Key(key) = event { if handle_key(&mut state, key) { break; // quit } } } // Render tick _ = tick.tick() => { if state.dirty { terminal.draw(|frame| render::render(frame, &state))?; state.dirty = false; } } } } Ok(()) } fn apply_stream_event(state: &mut DashboardState, event: StreamEvent) { match event { StreamEvent::TrainingUpdate { sessions, gpu } => { // Push first session's loss to history for sparkline if let Some(first) = sessions.first() { state.training.push_loss(first.epoch_loss as f64); } state.training.sessions = sessions; state.training.gpu = gpu.clone(); state.system.gpu = gpu; state.training.connected = true; } StreamEvent::OrderUpdate { order_id, symbol, status, filled_qty, last_fill_price, timestamp_nanos } => { let time = format_nanos(timestamp_nanos); let side = if filled_qty >= 0.0 { "BUY" } else { "SELL" }; state.trading.push_fill(super::state::FillRow { time, side: side.to_string(), symbol: symbol.clone(), quantity: filled_qty.abs(), price: last_fill_price, status, }); state.trading.connected = true; } StreamEvent::PositionsSnapshot { positions } => { state.trading.positions = positions; state.trading.unrealized_pnl = state.trading.positions.iter().map(|p| p.unrealized_pnl).sum(); state.trading.connected = true; } StreamEvent::RiskMetrics { var, max_drawdown, current_drawdown, sharpe } => { state.risk.portfolio_var = var; state.risk.max_drawdown = max_drawdown; state.risk.current_drawdown = current_drawdown; state.risk.sharpe_ratio = sharpe; state.risk.connected = true; } StreamEvent::RiskAlert { severity: _, symbol: _, message, threshold, current } => { // Update circuit breaker display from alerts let status = if current < threshold { "OK" } else { "TRIPPED" }; let found = state.risk.circuit_breakers.iter_mut().find(|cb| cb.name == message); if let Some(cb) = found { cb.status = status.to_string(); cb.current = format!("{:.2}", current); } else { state.risk.circuit_breakers.push(super::state::CircuitBreakerRow { name: message, status: status.to_string(), current: format!("{:.2}", current), limit: format!("{:.2}", threshold), }); } state.risk.connected = true; } StreamEvent::SystemStatus { service, status, message } => { let found = state.system.services.iter_mut().find(|s| s.name == service); if let Some(s) = found { s.status = status; s.message = message; } else { state.system.services.push(super::state::ServiceRow { name: service, status, latency_ms: 0.0, message, }); } state.system.connected = true; } StreamEvent::StreamDisconnected { stream_name } => { match stream_name.as_str() { "Training" => state.training.connected = false, "Trading" => state.trading.connected = false, "Risk" => state.risk.connected = false, "System" => state.system.connected = false, _ => {} } } } state.dirty = true; } fn handle_key(state: &mut DashboardState, key: KeyEvent) -> bool { match key.code { KeyCode::Char('q') => return true, KeyCode::Char('c') if key.modifiers.contains(KeyModifiers::CONTROL) => return true, KeyCode::Char('1') => state.select_tab(0), KeyCode::Char('2') => state.select_tab(1), KeyCode::Char('3') => state.select_tab(2), KeyCode::Char('4') => state.select_tab(3), KeyCode::Char('j') | KeyCode::Down => state.scroll_down(), KeyCode::Char('k') | KeyCode::Up => state.scroll_up(), _ => {} } false } fn format_nanos(nanos: i64) -> String { let secs = nanos / 1_000_000_000; let hour = (secs / 3600) % 24; let min = (secs / 60) % 60; let sec = secs % 60; format!("{:02}:{:02}:{:02}", hour, min, sec) } #[cfg(test)] #[allow(clippy::unwrap_used, clippy::expect_used)] mod tests { use super::*; use crossterm::event::KeyEventKind; fn key(code: KeyCode) -> KeyEvent { KeyEvent::new(code, KeyModifiers::NONE) } #[test] fn test_handle_key_quit() { let mut state = DashboardState::default(); assert!(handle_key(&mut state, key(KeyCode::Char('q')))); } #[test] fn test_handle_key_tab_switch() { let mut state = DashboardState::default(); assert!(!handle_key(&mut state, key(KeyCode::Char('3')))); assert_eq!(state.active_tab, 2); } #[test] fn test_handle_key_scroll() { let mut state = DashboardState::default(); handle_key(&mut state, key(KeyCode::Char('j'))); assert_eq!(state.training.scroll_offset, 1); handle_key(&mut state, key(KeyCode::Char('k'))); assert_eq!(state.training.scroll_offset, 0); } #[test] fn test_format_nanos() { assert_eq!(format_nanos(52321_000_000_000), "14:32:01"); } #[test] fn test_apply_training_update() { let mut state = DashboardState::default(); apply_stream_event(&mut state, StreamEvent::TrainingUpdate { sessions: vec![super::super::state::TrainingSession { model: "DQN".to_string(), fold: "1/5".to_string(), is_hyperopt: false, epoch: 10.0, epoch_loss: 0.05, validation_loss: 0.06, learning_rate: 1e-4, gpu_percent: 80.0, batches_per_second: 120.0, gradient_norm: 0.5, epoch_duration_seconds: 2.0, }], gpu: super::super::state::GpuInfo { utilization_percent: 80.0, memory_used_mb: 30000.0, memory_total_mb: 48000.0, temperature_celsius: 65.0, }, }); assert_eq!(state.training.sessions.len(), 1); assert!(state.training.connected); assert!(state.dirty); } #[test] fn test_apply_system_status_upsert() { let mut state = DashboardState::default(); apply_stream_event(&mut state, StreamEvent::SystemStatus { service: "api-gateway".to_string(), status: "HEALTHY".to_string(), message: String::new(), }); assert_eq!(state.system.services.len(), 1); // Second update replaces, not appends apply_stream_event(&mut state, StreamEvent::SystemStatus { service: "api-gateway".to_string(), status: "DEGRADED".to_string(), message: "high latency".to_string(), }); assert_eq!(state.system.services.len(), 1); assert_eq!(state.system.services[0].status, "DEGRADED"); } #[test] fn test_apply_stream_disconnected() { let mut state = DashboardState::default(); state.training.connected = true; apply_stream_event(&mut state, StreamEvent::StreamDisconnected { stream_name: "Training".to_string(), }); assert!(!state.training.connected); } } ``` **Step 2: Run tests** Run: `SQLX_OFFLINE=true cargo test -p fxt --lib watch::event_loop` Expected: 7 tests pass **Step 3: Commit** ```bash git add bin/fxt/src/commands/watch/event_loop.rs git commit -m "feat(fxt): add dashboard event loop with key handling" ``` --- ### Task 7: Wire Watch Command into CLI **Files:** - Modify: `bin/fxt/src/commands/mod.rs` - Modify: `bin/fxt/src/main.rs` **Step 1: Add watch module to commands/mod.rs** Add to `bin/fxt/src/commands/mod.rs`: ```rust pub mod watch; ``` **Step 2: Add Watch variant to Commands enum in main.rs** In `bin/fxt/src/main.rs`, add a new variant to the `Commands` enum: ```rust /// Live streaming dashboard Watch, ``` **Step 3: Add dispatch in the match block** In the `match cli.command { ... }` block in `main.rs`, add: ```rust Commands::Watch => commands::watch::run(&cli.api_gateway_url, &jwt_token).await, ``` **Step 4: Verify compilation** Run: `SQLX_OFFLINE=true cargo check -p fxt` Expected: compiles **Step 5: Commit** ```bash git add bin/fxt/src/commands/mod.rs bin/fxt/src/main.rs git commit -m "feat(fxt): wire fxt watch command into CLI" ``` --- ### Task 8: Wire Model List **Files:** - Modify: `bin/fxt/src/commands/model/list.rs` **Step 1: Replace stub with real gRPC call** Replace the entire contents of `bin/fxt/src/commands/model/list.rs` with a real implementation that calls `MLTrainingService::ListAvailableModels`. Follow the same pattern as `train/list.rs`: 1. `Channel::from_shared(url).connect().await` 2. `MlTrainingServiceClient::new(channel)` 3. `MetadataValue::try_from(format!("Bearer {}", jwt))` for auth 4. Call `list_available_models(request)` 5. Display results in a table using println Use imports from `crate::proto::ml_training::{ ml_training_service_client::MlTrainingServiceClient, ListAvailableModelsRequest }`. **Step 2: Verify compilation** Run: `SQLX_OFFLINE=true cargo check -p fxt` **Step 3: Run existing tests** Run: `SQLX_OFFLINE=true cargo test -p fxt --lib` Expected: all 134+ tests pass **Step 4: Commit** ```bash git add bin/fxt/src/commands/model/list.rs git commit -m "feat(fxt): wire model list to MLTrainingService::ListAvailableModels" ``` --- ### Task 9: Wire Model Approve **Files:** - Modify: `bin/fxt/src/commands/model/approve.rs` **Step 1: Replace stub with real gRPC call** Same pattern as Task 8 but calls the new `ApproveModel` RPC (added in Task 2). Import `ApproveModelRequest` from `crate::proto::ml_training`. Display success/failure message from the response. **Step 2: Verify + commit** ```bash git add bin/fxt/src/commands/model/approve.rs git commit -m "feat(fxt): wire model approve to MLTrainingService::ApproveModel" ``` --- ### Task 10: Wire Model Reject **Files:** - Modify: `bin/fxt/src/commands/model/reject.rs` **Step 1: Replace stub with real gRPC call** Same pattern, calls `RejectModel` RPC. Import `RejectModelRequest`. **Step 2: Verify + commit** ```bash git add bin/fxt/src/commands/model/reject.rs git commit -m "feat(fxt): wire model reject to MLTrainingService::RejectModel" ``` --- ### Task 11: Wire Agent Allocate-Portfolio (Replace Mock Assets) **Files:** - Modify: `bin/fxt/src/commands/agent.rs` **Step 1: Replace mock assets block** In `agent.rs`, find the mock assets block (around lines 192-212, marked with "Mock assets for testing"). Replace it with a real call to `TradingAgentService::GetSelectedAssets(GetSelectedAssetsRequest { universe_id: selection_id })`. Map the returned `AssetScore` messages to the existing `AssetScore` type used by `AllocatePortfolioRequest`. If the `selection_id` arg is not provided, use a default universe. **Step 2: Verify + commit** ```bash git add bin/fxt/src/commands/agent.rs git commit -m "feat(fxt): wire agent allocate-portfolio to GetSelectedAssets RPC" ``` --- ### Task 12: Wire Auth Login Non-Interactive **Files:** - Modify: `bin/fxt/src/commands/auth.rs` **Step 1: Replace simulated token** In the `execute_login` function, find the block around lines 113-127 that creates a "simulated token". Replace it with a real gRPC call using the `LoginClient` that is already constructed (line ~108-110). The interactive path already uses `login_client.interactive_login()`. For non-interactive, construct the appropriate request with username/password and call the login RPC directly. On success, store the returned token via `auth_manager.store_tokens(...)`. **Step 2: Verify + commit** ```bash git add bin/fxt/src/commands/auth.rs git commit -m "feat(fxt): wire non-interactive auth login to real gRPC" ``` --- ### Task 13: Uncomment tune_stream **Files:** - Modify: `bin/fxt/src/commands/mod.rs` **Step 1: Uncomment the module** In `bin/fxt/src/commands/mod.rs`, change: ```rust // TODO: Enable tune_stream when API Gateway implements streaming support // pub mod tune_stream; ``` to: ```rust pub mod tune_stream; ``` **Step 2: Wire into tune command if not already connected** Check `bin/fxt/src/commands/tune.rs` (or `tune/mod.rs`) to see if the `--watch` flag already dispatches to `tune_stream::watch_tuning_progress_streaming`. If not, add the dispatch. **Step 3: Verify compilation** Run: `SQLX_OFFLINE=true cargo check -p fxt` **Step 4: Commit** ```bash git add bin/fxt/src/commands/mod.rs bin/fxt/src/commands/tune.rs git commit -m "feat(fxt): enable tune_stream module for fxt tune --watch" ``` --- ### Task 14: Clippy + Full Test Suite **Files:** (none — verification only) **Step 1: Run clippy** Run: `SQLX_OFFLINE=true cargo clippy -p fxt -- -D warnings` Expected: 0 errors, 0 warnings. Fix any issues. **Step 2: Run all fxt tests** Run: `SQLX_OFFLINE=true cargo test -p fxt --lib` Expected: all tests pass (134 existing + new dashboard tests) **Step 3: Run workspace check** Run: `SQLX_OFFLINE=true cargo check --workspace` Expected: compiles **Step 4: Commit any fixes** ```bash git commit -am "fix(fxt): clippy and test fixes" ``` --- ### Task 15: Final Server-Side Handlers (ml_training_service) **Files:** - Modify: `services/ml_training_service/src/service.rs` **Step 1: Add ApproveModel handler** In `service.rs`, add handler for `approve_model` in the `MlTrainingService` impl. Pattern: accept `ApproveModelRequest`, log the approval, return `ApproveModelResponse { success: true, message }`. For now, the handler can update an in-memory map or return success — the actual checkpoint promotion logic can be wired later when the promotion_manager is mature. The important thing is the RPC exists and responds. **Step 2: Add RejectModel handler** Same pattern for `reject_model`. **Step 3: Verify compilation** Run: `SQLX_OFFLINE=true cargo check -p ml_training_service` **Step 4: Commit** ```bash git add services/ml_training_service/src/service.rs git commit -m "feat(ml_training_service): add ApproveModel and RejectModel handlers" ```