Bite-sized TDD tasks: ratatui deps, proto RPCs, state types, stream manager, renderer, event loop, CLI wiring, 6 stub fixes, clippy. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1693 lines
51 KiB
Markdown
1693 lines
51 KiB
Markdown
# FXT Dashboard & CLI Completion Implementation Plan
|
|
|
|
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
|
|
|
**Goal:** Finalize `fxt` as a fully functional CLI with a live ratatui streaming dashboard (`fxt watch`) and all stub commands wired to real gRPC backends.
|
|
|
|
**Architecture:** New `watch` subcommand with ratatui TUI, 4 tabs (Training/Trading/Risk/System) fed by gRPC server-side streams via `tokio::select!` event loop. Stub commands wired to existing RPCs. Two new proto RPCs (ApproveModel/RejectModel) added to ml_training.proto.
|
|
|
|
**Tech Stack:** Rust, ratatui 0.29, crossterm 0.28, tonic gRPC, tokio
|
|
|
|
---
|
|
|
|
### Task 1: Add Dependencies
|
|
|
|
**Files:**
|
|
- Modify: `bin/fxt/Cargo.toml`
|
|
|
|
**Step 1: Add ratatui and crossterm to dependencies**
|
|
|
|
In `bin/fxt/Cargo.toml`, add after the existing `[dependencies]` entries:
|
|
|
|
```toml
|
|
ratatui = "0.29"
|
|
crossterm = { version = "0.28", features = ["event-stream"] }
|
|
```
|
|
|
|
**Step 2: Verify compilation**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo check -p fxt`
|
|
Expected: compiles with 0 errors
|
|
|
|
**Step 3: Commit**
|
|
|
|
```bash
|
|
git add bin/fxt/Cargo.toml
|
|
git commit -m "feat(fxt): add ratatui and crossterm dependencies"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 2: Add ApproveModel / RejectModel Proto RPCs
|
|
|
|
**Files:**
|
|
- Modify: `bin/fxt/proto/ml_training.proto`
|
|
|
|
**Step 1: Add messages and RPCs**
|
|
|
|
In `bin/fxt/proto/ml_training.proto`, add two RPCs to the `MLTrainingService` service block (after the existing `StreamTuningProgress` RPC at line ~47):
|
|
|
|
```protobuf
|
|
// Model promotion management
|
|
rpc ApproveModel(ApproveModelRequest) returns (ApproveModelResponse);
|
|
rpc RejectModel(RejectModelRequest) returns (RejectModelResponse);
|
|
```
|
|
|
|
Add messages at the end of the file:
|
|
|
|
```protobuf
|
|
// Model promotion messages
|
|
message ApproveModelRequest {
|
|
string model_id = 1;
|
|
string promoted_to = 2; // e.g. "production", "staging", "canary"
|
|
}
|
|
|
|
message ApproveModelResponse {
|
|
bool success = 1;
|
|
string message = 2;
|
|
}
|
|
|
|
message RejectModelRequest {
|
|
string model_id = 1;
|
|
string reason = 2;
|
|
}
|
|
|
|
message RejectModelResponse {
|
|
bool success = 1;
|
|
string message = 2;
|
|
}
|
|
```
|
|
|
|
**Step 2: Verify proto compiles**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo check -p fxt`
|
|
Expected: compiles (tonic-prost-build picks up new messages automatically)
|
|
|
|
**Step 3: Commit**
|
|
|
|
```bash
|
|
git add bin/fxt/proto/ml_training.proto
|
|
git commit -m "feat(fxt): add ApproveModel and RejectModel proto RPCs"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 3: Dashboard State Types
|
|
|
|
**Files:**
|
|
- Create: `bin/fxt/src/commands/watch/state.rs`
|
|
- Create: `bin/fxt/src/commands/watch/mod.rs`
|
|
|
|
**Step 1: Create watch module directory**
|
|
|
|
```bash
|
|
mkdir -p bin/fxt/src/commands/watch
|
|
```
|
|
|
|
**Step 2: Write state types**
|
|
|
|
Create `bin/fxt/src/commands/watch/state.rs`:
|
|
|
|
```rust
|
|
use std::collections::VecDeque;
|
|
|
|
const CHART_HISTORY: usize = 200;
|
|
|
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
|
pub enum Tab {
|
|
Training,
|
|
Trading,
|
|
Risk,
|
|
System,
|
|
}
|
|
|
|
impl Tab {
|
|
pub const ALL: [Tab; 4] = [Tab::Training, Tab::Trading, Tab::Risk, Tab::System];
|
|
|
|
pub fn title(&self) -> &'static str {
|
|
match self {
|
|
Tab::Training => "Training",
|
|
Tab::Trading => "Trading",
|
|
Tab::Risk => "Risk",
|
|
Tab::System => "System",
|
|
}
|
|
}
|
|
|
|
pub fn index(&self) -> usize {
|
|
match self {
|
|
Tab::Training => 0,
|
|
Tab::Trading => 1,
|
|
Tab::Risk => 2,
|
|
Tab::System => 3,
|
|
}
|
|
}
|
|
}
|
|
|
|
#[derive(Debug, Default)]
|
|
pub struct DashboardState {
|
|
pub active_tab: usize,
|
|
pub training: TrainingTabState,
|
|
pub trading: TradingTabState,
|
|
pub risk: RiskTabState,
|
|
pub system: SystemTabState,
|
|
pub dirty: bool,
|
|
}
|
|
|
|
impl DashboardState {
|
|
pub fn current_tab(&self) -> Tab {
|
|
Tab::ALL.get(self.active_tab).copied().unwrap_or(Tab::Training)
|
|
}
|
|
|
|
pub fn select_tab(&mut self, index: usize) {
|
|
if index < Tab::ALL.len() {
|
|
self.active_tab = index;
|
|
self.dirty = true;
|
|
}
|
|
}
|
|
|
|
pub fn scroll_up(&mut self) {
|
|
match self.current_tab() {
|
|
Tab::Training => self.training.scroll_up(),
|
|
Tab::Trading => self.trading.scroll_up(),
|
|
Tab::Risk => self.risk.scroll_up(),
|
|
Tab::System => self.system.scroll_up(),
|
|
}
|
|
self.dirty = true;
|
|
}
|
|
|
|
pub fn scroll_down(&mut self) {
|
|
match self.current_tab() {
|
|
Tab::Training => self.training.scroll_down(),
|
|
Tab::Trading => self.trading.scroll_down(),
|
|
Tab::Risk => self.risk.scroll_down(),
|
|
Tab::System => self.system.scroll_down(),
|
|
}
|
|
self.dirty = true;
|
|
}
|
|
}
|
|
|
|
// ── Training Tab ──
|
|
|
|
#[derive(Debug, Clone)]
|
|
pub struct TrainingSession {
|
|
pub model: String,
|
|
pub fold: String,
|
|
pub is_hyperopt: bool,
|
|
pub epoch: f32,
|
|
pub epoch_loss: f32,
|
|
pub validation_loss: f32,
|
|
pub learning_rate: f32,
|
|
pub gpu_percent: f32,
|
|
pub batches_per_second: f32,
|
|
pub gradient_norm: f32,
|
|
pub epoch_duration_seconds: f32,
|
|
}
|
|
|
|
#[derive(Debug, Default)]
|
|
pub struct GpuInfo {
|
|
pub utilization_percent: f32,
|
|
pub memory_used_mb: f32,
|
|
pub memory_total_mb: f32,
|
|
pub temperature_celsius: f32,
|
|
}
|
|
|
|
#[derive(Debug, Default)]
|
|
pub struct TrainingTabState {
|
|
pub sessions: Vec<TrainingSession>,
|
|
pub gpu: GpuInfo,
|
|
pub loss_history: VecDeque<f64>,
|
|
pub scroll_offset: usize,
|
|
pub connected: bool,
|
|
}
|
|
|
|
impl TrainingTabState {
|
|
pub fn push_loss(&mut self, loss: f64) {
|
|
self.loss_history.push_back(loss);
|
|
if self.loss_history.len() > CHART_HISTORY {
|
|
self.loss_history.pop_front();
|
|
}
|
|
}
|
|
|
|
pub fn scroll_up(&mut self) {
|
|
self.scroll_offset = self.scroll_offset.saturating_sub(1);
|
|
}
|
|
|
|
pub fn scroll_down(&mut self) {
|
|
self.scroll_offset = self.scroll_offset.saturating_add(1);
|
|
}
|
|
}
|
|
|
|
// ── Trading Tab ──
|
|
|
|
#[derive(Debug, Clone)]
|
|
pub struct PositionRow {
|
|
pub symbol: String,
|
|
pub side: String,
|
|
pub quantity: f64,
|
|
pub entry_price: f64,
|
|
pub unrealized_pnl: f64,
|
|
pub status: String,
|
|
}
|
|
|
|
#[derive(Debug, Clone)]
|
|
pub struct FillRow {
|
|
pub time: String,
|
|
pub side: String,
|
|
pub symbol: String,
|
|
pub quantity: f64,
|
|
pub price: f64,
|
|
pub status: String,
|
|
}
|
|
|
|
#[derive(Debug, Default)]
|
|
pub struct TradingTabState {
|
|
pub positions: Vec<PositionRow>,
|
|
pub fills: VecDeque<FillRow>,
|
|
pub day_pnl: f64,
|
|
pub realized_pnl: f64,
|
|
pub unrealized_pnl: f64,
|
|
pub scroll_offset: usize,
|
|
pub connected: bool,
|
|
}
|
|
|
|
impl TradingTabState {
|
|
pub fn push_fill(&mut self, fill: FillRow) {
|
|
self.fills.push_front(fill);
|
|
if self.fills.len() > 50 {
|
|
self.fills.pop_back();
|
|
}
|
|
}
|
|
|
|
pub fn scroll_up(&mut self) {
|
|
self.scroll_offset = self.scroll_offset.saturating_sub(1);
|
|
}
|
|
|
|
pub fn scroll_down(&mut self) {
|
|
self.scroll_offset = self.scroll_offset.saturating_add(1);
|
|
}
|
|
}
|
|
|
|
// ── Risk Tab ──
|
|
|
|
#[derive(Debug, Clone)]
|
|
pub struct CircuitBreakerRow {
|
|
pub name: String,
|
|
pub status: String,
|
|
pub current: String,
|
|
pub limit: String,
|
|
}
|
|
|
|
#[derive(Debug, Default)]
|
|
pub struct RiskTabState {
|
|
pub portfolio_var: f64,
|
|
pub max_drawdown: f64,
|
|
pub current_drawdown: f64,
|
|
pub sharpe_ratio: f64,
|
|
pub circuit_breakers: Vec<CircuitBreakerRow>,
|
|
pub scroll_offset: usize,
|
|
pub connected: bool,
|
|
}
|
|
|
|
impl RiskTabState {
|
|
pub fn scroll_up(&mut self) {
|
|
self.scroll_offset = self.scroll_offset.saturating_sub(1);
|
|
}
|
|
|
|
pub fn scroll_down(&mut self) {
|
|
self.scroll_offset = self.scroll_offset.saturating_add(1);
|
|
}
|
|
}
|
|
|
|
// ── System Tab ──
|
|
|
|
#[derive(Debug, Clone)]
|
|
pub struct ServiceRow {
|
|
pub name: String,
|
|
pub status: String,
|
|
pub latency_ms: f64,
|
|
pub message: String,
|
|
}
|
|
|
|
#[derive(Debug, Default)]
|
|
pub struct SystemTabState {
|
|
pub services: Vec<ServiceRow>,
|
|
pub gpu: GpuInfo,
|
|
pub cpu_percent: f64,
|
|
pub ram_used_gb: f64,
|
|
pub ram_total_gb: f64,
|
|
pub scroll_offset: usize,
|
|
pub connected: bool,
|
|
}
|
|
|
|
impl SystemTabState {
|
|
pub fn scroll_up(&mut self) {
|
|
self.scroll_offset = self.scroll_offset.saturating_sub(1);
|
|
}
|
|
|
|
pub fn scroll_down(&mut self) {
|
|
self.scroll_offset = self.scroll_offset.saturating_add(1);
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
#[allow(clippy::unwrap_used, clippy::expect_used)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
#[test]
|
|
fn test_tab_selection() {
|
|
let mut state = DashboardState::default();
|
|
assert_eq!(state.current_tab(), Tab::Training);
|
|
state.select_tab(2);
|
|
assert_eq!(state.current_tab(), Tab::Risk);
|
|
assert!(state.dirty);
|
|
}
|
|
|
|
#[test]
|
|
fn test_tab_selection_out_of_bounds() {
|
|
let mut state = DashboardState::default();
|
|
state.select_tab(99);
|
|
assert_eq!(state.current_tab(), Tab::Training);
|
|
}
|
|
|
|
#[test]
|
|
fn test_loss_history_ring_buffer() {
|
|
let mut tab = TrainingTabState::default();
|
|
for i in 0..250 {
|
|
tab.push_loss(i as f64);
|
|
}
|
|
assert_eq!(tab.loss_history.len(), CHART_HISTORY);
|
|
assert!((tab.loss_history[0] - 50.0).abs() < f64::EPSILON);
|
|
}
|
|
|
|
#[test]
|
|
fn test_fill_ring_buffer() {
|
|
let mut tab = TradingTabState::default();
|
|
for i in 0..60 {
|
|
tab.push_fill(FillRow {
|
|
time: format!("{}", i),
|
|
side: "BUY".to_string(),
|
|
symbol: "ES".to_string(),
|
|
quantity: 1.0,
|
|
price: 5000.0,
|
|
status: "FILLED".to_string(),
|
|
});
|
|
}
|
|
assert_eq!(tab.fills.len(), 50);
|
|
}
|
|
|
|
#[test]
|
|
fn test_scroll() {
|
|
let mut state = DashboardState::default();
|
|
state.scroll_down();
|
|
assert_eq!(state.training.scroll_offset, 1);
|
|
state.scroll_up();
|
|
assert_eq!(state.training.scroll_offset, 0);
|
|
state.scroll_up(); // should not underflow
|
|
assert_eq!(state.training.scroll_offset, 0);
|
|
}
|
|
}
|
|
```
|
|
|
|
**Step 3: Create watch/mod.rs**
|
|
|
|
Create `bin/fxt/src/commands/watch/mod.rs`:
|
|
|
|
```rust
|
|
pub mod state;
|
|
mod streams;
|
|
mod render;
|
|
mod event_loop;
|
|
|
|
use anyhow::Result;
|
|
|
|
pub async fn run(api_gateway_url: &str, jwt_token: &str) -> Result<()> {
|
|
event_loop::run_dashboard(api_gateway_url, jwt_token).await
|
|
}
|
|
```
|
|
|
|
**Step 4: Run tests**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo test -p fxt --lib watch::state`
|
|
Expected: 5 tests pass
|
|
|
|
**Step 5: Commit**
|
|
|
|
```bash
|
|
git add bin/fxt/src/commands/watch/
|
|
git commit -m "feat(fxt): add dashboard state types with tests"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 4: gRPC Stream Manager
|
|
|
|
**Files:**
|
|
- Create: `bin/fxt/src/commands/watch/streams.rs`
|
|
|
|
**Step 1: Write stream manager**
|
|
|
|
Create `bin/fxt/src/commands/watch/streams.rs`. This module spawns one tokio task per gRPC stream, all feeding into a single `mpsc::Sender<StreamEvent>`.
|
|
|
|
```rust
|
|
use anyhow::{Context, Result};
|
|
use tokio::sync::mpsc;
|
|
use tokio::time::{sleep, Duration};
|
|
use tonic::metadata::MetadataValue;
|
|
use tonic::transport::Channel;
|
|
use tonic::Request;
|
|
use tracing::{debug, warn};
|
|
|
|
use crate::proto::monitoring::{
|
|
monitoring_service_client::MonitoringServiceClient,
|
|
StreamTrainingMetricsRequest,
|
|
};
|
|
use crate::proto::foxhunt_tli::{
|
|
trading_service_client::TradingServiceClient,
|
|
GetPositionsRequest, GetRiskMetricsRequest,
|
|
SubscribeOrderUpdatesRequest, SubscribeRiskAlertsRequest,
|
|
SubscribeSystemStatusRequest,
|
|
};
|
|
|
|
use super::state::*;
|
|
|
|
/// Events pushed from gRPC streams into the dashboard state.
|
|
#[derive(Debug)]
|
|
pub enum StreamEvent {
|
|
TrainingUpdate {
|
|
sessions: Vec<TrainingSession>,
|
|
gpu: GpuInfo,
|
|
},
|
|
OrderUpdate {
|
|
order_id: String,
|
|
symbol: String,
|
|
status: String,
|
|
filled_qty: f64,
|
|
last_fill_price: f64,
|
|
timestamp_nanos: i64,
|
|
},
|
|
PositionsSnapshot {
|
|
positions: Vec<PositionRow>,
|
|
},
|
|
RiskMetrics {
|
|
var: f64,
|
|
max_drawdown: f64,
|
|
current_drawdown: f64,
|
|
sharpe: f64,
|
|
},
|
|
RiskAlert {
|
|
severity: String,
|
|
symbol: String,
|
|
message: String,
|
|
threshold: f64,
|
|
current: f64,
|
|
},
|
|
SystemStatus {
|
|
service: String,
|
|
status: String,
|
|
message: String,
|
|
},
|
|
StreamDisconnected {
|
|
stream_name: String,
|
|
},
|
|
}
|
|
|
|
const MAX_BACKOFF_SECS: u64 = 30;
|
|
|
|
async fn connect_channel(url: &str) -> Result<Channel> {
|
|
Channel::from_shared(url.to_owned())
|
|
.context("Invalid API Gateway URL")?
|
|
.connect()
|
|
.await
|
|
.context("Failed to connect to API Gateway")
|
|
}
|
|
|
|
fn auth_metadata(jwt: &str) -> Result<tonic::metadata::MetadataValue<tonic::metadata::Ascii>> {
|
|
MetadataValue::try_from(format!("Bearer {}", jwt))
|
|
.context("Invalid JWT token format")
|
|
}
|
|
|
|
/// Spawn all stream tasks. Returns the receiving end.
|
|
pub fn spawn_all_streams(
|
|
api_gateway_url: String,
|
|
jwt_token: String,
|
|
) -> mpsc::Receiver<StreamEvent> {
|
|
let (tx, rx) = mpsc::channel(256);
|
|
|
|
// Training metrics stream (monitoring.proto)
|
|
{
|
|
let tx = tx.clone();
|
|
let url = api_gateway_url.clone();
|
|
let jwt = jwt_token.clone();
|
|
tokio::spawn(async move {
|
|
stream_training_loop(&url, &jwt, tx).await;
|
|
});
|
|
}
|
|
|
|
// Order updates stream (trading.proto)
|
|
{
|
|
let tx = tx.clone();
|
|
let url = api_gateway_url.clone();
|
|
let jwt = jwt_token.clone();
|
|
tokio::spawn(async move {
|
|
stream_order_updates_loop(&url, &jwt, tx).await;
|
|
});
|
|
}
|
|
|
|
// Risk alerts stream (trading.proto)
|
|
{
|
|
let tx = tx.clone();
|
|
let url = api_gateway_url.clone();
|
|
let jwt = jwt_token.clone();
|
|
tokio::spawn(async move {
|
|
stream_risk_alerts_loop(&url, &jwt, tx).await;
|
|
});
|
|
}
|
|
|
|
// System status stream (trading.proto)
|
|
{
|
|
let tx = tx.clone();
|
|
let url = api_gateway_url.clone();
|
|
let jwt = jwt_token.clone();
|
|
tokio::spawn(async move {
|
|
stream_system_status_loop(&url, &jwt, tx).await;
|
|
});
|
|
}
|
|
|
|
// Initial positions + risk snapshot (unary RPCs, then done)
|
|
{
|
|
let tx = tx.clone();
|
|
let url = api_gateway_url.clone();
|
|
let jwt = jwt_token.clone();
|
|
tokio::spawn(async move {
|
|
if let Err(e) = fetch_initial_state(&url, &jwt, &tx).await {
|
|
warn!("Failed to fetch initial state: {}", e);
|
|
}
|
|
});
|
|
}
|
|
|
|
rx
|
|
}
|
|
|
|
async fn stream_training_loop(url: &str, jwt: &str, tx: mpsc::Sender<StreamEvent>) {
|
|
let mut backoff = 1u64;
|
|
loop {
|
|
match try_stream_training(url, jwt, &tx).await {
|
|
Ok(()) => backoff = 1,
|
|
Err(e) => {
|
|
warn!("Training stream error: {}", e);
|
|
let _ = tx.send(StreamEvent::StreamDisconnected {
|
|
stream_name: "Training".to_string(),
|
|
}).await;
|
|
}
|
|
}
|
|
sleep(Duration::from_secs(backoff)).await;
|
|
backoff = (backoff * 2).min(MAX_BACKOFF_SECS);
|
|
}
|
|
}
|
|
|
|
async fn try_stream_training(
|
|
url: &str,
|
|
jwt: &str,
|
|
tx: &mpsc::Sender<StreamEvent>,
|
|
) -> Result<()> {
|
|
let channel = connect_channel(url).await?;
|
|
let mut client = MonitoringServiceClient::new(channel);
|
|
let mut request = Request::new(StreamTrainingMetricsRequest {
|
|
model_filter: String::new(),
|
|
interval_seconds: 3,
|
|
});
|
|
request.metadata_mut().insert("authorization", auth_metadata(jwt)?);
|
|
|
|
let mut stream = client
|
|
.stream_training_metrics(request)
|
|
.await
|
|
.context("StreamTrainingMetrics failed")?
|
|
.into_inner();
|
|
|
|
use tokio_stream::StreamExt;
|
|
while let Some(result) = stream.next().await {
|
|
let resp = result.context("Training stream message error")?;
|
|
let sessions: Vec<TrainingSession> = resp.sessions.iter().map(|s| TrainingSession {
|
|
model: s.model.clone(),
|
|
fold: s.fold.clone(),
|
|
is_hyperopt: s.is_hyperopt,
|
|
epoch: s.current_epoch,
|
|
epoch_loss: s.epoch_loss,
|
|
validation_loss: s.validation_loss,
|
|
learning_rate: s.learning_rate,
|
|
gpu_percent: 0.0, // filled from gpu snapshot
|
|
batches_per_second: s.batches_per_second,
|
|
gradient_norm: s.gradient_norm,
|
|
epoch_duration_seconds: s.epoch_duration_seconds,
|
|
}).collect();
|
|
let gpu = resp.gpu.as_ref().map(|g| GpuInfo {
|
|
utilization_percent: g.utilization_percent,
|
|
memory_used_mb: g.memory_used_mb,
|
|
memory_total_mb: g.memory_total_mb,
|
|
temperature_celsius: g.temperature_celsius,
|
|
}).unwrap_or_default();
|
|
tx.send(StreamEvent::TrainingUpdate { sessions, gpu }).await
|
|
.context("Dashboard receiver dropped")?;
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
async fn stream_order_updates_loop(url: &str, jwt: &str, tx: mpsc::Sender<StreamEvent>) {
|
|
let mut backoff = 1u64;
|
|
loop {
|
|
match try_stream_order_updates(url, jwt, &tx).await {
|
|
Ok(()) => backoff = 1,
|
|
Err(e) => {
|
|
warn!("Order updates stream error: {}", e);
|
|
let _ = tx.send(StreamEvent::StreamDisconnected {
|
|
stream_name: "Trading".to_string(),
|
|
}).await;
|
|
}
|
|
}
|
|
sleep(Duration::from_secs(backoff)).await;
|
|
backoff = (backoff * 2).min(MAX_BACKOFF_SECS);
|
|
}
|
|
}
|
|
|
|
async fn try_stream_order_updates(
|
|
url: &str,
|
|
jwt: &str,
|
|
tx: &mpsc::Sender<StreamEvent>,
|
|
) -> Result<()> {
|
|
let channel = connect_channel(url).await?;
|
|
let mut client = TradingServiceClient::new(channel);
|
|
let mut request = Request::new(SubscribeOrderUpdatesRequest { account_id: None });
|
|
request.metadata_mut().insert("authorization", auth_metadata(jwt)?);
|
|
|
|
let mut stream = client
|
|
.subscribe_order_updates(request)
|
|
.await
|
|
.context("SubscribeOrderUpdates failed")?
|
|
.into_inner();
|
|
|
|
use tokio_stream::StreamExt;
|
|
while let Some(result) = stream.next().await {
|
|
let ev = result.context("Order update stream message error")?;
|
|
tx.send(StreamEvent::OrderUpdate {
|
|
order_id: ev.order_id,
|
|
symbol: ev.symbol,
|
|
status: format!("{:?}", ev.status),
|
|
filled_qty: ev.filled_quantity,
|
|
last_fill_price: ev.last_fill_price,
|
|
timestamp_nanos: ev.timestamp_unix_nanos,
|
|
}).await.context("Dashboard receiver dropped")?;
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
async fn stream_risk_alerts_loop(url: &str, jwt: &str, tx: mpsc::Sender<StreamEvent>) {
|
|
let mut backoff = 1u64;
|
|
loop {
|
|
match try_stream_risk_alerts(url, jwt, &tx).await {
|
|
Ok(()) => backoff = 1,
|
|
Err(e) => {
|
|
warn!("Risk alerts stream error: {}", e);
|
|
let _ = tx.send(StreamEvent::StreamDisconnected {
|
|
stream_name: "Risk".to_string(),
|
|
}).await;
|
|
}
|
|
}
|
|
sleep(Duration::from_secs(backoff)).await;
|
|
backoff = (backoff * 2).min(MAX_BACKOFF_SECS);
|
|
}
|
|
}
|
|
|
|
async fn try_stream_risk_alerts(
|
|
url: &str,
|
|
jwt: &str,
|
|
tx: &mpsc::Sender<StreamEvent>,
|
|
) -> Result<()> {
|
|
let channel = connect_channel(url).await?;
|
|
let mut client = TradingServiceClient::new(channel);
|
|
let mut request = Request::new(SubscribeRiskAlertsRequest {
|
|
min_severity: vec![],
|
|
symbols: vec![],
|
|
});
|
|
request.metadata_mut().insert("authorization", auth_metadata(jwt)?);
|
|
|
|
let mut stream = client
|
|
.subscribe_risk_alerts(request)
|
|
.await
|
|
.context("SubscribeRiskAlerts failed")?
|
|
.into_inner();
|
|
|
|
use tokio_stream::StreamExt;
|
|
while let Some(result) = stream.next().await {
|
|
let ev = result.context("Risk alert stream message error")?;
|
|
tx.send(StreamEvent::RiskAlert {
|
|
severity: format!("{:?}", ev.severity),
|
|
symbol: ev.symbol,
|
|
message: ev.message,
|
|
threshold: ev.threshold_value,
|
|
current: ev.current_value,
|
|
}).await.context("Dashboard receiver dropped")?;
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
async fn stream_system_status_loop(url: &str, jwt: &str, tx: mpsc::Sender<StreamEvent>) {
|
|
let mut backoff = 1u64;
|
|
loop {
|
|
match try_stream_system_status(url, jwt, &tx).await {
|
|
Ok(()) => backoff = 1,
|
|
Err(e) => {
|
|
warn!("System status stream error: {}", e);
|
|
let _ = tx.send(StreamEvent::StreamDisconnected {
|
|
stream_name: "System".to_string(),
|
|
}).await;
|
|
}
|
|
}
|
|
sleep(Duration::from_secs(backoff)).await;
|
|
backoff = (backoff * 2).min(MAX_BACKOFF_SECS);
|
|
}
|
|
}
|
|
|
|
async fn try_stream_system_status(
|
|
url: &str,
|
|
jwt: &str,
|
|
tx: &mpsc::Sender<StreamEvent>,
|
|
) -> Result<()> {
|
|
let channel = connect_channel(url).await?;
|
|
let mut client = TradingServiceClient::new(channel);
|
|
let mut request = Request::new(SubscribeSystemStatusRequest {
|
|
service_names: vec![],
|
|
});
|
|
request.metadata_mut().insert("authorization", auth_metadata(jwt)?);
|
|
|
|
let mut stream = client
|
|
.subscribe_system_status(request)
|
|
.await
|
|
.context("SubscribeSystemStatus failed")?
|
|
.into_inner();
|
|
|
|
use tokio_stream::StreamExt;
|
|
while let Some(result) = stream.next().await {
|
|
let ev = result.context("System status stream message error")?;
|
|
tx.send(StreamEvent::SystemStatus {
|
|
service: ev.service_name,
|
|
status: format!("{:?}", ev.status),
|
|
message: ev.message,
|
|
}).await.context("Dashboard receiver dropped")?;
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
async fn fetch_initial_state(
|
|
url: &str,
|
|
jwt: &str,
|
|
tx: &mpsc::Sender<StreamEvent>,
|
|
) -> Result<()> {
|
|
let channel = connect_channel(url).await?;
|
|
let mut client = TradingServiceClient::new(channel);
|
|
let auth = auth_metadata(jwt)?;
|
|
|
|
// Fetch positions
|
|
let mut pos_req = Request::new(GetPositionsRequest { symbol: None });
|
|
pos_req.metadata_mut().insert("authorization", auth.clone());
|
|
if let Ok(resp) = client.get_positions(pos_req).await {
|
|
let positions = resp.into_inner().positions.iter().map(|p| PositionRow {
|
|
symbol: p.symbol.clone(),
|
|
side: if p.quantity >= 0.0 { "LONG" } else { "SHORT" }.to_string(),
|
|
quantity: p.quantity.abs(),
|
|
entry_price: p.average_cost,
|
|
unrealized_pnl: p.unrealized_pnl,
|
|
status: "OPEN".to_string(),
|
|
}).collect();
|
|
let _ = tx.send(StreamEvent::PositionsSnapshot { positions }).await;
|
|
}
|
|
|
|
// Fetch risk metrics
|
|
let mut risk_req = Request::new(GetRiskMetricsRequest {
|
|
portfolio_id: None,
|
|
start_time_unix_nanos: None,
|
|
end_time_unix_nanos: None,
|
|
});
|
|
risk_req.metadata_mut().insert("authorization", auth);
|
|
if let Ok(resp) = client.get_risk_metrics(risk_req).await {
|
|
let r = resp.into_inner();
|
|
let _ = tx.send(StreamEvent::RiskMetrics {
|
|
var: r.value_at_risk,
|
|
max_drawdown: r.max_drawdown,
|
|
current_drawdown: r.current_drawdown,
|
|
sharpe: r.sharpe_ratio,
|
|
}).await;
|
|
}
|
|
|
|
Ok(())
|
|
}
|
|
```
|
|
|
|
**Step 2: Verify compilation**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo check -p fxt`
|
|
Expected: compiles (streams.rs may need import path adjustments — the proto module paths depend on the `include_proto!` setup in lib.rs)
|
|
|
|
**Step 3: Commit**
|
|
|
|
```bash
|
|
git add bin/fxt/src/commands/watch/streams.rs
|
|
git commit -m "feat(fxt): add gRPC stream manager with reconnect"
|
|
```
|
|
|
|
**NOTE to implementer:** The exact proto module paths (`crate::proto::monitoring`, `crate::proto::foxhunt_tli`) must match the `include_proto!` setup in `bin/fxt/src/lib.rs`. Check `lib.rs` for the actual module paths and adjust imports accordingly. The package names are `monitoring` and `foxhunt.tli` (which becomes `foxhunt_tli` in Rust).
|
|
|
|
---
|
|
|
|
### Task 5: Renderer
|
|
|
|
**Files:**
|
|
- Create: `bin/fxt/src/commands/watch/render.rs`
|
|
|
|
**Step 1: Write renderer**
|
|
|
|
Create `bin/fxt/src/commands/watch/render.rs`:
|
|
|
|
```rust
|
|
use ratatui::{
|
|
Frame,
|
|
layout::{Constraint, Direction, Layout, Rect},
|
|
style::{Color, Modifier, Style},
|
|
text::{Line, Span},
|
|
widgets::{Block, Borders, Cell, Paragraph, Row, Sparkline, Table, Tabs},
|
|
};
|
|
|
|
use super::state::*;
|
|
|
|
pub fn render(frame: &mut Frame, state: &DashboardState) {
|
|
let chunks = Layout::default()
|
|
.direction(Direction::Vertical)
|
|
.constraints([
|
|
Constraint::Length(3), // tab bar
|
|
Constraint::Min(0), // tab content
|
|
Constraint::Length(1), // status bar
|
|
])
|
|
.split(frame.area());
|
|
|
|
render_tab_bar(frame, chunks[0], state);
|
|
|
|
match state.current_tab() {
|
|
Tab::Training => render_training_tab(frame, chunks[1], &state.training),
|
|
Tab::Trading => render_trading_tab(frame, chunks[1], &state.trading),
|
|
Tab::Risk => render_risk_tab(frame, chunks[1], &state.risk),
|
|
Tab::System => render_system_tab(frame, chunks[1], &state.system),
|
|
}
|
|
|
|
render_status_bar(frame, chunks[2]);
|
|
}
|
|
|
|
fn render_tab_bar(frame: &mut Frame, area: Rect, state: &DashboardState) {
|
|
let titles: Vec<Line> = Tab::ALL.iter().map(|t| {
|
|
Line::from(Span::styled(t.title(), Style::default().fg(Color::White)))
|
|
}).collect();
|
|
let tabs = Tabs::new(titles)
|
|
.block(Block::default().borders(Borders::ALL).title(" fxt watch "))
|
|
.select(state.active_tab)
|
|
.style(Style::default().fg(Color::DarkGray))
|
|
.highlight_style(Style::default().fg(Color::Cyan).add_modifier(Modifier::BOLD));
|
|
frame.render_widget(tabs, area);
|
|
}
|
|
|
|
fn render_status_bar(frame: &mut Frame, area: Rect) {
|
|
let bar = Paragraph::new(Line::from(vec![
|
|
Span::styled(" 1-4", Style::default().fg(Color::Cyan)),
|
|
Span::raw(" tabs "),
|
|
Span::styled("j/k", Style::default().fg(Color::Cyan)),
|
|
Span::raw(" scroll "),
|
|
Span::styled("q", Style::default().fg(Color::Cyan)),
|
|
Span::raw(" quit"),
|
|
]));
|
|
frame.render_widget(bar, area);
|
|
}
|
|
|
|
fn render_training_tab(frame: &mut Frame, area: Rect, tab: &TrainingTabState) {
|
|
let chunks = Layout::default()
|
|
.direction(Direction::Vertical)
|
|
.constraints([Constraint::Min(8), Constraint::Length(6)])
|
|
.split(area);
|
|
|
|
// Sessions table
|
|
let header = Row::new(vec!["Model", "Fold", "Epoch", "Loss", "Val Loss", "LR", "Batch/s", "Grad"])
|
|
.style(Style::default().fg(Color::Yellow).add_modifier(Modifier::BOLD));
|
|
|
|
let rows: Vec<Row> = tab.sessions.iter().skip(tab.scroll_offset).map(|s| {
|
|
Row::new(vec![
|
|
Cell::from(s.model.clone()),
|
|
Cell::from(s.fold.clone()),
|
|
Cell::from(format!("{:.0}", s.epoch)),
|
|
Cell::from(format!("{:.4}", s.epoch_loss)),
|
|
Cell::from(format!("{:.4}", s.validation_loss)),
|
|
Cell::from(format!("{:.1e}", s.learning_rate)),
|
|
Cell::from(format!("{:.1}", s.batches_per_second)),
|
|
Cell::from(format!("{:.2}", s.gradient_norm)),
|
|
])
|
|
}).collect();
|
|
|
|
let connected_indicator = if tab.connected { " [connected]" } else { " [disconnected]" };
|
|
let table = Table::new(rows, [
|
|
Constraint::Length(10), Constraint::Length(8), Constraint::Length(8),
|
|
Constraint::Length(10), Constraint::Length(10), Constraint::Length(10),
|
|
Constraint::Length(8), Constraint::Length(8),
|
|
])
|
|
.header(header)
|
|
.block(Block::default().borders(Borders::ALL).title(format!(" Training{} ", connected_indicator)));
|
|
frame.render_widget(table, chunks[0]);
|
|
|
|
// GPU + loss sparkline
|
|
let bottom = Layout::default()
|
|
.direction(Direction::Horizontal)
|
|
.constraints([Constraint::Percentage(50), Constraint::Percentage(50)])
|
|
.split(chunks[1]);
|
|
|
|
let gpu_text = format!(
|
|
"GPU: {:.0}% VRAM: {:.0}/{:.0} MB Temp: {:.0}C",
|
|
tab.gpu.utilization_percent, tab.gpu.memory_used_mb,
|
|
tab.gpu.memory_total_mb, tab.gpu.temperature_celsius,
|
|
);
|
|
let gpu_para = Paragraph::new(gpu_text)
|
|
.block(Block::default().borders(Borders::ALL).title(" GPU "));
|
|
frame.render_widget(gpu_para, bottom[0]);
|
|
|
|
let loss_data: Vec<u64> = tab.loss_history.iter().map(|v| {
|
|
(v.clamp(0.0, 10.0) * 100.0) as u64
|
|
}).collect();
|
|
let sparkline = Sparkline::default()
|
|
.block(Block::default().borders(Borders::ALL).title(" Loss History "))
|
|
.data(&loss_data)
|
|
.style(Style::default().fg(Color::Green));
|
|
frame.render_widget(sparkline, bottom[1]);
|
|
}
|
|
|
|
fn render_trading_tab(frame: &mut Frame, area: Rect, tab: &TradingTabState) {
|
|
let chunks = Layout::default()
|
|
.direction(Direction::Vertical)
|
|
.constraints([Constraint::Percentage(40), Constraint::Percentage(40), Constraint::Length(3)])
|
|
.split(area);
|
|
|
|
// Positions table
|
|
let header = Row::new(vec!["Symbol", "Side", "Qty", "Entry", "PnL", "Status"])
|
|
.style(Style::default().fg(Color::Yellow).add_modifier(Modifier::BOLD));
|
|
let rows: Vec<Row> = tab.positions.iter().map(|p| {
|
|
let pnl_color = if p.unrealized_pnl >= 0.0 { Color::Green } else { Color::Red };
|
|
Row::new(vec![
|
|
Cell::from(p.symbol.clone()),
|
|
Cell::from(p.side.clone()),
|
|
Cell::from(format!("{:.0}", p.quantity)),
|
|
Cell::from(format!("{:.2}", p.entry_price)),
|
|
Cell::from(Span::styled(format!("{:+.2}", p.unrealized_pnl), Style::default().fg(pnl_color))),
|
|
Cell::from(p.status.clone()),
|
|
])
|
|
}).collect();
|
|
let connected_indicator = if tab.connected { " [connected]" } else { " [disconnected]" };
|
|
let table = Table::new(rows, [
|
|
Constraint::Length(10), Constraint::Length(6), Constraint::Length(6),
|
|
Constraint::Length(10), Constraint::Length(12), Constraint::Length(8),
|
|
])
|
|
.header(header)
|
|
.block(Block::default().borders(Borders::ALL).title(format!(" Positions{} ", connected_indicator)));
|
|
frame.render_widget(table, chunks[0]);
|
|
|
|
// Recent fills
|
|
let fill_header = Row::new(vec!["Time", "Side", "Symbol", "Qty", "Price", "Status"])
|
|
.style(Style::default().fg(Color::Yellow).add_modifier(Modifier::BOLD));
|
|
let fill_rows: Vec<Row> = tab.fills.iter().skip(tab.scroll_offset).take(10).map(|f| {
|
|
Row::new(vec![
|
|
Cell::from(f.time.clone()),
|
|
Cell::from(f.side.clone()),
|
|
Cell::from(f.symbol.clone()),
|
|
Cell::from(format!("{:.0}", f.quantity)),
|
|
Cell::from(format!("{:.2}", f.price)),
|
|
Cell::from(f.status.clone()),
|
|
])
|
|
}).collect();
|
|
let fills_table = Table::new(fill_rows, [
|
|
Constraint::Length(10), Constraint::Length(6), Constraint::Length(10),
|
|
Constraint::Length(6), Constraint::Length(10), Constraint::Length(8),
|
|
])
|
|
.header(fill_header)
|
|
.block(Block::default().borders(Borders::ALL).title(" Recent Fills "));
|
|
frame.render_widget(fills_table, chunks[1]);
|
|
|
|
// PnL summary
|
|
let pnl_color = if tab.day_pnl >= 0.0 { Color::Green } else { Color::Red };
|
|
let pnl_text = format!(
|
|
"Day PnL: {:+.2} | Realized: {:+.2} Unrealized: {:+.2}",
|
|
tab.day_pnl, tab.realized_pnl, tab.unrealized_pnl,
|
|
);
|
|
let pnl_bar = Paragraph::new(Span::styled(pnl_text, Style::default().fg(pnl_color)))
|
|
.block(Block::default().borders(Borders::ALL));
|
|
frame.render_widget(pnl_bar, chunks[2]);
|
|
}
|
|
|
|
fn render_risk_tab(frame: &mut Frame, area: Rect, tab: &RiskTabState) {
|
|
let chunks = Layout::default()
|
|
.direction(Direction::Vertical)
|
|
.constraints([Constraint::Length(5), Constraint::Min(0)])
|
|
.split(area);
|
|
|
|
// Risk metrics summary
|
|
let connected_indicator = if tab.connected { " [connected]" } else { " [disconnected]" };
|
|
let summary = Paragraph::new(vec![
|
|
Line::from(format!("Portfolio VaR (95%): ${:.0} Max DD: {:.1}%", tab.portfolio_var, tab.max_drawdown * 100.0)),
|
|
Line::from(format!("Current DD: {:.1}% Sharpe: {:.2}", tab.current_drawdown * 100.0, tab.sharpe_ratio)),
|
|
])
|
|
.block(Block::default().borders(Borders::ALL).title(format!(" Risk Metrics{} ", connected_indicator)));
|
|
frame.render_widget(summary, chunks[0]);
|
|
|
|
// Circuit breakers
|
|
let header = Row::new(vec!["Breaker", "Status", "Current", "Limit"])
|
|
.style(Style::default().fg(Color::Yellow).add_modifier(Modifier::BOLD));
|
|
let rows: Vec<Row> = tab.circuit_breakers.iter().map(|cb| {
|
|
let status_color = if cb.status == "OK" { Color::Green } else { Color::Red };
|
|
Row::new(vec![
|
|
Cell::from(cb.name.clone()),
|
|
Cell::from(Span::styled(&cb.status, Style::default().fg(status_color))),
|
|
Cell::from(cb.current.clone()),
|
|
Cell::from(cb.limit.clone()),
|
|
])
|
|
}).collect();
|
|
let table = Table::new(rows, [
|
|
Constraint::Length(18), Constraint::Length(10),
|
|
Constraint::Length(15), Constraint::Length(15),
|
|
])
|
|
.header(header)
|
|
.block(Block::default().borders(Borders::ALL).title(" Circuit Breakers "));
|
|
frame.render_widget(table, chunks[1]);
|
|
}
|
|
|
|
fn render_system_tab(frame: &mut Frame, area: Rect, tab: &SystemTabState) {
|
|
let chunks = Layout::default()
|
|
.direction(Direction::Vertical)
|
|
.constraints([Constraint::Min(8), Constraint::Length(5)])
|
|
.split(area);
|
|
|
|
// Services table
|
|
let header = Row::new(vec!["Service", "Status", "Latency", "Message"])
|
|
.style(Style::default().fg(Color::Yellow).add_modifier(Modifier::BOLD));
|
|
let rows: Vec<Row> = tab.services.iter().map(|s| {
|
|
let status_color = match s.status.as_str() {
|
|
"HEALTHY" | "UP" => Color::Green,
|
|
"DEGRADED" => Color::Yellow,
|
|
_ => Color::Red,
|
|
};
|
|
Row::new(vec![
|
|
Cell::from(s.name.clone()),
|
|
Cell::from(Span::styled(&s.status, Style::default().fg(status_color))),
|
|
Cell::from(format!("{:.0}ms", s.latency_ms)),
|
|
Cell::from(s.message.clone()),
|
|
])
|
|
}).collect();
|
|
let connected_indicator = if tab.connected { " [connected]" } else { " [disconnected]" };
|
|
let table = Table::new(rows, [
|
|
Constraint::Length(22), Constraint::Length(12),
|
|
Constraint::Length(10), Constraint::Min(20),
|
|
])
|
|
.header(header)
|
|
.block(Block::default().borders(Borders::ALL).title(format!(" Services{} ", connected_indicator)));
|
|
frame.render_widget(table, chunks[0]);
|
|
|
|
// GPU/CPU summary
|
|
let sys_text = format!(
|
|
"GPU: {:.0}% VRAM: {:.0}/{:.0} MB | CPU: {:.0}% RAM: {:.1}/{:.1} GB",
|
|
tab.gpu.utilization_percent, tab.gpu.memory_used_mb, tab.gpu.memory_total_mb,
|
|
tab.cpu_percent, tab.ram_used_gb, tab.ram_total_gb,
|
|
);
|
|
let sys_para = Paragraph::new(sys_text)
|
|
.block(Block::default().borders(Borders::ALL).title(" Resources "));
|
|
frame.render_widget(sys_para, chunks[1]);
|
|
}
|
|
```
|
|
|
|
**Step 2: Verify compilation**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo check -p fxt`
|
|
Expected: compiles
|
|
|
|
**Step 3: Commit**
|
|
|
|
```bash
|
|
git add bin/fxt/src/commands/watch/render.rs
|
|
git commit -m "feat(fxt): add ratatui dashboard renderer for 4 tabs"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 6: Event Loop
|
|
|
|
**Files:**
|
|
- Create: `bin/fxt/src/commands/watch/event_loop.rs`
|
|
|
|
**Step 1: Write event loop**
|
|
|
|
Create `bin/fxt/src/commands/watch/event_loop.rs`:
|
|
|
|
```rust
|
|
use anyhow::Result;
|
|
use crossterm::{
|
|
event::{self, Event, KeyCode, KeyEvent, KeyModifiers, EventStream},
|
|
execute,
|
|
terminal::{disable_raw_mode, enable_raw_mode, EnterAlternateScreen, LeaveAlternateScreen},
|
|
};
|
|
use ratatui::prelude::*;
|
|
use std::io;
|
|
use tokio::time::{interval, Duration};
|
|
|
|
use super::render;
|
|
use super::state::DashboardState;
|
|
use super::streams::{self, StreamEvent};
|
|
|
|
pub async fn run_dashboard(api_gateway_url: &str, jwt_token: &str) -> Result<()> {
|
|
// Setup terminal
|
|
enable_raw_mode()?;
|
|
let mut stdout = io::stdout();
|
|
execute!(stdout, EnterAlternateScreen)?;
|
|
let backend = CrosstermBackend::new(stdout);
|
|
let mut terminal = Terminal::new(backend)?;
|
|
|
|
let result = run_event_loop(&mut terminal, api_gateway_url, jwt_token).await;
|
|
|
|
// Restore terminal
|
|
disable_raw_mode()?;
|
|
execute!(terminal.backend_mut(), LeaveAlternateScreen)?;
|
|
terminal.show_cursor()?;
|
|
|
|
result
|
|
}
|
|
|
|
async fn run_event_loop(
|
|
terminal: &mut Terminal<CrosstermBackend<io::Stdout>>,
|
|
api_gateway_url: &str,
|
|
jwt_token: &str,
|
|
) -> Result<()> {
|
|
let mut state = DashboardState::default();
|
|
state.dirty = true; // initial render
|
|
|
|
let mut stream_rx = streams::spawn_all_streams(
|
|
api_gateway_url.to_string(),
|
|
jwt_token.to_string(),
|
|
);
|
|
|
|
let mut tick = interval(Duration::from_millis(200));
|
|
let mut event_stream = EventStream::new();
|
|
|
|
use tokio_stream::StreamExt;
|
|
|
|
loop {
|
|
tokio::select! {
|
|
// gRPC stream events
|
|
Some(event) = stream_rx.recv() => {
|
|
apply_stream_event(&mut state, event);
|
|
}
|
|
// Terminal key events
|
|
Some(Ok(event)) = event_stream.next() => {
|
|
if let Event::Key(key) = event {
|
|
if handle_key(&mut state, key) {
|
|
break; // quit
|
|
}
|
|
}
|
|
}
|
|
// Render tick
|
|
_ = tick.tick() => {
|
|
if state.dirty {
|
|
terminal.draw(|frame| render::render(frame, &state))?;
|
|
state.dirty = false;
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
Ok(())
|
|
}
|
|
|
|
fn apply_stream_event(state: &mut DashboardState, event: StreamEvent) {
|
|
match event {
|
|
StreamEvent::TrainingUpdate { sessions, gpu } => {
|
|
// Push first session's loss to history for sparkline
|
|
if let Some(first) = sessions.first() {
|
|
state.training.push_loss(first.epoch_loss as f64);
|
|
}
|
|
state.training.sessions = sessions;
|
|
state.training.gpu = gpu.clone();
|
|
state.system.gpu = gpu;
|
|
state.training.connected = true;
|
|
}
|
|
StreamEvent::OrderUpdate { order_id, symbol, status, filled_qty, last_fill_price, timestamp_nanos } => {
|
|
let time = format_nanos(timestamp_nanos);
|
|
let side = if filled_qty >= 0.0 { "BUY" } else { "SELL" };
|
|
state.trading.push_fill(super::state::FillRow {
|
|
time,
|
|
side: side.to_string(),
|
|
symbol: symbol.clone(),
|
|
quantity: filled_qty.abs(),
|
|
price: last_fill_price,
|
|
status,
|
|
});
|
|
state.trading.connected = true;
|
|
}
|
|
StreamEvent::PositionsSnapshot { positions } => {
|
|
state.trading.positions = positions;
|
|
state.trading.unrealized_pnl = state.trading.positions.iter().map(|p| p.unrealized_pnl).sum();
|
|
state.trading.connected = true;
|
|
}
|
|
StreamEvent::RiskMetrics { var, max_drawdown, current_drawdown, sharpe } => {
|
|
state.risk.portfolio_var = var;
|
|
state.risk.max_drawdown = max_drawdown;
|
|
state.risk.current_drawdown = current_drawdown;
|
|
state.risk.sharpe_ratio = sharpe;
|
|
state.risk.connected = true;
|
|
}
|
|
StreamEvent::RiskAlert { severity: _, symbol: _, message, threshold, current } => {
|
|
// Update circuit breaker display from alerts
|
|
let status = if current < threshold { "OK" } else { "TRIPPED" };
|
|
let found = state.risk.circuit_breakers.iter_mut().find(|cb| cb.name == message);
|
|
if let Some(cb) = found {
|
|
cb.status = status.to_string();
|
|
cb.current = format!("{:.2}", current);
|
|
} else {
|
|
state.risk.circuit_breakers.push(super::state::CircuitBreakerRow {
|
|
name: message,
|
|
status: status.to_string(),
|
|
current: format!("{:.2}", current),
|
|
limit: format!("{:.2}", threshold),
|
|
});
|
|
}
|
|
state.risk.connected = true;
|
|
}
|
|
StreamEvent::SystemStatus { service, status, message } => {
|
|
let found = state.system.services.iter_mut().find(|s| s.name == service);
|
|
if let Some(s) = found {
|
|
s.status = status;
|
|
s.message = message;
|
|
} else {
|
|
state.system.services.push(super::state::ServiceRow {
|
|
name: service,
|
|
status,
|
|
latency_ms: 0.0,
|
|
message,
|
|
});
|
|
}
|
|
state.system.connected = true;
|
|
}
|
|
StreamEvent::StreamDisconnected { stream_name } => {
|
|
match stream_name.as_str() {
|
|
"Training" => state.training.connected = false,
|
|
"Trading" => state.trading.connected = false,
|
|
"Risk" => state.risk.connected = false,
|
|
"System" => state.system.connected = false,
|
|
_ => {}
|
|
}
|
|
}
|
|
}
|
|
state.dirty = true;
|
|
}
|
|
|
|
fn handle_key(state: &mut DashboardState, key: KeyEvent) -> bool {
|
|
match key.code {
|
|
KeyCode::Char('q') => return true,
|
|
KeyCode::Char('c') if key.modifiers.contains(KeyModifiers::CONTROL) => return true,
|
|
KeyCode::Char('1') => state.select_tab(0),
|
|
KeyCode::Char('2') => state.select_tab(1),
|
|
KeyCode::Char('3') => state.select_tab(2),
|
|
KeyCode::Char('4') => state.select_tab(3),
|
|
KeyCode::Char('j') | KeyCode::Down => state.scroll_down(),
|
|
KeyCode::Char('k') | KeyCode::Up => state.scroll_up(),
|
|
_ => {}
|
|
}
|
|
false
|
|
}
|
|
|
|
fn format_nanos(nanos: i64) -> String {
|
|
let secs = nanos / 1_000_000_000;
|
|
let hour = (secs / 3600) % 24;
|
|
let min = (secs / 60) % 60;
|
|
let sec = secs % 60;
|
|
format!("{:02}:{:02}:{:02}", hour, min, sec)
|
|
}
|
|
|
|
#[cfg(test)]
|
|
#[allow(clippy::unwrap_used, clippy::expect_used)]
|
|
mod tests {
|
|
use super::*;
|
|
use crossterm::event::KeyEventKind;
|
|
|
|
fn key(code: KeyCode) -> KeyEvent {
|
|
KeyEvent::new(code, KeyModifiers::NONE)
|
|
}
|
|
|
|
#[test]
|
|
fn test_handle_key_quit() {
|
|
let mut state = DashboardState::default();
|
|
assert!(handle_key(&mut state, key(KeyCode::Char('q'))));
|
|
}
|
|
|
|
#[test]
|
|
fn test_handle_key_tab_switch() {
|
|
let mut state = DashboardState::default();
|
|
assert!(!handle_key(&mut state, key(KeyCode::Char('3'))));
|
|
assert_eq!(state.active_tab, 2);
|
|
}
|
|
|
|
#[test]
|
|
fn test_handle_key_scroll() {
|
|
let mut state = DashboardState::default();
|
|
handle_key(&mut state, key(KeyCode::Char('j')));
|
|
assert_eq!(state.training.scroll_offset, 1);
|
|
handle_key(&mut state, key(KeyCode::Char('k')));
|
|
assert_eq!(state.training.scroll_offset, 0);
|
|
}
|
|
|
|
#[test]
|
|
fn test_format_nanos() {
|
|
assert_eq!(format_nanos(52321_000_000_000), "14:32:01");
|
|
}
|
|
|
|
#[test]
|
|
fn test_apply_training_update() {
|
|
let mut state = DashboardState::default();
|
|
apply_stream_event(&mut state, StreamEvent::TrainingUpdate {
|
|
sessions: vec![super::super::state::TrainingSession {
|
|
model: "DQN".to_string(),
|
|
fold: "1/5".to_string(),
|
|
is_hyperopt: false,
|
|
epoch: 10.0,
|
|
epoch_loss: 0.05,
|
|
validation_loss: 0.06,
|
|
learning_rate: 1e-4,
|
|
gpu_percent: 80.0,
|
|
batches_per_second: 120.0,
|
|
gradient_norm: 0.5,
|
|
epoch_duration_seconds: 2.0,
|
|
}],
|
|
gpu: super::super::state::GpuInfo {
|
|
utilization_percent: 80.0,
|
|
memory_used_mb: 30000.0,
|
|
memory_total_mb: 48000.0,
|
|
temperature_celsius: 65.0,
|
|
},
|
|
});
|
|
assert_eq!(state.training.sessions.len(), 1);
|
|
assert!(state.training.connected);
|
|
assert!(state.dirty);
|
|
}
|
|
|
|
#[test]
|
|
fn test_apply_system_status_upsert() {
|
|
let mut state = DashboardState::default();
|
|
apply_stream_event(&mut state, StreamEvent::SystemStatus {
|
|
service: "api-gateway".to_string(),
|
|
status: "HEALTHY".to_string(),
|
|
message: String::new(),
|
|
});
|
|
assert_eq!(state.system.services.len(), 1);
|
|
// Second update replaces, not appends
|
|
apply_stream_event(&mut state, StreamEvent::SystemStatus {
|
|
service: "api-gateway".to_string(),
|
|
status: "DEGRADED".to_string(),
|
|
message: "high latency".to_string(),
|
|
});
|
|
assert_eq!(state.system.services.len(), 1);
|
|
assert_eq!(state.system.services[0].status, "DEGRADED");
|
|
}
|
|
|
|
#[test]
|
|
fn test_apply_stream_disconnected() {
|
|
let mut state = DashboardState::default();
|
|
state.training.connected = true;
|
|
apply_stream_event(&mut state, StreamEvent::StreamDisconnected {
|
|
stream_name: "Training".to_string(),
|
|
});
|
|
assert!(!state.training.connected);
|
|
}
|
|
}
|
|
```
|
|
|
|
**Step 2: Run tests**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo test -p fxt --lib watch::event_loop`
|
|
Expected: 7 tests pass
|
|
|
|
**Step 3: Commit**
|
|
|
|
```bash
|
|
git add bin/fxt/src/commands/watch/event_loop.rs
|
|
git commit -m "feat(fxt): add dashboard event loop with key handling"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 7: Wire Watch Command into CLI
|
|
|
|
**Files:**
|
|
- Modify: `bin/fxt/src/commands/mod.rs`
|
|
- Modify: `bin/fxt/src/main.rs`
|
|
|
|
**Step 1: Add watch module to commands/mod.rs**
|
|
|
|
Add to `bin/fxt/src/commands/mod.rs`:
|
|
|
|
```rust
|
|
pub mod watch;
|
|
```
|
|
|
|
**Step 2: Add Watch variant to Commands enum in main.rs**
|
|
|
|
In `bin/fxt/src/main.rs`, add a new variant to the `Commands` enum:
|
|
|
|
```rust
|
|
/// Live streaming dashboard
|
|
Watch,
|
|
```
|
|
|
|
**Step 3: Add dispatch in the match block**
|
|
|
|
In the `match cli.command { ... }` block in `main.rs`, add:
|
|
|
|
```rust
|
|
Commands::Watch => commands::watch::run(&cli.api_gateway_url, &jwt_token).await,
|
|
```
|
|
|
|
**Step 4: Verify compilation**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo check -p fxt`
|
|
Expected: compiles
|
|
|
|
**Step 5: Commit**
|
|
|
|
```bash
|
|
git add bin/fxt/src/commands/mod.rs bin/fxt/src/main.rs
|
|
git commit -m "feat(fxt): wire fxt watch command into CLI"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 8: Wire Model List
|
|
|
|
**Files:**
|
|
- Modify: `bin/fxt/src/commands/model/list.rs`
|
|
|
|
**Step 1: Replace stub with real gRPC call**
|
|
|
|
Replace the entire contents of `bin/fxt/src/commands/model/list.rs` with a real implementation that calls `MLTrainingService::ListAvailableModels`. Follow the same pattern as `train/list.rs`:
|
|
|
|
1. `Channel::from_shared(url).connect().await`
|
|
2. `MlTrainingServiceClient::new(channel)`
|
|
3. `MetadataValue::try_from(format!("Bearer {}", jwt))` for auth
|
|
4. Call `list_available_models(request)`
|
|
5. Display results in a table using println
|
|
|
|
Use imports from `crate::proto::ml_training::{ ml_training_service_client::MlTrainingServiceClient, ListAvailableModelsRequest }`.
|
|
|
|
**Step 2: Verify compilation**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo check -p fxt`
|
|
|
|
**Step 3: Run existing tests**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo test -p fxt --lib`
|
|
Expected: all 134+ tests pass
|
|
|
|
**Step 4: Commit**
|
|
|
|
```bash
|
|
git add bin/fxt/src/commands/model/list.rs
|
|
git commit -m "feat(fxt): wire model list to MLTrainingService::ListAvailableModels"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 9: Wire Model Approve
|
|
|
|
**Files:**
|
|
- Modify: `bin/fxt/src/commands/model/approve.rs`
|
|
|
|
**Step 1: Replace stub with real gRPC call**
|
|
|
|
Same pattern as Task 8 but calls the new `ApproveModel` RPC (added in Task 2).
|
|
|
|
Import `ApproveModelRequest` from `crate::proto::ml_training`.
|
|
|
|
Display success/failure message from the response.
|
|
|
|
**Step 2: Verify + commit**
|
|
|
|
```bash
|
|
git add bin/fxt/src/commands/model/approve.rs
|
|
git commit -m "feat(fxt): wire model approve to MLTrainingService::ApproveModel"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 10: Wire Model Reject
|
|
|
|
**Files:**
|
|
- Modify: `bin/fxt/src/commands/model/reject.rs`
|
|
|
|
**Step 1: Replace stub with real gRPC call**
|
|
|
|
Same pattern, calls `RejectModel` RPC. Import `RejectModelRequest`.
|
|
|
|
**Step 2: Verify + commit**
|
|
|
|
```bash
|
|
git add bin/fxt/src/commands/model/reject.rs
|
|
git commit -m "feat(fxt): wire model reject to MLTrainingService::RejectModel"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 11: Wire Agent Allocate-Portfolio (Replace Mock Assets)
|
|
|
|
**Files:**
|
|
- Modify: `bin/fxt/src/commands/agent.rs`
|
|
|
|
**Step 1: Replace mock assets block**
|
|
|
|
In `agent.rs`, find the mock assets block (around lines 192-212, marked with "Mock assets for testing"). Replace it with a real call to `TradingAgentService::GetSelectedAssets(GetSelectedAssetsRequest { universe_id: selection_id })`.
|
|
|
|
Map the returned `AssetScore` messages to the existing `AssetScore` type used by `AllocatePortfolioRequest`.
|
|
|
|
If the `selection_id` arg is not provided, use a default universe.
|
|
|
|
**Step 2: Verify + commit**
|
|
|
|
```bash
|
|
git add bin/fxt/src/commands/agent.rs
|
|
git commit -m "feat(fxt): wire agent allocate-portfolio to GetSelectedAssets RPC"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 12: Wire Auth Login Non-Interactive
|
|
|
|
**Files:**
|
|
- Modify: `bin/fxt/src/commands/auth.rs`
|
|
|
|
**Step 1: Replace simulated token**
|
|
|
|
In the `execute_login` function, find the block around lines 113-127 that creates a "simulated token". Replace it with a real gRPC call using the `LoginClient` that is already constructed (line ~108-110).
|
|
|
|
The interactive path already uses `login_client.interactive_login()`. For non-interactive, construct the appropriate request with username/password and call the login RPC directly.
|
|
|
|
On success, store the returned token via `auth_manager.store_tokens(...)`.
|
|
|
|
**Step 2: Verify + commit**
|
|
|
|
```bash
|
|
git add bin/fxt/src/commands/auth.rs
|
|
git commit -m "feat(fxt): wire non-interactive auth login to real gRPC"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 13: Uncomment tune_stream
|
|
|
|
**Files:**
|
|
- Modify: `bin/fxt/src/commands/mod.rs`
|
|
|
|
**Step 1: Uncomment the module**
|
|
|
|
In `bin/fxt/src/commands/mod.rs`, change:
|
|
```rust
|
|
// TODO: Enable tune_stream when API Gateway implements streaming support
|
|
// pub mod tune_stream;
|
|
```
|
|
to:
|
|
```rust
|
|
pub mod tune_stream;
|
|
```
|
|
|
|
**Step 2: Wire into tune command if not already connected**
|
|
|
|
Check `bin/fxt/src/commands/tune.rs` (or `tune/mod.rs`) to see if the `--watch` flag already dispatches to `tune_stream::watch_tuning_progress_streaming`. If not, add the dispatch.
|
|
|
|
**Step 3: Verify compilation**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo check -p fxt`
|
|
|
|
**Step 4: Commit**
|
|
|
|
```bash
|
|
git add bin/fxt/src/commands/mod.rs bin/fxt/src/commands/tune.rs
|
|
git commit -m "feat(fxt): enable tune_stream module for fxt tune --watch"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 14: Clippy + Full Test Suite
|
|
|
|
**Files:** (none — verification only)
|
|
|
|
**Step 1: Run clippy**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo clippy -p fxt -- -D warnings`
|
|
Expected: 0 errors, 0 warnings. Fix any issues.
|
|
|
|
**Step 2: Run all fxt tests**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo test -p fxt --lib`
|
|
Expected: all tests pass (134 existing + new dashboard tests)
|
|
|
|
**Step 3: Run workspace check**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo check --workspace`
|
|
Expected: compiles
|
|
|
|
**Step 4: Commit any fixes**
|
|
|
|
```bash
|
|
git commit -am "fix(fxt): clippy and test fixes"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 15: Final Server-Side Handlers (ml_training_service)
|
|
|
|
**Files:**
|
|
- Modify: `services/ml_training_service/src/service.rs`
|
|
|
|
**Step 1: Add ApproveModel handler**
|
|
|
|
In `service.rs`, add handler for `approve_model` in the `MlTrainingService` impl. Pattern: accept `ApproveModelRequest`, log the approval, return `ApproveModelResponse { success: true, message }`.
|
|
|
|
For now, the handler can update an in-memory map or return success — the actual checkpoint promotion logic can be wired later when the promotion_manager is mature. The important thing is the RPC exists and responds.
|
|
|
|
**Step 2: Add RejectModel handler**
|
|
|
|
Same pattern for `reject_model`.
|
|
|
|
**Step 3: Verify compilation**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo check -p ml_training_service`
|
|
|
|
**Step 4: Commit**
|
|
|
|
```bash
|
|
git add services/ml_training_service/src/service.rs
|
|
git commit -m "feat(ml_training_service): add ApproveModel and RejectModel handlers"
|
|
```
|