foxhunt/docs/plans/2026-03-04-tui-live-data.md

TUI Live Data Wiring — Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Replace all hardcoded mock data in fxt watch with real gRPC streaming data from the API Gateway, using poll-to-stream adapters in the gateway for RPCs that lack native streaming.

Architecture: 9 new streaming RPCs added to protos, implemented as poll-to-stream adapters in the API Gateway proxies. 3 existing unimplemented monitoring streams get real implementations. TUI gets a DataFetcher module that opens all streams and sends StateUpdate enum variants through an mpsc channel to the event loop, which owns AppState exclusively.

Tech Stack: Rust, tonic 0.14, async-stream, tokio-stream, ratatui 0.29, crossterm 0.28

---

Stream Inventory

New streaming RPCs (to be added to protos + gateway adapters):

#	Proto	New RPC	Wraps	Interval	Cockpit
1	broker_gateway.proto	StreamAccountState	GetAccountState	3s	Trading
2	broker_gateway.proto	StreamSessionStatus	GetSessionStatus	5s	Trading
3	risk.proto	StreamCircuitBreakerStatus	GetCircuitBreakerStatus	2s	Risk
4	risk.proto	StreamRiskMetrics	GetRiskMetrics	3s	Risk
5	data_acquisition.proto	StreamDownloadStatus	ListDownloadJobs	5s	Data
6	ml.proto	StreamModelStatus	GetModelStatus	5s	Overview
7	trading_agent.proto	StreamAgentStatus	GetAgentStatus	3s	Overview
8	trading.proto	StreamPortfolioSummary	GetPortfolioSummary	3s	Trading
9	trading.proto	StreamOrderBook	GetOrderBook	1s	Trading

Existing unimplemented monitoring streams (implement in monitoring_handler.rs):

#	RPC	Status	Cockpit
10	monitoring.StreamSystemStatus	Proto exists, returns Status::unimplemented	Services
11	monitoring.StreamMetrics	Proto exists, returns Status::unimplemented	Services
12	monitoring.StreamAlerts	Proto exists, returns Status::unimplemented	Services

Existing working streams (already proxied, just need TUI clients):

Proto	Stream	Cockpit
trading.proto	StreamPositions	Trading
trading.proto	StreamOrders	Trading
trading.proto	StreamExecutions	Trading
trading.proto	StreamMarketData	Data
risk.proto	StreamVaRUpdates	Risk
risk.proto	StreamRiskAlerts	Risk
monitoring.proto	StreamTrainingMetrics	Training
ml.proto	StreamModelMetrics	Training
ml.proto	StreamSignalStrength	Overview
trading_agent.proto	StreamAgentActivity	Overview

---

Task 1: Add 9 new streaming RPCs to proto files

Files:

• Modify: proto/broker_gateway.proto
• Modify: proto/risk.proto
• Modify: proto/data_acquisition.proto
• Modify: proto/ml.proto
• Modify: proto/trading_agent.proto
• Modify: proto/trading.proto

What: Add rpc Stream* methods to existing service definitions. Each new stream reuses existing request/response types (no new messages needed). Use empty request messages for streams that don't need parameters.

Changes per proto:

broker_gateway.proto — add after HealthCheck RPC:

  // Server-streaming: polls GetAccountState at gateway level
  rpc StreamAccountState(StreamAccountStateRequest) returns (stream GetAccountStateResponse);
  // Server-streaming: polls GetSessionStatus at gateway level
  rpc StreamSessionStatus(StreamSessionStatusRequest) returns (stream GetSessionStatusResponse);
}

message StreamAccountStateRequest {
  uint32 interval_seconds = 1; // 0 = server default (3s)
}

message StreamSessionStatusRequest {
  uint32 interval_seconds = 1; // 0 = server default (5s)
}

risk.proto — add after GetCircuitBreakerStatus RPC:

  // Server-streaming: polls GetCircuitBreakerStatus at gateway level
  rpc StreamCircuitBreakerStatus(StreamCircuitBreakerStatusRequest) returns (stream GetCircuitBreakerStatusResponse);
  // Server-streaming: polls GetRiskMetrics at gateway level
  rpc StreamRiskMetrics(StreamRiskMetricsRequest) returns (stream GetRiskMetricsResponse);
}

message StreamCircuitBreakerStatusRequest {
  uint32 interval_seconds = 1; // 0 = server default (2s)
}

message StreamRiskMetricsRequest {
  uint32 interval_seconds = 1; // 0 = server default (3s)
}

data_acquisition.proto — add after HealthCheck RPC:

  // Server-streaming: polls ListDownloadJobs at gateway level
  rpc StreamDownloadStatus(StreamDownloadStatusRequest) returns (stream ListDownloadJobsResponse);
}

message StreamDownloadStatusRequest {
  uint32 interval_seconds = 1; // 0 = server default (5s)
}

ml.proto — add after StreamSignalStrength RPC:

  // Server-streaming: polls GetModelStatus at gateway level
  rpc StreamModelStatus(StreamModelStatusRequest) returns (stream GetModelStatusResponse);
}

message StreamModelStatusRequest {
  string model_name = 1;       // empty = all models
  uint32 interval_seconds = 2; // 0 = server default (5s)
}

trading_agent.proto — add after HealthCheck RPC:

  // Server-streaming: polls GetAgentStatus at gateway level
  rpc StreamAgentStatus(StreamAgentStatusRequest) returns (stream GetAgentStatusResponse);
}

message StreamAgentStatusRequest {
  uint32 interval_seconds = 1; // 0 = server default (3s)
}

trading.proto — add after GetRegimeTransitions RPC:

  // Server-streaming: polls GetPortfolioSummary at gateway level
  rpc StreamPortfolioSummary(StreamPortfolioSummaryRequest) returns (stream GetPortfolioSummaryResponse);
  // Server-streaming: polls GetOrderBook at gateway level
  rpc StreamOrderBook(StreamOrderBookRequest) returns (stream GetOrderBookResponse);
}

message StreamPortfolioSummaryRequest {
  uint32 interval_seconds = 1; // 0 = server default (3s)
}

message StreamOrderBookRequest {
  string symbol = 1;
  int32 depth = 2;
  uint32 interval_seconds = 3; // 0 = server default (1s)
}

Verify: SQLX_OFFLINE=true cargo check -p fxt 2>&1 | head -5 — new proto types should compile but gateway will have errors (expected — server trait now requires new methods).

Commit: feat(proto): add 9 streaming RPCs for TUI live data

---

Task 2: Gateway stream adapters — broker_gateway + data_acquisition

Files:

• Modify: services/api_gateway/src/grpc/broker_gateway_proxy.rs
• Modify: services/api_gateway/src/grpc/data_acquisition_proxy.rs

What: Add poll-to-stream adapter methods to existing proxy structs. Each adapter spawns an async_stream::stream! loop that polls the backend unary RPC and yields responses.

Pattern (used by all gateway stream adapters):

use std::time::Duration;

// In the impl block, add stream type alias:
type StreamAccountStateStream =
    Pin<Box<dyn Stream<Item = Result<GetAccountStateResponse, Status>> + Send>>;

// Then the method:
async fn stream_account_state(
    &self,
    request: Request<StreamAccountStateRequest>,
) -> Result<Response<Self::StreamAccountStateStream>, Status> {
    let req = request.into_inner();
    let interval_secs = if req.interval_seconds == 0 { 3 } else { req.interval_seconds.clamp(1, 60) };
    let mut client = self.client.clone();

    let stream = async_stream::stream! {
        let mut interval = tokio::time::interval(Duration::from_secs(u64::from(interval_secs)));
        loop {
            interval.tick().await;
            match client.get_account_state(Request::new(GetAccountStateRequest {})).await {
                Ok(resp) => yield Ok(resp.into_inner()),
                Err(e) => {
                    error!("Backend GetAccountState failed: {}", e);
                    yield Err(e);
                    break;
                }
            }
        }
    };

    Ok(Response::new(Box::pin(stream)))
}

broker_gateway_proxy.rs — add 2 methods:

• stream_account_state (default interval 3s, polls get_account_state)
• stream_session_status (default interval 5s, polls get_session_status)
• Add use std::time::Duration; at top

data_acquisition_proxy.rs — add 1 method:

• stream_download_status (default interval 5s, polls list_download_jobs with empty request)
• Add use std::time::Duration; at top

Verify: SQLX_OFFLINE=true cargo check -p api_gateway 2>&1 | grep "error\[" | head -10 — broker_gateway and data_acquisition errors should be gone. Other proxy errors expected.

Commit: feat(api_gateway): add broker_gateway + data_acquisition stream adapters

---

Task 3: Gateway stream adapters — risk + ml

Files:

• Modify: services/api_gateway/src/grpc/risk_proxy.rs
• Modify: services/api_gateway/src/grpc/ml_proxy.rs

risk_proxy.rs — add 2 methods:

• stream_circuit_breaker_status (default interval 2s, polls get_circuit_breaker_status)
• stream_risk_metrics (default interval 3s, polls get_risk_metrics)
• Import: use std::time::Duration;

ml_proxy.rs — add 1 method:

• stream_model_status (default interval 5s, polls get_model_status)
• Forward model_name from the stream request to each GetModelStatusRequest
• Import: use std::time::Duration;

Verify: SQLX_OFFLINE=true cargo check -p api_gateway 2>&1 | grep "error\[" | head -10

Commit: feat(api_gateway): add risk + ml stream adapters

---

Task 4: Gateway stream adapters — trading + trading_agent

Files:

• Modify: services/api_gateway/src/grpc/trading_direct_proxy.rs
• Modify: services/api_gateway/src/grpc/trading_agent_proxy.rs

trading_direct_proxy.rs — add 2 methods:

• stream_portfolio_summary (default interval 3s, polls get_portfolio_summary)
• stream_order_book (default interval 1s, polls get_order_book, forwards symbol + depth)
• Import: use std::time::Duration;

trading_agent_proxy.rs — add 1 method:

• stream_agent_status (default interval 3s, polls get_agent_status)
• Import: use std::time::Duration;

Verify: SQLX_OFFLINE=true cargo check -p api_gateway 2>&1 | grep "error\[" | head -10

Commit: feat(api_gateway): add trading + trading_agent stream adapters

---

Task 5: Implement monitoring handler streams

Files:

• Modify: services/api_gateway/src/grpc/monitoring_handler.rs

What: Replace 3 Status::unimplemented stubs with real implementations. Follow the existing stream_training_metrics pattern (clone internal state, async_stream::stream! loop).

stream_system_status (replaces stub at ~line 497):

• Clone self.backends, self.start_time
• In stream loop: fan-out health checks (same logic as get_system_status), yield SystemStatusEvent wrapping the GetSystemStatusResponse fields
• Default interval from request or 5s

stream_metrics (replaces stub at ~line 609):

• Clone self.prom
• In stream loop: query configured metric names from Prometheus, yield MetricsEvent wrapping metric list
• Default interval from request or 5s
• Use fixed metric list: ["cpu_usage", "memory_usage", "gpu_utilization"]

stream_alerts (replaces stub at ~line 639):

• This is a placeholder — Prometheus has an alertmanager API but the handler doesn't currently scrape alerts
• Implement as a stream that yields an empty AlertEvent each interval (no-data is better than unimplemented)
• Default interval from request or 10s

Verify: SQLX_OFFLINE=true cargo check -p api_gateway — should compile with 0 errors now

Commit: feat(api_gateway): implement monitoring stream RPCs

---

Task 6: Verify full gateway build

Verify:

SQLX_OFFLINE=true cargo check -p api_gateway
SQLX_OFFLINE=true cargo test -p api_gateway --lib
SQLX_OFFLINE=true cargo clippy -p api_gateway -- -D warnings

All must pass: 0 errors, 0 clippy warnings.

Commit (if any fixups needed): fix(api_gateway): clippy and test fixes for stream adapters

---

Task 7: TUI state.rs — ConnectionStatus, empty defaults, new data types

Files:

• Modify: bin/fxt/src/tui/state.rs

What:

1. Add ConnectionStatus enum
2. Add missing data types needed by streams (OrderData, VaRData, AlertData, etc.)
3. Change AppState::default() to start with empty vecs (not mock data)
4. Add connection_status: ConnectionStatus field to AppState

Add to top of state.rs:

/// Connection state for the gRPC streaming layer.
#[derive(Debug, Clone)]
pub enum ConnectionStatus {
    Connecting,
    Connected,
    Reconnecting { attempt: u32 },
    Error(String),
}

impl Default for ConnectionStatus {
    fn default() -> Self { Self::Connecting }
}

Replace impl Default for AppState — all vecs become Vec::new(), account defaults to zeros, risk defaults to zeros. Add connection_status: ConnectionStatus::default().

Add any new sub-struct types needed for streams not covered by existing types (e.g., AlertData, VaRData if not already present). Keep minimal — only fields the cockpits actually render.

Verify: SQLX_OFFLINE=true cargo check -p fxt

Commit: refactor(fxt): empty TUI state defaults + ConnectionStatus

---

Task 8: Create TUI data_fetcher.rs

Files:

• Create: bin/fxt/src/tui/data_fetcher.rs
• Modify: bin/fxt/src/tui/mod.rs (add pub mod data_fetcher;)

What: Core streaming data layer. StateUpdate enum + DataFetcher struct.

StateUpdate enum — one variant per data source, carrying the parsed TUI-layer types:

pub enum StateUpdate {
    // Trading cockpit
    Positions(Vec<PositionData>),
    Executions(Vec<ExecutionData>),
    AccountState(AccountData),
    PortfolioSummary(/* fields */),
    SessionStatus(/* fields */),
    // Training cockpit
    TrainingSessions(Vec<TrainingSessionData>),
    Gpu(GpuData),
    // Services cockpit
    Services(Vec<ServiceData>),
    SystemResources(SystemResources),
    ClusterEvents(Vec<ClusterEventData>),
    // Risk cockpit
    CircuitBreakers(Vec<CircuitBreakerData>),
    PositionLimits(Vec<PositionLimitData>),
    RiskMetrics(/* drawdown, VaR */),
    // Data cockpit
    DataFeeds(Vec<DataFeedData>),
    DownloadJobs(Vec</* download job data */>),
    // Meta
    Connection(ConnectionStatus),
}

DataFetcher struct:

pub struct DataFetcher {
    tx: mpsc::UnboundedSender<StateUpdate>,
    cancel: tokio_util::sync::CancellationToken,
}

impl DataFetcher {
    pub fn new(
        client: &FoxhuntClient,
        tx: mpsc::UnboundedSender<StateUpdate>,
    ) -> Self { ... }

    /// Spawn all stream tasks. Returns immediately.
    pub fn start(&self, client: &FoxhuntClient) { ... }

    /// Cancel all streams gracefully.
    pub fn stop(&self) { self.cancel.cancel(); }
}

Each stream task follows this pattern:

fn spawn_positions_stream(client: &FoxhuntClient, tx: UnboundedSender<StateUpdate>, cancel: CancellationToken) {
    let mut trading = client.trading();
    tokio::spawn(async move {
        let mut backoff = Duration::from_secs(1);
        loop {
            match trading.stream_positions(Request::new(StreamPositionsRequest {})).await {
                Ok(response) => {
                    backoff = Duration::from_secs(1); // reset on success
                    let mut stream = response.into_inner();
                    loop {
                        tokio::select! {
                            _ = cancel.cancelled() => return,
                            msg = stream.message() => {
                                match msg {
                                    Ok(Some(event)) => {
                                        let positions = convert_positions(event);
                                        let _ = tx.send(StateUpdate::Positions(positions));
                                    }
                                    Ok(None) => break, // stream ended, reconnect
                                    Err(e) => {
                                        let _ = tx.send(StateUpdate::Connection(
                                            ConnectionStatus::Error(format!("positions: {e}"))
                                        ));
                                        break;
                                    }
                                }
                            }
                        }
                    }
                }
                Err(e) => {
                    let _ = tx.send(StateUpdate::Connection(
                        ConnectionStatus::Reconnecting { attempt: backoff.as_secs() as u32 }
                    ));
                }
            }
            // Exponential backoff before reconnect
            tokio::select! {
                _ = cancel.cancelled() => return,
                _ = tokio::time::sleep(backoff) => {}
            }
            backoff = (backoff * 2).min(Duration::from_secs(30));
        }
    });
}

Conversion functions — map proto types to TUI state types. One per stream. Keep simple: extract fields, convert enums to strings, format numbers.

Spawn order in start(): monitoring first (service health), then trading, risk, training, data, agent — so the user sees connection status early.

Dependencies: Add tokio-util (for CancellationToken) to bin/fxt/Cargo.toml if not already present.

Verify: SQLX_OFFLINE=true cargo check -p fxt

Commit: feat(fxt): add TUI DataFetcher with streaming gRPC clients

---

Task 9: Wire event_loop.rs + watch.rs

Files:

• Modify: bin/fxt/src/tui/event_loop.rs
• Modify: bin/fxt/src/commands/watch.rs

event_loop.rs changes:

1. Change run signature:

pub async fn run(api_url: &str) -> anyhow::Result<()> {

(remove the _ prefix from api_url)

2. Inside run(), before event_loop():

// Connect to API Gateway
let client = FoxhuntClient::connect(api_url).await?;

// Auto-refresh auth token
let storage = FileTokenStorage::new()?;
let manager = AuthTokenManager::new(storage);
if manager.needs_refresh().await {
    let login_client = LoginClient::new(client.channel());
    if login_client.silent_login(&manager).await? {
        eprintln!("Token refreshed.");
    }
}

// Start streaming data layer
let (tx, rx) = tokio::sync::mpsc::unbounded_channel();
let fetcher = DataFetcher::new(&client, tx);
fetcher.start(&client);

3. Pass rx to event_loop().

4. Inside event_loop(), at the start of each loop iteration, drain the channel:

// Drain all pending state updates
while let Ok(update) = rx.try_recv() {
    apply_update(&mut state, update);
}

5. Add apply_update function that matches on StateUpdate variants and updates AppState fields.

6. On 'r' keypress (refresh), cancel and restart the fetcher.

watch.rs — no changes needed (already passes api_url to run()).

Verify: SQLX_OFFLINE=true cargo check -p fxt

Commit: feat(fxt): wire DataFetcher into TUI event loop

---

Task 10: Wire Overview + Training cockpits

Files:

• Modify: bin/fxt/src/tui/cockpits/overview.rs
• Modify: bin/fxt/src/tui/cockpits/training.rs

What: Handle empty/connecting state in rendering. When data hasn't arrived yet, show "Connecting..." instead of empty tables.

Overview cockpit changes:

• render_services: If state.services.is_empty(), show centered "Connecting..."
• render_resources: If all zero, show "Waiting for data..."
• render_training_summary: If state.training_sessions.is_empty(), show "No active training"
• render_portfolio: If account equity is 0.0, show "Connecting..."

Training cockpit changes:

• render_sessions_table: If state.training_sessions.is_empty(), show "No active training sessions"
• render_gpu_panel: If state.gpu.name == "N/A", show "No GPU data"
• render_epoch_metrics: Similar empty-state handling

Verify: SQLX_OFFLINE=true cargo check -p fxt

Commit: feat(fxt): wire Overview + Training cockpits to live data

---

Task 11: Wire Trading cockpit

Files:

• Modify: bin/fxt/src/tui/cockpits/trading.rs

What: Handle empty state for all 4 panels (positions, executions, account, broker health).

• render_positions: Empty → "No open positions"
• render_executions: Empty → "No recent executions"
• render_account: All zeros → "Connecting..."
• render_broker_health: Currently hardcoded. Wire to state.session_status or similar field. If no data → "Connecting..."

Verify: SQLX_OFFLINE=true cargo check -p fxt

Commit: feat(fxt): wire Trading cockpit to live data

---

Task 12: Wire Services + Risk cockpits

Files:

• Modify: bin/fxt/src/tui/cockpits/services.rs
• Modify: bin/fxt/src/tui/cockpits/risk.rs

Services cockpit:

• render_service_grid: Empty → "Connecting..."
• render_cluster_events: Empty → "No events"
• render_cluster_resources: All zeros → "Waiting for metrics..."

Risk cockpit:

• render_kill_switches: Handle default false values (already works)
• render_drawdown: All zeros → "No drawdown data"
• render_circuit_breakers: Empty → "No circuit breakers"
• render_position_limits: Empty → "No position limits"

Verify: SQLX_OFFLINE=true cargo check -p fxt

Commit: feat(fxt): wire Services + Risk cockpits to live data

---

Task 13: Wire Data cockpit

Files:

• Modify: bin/fxt/src/tui/cockpits/data.rs

What:

• render_feeds: If state.data_feeds.is_empty(), show "No data feeds" (market data stream may not be active if no symbols subscribed)
• render_cache_stats: All zeros → "No cache data"
• render_pipeline: Wire to download job status from DataFetcher. Show actual job states (PENDING, DOWNLOADING, etc.) instead of hardcoded "IDLE"/"RUNNING"

Verify: SQLX_OFFLINE=true cargo check -p fxt

Commit: feat(fxt): wire Data cockpit to live data

---

Task 14: Add connection status to status bar

Files:

• Modify: bin/fxt/src/tui/event_loop.rs

What: Update render_status_bar to show connection status from state.connection_status:

• Connecting → yellow "Connecting..."
• Connected → green "Connected"
• Reconnecting { attempt } → yellow "Reconnecting (attempt N)..."
• Error(msg) → red "Error: msg"

Place between the keymap hints and the "Updated: X ago" text.

Verify: SQLX_OFFLINE=true cargo check -p fxt

Commit: feat(fxt): show connection status in TUI status bar

---

Task 15: Full verification — tests + clippy

Verify:

SQLX_OFFLINE=true cargo test -p fxt --lib
SQLX_OFFLINE=true cargo test -p api_gateway --lib
SQLX_OFFLINE=true cargo clippy -p fxt -p api_gateway -- -D warnings

Fix any issues. All must pass with 0 errors, 0 warnings.

Commit: fix(fxt): test and clippy fixes for TUI live data

---

Task 16: Build + deploy api_gateway

Steps:

1. SQLX_OFFLINE=true cargo build --release -p api_gateway
2. Deploy using pod-writer-deploy skill (see /.claude/skills/pod-writer-deploy/)
3. Verify with: fxt service health

Commit: none (deployment only)

---

Task 17: Build fxt + test in tmux

Steps:

1. SQLX_OFFLINE=true cargo build --release -p fxt
2. cp target/release/fxt ~/.cargo/bin/fxt
3. fxt auth login --username admin (or use FXT_PASSWORD env var)
4. Launch: fxt watch --api-url https://api.fxhnt.ai
5. Test all 6 views (keys 1-6):
   • Overview: Should show real service health, real portfolio if authenticated
   • Training: Real training sessions if any running, real GPU data
   • Trading: Real positions from IBKR/broker-gateway, real executions
   • Services: Real health check results from all backends
   • Risk: Real circuit breaker status, VaR data if risk service running
   • Data: Real download job status, market data feed status
6. Test reconnection: temporarily disconnect, verify TUI shows "Reconnecting..."
7. Test help overlay (?), quit (q)

Verification checklist:

• All 6 views render without panic
• Connected status shows in status bar
• Data updates in real-time (positions change, metrics refresh)
• Empty/connecting state shown gracefully when backend data unavailable
• 'r' key forces reconnection
• Quit works cleanly (no terminal corruption)