foxhunt/docs/plans/2026-03-03-fxt-overhaul-design.md

fxt 2.0 — Foxhunt Operations Platform

Date: 2026-03-03 Branch: feature/fxt-overhaul Status: Approved

Summary

Clean rewrite of fxt from a CLI tool into a full-scale monitoring and operations platform for the foxhunt HFT system. Three interfaces: CLI commands with JSON output, MCP server for LLM integration, and purpose-built TUI cockpits. Single proto set at workspace root. Monitoring service removed — API Gateway serves everything.

Design Decisions

1. Clean rewrite in the feature/fxt-overhaul worktree (greenfield)
2. Single proto directory at proto/ (workspace root) — all services and fxt reference it
3. Remove monitoring_service — API Gateway absorbs Prometheus scraping + training metrics
4. API Gateway is the single gRPC entry point — fxt talks to one URL
5. Dual output: --json flag on every command + MCP server mode (fxt mcp serve)
6. Cockpit TUI: 6 purpose-built split layouts, no customization needed
7. Color theme: Purple (#8B5CF6) / Cyan (#06B6D4) / Dark navy background

Architecture

┌────────────────────────────────────────────────────────┐
│                      fxt binary                         │
│                                                         │
│  ┌──────────┐  ┌───────────┐  ┌──────────────────────┐ │
│  │ CLI      │  │ MCP       │  │ TUI Cockpits         │ │
│  │ (clap)   │  │ Server    │  │ (ratatui)            │ │
│  │ --json   │  │ stdin/out │  │ 6 views, 1-6 keys    │ │
│  └────┬─────┘  └─────┬─────┘  └──────────┬───────────┘ │
│       │              │                    │              │
│  ┌────▼──────────────▼────────────────────▼───────────┐ │
│  │                gRPC Client Layer                    │ │
│  │  Compiled from proto/ (workspace root)              │ │
│  │  Single endpoint: api.fxhnt.ai                     │ │
│  └──────────────────────┬─────────────────────────────┘ │
└─────────────────────────┼───────────────────────────────┘
                          │ gRPC / TLS
┌─────────────────────────▼───────────────────────────────┐
│                    API Gateway                           │
│  Routes: trading · ml_training · broker_gateway          │
│          trading_agent · data_acquisition                 │
│  Serves: config · health · monitoring (absorbed)         │
│  Scrapes: Prometheus (training metrics, GPU, system)     │
└──────────────────────────────────────────────────────────┘

Proto Consolidation

Current state (scattered)

Location	Package	Files	Issue
bin/fxt/proto/	foxhunt.tli, foxhunt.config, foxhunt.ml	8 files	Fat-client versions, diverged from services
services/trading_service/proto/	trading, config, ml, monitoring, risk	5 files	Authoritative for trading service
services/monitoring_service/proto/	monitoring	1 file	Training metrics (being removed)
services/ml_training_service/proto/	ml_training	1 file	Authoritative, ahead of fxt version
services/broker_gateway_service/proto/	broker_gateway	1 file	Identical to fxt
services/trading_agent_service/proto/	trading_agent	1 file	Identical to fxt
services/data_acquisition_service/proto/	data_acquisition	1 file	Standalone
services/api_gateway/proto/	foxhunt.config	1 file	Config service

Target state (consolidated)

proto/                              # Workspace root — single source of truth
├── trading.proto                   # package: trading
├── config.proto                    # package: config
├── ml.proto                        # package: ml
├── ml_training.proto               # package: ml_training
├── monitoring.proto                # package: monitoring (training metrics + system)
├── risk.proto                      # package: risk
├── broker_gateway.proto            # package: broker_gateway
├── trading_agent.proto             # package: trading_agent
├── data_acquisition.proto          # package: data_acquisition
├── config_service.proto            # package: foxhunt.config (API GW config)
└── health.proto                    # package: health

Every build.rs references ../../proto/ or ../proto/ relative to crate.

What happens to fxt fat-client protos

Deleted. The fat-client trading.proto (package foxhunt.tli, 895 lines) had a unified API design that merged trading/risk/ML/config into one service. This is replaced by fxt calling the decomposed service protos directly through the API Gateway.

The API Gateway continues to serve the foxhunt.tli package for backward compatibility with any existing clients, but fxt itself uses the service-native protos.

Command Tree

fxt
├── auth            # Authentication
│   ├── login       # Authenticate with API
│   ├── logout      # Clear credentials
│   └── status      # Check auth state
│
├── trade           # Order management
│   ├── submit      # Place order
│   ├── cancel      # Cancel order
│   ├── positions   # List positions
│   ├── orders      # List active orders
│   └── account     # Account state (balance, margin, P&L)
│
├── train           # ML training lifecycle
│   ├── start       # Start training job
│   ├── stop        # Stop training job
│   ├── status      # Job status + metrics
│   ├── list        # List all jobs
│   └── logs        # Stream training logs/metrics
│
├── tune            # Hyperparameter optimization
│   ├── start       # Start tuning (single or batch)
│   ├── stop        # Stop tuning job
│   ├── status      # Tuning progress
│   └── approve     # Approve/reject model promotion
│
├── model           # Model management
│   ├── list        # List available models
│   ├── status      # Model health/performance
│   ├── predict     # Get prediction
│   ├── ensemble    # Ensemble vote
│   └── promote     # Model promotion pipeline
│
├── agent           # Trading agent control
│   ├── start       # Start trading agent
│   ├── stop        # Stop trading agent
│   ├── status      # Agent state
│   └── config      # Agent configuration
│
├── backtest        # Backtesting
│   ├── run         # Run backtest
│   ├── status      # Backtest progress
│   └── results     # Get results
│
├── broker          # Broker connectivity
│   ├── status      # Session status (FIX, heartbeat RTT)
│   ├── connect     # Establish session
│   └── executions  # Stream executions
│
├── data            # Data pipeline management (NEW)
│   ├── download    # Download market data (Databento)
│   ├── status      # Feed status, data freshness
│   ├── cache       # Cache management
│   └── feeds       # Active feed list
│
├── service         # Service operations (NEW)
│   ├── list        # All services + health
│   ├── status      # Detailed service status
│   ├── logs        # Stream service logs
│   ├── restart     # Restart service (rollout)
│   ├── deploy      # Deploy binary (pod-writer)
│   └── health      # Health check all services
│
├── cluster         # Cluster operations (NEW)
│   ├── status      # Node/pod status
│   ├── resources   # CPU/RAM/GPU utilization
│   └── events      # K8s events
│
├── risk            # Risk management (NEW)
│   ├── status      # Kill switch state, circuit breakers
│   ├── limits      # Current risk limits
│   ├── drawdown    # Drawdown stats
│   ├── stress      # Run stress test
│   └── emergency   # Emergency controls (halt/resume)
│
├── config          # System configuration
│   ├── get         # Get config values
│   ├── set         # Set config values
│   ├── export      # Export full config
│   └── env         # Active environment
│
├── watch           # TUI cockpits (REDESIGNED)
│   (default = overview cockpit, 1-6 to switch)
│
└── mcp             # MCP server mode (NEW)
    └── serve       # Start MCP server (stdin/stdout JSON-RPC)

JSON Output

Every command supports --json / -j flag:

# Human-readable (default)
fxt service list
# → api-gateway      ● HEALTHY   12ms   4d 12h

# JSON for CI/LLM
fxt service list --json
# → [{"name":"api-gateway","status":"healthy","latency_ms":12,"uptime":"4d 12h"}]

TUI Cockpits

Six purpose-built cockpit views. Switch with number keys 1-6.

Color Theme (from logo)

Role	Color	Hex
Background	Dark navy	#0F0F23
Surface	Slightly lighter	#1A1A2E
Primary text	Light gray	#E0E0E0
Primary accent	Purple	#8B5CF6
Secondary accent	Cyan	#06B6D4
Success	Green	#10B981
Warning	Amber	#F59E0B
Error	Red	#EF4444
Muted text	Gray	#6B7280
Border	Purple dim	#4C1D95
Highlight	Bright cyan	#22D3EE

Cockpit 1 — Overview (default)

Four quadrants: Services | Resources | Training | Portfolio. Quick glance at entire system health.

Cockpit 2 — Training

Active sessions table + GPU panel + CPU/RAM panel + Loss sparkline + Epoch financial metrics. Designed for monitoring active training jobs.

Cockpit 3 — Trading

Positions table + Recent executions + Account summary + Broker session health. Designed for active trading operations.

Cockpit 4 — Services

Service health grid (all 7-8 services) + K8s events + Cluster resources. Designed for ops/SRE workflow.

Cockpit 5 — Risk

Kill switch status + Drawdown monitor + Circuit breakers + Position limits + Stress test results. Designed for risk management oversight.

Cockpit 6 — Data

Active feeds + Data cache stats + Pipeline status. Designed for data operations.

Navigation

Key	Action
1-6	Switch cockpit
q	Quit
?	Help overlay
r	Force refresh
/	Filter (contextual)

MCP Server Mode

fxt mcp serve starts an MCP server on stdin/stdout (JSON-RPC 2.0).

Every CLI command maps to an MCP tool. Tool names: fxt_{command}_{subcommand}. All tools return structured JSON.

Example tools:

• fxt_service_list — List all services with health
• fxt_service_status — Detailed service status
• fxt_train_start — Start training job
• fxt_trade_positions — Get positions
• fxt_risk_status — Risk system state
• fxt_cluster_resources — CPU/RAM/GPU

This allows Claude Code (or any MCP client) to query the entire foxhunt system programmatically.

Monitoring Service Removal

What it does today

• Prometheus-to-gRPC bridge (97 lines main.rs)
• Scrapes Prometheus for training metrics, GPU stats, system resources
• Serves 3 RPCs: GetLiveTrainingMetrics, StreamTrainingMetrics, GetEpochHistory

What replaces it

• API Gateway absorbs the Prometheus scraping logic
• The monitoring proto RPCs move to API Gateway's service layer
• fxt calls the same RPCs, just routed differently

Cleanup

• Delete services/monitoring_service/ directory
• Remove from Cargo.toml workspace members
• Remove nginx path routing for /monitoring.MonitoringService/
• Remove K8s deployment manifest
• Update CI pipeline (one fewer binary to compile)

Metrics Inventory (available via API)

Training (45+ fields per session)

epoch, loss, val_loss, sharpe, sortino, win_rate, max_drawdown, profit_factor, total_return, avg_return, total_trades, action distribution, q_value, policy_entropy, kl_divergence, gradient_norm, learning_rate, batches/s, GPU util/mem/temp/power, CPU%, RAM, checkpoint stats, NaN/explosion counts, hyperopt progress

Trading

account (equity, cash, margin, P&L, buying power, leverage), positions (symbol, qty, entry, market value, unrealized P&L), executions (fill price/qty, cum qty, avg price, timestamps), session (FIX state, heartbeat RTT, sequence numbers)

Risk

kill switch state (global/portfolio/strategy/instrument), drawdown (current%, max%, HWM, days), circuit breakers (daily loss, max DD, concentration), position limits (max position/order, leverage), emergency response, enforcement actions, recovery mode

System

service health (per-service latency, restarts, uptime), cluster resources (node CPU/RAM/GPU/disk), K8s events, data feeds (records/s, latency, status)

ML Models

model status, predictions, ensemble votes, feature importance, training job state, tuning progress, promotion pipeline