# fxt 2.0 — Foxhunt Operations Platform

**Date**: 2026-03-03
**Branch**: `feature/fxt-overhaul`
**Status**: Approved

## Summary

Clean rewrite of `fxt` from a CLI tool into a full-scale monitoring and operations platform for the foxhunt HFT system. Three interfaces: CLI commands with JSON output, MCP server for LLM integration, and purpose-built TUI cockpits. Single proto set at workspace root. Monitoring service removed — API Gateway serves everything.

## Design Decisions

1. **Clean rewrite** in the `feature/fxt-overhaul` worktree (greenfield)
2. **Single proto directory** at `proto/` (workspace root) — all services and fxt reference it
3. **Remove monitoring_service** — API Gateway absorbs Prometheus scraping + training metrics
4. **API Gateway is the single gRPC entry point** — fxt talks to one URL
5. **Dual output**: `--json` flag on every command + MCP server mode (`fxt mcp serve`)
6. **Cockpit TUI**: 6 purpose-built split layouts, no customization needed
7. **Color theme**: Purple (#8B5CF6) / Cyan (#06B6D4) / Dark navy background

## Architecture

```
┌────────────────────────────────────────────────────────┐
│                      fxt binary                         │
│                                                         │
│  ┌──────────┐  ┌───────────┐  ┌──────────────────────┐ │
│  │ CLI      │  │ MCP       │  │ TUI Cockpits         │ │
│  │ (clap)   │  │ Server    │  │ (ratatui)            │ │
│  │ --json   │  │ stdin/out │  │ 6 views, 1-6 keys    │ │
│  └────┬─────┘  └─────┬─────┘  └──────────┬───────────┘ │
│       │              │                    │              │
│  ┌────▼──────────────▼────────────────────▼───────────┐ │
│  │                gRPC Client Layer                    │ │
│  │  Compiled from proto/ (workspace root)              │ │
│  │  Single endpoint: api.fxhnt.ai                     │ │
│  └──────────────────────┬─────────────────────────────┘ │
└─────────────────────────┼───────────────────────────────┘
                          │ gRPC / TLS
┌─────────────────────────▼───────────────────────────────┐
│                    API Gateway                           │
│  Routes: trading · ml_training · broker_gateway          │
│          trading_agent · data_acquisition                 │
│  Serves: config · health · monitoring (absorbed)         │
│  Scrapes: Prometheus (training metrics, GPU, system)     │
└──────────────────────────────────────────────────────────┘
```

## Proto Consolidation

### Current state (scattered)

| Location | Package | Files | Issue |
|----------|---------|-------|-------|
| `bin/fxt/proto/` | `foxhunt.tli`, `foxhunt.config`, `foxhunt.ml` | 8 files | Fat-client versions, diverged from services |
| `services/trading_service/proto/` | `trading`, `config`, `ml`, `monitoring`, `risk` | 5 files | Authoritative for trading service |
| `services/monitoring_service/proto/` | `monitoring` | 1 file | Training metrics (being removed) |
| `services/ml_training_service/proto/` | `ml_training` | 1 file | Authoritative, ahead of fxt version |
| `services/broker_gateway_service/proto/` | `broker_gateway` | 1 file | Identical to fxt |
| `services/trading_agent_service/proto/` | `trading_agent` | 1 file | Identical to fxt |
| `services/data_acquisition_service/proto/` | `data_acquisition` | 1 file | Standalone |
| `services/api_gateway/proto/` | `foxhunt.config` | 1 file | Config service |

### Target state (consolidated)

```
proto/                              # Workspace root — single source of truth
├── trading.proto                   # package: trading
├── config.proto                    # package: config
├── ml.proto                        # package: ml
├── ml_training.proto               # package: ml_training
├── monitoring.proto                # package: monitoring (training metrics + system)
├── risk.proto                      # package: risk
├── broker_gateway.proto            # package: broker_gateway
├── trading_agent.proto             # package: trading_agent
├── data_acquisition.proto          # package: data_acquisition
├── config_service.proto            # package: foxhunt.config (API GW config)
└── health.proto                    # package: health
```

Every `build.rs` references `../../proto/` or `../proto/` relative to crate.

### What happens to fxt fat-client protos

Deleted. The fat-client `trading.proto` (package `foxhunt.tli`, 895 lines) had a unified API design that merged trading/risk/ML/config into one service. This is replaced by fxt calling the decomposed service protos directly through the API Gateway.

The API Gateway continues to serve the `foxhunt.tli` package for backward compatibility with any existing clients, but fxt itself uses the service-native protos.

## Command Tree

```
fxt
├── auth            # Authentication
│   ├── login       # Authenticate with API
│   ├── logout      # Clear credentials
│   └── status      # Check auth state
│
├── trade           # Order management
│   ├── submit      # Place order
│   ├── cancel      # Cancel order
│   ├── positions   # List positions
│   ├── orders      # List active orders
│   └── account     # Account state (balance, margin, P&L)
│
├── train           # ML training lifecycle
│   ├── start       # Start training job
│   ├── stop        # Stop training job
│   ├── status      # Job status + metrics
│   ├── list        # List all jobs
│   └── logs        # Stream training logs/metrics
│
├── tune            # Hyperparameter optimization
│   ├── start       # Start tuning (single or batch)
│   ├── stop        # Stop tuning job
│   ├── status      # Tuning progress
│   └── approve     # Approve/reject model promotion
│
├── model           # Model management
│   ├── list        # List available models
│   ├── status      # Model health/performance
│   ├── predict     # Get prediction
│   ├── ensemble    # Ensemble vote
│   └── promote     # Model promotion pipeline
│
├── agent           # Trading agent control
│   ├── start       # Start trading agent
│   ├── stop        # Stop trading agent
│   ├── status      # Agent state
│   └── config      # Agent configuration
│
├── backtest        # Backtesting
│   ├── run         # Run backtest
│   ├── status      # Backtest progress
│   └── results     # Get results
│
├── broker          # Broker connectivity
│   ├── status      # Session status (FIX, heartbeat RTT)
│   ├── connect     # Establish session
│   └── executions  # Stream executions
│
├── data            # Data pipeline management (NEW)
│   ├── download    # Download market data (Databento)
│   ├── status      # Feed status, data freshness
│   ├── cache       # Cache management
│   └── feeds       # Active feed list
│
├── service         # Service operations (NEW)
│   ├── list        # All services + health
│   ├── status      # Detailed service status
│   ├── logs        # Stream service logs
│   ├── restart     # Restart service (rollout)
│   ├── deploy      # Deploy binary (pod-writer)
│   └── health      # Health check all services
│
├── cluster         # Cluster operations (NEW)
│   ├── status      # Node/pod status
│   ├── resources   # CPU/RAM/GPU utilization
│   └── events      # K8s events
│
├── risk            # Risk management (NEW)
│   ├── status      # Kill switch state, circuit breakers
│   ├── limits      # Current risk limits
│   ├── drawdown    # Drawdown stats
│   ├── stress      # Run stress test
│   └── emergency   # Emergency controls (halt/resume)
│
├── config          # System configuration
│   ├── get         # Get config values
│   ├── set         # Set config values
│   ├── export      # Export full config
│   └── env         # Active environment
│
├── watch           # TUI cockpits (REDESIGNED)
│   (default = overview cockpit, 1-6 to switch)
│
└── mcp             # MCP server mode (NEW)
    └── serve       # Start MCP server (stdin/stdout JSON-RPC)
```

### JSON Output

Every command supports `--json` / `-j` flag:

```bash
# Human-readable (default)
fxt service list
# → api-gateway      ● HEALTHY   12ms   4d 12h

# JSON for CI/LLM
fxt service list --json
# → [{"name":"api-gateway","status":"healthy","latency_ms":12,"uptime":"4d 12h"}]
```

## TUI Cockpits

Six purpose-built cockpit views. Switch with number keys 1-6.

### Color Theme (from logo)

| Role | Color | Hex |
|------|-------|-----|
| Background | Dark navy | #0F0F23 |
| Surface | Slightly lighter | #1A1A2E |
| Primary text | Light gray | #E0E0E0 |
| Primary accent | Purple | #8B5CF6 |
| Secondary accent | Cyan | #06B6D4 |
| Success | Green | #10B981 |
| Warning | Amber | #F59E0B |
| Error | Red | #EF4444 |
| Muted text | Gray | #6B7280 |
| Border | Purple dim | #4C1D95 |
| Highlight | Bright cyan | #22D3EE |

### Cockpit 1 — Overview (default)

Four quadrants: Services | Resources | Training | Portfolio.
Quick glance at entire system health.

### Cockpit 2 — Training

Active sessions table + GPU panel + CPU/RAM panel + Loss sparkline + Epoch financial metrics.
Designed for monitoring active training jobs.

### Cockpit 3 — Trading

Positions table + Recent executions + Account summary + Broker session health.
Designed for active trading operations.

### Cockpit 4 — Services

Service health grid (all 7-8 services) + K8s events + Cluster resources.
Designed for ops/SRE workflow.

### Cockpit 5 — Risk

Kill switch status + Drawdown monitor + Circuit breakers + Position limits + Stress test results.
Designed for risk management oversight.

### Cockpit 6 — Data

Active feeds + Data cache stats + Pipeline status.
Designed for data operations.

### Navigation

| Key | Action |
|-----|--------|
| 1-6 | Switch cockpit |
| q | Quit |
| ? | Help overlay |
| r | Force refresh |
| / | Filter (contextual) |

## MCP Server Mode

`fxt mcp serve` starts an MCP server on stdin/stdout (JSON-RPC 2.0).

Every CLI command maps to an MCP tool. Tool names: `fxt_{command}_{subcommand}`. All tools return structured JSON.

Example tools:
- `fxt_service_list` — List all services with health
- `fxt_service_status` — Detailed service status
- `fxt_train_start` — Start training job
- `fxt_trade_positions` — Get positions
- `fxt_risk_status` — Risk system state
- `fxt_cluster_resources` — CPU/RAM/GPU

This allows Claude Code (or any MCP client) to query the entire foxhunt system programmatically.

## Monitoring Service Removal

### What it does today
- Prometheus-to-gRPC bridge (97 lines main.rs)
- Scrapes Prometheus for training metrics, GPU stats, system resources
- Serves 3 RPCs: GetLiveTrainingMetrics, StreamTrainingMetrics, GetEpochHistory

### What replaces it
- API Gateway absorbs the Prometheus scraping logic
- The monitoring proto RPCs move to API Gateway's service layer
- fxt calls the same RPCs, just routed differently

### Cleanup
- Delete `services/monitoring_service/` directory
- Remove from Cargo.toml workspace members
- Remove nginx path routing for `/monitoring.MonitoringService/`
- Remove K8s deployment manifest
- Update CI pipeline (one fewer binary to compile)

## Metrics Inventory (available via API)

### Training (45+ fields per session)
epoch, loss, val_loss, sharpe, sortino, win_rate, max_drawdown, profit_factor, total_return, avg_return, total_trades, action distribution, q_value, policy_entropy, kl_divergence, gradient_norm, learning_rate, batches/s, GPU util/mem/temp/power, CPU%, RAM, checkpoint stats, NaN/explosion counts, hyperopt progress

### Trading
account (equity, cash, margin, P&L, buying power, leverage), positions (symbol, qty, entry, market value, unrealized P&L), executions (fill price/qty, cum qty, avg price, timestamps), session (FIX state, heartbeat RTT, sequence numbers)

### Risk
kill switch state (global/portfolio/strategy/instrument), drawdown (current%, max%, HWM, days), circuit breakers (daily loss, max DD, concentration), position limits (max position/order, leverage), emergency response, enforcement actions, recovery mode

### System
service health (per-service latency, restarts, uptime), cluster resources (node CPU/RAM/GPU/disk), K8s events, data feeds (records/s, latency, status)

### ML Models
model status, predictions, ensemble votes, feature importance, training job state, tuning progress, promotion pipeline
