- StreamSystemStatus now queries real Prometheus data (was all zeros)
- StreamAlerts returns honest unimplemented (was sending empty events)
- Training epoch shows "N" not "N/0" (max_epochs not in proto)
- GPU power shows "300W" not "300W / 0W" when limit unavailable
- Services cockpit shows error reason when service is DOWN
- System status polling detects staleness after 3 consecutive failures
- Training cockpit header shows active K8s job count
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All 30 MCP tool handlers now return structured JSON error with
isError=true instead of fake "stub: would ..." placeholder responses.
Tool registrations preserved so list_tools still works.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
config_service.proto (package foxhunt.config, 4 RPCs) was a subset of
config.proto (package config, 12 RPCs). Migrated api_gateway to implement
ConfigService from config.proto directly. Removed dead config_service()
method from fxt client.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a new gRPC stream spawner that subscribes to SubscribeClusterPods
and groups pods by service name for the pods cockpit view. Follows the
same backoff/reconnect pattern used by all other stream spawners.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add K8s pod data structures to TUI state layer for the upcoming pods
cockpit view. PodData represents individual pods, PodGroupData groups
them by service with ready/total counts. The StateUpdate::Pods variant
and apply_update handler wire the data path from gRPC streams to state.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Flip 4 positions in the base64 payload instead of 1 to eliminate
any chance of environmental flakiness from OsRng contention.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Auto-refresh: before non-auth commands, if the access token is expired
but a refresh token exists, silently regenerate tokens. Prints
"Token refreshed." to stderr so users know what happened.
- FXT_PASSWORD: login reads password from this env var when set,
skipping the interactive prompt. Safer than a --password flag
(env vars don't appear in ps output or shell history).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
FoxhuntClient now holds an AuthInterceptor<FileTokenStorage> and
all 11 typed service accessors use with_interceptor() instead of
bare Channel::new(). This ensures every outgoing RPC automatically
includes the JWT Bearer token when one is cached on disk.
The bare channel() accessor is preserved for LoginClient which
needs unauthenticated access for the initial login RPC.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds accessor methods for all 11 service clients (trading, ml, ml_training,
broker_gateway, trading_agent, monitoring, config, data_acquisition, risk,
health, config_service). All clients share a single gRPC channel.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Port Prometheus scraping and training metrics RPCs directly into
api_gateway, eliminating the monitoring-service backend proxy.
MonitoringServiceProxy replaced with MonitoringServiceHandler that
queries Prometheus directly for training metrics, GPU stats, and
active job counts.
Changes:
- monitoring_proxy.rs -> monitoring_handler.rs with embedded PrometheusClient
- Remove MonitoringBackendConfig and setup_monitoring_proxy from server.rs
- main.rs reads PROMETHEUS_URL + MONITORING_STREAM_INTERVAL instead of
MONITORING_SERVICE_URL
- Remove monitoring-service from backend health check list
- Monitoring is always available (no graceful degradation needed)
- Fix fxt build.rs compile_protos type mismatch
- All 9 ported unit tests + 0 clippy warnings
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fxt now compiles from the same service protos as all backend services.
Deleted 8 fat-client protos (3,286 lines) from bin/fxt/proto/.
Updated include_proto! macros to use new package names (trading, ml,
config instead of foxhunt.tli, foxhunt.ml, foxhunt.config).
Added risk, data_acquisition, and config_service proto modules.
Fixed duplicate include_proto in agent.rs to use crate::proto.
Existing command implementations will be rewritten to use new proto
types in Task 6.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add print_financial_metrics() to the monitor command, displaying
per-model Sharpe ratio (color-coded green/yellow/red), win rate,
max drawdown, profit factor, total return, trade count, and
action distribution (BUY/SELL/HOLD percentages). Only shown when
sessions report non-zero financial data.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add Sharpe/Win% columns to training list table, financial summary lines
(Sharpe, Sortino, Win Rate, Max DD, PF, Return, Avg, Trades) to the
detail overview, action distribution (BUY/SELL/HOLD %) to current metrics,
and four new sparklines (Sharpe, Win Rate, Max DD, Total Return) to the
Metrics sub-tab.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 11 financial fields (sharpe, sortino, win_rate, max_drawdown,
profit_factor, total_return, avg_return, total_trades, action
distribution) to TrainingSession (fields 36-46), a new
GetEpochHistory RPC with request/response messages, and wire the
Prometheus metric mapping in the monitoring service with a
stub RPC handler for task 7.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- trade_ml.rs: Replace 3 mock data fallbacks (submit, predictions,
performance) with proper error propagation. Commands now fail
honestly when the API Gateway is unreachable instead of silently
returning fake data. Mark 3 integration tests as #[ignore].
- monitoring_service: Add tonic-health with set_serving for
MonitoringServiceServer. Enables grpc_health_probe readiness checks.
- ml_training_service: Add tonic-health with set_serving for
MlTrainingServiceServer. Wired into both TLS and non-TLS paths.
- data_acquisition_service: Add tonic-health with set_serving for
DataAcquisitionServiceServer.
- ml/cuda_streams: Fix pre-existing unused variable clippy warning.
All 8 services now have standard gRPC health checking enabled.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The fxt client was using a hardcoded dev fallback secret for JWT signing,
causing InvalidSignature errors against the API gateway. Now reads
jwt_secret from ~/.foxhunt/config.toml with fallback chain:
env var > config file > dev secret.
Also updates default api_gateway_url to https://api.fxhnt.ai.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DashboardState with 4 tab substates (Training/Trading/Risk/System),
ring buffers for loss history and fills, scroll support.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- ratatui 0.29, crossterm 0.28 (event-stream), tokio-stream to deps
- ApproveModel + RejectModel RPCs added to MLTrainingService
- 4 new proto messages for model promotion workflow
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add DNS record, nginx gRPC proxy block, network policy for Tailscale
ingress, and auto-detect fxhnt.ai in fxt monitor URL derivation.
Show GPU telemetry even when no training sessions are active.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a new CLI subcommand that connects to the monitoring service via
gRPC to display live training metrics. Supports single-snapshot mode
(--once), model filtering (--model), and configurable streaming
interval. Renders per-model epoch/loss table, GPU telemetry, hyperopt
progress, and health counters.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- ibapi handshake: retry once after 1s on first failure (IB Gateway
needs time to release stale client_id slots after disconnection)
- test_load_nonexistent_config → test_load_config_succeeds: remove
default-value assertions that fail when ~/.foxhunt/config.toml exists
(defaults already tested by test_serde_defaults and test_default_config)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Match the actual K8s broker-gateway service port discovered during
live validation of fxt broker check.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The IB Gateway JVM uses non-blocking I/O on port 4002, which causes
EAGAIN errors with ibapi's blocking Client::connect(). The gnzsnz
image provides socat on port 4004 as a stable proxy that bridges
blocking clients to the non-blocking gateway socket.
Port 4004 is the correct client-facing port for paper mode (4003 for
live). The K8s readiness probe stays on 4002 since TCP probes work
fine with non-blocking sockets.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add Broker variant to Commands enum with BrokerArgs (flatten),
route to execute_broker_command in match block (no JWT required),
exit(1) on check failure. Clone config.api_gateway_url to avoid
partial move. Add two CLI parsing tests for broker check.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copy broker_gateway.proto from broker_gateway_service into fxt/proto/
and wire it into build.rs compile_protos + lib.rs proto module so the
CLI can call BrokerGatewayService RPCs (health check, account state,
session status).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds IbkrConfig (host/port/client_id with defaults for paper trading)
and BrokerConfig (gateway_url + ibkr nested) to the TLI config file.
Both structs derive serde defaults so existing config files remain
backward-compatible.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Create 3 missing service READMEs (api_gateway, data_acquisition_service,
trading_agent_service). Create bin/fxt/README.md. Create
testing/service-integration/README.md. Update existing service and testing
READMEs to standard template. Delete 4 subdirectory READMEs from
testing/integration/ and testing/e2e/.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove stale test reports, quick-start guides, benchmark analyses,
profiling reports, and tool artifacts from across the workspace.
Keeps only root README.md per crate/service.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add nginx server block for api.fxhnt.ai with grpc_pass to the
api-gateway ClusterIP service. Add Scaleway DNS A record via Terraform
module. Fix fxt gRPC client to enable TLS when connecting to https://
endpoints (tonic requires explicit ClientTlsConfig).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add --hyperopt, --trials, and --parallel flags to `fxt train start` so
users can dispatch PSO hyperparameter optimization jobs alongside
regular training. The mode=hyperopt tag propagates through the service
layer which selects the correct binary (hyperopt_baseline_rl for DQN/PPO,
hyperopt_baseline_supervised for all others) and builds hyperopt-specific
CLI arguments via a new extra_args field on TrainingJobParams.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extend ml_training_service to dispatch GPU training jobs as K8s batch/v1
Jobs, collect results via a Rust sidecar uploader, and support model
promotion with operator approval via fxt CLI.
- K8s dispatcher creates Jobs on gpu-training pool with native sidecar
- training_uploader crate: watches DONE/FAILED marker, uploads to S3,
reports completion via ReportJobCompletion gRPC
- PromotionManager compares metrics, queues better models for approval
- 4 new proto RPCs: ReportJobCompletion, ListPendingPromotions,
ApprovePromotion, RejectPromotion
- fxt commands: train start, model list/approve/reject
- Training binaries write DONE/FAILED markers + metrics.json
- Dockerfile, K8s job template, and CI pipeline updated
- StartTraining gracefully falls back to in-process when outside K8s
- 27 new tests (16 service + 11 promotion), 141 total service tests pass
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Delete 22 dead/placeholder/broken example files (-3,489 lines code)
- Delete 4 tracked CSV files (-1.1M lines, were accidentally committed)
- Move baseline training data default from data/cache/ to test_data/
- Update 5 unified binary defaults, gitignore, k8s upload comment, docs
- Consolidate all training data under test_data/futures-baseline/
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
test_machine_id_derivation reads /etc/machine-id which doesn't exist
in container environments.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
