Commit Graph

9 Commits

Author SHA1 Message Date
jgrusewski
5842859d78 feat(monitoring): wire epoch financial metrics mapper + epoch history store
- Fix action distribution metric names (add epoch_ prefix)
- Implement epoch history ring buffer (50 epochs per session)
- Wire GetEpochHistory RPC with real data
- Add test for financial metric mapping

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 16:01:04 +01:00
jgrusewski
a471c913d7 feat(proto): add epoch financial metrics + GetEpochHistory to monitoring.proto
Add 11 financial fields (sharpe, sortino, win_rate, max_drawdown,
profit_factor, total_return, avg_return, total_trades, action
distribution) to TrainingSession (fields 36-46), a new
GetEpochHistory RPC with request/response messages, and wire the
Prometheus metric mapping in the monitoring service with a
stub RPC handler for task 7.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 15:57:27 +01:00
jgrusewski
d04b6c7023 fix(fxt,services): remove mock fallbacks and add gRPC health checks
- trade_ml.rs: Replace 3 mock data fallbacks (submit, predictions,
  performance) with proper error propagation. Commands now fail
  honestly when the API Gateway is unreachable instead of silently
  returning fake data. Mark 3 integration tests as #[ignore].

- monitoring_service: Add tonic-health with set_serving for
  MonitoringServiceServer. Enables grpc_health_probe readiness checks.

- ml_training_service: Add tonic-health with set_serving for
  MlTrainingServiceServer. Wired into both TLS and non-TLS paths.

- data_acquisition_service: Add tonic-health with set_serving for
  DataAcquisitionServiceServer.

- ml/cuda_streams: Fix pre-existing unused variable clippy warning.

All 8 services now have standard gRPC health checking enabled.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 14:14:51 +01:00
jgrusewski
ab218121b6 feat(monitoring): wire 12 new metric→proto field mappings in group_into_sessions
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 00:09:54 +01:00
jgrusewski
7983302a52 proto(monitoring): add 12 new TrainingSession fields for diagnostics
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 00:00:36 +01:00
jgrusewski
2f97402d5d fix(monitoring): use active_workers metric instead of kube_job query
CI training jobs run as GitLab runner pods, not K8s Jobs, so the
kube_job_status_active query always returned 0. Switch to
foxhunt_training_active_workers which is emitted by all training
binaries regardless of deployment method.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 23:21:15 +01:00
jgrusewski
85725abd8f feat(monitoring): per-epoch Prometheus metrics + CI scrape plumbing
- DQN/PPO trainers now record epoch, loss, val_loss, batch/s every epoch
- CI training jobs get Prometheus scrape annotations via runner overrides
- Allow Prometheus (foxhunt namespace) to scrape foxhunt-ci pods
- Skip no-model metrics in monitoring service session grouping

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 23:10:52 +01:00
jgrusewski
9e20337ee4 feat(monitoring): add monitoring_service crate with Prometheus-to-gRPC bridge
Prometheus client queries training/hyperopt/GPU/K8s metrics, gRPC service
exposes unary + server-streaming RPCs, 4 unit tests, zero clippy warnings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 21:35:38 +01:00
jgrusewski
33d3cf2dd2 feat(monitoring): add monitoring.proto with gRPC service definition
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 21:21:29 +01:00