00ef9e2866
Major Changes: - Migrated from 3-action TradingAction to 45-action FactoredAction - 45 actions: 5 exposure × 3 order types × 3 urgency levels - Absolute exposure model (target positions -1.0 to +1.0) - Transaction cost differentiation (Market 0.15%, LimitMaker 0.05%, IoC 0.10%) - Fixed action diversity threshold (1.11% → 0.5% for 45-action space) Bug Fixes: - Bug #15: Incomplete FactoredAction integration (code existed but unused) - Bug #16: Runtime crash in action diversity checking (hardcoded 3-action match) Code Changes (13 files, ~464 lines): - ml/src/dqn/action_space.rs: Core FactoredAction + 4 helper methods - ml/src/trainers/dqn.rs: Action diversity refactored (3→45 dynamic) - ml/src/dqn/reward.rs: calculate_reward() signature updated - ml/src/dqn/portfolio_tracker.rs: execute_action() absolute exposure - ml/src/dqn/dqn.rs: WorkingDQN action selection migrated - ml/tests/*.rs: 9 test files updated with FactoredAction assertions Test Results: - 1-epoch smoke test: 100% action diversity (45/45 actions, 80.2s) - 10-epoch production: 87.8% readiness (79/90 scorecard, 14.0 min) - Loss convergence: 96.9% reduction (119K → 3.6K) - Action diversity: 100% → 44% (healthy specialization) - Checkpoint reliability: 12/12 files saved (100%) - DQN tests: 195/195 passing (100%) - ML baseline: 1,514/1,515 passing (99.93%) Production Status: ✅ CERTIFIED (87.8% readiness) Go/No-Go: ✅ GO FOR 100-EPOCH PRODUCTION TRAINING 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
409 lines
17 KiB
YAML
409 lines
17 KiB
YAML
# Prometheus Alerting Rules for Broker Gateway Service
|
|
#
|
|
# These rules define critical alerts for FIX session health, order routing
|
|
# performance, error rates, and position management.
|
|
#
|
|
# Alert Severities:
|
|
# - CRITICAL: Immediate action required (paged)
|
|
# - WARNING: Attention needed (ticket created)
|
|
#
|
|
# Integration:
|
|
# - Alerts are sent to Alertmanager
|
|
# - CRITICAL alerts trigger PagerDuty
|
|
# - WARNING alerts create Jira tickets
|
|
|
|
groups:
|
|
- name: broker_gateway_fix_session
|
|
interval: 30s
|
|
rules:
|
|
# Alert 1: FIX session disconnected for >60s (CRITICAL)
|
|
- alert: BrokerGatewayFIXSessionDisconnected
|
|
expr: broker_gateway_fix_session_status == 0
|
|
for: 60s
|
|
labels:
|
|
severity: CRITICAL
|
|
service: broker_gateway_service
|
|
component: fix_session
|
|
impact: trading_halted
|
|
annotations:
|
|
summary: "FIX session {{ $labels.session_id }} is DISCONNECTED"
|
|
description: |
|
|
FIX session {{ $labels.session_id }} has been disconnected for more than 60 seconds.
|
|
|
|
**Impact**: All order routing is halted. No orders can be submitted to the broker.
|
|
|
|
**Action Required**:
|
|
1. Check FIX engine logs for connection errors
|
|
2. Verify network connectivity to AMP Futures gateway
|
|
3. Check firewall rules and VPN tunnel
|
|
4. Verify FIX credentials and session configuration
|
|
5. Attempt manual FIX session reconnect
|
|
|
|
**Current Status**: {{ $value }}
|
|
- 0 = DISCONNECTED
|
|
- 1 = CONNECTED
|
|
- 2 = RECONNECTING
|
|
runbook_url: "https://wiki.foxhunt.com/runbooks/fix_session_reconnect"
|
|
dashboard_url: "https://grafana.foxhunt.com/d/broker_gateway_dashboard"
|
|
|
|
# Alert 2: Order latency P95 > 100ms for 5m (WARNING)
|
|
- alert: BrokerGatewayHighOrderLatency
|
|
expr: |
|
|
histogram_quantile(0.95,
|
|
sum by(le, order_type) (rate(broker_gateway_order_latency_seconds_bucket[5m]))
|
|
) * 1000 > 100
|
|
for: 5m
|
|
labels:
|
|
severity: WARNING
|
|
service: broker_gateway_service
|
|
component: order_routing
|
|
impact: performance_degraded
|
|
annotations:
|
|
summary: "High order latency detected: P95 > 100ms for {{ $labels.order_type }}"
|
|
description: |
|
|
Order routing latency (P95) has exceeded 100ms for more than 5 minutes.
|
|
|
|
**Current P95 Latency**: {{ $value | humanizeDuration }}
|
|
**Order Type**: {{ $labels.order_type }}
|
|
**Target**: < 100ms
|
|
|
|
**Possible Causes**:
|
|
- Network congestion to broker gateway
|
|
- High FIX message processing load
|
|
- Database query slowdown
|
|
- Increased order volume
|
|
|
|
**Recommended Actions**:
|
|
1. Check FIX heartbeat RTT metric
|
|
2. Review database query performance
|
|
3. Verify CPU/memory usage on broker gateway service
|
|
4. Check for network packet loss
|
|
5. Review recent order volume trends
|
|
runbook_url: "https://wiki.foxhunt.com/runbooks/order_latency_investigation"
|
|
dashboard_url: "https://grafana.foxhunt.com/d/broker_gateway_dashboard"
|
|
|
|
# Alert 3: Error rate > 5% for 5m (WARNING)
|
|
- alert: BrokerGatewayHighErrorRate
|
|
expr: |
|
|
(
|
|
sum by(severity) (rate(broker_gateway_error_total[5m]))
|
|
/
|
|
(
|
|
sum(rate(broker_gateway_orders_submitted_total[5m]))
|
|
+ 0.001
|
|
)
|
|
) > 0.05
|
|
for: 5m
|
|
labels:
|
|
severity: WARNING
|
|
service: broker_gateway_service
|
|
component: error_handling
|
|
impact: reliability_degraded
|
|
annotations:
|
|
summary: "High error rate detected: {{ $value | humanizePercentage }} for {{ $labels.severity }}"
|
|
description: |
|
|
Error rate has exceeded 5% for more than 5 minutes.
|
|
|
|
**Current Error Rate**: {{ $value | humanizePercentage }}
|
|
**Error Severity**: {{ $labels.severity }}
|
|
**Target**: < 5%
|
|
|
|
**Impact**: Increased order rejections and failed operations
|
|
|
|
**Recommended Actions**:
|
|
1. Review error logs for common error types
|
|
2. Check broker_gateway_error_total metric by error_type
|
|
3. Verify database connectivity
|
|
4. Review FIX session stability
|
|
5. Check for invalid order parameters from trading agent
|
|
|
|
**Common Error Types**:
|
|
- DATABASE_ERROR: Check PostgreSQL connection pool
|
|
- FIX_ERROR: Review FIX session logs
|
|
- VALIDATION_ERROR: Check order parameter validation
|
|
- ORDER_REJECTED: Review broker rejection reasons
|
|
runbook_url: "https://wiki.foxhunt.com/runbooks/error_rate_investigation"
|
|
dashboard_url: "https://grafana.foxhunt.com/d/broker_gateway_dashboard"
|
|
|
|
# Alert 4: No orders in 10m during market hours (WARNING)
|
|
- alert: BrokerGatewayNoOrderActivity
|
|
expr: |
|
|
(time() - broker_gateway_last_order_time) > 600
|
|
and
|
|
(hour() >= 9 and hour() < 16)
|
|
for: 1m
|
|
labels:
|
|
severity: WARNING
|
|
service: broker_gateway_service
|
|
component: order_flow
|
|
impact: trading_inactive
|
|
annotations:
|
|
summary: "No order activity for 10+ minutes during market hours"
|
|
description: |
|
|
No orders have been submitted for more than 10 minutes during market hours.
|
|
|
|
**Last Order Timestamp**: {{ $value | humanizeTimestamp }}
|
|
**Time Since Last Order**: {{ with printf "broker_gateway_last_order_time" | query }}{{ . | first | value | humanizeDuration }}{{ end }}
|
|
|
|
**Possible Causes**:
|
|
- Trading Agent Service stopped sending orders
|
|
- All ML models producing HOLD signals
|
|
- Risk limits preventing order submission
|
|
- Market data feed issues
|
|
- Circuit breaker activated
|
|
|
|
**Recommended Actions**:
|
|
1. Check Trading Agent Service health
|
|
2. Review ML model predictions (check for all HOLD signals)
|
|
3. Verify risk limit settings
|
|
4. Check market data feed connectivity
|
|
5. Review circuit breaker status
|
|
6. Check gRPC connectivity between services
|
|
runbook_url: "https://wiki.foxhunt.com/runbooks/no_order_activity"
|
|
dashboard_url: "https://grafana.foxhunt.com/d/broker_gateway_dashboard"
|
|
|
|
# Alert 5: Position mismatch > $10K (CRITICAL)
|
|
- alert: BrokerGatewayPositionMismatch
|
|
expr: |
|
|
abs(
|
|
broker_gateway_position_value_usd
|
|
-
|
|
on(symbol, account_id) group_left()
|
|
trading_service_position_value_usd
|
|
) > 10000
|
|
for: 2m
|
|
labels:
|
|
severity: CRITICAL
|
|
service: broker_gateway_service
|
|
component: position_management
|
|
impact: data_integrity
|
|
annotations:
|
|
summary: "Position mismatch detected: {{ $labels.symbol }} ({{ $labels.account_id }})"
|
|
description: |
|
|
Position value mismatch detected between Broker Gateway and Trading Service.
|
|
|
|
**Symbol**: {{ $labels.symbol }}
|
|
**Account**: {{ $labels.account_id }}
|
|
**Mismatch Amount**: ${{ $value | humanize }}
|
|
**Threshold**: $10,000
|
|
|
|
**Impact**: Position tracking inaccuracy can lead to:
|
|
- Incorrect risk calculations
|
|
- Over-leveraged positions
|
|
- PnL reporting errors
|
|
- Regulatory compliance issues
|
|
|
|
**IMMEDIATE ACTIONS REQUIRED**:
|
|
1. HALT automated trading on affected symbol/account
|
|
2. Reconcile position with broker's actual position
|
|
3. Review recent execution reports for missing fills
|
|
4. Check for FIX message sequence gaps
|
|
5. Verify database transaction integrity
|
|
6. Manual position adjustment if necessary
|
|
|
|
**Data Sources**:
|
|
- Broker Gateway Position: {{ with printf "broker_gateway_position_value_usd{symbol=\"%s\",account_id=\"%s\"}" $labels.symbol $labels.account_id | query }}{{ . | first | value | humanize }}{{ end }}
|
|
- Trading Service Position: {{ with printf "trading_service_position_value_usd{symbol=\"%s\",account_id=\"%s\"}" $labels.symbol $labels.account_id | query }}{{ . | first | value | humanize }}{{ end }}
|
|
runbook_url: "https://wiki.foxhunt.com/runbooks/position_reconciliation"
|
|
dashboard_url: "https://grafana.foxhunt.com/d/broker_gateway_dashboard"
|
|
pagerduty_severity: "critical"
|
|
|
|
# Alert 6: Sequence number gap detected (WARNING)
|
|
- alert: BrokerGatewayFIXSequenceGap
|
|
expr: |
|
|
rate(broker_gateway_sequence_number_gap_total[5m]) > 0
|
|
for: 1m
|
|
labels:
|
|
severity: WARNING
|
|
service: broker_gateway_service
|
|
component: fix_session
|
|
impact: message_loss
|
|
annotations:
|
|
summary: "FIX sequence number gap detected on {{ $labels.session_id }}"
|
|
description: |
|
|
FIX sequence number gap detected, indicating potential message loss.
|
|
|
|
**FIX Session**: {{ $labels.session_id }}
|
|
**Gap Rate**: {{ $value | humanize }} gaps/sec
|
|
|
|
**Impact**: Sequence gaps can cause:
|
|
- Lost execution reports
|
|
- Missed order acknowledgments
|
|
- Position tracking errors
|
|
- Delayed order status updates
|
|
|
|
**Possible Causes**:
|
|
- Network packet loss
|
|
- FIX gateway message drop
|
|
- Session disconnect/reconnect
|
|
- High message volume overload
|
|
|
|
**Recommended Actions**:
|
|
1. Check FIX session logs for Resend Request messages
|
|
2. Verify network stability (check packet loss metrics)
|
|
3. Review FIX heartbeat RTT metric
|
|
4. Check broker_gateway_fix_sender_seq_num and broker_gateway_fix_target_seq_num
|
|
5. Initiate FIX message recovery if necessary
|
|
6. Monitor for position reconciliation issues
|
|
|
|
**Current Sequence Numbers**:
|
|
- Sender SeqNum: {{ with printf "broker_gateway_fix_sender_seq_num{session_id=\"%s\"}" $labels.session_id | query }}{{ . | first | value }}{{ end }}
|
|
- Target SeqNum: {{ with printf "broker_gateway_fix_target_seq_num{session_id=\"%s\"}" $labels.session_id | query }}{{ . | first | value }}{{ end }}
|
|
runbook_url: "https://wiki.foxhunt.com/runbooks/fix_sequence_gap_recovery"
|
|
dashboard_url: "https://grafana.foxhunt.com/d/broker_gateway_dashboard"
|
|
|
|
- name: broker_gateway_performance
|
|
interval: 60s
|
|
rules:
|
|
# Alert 7: High FIX heartbeat RTT (WARNING)
|
|
- alert: BrokerGatewayHighFIXHeartbeatRTT
|
|
expr: broker_gateway_fix_heartbeat_rtt_ms > 50
|
|
for: 5m
|
|
labels:
|
|
severity: WARNING
|
|
service: broker_gateway_service
|
|
component: fix_session
|
|
impact: network_latency
|
|
annotations:
|
|
summary: "High FIX heartbeat RTT detected: {{ $value }}ms on {{ $labels.session_id }}"
|
|
description: |
|
|
FIX heartbeat round-trip time (RTT) has exceeded 50ms for more than 5 minutes.
|
|
|
|
**Current RTT**: {{ $value }}ms
|
|
**FIX Session**: {{ $labels.session_id }}
|
|
**Target**: < 50ms
|
|
|
|
**Impact**: High RTT indicates network latency which can:
|
|
- Delay order execution
|
|
- Slow down position updates
|
|
- Increase order routing latency
|
|
|
|
**Recommended Actions**:
|
|
1. Check network path to broker gateway (traceroute)
|
|
2. Verify VPN tunnel stability
|
|
3. Review network congestion metrics
|
|
4. Check for packet loss
|
|
5. Contact network operations if persistent
|
|
runbook_url: "https://wiki.foxhunt.com/runbooks/network_latency_investigation"
|
|
dashboard_url: "https://grafana.foxhunt.com/d/broker_gateway_dashboard"
|
|
|
|
# Alert 8: High order rejection rate (WARNING)
|
|
- alert: BrokerGatewayHighOrderRejectionRate
|
|
expr: |
|
|
(
|
|
sum by(symbol) (rate(broker_gateway_orders_rejected_total[5m]))
|
|
/
|
|
(
|
|
sum by(symbol) (rate(broker_gateway_orders_submitted_total[5m]))
|
|
+ 0.001
|
|
)
|
|
) > 0.10
|
|
for: 5m
|
|
labels:
|
|
severity: WARNING
|
|
service: broker_gateway_service
|
|
component: order_validation
|
|
impact: order_flow_disrupted
|
|
annotations:
|
|
summary: "High order rejection rate: {{ $value | humanizePercentage }} for {{ $labels.symbol }}"
|
|
description: |
|
|
Order rejection rate has exceeded 10% for more than 5 minutes.
|
|
|
|
**Symbol**: {{ $labels.symbol }}
|
|
**Rejection Rate**: {{ $value | humanizePercentage }}
|
|
**Target**: < 10%
|
|
|
|
**Impact**: High rejection rate reduces trading efficiency and profitability
|
|
|
|
**Common Rejection Reasons**:
|
|
- RISK_LIMIT: Position or loss limits exceeded
|
|
- INSUFFICIENT_MARGIN: Not enough margin for order
|
|
- INVALID_PRICE: Limit price outside allowed range
|
|
- MARKET_CLOSED: Orders submitted outside trading hours
|
|
|
|
**Recommended Actions**:
|
|
1. Review rejection reasons in broker_gateway_orders_rejected_total
|
|
2. Check risk limit settings
|
|
3. Verify account margin availability
|
|
4. Review ML model price predictions
|
|
5. Validate order parameter generation logic
|
|
runbook_url: "https://wiki.foxhunt.com/runbooks/order_rejection_investigation"
|
|
dashboard_url: "https://grafana.foxhunt.com/d/broker_gateway_dashboard"
|
|
|
|
# Alert 9: Database query slow (WARNING)
|
|
- alert: BrokerGatewaySlowDatabaseQueries
|
|
expr: |
|
|
histogram_quantile(0.95,
|
|
sum by(le) (rate(database_query_duration_seconds_bucket{service="broker_gateway_service"}[5m]))
|
|
) > 0.100
|
|
for: 5m
|
|
labels:
|
|
severity: WARNING
|
|
service: broker_gateway_service
|
|
component: database
|
|
impact: performance_degraded
|
|
annotations:
|
|
summary: "Slow database queries detected: P95 > 100ms"
|
|
description: |
|
|
Database query latency (P95) has exceeded 100ms for more than 5 minutes.
|
|
|
|
**Current P95 Latency**: {{ $value | humanizeDuration }}
|
|
**Target**: < 100ms
|
|
|
|
**Impact**: Slow queries can:
|
|
- Increase order routing latency
|
|
- Delay position updates
|
|
- Cause gRPC request timeouts
|
|
|
|
**Recommended Actions**:
|
|
1. Check PostgreSQL slow query log
|
|
2. Review query execution plans
|
|
3. Check database connection pool utilization
|
|
4. Verify database server CPU/memory/disk I/O
|
|
5. Consider adding indexes on frequently queried columns
|
|
6. Review recent database schema changes
|
|
runbook_url: "https://wiki.foxhunt.com/runbooks/database_performance_tuning"
|
|
dashboard_url: "https://grafana.foxhunt.com/d/broker_gateway_dashboard"
|
|
|
|
# Alert 10: High margin usage (WARNING)
|
|
- alert: BrokerGatewayHighMarginUsage
|
|
expr: |
|
|
(
|
|
broker_gateway_margin_used_usd
|
|
/
|
|
(broker_gateway_cash_balance_usd + 0.001)
|
|
) > 0.80
|
|
for: 5m
|
|
labels:
|
|
severity: WARNING
|
|
service: broker_gateway_service
|
|
component: risk_management
|
|
impact: margin_pressure
|
|
annotations:
|
|
summary: "High margin usage: {{ $value | humanizePercentage }} for {{ $labels.account_id }}"
|
|
description: |
|
|
Margin usage has exceeded 80% for more than 5 minutes.
|
|
|
|
**Account**: {{ $labels.account_id }}
|
|
**Margin Usage**: {{ $value | humanizePercentage }}
|
|
**Threshold**: 80%
|
|
|
|
**Current Status**:
|
|
- Margin Used: ${{ with printf "broker_gateway_margin_used_usd{account_id=\"%s\"}" $labels.account_id | query }}{{ . | first | value | humanize }}{{ end }}
|
|
- Cash Balance: ${{ with printf "broker_gateway_cash_balance_usd{account_id=\"%s\"}" $labels.account_id | query }}{{ . | first | value | humanize }}{{ end }}
|
|
|
|
**Impact**: High margin usage increases risk of:
|
|
- Margin calls
|
|
- Forced position liquidation
|
|
- Inability to open new positions
|
|
|
|
**Recommended Actions**:
|
|
1. Review open positions and unrealized PnL
|
|
2. Consider reducing position sizes
|
|
3. Close losing positions to free up margin
|
|
4. Increase account cash balance if necessary
|
|
5. Review risk limit settings
|
|
6. Notify risk management team
|
|
runbook_url: "https://wiki.foxhunt.com/runbooks/margin_management"
|
|
dashboard_url: "https://grafana.foxhunt.com/d/broker_gateway_dashboard"
|